os-test: Measuring POSIX compliance on every single OS

What happens if you run tests on every POSIX system? You find a lot of bugs in every single OS. I parsed the new POSIX.1-2024 standard into API definitions, generated tests, and measured exactly how much of the standard is implemented. I invoked every libc function to see if they work, and began writing detailed test suites. As it turns out, if there's a sentence in POSIX, someone probably implemented it incorrectly. I found missing interfaces, incompatible declarations, namespace pollution, a lot of bugs, interesting benign differences, and many more issues. I published all tests and results as os-test. The volume of test failures makes it virtually impossible to report all individual issues to each upstream. However as all of the data is publicly available online, vendors are now beginning to incorporate os-test feedback into their development and testing process, which ultimately leads to improved POSIX compliance and software interoperability.

In this talk we will dive into the challenges of testing 16 different operating systems, survey the main findings of os-test, and finally determine which operating system takes the lead when it comes to POSIX conformance.

https://sortix.org/os-test/ https://sortix.org/blog/os-testing-posix-headers/

os-test is currently funded by Next Generation Internet Zero Commons.