Professional PHP

I’ve long wondered about the quality and extent of the PHP test suite. During my posting hiatus, John Coggeshall addressed my question by posting a coverage report generated by running the test suite. Unless I am mistaken, he also implies that the report will at some point become automated and available at http://cov.php.net/. I see the availability of this report as a big step forward for PHPs future stability and maturity.

Agile software development methods including extreme programming (XP) have been growing in popularity and maturity alongside an increase in the popularity of open source projects. One of the fundamental principles of XP is early and often feedback. The value of feedback is obvious to anyone who has used a strange shower. The shorter the time between turning the knob and feeling the temperature change, the faster you can arrive at the perfect temperature. Nobody likes flailing between too hot and too cold in a lazy shower. In the same way, immediate feedback on a development project can avoid wasted effort and produce a better result faster.

In XP and other Agile programming methods, an automated test suite provides important early and often feedback. It is far better to catch a bug in the test suite than for the customer to find it. But, how do you know if your tests are testing all of the features of your program?

The XP practice of test driven development (TDD) demands that you only add features for which you have a failing test case. Therefore, you write the test first, and then write the code to satisfy it. For a theoretical perspective, this should give your code 100% code coverage. Every line of code should be covered by a test. Unfortunately, developers are human and any technique that relies of the perfection of people is doomed to fail. Unfortunately, the converse of the TDD coverage assertion is simply not helpful: If you don’t have 100% test coverage, you aren’t correctly doing TDD. If you are not doing TDD, how do you gauge the quality of your test suite?

Open source projects present a special challenge for the concept of TDD. No project wants to reject useful contributions, yet many contributors may not write tests for their contributions. Even if tests are required, there is no way to tell if they were written first, as in TDD, or after the fact. XP would have the code author writing the unit tests. In an open source project, this might not be possible or likely, although still desirable.

While certainly not perfect, code coverage tools do provide an indicator of how well tested a section of code is. Code coverage reports are a feedback loop to help test writers find weak areas and to better target tests to specific lines of code. Just picking a file at random from PHP reveals low-hanging testing fruit: zend_operators.c.

I feel that a coverage report can be an important tool in an open source project. More so than in other kinds of projects. Test writing based on a coverage report inherently lends itself to the distributed nature of open source development. Many people have a stake in the stability, robustness and reputation of PHP. Tests are easy enough to write, easier perhaps than many other ways to contribute. Hopefully easily available coverage reporting would help coordinate testing activities and draw people in that might not otherwise participate.

Ideally, contributions would be accompanied by test code contributed by the same author. I also don’t think it is out of line to look at untested contributions with increased skepticism. Coverage reports can help identify contributions that might cause future problems. One can hope the existence of coverage reports might inspire contributors to include better tests by exposing their code as untested. My (unverified) theory and hope is that the feedback of coverage reports can help inspire a culture of testing on an open source project.

While there are areas (such as GUI apps) where unit testing is difficult at best (thread safety may be another), PHP overall is very testable. The stability and reputation of PHP can only be improved by increasing the coverage of its unit tests.

I am looking forward to seeing the availability of an automated http://cov.php.net/, if indeed that is planned.