I’ve never been a PEAR fan. My experiences being peripherally involved with the XML_HTMLSax package weren’t encouraging. However, my opinion of PEAR has turned around in recent years thanks to work such as channels in the pear installer and MDB2.

I hope this doesn’t end up a fork. I see little sense in that. I’ll be interested in seeing the PEAR response.

Agile software development methods including extreme programming (XP) have been growing in popularity and maturity alongside an increase in the popularity of open source projects. One of the fundamental principles of XP is early and often feedback. The value of feedback is obvious to anyone who has used a strange shower. The shorter the time between turning the knob and feeling the temperature change, the faster you can arrive at the perfect temperature. Nobody likes flailing between too hot and too cold in a lazy shower. In the same way, immediate feedback on a development project can avoid wasted effort and produce a better result faster.

In XP and other Agile programming methods, an automated test suite provides important early and often feedback. It is far better to catch a bug in the test suite than for the customer to find it. But, how do you know if your tests are testing all of the features of your program?

The XP practice of test driven development (TDD) demands that you only add features for which you have a failing test case. Therefore, you write the test first, and then write the code to satisfy it. For a theoretical perspective, this should give your code 100% code coverage. Every line of code should be covered by a test. Unfortunately, developers are human and any technique that relies of the perfection of people is doomed to fail. Unfortunately, the converse of the TDD coverage assertion is simply not helpful: If you don’t have 100% test coverage, you aren’t correctly doing TDD. If you are not doing TDD, how do you gauge the quality of your test suite?

Open source projects present a special challenge for the concept of TDD. No project wants to reject useful contributions, yet many contributors may not write tests for their contributions. Even if tests are required, there is no way to tell if they were written first, as in TDD, or after the fact. XP would have the code author writing the unit tests. In an open source project, this might not be possible or likely, although still desirable.

While certainly not perfect, code coverage tools do provide an indicator of how well tested a section of code is. Code coverage reports are a feedback loop to help test writers find weak areas and to better target tests to specific lines of code. Just picking a file at random from PHP reveals low-hanging testing fruit: zend_operators.c.

I feel that a coverage report can be an important tool in an open source project. More so than in other kinds of projects. Test writing based on a coverage report inherently lends itself to the distributed nature of open source development. Many people have a stake in the stability, robustness and reputation of PHP. Tests are easy enough to write, easier perhaps than many other ways to contribute. Hopefully easily available coverage reporting would help coordinate testing activities and draw people in that might not otherwise participate.

Ideally, contributions would be accompanied by test code contributed by the same author. I also don’t think it is out of line to look at untested contributions with increased skepticism. Coverage reports can help identify contributions that might cause future problems. One can hope the existence of coverage reports might inspire contributors to include better tests by exposing their code as untested. My (unverified) theory and hope is that the feedback of coverage reports can help inspire a culture of testing on an open source project.

While there are areas (such as GUI apps) where unit testing is difficult at best (thread safety may be another), PHP overall is very testable. The stability and reputation of PHP can only be improved by increasing the coverage of its unit tests.

I am looking forward to seeing the availability of an automated http://cov.php.net/, if indeed that is planned.

They have released a coverage reporting tool, Spike PHP Coverage, for PHP that works with XDebug coverage data. It works with Simple Test and it seems to be able aggregate the results of remote test runs, such as for web based tests. I have wanted something exactly like this. I can’t wait to get the chance to generate a consolidated coverage report for WACT.

It seems that they have the capability for generating and aggregating code coverage reports for PHP itself. I haven’t had a chance to check this out, but its something that I have always wondered about.

Additionally, it looks like they accept contributions of Test Suites. If I understand this correctly, it means that I can upload the WACT test suite and that it can be used to test and generate coverage reports for PHP 4 and for PHP 5 as well as the framework. If this is true, this is a big deal. As there are more and more applications with automated test suites, it only makes sense to aggregate them to test PHP itself.

It also seems that they have embraced Simple Test for php testing. Congratulations, Marcus. Simple Test is a fine piece of software.

I think code coverage measurements are important for gauging the quality of a test suite on open source projects, where there can be less formal development practices and a large number of contributers of varying skill levels and motivation.