Professional PHP

PHP Programming, Web Development, PHP Advocacy and PHP Best Practices.
« Backward compatibilty and web host adoption of PHP 5
Zend PHP Framework »

Code Coverage, Feedback and Open Source

October 4th, 2005

I’ve long wondered about the quality and extent of the PHP test suite. During my posting hiatus, John Coggeshall addressed my question by posting a coverage report generated by running the test suite. Unless I am mistaken, he also implies that the report will at some point become automated and available at http://cov.php.net/. I see the availability of this report as a big step forward for PHPs future stability and maturity.

Agile software development methods including extreme programming (XP) have been growing in popularity and maturity alongside an increase in the popularity of open source projects. One of the fundamental principles of XP is early and often feedback. The value of feedback is obvious to anyone who has used a strange shower. The shorter the time between turning the knob and feeling the temperature change, the faster you can arrive at the perfect temperature. Nobody likes flailing between too hot and too cold in a lazy shower. In the same way, immediate feedback on a development project can avoid wasted effort and produce a better result faster.

In XP and other Agile programming methods, an automated test suite provides important early and often feedback. It is far better to catch a bug in the test suite than for the customer to find it. But, how do you know if your tests are testing all of the features of your program?

The XP practice of test driven development (TDD) demands that you only add features for which you have a failing test case. Therefore, you write the test first, and then write the code to satisfy it. For a theoretical perspective, this should give your code 100% code coverage. Every line of code should be covered by a test. Unfortunately, developers are human and any technique that relies of the perfection of people is doomed to fail. Unfortunately, the converse of the TDD coverage assertion is simply not helpful: If you don’t have 100% test coverage, you aren’t correctly doing TDD. If you are not doing TDD, how do you gauge the quality of your test suite?

Open source projects present a special challenge for the concept of TDD. No project wants to reject useful contributions, yet many contributors may not write tests for their contributions. Even if tests are required, there is no way to tell if they were written first, as in TDD, or after the fact. XP would have the code author writing the unit tests. In an open source project, this might not be possible or likely, although still desirable.

While certainly not perfect, code coverage tools do provide an indicator of how well tested a section of code is. Code coverage reports are a feedback loop to help test writers find weak areas and to better target tests to specific lines of code. Just picking a file at random from PHP reveals low-hanging testing fruit: zend_operators.c.

I feel that a coverage report can be an important tool in an open source project. More so than in other kinds of projects. Test writing based on a coverage report inherently lends itself to the distributed nature of open source development. Many people have a stake in the stability, robustness and reputation of PHP. Tests are easy enough to write, easier perhaps than many other ways to contribute. Hopefully easily available coverage reporting would help coordinate testing activities and draw people in that might not otherwise participate.

Ideally, contributions would be accompanied by test code contributed by the same author. I also don’t think it is out of line to look at untested contributions with increased skepticism. Coverage reports can help identify contributions that might cause future problems. One can hope the existence of coverage reports might inspire contributors to include better tests by exposing their code as untested. My (unverified) theory and hope is that the feedback of coverage reports can help inspire a culture of testing on an open source project.

While there are areas (such as GUI apps) where unit testing is difficult at best (thread safety may be another), PHP overall is very testable. The stability and reputation of PHP can only be improved by increasing the coverage of its unit tests.

I am looking forward to seeing the availability of an automated http://cov.php.net/, if indeed that is planned.

Filed Under

  • Agile Methods, PHP, Software Design

Related Posts

  • php testing and coverage
  • PDO Design Evolution
  • ZendCon: Writing Maintainable PHP Code
  • Zend Framework Webcast
  • The PHP scalability saga continues
You can leave a response, or trackback from your own site.

One Response to “Code Coverage, Feedback and Open Source”

  1. mayur says:
    3/9/2008 at 7:42 pm

    it is better

Leave a Reply

Click here to cancel reply.

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

code: use [code=php][/code].

Comment Preview

    Subscribe Feed
    Share Subscribe to this blog…
    Share Bookmark or share this page…
  • About

    My name is Jeff Moore. I'm a PHP programmer living in San Francico and working for a startup.

    More about me…

  • Categories (Home)

    • Agile Methods (14)
    • Mac (14)
    • Misc (17)
    • Open Source (14)
    • PHP (98)
    • Software Design (29)
    • Usability (14)
    • Web Design (20)
  • Recent Comments

    • Benchmarking PHP's Magic Methods  12
      Article Marketing Strategy, Magento Development, Vance Lucas [...]
    • Improved Error Messages in PHP 5  32
      Terry Nessel, css, Amsterdam [...]
    • Keywords and Language Simplicity  11
      kim kardashian sex tape price, nokia, per Rechnung bestellen [...]
    • flickr case study  3
      bookmarking demon software, php-trivandrum.org, Harry Fuecks
    • rsync to remote server via ssh  7
      Burton Haynes, James, Mike [...]
    • Yahoo YUI wins JavaScript Library Wars  10
      Lera Bride, Scott, Jeff [...]
    • OOP is Mature, not Dead  15
      Avery Depew, Fernando, deltawing [...]
    • Ruby versus PHP or There and Back Again  10
      Solar Pumps, Amsterdam, Hari K T [...]
    • Looking Towards the Cloud  22
      Driver License, Jamel Sawyer, enculez. [...]
    • ZendCon: Writing Maintainable PHP Code  8
      IT Ninja, nicopico, Arif [...]
  • Recent Posts

    • ZendCon: Writing Maintainable PHP Code
    • Looking Towards the Cloud
    • Holiday Tech Support
    • Closures are coming to PHP
    • php | tek Wrapup
    • php | tek 2008
    • Sarah Snow Stever
    • Benchmarking PHP’s Magic Methods
    • The Endpoints of the Scale of Stupidity on Video
    • Working with PHP 5 in Mac OS X 10.5 (Leopard)
  • Site

    • Archives
    • Log in
  • Search