I’ll continue one argument I was having with Jon in the open, just to see what happens. The premise: Open collaboration can relentlessly commoditize things that require a lot of work — like deep integration — just because it can mobilize the resources.

My response: In my experience, that hasn’t been the case. In fact, the “open collaboration” model tends to leave things at the “good enough” point too often. Look at most open source software efforts: very few large-scale projects get beyond looking like bunches of different applications created by different people. Nearly none actually demonstrate things like common user experience paradigms, consistent APIs, or uniform management.

John goes on to describe four types of open source projects:

• Cloners – so many of these.
• Standards built – such as apache or mozilla.
• Competitive Devaluation – closed source projects that become open source to gather a larger audience.
• Invention – “Not quite like anything that came before.”

I think its worth noting that of these four kinds of projects, the top three share the characteristic of having well defined requirements. These are the projects where parallel collaborative development can excel.

The fourth kind of project, Invention, is a different beast entirely. I’ll argue that the open source projects in this category that end up actually succeeding are the ones run by a single benevolent dictator, or a small tight team. Thats because large groups are not good at setting cohesive goals. (ever watch CSPAN?) A strong goal and a strong vision, more than any other factor, determines the success of a software project.

Again and again I see various forums, mailing lists and news groups spawn open source projects, but if they don’t fit into the clone this or implement this standard variety, they usually die away. Thats because they lack a clear vision and a participants lack a willingness to commit resources to achieve that vision.

Communicating a vision such that people are willing to invest resources in realizing that vision is hard. The coin of the realm in open source is, of course, source code. The best communicator of a vision is code that solves a problem or provides value. Without that seed of working, valuable code, most open source projects never move beyond their creator.

Innovative open source projects face the same problem that a small business faces trying to get a a loan from the bank. The more you need the money, the less likely the bank is to give it to you. Similarly, potential contributers to an innovative open source project need to see that the goal of the project offers them value and that the project already has enough resources committed to potentially achieve that goal. Only then are they willing to commit their own resources to the stone soup.

Given the difficulty of communicating a common vision, I think its no surprise that “common user experience paradigms, consistent APIs, or uniform management” are not necessarily natural strong points of open source software.

Agile software development methods including extreme programming (XP) have been growing in popularity and maturity alongside an increase in the popularity of open source projects. One of the fundamental principles of XP is early and often feedback. The value of feedback is obvious to anyone who has used a strange shower. The shorter the time between turning the knob and feeling the temperature change, the faster you can arrive at the perfect temperature. Nobody likes flailing between too hot and too cold in a lazy shower. In the same way, immediate feedback on a development project can avoid wasted effort and produce a better result faster.

In XP and other Agile programming methods, an automated test suite provides important early and often feedback. It is far better to catch a bug in the test suite than for the customer to find it. But, how do you know if your tests are testing all of the features of your program?

The XP practice of test driven development (TDD) demands that you only add features for which you have a failing test case. Therefore, you write the test first, and then write the code to satisfy it. For a theoretical perspective, this should give your code 100% code coverage. Every line of code should be covered by a test. Unfortunately, developers are human and any technique that relies of the perfection of people is doomed to fail. Unfortunately, the converse of the TDD coverage assertion is simply not helpful: If you don’t have 100% test coverage, you aren’t correctly doing TDD. If you are not doing TDD, how do you gauge the quality of your test suite?

Open source projects present a special challenge for the concept of TDD. No project wants to reject useful contributions, yet many contributors may not write tests for their contributions. Even if tests are required, there is no way to tell if they were written first, as in TDD, or after the fact. XP would have the code author writing the unit tests. In an open source project, this might not be possible or likely, although still desirable.

While certainly not perfect, code coverage tools do provide an indicator of how well tested a section of code is. Code coverage reports are a feedback loop to help test writers find weak areas and to better target tests to specific lines of code. Just picking a file at random from PHP reveals low-hanging testing fruit: zend_operators.c.

I feel that a coverage report can be an important tool in an open source project. More so than in other kinds of projects. Test writing based on a coverage report inherently lends itself to the distributed nature of open source development. Many people have a stake in the stability, robustness and reputation of PHP. Tests are easy enough to write, easier perhaps than many other ways to contribute. Hopefully easily available coverage reporting would help coordinate testing activities and draw people in that might not otherwise participate.

Ideally, contributions would be accompanied by test code contributed by the same author. I also don’t think it is out of line to look at untested contributions with increased skepticism. Coverage reports can help identify contributions that might cause future problems. One can hope the existence of coverage reports might inspire contributors to include better tests by exposing their code as untested. My (unverified) theory and hope is that the feedback of coverage reports can help inspire a culture of testing on an open source project.

While there are areas (such as GUI apps) where unit testing is difficult at best (thread safety may be another), PHP overall is very testable. The stability and reputation of PHP can only be improved by increasing the coverage of its unit tests.

I am looking forward to seeing the availability of an automated http://cov.php.net/, if indeed that is planned.