Cognitive science research on problem solving tries to examine the difference between experts and novices in a domain. Rather than distinguish between developers and architects, I think it is better to distinguish between experts and novices at programming. The Dreyfus model of skill acquisition details five skill levels to help in this task. Here is a summary from Coding Horror:

Level 1: Beginner

• Little or no previous experience
• Doesn’t want to learn: wants to accomplish a goal
• No discretionary judgement
• Rigid adherence to rules

Level 2: Advanced Beginner

• Starts trying tasks on their own
• Has difficulty troubleshooting
• Wants information fast
• Can place some advice in context required
• Uses guidelines, but without holisitic understanding

Level 3: Competent

• Develops conceptual models
• Troubleshoots on their own
• Seeks out expert advice
• Sees actions at least partially in terms of long-term plans and goals

Level 4: Proficient

• Guided by maxims applied to the current situation
• Sees situations holistically
• Will self-correct based on previous performance
• Learns from the experience of others
• Frustrated by oversimplified information

Level 5: Expert

• No longer relies on rules, guidelines, or maxims
• Works primarily from intuition
• Analytic approaches only used in novel situations or when problems occur
• When forced to follow set rules, performance is degraded

The Java World article laments about companies that advertise for experts, but don’t interview for it. I have to say this struck a nerve with me. Multiple choice tests like the Zend Certification disappoint me. Tests like this don’t measure skill on this scale at all, they measure exposure.

In the past, I’ve used a coding sample as an interview question. The code, about 250 lines, was distilled from an existing system and is horribly bad in so very many ways, but functional. By showing the code to an interviewee and asking them what they would do to improve it, I found I could get a good idea of their skill level. Novices simply had no idea what to do with it and would just move code around. Sometimes they would insert comments, trying to make the code “better.” Some candidates only found and fixed one problem (there were several major ones.) No one fixed all the problems. The one person that I interviewed and didn’t have look at the code (for time reasons), we had to let go. He simply fooled us at his interview about his skill level. The thing is that I am certain he could have passed a Delphi certification test. (like the one I took at Brain Bench.)

Moving from novice to expert programmer takes a very long time. From Teach Yourself Programming in Ten Years:

Researchers (Hayes, Bloom) have shown it takes about ten years to develop expertise in any of a wide variety of areas, including chess playing, music composition, painting, piano playing, swimming, tennis, and research in neuropsychology and topology. There appear to be no real shortcuts: even Mozart, who was a musical prodigy at age 4, took 13 more years before he began to produce world-class music. In another genre, the Beatles seemed to burst onto the scene with a string of #1 hits and an appearance on the Ed Sullivan show in 1964. But they had been playing small clubs in Liverpool and Hamburg since 1957, and while they had mass appeal early on, their first great critical success, Sgt. Peppers, was released in 1967. Samuel Johnson thought it took longer than ten years: “Excellence in any department can be attained only by the labor of a lifetime; it is not to be purchased at a lesser price.” And Chaucer complained “the lyf so short, the craft so long to lerne.”

So you can’t become an expert without experience. However, you can have experience without becoming an expert. Some people just put in their time and never develop themselves. So these people may have the answers to the certification trivia question for their specific environment and be able to get past the HR resume screeners with their buzzword detectors, but they will not have the impact that a true expert would have in the same situation. Just like our Delphi Dud.