BBCODE was designed for this purpose. There is no actual standard, but the core syntax seems fairly uniform. It’s good for those used to forums, where it seems to norm.

HTML markup is nice because it is a standard, even if varying subsets are supported. Learning a little HTML isn’t going to hurt anyone, at least for the next 20 years or so. The problem is that HTML was never intended to be hand edited. The syntax is not the most inviting, and different HTML-like markup languages handle whitespace differently than the HTML standard.

Wiki markup syntaxes were designed to be human friendly. The main problem I have with wiki syntax is that there is no standard. It seems like every wiki has a different way to formulate a link, for example. I guess there is some progress with Wiki Creole, but I still have a bad taste in my mouth.

The other problem I have with wiki markup is that I find it to be non-deterministic. When I edit any given wiki and try to use more than basic formatting, I never know what I am going to get. Most of the markup processing engines for these wikis are impenetrable morasses of regular expressions. It can be hard to gauge interactions. Are you really sure they are secure?

Speaking of impenetrable morasses of regular expressions, have you ever looked at WordPress’s input path? I’m sure every one with a WordPress blog who likes to blog about PHP code knows that it is a code eater. I’ve been particularly disappointed with WordPress in this area. Most the “code formatting” plugins still have problems protecting code from WordPress’ heavy hand.

But the WordPress preg_replace gauntlet doesn’t just mangle code. I have a post which has been sitting in draft mode for several weeks because I can’t figure out how to give it the proper markup. WordPress is somehow taking my perfectly balanced input markup and producing “unbalanced” output markup. I haven’t yet tracked down the problem to either submit a fix or to do a good bug report. Frankly, I’m not looking forward to trudging through all those regular expressions.

In Chris’ post, he takes the regular expression approach. Folks in the comments have pointed out a few problems with his approach, including the problem of interleaved tags. If you can’t tell by now, I am not a fan of the regular expression gauntlet approach to markup languages. I prefer a defined syntax and a traditional computer science style parser (which may use regular expressions).

The other must-have is a preview option. With so much variation in markup languages, not having a preview leaves the user to play Russian roulette with their submitted content. I’ve talked about that before in the usability of input filtering. This is another area where WordPress leaves the user high and dry.

The complex input path in WordPress combined with its reliance on global variables seems to leave it unable to do an in-page preview. The admin area preview is an IFRAME so that it launches a separate request. The various live preview plugins are JavaScript based and don’t work when it is disabled. They also don’t pass the input through the same input path that WordPress uses, so they are not a true preview.

I don’t mean for this to be a WordPress rant, on the whole, I like WordPress. Rather, I just wanted to point out how hard it can be to do good input filtering, that is safe, reliable, deterministic, and usable.

My Grandma is doing well. They used an experimental new procedure called radio frequency ablation to remove the meta-static colon cancer tumors from her lungs. This procedure is amazing compared to the standard treatment. The doctors at the University of Michigan were impressive. We’ll know the results in a couple months when her lungs look a little less like scrambled eggs. We’re hopeful.

I’m not much for retrospectives. Looking forward into 2007, I have a few major goals. I joined a gym today. I’m going to get a new laptop and refresh my development environment next week after MacWorld. I want to get at least a beta release of WACT out by May. I have to prepare for php|tek. I need to find a new place to live by this fall. (Ann Arbor?) I want to move by the end of the year.

I loved all my christmas and birthday gifts this year. (My birthday is December 28th.) This year I pointed everyone to my Amazon.com wishlist and I ended up with a ton of good books to read. Jason Gillmore from Apress also sent me some web development books. My to-read stack for 2007 includes:

• The Promise of Sleep - A survey of the subject of sleep for laymen, written by a top sleep researcher. I’m almost done with this one. This book has a bunch of sleep deprivation horror stories and a good survey of what is known about sleep, which is not much. Its incredible that we know so little about something we spend so much time doing. Its also amazing how many people have easily treatable sleep disorders that don’t even know it. Do you snore?
• Don’t make me Think - Looks like a nice overview book on web usability.
• Domain Driven Design - Recommended by Jason and Marcus. How did I get this far without reading this book?
• Da Vinci Code - Wasn’t on my wishlist, but I’ll read it anyway. I read so little fiction these days. Where is a beach when you need one?
• Getting Things Done - I’m almost through this one. It is a testimony to the power of the ideas that this book expresses that so many people recommend it, despite its being so incredibly dull. Useful? Yes. Inspiring? No. But, then I’ve read enough of these self help / personal productivity type books for a lifetime. Anyone want to buy a Franklin Planner? I used mine until I got a cell phone.
• Practical Subversion - I’m really liking subversion. If you haven’t tried it, do so. I’m hoping to combine this with Greg Beaver’s book, The PEAR installer manifesto — the book on my wishlist I most wanted that I didn’t get, to create a new deployment process.
• Pro CSS Techniques - A CSS book that tackles maintainability? I’m really looking forward to this one.
• Pro MySQL - The last MySQL book I read was a couple years ago, yet I use it almost every day. I’m due for a refresh. This one looks good.
• Pro PHP Security - Never hurts to brush up. This one looks like it has alot on encryption, SSL and SSH; not strong areas for me.
• Pattern-Oriented Software ARchitecture Volume 2 - The first volume, A system of patterns, is one of my “always within reach when developing” books. Nice to add to the set.

Thanks for the books, guys. I’ll have in-depth reviews of some of these here in the future.

Happy New Year.

I watched a bunch of google tech talk presentations a few months ago. By far the best presentation was Barry Schwartz explaining The paradox of Choice. This video is worth watching.

I think these are important ideas in the quest for extreme simplicity.

10 fundamental rules for the age of user experience technology:

1. More features isn’t better, it’s worse.
2. You can’t make things easier by adding to them.
3. Confusion is the ultimate deal-breaker.
4. Style matters
5. Only features that provide a good user experience will be used.
6. Any feature that requires learning will only be adopted by a small fraction of users.
7. Unused features are not only useless, they can slow you down and diminish ease of use
8. Users do not want to think about technology: what really counts is what it does for them.
9. Forget about the killer feature. Welcome to the age of the killer user-experience.
10. Less is difficult, that’s why less is more

Consumer devices or frameworks, we are a product of our times.

Watch and contemplate.

• The departure of the hyper-enthusiasts - “The Java hyper-enthusiasts have left the building” (along the lines of this.)
• The New Methodology - Martin Fowler describes Agile methodologies — recently updated.
• PHP on Caucho - PHP on the JVM.
• XML 2.0 - some thoughts on XML 2.0.
• Web Patterns - Under construction — check back later.
• Web Design Patterns.
• Carnival of the Agilists.

I’m currently adding UTF-8 support to and generally improving WACT’s “liberal” xml/html parser. A few sources of tests cases and information:

• Who knows a title from a hole in the ground?
• FeedBurner feeds give heartburn to PHP XML parsers?
• Web SGML and HTML 4.0 explained
• CDATA confusion
• Comment syntax in SGML and HTML
• Empty elements in SGML, HTML, XML, and XHTML
• Comparison of SGML and XML
• Extensible Markup Language (XML) Conformance Test Suites
• Ian Hixie’s HTML parsing test cases
• Conformance Testing for XML and Related Technologies
• Universal Feed Parser liberal feed parser with many test cases.
• XHTML test cases

• UTF-8 and Unicode FAQ for Unix/Linux
• bad UTF-8 test files
• The W3C Markup Validation Service: Tests

I think the fundamental problem with input filtering and especially XSS filtering is that it violates the principle of least surprise. User input is silently modified without the user’s knowledge. If the violation is innocent, then the software surprises the user. This is bad. At least with validation, the user gets a heads up on the problem.

Let me try to name and enumerate some scenarios:

Direct Filter
This is what WordPress did in the example post. It simply accepted the user input and silently changed it. The filtered value is stored directly into the database. The original input is lost. There is no preview. I think this has to be a usability worse case scenario.

Filter with Preview
This scenario adds a preview capability to the last. The filter is still applied. A validation failure or explicit preview button causes the form values to be re-displayed and a preview panel to be shown. However, the previous input value is silently modified and sent back to the user. The user may or may not realize that his original input has been changed during the round trip.

This is also seems like a usability problem, but every once and a while it happens to me when entering legitimate input into professionally written programs.

Filter with Buffered Preview
This scenario adds an additional buffer to the last. The filter is applied, but the original input is sent back to the user in the form field. However, the preview panel shows the modified value.

I don’t really see this very often outside of fields with a dedicated markup language (for example BBCode).

Filter with Forced Preview
The input value is silently filtered. However, the user is forced to preview the output at least once. Its up to the user to notice the results of the filter.

I think slashdot does this.

Filter with Confirmation
A stricter variation of Forced Preview where as the last stage, the user must confirm their input once without the ability to change it. It is up to the user to notice the results of the filter.

I think this is popular as the last stage of a wizard style interface.

Filter with Confirmation and Warning
The filter is applied and the user’s input is changed, however, the user is warned exactly which value was changed by the filter.

I don’t think I’ve ever seen this one.

Validation
The program notifies the user that the input value is bad, but does not modify it. The user must change the value to proceed.

I tend to use this one. I escape all output, so I don’t worry too much about displaying XSS in the preview panel.

Obviously, you can mix and match scenarios for different input rules and fields. I’m sure there are other scenarios. Please suggest some.

I guess I’ve been programming for about 23 years now. The longer I do it, the more reluctant I am to be strict with user input. Ultra sanitized, ultra structured data may seem attractive to the programmer, but its a pain for the user and its only a matter of time before a legitimate exception comes along. A European phone number, the 51rst state, a canadian postal code, a new millennium, etc. The exception is the rule. Understandably, XSS must be prevented, but its easy to go too far.

Which of these scenarios do you think are best from the user’s perspective? From the programmers perspective?

1. Download.
2. Locate the application icon in your download directory and optionally move it to another location.
3. Double click on the application icon to run.

This is the essence of what apple calls Drag and Drop installation.

Unfortunately, installing web applications has not reached this level of ease of use refinement. An end user oriented web application might require that you:

1. Download to your local machine (Using browser)
2. Unpack (using 3rd party archive tool)
3. Locate and edit configuration files, potentially requiring a knowledge of file system paths on the server, database names, etc.
4. Upload (Using yet another tool, such as ftp)
5. Change file permissions to allow files to be written to some directories on the server
6. Install external components, such as pear modules
7. Compile PHP to include required modules
8. Edit .htaccess to setup include_path’s and mod_rewrite

Hardly “Drag and Drop.” It would be nice to have something like a drag and drop capability for end users. In this case, the applications could just be moved to the desired location in DOCUMENT_ROOT and run by pointing the browser to it.

Apple has several supporting technologies that facilitate drag and drop install:

Grouping

“Bundles provide an elegant solution to the problem of grouping related code and resources together.” PHP has the tarball or the zip file, but this requires an unpacking stage. If the user lacks shell access, then the unpacking stage must occur offline. Where is the PHP equivelent of the .JAR or the .WAR? How can the unpack stage be eliminated?

Hiding

One of the advantages of the bundle is the ability to hide internal resources from the end user. Web applications have the need to hide internal resources. Typically, this can be done with PHP by moving files outside of your DOCUMENT_ROOT (adding another installation stage), hiding resources with .htaccess, or giving them .php extensions when possible (typically for config files). Hardly end user friendly techniques.

Sharing

Frameworks are a special type of bundle designed for distributing shared code and resources.

Frameworks have a version capability that helps manage DLL Hell situations, where installing or upgrading one application breaks another via shared dependencies. A Framework bundle may contain multiple versions of the same code, and the OS will link to the correct one. Additionally, framework bundles can be embedded into Application Bundles, so that application will always have a usable version available. Embedding helps facilitate drag and drop install because an additional step is not needed to install shared resources.

I think embedding shared resources (libraries like adodb and wact) is the best way to go for application deployment.

Unfortunately, I think PEAR takes a shared commons attitude and resists embedding because of its desire to be in the include_path.

Preferences

User configuration in OS X is stored in preference files, which are independent files named by application. Well behaved OS X application operate even with their preferences files missing. deleting an applications preference file causes the application to recreate it with the default preferences. (Try deleting an application’s registry keys under windows and see how things go.)

Drag and Drop Web Applications
I think a drag and drop PHP web application should meet the following criteria:

• Make no requirements for PHP_INI_SYSTEM or PHP_INI_PERDIR configuration.
• Make no requirements for .htaccess configuration
• Work with default application level configuration
• Not require writable filesystem permissions
• Not require installing external software
• Not require unarchiving (a wish, i know)
• Not require obscure PHP modules

A wish list, i am sure.

What are best practices for deploying PHP web applications? What applications do this well?

If you start composing an Email message and your dialup connection is disconnected while composing, even if you reconnect before hitting send, that email can not be sent. Somehow Mail associates it with the wrong status, caches some sort of stale DNS or status information. The email goes into your outbox and Mail refuses to deliver it, ever. It does bring up a cryptic message when it tries, even though there is no reason Mail could not deliver the message. The solution is to go into your outbox, double click on the email and manually hit send again.

Unfortunately, this is too much for my grandmother. She can never quite remember how to fix this problem. It happens frequently enough to be really annoying, but not frequently enough to learn the work around.

So I went over again today and unconstipated her outbox. She said “Let me write down how to do that.” However, sitting next to the computer were the instructions for doing it from the last time this happened and she wrote it down. I really hope Apple fixes this usability issue in the next OS release.

First, it appears performance really was a major problem for friendster. Friendster Quickly Gathering Foes:

The key issues behind the Friendster abandonment trend, according to users, are the service’s inability to do anything about its habitual server lag problems, and its growing reputation for heavy-handed moral policies and unilateral decisions it makes on behalf of its members.

Performance problems show up in this usuability study of Friendster. more discussion of friendster performance. This quote from Cracking the code to Romance seems to show they are aware of the issue.

Notified of the security holes Moore and Chisholm exploit, Friendster rep Lisa Kopp insists, “We have a policy that we are not being hacked.” When I explain that, policy or no, they are being hacked, she says, “Security isn’t a priority for us. We’re mostly focused on making the site go faster.”

While the consensus seems to be that you can write scalable applications in either Java or PHP, MySQL is another major part of Friendster’s architecture. Why Friendster is so slow makes an educated guess on the cause of Friendsters performance problems, laying the blame on an inappropriate use of MySQL. I wonder how much of this is educated and how much of this is guess.

Philip Greenspun suggests that Friendster “flush MySQL and replace with Oracle 10g.”

It would seem that the friendster folks have a mysql support contract and Attend MySQL conferences:

between sessions, three young men from Friendster are chatting with a neat-looking person in a MySQL shirt, who introduces them to another neat-looking person in a MySQL shirt and says, “he’ll be your primary support contact.”

If the natural architecture of PHP is to push scalability issues out of the language and into the database, the Friendster case seems to raise the question, does MySQL scale?

I would like to see more official information out of Friendster regarding this case.