Professional PHP » input-filtering http://www.procata.com/blog PHP Programming, Web Development, PHP Advocacy and PHP Best Practices. Fri, 10 Dec 2010 17:23:30 +0000 http://wordpress.org/?v=2.8.6 en hourly 1 The Problem with Markup Languages http://www.procata.com/blog/archives/2007/03/14/the-problem-with-markup-languages/ http://www.procata.com/blog/archives/2007/03/14/the-problem-with-markup-languages/#comments Wed, 14 Mar 2007 17:30:14 +0000 Jeff http://www.procata.com/blog/archives/2007/03/14/the-problem-with-markup-languages/ Chris Shiflett has a post today, Allowing HTML and Preventing XSS. The problem is how to allow users to format their contributed content without introducing security vulnerabilities. The answer is usually some sort of markup language or filtering and sanitization of HTML.

BBCODE was designed for this purpose. There is no actual standard, but the core syntax seems fairly uniform. It’s good for those used to forums, where it seems to norm.

HTML markup is nice because it is a standard, even if varying subsets are supported. Learning a little HTML isn’t going to hurt anyone, at least for the next 20 years or so. The problem is that HTML was never intended to be hand edited. The syntax is not the most inviting, and different HTML-like markup languages handle whitespace differently than the HTML standard.

Wiki markup syntaxes were designed to be human friendly. The main problem I have with wiki syntax is that there is no standard. It seems like every wiki has a different way to formulate a link, for example. I guess there is some progress with Wiki Creole, but I still have a bad taste in my mouth.

The other problem I have with wiki markup is that I find it to be non-deterministic. When I edit any given wiki and try to use more than basic formatting, I never know what I am going to get. Most of the markup processing engines for these wikis are impenetrable morasses of regular expressions. It can be hard to gauge interactions. Are you really sure they are secure?

Speaking of impenetrable morasses of regular expressions, have you ever looked at WordPress’s input path? I’m sure every one with a WordPress blog who likes to blog about PHP code knows that it is a code eater. I’ve been particularly disappointed with WordPress in this area. Most the “code formatting” plugins still have problems protecting code from WordPress’ heavy hand.

But the WordPress preg_replace gauntlet doesn’t just mangle code. I have a post which has been sitting in draft mode for several weeks because I can’t figure out how to give it the proper markup. WordPress is somehow taking my perfectly balanced input markup and producing “unbalanced” output markup. I haven’t yet tracked down the problem to either submit a fix or to do a good bug report. Frankly, I’m not looking forward to trudging through all those regular expressions.

In Chris’ post, he takes the regular expression approach. Folks in the comments have pointed out a few problems with his approach, including the problem of interleaved tags. If you can’t tell by now, I am not a fan of the regular expression gauntlet approach to markup languages. I prefer a defined syntax and a traditional computer science style parser (which may use regular expressions).

The other must-have is a preview option. With so much variation in markup languages, not having a preview leaves the user to play Russian roulette with their submitted content. I’ve talked about that before in the usability of input filtering. This is another area where WordPress leaves the user high and dry.

The complex input path in WordPress combined with its reliance on global variables seems to leave it unable to do an in-page preview. The admin area preview is an IFRAME so that it launches a separate request. The various live preview plugins are JavaScript based and don’t work when it is disabled. They also don’t pass the input through the same input path that WordPress uses, so they are not a true preview.

I don’t mean for this to be a WordPress rant, on the whole, I like WordPress. Rather, I just wanted to point out how hard it can be to do good input filtering, that is safe, reliable, deterministic, and usable.

]]> http://www.procata.com/blog/archives/2007/03/14/the-problem-with-markup-languages/feed/ 14 The Usability of Input Filtering http://www.procata.com/blog/archives/2005/03/31/the-usability-of-input-filtering/ http://www.procata.com/blog/archives/2005/03/31/the-usability-of-input-filtering/#comments Fri, 01 Apr 2005 06:21:05 +0000 Jeff http://www.procata.com/blog/archives/2005/03/31/the-usability-of-input-filtering/ There seems to be much interest lately in input filtering in PHP, especially in cross site scripting prevention. I’ve always preferred input validation to input filtering, but I am giving filtering a new examination. My problem with filtering is with usability. The comments to this post are a good example. There are obviously some usability issues going on here.

I think the fundamental problem with input filtering and especially XSS filtering is that it violates the principle of least surprise. User input is silently modified without the user’s knowledge. If the violation is innocent, then the software surprises the user. This is bad. At least with validation, the user gets a heads up on the problem.

Let me try to name and enumerate some scenarios:

Direct Filter
This is what WordPress did in the example post. It simply accepted the user input and silently changed it. The filtered value is stored directly into the database. The original input is lost. There is no preview. I think this has to be a usability worse case scenario.

Filter with Preview
This scenario adds a preview capability to the last. The filter is still applied. A validation failure or explicit preview button causes the form values to be re-displayed and a preview panel to be shown. However, the previous input value is silently modified and sent back to the user. The user may or may not realize that his original input has been changed during the round trip.

This is also seems like a usability problem, but every once and a while it happens to me when entering legitimate input into professionally written programs.

Filter with Buffered Preview
This scenario adds an additional buffer to the last. The filter is applied, but the original input is sent back to the user in the form field. However, the preview panel shows the modified value.

I don’t really see this very often outside of fields with a dedicated markup language (for example BBCode).

Filter with Forced Preview
The input value is silently filtered. However, the user is forced to preview the output at least once. Its up to the user to notice the results of the filter.

I think slashdot does this.

Filter with Confirmation
A stricter variation of Forced Preview where as the last stage, the user must confirm their input once without the ability to change it. It is up to the user to notice the results of the filter.

I think this is popular as the last stage of a wizard style interface.

Filter with Confirmation and Warning
The filter is applied and the user’s input is changed, however, the user is warned exactly which value was changed by the filter.

I don’t think I’ve ever seen this one.

Validation
The program notifies the user that the input value is bad, but does not modify it. The user must change the value to proceed.

I tend to use this one. I escape all output, so I don’t worry too much about displaying XSS in the preview panel.

Obviously, you can mix and match scenarios for different input rules and fields. I’m sure there are other scenarios. Please suggest some.

I guess I’ve been programming for about 23 years now. The longer I do it, the more reluctant I am to be strict with user input. Ultra sanitized, ultra structured data may seem attractive to the programmer, but its a pain for the user and its only a matter of time before a legitimate exception comes along. A European phone number, the 51rst state, a canadian postal code, a new millennium, etc. The exception is the rule. Understandably, XSS must be prevented, but its easy to go too far.

Which of these scenarios do you think are best from the user’s perspective? From the programmers perspective?

]]> http://www.procata.com/blog/archives/2005/03/31/the-usability-of-input-filtering/feed/ 26 Even the Big Guys Get Validation Wrong http://www.procata.com/blog/archives/2004/05/13/even-the-big-guys-get-validation-wrong/ http://www.procata.com/blog/archives/2004/05/13/even-the-big-guys-get-validation-wrong/#comments Thu, 13 May 2004 12:46:17 +0000 Jeff http://www.procata.com/blog/archives/2004/05/13/even-the-big-guys-get-validation-wrong/ I ordered a computer for someone from Dell last night. When I got to the end of the order, I mistyped a digit on the credit card number and the form was redisplayed with an “invalid credit card number” error. I added spaces between the digits (as they appear on the card) to check the number. Sure enough, one digit was wrong. I re-submitted, but the “invalid card number” error remained. I was sure the card was valid and that I typed in the correct number. After a little experimenting, it turns out that the order form could not handle the spaces that I added. The person I was ordering the computer for was looking over my shoulder and said that he would have never figured out to remove the spaces. I wonder how many people enter their credit card numbers as XXXX XXXX XXXX XXXX, just as it appears on the card.

(P.S. Dell seems to be a master at “Do you want fries with that?”)

]]> http://www.procata.com/blog/archives/2004/05/13/even-the-big-guys-get-validation-wrong/feed/ 4