First, it appears performance really was a major problem for friendster. Friendster Quickly Gathering Foes:

The key issues behind the Friendster abandonment trend, according to users, are the service’s inability to do anything about its habitual server lag problems, and its growing reputation for heavy-handed moral policies and unilateral decisions it makes on behalf of its members.

Performance problems show up in this usuability study of Friendster. more discussion of friendster performance. This quote from Cracking the code to Romance seems to show they are aware of the issue.

Notified of the security holes Moore and Chisholm exploit, Friendster rep Lisa Kopp insists, “We have a policy that we are not being hacked.” When I explain that, policy or no, they are being hacked, she says, “Security isn’t a priority for us. We’re mostly focused on making the site go faster.”

While the consensus seems to be that you can write scalable applications in either Java or PHP, MySQL is another major part of Friendster’s architecture. Why Friendster is so slow makes an educated guess on the cause of Friendsters performance problems, laying the blame on an inappropriate use of MySQL. I wonder how much of this is educated and how much of this is guess.

Philip Greenspun suggests that Friendster “flush MySQL and replace with Oracle 10g.”

It would seem that the friendster folks have a mysql support contract and Attend MySQL conferences:

between sessions, three young men from Friendster are chatting with a neat-looking person in a MySQL shirt, who introduces them to another neat-looking person in a MySQL shirt and says, “he’ll be your primary support contact.”

If the natural architecture of PHP is to push scalability issues out of the language and into the database, the Friendster case seems to raise the question, does MySQL scale?

I would like to see more official information out of Friendster regarding this case.

1) We had not one but TWO guys here who had written bestselling JSP books. Not that this necessarily means they’re great Java devs, but I actually think our guys were as good as any team.

2) We tried rewriting the site in Java twice, using MVC and all available best practices. It actually got slower. Anyway, what does MVC have to do with speed or scalability? I thought it was a design cleanliness and maintainability thing.

3) We tried different app servers, different JVMs, different machines.

4) Anything that money could do, it did.

George Schlossnagle has the best coverage of the topic, saying It’s the App code, stupid (As an Adult Swim fan, I have to appreciate the Big O reference):

My personal bias against java is that many Java programmers seem to be thread-crazy, and as has been noted, humans just aren’t smart enough to program threads.

I once lost a job by saying “Threads are dangerous” at a job interview. Everything was going well, until they asked me what I thought about threading. (this was a business application) Obviously, my opinion didn’t agree with this companies vision for their software. Ironically, I was being interviewed to help them fix their problems with frequent unrepeatable lockups and GPFs. They did not see the connection.

One advantage of PHP is that it simplifies your life by not bothering you with threading, just as it doesn’t bother you with memory management. That doesn’t mean that concurrency issues go away, any multi user application will have them, it just means that they are made explicit at the application level.

George Schlossnagle:

Share nothing’ (which if you don’t want to click on the link basically means not performing much inter-process or inter-thread data sharing or pooling) is not something unique to PHP. In fact, I’m quite sure that you can implement it in Java as well. The problem is that Java gives you a number of powerful facilities with which to shoot yourself in the foot

Here is a java performance article that recommends against object pooling. The struts vs. webwork comparison has an interesting take on this:

Struts Actions must be thread-safe because there will only be one instance to handle all requests. This places restrictions on what can be done with Struts Actions as any resources held must be thread-safe or access to them must be synchronized.

WebWork Actions are instantiated for each request, so there are no thread-safety issues. In practice, Servlet containers generate many throw-away objects per request, and one more Object does not prove to be a problem for performance or garbage collection.

It seems like the Java culture, if not the java language seems to encourage premature optimization in the form of creating pools of shared information.

John Lim expresses some skepticism about the inherent scalability properties of any language, claiming its the developer skill not the language.

I think I’ll end this post with heresy. The field of web development seems to have a mental model of application development forged from the dot-com boom era. We operate with the vision that our applications are going to experience exponential usage growth. Perhaps this leads to an unhealthy focus on scalability in web applications versus other requirements. Perhaps this also leads us to employ optimizations prematurely before we can even understand their impact or even have a need for them. Perhaps these premature optimizations even hurt scalability and performance and needlessly complicate our applications.

Perhaps the Java Culture is more infected with “dot-com-itis” than the php culture?

Scalability – YAGNI?

I hope that more information will come out about this. It would be nice to have another large scale case study alongside the Yahoo talks: Large Scale PHP, One year of PHP at Yahoo, and Making the case for PHP at Yahoo.

Another interesting case study is the Gaia Online discussion board. One of the largest forums on the net, on a recent check, it claims to have 70 million messages online with roughly 9,000 simultaneous users, and 750,000 registered users. The sitepoint post that brought this to my attention claims they are getting over 700,000 new posts per day. All of this running on a modified version of open source phpBB. Here is an interview with the board owner and founder.

The thread that talks about the modifications to phpBB to support Gaia Online’s volume is very interesting. It focuses mostly on query optimization. Not surprising in light of the previous discussion on PHP’s shared nothing architecture. If PHP encourages moving concurrency management to the database, then application scalability is going to become a matter of database scalability.

Switching from scalability to performance, John Lim talks about his experiences optimizing PHP code using xdebug. His article makes me want to check out xdebug, but I have too much on my plate right now to even risk getting near my PHP configuration. I am hoping that I will have time to do this before the upcoming WACT release.