To Infinity and Beyond!
February 22, 2009
By Jeff Freund
"Web Scale" is a phrase I use quite frequently. I encourage engineers to think in "Web Scale", and consider such things as "how will this work when things get really, really big?", or "what happens when you have 10x or 100x that many users?"
I believe that there are three categories of scale:
- Prototype Scale - This is the first phase of things. You never quite know what you get when you start out building a product or application, but by following some good technology practices (ie. using database indexes correctly) you can be reasonably confident of getting something that works that you can provide to some real customers and users. Only then can you truly start figuring out what needs optimizations and scalability work.
- Enterprise Scale - You have hit Enterprise Scale when you are confident that you can service the largest customer that you are likely to get. For installed software, you may actually be able to stop here. But for those providing SaaS solutions or Internet services, there is a critical next step.
- Web Scale - When you hit Web Scale, you have identified and solved all scaling challenges. Everything in the platform will scale linearly such that you can not only service your largest single potential customer, but you can service any number of such customers of that size.
There are several big name internet companies that pop to my mind when I think about Web Scale: Google, Yahoo!, Amazon. There are even some SaaS platforms that can make that claim, like Salesforce.com and Omniture. One thing that these companies all share is a well architected and crafted platform that uses distributed computing and sharding to take HUGE challenges and make them small and repeatable.