Infrastructure Scaling: The NewsBlur Story


For those who know what RSS is, you probably also know by now that Google Reader was announced to be shutting down soon last week. RSS is simply a way for people to go to a web site and subscribe to updates, then keep track of all of these feeds through a single location. Google used to have the most popular reader out there, and many devout fans. But lately, the company has been trying to shut down initiatives it doesn't see as core to its business. This unfortunately includes Google Reader. But this in turn proved to be a great boon for young entrepreneurs, inventors who had created their own competing products, people like NewsBlur. As their daily userbase went from 1,500 users to 50,000 overnight, this caused significant problems.

Now in a blog post, the company executive talk about their experience, and what other entrepreneurs should know in case something like that happens to them. The first issue was scaling, where their servers were not prepared to handle the type of user load they received. For one thing, nothing was cached. Having redundant, distributed caching became a priority. Their host basically crumbled under the load and they had to quickly switch around. Even their email registration process could not cope, since they had a maximum amount of mail per day which they never hit before.

The takeaway is that any product, any invention that provides some type of service to the web, should be built from the ground up for scalability. You never know when you will get a surge of new users. Try to have as few bottlenecks as possible. But another lesson is that even if you plan, there is no way to be completely prepared for the unknown. So you need to be fast and able to move quickly.