Once a pragmatic lurker, now a nervous participant.
I’d like to have a book on practical ways to set up a full computing environment, so that I can build reliable and scalable (and inexpensive) online applications. Over the last five years I designed and helped implement a complete data center, starting with an empty secure room and a half-dozen desks for future staff. The result was a 50-server, open-source data center providing online e-commerce applications for its owner, running dozens of infrastructure services for availability and security and a dozen other “ities”, and crewed by the steadfast operations team.
I learned a lot about scalability, security, high availability, and performance that I never really understood as a programmer. Today a developer doesn’t need to buy hardware to set up a “production” environment - Amazon and the other cloud platforms are a few minutes and a credit-card imprint away. But getting access to boxes is not the hard problem for developers or web entrepreneurs. Designing a full ecosystem of servers and infrastructure applications to keep services running and customer data private is the real challenge, and it doesn’t get solved when you click on “rent me a server instance” at RackSpace or Linode.
A good book for this subject would show web entrepreneurs, “app” developers, and practical programmers how to put together a completely professional delivery environment, based on either real or virtualized server hardware. It would tackle the hard problems in detail - particularly high availability, scalability, performance, and security.
These problems don’t come pre-solved by the rack center or cloud provider. In the last couple of years, major players in IT and the cloud have experienced public failures for themselves and for their customers. On my wall of shame I see Rackspace crashing customer systems (June-July 2009), Microsoft permanently losing data uploaded by T-Mobile Sidekick phone users (October 2009), Facebook’s unexpected downtime (September 2010), and the Amazon EC2 failure that brought down Reddit and Quora among many others (April 2011). For blessed brevity I won’t list the number of security breaches major providers have suffered over the same period.
If Amazon won’t make your system unbreakable and your customer data secure, who will? It’s up to the developer. Perhaps not the actual programmers, but at least the development team as a whole must be able to specify, acquire, and assemble a trustworthy platform for to deliver the programmer’s next big thing. As a programmer, I was happy to find out that the problem of building a full computing environment was not too tough, and being involved in designing it made me create better applications – more stable, simpler, easier to distribute for scaling, more secure, and pretty much impervious to hardware and software failures in the infrastructure.
These applications could run on Amazon and keep delivering services through a zone failure, or be built with cheap commodity servers and still “survive a comet strike in the East” without dropping customer sessions or stored data. They would foil common hacking attempts and secure customer data both on-disk and on the network. They would monitor themselves and support remote management to make sure they could run even when located in other countries and “availability zones”.
I haven’t yet seen the book that will give a practical overview and detailed advice on delivering applications like this. I’d like to have it, for all my enterprise development, my distributed programming and testing environment, even my dreams of striking it rich with “app” creation. It’s evident I don’t have a good elevator pitch for this idea yet, but the Pragmatic Programmers are helping me step up to the bar. I will draft a table of contents based on the preceding rant and perhaps complete a chapter this month.