Scaling Web Applications
Sam Ruby, via Tim Bray, has collected a list of scaling web applications presentations and documents. As Tim said, this is “everything anybody knows” on the subject.
I’m interested in large scale data crunching as we build out our data warehouse. It’s tricky for us, as we have one machine to do all of our data crunching, so we are definitely constrained by I/O. To really solve this issue on a single machine, we need to be smart with our disks and spread the data out to ensure parallel reads.
As I read through these presentations and reports, I’m always trying to map it back down to one machine with maybe four discs and two dual core processors.
Of course, I can just rent a Hadoop cluster.
Note to Amazon EC2: Install a EC2 instance on the DoD .mil network so we can use it, too!
