I just read Google's Bigtable paper. It is relevant here because it talks about keeping petabyte scale (1024TB) tables on a variable size cluster of machines. I have talked about partitioning versus distributed cache in the second to last post. The problem in short is that you do not expect a dba to really know how to partition things and even if the indices are correctly partitioned repartitionin ... Continue reading