Except major compactions are not that rare if you have a cluster which you need to add capacity to. Anytime to add nodes with bootstrap it is recommended you run cleanup on nodes which you removed data from (and this is useful to see how much space you are now using). Cleanup does a major compaction and if you happen to have one disk larger than others, most of the data ends up there (this happened to me when I added some ebs's to some ec2 nodes, I distributed the sstables and everything was cool, I had more io, things were great, then I need to add another node, did a cleanup, boom everything is on one disk and io sucks again). I also don't quite know what happens when a major compaction occurs which would combine sstables and fill up the largest disk?
However after discussion I completely understand why things were done this way, it's difficut to manage the space and really it should be relegated to the disk subsystem of the OS (ie, RAID0, JBOD, LVM, etc). I would almost recommend just keeping things simple and removing multiple data directories from the config altogether and just documenting that you should plan on using OS level mechanisms for growing diskspace and io. -Anthony On Wed, Mar 10, 2010 at 04:43:36PM -0600, Stu Hood wrote: > Yea, I suppose major compactions are the wildcard here. Nonetheless, the > situation where you only have 1 SSTable should be very rare. > > I'll open a ticket though, because we really ought to be able to utilize > those disks more thoroughly, and I have some ideas there. > > > -----Original Message----- > From: "Anthony Molinaro" <antho...@alumni.caltech.edu> > Sent: Wednesday, March 10, 2010 3:38pm > To: cassandra-user@incubator.apache.org > Subject: Re: Effective allocation of multiple disks > > This is incorrect, as discussed a few weeks ago. I have a setup with multiple > disks, and as soon as compaction occurs all the data ends up on one disk. If > you need the additional io, you will want raid0. But simply listing multiple > DataFileDirectories will not work. > > -Anthony > > On Wed, Mar 10, 2010 at 02:08:13AM -0600, Stu Hood wrote: > > You can list multiple DataFileDirectories, and Cassandra will scatter files > > across all of them. Use 1 disk for the commitlog, and 3 disks for data > > directories. > > > > See http://wiki.apache.org/cassandra/CassandraHardware#Disk > > > > Thanks, > > Stu > > > > -----Original Message----- > > From: "Eric Rosenberry" <epros...@gmail.com> > > Sent: Wednesday, March 10, 2010 2:00am > > To: cassandra-user@incubator.apache.org > > Subject: Effective allocation of multiple disks > > > > Based on the documentation, it is clear that with Cassandra you want to have > > one disk for commitlog, and one disk for data. > > > > My question is: If you think your workload is going to require more io > > performance to the data disks than a single disk can handle, how would you > > recommend effectively utilizing additional disks? > > > > It would seem a number of vendors sell 1U boxes with four 3.5 inch disks. > > If we use one for commitlog, is there a way to have Cassandra itself > > equally split data across the three remaining disks? Or is this something > > that needs to be handled by the hardware level, or operating system/file > > system level? > > > > Options include a hardware RAID controller in a RAID 0 stripe (this is more > > $$$ and for what gain?), or utilizing a volume manager like LVM. > > > > Along those same lines, if you do implement some type of striping, what RAID > > stripe size is recommended? (I think Todd Burruss asked this earlier but I > > did not see a response) > > > > Thanks for any input! > > > > -Eric > > > > > > -- > ------------------------------------------------------------------------ > Anthony Molinaro <antho...@alumni.caltech.edu> > > -- ------------------------------------------------------------------------ Anthony Molinaro <antho...@alumni.caltech.edu>
