> > To confirm - are you saying the data directory size is huge, but the > live size as reported by nodetool ring and nodetool info does NOT > reflect this inflated size? >
That's correct. > What files *do* you have in the data directory? Any left-over *tmp* > files for example? > > The files that consumed most do disk space were of the CF that was 23+GB. There were multiple copies of 23GB files of that CF, and they did not have any Compacted file associated with them nor were they of *tmp* files. > Are you sure you're only running a single repair at a time? (Sorry if > this was covered, I did a quick swipe through thread history because I > was unsure whether I was confusing two different threads, and I don't > think so.) > > Absolutely sure I was running single repair. In fact, after few data reset (restore from backup), and retries (reset data on each retry, except one time I did not reset data for different test) on the same node, I thought may be there was some kind of hardware issue but OS not reported; so I reset data and try repair on diff node, I ran into the same issue. This morning I restored data from backup and kicked off repair with version 0.6.11, and it worked fine. > The question is what's taking the space. If it's sstables, they really > should be either compacted onces that are marked for deletion but > being retained, or "live" sstables in which case they should show up > as load in nodetool. > > I also tested compaction once, and it did reduce the size, but I could not wait for it to finish as the size of the directory containing the sstables grew over 700GB, and it was just taking too long to compact down to the expected size. > What else... maybe streams are being re-tried from the source nodes > and the disk space is coming from a bunch of half-finished streams of > the same data. But if so, those should be *tmp* files IIRC. > As mentioned above, most of disk space used were from multiple files of a big CF. Is there any chance that the entire file from source node got streamed to destination node even though only small amount of data in hte file from source node is supposed to be streamed destination node? > > I'm just wildly speculation, but it would be nice to get to the bottom of > this. > > -- > / Peter Schuller (@scode on twitter) > -- Huy Le Spring Partners, Inc. http://springpadit.com
