Re: force gc?
What version are you on ? Check the result of you major compaction by looking for log lines such as Compacted to… They will say how much smaller the new file is. After a major compaction there should be a single SSTable, the ks-cf-he-1234 part with multiple components such as -Data.db. How many files do you have for the CF on disk ? Are you also using secondary indexes ? Reducing gc and forcing a major compaction should result in all purgable space being removed. There are times when minor compaction cannot purge expired tombstones because the rows are spread out over multiple size tiers. Expanding from 5GB to 70Gb is out of the normal expections I would say. You may want to check it the db contains what you expect it to. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 3/09/2012, at 5:32 PM, Alexander Shutyaev shuty...@gmail.com wrote: Hi Derek, I'm using size-tiered compaction. 2012/9/3 Derek Williams de...@fyrie.net On Fri, Aug 31, 2012 at 7:03 AM, Alexander Shutyaev shuty...@gmail.com wrote: Does anyone have any suggestions on how can I analyze my problem? Or maybe I'm doing something wrong and there is another way to force gc on an existing column family. Are you using leveled compaction? I haven't looked into it too much, but I think forcing a major compaction when using leveled strategy doesn't have the same effect as with size tiered. -- Derek Williams
Re: force gc?
Hi Jeffrey, I think I described the problem wrong :) I don't want to do Java's memory GC. I want to do cassandra's GC - that is I want to really remove deleted rows from a column family and get my disc space back. 2012/8/31 Jeffrey Kesselman jef...@gmail.com Cassandra at least used to do disc cleanup as a side effect of garbage collection through finalizers. (This is a mistake for the reason outlined below.) It is important to understand that you can *never* force* a gc in java. Even calling System.gc() is merely a hint to the VM. What you are doing is telling the VM that you are * willing* to give up some processor time right now to gc, how much it choses to actually collect or not collect is totally up to the VM. The *only* garbage collection guarantee in java is that it will make a best effort to collect what it can to avoid an out of memory exception at the time that it runs out of memory. You are not guaranteed when *if ever*, a given object will actually be collected. Since finalizers happen when an object is collected, and not when it becomes a candidate for collection, the same is true of the finalizer. You are not guaranteed when, if ever, it will run. On Fri, Aug 31, 2012 at 9:03 AM, Alexander Shutyaev shuty...@gmail.comwrote: Hi All! I have a problem with using cassandra. Our application does a lot of overwrites and deletes. If I understand correctly cassandra does not actually delete these objects until gc_grace seconds have passed. I tried to force gc by setting gc_grace to 0 on an existing column family and running major compaction afterwards. However I did not get disk space back, although I'm pretty much sure that my column family should occupy many times fewer space. We have also a PostgreSQL db and we duplicate each operation with data in both dbs. And the PosgreSQL table is much more smaller than the corresponding cassandra's column family. Does anyone have any suggestions on how can I analyze my problem? Or maybe I'm doing something wrong and there is another way to force gc on an existing column family. Thanks in advance, Alexander -- It's always darkest just before you are eaten by a grue.
Re: force gc?
I think I described the problem wrong :) I don't want to do Java's memory GC. I want to do cassandra's GC - that is I want to really remove deleted rows from a column family and get my disc space back. I think that was clear from your post. I don't see a problem with your process. Setting gc grace to 0 and forcing compaction should indeed return you to the smallest possible on-disk size. Did you really not see a *decrease*, or are you just comparing the final size with that of PostgreSQL? Keep in mind that in many cases (especially if not using compression) the Cassandra on-disk format is not as compact as PostgreSQL. For example column names are duplicated in each row, and the row key is duplicated twice (once in index, once in data). -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: force gc?
I think that was clear from your post. I don't see a problem with your process. Setting gc grace to 0 and forcing compaction should indeed return you to the smallest possible on-disk size. (But may be unsafe as documented; can cause deleted data to pop back up, etc.) -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: force gc?
Hi Peter, I don't compare it with PosgreSQL size, I just make some estimations.. This table / column family stores some xml documents with average raw size of 2Mb each and total size about 5Gb. However the space cassandra occupies on disc is 70Gb (after gc_grace was set to 0 and major compaction was run). Maybe there is some tool to analyze it? It would be great if I could somehow export each row of a column family into a separate file - so I could see their count and sizes. Is there any such tool? Or maybe you have some better thoughts... 2012/9/3 Peter Schuller peter.schul...@infidyne.com I think that was clear from your post. I don't see a problem with your process. Setting gc grace to 0 and forcing compaction should indeed return you to the smallest possible on-disk size. (But may be unsafe as documented; can cause deleted data to pop back up, etc.) -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: force gc?
Maybe there is some tool to analyze it? It would be great if I could somehow export each row of a column family into a separate file - so I could see their count and sizes. Is there any such tool? Or maybe you have some better thoughts... Use something like pycassa to non-obnoxiously iterate over all rows: for row_id, row in your_column_family.get_range(): https://github.com/pycassa/pycassa -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: force gc?
On Fri, Aug 31, 2012 at 7:03 AM, Alexander Shutyaev shuty...@gmail.comwrote: Does anyone have any suggestions on how can I analyze my problem? Or maybe I'm doing something wrong and there is another way to force gc on an existing column family. Are you using leveled compaction? I haven't looked into it too much, but I think forcing a major compaction when using leveled strategy doesn't have the same effect as with size tiered. -- Derek Williams
Re: force gc?
Hi Derek, I'm using size-tiered compaction. 2012/9/3 Derek Williams de...@fyrie.net On Fri, Aug 31, 2012 at 7:03 AM, Alexander Shutyaev shuty...@gmail.comwrote: Does anyone have any suggestions on how can I analyze my problem? Or maybe I'm doing something wrong and there is another way to force gc on an existing column family. Are you using leveled compaction? I haven't looked into it too much, but I think forcing a major compaction when using leveled strategy doesn't have the same effect as with size tiered. -- Derek Williams
Re: force gc?
Cassandra at least used to do disc cleanup as a side effect of garbage collection through finalizers. (This is a mistake for the reason outlined below.) It is important to understand that you can *never* force* a gc in java. Even calling System.gc() is merely a hint to the VM. What you are doing is telling the VM that you are * willing* to give up some processor time right now to gc, how much it choses to actually collect or not collect is totally up to the VM. The *only* garbage collection guarantee in java is that it will make a best effort to collect what it can to avoid an out of memory exception at the time that it runs out of memory. You are not guaranteed when *if ever*, a given object will actually be collected. Since finalizers happen when an object is collected, and not when it becomes a candidate for collection, the same is true of the finalizer. You are not guaranteed when, if ever, it will run. On Fri, Aug 31, 2012 at 9:03 AM, Alexander Shutyaev shuty...@gmail.comwrote: Hi All! I have a problem with using cassandra. Our application does a lot of overwrites and deletes. If I understand correctly cassandra does not actually delete these objects until gc_grace seconds have passed. I tried to force gc by setting gc_grace to 0 on an existing column family and running major compaction afterwards. However I did not get disk space back, although I'm pretty much sure that my column family should occupy many times fewer space. We have also a PostgreSQL db and we duplicate each operation with data in both dbs. And the PosgreSQL table is much more smaller than the corresponding cassandra's column family. Does anyone have any suggestions on how can I analyze my problem? Or maybe I'm doing something wrong and there is another way to force gc on an existing column family. Thanks in advance, Alexander -- It's always darkest just before you are eaten by a grue.
