Repair has now effect
Hey folks, perhaps a dumb question but I ran into this situation and it's a bit unclear what's going on. We are running a 3 nodes cluster with RF 3 (cass 1.0.11). We had one issue with one node and it was down for like 1 hour. I brought the node up again as soon as I realized it was down (and ganglia + nagios triggered some alerts). But when running repair to make sure it had all the data, I get this message: INFO [AntiEntropySessions:6] 2012-09-02 15:46:23,022 AntiEntropyService.java (line 663) [repair #%s] No neighbors to repair with on range %s: session completed Am I missing something ? thanks
Re: Repair has now effect
What does nt ring show (on this node and on the other two)? That may provide some clues. From: Patricio Echagüe patric...@gmail.commailto:patric...@gmail.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Sun, 2 Sep 2012 15:50:31 -0700 To: cassandra-u...@incubator.apache.orgmailto:cassandra-u...@incubator.apache.org Subject: Repair has now effect Hey folks, perhaps a dumb question but I ran into this situation and it's a bit unclear what's going on. We are running a 3 nodes cluster with RF 3 (cass 1.0.11). We had one issue with one node and it was down for like 1 hour. I brought the node up again as soon as I realized it was down (and ganglia + nagios triggered some alerts). But when running repair to make sure it had all the data, I get this message: INFO [AntiEntropySessions:6] 2012-09-02 15:46:23,022 AntiEntropyService.java (line 663) [repair #%s] No neighbors to repair with on range %s: session completed Am I missing something ? thanks
Re: performance is drastically degraded after 0.7.8 -- 1.0.11 upgrade
The whole test run is taking longer ? So it could be slower queries or slower test setup / tear down? If you are creating and truncate the KS for each of the 500 tests is that taking longer ? (Schema code has changed a lot 0.7 1.0) Can you log the execution time for tests and find ones that are taking longer ? There are full request metrics available on the StorageProxy JMX object. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 31/08/2012, at 4:45 PM, Илья Шипицин chipits...@gmail.com wrote: we are using functional tests ( ~500 tests in time). it is hard to tell which query is slower, it is slower in general. same hardware. 1 node, 32Gb RAM, 8Gb heap. default cassandra settings. as we are talking about functional tests, so we recreate KS just before tests are run. I do not know how to record queries (there are a lot of them), if you are interested, I can set up a special stand for you. 2012/8/31 aaron morton aa...@thelastpickle.com we are running somewhat queue-like with aggressive write-read patterns. We'll need some more details… How much data ? How many machines ? What is the machine spec ? How many clients ? Is there an example of a slow request ? How are you measuring that it's slow ? Is there anything unusual in the log ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 31/08/2012, at 3:30 AM, Edward Capriolo edlinuxg...@gmail.com wrote: If you move from 7.X to 0.8X or 1.0X you have to rebuild sstables as soon as possible. If you have large bloomfilters you can hit a bug where the bloom filters will not work properly. On Thu, Aug 30, 2012 at 9:44 AM, Илья Шипицин chipits...@gmail.com wrote: we are running somewhat queue-like with aggressive write-read patterns. I was looking for scripting queries from live Cassandra installation, but I didn't find any. is there something like thrift-proxy or other query logging/scripting engine ? 2012/8/30 aaron morton aa...@thelastpickle.com in terms of our high-rate write load cassandra1.0.11 is about 3 (three!!) times slower than cassandra-0.7.8 We've not had any reports of a performance drop off. All tests so far have show improvements in both read and write performance. I agree, such digests save some network IO, but they seem to be very bad in terms of CPU and disk IO. The sha1 is created so we can diagnose corruptions in the -Data component of the SSTables. They are not used to save network IO. It is calculated while streaming the Memtable to disk so has no impact on disk IO. While not the fasted algorithm I would assume it's CPU overhead in this case is minimal. there's already relatively small Bloom filter file, which can be used for saving network traffic instead of sha1 digest. Bloom filters are used to test if a row key may exist in an SSTable. any explanation ? If you can provide some more information on your use case we may be able to help. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 30/08/2012, at 5:18 AM, Илья Шипицин chipits...@gmail.com wrote: in terms of our high-rate write load cassandra1.0.11 is about 3 (three!!) times slower than cassandra-0.7.8 after some investigation carried out I noticed files with sha1 extension (which are missing for Cassandra-0.7.8) in maybeWriteDigest() function I see no option fot switching sha1 digests off. I agree, such digests save some network IO, but they seem to be very bad in terms of CPU and disk IO. why to use one more digest (which have to be calculated), there's already relatively small Bloom filter file, which can be used for saving network traffic instead of sha1 digest. any explanation ? Ilya Shipitsin
Re: force gc?
Hi Jeffrey, I think I described the problem wrong :) I don't want to do Java's memory GC. I want to do cassandra's GC - that is I want to really remove deleted rows from a column family and get my disc space back. 2012/8/31 Jeffrey Kesselman jef...@gmail.com Cassandra at least used to do disc cleanup as a side effect of garbage collection through finalizers. (This is a mistake for the reason outlined below.) It is important to understand that you can *never* force* a gc in java. Even calling System.gc() is merely a hint to the VM. What you are doing is telling the VM that you are * willing* to give up some processor time right now to gc, how much it choses to actually collect or not collect is totally up to the VM. The *only* garbage collection guarantee in java is that it will make a best effort to collect what it can to avoid an out of memory exception at the time that it runs out of memory. You are not guaranteed when *if ever*, a given object will actually be collected. Since finalizers happen when an object is collected, and not when it becomes a candidate for collection, the same is true of the finalizer. You are not guaranteed when, if ever, it will run. On Fri, Aug 31, 2012 at 9:03 AM, Alexander Shutyaev shuty...@gmail.comwrote: Hi All! I have a problem with using cassandra. Our application does a lot of overwrites and deletes. If I understand correctly cassandra does not actually delete these objects until gc_grace seconds have passed. I tried to force gc by setting gc_grace to 0 on an existing column family and running major compaction afterwards. However I did not get disk space back, although I'm pretty much sure that my column family should occupy many times fewer space. We have also a PostgreSQL db and we duplicate each operation with data in both dbs. And the PosgreSQL table is much more smaller than the corresponding cassandra's column family. Does anyone have any suggestions on how can I analyze my problem? Or maybe I'm doing something wrong and there is another way to force gc on an existing column family. Thanks in advance, Alexander -- It's always darkest just before you are eaten by a grue.
Re: Composite row keys with SSTableSimpleUnsortedWriter for Cassandra 1.0?
I think you want the o.a.c.db.marshal.TypeParser. You can pass a CLI format composite type to the parse() func. It's in 1.0X Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/09/2012, at 6:44 AM, Jeff Schmidt j...@535consulting.com wrote: Hello: I'm using DataStax Enterprise 2.1, which is based on Cassandra 1.0.10 from what I can tell. For my project, I perform a content build that generates a number of SSTables using SSTableSimpleUnsortedWriter. These are loaded using either JMX or sstableloader depending on the environment. I want to introduce a composite row key into some of the generated SSTables. Also, I will be referring to these keys by using composite column names. I can define the desired composite time and provide it to the SSTableSimpleUnsortedWriter constructor: ListAbstractType? compositeList = new ArrayListAbstractType?(); compositeList.add(UTF8Type.instance) compositeList.add(UTF8Type.instance) compositeUtf8Utf8Type = CompositeType.getInstance(compositeList) articleWriter = new SSTableSimpleUnsortedWriter( cassandraOutputDir, IngenuityContent, Articles, compositeUtf8Utf8Type, null, 64) I then figured I could use compositeUtf8Utf8Type when creating composite row keys and column names of the kind I require. Cassandra 1.1.x introduces the CompositeType.Builder class for creating actual composite values, but that's not available to me. I've also seen examples of using Hector's Composite to create composite values. But, I need to create these values using the various classes within Cassandra 1.0 itself to work with SSTableSimpleUnsortedWriter. For that, I'm not finding any examples on how one does that. As far as I can tell, composite columns at least have been around since Cassandra 0.8.x? Is there the support I need in Cassandra 1.0.x? Many thanks! Jeff -- Jeff Schmidt 535 Consulting j...@535consulting.com http://www.535consulting.com (650) 423-1068
Re: force gc?
I think I described the problem wrong :) I don't want to do Java's memory GC. I want to do cassandra's GC - that is I want to really remove deleted rows from a column family and get my disc space back. I think that was clear from your post. I don't see a problem with your process. Setting gc grace to 0 and forcing compaction should indeed return you to the smallest possible on-disk size. Did you really not see a *decrease*, or are you just comparing the final size with that of PostgreSQL? Keep in mind that in many cases (especially if not using compression) the Cassandra on-disk format is not as compact as PostgreSQL. For example column names are duplicated in each row, and the row key is duplicated twice (once in index, once in data). -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: force gc?
I think that was clear from your post. I don't see a problem with your process. Setting gc grace to 0 and forcing compaction should indeed return you to the smallest possible on-disk size. (But may be unsafe as documented; can cause deleted data to pop back up, etc.) -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: force gc?
Hi Peter, I don't compare it with PosgreSQL size, I just make some estimations.. This table / column family stores some xml documents with average raw size of 2Mb each and total size about 5Gb. However the space cassandra occupies on disc is 70Gb (after gc_grace was set to 0 and major compaction was run). Maybe there is some tool to analyze it? It would be great if I could somehow export each row of a column family into a separate file - so I could see their count and sizes. Is there any such tool? Or maybe you have some better thoughts... 2012/9/3 Peter Schuller peter.schul...@infidyne.com I think that was clear from your post. I don't see a problem with your process. Setting gc grace to 0 and forcing compaction should indeed return you to the smallest possible on-disk size. (But may be unsafe as documented; can cause deleted data to pop back up, etc.) -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: force gc?
Maybe there is some tool to analyze it? It would be great if I could somehow export each row of a column family into a separate file - so I could see their count and sizes. Is there any such tool? Or maybe you have some better thoughts... Use something like pycassa to non-obnoxiously iterate over all rows: for row_id, row in your_column_family.get_range(): https://github.com/pycassa/pycassa -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: force gc?
On Fri, Aug 31, 2012 at 7:03 AM, Alexander Shutyaev shuty...@gmail.comwrote: Does anyone have any suggestions on how can I analyze my problem? Or maybe I'm doing something wrong and there is another way to force gc on an existing column family. Are you using leveled compaction? I haven't looked into it too much, but I think forcing a major compaction when using leveled strategy doesn't have the same effect as with size tiered. -- Derek Williams
Re: force gc?
Hi Derek, I'm using size-tiered compaction. 2012/9/3 Derek Williams de...@fyrie.net On Fri, Aug 31, 2012 at 7:03 AM, Alexander Shutyaev shuty...@gmail.comwrote: Does anyone have any suggestions on how can I analyze my problem? Or maybe I'm doing something wrong and there is another way to force gc on an existing column family. Are you using leveled compaction? I haven't looked into it too much, but I think forcing a major compaction when using leveled strategy doesn't have the same effect as with size tiered. -- Derek Williams
