Re: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released
On Mon, Apr 16, 2012 at 10:45 PM, Bryce Godfrey bryce.godf...@azaleos.com wrote: I keep running into this with my testing (on a windows box), Is this just a OOM for RAM? How much RAM do you have? Do you use completely standard settings? Do you also OOM if you try the same test with Cassandra 1.0.9? -- Sylvain ERROR [COMMIT-LOG-ALLOCATOR] 2012-04-16 13:36:18,790 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Map failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.freshSegment(CommitLogSegment.java:80) at org.apache.cassandra.db.commitlog.CommitLogAllocator.createFreshSegment(CommitLogAllocator.java:244) at org.apache.cassandra.db.commitlog.CommitLogAllocator.access$500(CommitLogAllocator.java:49) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:104) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(Unknown Source) at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:119) ... 6 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) ... 8 more INFO [StorageServiceShutdownHook] 2012-04-16 13:36:18,961 CassandraDaemon.java (line 218) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2012-04-16 13:36:18,961 MessagingService.java (line 539) Waiting for messaging service to quiesce INFO [ACCEPT-/10.47.1.15] 2012-04-16 13:36:18,977 MessagingService.java (line 695) MessagingService shutting down server thread. -Original Message- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: Friday, April 13, 2012 9:41 AM To: user@cassandra.apache.org Subject: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released The Cassandra team is pleased to announce the release of the first release candidate for the future Apache Cassandra 1.1. Please first note that this is a release candidate, *not* the final release yet. All help in testing this release candidate will be greatly appreciated. Please report any problem you may encounter[3,4] and have a look at the change log[1] and the release notes[2] to see where Cassandra 1.1 differs from the previous series. Apache Cassandra 1.1.0-rc1[5] is available as usual from the cassandra website (http://cassandra.apache.org/download/) and a debian package is available using the 11x branch (see http://wiki.apache.org/cassandra/DebianPackaging). Thank you for your help in testing and have fun with it. [1]: http://goo.gl/XwH7J (CHANGES.txt) [2]: http://goo.gl/JocLX (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA [4]: user@cassandra.apache.org [5]: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/cassandra-1.1.0-rc1
Re: Poor write performance with seconrady index
Secondary indexes require a read and a write (potentially two) for every
update. Regular mutations are no look writes and are much faster.
Just like in a RDBMS, it's more efficient to insert data and then create the
index than to insert data with the index present.
An alternative is to create SSTables in the hadoop jobs and bulk load them into
the cluster.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 17/04/2012, at 2:51 AM, Patrik Modesto wrote:
Hi,
I've a 4 node test cluster running Cassandra 1.0.9, 32GB memory, 4x
1TB disks. I've two keyspaces, rfTest2 (RF=2) and rfTest3 (RF=3).
There are two CF, one with source data and one with secondary index:
create column family UrlGroup
with column_type=Standard
and comparator=UTF8Type
and default_validation_class=UTF8Type
and key_validation_class=UTF8Type
and column_metadata=
[{
column_name: groupId,
validation_class: UTF8Type,
index_type: KEYS
}];
I'm running Hadoop mapreduce job, reading the source CF and creating 3
mutations for each row-key in the UrlGroup CF.
The mapreduce runs for 30minutes. When I remove the secondary index,
the mapreduce runs just 10minutes. There are 26,273,544 mutations
total.
Also with the secondary index, the nodes show very high load 50+ and
iowait 70%+. Without secondary index the load is ~5 and iowait ~10%.
What may be the problem?
Regards,
Patrik
Re: Cassandra Network Protocols
Hand rolled headers and binary serialisation over TCP. See o.a.c.net.IncomingTcpConnection for the starting point for an incoming connection. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 9:49 AM, Tom Duffield (Mailing Lists) wrote: When you submit a write request with a constancy of EACH_QUORUM, what network protocol does Cassandra use to write the data to the other DCs? Similarly, what network protocol does Cassandra use when writing data as part of replication? -- Tom Duffield (Mailing Lists) Sent with Sparrow
Re: java.nio.BufferOverflowException from cassandra server
What version are you on ? it's on odd error for sure. Try eliminates changes until you get to a stable base: use the default install out of the box and the tools/stress app in the source distro. If that works try it with your C++ client. Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 11:43 AM, Aniket Chakrabarti wrote: Hi, I have set up a 4 node cassandra cluster. I am using the Thrift C++ API to write a simple C++ application with creates a 50% READ 50% WRITE requests. Every time near about a thousand request mark, I am getting the following exception and my connection is broken: === ERROR 17:30:27,647 Error occurred during processing of message. java.nio.BufferOverflowException at java.nio.charset.CoderResult.throwException(Unknown Source) at java.lang.StringCoding$StringEncoder.encode(Unknown Source) at java.lang.StringCoding.encode(Unknown Source) at java.lang.String.getBytes(Unknown Source) at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:185) at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:92) at org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:3302) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) == Some info about the config I am using: - It is a 4 node cluster with only 1 seed. -The consistency level is also set to ONE. -The max heap size and new heap size is set to 4G and 800M(I tried without setting them as well) -Java is run in the interpreted mode(-Xint) -I'm using user mode linux Any pointers to what I might be doing wrong will be very helpful. Thanks in advance, Aniket
Re: Is the secondary index re-built under compaction?
Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim
Re: Poor write performance with seconrady index
Hi Aaron, thanks for the reply. I suspected it might be the
read-and-write that causes the slower updates.
Regards,
P.
On Tue, Apr 17, 2012 at 11:52, aaron morton aa...@thelastpickle.com wrote:
Secondary indexes require a read and a write (potentially two) for every
update. Regular mutations are no look writes and are much faster.
Just like in a RDBMS, it's more efficient to insert data and then create the
index than to insert data with the index present.
An alternative is to create SSTables in the hadoop jobs and bulk load them
into the cluster.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 17/04/2012, at 2:51 AM, Patrik Modesto wrote:
Hi,
I've a 4 node test cluster running Cassandra 1.0.9, 32GB memory, 4x
1TB disks. I've two keyspaces, rfTest2 (RF=2) and rfTest3 (RF=3).
There are two CF, one with source data and one with secondary index:
create column family UrlGroup
with column_type=Standard
and comparator=UTF8Type
and default_validation_class=UTF8Type
and key_validation_class=UTF8Type
and column_metadata=
[{
column_name: groupId,
validation_class: UTF8Type,
index_type: KEYS
}];
I'm running Hadoop mapreduce job, reading the source CF and creating 3
mutations for each row-key in the UrlGroup CF.
The mapreduce runs for 30minutes. When I remove the secondary index,
the mapreduce runs just 10minutes. There are 26,273,544 mutations
total.
Also with the secondary index, the nodes show very high load 50+ and
iowait 70%+. Without secondary index the load is ~5 and iowait ~10%.
What may be the problem?
Regards,
Patrik
User authorized for modify-keyspace cannot create CFs
Hi, I'm suffering a problem, which maybe is a feature ( ;) ), but for me it's rather an annoying problem. I use SimpleAuthenticator and I have user who should be a kind of Cassandra's keyspace root - he should be allowed to do everything. So I set: modify-keyspaces=master Unluckily, when I try to create CFs (with a script) I get: pycassa.cassandra.c10.ttypes.InvalidRequestException: InvalidRequestException(why='#User master groups=[] does not have permission WRITE for /cassandra/keyspaces/test') It's not a problem of logging in, because if I provide wrong password I get different error: pycassa.cassandra.c10.ttypes.AuthenticationException: AuthenticationException(why='Given password in password mode MD5 could not be validated for user master') When I go back to AllowAllAuthority for a while, create CFs and then move back to SimpleAuthority (user has rw on all CFs) it's working perfectly to manipulate the data in CFs in this keyspace. But altering CFs settings still fails. So, the question is - is it intended to work this way, that user with modify-keyspace permission cannot - sic! - modify the keyspace in terms of creating/altering CF? Is it a bug, or a feature? If feature - why? :) Or maybe this setting is for other purposes and I just don't understand the idea? ;) Kind regards, Michał M.
Re: Is the secondary index re-built under compaction?
Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim
Re: Is the secondary index re-built under compaction?
No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.gov wrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim -- http://twitter.com/tjake
Re: exceptions after upgrading from 1.0.7 to 1.0.9
You need to install cql driver for python as it says. % easy_install cql If you don't have easy_install, you need to install it first. You will be able to find easy_install by querying easy_install python on google. maki On 2012/04/17, at 20:18, Tamar Fraenkel ta...@tok-media.com wrote: Thanks for answering! I unzipped the cassandra taken from http://off.co.il/apache/cassandra/1.0.9/apache-cassandra-1.0.9-bin.tar.gz. I changed the cassandra init script to run from the new installation (CASSANDRA_HOME). I retried it, and found out that the reason Cassandra didn't start the previous time was a typo in the init script. So now Cassandra 1.0.9 is up, but cqlsh still give me the following error, even when I make sure it is started from the 1.0.9 bin (Cassandra-cli works well): Python CQL driver not installed, or not on PYTHONPATH. You might try easy_install cql. Python: /usr/bin/python Module load path: ['/usr/share/apache-cassandra-1.0.9/bin/../pylib', '/usr/share/apache-cassandra-1.0.9/bin', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/ubuntuone-client', '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel', '/usr/lib/python2.7/dist-packages/ubuntuone-couch', '/usr/lib/python2.7/dist-packages/ubuntuone-installer', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol'] Error: No module named cql Also, do you have a good step by step upgrade guide for tar.gz? Thanks, Tamar Fraenkel Senior Software Engineer, TOK Media ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Tue, Apr 17, 2012 at 12:15 PM, aaron morton aa...@thelastpickle.com wrote: 1. No cql is installed now. Do I need to download and install separately? If you have just unzipped the bin distrobution nothing will be installed, it will only be in the unzipped file locations. cqlsh is in the bin/ directory. As well as the datastax packages there are Debian packages from Apache, see http://cassandra.apache.org/download/ 2. Cassandra won't start, I got the following exception below. This message INFO [RMI TCP Connection(2)-127.0.0.1] 2012-04-16 00:25:37,980 StorageService.java (line 667) DRAINED Says that the sever was drained via the JMX / Node Tool interface. The RejectedExecutionError that followed are a result of the server shutting down. 3. While at it, does someone know if Hector 1.0.3 supports Cassandra 1.0.9? It should do, cassandra 1.0.9 is compatible with previous 1.0.X releases. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 16/04/2012, at 7:54 PM, Tamar Fraenkel wrote: Hi! I had datastax 1.0.7 installed on ubuntu. I downloaded http://off.co.il/apache/cassandra/1.0.9/apache-cassandra-1.0.9-bin.tar.gz and unzipped it. I left both versions installed, but changed my service script to start the 1.0.9. Two problems: 1. No cql is installed now. Do I need to download and install separately? 2. Cassandra won't start, I got the following exception below. 3. While at it, does someone know if Hector 1.0.3 supports Cassandra 1.0.9? Thanks, Tamar INFO [FlushWriter:2] 2012-04-16 00:25:37,879 Memtable.java (line 283) Completed flushing /var/lib/cassandra/data/OpsCenter/events_timeline-hc-28-Data.db (79 bytes) INFO [RMI TCP Connection(2)-127.0.0.1] 2012-04-16 00:25:37,980 StorageService.java (line 667) DRAINED ERROR [CompactionExecutor:3] 2012-04-16 00:25:38,021 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:3,1,RMI Runtime] java.util.concurrent.RejectedExecutionException at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215) at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397) at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470) at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:67) at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:742) at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:316)
Re: Is the secondary index re-built under compaction?
Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction stats, and tons of index files created in the data directory. Plus it does tell me that it is building the secondary index, and that seems to be happening at an amazingly glacial pace. I have 2 CFs there, with multiple secondary indexes. I'll try to compact the CF one by one, reboot and see if that helps. Maxim On 4/17/2012 9:53 AM, Jake Luciani wrote: No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim -- http://twitter.com/tjake
Re: Is the secondary index re-built under compaction?
Well, the since the secondary indexes are themselves column families they too are compacted along with everything else. On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin potek...@bnl.gov wrote: Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction stats, and tons of index files created in the data directory. Plus it does tell me that it is building the secondary index, and that seems to be happening at an amazingly glacial pace. I have 2 CFs there, with multiple secondary indexes. I'll try to compact the CF one by one, reboot and see if that helps. Maxim On 4/17/2012 9:53 AM, Jake Luciani wrote: No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.gov wrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim -- http://twitter.com/tjake -- http://twitter.com/tjake
Re: Is the secondary index re-built under compaction?
I understand that indexes are CFs. But the compaction stats says it's building the index, not compacting the corresponding CF. Either that's an ambiguous diagnostic, or indeed something is not right with my rig as of late. Maxim On 4/17/2012 10:05 AM, Jake Luciani wrote: Well, the since the secondary indexes are themselves column families they too are compacted along with everything else. On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction stats, and tons of index files created in the data directory. Plus it does tell me that it is building the secondary index, and that seems to be happening at an amazingly glacial pace. I have 2 CFs there, with multiple secondary indexes. I'll try to compact the CF one by one, reboot and see if that helps. Maxim On 4/17/2012 9:53 AM, Jake Luciani wrote: No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim -- http://twitter.com/tjake -- http://twitter.com/tjake
Re: Is the secondary index re-built under compaction?
Hmm that does sound fishy. When you run show keyspaces from cassandra-cli it shows which indexes are built. Are they marked built in your column family? -Jake On Tue, Apr 17, 2012 at 10:09 AM, Maxim Potekhin potek...@bnl.gov wrote: I understand that indexes are CFs. But the compaction stats says it's building the index, not compacting the corresponding CF. Either that's an ambiguous diagnostic, or indeed something is not right with my rig as of late. Maxim On 4/17/2012 10:05 AM, Jake Luciani wrote: Well, the since the secondary indexes are themselves column families they too are compacted along with everything else. On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin potek...@bnl.gov wrote: Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction stats, and tons of index files created in the data directory. Plus it does tell me that it is building the secondary index, and that seems to be happening at an amazingly glacial pace. I have 2 CFs there, with multiple secondary indexes. I'll try to compact the CF one by one, reboot and see if that helps. Maxim On 4/17/2012 9:53 AM, Jake Luciani wrote: No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.gov wrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim -- http://twitter.com/tjake -- http://twitter.com/tjake -- http://twitter.com/tjake
Re: Is the secondary index re-built under compaction?
Yes. Sorry I didn't mention this, but of course I'm checking on indexes once in a while. So yes, they are marked as built. All of this started happening after a few days of continuous loading process. Since the nodes have good hardware (24 cores + SSD), the apparent load on each node was nothing remarkable, even at 20kHz insertion rate. But maybe I'm being overoptimistic. Maxim On 4/17/2012 10:12 AM, Jake Luciani wrote: Hmm that does sound fishy. When you run show keyspaces from cassandra-cli it shows which indexes are built. Are they marked built in your column family? -Jake On Tue, Apr 17, 2012 at 10:09 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: I understand that indexes are CFs. But the compaction stats says it's building the index, not compacting the corresponding CF. Either that's an ambiguous diagnostic, or indeed something is not right with my rig as of late. Maxim On 4/17/2012 10:05 AM, Jake Luciani wrote: Well, the since the secondary indexes are themselves column families they too are compacted along with everything else. On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction stats, and tons of index files created in the data directory. Plus it does tell me that it is building the secondary index, and that seems to be happening at an amazingly glacial pace. I have 2 CFs there, with multiple secondary indexes. I'll try to compact the CF one by one, reboot and see if that helps. Maxim On 4/17/2012 9:53 AM, Jake Luciani wrote: No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim -- http://twitter.com/tjake -- http://twitter.com/tjake -- http://twitter.com/tjake
Re: Is the secondary index re-built under compaction?
How many indexes are there? On Tue, Apr 17, 2012 at 10:16 AM, Maxim Potekhin potek...@bnl.gov wrote: Yes. Sorry I didn't mention this, but of course I'm checking on indexes once in a while. So yes, they are marked as built. All of this started happening after a few days of continuous loading process. Since the nodes have good hardware (24 cores + SSD), the apparent load on each node was nothing remarkable, even at 20kHz insertion rate. But maybe I'm being overoptimistic. Maxim On 4/17/2012 10:12 AM, Jake Luciani wrote: Hmm that does sound fishy. When you run show keyspaces from cassandra-cli it shows which indexes are built. Are they marked built in your column family? -Jake On Tue, Apr 17, 2012 at 10:09 AM, Maxim Potekhin potek...@bnl.govwrote: I understand that indexes are CFs. But the compaction stats says it's building the index, not compacting the corresponding CF. Either that's an ambiguous diagnostic, or indeed something is not right with my rig as of late. Maxim On 4/17/2012 10:05 AM, Jake Luciani wrote: Well, the since the secondary indexes are themselves column families they too are compacted along with everything else. On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin potek...@bnl.govwrote: Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction stats, and tons of index files created in the data directory. Plus it does tell me that it is building the secondary index, and that seems to be happening at an amazingly glacial pace. I have 2 CFs there, with multiple secondary indexes. I'll try to compact the CF one by one, reboot and see if that helps. Maxim On 4/17/2012 9:53 AM, Jake Luciani wrote: No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.govwrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim -- http://twitter.com/tjake -- http://twitter.com/tjake -- http://twitter.com/tjake -- http://twitter.com/tjake
Re: Is the secondary index re-built under compaction?
The offending CF only has one. The other one, that seems to behave well, has nine. Maxim On 4/17/2012 10:20 AM, Jake Luciani wrote: How many indexes are there? On Tue, Apr 17, 2012 at 10:16 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Yes. Sorry I didn't mention this, but of course I'm checking on indexes once in a while. So yes, they are marked as built. All of this started happening after a few days of continuous loading process. Since the nodes have good hardware (24 cores + SSD), the apparent load on each node was nothing remarkable, even at 20kHz insertion rate. But maybe I'm being overoptimistic. Maxim On 4/17/2012 10:12 AM, Jake Luciani wrote: Hmm that does sound fishy. When you run show keyspaces from cassandra-cli it shows which indexes are built. Are they marked built in your column family? -Jake On Tue, Apr 17, 2012 at 10:09 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: I understand that indexes are CFs. But the compaction stats says it's building the index, not compacting the corresponding CF. Either that's an ambiguous diagnostic, or indeed something is not right with my rig as of late. Maxim On 4/17/2012 10:05 AM, Jake Luciani wrote: Well, the since the secondary indexes are themselves column families they too are compacted along with everything else. On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction stats, and tons of index files created in the data directory. Plus it does tell me that it is building the secondary index, and that seems to be happening at an amazingly glacial pace. I have 2 CFs there, with multiple secondary indexes. I'll try to compact the CF one by one, reboot and see if that helps. Maxim On 4/17/2012 9:53 AM, Jake Luciani wrote: No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary index while I initiate compaction. Is this to be expected? Cassandra version 0.8.8. Thank you Maxim -- http://twitter.com/tjake -- http://twitter.com/tjake -- http://twitter.com/tjake -- http://twitter.com/tjake
Re: exceptions after upgrading from 1.0.7 to 1.0.9
Thanks!!! Two simple actions 1. sudo apt-get install python-setuptools 2. sudo easy_install cql And it did the trick! But just to be on the safe side, before I move to upgrade our staging environment, does anyone know a detailed description of how to upgrade cassandra installed from tar.gz? or how to upgrade Amazon EC2 datastax AMI? Thanks, *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Tue, Apr 17, 2012 at 4:56 PM, Watanabe Maki watanabe.m...@gmail.comwrote: You need to install cql driver for python as it says. % easy_install cql If you don't have easy_install, you need to install it first. You will be able to find easy_install by querying easy_install python on google. maki On 2012/04/17, at 20:18, Tamar Fraenkel ta...@tok-media.com wrote: Thanks for answering! I unzipped the cassandra taken from http://off.co.il/apache/cassandra/1.0.9/apache-cassandra-1.0.9-bin.tar.gz. I changed the cassandra init script to run from the new installation (CASSANDRA_HOME). I retried it, and found out that the reason Cassandra didn't start the previous time was a typo in the init script. So now Cassandra 1.0.9 is up, but cqlsh still give me the following error, even when I make sure it is started from the 1.0.9 bin (Cassandra- cli works well): Python CQL driver not installed, or not on PYTHONPATH. You might try easy_install cql. Python: /usr/bin/python Module load path: ['/usr/share/apache-cassandra-1.0.9/bin/../pylib', '/usr/share/apache-cassandra-1.0.9/bin', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/ubuntuone-client', '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel', '/usr/lib/python2.7/dist-packages/ubuntuone-couch', '/usr/lib/python2.7/dist-packages/ubuntuone-installer', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol'] Error: No module named cql Also, do you have a good step by step upgrade guide for tar.gz? Thanks, *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Tue, Apr 17, 2012 at 12:15 PM, aaron morton aa...@thelastpickle.comwrote: 1. No cql is installed now. Do I need to download and install separately? If you have just unzipped the bin distrobution nothing will be installed, it will only be in the unzipped file locations. cqlsh is in the bin/ directory. As well as the datastax packages there are Debian packages from Apache, see http://cassandra.apache.org/download/ 2. Cassandra won't start, I got the following exception below. This message INFO [RMI TCP Connection(2)-127.0.0.1] 2012-04-16 00:25:37,980 StorageService.java (line 667) DRAINED Says that the sever was drained via the JMX / Node Tool interface. The RejectedExecutionError that followed are a result of the server shutting down. 3. While at it, does someone know if Hector 1.0.3 supports Cassandra 1.0.9? It should do, cassandra 1.0.9 is compatible with previous 1.0.X releases. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 16/04/2012, at 7:54 PM, Tamar Fraenkel wrote: Hi! I had datastax 1.0.7 installed on ubuntu. I downloaded http://off.co.il/apache/cassandra/1.0.9/apache-cassandra-1.0.9-bin.tar.gzand unzipped it. I left both versions installed, but changed my service script to start the 1.0.9. Two problems: 1. No cql is installed now. Do I need to download and install separately? 2. Cassandra won't start, I got the following exception below. 3. While at it, does someone know if Hector 1.0.3 supports Cassandra 1.0.9? Thanks, Tamar INFO [FlushWriter:2] 2012-04-16 00:25:37,879 Memtable.java (line 283) Completed flushing /var/lib/cassandra /data/OpsCenter/events_timeline-hc-28-Data.db (79 bytes) INFO [RMI TCP Connection(2)-127.0.0.1] 2012-04-16 00:25:37,980 StorageService.java (line 667) DRAINED ERROR [CompactionExecutor:3] 2012-04-16 00:25:38,021 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ CompactionExecutor:3,1,RMI Runtime] java.util.concurrent.RejectedExecutionException at java.util.concurrent.ThreadPoolExecutor$AbortPolicy. rejectedExecution(ThreadPoolExecutor.java:1768) at java.util.concurrent.ThreadPoolExecutor.reject( ThreadPoolExecutor.java:767)
Multi Master replication : rejoining a node after split network
Hi all, We are evaluating Cassandra for a geographically distributed deployment that requires multi master replication. We have a few questions regarding how replication is handled in Cassandra, like: 1. Which mechanism is used to replicate the changes from one system to another: statement distribution or recording the changeset via triggers or storing the changeset in transaction log? 2. Since replication is continuous copying of changes from one node to another, these changes would have to be snapshotted in order to sustain temporary network failures so that replication can resume after the network problem is healed. is there a mechanism to define how long we can store/archive the snaphotted changes before we discard and would demand a recreation of node from the scratch rather than rejoin 3. What options are available for conflict resolution since we are talking about master-master replication across tens of nodes? 4. If a node is rejoined after a split network where same records would have been modified on multiple nodes, is there a mechanism to merge the data, resolve conflicts and eventually reach to a consistent state? Thanks and Regards, Samba
Counter column family
Hi! I want to understand how incrementing of counter works. - I have a 3 node ring, - I use FailoverPolicy.FAIL_FAST, - RF is 2, I have the following counter column family ColumnFamily: tk_counters Key Validation Class: org.apache.cassandra.db.marshal.CompositeType( org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal. UUIDType) Default column value validator: org.apache.cassandra.db.marshal. CounterColumnType Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache. SerializingCacheProvider Key cache size / save period in seconds: 0.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Bloom Filter FP chance: default Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy My CL for this column family is Write=2, Read=1. When I increment a counter (using hector mutator), and execute returns without errors, what is the status of the nodes at that stage. Can execute return before the nodes are really updated? So that if a read is done immediately after the increment it will still read the previous values? Thanks, *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 tokLogo.png
RE: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released
Sorry, I found the issue. The server I was using had 32bit java installed. -Original Message- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: Monday, April 16, 2012 11:39 PM To: user@cassandra.apache.org Subject: Re: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released On Mon, Apr 16, 2012 at 10:45 PM, Bryce Godfrey bryce.godf...@azaleos.com wrote: I keep running into this with my testing (on a windows box), Is this just a OOM for RAM? How much RAM do you have? Do you use completely standard settings? Do you also OOM if you try the same test with Cassandra 1.0.9? -- Sylvain ERROR [COMMIT-LOG-ALLOCATOR] 2012-04-16 13:36:18,790 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Map failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSeg ment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.freshSegment(Commit LogSegment.java:80) at org.apache.cassandra.db.commitlog.CommitLogAllocator.createFreshSegmen t(CommitLogAllocator.java:244) at org.apache.cassandra.db.commitlog.CommitLogAllocator.access$500(Commit LogAllocator.java:49) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(Com mitLogAllocator.java:104) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30 ) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(Unknown Source) at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSeg ment.java:119) ... 6 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) ... 8 more INFO [StorageServiceShutdownHook] 2012-04-16 13:36:18,961 CassandraDaemon.java (line 218) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2012-04-16 13:36:18,961 MessagingService.java (line 539) Waiting for messaging service to quiesce INFO [ACCEPT-/10.47.1.15] 2012-04-16 13:36:18,977 MessagingService.java (line 695) MessagingService shutting down server thread. -Original Message- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: Friday, April 13, 2012 9:41 AM To: user@cassandra.apache.org Subject: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released The Cassandra team is pleased to announce the release of the first release candidate for the future Apache Cassandra 1.1. Please first note that this is a release candidate, *not* the final release yet. All help in testing this release candidate will be greatly appreciated. Please report any problem you may encounter[3,4] and have a look at the change log[1] and the release notes[2] to see where Cassandra 1.1 differs from the previous series. Apache Cassandra 1.1.0-rc1[5] is available as usual from the cassandra website (http://cassandra.apache.org/download/) and a debian package is available using the 11x branch (see http://wiki.apache.org/cassandra/DebianPackaging). Thank you for your help in testing and have fun with it. [1]: http://goo.gl/XwH7J (CHANGES.txt) [2]: http://goo.gl/JocLX (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA [4]: user@cassandra.apache.org [5]: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=re fs/tags/cassandra-1.1.0-rc1
Re: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released
64bit is recommended where that's available. If you actually did have a 32bit machine or VM, then you should dramatically reduce the commitlog space cap to the minimum of 128MB so it doesn't need to mmap so much. On Tue, Apr 17, 2012 at 1:45 PM, Bryce Godfrey bryce.godf...@azaleos.com wrote: Sorry, I found the issue. The server I was using had 32bit java installed. -Original Message- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: Monday, April 16, 2012 11:39 PM To: user@cassandra.apache.org Subject: Re: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released On Mon, Apr 16, 2012 at 10:45 PM, Bryce Godfrey bryce.godf...@azaleos.com wrote: I keep running into this with my testing (on a windows box), Is this just a OOM for RAM? How much RAM do you have? Do you use completely standard settings? Do you also OOM if you try the same test with Cassandra 1.0.9? -- Sylvain ERROR [COMMIT-LOG-ALLOCATOR] 2012-04-16 13:36:18,790 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Map failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSeg ment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.freshSegment(Commit LogSegment.java:80) at org.apache.cassandra.db.commitlog.CommitLogAllocator.createFreshSegmen t(CommitLogAllocator.java:244) at org.apache.cassandra.db.commitlog.CommitLogAllocator.access$500(Commit LogAllocator.java:49) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(Com mitLogAllocator.java:104) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30 ) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(Unknown Source) at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSeg ment.java:119) ... 6 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) ... 8 more INFO [StorageServiceShutdownHook] 2012-04-16 13:36:18,961 CassandraDaemon.java (line 218) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2012-04-16 13:36:18,961 MessagingService.java (line 539) Waiting for messaging service to quiesce INFO [ACCEPT-/10.47.1.15] 2012-04-16 13:36:18,977 MessagingService.java (line 695) MessagingService shutting down server thread. -Original Message- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: Friday, April 13, 2012 9:41 AM To: user@cassandra.apache.org Subject: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released The Cassandra team is pleased to announce the release of the first release candidate for the future Apache Cassandra 1.1. Please first note that this is a release candidate, *not* the final release yet. All help in testing this release candidate will be greatly appreciated. Please report any problem you may encounter[3,4] and have a look at the change log[1] and the release notes[2] to see where Cassandra 1.1 differs from the previous series. Apache Cassandra 1.1.0-rc1[5] is available as usual from the cassandra website (http://cassandra.apache.org/download/) and a debian package is available using the 11x branch (see http://wiki.apache.org/cassandra/DebianPackaging). Thank you for your help in testing and have fun with it. [1]: http://goo.gl/XwH7J (CHANGES.txt) [2]: http://goo.gl/JocLX (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA [4]: user@cassandra.apache.org [5]: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=re fs/tags/cassandra-1.1.0-rc1 -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: size tiered compaction - improvement
On Sat, Apr 14, 2012 at 3:27 AM, Radim Kolar h...@filez.com wrote: forceUserDefinedCompaction would be more usefull if you could do compaction on 2 tables. You absolutely can. That's what the user defined part is: you give it the exact list of sstables you want compacted. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: size tiered compaction - improvement
On Sat, Apr 14, 2012 at 4:08 AM, Igor i...@4friends.od.ua wrote: Assume I insert all my data with TTL=2weeks and let we have sstable A which was created week ago at the time T, so I know that right now it contain: 1) some data that were inserted not later than T and may-be not expired yet 2) some amount of data that were already close to expiration due TTL at the time T, but still had no chances to be wiped out because up to the current moment size-tiered compaction did not involve A into compactions. Large amount of data from 2) became expired in a week after time T and probably passed gc_grace period, so it shoould be wiped at any compaction on table A. Any compaction pass over A will first convert the TTL data into tombstones. Then, any subsequent pass that includes A *and all other sstables containing rows with the same key* will drop the tombstones. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Off-heap row cache and mmapped sstables
Absolutely. Best practice is still to disable swap entirely on server machines; mlockall is just our best attempt to at least keep your JVM from swapping if you've forgotten this. On Thu, Apr 12, 2012 at 11:15 AM, Omid Aladini omidalad...@gmail.com wrote: Hi, Cassandra issues an mlockall [1] before mmap-ing sstables to prevent the kernel from paging out heap space in favor of memory-mapped sstables. I was wondering, what happens to the off-heap row cache (saved or unsaved)? Is it possible that the kernel pages out off-heap row cache in favor of resident mmap-ed sstable pages? Thanks, Omid [1] http://pubs.opengroup.org/onlinepubs/007908799/xsh/mlockall.html -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Long start-up times
On Sun, Apr 15, 2012 at 2:47 PM, sj.climber sj.clim...@gmail.com wrote: Also, I see in 1.0.9 there's a fix for a potentially related issue (see https://issues.apache.org/jira/browse/CASSANDRA-4023). Any thoughts on this? My thought is, upgrading is a no-brainer if that's a pain point for you. :) -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: swap grows
Swappiness is actually a fairly weak hint to linux: http://www.linuxvox.com/2009/10/what-is-the-linux-kernel-parameter-vm-swappiness On Sat, Apr 14, 2012 at 1:39 PM, aaron morton aa...@thelastpickle.com wrote: From https://help.ubuntu.com/community/SwapFaq swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible If you have swap enabled at some point the OS may swap out pages, even if swappiness is 0 and you have free memory. Disable swap entirely if you want to avoid this. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 15/04/2012, at 1:37 AM, R. Verlangen wrote: Maybe it has got something to do with swapiness, it's something you can configure, more info here: https://www.linux.com/news/software/applications/8208-all-about-linux-swap-space 2012/4/14 ruslan usifov ruslan.usi...@gmail.com I know:-) but this is not answer:-(. I found that on other nodes there still about 3GB (on node with JAVA_HEAP=6GB free memory also 3GB) of free memory but there JAVA_HEAP=5G, so this looks like some sysctl (/proc/sys/vm???) ratio (about 10%(3 / 24 * 100)), i don't known which, anybody can explain this situation 2012/4/14 R. Verlangen ro...@us2.nl Its recommended to disable swap entirely when you run Cassandra on a server. 2012/4/14 ruslan usifov ruslan.usi...@gmail.com I forgot to say that system have 24GB of phis memory 2012/4/14 ruslan usifov ruslan.usi...@gmail.com Hello We have 6 node cluster (cassandra 0.8.10). On one node i increase java heap size to 6GB, and now at this node begin grows swap, but system have about 3GB of free memory: root@6wd003:~# free total used free shared buffers cached Mem: 24733664 21702812 3030852 0 6792 13794724 -/+ buffers/cache: 7901296 16832368 Swap: 1998840 2352 1996488 And swap space slowly grows, but i misunderstand why? PS: We have JNA mlock, and set vm.swappiness = 0 PS: OS ubuntu 10.0.4(2.6.32-40-generic) -- With kind regards, Robin Verlangen www.robinverlangen.nl -- With kind regards, Robin Verlangen www.robinverlangen.nl -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: java.nio.BufferOverflowException from cassandra server
If I were to take a wild guess, it would be that you're using a single Thrift connection in multiple threads, which isn't supported. On Mon, Apr 16, 2012 at 6:43 PM, Aniket Chakrabarti chakr...@cse.ohio-state.edu wrote: Hi, I have set up a 4 node cassandra cluster. I am using the Thrift C++ API to write a simple C++ application with creates a 50% READ 50% WRITE requests. Every time near about a thousand request mark, I am getting the following exception and my connection is broken: === ERROR 17:30:27,647 Error occurred during processing of message. java.nio.BufferOverflowException at java.nio.charset.CoderResult.throwException(Unknown Source) at java.lang.StringCoding$StringEncoder.encode(Unknown Source) at java.lang.StringCoding.encode(Unknown Source) at java.lang.String.getBytes(Unknown Source) at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:185) at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:92) at org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:3302) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) == Some info about the config I am using: - It is a 4 node cluster with only 1 seed. -The consistency level is also set to ONE. -The max heap size and new heap size is set to 4G and 800M(I tried without setting them as well) -Java is run in the interpreted mode(-Xint) -I'm using user mode linux Any pointers to what I might be doing wrong will be very helpful. Thanks in advance, Aniket -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: size tiered compaction - improvement
Thank you Jonatathan, I missed this point about converting TTL data to tombstones first. When you say: You absolutely can. That's what the user defined part is: you give it the exact list of sstables you want compacted. does it mean that I can use list (not just one) of sstables as second parameter for userDefinedCompaction? On 04/18/2012 05:53 AM, Jonathan Ellis wrote: On Sat, Apr 14, 2012 at 4:08 AM, Igori...@4friends.od.ua wrote: Assume I insert all my data with TTL=2weeks and let we have sstable A which was created week ago at the time T, so I know that right now it contain: 1) some data that were inserted not later than T and may-be not expired yet 2) some amount of data that were already close to expiration due TTL at the time T, but still had no chances to be wiped out because up to the current moment size-tiered compaction did not involve A into compactions. Large amount of data from 2) became expired in a week after time T and probably passed gc_grace period, so it shoould be wiped at any compaction on table A. Any compaction pass over A will first convert the TTL data into tombstones. Then, any subsequent pass that includes A *and all other sstables containing rows with the same key* will drop the tombstones.
Re: size tiered compaction - improvement
On Tue, Apr 17, 2012 at 11:26 PM, Igor i...@4friends.od.ua wrote: You absolutely can. That's what the user defined part is: you give it the exact list of sstables you want compacted. does it mean that I can use list (not just one) of sstables as second parameter for userDefinedCompaction? If you want them all compacted together into one big sstable, yes. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
