Re: Loading data on-demand in Cassandra
Hi, You do not need to have 700 Go of data in RAM. Cassandra is able to store on disks and query from there if data is not cached in memory. Caches are maintained by C* by itself but you still have to some configuration. Supposing you want to store around 800 Go and with a RF=3, you will need at least 6 servers if you want to store all data of your db (keeping max 400 Go per server) : 800x3/400=6. There is no native implementation of trigger in C*. Anyway, there is an extension bringing this feature: https://github.com/hmsonline/cassandra-triggers. This should allow you to be notified of mutations (ie: not query). Some peoples on this ML are involved in this, maybe they could help on this. Cheers, - Pierre -Original Message- From: Oliver Plohmann Date: Sun, 12 Aug 2012 21:24:43 To: Reply-To: user@cassandra.apache.org Subject: Loading data on-demand in Cassandra Hello, I'm looking a bit into Cassandra to see whether it would be something to go with for my company. I searched through the Internet, looked through the FAQs, etc. but there are still some few open questions. Hope I don't bother anybody with the usual beginner questions ... Is there a way to do load-on-demand of data in Cassandra? For the time being, we cannot afford to built up a cluster that holds our 700 GB SQL-Database in RAM. So we need to be able to load data on-demand from our relational database. Can this be done in Cassandra? Then there also needs to be a way to unload data in order to reclaim RAM space. Would be nice if it were possible to register for an asynchronous notification in case some value was changed. Can this be done? Thanks for any answers. Regards, Oliver
Re: DSE solr HA
You may have more luck on the DS forums http://www.datastax.com/support-forums/ Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/08/2012, at 6:00 AM, Mohit Anchlia wrote: > > Going through this page and it looks like indexes are stored locally > http://www.datastax.com/dev/blog/cassandra-with-solr-integration-details . My > question is what happens if one of the solr nodes crashes? Is the data > indexed again on those nodes? > > Also, if RF > 1 then is the same data being indexed on all RF nodes or is > that RF only for document replication? >
Re: [gem] does "disconnect!" work properly?
My rough understanding of the ruby code is that it auto connects if necessary
when you make a request.
i.e. your get turns into a _multiget call here
https://github.com/twitter/cassandra/blob/master/lib/cassandra/protocol.rb#L52
and that result is in a client connection if necessary
https://github.com/twitter/cassandra/blob/master/lib/cassandra/cassandra.rb#L1043
If you still feel there is an problem the Issue list of github is probably the
place to go https://github.com/twitter/cassandra/issues
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 13/08/2012, at 2:27 PM, Satoshi Yamada wrote:
>
> hi,
>
> I wonder if disconnect! method works properly in gem cassandra
> because the code below does not cause exception.
>
> -
>
> client = Cassandra.new('pool', host_ip)
>
> ret = client.get(:db, 'test', key, option_one)
> p ret
> client.disconnect!
>
> ret = client.get(:db, 'test', key, option_one)
> p ret
>
> -
>
> I use gem cassandra 0.14.0
> http://rubygems.org/gems/cassandra/versions/0.14.0
>
> thanks in advance,
> satoshi
Re: Problem with cassandra startup on Linux
Hi Dwight,
I can confirm that issue on my MBP under Mountain Lion.
Can you create a ticker at https://issues.apache.org/jira/browse/CASSANDRA and
include the platform you are running on.
For reference the change was added by
https://issues.apache.org/jira/browse/CASSANDRA-4447
The change is only relevant if you are running on Java 7. As a work around
change the relevant section of cassandra-env.sh to look like
#startswith () [ "${1#$2}" != "$1" ]
if [ "`uname`" = "Linux" ] ; then
# reduce the per-thread stack size to minimize the impact of Thrift
# thread-per-client. (Best practice is for client connections to
# be pooled anyway.) Only do so on Linux where it is known to be
# supported.
#if startswith "$JVM_VERSION" '1.7.'
#then
#JVM_OPTS="$JVM_OPTS -Xss160k"
#else
JVM_OPTS="$JVM_OPTS -Xss128k"
#fi
fi
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 13/08/2012, at 5:16 PM, Dwight Smith wrote:
> Installed 1.1.3 on my Linux cluster – the JVM_OPTS were truncated due to a
> script error in Cassandra-env.sh:
>
> Invalid token in the following.
>
>startswith () [ "${1#$2}" != "$1" ]
>
RE: RE Restore snapshot
> De : Sylvain Lebresne [mailto:sylv...@datastax.com] > > 2) copy the snapshot sstable in the right place and call the JMX > > method > > loadNewSSTables() (in the column family MBean, which mean you need to > > do that per-CF). > > How does this affect the contents of the CommitLogs? I mean, I > imagine using this for rollback'ing errors in the app side, like > some/many/all objects deleted. How will updates/deletes saved in > CommitLogs impact the data restored this way? Also: while I'm restoring > a KS of CF, is it possible to cut write requests but not reads? I hope > I'm explaining myself clearly... In particular, I'm thinking on a restore like this: * the app does something stupid. * (if possible) I stop writes to the KS or CF. * remove the db files with the wrong data. * I put the db files from the restore. * nodetool refresh * profit! But on Friday I did it slightly different (too tired, I was confused), and only did the refresh on one node and then a repair on the same, and it failed like this: * state just after the app did something stupid: root@pnscassandra04:~# tree /var/opt/hosting/db/cassandra/data/one_cf/cf_1/ /var/opt/hosting/db/cassandra/data/one_cf/cf_1/ |-- one_cf-cf_1-hd-13-CompressionInfo.db |-- one_cf-cf_1-hd-13-Data.db |-- one_cf-cf_1-hd-13-Filter.db |-- one_cf-cf_1-hd-13-Index.db |-- one_cf-cf_1-hd-13-Statistics.db |-- one_cf-cf_1-hd-14-CompressionInfo.db |-- one_cf-cf_1-hd-14-Data.db |-- one_cf-cf_1-hd-14-Filter.db |-- one_cf-cf_1-hd-14-Index.db |-- one_cf-cf_1-hd-14-Statistics.db |-- one_cf-cf_1-hd-15-CompressionInfo.db |-- one_cf-cf_1-hd-15-Data.db |-- one_cf-cf_1-hd-15-Filter.db |-- one_cf-cf_1-hd-15-Index.db |-- one_cf-cf_1-hd-15-Statistics.db `-- snapshots `-- 1344609231532 |-- one_cf-cf_1-hd-10-CompressionInfo.db |-- one_cf-cf_1-hd-10-Data.db |-- one_cf-cf_1-hd-10-Filter.db |-- one_cf-cf_1-hd-10-Index.db |-- one_cf-cf_1-hd-10-Statistics.db |-- one_cf-cf_1-hd-11-CompressionInfo.db |-- one_cf-cf_1-hd-11-Data.db |-- one_cf-cf_1-hd-11-Filter.db |-- one_cf-cf_1-hd-11-Index.db |-- one_cf-cf_1-hd-11-Statistics.db |-- one_cf-cf_1-hd-12-CompressionInfo.db |-- one_cf-cf_1-hd-12-Data.db |-- one_cf-cf_1-hd-12-Filter.db |-- one_cf-cf_1-hd-12-Index.db |-- one_cf-cf_1-hd-12-Statistics.db |-- one_cf-cf_1-hd-9-CompressionInfo.db |-- one_cf-cf_1-hd-9-Data.db |-- one_cf-cf_1-hd-9-Filter.db |-- one_cf-cf_1-hd-9-Index.db `-- one_cf-cf_1-hd-9-Statistics.db * I remove the 'wrong' databases root@pnscassandra04:~# rm -v /var/opt/hosting/db/cassandra/data/one_cf/cf_1/*.db removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-CompressionInfo.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Data.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Filter.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Index.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Statistics.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-CompressionInfo.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Data.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Filter.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Index.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Statistics.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-CompressionInfo.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Data.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Filter.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Index.db' removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Statistics.db' * restore root@pnscassandra04:~# for i in /var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/*; do ln -v $i /var/opt/hosting/db/cassandra/data/one_cf/cf_1/; done `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-CompressionInfo.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-CompressionInfo.db' `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Data.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Data.db' `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Filter.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Filter.db' `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Index.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Index.db' `/var/opt/hosting/db/cassandra/data/one_
RE: RE Restore snapshot
De : mdione@orange.com [mailto:mdione@orange.com] > > De : Sylvain Lebresne [mailto:sylv...@datastax.com] > > > 2) copy the snapshot sstable in the right place and call the JMX > > > method > > > loadNewSSTables() (in the column family MBean, which mean you need > > > to do that per-CF). > > > > How does this affect the contents of the CommitLogs? I mean, I > > imagine using this for rollback'ing errors in the app side, like > > some/many/all objects deleted. How will updates/deletes saved in > > CommitLogs impact the data restored this way? Also: while I'm > > restoring a KS of CF, is it possible to cut write requests but not > > reads? I hope I'm explaining myself clearly... > > In particular, I'm thinking on a restore like this: > > * the app does something stupid. > * (if possible) I stop writes to the KS or CF. > * remove the db files with the wrong data. > * I put the db files from the restore. > * nodetool refresh > * profit! > [...] > * and here's the error at repair time: > > root@pnscassandra04:~# nodetool repair one_cf cf_1 > > INFO [RMI TCP Connection(56)-10.234.169.244] 2012-08-10 17:03:12,354 > StorageService.java (line 1925) Starting repair command #3, repairing 5 > ranges. > INFO [AntiEntropySessions:5] 2012-08-10 17:03:12,358 > AntiEntropyService.java (line 666) [repair #811be330-e2fc-11e1-- > b64817724cbd] new session: will sync > pnscassandra04.vdev.bas.s1.p.fti.net/10.234.169.244, /10.234.169.245, > /10.234.169.241, /10.234.169.242, /10.234.169.243 on range > (34028236692093846346337460743176821145,6805647338418769269267492148635 > 3642291] for one_cf.[cf_1] INFO [AntiEntropySessions:5] 2012-08-10 > 17:03:12,359 AntiEntropyService.java (line 871) [repair #811be330-e2fc- > 11e1--b64817724cbd] requesting merkle trees for cf_1 (to > [/10.234.169.245, /10.234.169.241, /10.234.169.242, /10.234.169.243, > pnscassandra04.vdev.bas.s1.p.fti.net/10.234.169.244]) > ERROR [ValidationExecutor:2] 2012-08-10 17:03:12,361 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[ValidationExecutor:2,1,main] > java.io.IOError: java.io.FileNotFoundException: > /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13- > Data.db (No such file or directory) [...] > > I understand that this procedure (refresh+repair in one node) makes > absolutely no sense, but the error leads me to think that after > refresh, the 'wrong' databases are still taken in account. Is it that > the whole procedure is bad from the start and I should use > SSTableLoader instead? Ok, I just did the refresh without the repair on all the nodes at the same time, and the error is still there. Should I drop the CF or KS before doing this kind of restore? -- Marcos Dione SysAdmin Astek Sud-Est pour FT/TGPF/OPF/PORTAIL/DOP/HEBEX @ Marco Polo 04 97 12 62 45 - mdione@orange.com _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified. Thank you.
1.1.3 crasch when initializing column family
Hi
I'm trying the cassandra 1.1.3 on a DualCore SLES10SP4 32bit machine.
The setup is just one single node. The cassandra receives an exception
(see below) and cores when trying to initialize a keyspace/column
family. I can reproduce the crasch with the setup/config files included
in the tar.gz. distribution.
1. remove any old crap from /var/lib/cassandra
2. start the cassandra and observe output: bin/cassandra -f
3. In another shell, exec bin/cassandra-cli -h 127.0.0.1 -f datadef.txt
Where datadef.txt:
--->
create keyspace tst4
with placement_strategy =
'org.apache.cassandra.locator.SimpleStrategy'
and strategy_options = {replication_factor:1};
use tst4;
create column family dirs_ch
with comparator = AsciiType
and column_metadata = [
{ column_name:'str', validation_class:AsciiType},
{ column_name:'str1', validation_class:AsciiType},
{ column_name:'assoc_str', validation_class:AsciiType}];
<
Here is (part) of the output from the when cassandra crasches:
->
Unhandled exception
Type=Segmentation error vmState=0x
J9Generic_Signal_Number=0004 Signal_Number=000b
Error_Value= Signal_Code=0001
Handler1=B75B1443 Handler2=B758A859 InaccessibleAddress=00B4
EDI=B760F560 ESI=71EC1532 EAX=0022 EBX=71D35BA0
ECX=086050EC EDX=0080
EIP=B6584E15 ES=007B DS=007B ESP=08605080
EFlags=00010202 CS=0073 SS=007B EBP=082FCB00
Module=/usr/lib/jvm/java-1.6.0-ibm-1.6.0/jre/lib/i386/libjclscar_24.so
Module_base_address=B653F000
Symbol=sun_misc_Unsafe_getLong__Ljava_lang_Object_2J
Symbol_address=B6584DCC
Target=2_40_20101124_069295 (Linux 2.6.16.60-0.85.1-bigsmp)
CPU=x86 (2 logical CPUs) (0xf4ddb000 RAM)
<---
If I do exactly the same but with the following file i.e just one less
column_metadata, cassandra survives.
--->
create keyspace tst4
with placement_strategy =
'org.apache.cassandra.locator.SimpleStrategy'
and strategy_options = {replication_factor:1};
use tst4;
create column family dirs_ch
with comparator = AsciiType
and column_metadata = [
{ column_name:'str', validation_class:AsciiType},
{ column_name:'assoc_str', validation_class:AsciiType}];
<-
I've tried with Cassandra 1.0.9 and it is working just fine !!!
Any idea ??
rds /Robban
Re: DSE solr HA
> >> >> Going through this page and it looks like indexes are stored locally >> http://www.datastax.com/dev/blog/cassandra-with-solr-integration-details . >> My question is what happens if one of the solr nodes crashes? Is the data >> indexed again on those nodes? >> Yes the data is indexed again on the node. Either from the commitlog or hints or repair. Same as Cassandra. >> Also, if RF > 1 then is the same data being indexed on all RF nodes or is >> that RF only for document replication? The former. Each Replica has a indexed copy. We remove duplicates on read.
RE: Problem with cassandra startup on Linux
Aaron
Yes will do - I had already made the suggested change - cluster is up
and running.
Thanks
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Monday, August 13, 2012 1:56 AM
To: user@cassandra.apache.org
Subject: Re: Problem with cassandra startup on Linux
Hi Dwight,
I can confirm that issue on my MBP under Mountain Lion.
Can you create a ticker at
https://issues.apache.org/jira/browse/CASSANDRA and include the platform
you are running on.
For reference the change was added by
https://issues.apache.org/jira/browse/CASSANDRA-4447
The change is only relevant if you are running on Java 7. As a work
around change the relevant section of cassandra-env.sh to look like
#startswith () [ "${1#$2}" != "$1" ]
if [ "`uname`" = "Linux" ] ; then
# reduce the per-thread stack size to minimize the impact of Thrift
# thread-per-client. (Best practice is for client connections to
# be pooled anyway.) Only do so on Linux where it is known to be
# supported.
#if startswith "$JVM_VERSION" '1.7.'
#then
#JVM_OPTS="$JVM_OPTS -Xss160k"
#else
JVM_OPTS="$JVM_OPTS -Xss128k"
#fi
fi
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 13/08/2012, at 5:16 PM, Dwight Smith
wrote:
Installed 1.1.3 on my Linux cluster - the JVM_OPTS were truncated due to
a script error in Cassandra-env.sh:
Invalid token in the following.
startswith () [ "${1#$2}" != "$1" ]
RE: Problem with cassandra startup on Linux
Aaron
https://issues.apache.org/jira/browse/CASSANDRA-4535
created.
Thanks much
From: Dwight Smith [mailto:dwight.sm...@genesyslab.com]
Sent: Monday, August 13, 2012 7:54 AM
To: user@cassandra.apache.org
Subject: RE: Problem with cassandra startup on Linux
Aaron
Yes will do - I had already made the suggested change - cluster is up
and running.
Thanks
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Monday, August 13, 2012 1:56 AM
To: user@cassandra.apache.org
Subject: Re: Problem with cassandra startup on Linux
Hi Dwight,
I can confirm that issue on my MBP under Mountain Lion.
Can you create a ticker at
https://issues.apache.org/jira/browse/CASSANDRA and include the platform
you are running on.
For reference the change was added by
https://issues.apache.org/jira/browse/CASSANDRA-4447
The change is only relevant if you are running on Java 7. As a work
around change the relevant section of cassandra-env.sh to look like
#startswith () [ "${1#$2}" != "$1" ]
if [ "`uname`" = "Linux" ] ; then
# reduce the per-thread stack size to minimize the impact of Thrift
# thread-per-client. (Best practice is for client connections to
# be pooled anyway.) Only do so on Linux where it is known to be
# supported.
#if startswith "$JVM_VERSION" '1.7.'
#then
#JVM_OPTS="$JVM_OPTS -Xss160k"
#else
JVM_OPTS="$JVM_OPTS -Xss128k"
#fi
fi
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 13/08/2012, at 5:16 PM, Dwight Smith
wrote:
Installed 1.1.3 on my Linux cluster - the JVM_OPTS were truncated due to
a script error in Cassandra-env.sh:
Invalid token in the following.
startswith () [ "${1#$2}" != "$1" ]
Re: 1.1.3 crasch when initializing column family
It works for me on Sun's jvm. It appears to be similar to:
http://www-01.ibm.com/support/docview.wss?uid=swg1IV12627
Have you tried upgrading IBM's java or using Sun's?
-- Omid
On Mon, Aug 13, 2012 at 1:34 PM, Robert Hellmans wrote:
> **
> Hi
>
> I'm trying the cassandra 1.1.3 on a DualCore SLES10SP4 32bit machine. The
> setup is just one single node. The cassandra receives an exception (see
> below) and cores when trying to initialize a keyspace/column family. I can
> reproduce the crasch with the setup/config files included in the tar.gz.
> distribution.
>
>
> 1. remove any old crap from /var/lib/cassandra
> 2. start the cassandra and observe output: bin/cassandra -f
> 3. In another shell, exec bin/cassandra-cli -h 127.0.0.1 -f datadef.txt
>
> Where datadef.txt:
> --->
> create keyspace tst4
>with placement_strategy =
> 'org.apache.cassandra.locator.SimpleStrategy'
>and strategy_options = {replication_factor:1};
>
> use tst4;
>
> create column family dirs_ch
>with comparator = AsciiType
>and column_metadata = [
>{ column_name:'str', validation_class:AsciiType},
>{ column_name:'str1', validation_class:AsciiType},
>{ column_name:'assoc_str', validation_class:AsciiType}];
> <
>
>
> Here is (part) of the output from the when cassandra crasches:
> ->
> Unhandled
> exception
>
> Type=Segmentation error
> vmState=0x
> J9Generic_Signal_Number=0004 Signal_Number=000b
> Error_Value= Signal_Code=0001
> Handler1=B75B1443 Handler2=B758A859
> InaccessibleAddress=00B4
> EDI=B760F560 ESI=71EC1532 EAX=0022
> EBX=71D35BA0
> ECX=086050EC
> EDX=0080
>
> EIP=B6584E15 ES=007B DS=007B ESP=08605080
>
> EFlags=00010202 CS=0073 SS=007B
> EBP=082FCB00
> Module=/usr/lib/jvm/java-1.6.0-ibm-1.6.0/jre/lib/i386/libjclscar_24.so
>
> Module_base_address=B653F000
> Symbol=sun_misc_Unsafe_getLong__Ljava_lang_Object_2J
> Symbol_address=B6584DCC
>
> Target=2_40_20101124_069295 (Linux
> 2.6.16.60-0.85.1-bigsmp)
> CPU=x86 (2 logical CPUs) (0xf4ddb000 RAM)
> <---
>
>
>
>
>
> If I do exactly the same but with the following file i.e just one less
> column_metadata, cassandra survives.
> --->
> create keyspace tst4
>with placement_strategy =
> 'org.apache.cassandra.locator.SimpleStrategy'
>and strategy_options = {replication_factor:1};
>
> use tst4;
>
> create column family dirs_ch
>with comparator = AsciiType
>and column_metadata = [
>{ column_name:'str', validation_class:AsciiType},
>{ column_name:'assoc_str', validation_class:AsciiType}];
> <-
>
>
> I've tried with Cassandra 1.0.9 and it is working just fine !!!
> Any idea ??
>
> rds /Robban
>
>
>
>
>
>
>
>
Re: Cassandra OOM crash while mapping commitlog
3 hours ago I finished the upgraded of our cluster. Currently it runs quite smooth. I'll give an update within a week if this really solved our issues. Cheers! 2012/8/13 Robin Verlangen > @Tyler: We were already running most of our machines in 64bit JVM (Sun, > not the OpenJDK). Those also crashed. > > @Holger: Good to hear that. I'll schedule an update for our Cassandra > cluster. > > Thank you both for your time. > > > 2012/8/13 Holger Hoffstaette > >> On Sun, 12 Aug 2012 13:36:42 +0200, Robin Verlangen wrote: >> >> > Hmm, is issue caused by some 1.x version? Before it never occurred to >> us. >> >> This bug was introduced in 1.1.0 and has been fixed in 1.1.3, where the >> closed/recycled segments are now closed & unmapped properly. The default >> sizes are also smaller. >> Of course the question remains why an append-only commitlog needs to be >> mmap'ed in the first place, especially for writing.. >> >> -h >> >> >> > > > -- > With kind regards, > > Robin Verlangen > *Software engineer* > * > * > W http://www.robinverlangen.nl > E ro...@us2.nl > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. > > -- With kind regards, Robin Verlangen *Software engineer* * * W http://www.robinverlangen.nl E ro...@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Custom Partitioner Type
Is it possible to use a custom Partitioner type (other than RP or BOP) ? Say if my rowkeys are all Integers and I want all even keys to go to node1 and odd keys to node2, is it feasible ? How would I go about ? Thanks.
Re: Custom Partitioner Type
Yes, you need to implement the org.apache.cassandra.dht.IPartitioner interface,
there are a couple of abstract implementations you could base it on.
> I want all even keys to go to
> node1 and odd keys to node2, is it feasible ?
I'm not endorsing the idea of doing this, but as a hack to see if the effects
you could use the BOP and format the keys as (python):
str(key % 2) + "{0:0>#10}".format(key)
So all keys are 11 digit strings, even keys start with 0 and odd with 1.
Hope that helps.
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 14/08/2012, at 7:33 AM, A J wrote:
> Is it possible to use a custom Partitioner type (other than RP or BOP) ?
> Say if my rowkeys are all Integers and I want all even keys to go to
> node1 and odd keys to node2, is it feasible ? How would I go about ?
>
> Thanks.
GCInspector info messages in cassandra log
Hi! I have 3 nodes ring running on Amazon EC2. About once a week I see in the logs compaction messages and around the same time info messages about GC (see below) that I think means it is taking too long and happening too often. Does it mean I have to reduce my cache size? Thanks, Tamar INFO [ScheduledTasks:1] 2012-08-13 12:50:57,593 GCInspector.java (line 122) GC for ParNew: 242 ms for 1 collections, 1541590352 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:27,740 GCInspector.java (line 122) GC for ParNew: 291 ms for 1 collections, 1458227032 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:29,741 GCInspector.java (line 122) GC for ParNew: 261 ms for 1 collections, 1228861368 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:30,833 GCInspector.java (line 122) GC for ParNew: 319 ms for 1 collections, 1120131360 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:32,863 GCInspector.java (line 122) GC for ParNew: 241 ms for 1 collections, 983144216 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:33,864 GCInspector.java (line 122) GC for ParNew: 215 ms for 1 collections, 967702720 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:34,964 GCInspector.java (line 122) GC for ParNew: 248 ms for 1 collections, 973803344 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:41,211 GCInspector.java (line 122) GC for ParNew: 265 ms for 1 collections, 1071933560 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:43,212 GCInspector.java (line 122) GC for ParNew: 326 ms for 1 collections, 1217367792 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:44,212 GCInspector.java (line 122) GC for ParNew: 245 ms for 1 collections, 1203481536 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:45,213 GCInspector.java (line 122) GC for ParNew: 209 ms for 1 collections, 1208819416 used; max is 1937768448 INFO [ScheduledTasks:1] 2012-08-13 12:51:46,237 GCInspector.java (line 122) GC for ParNew: 248 ms for 1 collections, 1338361648 used; max is 1937768448 *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 <>
What are the basic steps to improve Cassandra performance
hi all, I'm a bit new to Cassandra and was wondering what are the basic steps that we must follow to optimize the Cassandra for performance in general and how to improve the performance for "select * from X" type of queries. Any help would be much appreciated. Note that, we have huge data sitting in our schema. Thanks, DK
