C*/Nodejs REST API integration
Hi I am using REST API to insert data into C* using below reference link. http://docs.datastax.com/en/developer/nodejs-driver/2.0/nodejs-driver/quick_start/qsSimpleClientExecutingQueries.html However the insert performance is extremely lagging. *Can anyone has experience with Nodejs driver for cassandra and using Async for better performance while inserting data through Nodejs REST API?* Thanks Amit
Running java-driver in parallel cassandra-driver-core 2.1.5, multithreading wokrs extremely slow.
Hi, I have weird driver behaviour. Can you help me please to find the problem? Problem: I try to insert data using 10 threads. I see that 10 thread starts, they start to insert some data and then they hung. It takes enormous amount of time to insert (seconds for 1K inserts). It runs 1K per second if I use single thread to insert. Here is my code: https://gist.github.com/seregasheypak/6ddf5b24cd1c195f5355
Re: Error while adding a new node.
any help? On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi nehajtriv...@gmail.com wrote: also: root@cas03:~# sudo service cassandra start root@cas03:~# lsof -n | grep java | wc -l 5315 root@cas03:~# lsof -n | grep java | wc -l 977317 root@cas03:~# lsof -n | grep java | wc -l 880240 root@cas03:~# lsof -n | grep java | wc -l 882402 On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi nehajtriv...@gmail.com wrote: One of the column family has SStable count as under : SSTable count: 98506 Can it be because of 2.1.3 version of cassandra.. I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964 regards Neha On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee peich...@gmail.com wrote: nodetool cfstats? On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi nehajtriv...@gmail.com wrote: Hey.. nodetool compactionstats pending tasks: 0 no pending tasks. Dont have opscenter. how do I monitor sstables? On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: You also might want to check if you have compactions pending (Opscenter / nodetool compactionstats). Also you can monitor the number of sstables. C*heers Alain 2015-07-01 11:53 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com: Thanks I will checkout. I increased the ulimit to 10, but I am getting the same error, but after a while. regards Neha On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: Just check the process owner to be sure (top, htop, ps, ...) http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits C*heers, Alain 2015-07-01 7:33 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com: Arun, I am logging on to Server as root and running (sudo service cassandra start) regards Neha On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi nehajtriv...@gmail.com wrote: Thanks Arun ! I will try and get back ! On Wed, Jul 1, 2015 at 10:32 AM, Arun arunsi...@gmail.com wrote: Looks like you have too many open files issue. Increase the ulimit for the user. If you are starting the cassandra daemon using user cassandra, increase the ulimit for that user. On Jun 30, 2015, at 21:16, Neha Trivedi nehajtriv...@gmail.com wrote: Hello, I have a 4 node cluster with SimpleSnitch. Cassandra : Cassandra 2.1.3 I am trying to add a new node (cassandra 2.1.7) and I get the following error. ERROR [STREAM-IN-] 2015-06-30 05:13:48,516 JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting forcefully due to: java.io.FileNotFoundException: /var/lib/cassandra/data/-Index.db (Too many open files) I increased the MAX_HEAP_SIZE then I get : ERROR [CompactionExecutor:9] 2015-06-30 23:31:44,792 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:9,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/cassandra/data/-Data.db (Too many open files) at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52) ~[apache-cassandra-2.1.7.jar:2.1.7] Is it because of the different version of Cassandra (2.1.3 and 2.17) ? regards N
Re: Running java-driver in parallel cassandra-driver-core 2.1.5, multithreading wokrs extremely slow.
Did you tried to use BatchStatement? On Jul 2, 2015 11:00 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, I have weird driver behaviour. Can you help me please to find the problem? Problem: I try to insert data using 10 threads. I see that 10 thread starts, they start to insert some data and then they hung. It takes enormous amount of time to insert (seconds for 1K inserts). It runs 1K per second if I use single thread to insert. Here is my code: https://gist.github.com/seregasheypak/6ddf5b24cd1c195f5355
Re: Running java-driver in parallel cassandra-driver-core 2.1.5, multithreading wokrs extremely slow.
What is the reason to do that? I understand BatchStatement as a kind of atomic insert hack. How it can help me to solve concurrency problem? 1 thread with sync insert gives me 1K ops/sec. 10 threads give me 20 ops/sec :) Here are metrics for single thread async insert: -- Timers -- insertTimer count = 4641205 mean rate = 14971.58 calls/second 1-minute rate = 18410.90 calls/second 5-minute rate = 10555.73 calls/second 15-minute rate = 4511.58 calls/second min = 0.00 milliseconds max = 0.12 milliseconds mean = 0.01 milliseconds stddev = 0.01 milliseconds median = 0.01 milliseconds 75% = 0.01 milliseconds 95% = 0.01 milliseconds 98% = 0.02 milliseconds 99% = 0.02 milliseconds 99.9% = 0.12 milliseconds what should I do to reach better performance when i use several threads? 2015-07-02 10:34 GMT+02:00 Vova Shelgunov vvs...@gmail.com: Did you tried to use BatchStatement? On Jul 2, 2015 11:00 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, I have weird driver behaviour. Can you help me please to find the problem? Problem: I try to insert data using 10 threads. I see that 10 thread starts, they start to insert some data and then they hung. It takes enormous amount of time to insert (seconds for 1K inserts). It runs 1K per second if I use single thread to insert. Here is my code: https://gist.github.com/seregasheypak/6ddf5b24cd1c195f5355
Re: Experiencing Timeouts on one node
you should check the network connectivity for this node and also its system average load. is that typo or literary what it is, cassandra 1.2.15.*1* and java 6 update *85* ? On Thu, Jul 2, 2015 at 12:59 AM, Shashi Yachavaram shashi...@gmail.com wrote: We have a 28 node cluster, out of which only one node is experiencing timeouts. We thought it was the raid, but there are two other nodes on the same raid without any problem. Also The problem goes away if we reboot the node, and then reappears after seven days. The following hinted hand-off timeouts are seen on the node experiencing the timeouts. Also we did not notice any gossip errors. I was wondering if anyone has seen this issue and how they resolved it. Cassandra Version: 1.2.15.1 OS: Linux cm 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_85 INFO [HintedHandoff:2] 2015-06-17 22:52:08,130 HintedHandOffManager.java (line 296) Started hinted handoff for host: 4fe86051-6bca-4c28-b09c-1b0f073c1588 with IP: /192.168.1.122 INFO [HintedHandoff:1] 2015-06-17 22:52:08,131 HintedHandOffManager.java (line 296) Started hinted handoff for host: bbf0878b-b405-4518-b649-f6cf7c9a6550 with IP: /192.168.1.119 INFO [HintedHandoff:2] 2015-06-17 22:52:17,634 HintedHandOffManager.java (line 422) Timed out replaying hints to /192.168.1.122; aborting (0 delivered) INFO [HintedHandoff:2] 2015-06-17 22:52:17,635 HintedHandOffManager.java (line 296) Started hinted handoff for host: f7b7ab10-4d42-4f0c-af92-2934a075bee3 with IP: /192.168.1.108 INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java (line 422) Timed out replaying hints to /192.168.1.119; aborting (0 delivered) INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java (line 296) Started hinted handoff for host: ddb79f35-3e2b-4be8-84d8-7942086e2b73 with IP: /192.168.1.104 INFO [HintedHandoff:2] 2015-06-17 22:52:27,143 HintedHandOffManager.java (line 422) Timed out replaying hints to /192.168.1.108; aborting (0 delivered) INFO [HintedHandoff:2] 2015-06-17 22:52:27,144 HintedHandOffManager.java (line 296) Started hinted handoff for host: 6a2fa431-4a51-44cb-af19-1991c960e075 with IP: /192.168.1.117 INFO [HintedHandoff:1] 2015-06-17 22:52:27,153 HintedHandOffManager.java (line 422) Timed out replaying hints to /192.168.1.104; aborting (0 delivered) INFO [HintedHandoff:1] 2015-06-17 22:52:27,154 HintedHandOffManager.java (line 296) Started hinted handoff for host: cf03174a-533c-44d6-a679-e70090ad2bc5 with IP: /192.168.1.107 Thanks -shashi..
Re: [MASSMAIL]Re: Error while adding a new node.
thanks for the reply.!! I will update it to 2.1.7 and checkout. On Thu, Jul 2, 2015 at 6:59 PM, Carlos Rolo r...@pythian.com wrote: Marco you should also avoid 2.1.5 and 2.1.6 because of https://issues.apache.org/jira/browse/CASSANDRA-9549 I know (And often don't recommend last versions, I'm still recommending 2.0.x series unless someone is already in 2.1.x) but given the above bug, 2.1.7 is the best option. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On Thu, Jul 2, 2015 at 3:20 PM, Marcos Ortiz mlor...@uci.cu wrote: The recommended version to use is 2.1.5 because, like you Carlos said, 2.1.6 and 2.1.7 are very new to consider them like stable. On 02/07/15 08:55, Carlos Rolo wrote: Indeed you should upgrade to 2.1.7. And then report if you are still facing problems. Versions up to 2.1.5 (in the 2.1.x series) are not considered stable. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On Thu, Jul 2, 2015 at 11:40 AM, Neha Trivedi nehajtriv...@gmail.com wrote: any help? On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi nehajtriv...@gmail.com wrote: also: root@cas03:~# sudo service cassandra start root@cas03:~# lsof -n | grep java | wc -l 5315 root@cas03:~# lsof -n | grep java | wc -l 977317 root@cas03:~# lsof -n | grep java | wc -l 880240 root@cas03:~# lsof -n | grep java | wc -l 882402 On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi nehajtriv...@gmail.com wrote: One of the column family has SStable count as under : SSTable count: 98506 Can it be because of 2.1.3 version of cassandra.. I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964 regards Neha On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee peich...@gmail.com wrote: nodetool cfstats? On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi nehajtriv...@gmail.com wrote: Hey.. nodetool compactionstats pending tasks: 0 no pending tasks. Dont have opscenter. how do I monitor sstables? On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: You also might want to check if you have compactions pending (Opscenter / nodetool compactionstats). Also you can monitor the number of sstables. C*heers Alain 2015-07-01 11:53 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com: Thanks I will checkout. I increased the ulimit to 10, but I am getting the same error, but after a while. regards Neha On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: Just check the process owner to be sure (top, htop, ps, ...) http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits C*heers, Alain 2015-07-01 7:33 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com: Arun, I am logging on to Server as root and running (sudo service cassandra start) regards Neha On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi nehajtriv...@gmail.com wrote: Thanks Arun ! I will try and get back ! On Wed, Jul 1, 2015 at 10:32 AM, Arun arunsi...@gmail.com wrote: Looks like you have too many open files issue. Increase the ulimit for the user. If you are starting the cassandra daemon using user cassandra, increase the ulimit for that user. On Jun 30, 2015, at 21:16, Neha Trivedi nehajtriv...@gmail.com wrote: Hello, I have a 4 node cluster with SimpleSnitch. Cassandra : Cassandra 2.1.3 I am trying to add a new node (cassandra 2.1.7) and I get the following error. ERROR [STREAM-IN-] 2015-06-30 05:13:48,516 JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting forcefully due to: java.io.FileNotFoundException: /var/lib/cassandra/data/-Index.db (Too many open files) I increased the MAX_HEAP_SIZE then I get : ERROR [CompactionExecutor:9] 2015-06-30 23:31:44,792 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:9,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/cassandra/data/-Data.db (Too many open files) at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52) ~[apache-cassandra-2.1.7.jar:2.1.7] Is it because of the different version of Cassandra (2.1.3 and 2.17) ? regards N -- -- Marcos Ortiz http://about.me/marcosortiz, Sr. Product Manager (Data Infrastructure) at UCI @marcosluis2186 http://twitter.com/marcosluis2186 --
Re: Error while adding a new node.
Indeed you should upgrade to 2.1.7. And then report if you are still facing problems. Versions up to 2.1.5 (in the 2.1.x series) are not considered stable. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On Thu, Jul 2, 2015 at 11:40 AM, Neha Trivedi nehajtriv...@gmail.com wrote: any help? On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi nehajtriv...@gmail.com wrote: also: root@cas03:~# sudo service cassandra start root@cas03:~# lsof -n | grep java | wc -l 5315 root@cas03:~# lsof -n | grep java | wc -l 977317 root@cas03:~# lsof -n | grep java | wc -l 880240 root@cas03:~# lsof -n | grep java | wc -l 882402 On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi nehajtriv...@gmail.com wrote: One of the column family has SStable count as under : SSTable count: 98506 Can it be because of 2.1.3 version of cassandra.. I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964 regards Neha On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee peich...@gmail.com wrote: nodetool cfstats? On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi nehajtriv...@gmail.com wrote: Hey.. nodetool compactionstats pending tasks: 0 no pending tasks. Dont have opscenter. how do I monitor sstables? On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: You also might want to check if you have compactions pending (Opscenter / nodetool compactionstats). Also you can monitor the number of sstables. C*heers Alain 2015-07-01 11:53 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com: Thanks I will checkout. I increased the ulimit to 10, but I am getting the same error, but after a while. regards Neha On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: Just check the process owner to be sure (top, htop, ps, ...) http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits C*heers, Alain 2015-07-01 7:33 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com: Arun, I am logging on to Server as root and running (sudo service cassandra start) regards Neha On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi nehajtriv...@gmail.com wrote: Thanks Arun ! I will try and get back ! On Wed, Jul 1, 2015 at 10:32 AM, Arun arunsi...@gmail.com wrote: Looks like you have too many open files issue. Increase the ulimit for the user. If you are starting the cassandra daemon using user cassandra, increase the ulimit for that user. On Jun 30, 2015, at 21:16, Neha Trivedi nehajtriv...@gmail.com wrote: Hello, I have a 4 node cluster with SimpleSnitch. Cassandra : Cassandra 2.1.3 I am trying to add a new node (cassandra 2.1.7) and I get the following error. ERROR [STREAM-IN-] 2015-06-30 05:13:48,516 JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting forcefully due to: java.io.FileNotFoundException: /var/lib/cassandra/data/-Index.db (Too many open files) I increased the MAX_HEAP_SIZE then I get : ERROR [CompactionExecutor:9] 2015-06-30 23:31:44,792 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:9,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/cassandra/data/-Data.db (Too many open files) at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52) ~[apache-cassandra-2.1.7.jar:2.1.7] Is it because of the different version of Cassandra (2.1.3 and 2.17) ? regards N -- --
Re: [MASSMAIL]Re: Error while adding a new node.
The recommended version to use is 2.1.5 because, like you Carlos said, 2.1.6 and 2.1.7 are very new to consider them like stable. On 02/07/15 08:55, Carlos Rolo wrote: Indeed you should upgrade to 2.1.7. And then report if you are still facing problems. Versions up to 2.1.5 (in the 2.1.x series) are not considered stable. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: _linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo_ Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com http://www.pythian.com/ On Thu, Jul 2, 2015 at 11:40 AM, Neha Trivedi nehajtriv...@gmail.com mailto:nehajtriv...@gmail.com wrote: any help? On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi nehajtriv...@gmail.com mailto:nehajtriv...@gmail.com wrote: also: root@cas03:~# sudo service cassandra start root@cas03:~# lsof -n | grep java | wc -l 5315 root@cas03:~# lsof -n | grep java | wc -l 977317 root@cas03:~# lsof -n | grep java | wc -l 880240 root@cas03:~# lsof -n | grep java | wc -l 882402 On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi nehajtriv...@gmail.com mailto:nehajtriv...@gmail.com wrote: One of the column family has SStable count as under : SSTable count: 98506 Can it be because of 2.1.3 version of cassandra.. I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964 regards Neha On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee peich...@gmail.com mailto:peich...@gmail.com wrote: nodetool cfstats? On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi nehajtriv...@gmail.com mailto:nehajtriv...@gmail.com wrote: Hey.. nodetool compactionstats pending tasks: 0 no pending tasks. Dont have opscenter. how do I monitor sstables? On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ arodr...@gmail.com mailto:arodr...@gmail.com wrote: You also might want to check if you have compactions pending (Opscenter / nodetool compactionstats). Also you can monitor the number of sstables. C*heers Alain 2015-07-01 11:53 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com mailto:nehajtriv...@gmail.com: Thanks I will checkout. I increased the ulimit to 10, but I am getting the same error, but after a while. regards Neha On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ arodr...@gmail.com mailto:arodr...@gmail.com wrote: Just check the process owner to be sure (top, htop, ps, ...) http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits C*heers, Alain 2015-07-01 7:33 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com mailto:nehajtriv...@gmail.com: Arun, I am logging on to Server as root and running (sudo service cassandra start) regards Neha On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi nehajtriv...@gmail.com mailto:nehajtriv...@gmail.com wrote: Thanks Arun ! I will try and get back ! On Wed, Jul 1, 2015 at 10:32 AM, Arun arunsi...@gmail.com mailto:arunsi...@gmail.com wrote: Looks like you have too many open files issue. Increase the ulimit for
Re: [MASSMAIL]Re: Error while adding a new node.
Marco you should also avoid 2.1.5 and 2.1.6 because of https://issues.apache.org/jira/browse/CASSANDRA-9549 I know (And often don't recommend last versions, I'm still recommending 2.0.x series unless someone is already in 2.1.x) but given the above bug, 2.1.7 is the best option. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On Thu, Jul 2, 2015 at 3:20 PM, Marcos Ortiz mlor...@uci.cu wrote: The recommended version to use is 2.1.5 because, like you Carlos said, 2.1.6 and 2.1.7 are very new to consider them like stable. On 02/07/15 08:55, Carlos Rolo wrote: Indeed you should upgrade to 2.1.7. And then report if you are still facing problems. Versions up to 2.1.5 (in the 2.1.x series) are not considered stable. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On Thu, Jul 2, 2015 at 11:40 AM, Neha Trivedi nehajtriv...@gmail.com wrote: any help? On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi nehajtriv...@gmail.com wrote: also: root@cas03:~# sudo service cassandra start root@cas03:~# lsof -n | grep java | wc -l 5315 root@cas03:~# lsof -n | grep java | wc -l 977317 root@cas03:~# lsof -n | grep java | wc -l 880240 root@cas03:~# lsof -n | grep java | wc -l 882402 On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi nehajtriv...@gmail.com wrote: One of the column family has SStable count as under : SSTable count: 98506 Can it be because of 2.1.3 version of cassandra.. I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964 regards Neha On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee peich...@gmail.com wrote: nodetool cfstats? On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi nehajtriv...@gmail.com wrote: Hey.. nodetool compactionstats pending tasks: 0 no pending tasks. Dont have opscenter. how do I monitor sstables? On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: You also might want to check if you have compactions pending (Opscenter / nodetool compactionstats). Also you can monitor the number of sstables. C*heers Alain 2015-07-01 11:53 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com: Thanks I will checkout. I increased the ulimit to 10, but I am getting the same error, but after a while. regards Neha On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: Just check the process owner to be sure (top, htop, ps, ...) http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits C*heers, Alain 2015-07-01 7:33 GMT+02:00 Neha Trivedi nehajtriv...@gmail.com: Arun, I am logging on to Server as root and running (sudo service cassandra start) regards Neha On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi nehajtriv...@gmail.com wrote: Thanks Arun ! I will try and get back ! On Wed, Jul 1, 2015 at 10:32 AM, Arun arunsi...@gmail.com wrote: Looks like you have too many open files issue. Increase the ulimit for the user. If you are starting the cassandra daemon using user cassandra, increase the ulimit for that user. On Jun 30, 2015, at 21:16, Neha Trivedi nehajtriv...@gmail.com wrote: Hello, I have a 4 node cluster with SimpleSnitch. Cassandra : Cassandra 2.1.3 I am trying to add a new node (cassandra 2.1.7) and I get the following error. ERROR [STREAM-IN-] 2015-06-30 05:13:48,516 JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting forcefully due to: java.io.FileNotFoundException: /var/lib/cassandra/data/-Index.db (Too many open files) I increased the MAX_HEAP_SIZE then I get : ERROR [CompactionExecutor:9] 2015-06-30 23:31:44,792 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:9,1,main] java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/cassandra/data/-Data.db (Too many open files) at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52) ~[apache-cassandra-2.1.7.jar:2.1.7] Is it because of the different version of Cassandra (2.1.3 and 2.17) ? regards N -- -- Marcos Ortiz http://about.me/marcosortiz, Sr. Product Manager (Data Infrastructure) at UCI @marcosluis2186 http://twitter.com/marcosluis2186 -- --
Re: Running java-driver in parallel cassandra-driver-core 2.1.5, multithreading wokrs extremely slow.
If you post the code you're using to test, it would be helpful. You should also use cassandra-stress to see if you get similar results. On Thu, Jul 2, 2015 at 1:39 AM Serega Sheypak serega.shey...@gmail.com wrote: What is the reason to do that? I understand BatchStatement as a kind of atomic insert hack. How it can help me to solve concurrency problem? 1 thread with sync insert gives me 1K ops/sec. 10 threads give me 20 ops/sec :) Here are metrics for single thread async insert: -- Timers -- insertTimer count = 4641205 mean rate = 14971.58 calls/second 1-minute rate = 18410.90 calls/second 5-minute rate = 10555.73 calls/second 15-minute rate = 4511.58 calls/second min = 0.00 milliseconds max = 0.12 milliseconds mean = 0.01 milliseconds stddev = 0.01 milliseconds median = 0.01 milliseconds 75% = 0.01 milliseconds 95% = 0.01 milliseconds 98% = 0.02 milliseconds 99% = 0.02 milliseconds 99.9% = 0.12 milliseconds what should I do to reach better performance when i use several threads? 2015-07-02 10:34 GMT+02:00 Vova Shelgunov vvs...@gmail.com: Did you tried to use BatchStatement? On Jul 2, 2015 11:00 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, I have weird driver behaviour. Can you help me please to find the problem? Problem: I try to insert data using 10 threads. I see that 10 thread starts, they start to insert some data and then they hung. It takes enormous amount of time to insert (seconds for 1K inserts). It runs 1K per second if I use single thread to insert. Here is my code: https://gist.github.com/seregasheypak/6ddf5b24cd1c195f5355
Re: Experiencing Timeouts on one node
Jason, The load was evenly distributed. And regarding network connectivity, our applications were successfully able to connect to the node, but the read and write operations were timing out. Also we were able to ssh to this node. I just pasted /bin/nodetool -h node version and java -version. Thanks shashi On Thu, Jul 2, 2015 at 8:42 AM, Jason Wee peich...@gmail.com wrote: you should check the network connectivity for this node and also its system average load. is that typo or literary what it is, cassandra 1.2.15.*1* and java 6 update *85* ? On Thu, Jul 2, 2015 at 12:59 AM, Shashi Yachavaram shashi...@gmail.com wrote: We have a 28 node cluster, out of which only one node is experiencing timeouts. We thought it was the raid, but there are two other nodes on the same raid without any problem. Also The problem goes away if we reboot the node, and then reappears after seven days. The following hinted hand-off timeouts are seen on the node experiencing the timeouts. Also we did not notice any gossip errors. I was wondering if anyone has seen this issue and how they resolved it. Cassandra Version: 1.2.15.1 OS: Linux cm 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_85 INFO [HintedHandoff:2] 2015-06-17 22:52:08,130 HintedHandOffManager.java (line 296) Started hinted handoff for host: 4fe86051-6bca-4c28-b09c-1b0f073c1588 with IP: /192.168.1.122 INFO [HintedHandoff:1] 2015-06-17 22:52:08,131 HintedHandOffManager.java (line 296) Started hinted handoff for host: bbf0878b-b405-4518-b649-f6cf7c9a6550 with IP: /192.168.1.119 INFO [HintedHandoff:2] 2015-06-17 22:52:17,634 HintedHandOffManager.java (line 422) Timed out replaying hints to /192.168.1.122; aborting (0 delivered) INFO [HintedHandoff:2] 2015-06-17 22:52:17,635 HintedHandOffManager.java (line 296) Started hinted handoff for host: f7b7ab10-4d42-4f0c-af92-2934a075bee3 with IP: /192.168.1.108 INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java (line 422) Timed out replaying hints to /192.168.1.119; aborting (0 delivered) INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java (line 296) Started hinted handoff for host: ddb79f35-3e2b-4be8-84d8-7942086e2b73 with IP: /192.168.1.104 INFO [HintedHandoff:2] 2015-06-17 22:52:27,143 HintedHandOffManager.java (line 422) Timed out replaying hints to /192.168.1.108; aborting (0 delivered) INFO [HintedHandoff:2] 2015-06-17 22:52:27,144 HintedHandOffManager.java (line 296) Started hinted handoff for host: 6a2fa431-4a51-44cb-af19-1991c960e075 with IP: /192.168.1.117 INFO [HintedHandoff:1] 2015-06-17 22:52:27,153 HintedHandOffManager.java (line 422) Timed out replaying hints to /192.168.1.104; aborting (0 delivered) INFO [HintedHandoff:1] 2015-06-17 22:52:27,154 HintedHandOffManager.java (line 296) Started hinted handoff for host: cf03174a-533c-44d6-a679-e70090ad2bc5 with IP: /192.168.1.107 Thanks -shashi..
joining a node caused loads on some existing nodes to skyrocket
We had six node clusters and when we attempted to join a node to this, cpu load on two gradually climbed to abnormally high number. Stopping the join and shutting down cassandra on two high-load nodes restored the cluster health (we have RF=3) Anyone have any insight on this cassandra behavior? We have done node join many times before; most recent was just 4 days before. The The following unusual messages in the relevant time period for two nodes. We are using cassandra 2.0.10 Jun 30 16:47:30 cass-22.pelotime.com cassandra-serverERROR [GossipStage:1] CassandraDaemon.java (line 199) Exception in thread Thread[GossipStage:1,5,main] Jun 30 16:47:30 cass-22.pelotime.com java.lang.NullPointerException Jun 30 16:47:30 cass-22.pelotime.com at org.apache.cassandra.gms.Gossiper.convict(Gossiper.java:301) Jun 30 16:47:30 cass-22.pelotime.com at org.apache.cassandra.gms.FailureDetector.forceConviction(FailureDetector.java:251) Jun 30 16:47:30 cass-22.pelotime.com at org.apache.cassandra.gms.GossipShutdownVerbHandler.doVerb(GossipShutdownVerbHandler.java:37) Jun 30 16:47:30 cass-22.pelotime.com at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) Jun 30 16:47:30 cass-22.pelotime.com at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) Jun 30 16:47:30 cass-22.pelotime.com at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) Jun 30 16:47:30 cass-22.pelotime.com at java.lang.Thread.run(Unknown Source) Jun 30 16:47:30 cass-22.pelotime.com cassandra-server INFO [GossipStage:2] Gossiper.java (line 910) Node /10.0.251.77 is now part of the cluster Jun 30 16:47:35 cass-22.pelotime.com cassandra-server INFO [HANDSHAKE-/ 10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking version with /10.0.251.77 Jun 30 16:47:35 cass-22.pelotime.com cassandra-server INFO [RequestResponseStage:138] Gossiper.java (line 876) InetAddress /10.0.251.77 is now UP Jun 30 16:47:38 cass-22.pelotime.com cassandra-server INFO [GossipStage:2] Gossiper.java (line 890) InetAddress /10.0.251.77 is now DOWN Jun 30 16:48:02 cass-22.pelotime.com cassandra-server INFO [HANDSHAKE-/ 10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking version with /10.0.251.77 Jun 30 16:48:05 cass-22.pelotime.com cassandra-server INFO [GossipTasks:1] Gossiper.java (line 658) FatClient /10.0.251.77 has been silent for 3ms, removing from gossip Jun 30 16:48:05 cass-22.pelotime.com cassandra-server INFO [HANDSHAKE-/ 10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking ve Jun 30 16:48:59 cass-24.pelotime.com cassandra-server INFO [HANDSHAKE-/ 10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking version with /10.0.251.77 Jun 30 16:48:59 cass-24.pelotime.com cassandra-server INFO [RequestResponseStage:26] Gossiper.java (line 876) InetAddress /10.0.251.77 is now UP Jun 30 16:48:59 cass-24.pelotime.com cassandra-server INFO [HANDSHAKE-/ 10.0.251.77] OutboundTcpConnection.java (line 386) Handshaking version with /10.0.251.77 Jun 30 16:50:52 cass-24.pelotime.com cassandra-serverERROR [STREAM-OUT-/ 10.0.251.77] StreamSession.java (line 454) [Stream #5f2251e0-1f69-11e5-94c0-d9033a25abe9] Streaming error occurred Jun 30 16:50:52 cass-24.pelotime.com java.io.IOException: Broken pipe Jun 30 16:50:52 cass-24.pelotime.com at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) Jun 30 16:50:52 cass-24.pelotime.com at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source) Jun 30 16:50:52 cass-24.pelotime.com at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source) Jun 30 16:50:52 cass-24.pelotime.com at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:74) Jun 30 16:50:52 cass-24.pelotime.com at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59) Jun 30 16:50:52 cass-24.pelotime.com at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) Jun 30 16:50:52 cass-24.pelotime.com at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45) Jun 30 16:50:52 cass-24.pelotime.com at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339) Jun 30 16:50:52 cass-24.pelotime.com at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:319) Jun 30 16:50:52 cass-24.pelotime.com at java.lang.Thread.run(Unknown Source) Jun 30 16:50:52 cass-24.pelotime.com cassandra-serverERROR [STREAM-OUT-/ 10.0.251.77] StreamSession.java (line 454) [Stream #5f2251e0-1f69-11e5-94c0-d9033a25abe9] Streaming error occurred Jun 30 16:50:52 cass-24.pelotime.com java.io.IOException: Broken pipe Jun 30 16:50:52 cass-24.pelotime.com at sun.nio.ch.FileDispatcherImpl.write0(Native Method) Jun 30 16:50:52 cass-24.pelotime.com at
What are problems with schema disagreement
Hi. Here is a schema disagreement we encountered. Schema versions: b6467059-5897-3cc1-9ee2-73f31841b0b0: [10.0.1.100, 10.0.1.109] c8971b2d-0949-3584-aa87-0050a4149bbd: [10.0.1.55, 10.0.1.16, 10.0.1.77] c733920b-2a31-30f0-bca1-45a8c9130a2c: [10.0.1.221] We deployed an application which would send a schema update (DDL=auto). We found this prod cluster had 3 schema difference. Other existing applications were fine, so some people were curious what if we left this problem alone until off hours. Is there any concerns with not resolve schema disagreement right away? FWIW we went ahead and restarted 221 first, and continue with the rest of the minors. Thanks. John
Re: What are problems with schema disagreement
What version of C* are you running? Some versions of 2.0.x might occasionally fail to propagate schema changes in a timely fashion (though they would fix themselves eventually - in the order of a few minutes) On Jul 2, 2015, at 9:37 PM, John Wong gokoproj...@gmail.com wrote: Hi. Here is a schema disagreement we encountered. Schema versions: b6467059-5897-3cc1-9ee2-73f31841b0b0: [10.0.1.100, 10.0.1.109] c8971b2d-0949-3584-aa87-0050a4149bbd: [10.0.1.55, 10.0.1.16, 10.0.1.77] c733920b-2a31-30f0-bca1-45a8c9130a2c: [10.0.1.221] We deployed an application which would send a schema update (DDL=auto). We found this prod cluster had 3 schema difference. Other existing applications were fine, so some people were curious what if we left this problem alone until off hours. Is there any concerns with not resolve schema disagreement right away? FWIW we went ahead and restarted 221 first, and continue with the rest of the minors. Thanks. John smime.p7s Description: S/MIME cryptographic signature
Re: What are problems with schema disagreement
On Thu, Jul 2, 2015 at 11:01 PM, graham sanderson gra...@vast.com wrote: What version of C* are you running? Some versions of 2.0.x might occasionally fail to propagate schema changes in a timely fashion (though they would fix themselves eventually - in the order of a few minutes) Hi Graham. Thanks. We are still running on 1.2.16, but we do plan to upgrade in the near future. The load on the cluster at the time was very very low. All nodes were responsive, except nothing was show up in the logs after certain time, which led me to believe something happened internal, although that was a poor wild guess. But is it safe to be okay with schema disagreement? I worry about data consistency if I let it sit too long. Thanks. John On Jul 2, 2015, at 9:37 PM, John Wong gokoproj...@gmail.com wrote: Hi. Here is a schema disagreement we encountered. Schema versions: b6467059-5897-3cc1-9ee2-73f31841b0b0: [10.0.1.100, 10.0.1.109] c8971b2d-0949-3584-aa87-0050a4149bbd: [10.0.1.55, 10.0.1.16, 10.0.1.77] c733920b-2a31-30f0-bca1-45a8c9130a2c: [10.0.1.221] We deployed an application which would send a schema update (DDL=auto). We found this prod cluster had 3 schema difference. Other existing applications were fine, so some people were curious what if we left this problem alone until off hours. Is there any concerns with not resolve schema disagreement right away? FWIW we went ahead and restarted 221 first, and continue with the rest of the minors. Thanks. John
Re: Experiencing Timeouts on one node
Hi, I am not sure about what is happening (I have never seen this error before). Yet from https://github.com/apache/cassandra/blob/cassandra-1.2/CHANGES.txt it looks like some bugs were fixed in late revision of 1.2.x. I would advice you upgrading to last 1.2.19 (It is an old and stable version, I see no reason not doing it). The problem goes away if we reboot the node, and then reappears after seven days -- Have you TTLs on any table ? (Set to 7 days ?) -- Do you see any GC warn / heap pressures ? C*heers, Alain 2015-07-02 16:20 GMT+02:00 Shashi Yachavaram shashi...@gmail.com: Jason, The load was evenly distributed. And regarding network connectivity, our applications were successfully able to connect to the node, but the read and write operations were timing out. Also we were able to ssh to this node. I just pasted /bin/nodetool -h node version and java -version. Thanks shashi On Thu, Jul 2, 2015 at 8:42 AM, Jason Wee peich...@gmail.com wrote: you should check the network connectivity for this node and also its system average load. is that typo or literary what it is, cassandra 1.2.15.*1* and java 6 update *85* ? On Thu, Jul 2, 2015 at 12:59 AM, Shashi Yachavaram shashi...@gmail.com wrote: We have a 28 node cluster, out of which only one node is experiencing timeouts. We thought it was the raid, but there are two other nodes on the same raid without any problem. Also The problem goes away if we reboot the node, and then reappears after seven days. The following hinted hand-off timeouts are seen on the node experiencing the timeouts. Also we did not notice any gossip errors. I was wondering if anyone has seen this issue and how they resolved it. Cassandra Version: 1.2.15.1 OS: Linux cm 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_85 INFO [HintedHandoff:2] 2015-06-17 22:52:08,130 HintedHandOffManager.java (line 296) Started hinted handoff for host: 4fe86051-6bca-4c28-b09c-1b0f073c1588 with IP: /192.168.1.122 INFO [HintedHandoff:1] 2015-06-17 22:52:08,131 HintedHandOffManager.java (line 296) Started hinted handoff for host: bbf0878b-b405-4518-b649-f6cf7c9a6550 with IP: /192.168.1.119 INFO [HintedHandoff:2] 2015-06-17 22:52:17,634 HintedHandOffManager.java (line 422) Timed out replaying hints to / 192.168.1.122; aborting (0 delivered) INFO [HintedHandoff:2] 2015-06-17 22:52:17,635 HintedHandOffManager.java (line 296) Started hinted handoff for host: f7b7ab10-4d42-4f0c-af92-2934a075bee3 with IP: /192.168.1.108 INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java (line 422) Timed out replaying hints to / 192.168.1.119; aborting (0 delivered) INFO [HintedHandoff:1] 2015-06-17 22:52:17,643 HintedHandOffManager.java (line 296) Started hinted handoff for host: ddb79f35-3e2b-4be8-84d8-7942086e2b73 with IP: /192.168.1.104 INFO [HintedHandoff:2] 2015-06-17 22:52:27,143 HintedHandOffManager.java (line 422) Timed out replaying hints to / 192.168.1.108; aborting (0 delivered) INFO [HintedHandoff:2] 2015-06-17 22:52:27,144 HintedHandOffManager.java (line 296) Started hinted handoff for host: 6a2fa431-4a51-44cb-af19-1991c960e075 with IP: /192.168.1.117 INFO [HintedHandoff:1] 2015-06-17 22:52:27,153 HintedHandOffManager.java (line 422) Timed out replaying hints to / 192.168.1.104; aborting (0 delivered) INFO [HintedHandoff:1] 2015-06-17 22:52:27,154 HintedHandOffManager.java (line 296) Started hinted handoff for host: cf03174a-533c-44d6-a679-e70090ad2bc5 with IP: /192.168.1.107 Thanks -shashi..
