Re: PHP - Cassandra integration

[Spencer Brown]

I'm using /McFrazier/PhpBinaryCql/


On Mon, Nov 10, 2014 at 1:48 AM, Akshay Ballarpure <
akshay.ballarp...@tcs.com> wrote:

> Hello,
> I am working on PHP cassandra integration, please let me know which
> library is good from scalability and performance perspective ?
>
> Best Regards
> Akshay Ballarpure
> Tata Consultancy Services
> Cell:- 9985084075
> Mailto: akshay.ballarp...@tcs.com
> Website: http://www.tcs.com
> 
> Experience certainty.IT Services
>Business Solutions
>Consulting
> 
>
> =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>

What actually causing java.lang.OutOfMemoryError: unable to create new native thread

[Jason Wee]

Hello people, below is an extraction from cassandra system log.

ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
at
org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)

Investigated into the call until the java native call,
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698

  if (native_thread->osthread() == NULL) {
// No one should hold a reference to the 'native_thread'.
delete native_thread;
if (JvmtiExport::should_post_resource_exhausted()) {
  JvmtiExport::post_resource_exhausted(
JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
JVMTI_RESOURCE_EXHAUSTED_THREADS,
"unable to create new native thread");
}
THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
  "unable to create new native thread");
  }

Question. Is that out of memory error due to native os memory or java heap?
Stacked size to the jvm is -Xss128k. Operating system file descriptor max
user processes 26. open files capped at 65536

Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
and  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?

Thank you.

Jason

Increase in dropped mutations after major upgrade from 1.2.18 to 2.0.10

[Paulo Ricardo Motta Gomes]

Hey,

We've seen a considerable increase in the number of dropped mutations after
a major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to
the extra load incurred by upgradesstables, but the dropped mutations
continue even after all sstables are upgraded.

Additional info: Overall (read, write and range) latency improved with the
upgrade, which is great, but I don't understand why dropped mutations has
increased. I/O and CPU load is pretty much the same, number of completed
tasks is the only metric that increased together with dropped mutations.

I also noticed that the number of "all time blocked" FlushWriter operations
is about 5% of completed operations, don't know if this is related, but in
case it helps out...

Anyone has a clue on what could that be? Or what should we monitor to find
out? Any help or JIRA pointers would be kindly appreciated.

Cheers,

-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br *
+55 48 3232.3200

Re: Increase in dropped mutations after major upgrade from 1.2.18 to 2.0.10

[Duncan Sands]

Hi Paulo,

On 10/11/14 15:18, Paulo Ricardo Motta Gomes wrote:

Hey,

We've seen a considerable increase in the number of dropped mutations after a
major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to the extra
load incurred by upgradesstables, but the dropped mutations continue even after
all sstables are upgraded.


are the clocks on all your nodes synchronized with each other?

Ciao, Duncan.



Additional info: Overall (read, write and range) latency improved with the
upgrade, which is great, but I don't understand why dropped mutations has
increased. I/O and CPU load is pretty much the same, number of completed tasks
is the only metric that increased together with dropped mutations.

I also noticed that the number of "all time blocked" FlushWriter operations is
about 5% of completed operations, don't know if this is related, but in case it
helps out...

Anyone has a clue on what could that be? Or what should we monitor to find out?
Any help or JIRA pointers would be kindly appreciated.

Cheers,

--
*Paulo Motta*

Chaordic | /Platform/
_www.chaordic.com.br _
+55 48 3232.3200

Re: Increase in dropped mutations after major upgrade from 1.2.18 to 2.0.10

[Paulo Ricardo Motta Gomes]

On Mon, Nov 10, 2014 at 12:46 PM, Duncan Sands 
wrote:

> Hi Paulo,
>
> On 10/11/14 15:18, Paulo Ricardo Motta Gomes wrote:
>
>> Hey,
>>
>> We've seen a considerable increase in the number of dropped mutations
>> after a
>> major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to
>> the extra
>> load incurred by upgradesstables, but the dropped mutations continue even
>> after
>> all sstables are upgraded.
>>
>
> are the clocks on all your nodes synchronized with each other?
>
> Ciao, Duncan.
>

Yes, the servers are synchronized via NTP.

Cheers!


>
>
>> Additional info: Overall (read, write and range) latency improved with the
>> upgrade, which is great, but I don't understand why dropped mutations has
>> increased. I/O and CPU load is pretty much the same, number of completed
>> tasks
>> is the only metric that increased together with dropped mutations.
>>
>> I also noticed that the number of "all time blocked" FlushWriter
>> operations is
>> about 5% of completed operations, don't know if this is related, but in
>> case it
>> helps out...
>>
>> Anyone has a clue on what could that be? Or what should we monitor to
>> find out?
>> Any help or JIRA pointers would be kindly appreciated.
>>
>> Cheers,
>>
>> --
>> *Paulo Motta*
>>
>> Chaordic | /Platform/
>> _www.chaordic.com.br _
>> +55 48 3232.3200
>>
>
>


-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br *
+55 48 3232.3200

Re: query tracing

[Johnny Miller]

Be cautious enabling query tracing. Great tool for dev/testing/diagnosing etc.. 
- but it does persist data to the system_traces keyspace with a TTL of 24 hours 
and will, as a consequence, consume resources.

http://www.datastax.com/dev/blog/advanced-request-tracing-in-cassandra-1-2 



> On 7 Nov 2014, at 20:20, Jonathan Haddad  wrote:
> 
> Personally I've found that using query timing + log aggregation on the client 
> side is more effective than trying to mess with tracing probability in order 
> to find a single query which has recently become a problem.  I recommend 
> wrapping your session with something that can automatically log the statement 
> on a slow query, then use tracing to identify exactly what happened.  This 
> way finding your problem is not a matter of chance.
> 
> 
> 
> On Fri Nov 07 2014 at 9:41:38 AM Chris Lohfink  > wrote:
> It saves a lot of information for each request thats traced so there is 
> significant overhead.  If you start at a low probability and move it up based 
> on the load impact it will provide a lot of insight and you can control the 
> cost.
> 
> ---
> Chris Lohfink
> 
> On Fri, Nov 7, 2014 at 11:35 AM, Jimmy Lin  > wrote:
> is there any significant  performance penalty if one turn on Cassandra query 
> tracing, through DataStax java driver (say, per every query request of some 
> trouble query)?
> 
> More sampling seems better but then doing so may also slow down the system in 
> some other ways?
> 
> thanks
> 
> 
> 

Re: query tracing

[DuyHai Doan]

As Jonathan said, it's better to activate query tracing client side. It'll
give you better flexibility of when to turn on & off tracing and on which
table. Server-side tracing is global (all tables) and probabilistic, thus
may not give satisfactory level of debugging.

 Programmatically it's pretty simple to achieve and coupled with a good
logging framework (LogBack for Java), you'll even have dynamic logging on
production without having to redeploy client code. I have implemented it in
Achilles very easily by wrapping over the Regular/Bound/Simple statements
of Java driver and display the bound values at runtime :
https://github.com/doanduyhai/Achilles/wiki/Statements-Logging-and-Tracing#dynamic-statements-logging

On Mon, Nov 10, 2014 at 3:52 PM, Johnny Miller 
wrote:

> Be cautious enabling query tracing. Great tool for dev/testing/diagnosing
> etc.. - but it does persist data to the system_traces keyspace with a TTL
> of 24 hours and will, as a consequence, consume resources.
>
> http://www.datastax.com/dev/blog/advanced-request-tracing-in-cassandra-1-2
>
>
> On 7 Nov 2014, at 20:20, Jonathan Haddad  wrote:
>
> Personally I've found that using query timing + log aggregation on the
> client side is more effective than trying to mess with tracing probability
> in order to find a single query which has recently become a problem.  I
> recommend wrapping your session with something that can automatically log
> the statement on a slow query, then use tracing to identify exactly what
> happened.  This way finding your problem is not a matter of chance.
>
>
>
> On Fri Nov 07 2014 at 9:41:38 AM Chris Lohfink 
> wrote:
>
>> It saves a lot of information for each request thats traced so there is
>> significant overhead.  If you start at a low probability and move it up
>> based on the load impact it will provide a lot of insight and you can
>> control the cost.
>>
>> ---
>> Chris Lohfink
>>
>> On Fri, Nov 7, 2014 at 11:35 AM, Jimmy Lin  wrote:
>>
>>> is there any significant  performance penalty if one turn on Cassandra
>>> query tracing, through DataStax java driver (say, per every query request
>>> of some trouble query)?
>>>
>>> More sampling seems better but then doing so may also slow down the
>>> system in some other ways?
>>>
>>> thanks
>>>
>>>
>>>
>>
>

Re: PHP - Cassandra integration

[Robert McFrazier]

Hi Akshay,

Please let me know if you have any issues using PhpBinaryCql, and submit
pull requests if you fix any bugs.

Thanks,
Robert McFrazier

On Mon, Nov 10, 2014 at 6:04 AM, Spencer Brown  wrote:

> I'm using /McFrazier/PhpBinaryCql/
>
>
> On Mon, Nov 10, 2014 at 1:48 AM, Akshay Ballarpure <
> akshay.ballarp...@tcs.com> wrote:
>
>> Hello,
>> I am working on PHP cassandra integration, please let me know which
>> library is good from scalability and performance perspective ?
>>
>> Best Regards
>> Akshay Ballarpure
>> Tata Consultancy Services
>> Cell:- 9985084075
>> Mailto: akshay.ballarp...@tcs.com
>> Website: http://www.tcs.com
>> 
>> Experience certainty.IT Services
>>Business Solutions
>>Consulting
>> 
>>
>> =-=-=
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>>
>>
>


-- 
Thanks,
Robert McFrazier

[RELEASE] Apache Cassandra 2.1.2 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 2.1.2.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 2.1 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problems.

Enjoy!

[1]: http://goo.gl/pi45XF (CHANGES.txt)
[2]: http://goo.gl/vtSXzZ (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread

[graham sanderson]

First question are you running 32bit or 64bit… on 32bit you can easily run out 
of virtual address space for thread stacks.

> On Nov 10, 2014, at 8:25 AM, Jason Wee  wrote:
> 
> Hello people, below is an extraction from cassandra system log.
> 
> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java (line 
> 139) Fatal exception in thread Thread[Thread-273,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at 
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
> at 
> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
> 
> Investigated into the call until the java native call, 
> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
>  
> 
> 
>   if (native_thread->osthread() == NULL) {
> // No one should hold a reference to the 'native_thread'.
> delete native_thread;
> if (JvmtiExport::should_post_resource_exhausted()) {
>   JvmtiExport::post_resource_exhausted(
> JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS,
> "unable to create new native thread");
> }
> THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
>   "unable to create new native thread");
>   }
> 
> Question. Is that out of memory error due to native os memory or java heap? 
> Stacked size to the jvm is -Xss128k. Operating system file descriptor max 
> user processes 26. open files capped at 65536
> 
> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR and 
>  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
> 
> Thank you.
> 
> Jason



smime.p7s
Description: S/MIME cryptographic signature

Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread

[Chris Lohfink]

if your using 64 bit, check output of:

cat /proc/{cassandra pid}/limits

some older linux kernels wont work with above so if it doesnt exist check
the ulimit -a output for the cassandra user. max processes per user may be
your issue as well.

---
Chris Lohfink


On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson  wrote:

> First question are you running 32bit or 64bit… on 32bit you can easily run
> out of virtual address space for thread stacks.
>
> On Nov 10, 2014, at 8:25 AM, Jason Wee  wrote:
>
> Hello people, below is an extraction from cassandra system log.
>
> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
> (line 139) Fatal exception in thread Thread[Thread-273,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
> at
> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
>
> Investigated into the call until the java native call,
> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
>
>   if (native_thread->osthread() == NULL) {
> // No one should hold a reference to the 'native_thread'.
> delete native_thread;
> if (JvmtiExport::should_post_resource_exhausted()) {
>   JvmtiExport::post_resource_exhausted(
> JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
> JVMTI_RESOURCE_EXHAUSTED_THREADS,
> "unable to create new native thread");
> }
> THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
>   "unable to create new native thread");
>   }
>
> Question. Is that out of memory error due to native os memory or java
> heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor
> max user processes 26. open files capped at 65536
>
> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
> and  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
>
> Thank you.
>
> Jason
>
>
>

Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread

[Jason Wee]

Hi, thank you for response.

using 64bit and kernel 2.6.32-358.18.1.el6.x86_64.


# cat /proc/13405/limits
Limit Soft Limit   Hard Limit   Units

Max cpu time  unlimitedunlimitedseconds

Max file size unlimitedunlimitedbytes

Max data size unlimitedunlimitedbytes

Max stack size10485760 unlimitedbytes

Max core file size0unlimitedbytes

Max resident set  unlimitedunlimitedbytes

Max processes 26   26
processes
Max open files6553665536files

Max locked memory 6553665536bytes

Max address space unlimitedunlimitedbytes

Max file locksunlimitedunlimitedlocks

Max pending signals   255762   255762   signals

Max msgqueue size 819200   819200   bytes

Max nice priority 00
Max realtime priority 00
Max realtime timeout  unlimitedunlimitedus




Is the stack size and / or Max open files is sufficient? The rest fd pretty
much infinite.

Jason

On Tue, Nov 11, 2014 at 4:09 AM, Chris Lohfink  wrote:

> if your using 64 bit, check output of:
>
> cat /proc/{cassandra pid}/limits
>
> some older linux kernels wont work with above so if it doesnt exist check
> the ulimit -a output for the cassandra user. max processes per user may
> be your issue as well.
>
> ---
> Chris Lohfink
>
>
> On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson 
> wrote:
>
>> First question are you running 32bit or 64bit… on 32bit you can easily
>> run out of virtual address space for thread stacks.
>>
>> On Nov 10, 2014, at 8:25 AM, Jason Wee  wrote:
>>
>> Hello people, below is an extraction from cassandra system log.
>>
>> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
>> (line 139) Fatal exception in thread Thread[Thread-273,5,main]
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:640)
>> at
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
>> at
>> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
>> at
>> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
>>
>> Investigated into the call until the java native call,
>> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
>>
>>   if (native_thread->osthread() == NULL) {
>> // No one should hold a reference to the 'native_thread'.
>> delete native_thread;
>> if (JvmtiExport::should_post_resource_exhausted()) {
>>   JvmtiExport::post_resource_exhausted(
>> JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
>> JVMTI_RESOURCE_EXHAUSTED_THREADS,
>> "unable to create new native thread");
>> }
>> THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
>>   "unable to create new native thread");
>>   }
>>
>> Question. Is that out of memory error due to native os memory or java
>> heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor
>> max user processes 26. open files capped at 65536
>>
>> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
>> and  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
>>
>> Thank you.
>>
>> Jason
>>
>>
>>
>