Re: Re : Optimum value for native_transport_max_concurrent_connections and native_transport_max_concurrent_connections_per_ip

[sai krishnam raju potturi]

thanks Alain. We were planning to move to 2.0.17 to begin with, and later
move to 2.1. We will try tweaking the number of connections from the
application side to begin with.

thanks
Sai

On Wed, May 11, 2016 at 10:12 AM, Alain RODRIGUEZ 
wrote:

> Hi,
>
> tl;dr: Would you like to give Cassandra 2.1 a try?
>
> Longer answer:
>
> With the good versions of both the driver and Cassandra, you could be
> using the V3 protocol, which dramatically improved way connections work.
>
> http://www.datastax.com/dev/blog/java-driver-2-1-2-native-protocol-v3
>
> Also, in  C*2.1, nodetool tpstats outputs the number of blocked native
> requests, allowing to better tune the
> native_transport_max_concurrent_connections.
>
> If you don't want to upgrade to 2.1, just try tuning the number of
> connection on the client side and monitor. About
> native_transport_max_concurrent_connections, as for other parameters,
> starting from defaults and monitoring the changes, working as much as
> possible on a canary node is often the best way to go.
>
> Hope this helps,
>
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2016-04-29 16:39 GMT+02:00 sai krishnam raju potturi 
> :
>
>> hi;
>>   we are upgrading our cluster from apache-cassandra 2.0.14 to 2.0.17.
>> We have been facing SYN flooding issue (port 9042) in our current
>> version of Cassandra at times. We are hoping to tackle the SYN flooding
>> issues with the following attributes in the YAML file for 2.0.17
>>
>> native_transport_max_concurrent_connections
>>
>> native_transport_max_concurrent_connections_per_ip
>>
>>
>> Are there any observed limitations for the above mentioned attributes.
>> During the peak hours each node serves around 1500 connections. Please
>> suggest optimal values for the mentioned attributes.
>>
>>
>> thanks
>>
>
>

Re: Re : Optimum value for native_transport_max_concurrent_connections and native_transport_max_concurrent_connections_per_ip

[Alain RODRIGUEZ]

Hi,

tl;dr: Would you like to give Cassandra 2.1 a try?

Longer answer:

With the good versions of both the driver and Cassandra, you could be using
the V3 protocol, which dramatically improved way connections work.

http://www.datastax.com/dev/blog/java-driver-2-1-2-native-protocol-v3

Also, in  C*2.1, nodetool tpstats outputs the number of blocked native
requests, allowing to better tune the
native_transport_max_concurrent_connections.

If you don't want to upgrade to 2.1, just try tuning the number of
connection on the client side and monitor. About
native_transport_max_concurrent_connections, as for other parameters,
starting from defaults and monitoring the changes, working as much as
possible on a canary node is often the best way to go.

Hope this helps,

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-04-29 16:39 GMT+02:00 sai krishnam raju potturi :

> hi;
>   we are upgrading our cluster from apache-cassandra 2.0.14 to 2.0.17. We
> have been facing SYN flooding issue (port 9042) in our current version of
> Cassandra at times. We are hoping to tackle the SYN flooding issues with
> the following attributes in the YAML file for 2.0.17
>
> native_transport_max_concurrent_connections
>
> native_transport_max_concurrent_connections_per_ip
>
>
> Are there any observed limitations for the above mentioned attributes.
> During the peak hours each node serves around 1500 connections. Please
> suggest optimal values for the mentioned attributes.
>
>
> thanks
>

Re: Cassandra 2.0.x OOM during startsup - schema version inconsistency after reboot

[Alain RODRIGUEZ]

Hi Michaels :-),

My guess is this ticket will be closed with a "Won't Fix" resolution.

Cassandra 2.0 is no longer supported and I have seen tickets being rejected
like CASSANDRA-10510 
.

Would you like to upgrade to 2.1.last and see if you still have the issue?

About your issue, do you stop your node using a command like the following
one?

nodetool disablethrift && nodetool disablebinary && sleep 5 && nodetool
disablegossip && sleep 10 && nodetool drain && sleep 10 && sudo service
cassandra stop

or even flushing:

nodetool disablethrift && nodetool disablebinary && sleep 5 && nodetool
disablegossip && sleep 10 && nodetool flush && nodetool drain && sleep 10
&& sudo service cassandra stop

Are commitlogs empty when you start cassandra?

C*heers,

---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-05-11 5:35 GMT+02:00 Michael Fong :

> Hi,
>
> Thanks for your recommendation.
> I also opened a ticket to keep track @
> https://issues.apache.org/jira/browse/CASSANDRA-11748
> Hope this could brought someone's attention to take a look. Thanks.
>
> Sincerely,
>
> Michael Fong
>
> -Original Message-
> From: Michael Kjellman [mailto:mkjell...@internalcircle.com]
> Sent: Monday, May 09, 2016 11:57 AM
> To: d...@cassandra.apache.org
> Cc: user@cassandra.apache.org
> Subject: Re: Cassandra 2.0.x OOM during startsup - schema version
> inconsistency after reboot
>
> I'd recommend you create a JIRA! That way you can get some traction on the
> issue. Obviously an OOM is never correct, even if your process is wrong in
> some way!
>
> Best,
> kjellman
>
> Sent from my iPhone
>
> > On May 8, 2016, at 8:48 PM, Michael Fong <
> michael.f...@ruckuswireless.com> wrote:
> >
> > Hi, all,
> >
> >
> > Haven't heard any responses so far, and this isue has troubled us for
> quite some time. Here is another update:
> >
> > We have noticed several times that The schema version may change after
> migration and reboot:
> >
> > Here is the scenario:
> >
> > 1.   Two node cluster (1 & 2).
> >
> > 2.   There are some schema changes, i.e. create a few new
> columnfamily. The cluster will wait until both nodes have schema version in
> sync (describe cluster) before moving on.
> >
> > 3.   Right before node2 is rebooted, the schema version is
> consistent; however, after ndoe2 reboots and starts servicing, the
> MigrationManager would gossip different schema version.
> >
> > 4.   Afterwards, both nodes starts exchanging schema  message
> indefinitely until one of the node dies.
> >
> > We currently suspect the change of schema is due to replying the old
> entry in commit log. We wish to continue dig further, but need experts help
> on this.
> >
> > I don't know if anyone has seen this before, or if there is anything
> wrong with our migration flow though..
> >
> > Thanks in advance.
> >
> > Best regards,
> >
> >
> > Michael Fong
> >
> > From: Michael Fong [mailto:michael.f...@ruckuswireless.com]
> > Sent: Thursday, April 21, 2016 6:41 PM
> > To: user@cassandra.apache.org; d...@cassandra.apache.org
> > Subject: RE: Cassandra 2.0.x OOM during bootstrap
> >
> > Hi, all,
> >
> > Here is some more information on before the OOM happened on the rebooted
> node in a 2-node test cluster:
> >
> >
> > 1.   It seems the schema version has changed on the rebooted node
> after reboot, i.e.
> > Before reboot,
> > Node 1: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,326
> > MigrationManager.java (line 328) Gossiping my schema version
> > 4cb463f8-5376-3baf-8e88-a5cc6a94f58f
> > Node 2: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,122
> > MigrationManager.java (line 328) Gossiping my schema version
> > 4cb463f8-5376-3baf-8e88-a5cc6a94f58f
> >
> > After rebooting node 2,
> > Node 2: DEBUG [main] 2016-04-19 11:18:18,016 MigrationManager.java
> > (line 328) Gossiping my schema version
> > f5270873-ba1f-39c7-ab2e-a86db868b09b
> >
> >
> >
> > 2.   After reboot, both nods repeatedly send MigrationTask to each
> other - we suspect it is related to the schema version (Digest) mismatch
> after Node 2 rebooted:
> > The node2  keeps submitting the migration task over 100+ times to the
> other node.
> > INFO [GossipStage:1] 2016-04-19 11:18:18,261 Gossiper.java (line 1011)
> > Node /192.168.88.33 has restarted, now UP INFO [GossipStage:1]
> > 2016-04-19 11:18:18,262 TokenMetadata.java (line 414) Updating
> > topology for /192.168.88.33 INFO [GossipStage:1] 2016-04-19
> > 11:18:18,263 StorageService.java (line 1544) Node /192.168.88.33 state
> > jump to normal INFO [GossipStage:1] 2016-04-19 11:18:18,264
> > TokenMetadata.java (line 414) Updating topology for /192.168.88.33
> DEBUG [GossipStage:1] 2016-04-19 11:18:18,265 MigrationManager.java (line
> 102) Submitting migration task for /192.168.88.33 DEBUG [GossipStage:1]
> 2016-04-19 11:18:18,265