Re: unable to gossip with peers exception when internode encryption is set to any setting other than 'none'
which exact version of OpenJDK are you using? Is it possible you don't have JCE on those nodes? (I believe more recent versions of Java 8 has this baked in so that might not be it) *Marc Selwan | *DataStax *| *PM, Server Team *|* *(925) 413-7079* *|* Twitter <https://twitter.com/MarcSelwan> * Quick links | *DataStax <http://www.datastax.com> *| *Training <http://www.academy.datastax.com> *| *Documentation <http://www.datastax.com/documentation/getting_started/doc/getting_started/gettingStartedIntro_r.html> *| *Downloads <http://www.datastax.com/download> On Mon, Aug 26, 2019 at 1:56 PM Michael Carlise wrote: > > I originally opened this issue on stackoverflow ( > https://stackoverflow.com/questions/57516660/cassandra-node-to-node-encryption-throws-unable-to-gossip-with-peers-exception > <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_57516660_cassandra-2Dnode-2Dto-2Dnode-2Dencryption-2Dthrows-2Dunable-2Dto-2Dgossip-2Dwith-2Dpeers-2Dexception=DwMFaQ=adz96Xi0w1RHqtPMowiL2g=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk=KdhQDpMbz8v1GYrbdYL_opGq-GBPXftrEYEkgcGeMp0=4CR8PRQopb4FyCLj8PDI44mSouBz65Yx8THnH8tOb7o=> > ). > > However, I haven't gotten any responses in over a week. I'm going to post > it here and maybe someone will have an idea on where I can look. > > We currently run a multi region cassandra cluster in AWS. It runs in four > regions, 12 nodes per region. It runs without node to node encryption (or > client encryption either). We are trying to enable inter datacenter node to > node encryption. However, when we flip encryption over we get an exception > that nodes are unable to gossip with any peers. > > It could possibly be that we didn't build our jks keystore/truststores > correctly (more on how we built these files below). But, we additionally do > not see intra datacenter communication working (which should be set to > unencrypted communication). Additionally, cqlsh cannot connect to the node > either; even though we have (by default) client_auth_required set to false > . > > ERROR [main] 2019-08-15 18:46:32,241 CassandraDaemon.java:749 - Exception > encountered during startup > java.lang.RuntimeException: Unable to gossip with any peers > at > org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) > [apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) > [apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) > [apache-cassandra-3.11.4.jar:3.11.4] > INFO [main] 2019-08-15 18:47:07,384 YamlConfigurationLoader.java:89 - > Configuration location: file:/etc/cassandra/cassandra.yaml > > > Something to note is that this error message occurs after a few minutes of > the node being up. (i.e. there is a delay between start up before this > exception is thrown). > > *Information about our cassandra setup* > > cassandra version: 3.11.4 > JDK version: openjdk-8. > Linux: Ubuntu 18.04 (bionic). > > *cassandra.yaml* > > endpoint_snitch: Ec2MultiRegionSnitch > > server_encryption_options: > internode_encryption: dc > keystore: > keystore_password: > truststore: > truststore_password: > > client_encryption_options: > enabled: false > > *cassandra-rackdc.properties* > > prefer_local=true > > *No obvious errors with SSH output* > > When starting cassandra with JVM_OPTS="$JVM_OPTS -Djavax.net.debug=ssl" added > to cassandra-env.sh we see SSL logs printed to stdout (*Note: Subject and > Issuer were omitted on purpose)*. > > found key for : cassy-us-west-2 > adding as trusted cert: > Subject: ... > Issuer: ... > Algorithm: RSA; Serial number: 0xdad28d843fc73325d4c1a75207d4e74 > Valid from Fri May 27 00:00:00 UTC 2016 until Tue May 26 23:59:59 UTC 2026 > > ... > > trigger seeding of SecureRandom > done seeding SecureRandom > > Looking at Java SE
Re: Cassandra-stress testing
Cassandra, being a scale-out database, can load any arbitrary number of records per hour. The best way to do this is for your given data model, find what your max throughput is on a single node by scaling the number of clients until you start seeing errors (or hit your latency SLA) then pull back by 15-20%. >From there, it's a matter of linearly scaling clients and nodes until you hit your desired throughput. I recommend taking a look at TLP-Stress as it's a bit easier to use and understand: https://thelastpickle.com/blog/2018/10/31/tlp-stress-intro.html Best. *Marc Selwan | *DataStax *| *PM, Server Team *|* *(925) 413-7079* *|* Twitter <https://twitter.com/MarcSelwan> * Quick links | *DataStax <http://www.datastax.com> *| *Training <http://www.academy.datastax.com> *| *Documentation <http://www.datastax.com/documentation/getting_started/doc/getting_started/gettingStartedIntro_r.html> *| *Downloads <http://www.datastax.com/download> On Tue, Aug 20, 2019 at 7:16 AM Surbhi Gupta wrote: > Have you tried ycsa? > It is a tool from yahoo for stress testing nosql databases. > > On Tue, Aug 20, 2019 at 3:34 AM wrote: > >> Hi Everyone, >> >> >> >> Anyone before who have bused Cassandra-stress. I want to test if it’s >> possible to load 600 milllions records per hour in Cassandra or >> >> Find a better way to optimize Cassandra for this case. >> >> Any help will be highly appreciated. >> >> >> >> Sent from Mail >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__go.microsoft.com_fwlink_-3FLinkId-3D550986=DwMFaQ=adz96Xi0w1RHqtPMowiL2g=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk=qz4MqEErkPhY1u6JLqEJUgJmIIjmnMQjptddjTPJE_M=87TbqmPgsIH-JP0fbsUYHhpSQyxeHVdqioQud3BHygc=> >> for Window >> >
Re: Unpair cassandra datacenters
Hi Kunal, Did you edit the cassandra.yaml file in each data center to remove the seed nodes? On which ever data center is starting from scratch (I think it's B in your case), you may want to also change the cluster name. Best, *Marc Selwan | *DataStax *| *PM, Server Team *|* *(925) 413-7079* *|* Twitter <https://twitter.com/MarcSelwan> * Quick links | *DataStax <http://www.datastax.com> *| *Training <http://www.academy.datastax.com> *| *Documentation <http://www.datastax.com/documentation/getting_started/doc/getting_started/gettingStartedIntro_r.html> *| *Downloads <http://www.datastax.com/download> On Mon, Apr 22, 2019 at 3:38 PM Kunal wrote: > Hi Friends, > > I need small help in unpairing two datacenters. > We have 2 datacenters (say A and B ) with 3 nodes in each datacenter. We > want to remove one whole data center (B) (3 nodes) from the other one (B). > basically, want to unpair both datacenter and want to use them both > individually. > We are trying this using nodetool decommission and it is removing the 3 > nodes from B datacenter. But when we are trying to bring up datacenter B to > use it separately from Datacenter A, it is joining back to datacenter A. We > noticed in debug.log, nodes from datacenter A keeps looking for nodes in > datacenter B and getting connection refused error when the nodes of > datacenter B are down, but as soon as nodes comes back, they are joining to > the cluster. > We don't want nodes from datacenter B to join datacenter A once they are > decommissioned. > > Can you please let me know if i am missing anything. > > Thanks in advance. > > Regards, > Kunal Vaid >
Re: Cassandra single unreachable node causing total cluster outage
e are abusing >Cassandra here) with gc grace period of 15 minutes (I know it sounds >ridiculous). Level-tiered compaction strategy. >2. Timeseries data, no updates (short lived) (1 hr). TTLed out using >Date-tiered compaction strategy. >3. Timeseries data, no updates (long lived) (7 days). TTLed out using >Date-tiered compaction strategy. > > > > Overall high read and write throughput (10/second) > > > > *Problem:* > >1. The EC2 machine becomes unreachable (we reproduced the issue by >taking down network card) and the entire cluster becomes unstable for the >time until the down node is removed from the cluster. The node is shown as >DN node while doing nodetool status. Our understanding was that a single >node down in one AZ should not impact other nodes. We are unable to >understand why a single node going down is causing entire cluster to become >unstable. Is there any open bug around this? >2. We tried another experiment by killing Cassandra process but in >this case we only see a blip in latencies but all the other nodes are still >healthy and responsive (as expected). > > > > Any thoughts/comments on what could be the issue here? > > > > Thanks, > Pratik > > > > > > > > -- Marc Selwan | DataStax | Product Management | (925) 413-7079
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Hi Shravan, Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before doing the major upgrade? It's generally favorable to go to the latest patch release as often times they include fixes that smooth over the upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current version) Best, Marc On Fri, Nov 30, 2018 at 3:13 PM Shravan R wrote: > Hello, > > I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9. > I came up with the version based on [1]. I followed upgrade steps as in > [2]. I was testing the same in the lab and encountered issues (streaming > just fails and hangs for ever) with bootstrapping a 3.0.9 node on a > partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The > production cluster that I am supporting is pretty large and I am > anticipating to end up in a situation like this (Hope not) and would like > to be prepared. > > 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded > cluster? > 2) How to bootstrap a 3.x node to a partially upgraded cluster? > 3) Is there an alternative approach to the upgrade large clusters. i.e > instead of going through nodetool upgradesstables on each node in rolling > fashion > > > As per [1] the general restriction is to avoid decommissioning or adding > nodes but in reality there can be failures or maintenance that warrants us > to do so. > > > Please point me in the right direction. > > > Thanks, > Shravan > > > [1] > https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change > > [2] > https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/ > <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_=DwMFaQ=adz96Xi0w1RHqtPMowiL2g=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg=> > > -- Marc Selwan | DataStax | Product Management | (925) 413-7079
Re: How do you do automatic restacking of AWS instance for cassandra?
Hi Surbhi, The only time I've heard of restacking, it was a specific term a financial services company used internally to describe a security related procedure specific to them. If this sounds like you/the company you work for, send me a PM because I don't believe I can share those details in a public mailing list outside of that organization. Best, Marc On Thu, May 25, 2017, 11:22 AM daemeon reiydelle <daeme...@gmail.com> wrote: > What is restacking? > > > > > > *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* > > > *“All men dream, but not equally. Those who dream by night in the dusty > recesses of their minds wake up in the day to find it was vanity, but the > dreamers of the day are dangerous men, for they may act their dreams with > open eyes, to make it possible.” — T.E. Lawrence* > > > On Thu, May 25, 2017 at 10:24 AM, Surbhi Gupta <surbhi.gupt...@gmail.com> > wrote: > >> Hi, >> >> Wanted to understand, how do you do automatic restacking of cassandra >> nodes on AWS? >> >> Thanks >> Surbhi >> > > -- Marc Selwan | DataStax | Solutions Engineer | (925) 413-7079
