Hi Reid,
Thank you for your extensive response. I don't think that we have such a
person and in any case, even if I am a Software Engineer I would be curious
to deep dive into the problem and understand the reason. The only
observation that I have right now is that I have in the same cluster 2
Hi Sergio,
I’m definitely not enough of a network wonk to make definitive statements on
network configuration, finding your in-company network expert is definitely
going to be a lot more productive. I’ve forgotten if you are on-prem or in
AWS, so if in AWS replace “your network wonk” with
There is definitely a resource risk to having thousands of open connections to
each node. Some of the drivers have (had?) less than optimal default settings,
like acquiring 50 connections per Cassandra node. This is usually overkill. I
think 5-10/node is much more reasonable. It depends on your
Hi Reid,
I don't have anymore this loading problem.
I solved by changing the Cassandra Driver Configuration.
Now my cluster is pretty stable and I don't have machines with crazy CPU
Load.
The only thing not urgent but I need to investigate is the number of
ESTABLISHED TCP connections. I see just
Oh nvm, didn't see the later msg about just posting what your fix was.
R
On 10/30/19, 4:24 PM, "Reid Pinchback" wrote:
Message from External Sender
Hi Sergio,
Assuming nobody is actually mounting a SYN flood attack, then this sounds
like you're either being hammered
Hi Sergio,
Assuming nobody is actually mounting a SYN flood attack, then this sounds like
you're either being hammered with connection requests in very short periods of
time, or your TCP backlog tuning is off. At least, that's where I'd start
looking. If you take that log message and google
https://docs.datastax.com/en/drivers/java/2.2/com/datastax/driver/core/policies/LatencyAwarePolicy.html
I had to change the Policy in the Cassandra Driver. I solved this problem few
weeks ago. I am just posting the solution for anyone that could hit the same
issue.
Best,
Sergio
On 2019/10/17
Hello guys!
I performed a thread dump
https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMTAvMTcvLS1kdW1wLnR4dC0tMC0zMC00MA==;
while try to join the node with
-Dcassandra.join_ring=false
OR
-Dcassandra.join.ring=false
OR
-Djoin.ring=false
because the node spiked in load and
