[jira] [Comment Edited] (CASSANDRA-13114) 3.0.x: update netty
[
https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851430#comment-15851430
]
Stefan Podkowinski edited comment on CASSANDRA-13114 at 2/3/17 12:37 PM:
-
Any exceptions caught by Netty will be logged by
{{UnexpectedChannelExceptionHandler}}. Before the changes to {{Message}}, this
would only happen when the channel would still be open. In Netty 4.0.23 this
seems to be the case, in 4.0.44 the channel is now closed (at least for
handshake errors, which makes sense) when {{exceptionCaught}} is called. What
the patch does is simply to log the exception in both cases, while leaving the
error reply sending case as it was (keep sending reply only in case channel is
still open). I've noticed the changed behaviour through a dtest, [see
results|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-3.11-dtest/1/testReport/junit/native_transport_ssl_test/NativeTransportSSL/connect_to_ssl_test/].
As for the mentioned performance regression tests, I've only have a single
physical host available for cstar_perf, so I can't really stress test
multi-node setups. A single node tests with and without ssl didn't show any
significant changes, but the results are little meaningful. Besides I also
can't find any way to test the actual memory leak issue at hand by simulating
lots of connecting and deconnecting clients.
was (Author: spo...@gmail.com):
Any exceptions caught be Netty will be logged by
{{UnexpectedChannelExceptionHandler}}. Before the changes to {{Message}}, this
would only happen when the channel would still be open. In Netty 4.0.23 this
seems to be the case, in 4.0.44 the channel is now closed when
{{exceptionCaught}} is called. What the patch does is simply to log the
exception in both cases, while leaving the error reply sending case as it was
(keep sending reply only in case channel is still open). I've noticed the
changed behaviour through a dtest, [see
results|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-3.11-dtest/1/testReport/junit/native_transport_ssl_test/NativeTransportSSL/connect_to_ssl_test/].
As for the mentioned performance regression tests, I've only have a single
physical host available for cstar_perf, so I can't really stress test
multi-node setups. A single node tests with and without ssl didn't show any
significant changes, but the results are little meaningful. Besides I also
can't find any way to test the actual memory leak issue at hand by simulating
lots of connecting and deconnecting clients.
> 3.0.x: update netty
> ---
>
> Key: CASSANDRA-13114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13114
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tom van der Woerdt
>Assignee: Stefan Podkowinski
> Attachments: 13114_netty-4.0.44_2.x-3.0.patch,
> 13114_netty-4.0.44_3.11.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for
> Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with
> Cassandra 3.0.x, has some serious bugs around memory handling for SSL
> connections.
> It would be nice if both were updated to 4.0.42, a version released this year.
> 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory
> every ~30 minutes. This was fixed in 4.0.27.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13114) 3.0.x: update netty
[
https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851430#comment-15851430
]
Stefan Podkowinski commented on CASSANDRA-13114:
Any exceptions caught be Netty will be logged by
{{UnexpectedChannelExceptionHandler}}. Before the changes to {{Message}}, this
would only happen when the channel would still be open. In Netty 4.0.23 this
seems to be the case, in 4.0.44 the channel is now closed when
{{exceptionCaught}} is called. What the patch does is simply to log the
exception in both cases, while leaving the error reply sending case as it was
(keep sending reply only in case channel is still open). I've noticed the
changed behaviour through a dtest, [see
results|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-3.11-dtest/1/testReport/junit/native_transport_ssl_test/NativeTransportSSL/connect_to_ssl_test/].
As for the mentioned performance regression tests, I've only have a single
physical host available for cstar_perf, so I can't really stress test
multi-node setups. A single node tests with and without ssl didn't show any
significant changes, but the results are little meaningful. Besides I also
can't find any way to test the actual memory leak issue at hand by simulating
lots of connecting and deconnecting clients.
> 3.0.x: update netty
> ---
>
> Key: CASSANDRA-13114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13114
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tom van der Woerdt
>Assignee: Stefan Podkowinski
> Attachments: 13114_netty-4.0.44_2.x-3.0.patch,
> 13114_netty-4.0.44_3.11.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for
> Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with
> Cassandra 3.0.x, has some serious bugs around memory handling for SSL
> connections.
> It would be nice if both were updated to 4.0.42, a version released this year.
> 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory
> every ~30 minutes. This was fixed in 4.0.27.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13171) Setting -Dcassandra.join_ring=false is ambiguous
[ https://issues.apache.org/jira/browse/CASSANDRA-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848503#comment-15848503 ] Stefan Podkowinski commented on CASSANDRA-13171: There's a option for preventing gossip during startup as described in CASSANDRA-10809. > Setting -Dcassandra.join_ring=false is ambiguous > > > Key: CASSANDRA-13171 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13171 > Project: Cassandra > Issue Type: Bug >Reporter: Joe Olson > Attachments: log.txt > > > Setting -Dcassandra.join_ring=false in /etc/cassandra/jvm.options has > questionable results. From the log snippet below, the value is set to false, > and read. However, everything seems to happen as if it weren'tgossip > occurs, and streaming data via a bulk load comes in. > (see attached log) > I really need this node not to join the cluster, because it is behind on its > compaction, and will immediately crash (too many files open - OS ulimit is > already set to 30 temporarily). I know of no other way to do an "offline > compaction" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848416#comment-15848416 ] Stefan Podkowinski commented on CASSANDRA-13114: Updated the patch to 4.0.44 and latest CI results already reflect the version change. [~snazy], would you review the patch? > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt >Assignee: Stefan Podkowinski > Attachments: 13114_netty-4.0.44_2.x-3.0.patch, > 13114_netty-4.0.44_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13114: --- Attachment: (was: 13114_netty-4.0.43_3.11.patch) > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt >Assignee: Stefan Podkowinski > Attachments: 13114_netty-4.0.44_2.x-3.0.patch, > 13114_netty-4.0.44_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13114: --- Attachment: 13114_netty-4.0.44_2.x-3.0.patch 13114_netty-4.0.44_3.11.patch > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt >Assignee: Stefan Podkowinski > Attachments: 13114_netty-4.0.43_3.11.patch, > 13114_netty-4.0.44_2.x-3.0.patch, 13114_netty-4.0.44_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13114: --- Attachment: (was: 13114_netty-4.0.43_2.x-3.0.patch) > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt >Assignee: Stefan Podkowinski > Attachments: 13114_netty-4.0.43_3.11.patch, > 13114_netty-4.0.44_2.x-3.0.patch, 13114_netty-4.0.44_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CASSANDRA-13175) Integrate "Error Prone" Code Analyzer
Stefan Podkowinski created CASSANDRA-13175: -- Summary: Integrate "Error Prone" Code Analyzer Key: CASSANDRA-13175 URL: https://issues.apache.org/jira/browse/CASSANDRA-13175 Project: Cassandra Issue Type: Improvement Reporter: Stefan Podkowinski Assignee: Stefan Podkowinski Attachments: 0001-Add-Error-Prone-code-analyzer.patch, checks-2_2.out, checks-3_0.out, checks-trunk.out I've been playing with [Error Prone|http://errorprone.info/] by integrating it into the build process and to see what kind of warnings it would produce. So far I'm positively impressed by the coverage and usefulness of some of the implemented checks. See attachments for results. Unfortunately there are still some issues on how the analyzer is effecting generated code and used guava versions, see [#492|https://github.com/google/error-prone/issues/492]. In case those issues have been solved and the resulting code isn't affected by the analyzer, I'd suggest to add it to trunk with warn only behaviour and some less useful checks disabled. Alternatively a new ant target could be added, maybe with build breaking checks and CI integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13114: --- Status: Patch Available (was: Open) > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt >Assignee: Stefan Podkowinski > Attachments: 13114_netty-4.0.43_2.x-3.0.patch, > 13114_netty-4.0.43_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13114: --- Attachment: (was: 13114_netty-4.0.43_3.11.patch) > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt >Assignee: Stefan Podkowinski > Attachments: 13114_netty-4.0.43_2.x-3.0.patch, > 13114_netty-4.0.43_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13114: --- Attachment: 13114_netty-4.0.43_2.x-3.0.patch 13114_netty-4.0.43_3.11.patch > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt >Assignee: Stefan Podkowinski > Attachments: 13114_netty-4.0.43_2.x-3.0.patch, > 13114_netty-4.0.43_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13114) 3.0.x: update netty
[
https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842958#comment-15842958
]
Stefan Podkowinski commented on CASSANDRA-13114:
Some insights on dtest results:
* {{local_quorum_bootstrap_test}}
See CASSANDRA-12437, [PR|https://github.com/riptano/cassandra-dtest/pull/1429]
created
* {{test_tombstone_failure_v3}}
Seems to be [failing on constant
basis|http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_dtest/lastCompletedBuild/testReport/read_failures_test/TestReadFailures/test_tombstone_failure_v3/],
I've opened a [PR| https://github.com/riptano/cassandra-dtest/pull/1430]
* {{cqlsh_tests.cqlsh_tests.CqlshSmokeTest.test_alter_table}} and
* {{cql_tests.MiscellaneousCQLTester.prepared_statement_invalidation_test}}
Fallout from CASSANDRA-12443 and already addressed in
[PR|https://github.com/riptano/cassandra-dtest/pull/1427]
Other tests seem to be flaky or have known issues.
> 3.0.x: update netty
> ---
>
> Key: CASSANDRA-13114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13114
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tom van der Woerdt
>Assignee: Stefan Podkowinski
> Attachments: 13114_netty-4.0.43_3.11.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for
> Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with
> Cassandra 3.0.x, has some serious bugs around memory handling for SSL
> connections.
> It would be nice if both were updated to 4.0.42, a version released this year.
> 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory
> every ~30 minutes. This was fixed in 4.0.27.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13114: --- Attachment: (was: 13114_netty-4.0.43_2.x-3.0.patch) > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt >Assignee: Stefan Podkowinski > Attachments: 13114_netty-4.0.43_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12437) dtest failure in bootstrap_test.TestBootstrap.local_quorum_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842443#comment-15842443
]
Stefan Podkowinski commented on CASSANDRA-12437:
Looks like stress is trying to connect using the default 7199 port, while we
start with 7100 for the first node when using ccm.
As this is broken for quite a while and no other tests seem to be affected, I'd
assume it's okay to simply add the error to the list of ignore patterns unless
we actual need the additional stress output.
[PR|https://github.com/riptano/cassandra-dtest/pull/1429] has been created.
/cc [~philipthompson]
> dtest failure in bootstrap_test.TestBootstrap.local_quorum_bootstrap_test
> -
>
> Key: CASSANDRA-12437
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12437
> Project: Cassandra
> Issue Type: Test
>Reporter: Craig Kodman
>Assignee: DS Test Eng
> Labels: dtest, windows
> Attachments: node1_debug.log, node1_gc.log, node1.log,
> node2_debug.log, node2_gc.log, node2.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.2_dtest_win32/281/testReport/bootstrap_test/TestBootstrap/local_quorum_bootstrap_test
> {code}
> Stacktrace
> File "C:\tools\python2\lib\unittest\case.py", line 329, in run
> testMethod()
> File
> "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\bootstrap_test.py",
> line 389, in local_quorum_bootstrap_test
> 'ops(insert=1)', '-rate', 'threads=50'])
> File "D:\jenkins\workspace\cassandra-2.2_dtest_win32\ccm\ccmlib\node.py",
> line 1244, in stress
> return handle_external_tool_process(p, ['stress'] + stress_options)
> File "D:\jenkins\workspace\cassandra-2.2_dtest_win32\ccm\ccmlib\node.py",
> line 1955, in handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> 'Subprocess [\'stress\', \'user\', \'profile=d:temp2tmp8sf4da\',
> \'n=2M\', \'no-warmup\', \'ops(insert=1)\', \'-rate\', \'threads=50\'] exited
> with non-zero status; exit status: 1; \nstderr: Exception in thread "main"
> java.io.IOError: java.io.FileNotFoundException: d:\\temp\\2\\tmp8sf4da (The
> process cannot access the file because it is being used by another
> process)\r\n\tat
> org.apache.cassandra.stress.StressProfile.load(StressProfile.java:574)\r\n\tat
>
> org.apache.cassandra.stress.settings.SettingsCommandUser.(SettingsCommandUser.java:58)\r\n\tat
>
> org.apache.cassandra.stress.settings.SettingsCommandUser.build(SettingsCommandUser.java:127)\r\n\tat
>
> org.apache.cassandra.stress.settings.SettingsCommand.get(SettingsCommand.java:195)\r\n\tat
>
> org.apache.cassandra.stress.settings.StressSettings.get(StressSettings.java:249)\r\n\tat
>
> org.apache.cassandra.stress.settings.StressSettings.parse(StressSettings.java:220)\r\n\tat
> org.apache.cassandra.stress.Stress.main(Stress.java:63)\r\nCaused by:
> java.io.FileNotFoundException: d:\\temp\\2\\tmp8sf4da (The process cannot
> access the file because it is being used by another process)\r\n\tat
> java.io.FileInputStream.open0(Native Method)\r\n\tat
> java.io.FileInputStream.open(FileInputStream.java:195)\r\n\tat
> java.io.FileInputStream.(FileInputStream.java:138)\r\n\tat
> java.io.FileInputStream.(FileInputStream.java:93)\r\n\tat
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)\r\n\tat
>
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)\r\n\tat
> java.net.URL.openStream(URL.java:1038)\r\n\tat
> org.apache.cassandra.stress.StressProfile.load(StressProfile.java:560)\r\n\t...
> 6 more\r\n\n >> begin captured logging <<
> \ndtest: DEBUG: cluster ccm directory:
> d:\\temp\\2\\dtest-wsze0r\ndtest: DEBUG: Done setting configuration
> options:\n{ \'initial_token\': None,\n\'num_tokens\': \'32\',\n
> \'phi_convict_threshold\': 5,\n\'start_rpc\':
> \'true\'}\n- >> end captured logging <<
> -'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13153) Reappeared Data when Mixing Incremental and Full Repairs
[ https://issues.apache.org/jira/browse/CASSANDRA-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839622#comment-15839622 ] Stefan Podkowinski commented on CASSANDRA-13153: Thanks reporting this, [~Amanda.Debrot]! Let me try to wrap-up again what's happending here.. I think the assumption was that anti-compaction will isolate repaired ranges into the repaired set of sstables, while parts of sstables not covered by the repair will stay in the unrepaired set. As described by Amanda, trouble starts when anti-compaction is taking place exclusively on already repaired sstables. Once we've finished repairing a certain range using full repair, anti-compaction will move unaffected ranges in overlapping sstables from the repaired into unrepaired set again, even if ranges have actually already been repaired before. As the overlap between ranges and sstables is non-deterministic, we could either see regular cells, tombstones or both being move to unrepaired, based on whether the sstable happens to overlap or not. Unfortunately this is not the only way that this could happen. As described in CASSANDRA-9143, compactions during the repairs can prevent anti-compaction for individual sstables and tombstones and data could end up in different sets in this case as well. bq. I've only tested it on Cassandra version 2.2 but it most likely also affects all Cassandra versions with incremental repair - like 2.1 and 3.0. I think 2.1 should not be affected, as we started doing anti-compactions for full repairs in 2.2. > Reappeared Data when Mixing Incremental and Full Repairs > > > Key: CASSANDRA-13153 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13153 > Project: Cassandra > Issue Type: Bug > Components: Compaction, Tools > Environment: Apache Cassandra 2.2 >Reporter: Amanda Debrot > Labels: Cassandra > Attachments: log-Reappeared-Data.txt, > Step-by-Step-Simulate-Reappeared-Data.txt > > > This happens for both LeveledCompactionStrategy and > SizeTieredCompactionStrategy. I've only tested it on Cassandra version 2.2 > but it most likely also affects all Cassandra versions with incremental > repair - like 2.1 and 3.0. > When mixing incremental and full repairs, there are a few scenarios where the > Data SSTable is marked as unrepaired and the Tombstone SSTable is marked as > repaired. Then if it is past gc_grace, and the tombstone and data has been > compacted out on other replicas, the next incremental repair will push the > Data to other replicas without the tombstone. > Simplified scenario: > 3 node cluster with RF=3 > Intial config: > Node 1 has data and tombstone in separate SSTables. > Node 2 has data and no tombstone. > Node 3 has data and tombstone in separate SSTables. > Incremental repair (nodetool repair -pr) is run every day so now we have > tombstone on each node. > Some minor compactions have happened since so data and tombstone get merged > to 1 SSTable on Nodes 1 and 3. > Node 1 had a minor compaction that merged data with tombstone. 1 > SSTable with tombstone. > Node 2 has data and tombstone in separate SSTables. > Node 3 had a minor compaction that merged data with tombstone. 1 > SSTable with tombstone. > Incremental repairs keep running every day. > Full repairs run weekly (nodetool repair -full -pr). > Now there are 2 scenarios where the Data SSTable will get marked as > "Unrepaired" while Tombstone SSTable will get marked as "Repaired". > Scenario 1: > Since the Data and Tombstone SSTable have been marked as "Repaired" > and anticompacted, they have had minor compactions with other SSTables > containing keys from other ranges. During full repair, if the last node to > run it doesn't own this particular key in it's partitioner range, the Data > and Tombstone SSTable will get anticompacted and marked as "Unrepaired". Now > in the next incremental repair, if the Data SSTable is involved in a minor > compaction during the repair but the Tombstone SSTable is not, the resulting > compacted SSTable will be marked "Unrepaired" and Tombstone SSTable is marked > "Repaired". > Scenario 2: > Only the Data SSTable had minor compaction with other SSTables > containing keys from other ranges after being marked as "Repaired". The > Tombstone SSTable was never involved in a minor compaction so therefore all > keys in that SSTable belong to 1 particular partitioner range. During full > repair, if the last node to run it doesn't own this particular key in it's > partitioner range, the Data SSTable will get anticompacted and marked as > "Unrepaired". The Tombstone SSTable stays marked as Repaired. > Then it’s past gc_grace. Since Node’s #1 and #3 only have 1 SSTable
[jira] [Updated] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13114: --- Attachment: 13114_netty-4.0.43_3.11.patch 13114_netty-4.0.43_2.x-3.0.patch > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt > Attachments: 13114_netty-4.0.43_2.x-3.0.patch, > 13114_netty-4.0.43_3.11.patch > > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837856#comment-15837856 ] Stefan Podkowinski commented on CASSANDRA-13114: I admit that I don't really have an opinion on using any particular version. Any reasons not to go with the latest stable 4.0 release? Let's try a CI run using 4.0.43: ||2.1||2.2||3.0||3.11||trunk|| |[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13114-2.1]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13114-2.2]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13114-3.0]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13114-3.11]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13114-trunk]| |[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-2.1-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-3.11-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-trunk-dtest/]| |[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-2.1-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-3.11-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13114-trunk-testall/]| [~jasobrown], you have any strong preference for netty releases and gossip 2.0? [~norman], anything to double check or look out for from our side while updating from 4.0.23 -> 4.0.43? > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10689) java.lang.OutOfMemoryError: Direct buffer memory
[
https://issues.apache.org/jira/browse/CASSANDRA-10689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837642#comment-15837642
]
Stefan Podkowinski edited comment on CASSANDRA-10689 at 1/25/17 12:24 PM:
--
If I understand correctly, the setting will reduce the potential size of
allocated native memory per thread for non-direct buffers, but the total
allocation size should still be capped by the constant number of pooled threads
we use, multiplied by the max. allocated non-direct ByteBuffer size by each
thread. What I'm wondering here is where are we using non-direct buffers and if
large buffer allocations that would be affected by the setting, are intentional
and shouldn't we be able to use smaller buffers instead?
Although the maxCachedBufferSize setting could help avoiding OOM errors, it
sounds more like a band-aid to me, for a situation where increased native
memory pressure is taking place caused by a memory leak somewhere else, e.g. in
the Netty SSL handler (CASSANDRA-13114). Once memory has been depleted, an OOME
could as well be raise during mentioned non-direct buffer allocations, even of
moderate sizes.
was (Author: spo...@gmail.com):
If I understand correctly, the setting will reduce the potential size of
allocated native memory per thread for non-direct buffers, but the total
allocation size should still be capped by the constant number of pooled threads
we use, multiplied by the max. allocated non-direct ByteBuffer size by each
thread. What I'm wondering here is where are we using non-direct buffers and if
large buffer allocations that would be affected by the setting, are intentional
and shouldn't we be able to use smaller buffers instead?
Although the maxCachedBufferSize setting could help avoiding OOM errors, it
sounds more like a band-aid to me, for a situation where increased native
memory pressure is taking place caused by a memory leak somewhere else, e.g. in
Netty (CASSANDRA-13114). Once memory has been depleted, an OOME could as well
be raise during mentioned non-direct buffer allocations, even of moderate sizes.
> java.lang.OutOfMemoryError: Direct buffer memory
>
>
> Key: CASSANDRA-10689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10689
> Project: Cassandra
> Issue Type: Bug
>Reporter: mlowicki
>
> {code}
> ERROR [SharedPool-Worker-63] 2015-11-11 17:53:16,161
> JVMStabilityInspector.java:117 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> ~[na:1.7.0_80]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> ~[na:1.7.0_80]
> at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
> ~[na:1.7.0_80]
> at sun.nio.ch.IOUtil.read(IOUtil.java:195) ~[na:1.7.0_80]
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:149)
> ~[na:1.7.0_80]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:104)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:81)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:310)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:64)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1894)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.setToRowStart(IndexedSliceReader.java:107)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:83)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:246)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
>
[jira] [Commented] (CASSANDRA-10689) java.lang.OutOfMemoryError: Direct buffer memory
[
https://issues.apache.org/jira/browse/CASSANDRA-10689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837642#comment-15837642
]
Stefan Podkowinski commented on CASSANDRA-10689:
If I understand correctly, the setting will reduce the potential size of
allocated native memory per thread for non-direct buffers, but the total
allocation size should still be capped by the constant number of pooled threads
we use, multiplied by the max. allocated non-direct ByteBuffer size by each
thread. What I'm wondering here is where are we using non-direct buffers and if
large buffer allocations that would be affected by the setting, are intentional
and shouldn't we be able to use smaller buffers instead?
Although the maxCachedBufferSize setting could help avoiding OOM errors, it
sounds more like a band-aid to me, for a situation where increased native
memory pressure is taking place caused by a memory leak somewhere else, e.g. in
Netty (CASSANDRA-13114). Once memory has been depleted, an OOME could as well
be raise during mentioned non-direct buffer allocations, even of moderate sizes.
> java.lang.OutOfMemoryError: Direct buffer memory
>
>
> Key: CASSANDRA-10689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10689
> Project: Cassandra
> Issue Type: Bug
>Reporter: mlowicki
>
> {code}
> ERROR [SharedPool-Worker-63] 2015-11-11 17:53:16,161
> JVMStabilityInspector.java:117 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> ~[na:1.7.0_80]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> ~[na:1.7.0_80]
> at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
> ~[na:1.7.0_80]
> at sun.nio.ch.IOUtil.read(IOUtil.java:195) ~[na:1.7.0_80]
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:149)
> ~[na:1.7.0_80]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:104)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:81)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:310)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:64)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1894)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.setToRowStart(IndexedSliceReader.java:107)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:83)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:246)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:270)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1994)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1837)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:353)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
>
[jira] [Commented] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837609#comment-15837609 ] Stefan Podkowinski commented on CASSANDRA-13114: [~snazy], since you've been working on a couple of related issues (CASSANDRA-12032, CASSANDRA-12033, CASSANDRA-12034, CASSANDRA-11937, CASSANDRA-11818).. what would be your suggestion when it comes to back-porting some of these to further stabilize memory management in 2.x and 3.0? Do you see any issues updating netty to 4.0.42 on all branches, regardless of any back-porting decisions for the listed tickets? > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6964) error in logs: ByteBuf.release() was not called before it's garbage-collected
[
https://issues.apache.org/jira/browse/CASSANDRA-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837587#comment-15837587
]
Stefan Podkowinski commented on CASSANDRA-6964:
---
[~rha], do you have ssl enabled for native transport connections? If so,
CASSANDRA-13114 could be relevant for you here.
> error in logs: ByteBuf.release() was not called before it's garbage-collected
> -
>
> Key: CASSANDRA-6964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6964
> Project: Cassandra
> Issue Type: Bug
>Reporter: Russ Hatch
> Labels: qa-resolved
> Attachments: node1.log
>
>
> Running some of our paging tests against 2.1 several of these exceptions are
> triggered:
> {noformat}
> run_tests:
> [java] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> [java] SLF4J: Defaulting to no-operation (NOP) logger implementation
> [java] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further details.
> [java] test_zero_page_size_ignored (__main__.TestPagingSize) ... ERROR
> [java]
> [java]
> ==
> [java] ERROR: test_zero_page_size_ignored (__main__.TestPagingSize)
> [java]
> --
> [java] Traceback (most recent call last):
> [java] File
> "/home/rhatch/git/knifewine/cassandra-dtest-jython/base.py", line 152, in
> tearDown
> [java] raise AssertionError('Unexpected error in %s node log: %s' %
> (node.name, errors))
> [java] AssertionError: Unexpected error in node1 node log: ["ERROR
> [nioEventLoopGroup-3-2] 2014-03-31 17:16:44,337 Slf4JLogger.java:176 - LEAK:
> ByteBuf.release() was not called before it's garbage-collected. Enable
> advanced leak reporting to find out where the leak occurred. To enable
> advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetectionLevel=advanced' or call
> ResourceLeakDetector.setLevel()\n"]
> [java]
> [java]
> --
> [java] Ran 1 test in 37.876s
> [java]
> [java] FAILED (errors=1)
> {noformat}
> These tests are run through jython with the java driver, so there's a little
> bit of setup needed (if you have ccm and dtest you are most of the way there):
> 1. clone and set up https://github.com/riptano/cassandra-dtest-jython . You
> may need to install ivy and copy ivy.jar to ~/.ant/lib/ivy.jar
> 2. you should have ccm, and CASSANDRA_DIR should be set in your environment
> 3. from the root of cassandra-dtest-jython run the tests with 'ant
> run_tests'. The tests take about 10 minutes run completely.
> 4. if you don't want to wait for the entire test suite to run, change the
> bottom of paging_test.py to just run a single test like so:
> {noformat}
> if __name__ == '__main__':
> suite = unittest.TestSuite()
> suite.addTest(TestPagingSize("test_zero_page_size_ignored"))
>
> unittest.TextTestRunner(verbosity=2).run(suite)
>
> exit(0)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13139) test failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834070#comment-15834070
]
Stefan Podkowinski commented on CASSANDRA-13139:
I think there's already a
[PR|https://github.com/riptano/cassandra-dtest/pull/1423] out there by
[~pauloricardomg] that should fix the test.
> test failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> -
>
> Key: CASSANDRA-13139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13139
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Sean McCarthy
> Labels: dtest, test-failure
> Attachments: node1_debug.log, node1_gc.log, node1.log,
> node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log,
> node3.log, node4_debug.log, node4_gc.log, node4.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/503/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7200', ['decommission']]
> exited with non-zero status; exit status: 1;
> stdout: nodetool: Unsupported operation: Not enough live nodes to maintain
> replication factor in keyspace system_distributed (RF = 3, N = 3). Perform a
> forceful decommission to ignore.
> See 'nodetool help' or 'nodetool help '.
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 169, in
> hintedhandoff_decom_test
> node2.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13124) Send ack when received hint is non-local
[
https://issues.apache.org/jira/browse/CASSANDRA-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825832#comment-15825832
]
Stefan Podkowinski commented on CASSANDRA-13124:
bq. Does this mean retry is broken on 3.0? Should we file a ticket for this?
From what I understood CASSANDRA-11960 only affected 3.X so I guess retry
should be working on 3.0?
You can just test it by commenting out all code in {{HintVerbHandler}}.
Afterwards the {{hintedhandoff_decom_test}} will fail due to data loss (that
was supposed to be prevented by the transferred hints). Although no hints have
been handled at all, decom will happily proceed and there won't be any
exceptions or errors in the logs.
bq. If my understanding above is correct we should probably also fix this on
3.0 to reinstate the same behavior before CASSANDRA-12905? Can you create a 3.0
patch and submit CI?
This plus removing the retry loop in the dispatcher, due to the broken
behaviour described in [my
comment|https://issues.apache.org/jira/browse/CASSANDRA-13058?focusedCommentId=15818371=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15818371],
which is what has been done for CASSANDRA-11960 as well. Afterwards retrying
hints on page basis should happen just once as initiated by
{{TransferHintsTask}} and periodically for regular hinting, executed by the
HintsDispatcher executor.
||3.0||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13124-3.0]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13124-3.0-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13124-3.0-testall/]|
> Send ack when received hint is non-local
>
>
> Key: CASSANDRA-13124
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13124
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Priority: Minor
>
> CASSANDRA-12905 changed the hints path to be asynchronous in the normal case,
> but when the hint is non-local and should be stored (ie. on decommission) an
> ack is not sent so the operation does not complete successfully.
> CASSANDRA-13058 fixed this for 3.0+, but for 3.11+ but there is an additional
> concern which on 3.0 is that non-acked hints are being completed successfully
> so we should probably look into this as well here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-13020) Stuck in LEAVING state (Transferring all hints to null)
[
https://issues.apache.org/jira/browse/CASSANDRA-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821557#comment-15821557
]
Stefan Podkowinski edited comment on CASSANDRA-13020 at 1/13/17 10:19 AM:
--
The issues seems to be introduced in CASSANDRA-10198 and caused by not using
the broadcast IP for retrieving the host ID from TokenMetadata. See
[StorageService.getPreferredHintsStreamTarget()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L3972].
/cc [~krummas]
was (Author: spo...@gmail.com):
The issues seems to be caused by not using the broadcast IP for retrieving the
host ID from TokenMetadata. See
[StorageService.getPreferredHintsStreamTarget()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L3972].
/cc [~krummas]
> Stuck in LEAVING state (Transferring all hints to null)
> ---
>
> Key: CASSANDRA-13020
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13020
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: v3.0.9
>Reporter: Aleksandr Ivanov
> Labels: decommission, hints
>
> I tried to decommission one node.
> Node sent all data to another node and got stuck in LEAVING state.
> Log message shows Exception in HintsDispatcher thread.
> Could it be reason of stuck in LEAVING state?
> command output:
> {noformat}
> root@cas-node6:~# time nodetool decommission
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
> at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:203)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:168)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> real147m7.483s
> user0m17.388s
> sys 0m1.968s
> {noformat}
> nodetool netstats:
> {noformat}
> root@cas-node6:~# nodetool netstats
> Mode: LEAVING
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 35082
> Mismatch (Blocking): 18
> Mismatch (Background): 0
> Pool NameActive Pending Completed Dropped
> Large messages n/a 1 0 0
> Small messages n/a 0 16109860 112
> Gossip messages n/a 0 287074 0
> {noformat}
> Log:
> {noformat}
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:52:59,467
> StorageService.java:1170 - LEAVING: sleeping 3 ms for batch processing
> and pending range setup
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,455
> StorageService.java:1170 - LEAVING: replaying batch log and streaming data to
> other nodes
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,910
> StreamResultFuture.java:87 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Executing streaming plan for Unbootstrap
> INFO [StreamConnectionEstablisher:1] 2016-12-07 12:53:39,911
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.17
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,911
> StreamSession.java:232 - [Stream
[jira] [Commented] (CASSANDRA-13020) Stuck in LEAVING state (Transferring all hints to null)
[
https://issues.apache.org/jira/browse/CASSANDRA-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821557#comment-15821557
]
Stefan Podkowinski commented on CASSANDRA-13020:
The issues seems to be caused by not using the broadcast IP for retrieving the
host ID from TokenMetadata. See
[StorageService.getPreferredHintsStreamTarget()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L3972].
/cc [~krummas]
> Stuck in LEAVING state (Transferring all hints to null)
> ---
>
> Key: CASSANDRA-13020
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13020
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: v3.0.9
>Reporter: Aleksandr Ivanov
> Labels: decommission, hints
>
> I tried to decommission one node.
> Node sent all data to another node and got stuck in LEAVING state.
> Log message shows Exception in HintsDispatcher thread.
> Could it be reason of stuck in LEAVING state?
> command output:
> {noformat}
> root@cas-node6:~# time nodetool decommission
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
> at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:203)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:168)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> real147m7.483s
> user0m17.388s
> sys 0m1.968s
> {noformat}
> nodetool netstats:
> {noformat}
> root@cas-node6:~# nodetool netstats
> Mode: LEAVING
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 35082
> Mismatch (Blocking): 18
> Mismatch (Background): 0
> Pool NameActive Pending Completed Dropped
> Large messages n/a 1 0 0
> Small messages n/a 0 16109860 112
> Gossip messages n/a 0 287074 0
> {noformat}
> Log:
> {noformat}
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:52:59,467
> StorageService.java:1170 - LEAVING: sleeping 3 ms for batch processing
> and pending range setup
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,455
> StorageService.java:1170 - LEAVING: replaying batch log and streaming data to
> other nodes
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,910
> StreamResultFuture.java:87 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Executing streaming plan for Unbootstrap
> INFO [StreamConnectionEstablisher:1] 2016-12-07 12:53:39,911
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.17
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,911
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:3] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
>
[jira] [Commented] (CASSANDRA-13020) Stuck in LEAVING state (Transferring all hints to null)
[
https://issues.apache.org/jira/browse/CASSANDRA-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821240#comment-15821240
]
Stefan Podkowinski commented on CASSANDRA-13020:
Did you set listen_address in your cassandra.yaml to the internal IP for all
nodes in your cluster?
> Stuck in LEAVING state (Transferring all hints to null)
> ---
>
> Key: CASSANDRA-13020
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13020
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: v3.0.9
>Reporter: Aleksandr Ivanov
> Labels: decommission, hints
>
> I tried to decommission one node.
> Node sent all data to another node and got stuck in LEAVING state.
> Log message shows Exception in HintsDispatcher thread.
> Could it be reason of stuck in LEAVING state?
> command output:
> {noformat}
> root@cas-node6:~# time nodetool decommission
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
> at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:203)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:168)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> real147m7.483s
> user0m17.388s
> sys 0m1.968s
> {noformat}
> nodetool netstats:
> {noformat}
> root@cas-node6:~# nodetool netstats
> Mode: LEAVING
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 35082
> Mismatch (Blocking): 18
> Mismatch (Background): 0
> Pool NameActive Pending Completed Dropped
> Large messages n/a 1 0 0
> Small messages n/a 0 16109860 112
> Gossip messages n/a 0 287074 0
> {noformat}
> Log:
> {noformat}
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:52:59,467
> StorageService.java:1170 - LEAVING: sleeping 3 ms for batch processing
> and pending range setup
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,455
> StorageService.java:1170 - LEAVING: replaying batch log and streaming data to
> other nodes
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,910
> StreamResultFuture.java:87 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Executing streaming plan for Unbootstrap
> INFO [StreamConnectionEstablisher:1] 2016-12-07 12:53:39,911
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.17
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,911
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:3] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,912
> StorageService.java:1170 - LEAVING: streaming hints to other nodes
> INFO [StreamConnectionEstablisher:2] 2016-12-07
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821190#comment-15821190
]
Stefan Podkowinski commented on CASSANDRA-12888:
Can you please take this kind of discussion to u...@cassandra.apache.org? This
is an issue tracker and you're email spamming all people subscribed to the
ticket to get updates on the actual ticket progress. Thank you.
> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> SSTables streamed during the repair process will first be written locally and
> afterwards either simply added to the pool of existing sstables or, in case
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we
> must put all partitions through the same write path as normal mutations. This
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the
> streamed rows will end up in the unrepaired set, in contrast to the rows on
> the sender site moved to the repaired set. The next repair run will stream
> the same data back again, causing rows to bounce on and on between nodes on
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this
> manually using ccm can be found
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12653) In-flight shadow round requests
[
https://issues.apache.org/jira/browse/CASSANDRA-12653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818621#comment-15818621
]
Stefan Podkowinski commented on CASSANDRA-12653:
The firstSynSendAt value is only set while sending the first syn during regular
gossip. All ACKs will be ignored that have been received and queued before that
point. Please keep in mind that the first SYN is send by GossipTasks by using
delayed execution, so it's not enough to simply check if the Gossiper has
already been started.
You are correct in pointing out that the timestamp of the incoming ACK will be
a local nanoTime() value. In-flight ACKs triggered during the shadow round will
not be filtered by the timestamp check unless already deserialized. But I'd
prefer comparing local timestamps anyways, as they will not require some kind
of clock skew tolerance, which would be hard to come up with given the kind of
races we like to address. Also this shouldn't be a problem as, like you
mentioned, the current handling of empty shadow round ACK replies should be
safe. But it's not a very clean solution and I'd rather be able to either have
a causal relationship to the corresponding SYN/Gossip round or even a custom
ShadowGossipMessage type. But this is probably better taken care of in e.g.
CASSANDRA-12345 instead of this bug fix.
> In-flight shadow round requests
> ---
>
> Key: CASSANDRA-12653
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12653
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Attachments: 12653-2.2.patch, 12653-3.0.patch, 12653-trunk.patch
>
>
> Bootstrapping or replacing a node in the cluster requires to gather and check
> some host IDs or tokens by doing a gossip "shadow round" once before joining
> the cluster. This is done by sending a gossip SYN to all seeds until we
> receive a response with the cluster state, from where we can move on in the
> bootstrap process. Receiving a response will call the shadow round done and
> calls {{Gossiper.resetEndpointStateMap}} for cleaning up the received state
> again.
> The issue here is that at this point there might be other in-flight requests
> and it's very likely that shadow round responses from other seeds will be
> received afterwards, while the current state of the bootstrap process doesn't
> expect this to happen (e.g. gossiper may or may not be enabled).
> One side effect will be that MigrationTasks are spawned for each shadow round
> reply except the first. Tasks might or might not execute based on whether at
> execution time {{Gossiper.resetEndpointStateMap}} had been called, which
> effects the outcome of {{FailureDetector.instance.isAlive(endpoint))}} at
> start of the task. You'll see error log messages such as follows when this
> happend:
> {noformat}
> INFO [SharedPool-Worker-1] 2016-09-08 08:36:39,255 Gossiper.java:993 -
> InetAddress /xx.xx.xx.xx is now UP
> ERROR [MigrationStage:1]2016-09-08 08:36:39,255 FailureDetector.java:223
> - unknown endpoint /xx.xx.xx.xx
> {noformat}
> Although is isn't pretty, I currently don't see any serious harm from this,
> but it would be good to get a second opinion (feel free to close as "wont
> fix").
> /cc [~Stefania] [~thobbs]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13020) Stuck in LEAVING state (Transferring all hints to null)
[
https://issues.apache.org/jira/browse/CASSANDRA-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818454#comment-15818454
]
Stefan Podkowinski commented on CASSANDRA-13020:
Is your cluster hosted in EC2 or did you enable the prefer_local option in your
cassandra-rackdc.properties?
> Stuck in LEAVING state (Transferring all hints to null)
> ---
>
> Key: CASSANDRA-13020
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13020
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: v3.0.9
>Reporter: Aleksandr Ivanov
> Labels: decommission, hints
>
> I tried to decommission one node.
> Node sent all data to another node and got stuck in LEAVING state.
> Log message shows Exception in HintsDispatcher thread.
> Could it be reason of stuck in LEAVING state?
> command output:
> {noformat}
> root@cas-node6:~# time nodetool decommission
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
> at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:203)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:168)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> real147m7.483s
> user0m17.388s
> sys 0m1.968s
> {noformat}
> nodetool netstats:
> {noformat}
> root@cas-node6:~# nodetool netstats
> Mode: LEAVING
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 35082
> Mismatch (Blocking): 18
> Mismatch (Background): 0
> Pool NameActive Pending Completed Dropped
> Large messages n/a 1 0 0
> Small messages n/a 0 16109860 112
> Gossip messages n/a 0 287074 0
> {noformat}
> Log:
> {noformat}
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:52:59,467
> StorageService.java:1170 - LEAVING: sleeping 3 ms for batch processing
> and pending range setup
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,455
> StorageService.java:1170 - LEAVING: replaying batch log and streaming data to
> other nodes
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,910
> StreamResultFuture.java:87 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Executing streaming plan for Unbootstrap
> INFO [StreamConnectionEstablisher:1] 2016-12-07 12:53:39,911
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.17
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,911
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:3] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,912
> StorageService.java:1170 - LEAVING: streaming hints to other nodes
> INFO [StreamConnectionEstablisher:2]
[jira] [Comment Edited] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818371#comment-15818371
]
Stefan Podkowinski edited comment on CASSANDRA-13058 at 1/11/17 1:52 PM:
-
The test is passing because hint delivery is not resumable in 3.0
(CASSANDRA-6230). This has only been fixed lately in 3.10 (CASSANDRA-11960).
As due to the missing reply messages addressed in the patch, node2 would never
respond, while handling non-local hints. That will in turn cause all callbacks
on node1 to time out and the HintDispatcher will retry hint delievery. At this
point, the FailureDetector is correctly reporting the node as alive and the
dispatch process will not be aborted but simply try to consume the next hints
from a now empty iterator and terminate with a successful return value
afterwards. The log file will contain a "Finished hinted handoff of file
[].hints to endpoint []", which is technically correct, but is probably a bit
misleading in case all writes just timed out.
Even with the FailureDetector reporting the target node as unavailable and
running into the ABORT case, we'd still get the Exception reported in
CASSANDRA-11960. All things considered chances are high that hints will be lost
in 3.0 in case of any errors during delivery.
was (Author: spo...@gmail.com):
The test is passing because hint delivery is not resumable in 3.0
(https://issues.apache.org/jira/browse/CASSANDRA-6230). This has only been
fixed lately in 3.10 (CASSANDRA-11960).
As due to the missing reply messages addressed in the patch, node2 would never
respond, while handling non-local hints. That will in turn cause all callbacks
on node1 to time out and the HintDispatcher will retry hint delievery. At this
point, the FailureDetector is correctly reporting the node as alive and the
dispatch process will not be aborted but simply try to consume the next hints
from a now empty iterator and terminate with a successful return value
afterwards. The log file will contain a "Finished hinted handoff of file
[].hints to endpoint []", which is technically correct, but is probably a bit
misleading in case all writes just timed out.
Even with the FailureDetector reporting the target node as unavailable and
running into the ABORT case, we'd still get the Exception reported in
CASSANDRA-11960. All things considered chances are high that hints will be lost
in 3.0 in case of any errors during delivery.
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Assignee: Stefan Podkowinski
>Priority: Blocker
> Labels: dtest, test-failure
> Fix For: 3.10
>
> Attachments: 13058-3.x.patch, node1.log, node1_debug.log,
> node1_gc.log, node2.log, node2_debug.log, node2_gc.log, node3.log,
> node3_debug.log, node3_gc.log, node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']]
> exited with non-zero status; exit status: 2;
> stderr: error: Error while decommissioning node: Failed to transfer all hints
> to 59f20b4f-0215-4e18-be1b-7e00f2901629
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in
> hintedhandoff_decom_test
> node1.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}{code}
> java.lang.RuntimeException: Error while decommissioning node: Failed to
> transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
> at
> org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Commented] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818371#comment-15818371
]
Stefan Podkowinski commented on CASSANDRA-13058:
The test is passing because hint delivery is not resumable in 3.0
(https://issues.apache.org/jira/browse/CASSANDRA-6230). This has only been
fixed lately in 3.10 (CASSANDRA-11960).
As due to the missing reply messages addressed in the patch, node2 would never
respond, while handling non-local hints. That will in turn cause all callbacks
on node1 to time out and the HintDispatcher will retry hint delievery. At this
point, the FailureDetector is correctly reporting the node as alive and the
dispatch process will not be aborted but simply try to consume the next hints
from a now empty iterator and terminate with a successful return value
afterwards. The log file will contain a "Finished hinted handoff of file
[].hints to endpoint []", which is technically correct, but is probably a bit
misleading in case all writes just timed out.
Even with the FailureDetector reporting the target node as unavailable and
running into the ABORT case, we'd still get the Exception reported in
CASSANDRA-11960. All things considered chances are high that hints will be lost
in 3.0 in case of any errors during delivery.
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Assignee: Stefan Podkowinski
>Priority: Blocker
> Labels: dtest, test-failure
> Fix For: 3.10
>
> Attachments: 13058-3.x.patch, node1.log, node1_debug.log,
> node1_gc.log, node2.log, node2_debug.log, node2_gc.log, node3.log,
> node3_debug.log, node3_gc.log, node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']]
> exited with non-zero status; exit status: 2;
> stderr: error: Error while decommissioning node: Failed to transfer all hints
> to 59f20b4f-0215-4e18-be1b-7e00f2901629
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in
> hintedhandoff_decom_test
> node1.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}{code}
> java.lang.RuntimeException: Error while decommissioning node: Failed to
> transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
> at
> org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
>
[jira] [Commented] (CASSANDRA-13059) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.rolling_upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814881#comment-15814881
]
Stefan Podkowinski commented on CASSANDRA-13059:
Test seems to pass locally with c612cd8d7 (CASSANDRA-12620) reverted.
/cc [~blerer]
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.rolling_upgrade_test
>
>
> Key: CASSANDRA-13059
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13059
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Assignee: T Jake Luciani
> Labels: dtest, test-failure
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_large_dtest/16/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/rolling_upgrade_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['upgradesstables',
> '-a']] exited with non-zero status; exit status: 2;
> stderr: error: null
> -- StackTrace --
> java.lang.AssertionError
> at org.apache.cassandra.db.rows.Rows.collectStats(Rows.java:88)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$StatsCollector.applyToRow(BigTableWriter.java:237)
> at
> org.apache.cassandra.db.transform.BaseRows.applyOne(BaseRows.java:120)
> at org.apache.cassandra.db.transform.BaseRows.add(BaseRows.java:110)
> at
> org.apache.cassandra.db.transform.UnfilteredRows.add(UnfilteredRows.java:44)
> at
> org.apache.cassandra.db.transform.Transformation.add(Transformation.java:174)
> at
> org.apache.cassandra.db.transform.Transformation.apply(Transformation.java:140)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:171)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:141)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:199)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:420)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:312)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$204(NamedThreadFactory.java:81)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1992550266.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 279, in rolling_upgrade_test
> self.upgrade_scenario(rolling=True)
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 345, in upgrade_scenario
> self.upgrade_to_version(version_meta, partial=True, nodes=(node,))
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 446, in upgrade_to_version
> node.nodetool('upgradesstables -a')
> File
> "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line
> 783, in nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File
> "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line
> 1993, in handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}
> Related failures:
>
[jira] [Commented] (CASSANDRA-13114) 3.0.x: update netty
[ https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814421#comment-15814421 ] Stefan Podkowinski commented on CASSANDRA-13114: Netty 4.0.23 has been introduced in CASSANDRA-7761 and is used since 2.1.1. Looking at the [4.0.27 release notes|http://netty.io/news/2015/04/02/4-0-27-Final.html] I assume that [#3567|https://github.com/netty/netty/issues/3567] fixes the memory issue you're referring to? If I read the ticket correctly, the bug does not manifest in 4.0.23, but only after updating to 4.0.25/26. Can you link the actual issue that makes you believe 4.0.23 is affected in case there's something I'm missing here? > 3.0.x: update netty > --- > > Key: CASSANDRA-13114 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13114 > Project: Cassandra > Issue Type: Bug >Reporter: Tom van der Woerdt > > https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for > Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with > Cassandra 3.0.x, has some serious bugs around memory handling for SSL > connections. > It would be nice if both were updated to 4.0.42, a version released this year. > 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory > every ~30 minutes. This was fixed in 4.0.27. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812110#comment-15812110
]
Stefan Podkowinski commented on CASSANDRA-13058:
[~pauloricardomg], is this something you could review?
The failure was triggered by 48abc03697, see attached patch for the fix. This
should make the test work for 3.11. Unfortunately I haven't found a way to
trigger dtests on jenkins with the "--vnodes false" option. I guess someone
needs to setup a job manually for that. You can test locally using:
{{./run_dtests.py --vnodes false --nose-options -v
hintedhandoff_test.py:TestHintedHandoff.hintedhandoff_decom_test}}.
I've also notice that the same dtest will stop working again after
CASSANDRA-12510. This needs to be addressed as well, but should not further
block 3.10.
That being said, it's less than ideal that dtests annotated with "@no_vnodes"
are simply skipped during ad-hoc job scheduling during reviews. Eventually
those failures will be caught by the cassandra_*_novnode_dtest job, but
investigating those will be more painful compared to the situation where we'd
get these failures through a ticket related, ad-hoc job during review.
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Assignee: Stefan Podkowinski
>Priority: Blocker
> Labels: dtest, test-failure
> Fix For: 3.10
>
> Attachments: 13058-3.x.patch, node1.log, node1_debug.log,
> node1_gc.log, node2.log, node2_debug.log, node2_gc.log, node3.log,
> node3_debug.log, node3_gc.log, node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']]
> exited with non-zero status; exit status: 2;
> stderr: error: Error while decommissioning node: Failed to transfer all hints
> to 59f20b4f-0215-4e18-be1b-7e00f2901629
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in
> hintedhandoff_decom_test
> node1.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}{code}
> java.lang.RuntimeException: Error while decommissioning node: Failed to
> transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
> at
> org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
>
[jira] [Updated] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13058:
---
Status: Patch Available (was: Open)
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Assignee: Stefan Podkowinski
>Priority: Blocker
> Labels: dtest, test-failure
> Fix For: 3.10
>
> Attachments: 13058-3.x.patch, node1.log, node1_debug.log,
> node1_gc.log, node2.log, node2_debug.log, node2_gc.log, node3.log,
> node3_debug.log, node3_gc.log, node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']]
> exited with non-zero status; exit status: 2;
> stderr: error: Error while decommissioning node: Failed to transfer all hints
> to 59f20b4f-0215-4e18-be1b-7e00f2901629
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in
> hintedhandoff_decom_test
> node1.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}{code}
> java.lang.RuntimeException: Error while decommissioning node: Failed to
> transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
> at
> org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:828)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at
[jira] [Updated] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13058:
---
Attachment: 13058-3.x.patch
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Assignee: Stefan Podkowinski
>Priority: Blocker
> Labels: dtest, test-failure
> Fix For: 3.10
>
> Attachments: 13058-3.x.patch, node1.log, node1_debug.log,
> node1_gc.log, node2.log, node2_debug.log, node2_gc.log, node3.log,
> node3_debug.log, node3_gc.log, node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']]
> exited with non-zero status; exit status: 2;
> stderr: error: Error while decommissioning node: Failed to transfer all hints
> to 59f20b4f-0215-4e18-be1b-7e00f2901629
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in
> hintedhandoff_decom_test
> node1.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}{code}
> java.lang.RuntimeException: Error while decommissioning node: Failed to
> transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
> at
> org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:828)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at
[jira] [Comment Edited] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811534#comment-15811534
]
Stefan Podkowinski edited comment on CASSANDRA-13058 at 1/9/17 3:56 PM:
This seems to be caused by CASSANDRA-12905, as hints for which the node is not
the final destination are no longer ACKed.
||3.11||3.x||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13058-3.11]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13058-3.x]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13058-3.11-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13058-3.x-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13058-3.11-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13058-3.x-testall/]|
was (Author: spo...@gmail.com):
This seems to be caused by CASSANDRA-12905, as hints for which the node is not
the final destination are no longer ACKed.
||3.x||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13058-3.x]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13058-3.x-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13058-3.x-testall/]|
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Assignee: Stefan Podkowinski
>Priority: Blocker
> Labels: dtest, test-failure
> Fix For: 3.10
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log,
> node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']]
> exited with non-zero status; exit status: 2;
> stderr: error: Error while decommissioning node: Failed to transfer all hints
> to 59f20b4f-0215-4e18-be1b-7e00f2901629
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in
> hintedhandoff_decom_test
> node1.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}{code}
> java.lang.RuntimeException: Error while decommissioning node: Failed to
> transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
> at
> org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
>
[jira] [Assigned] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski reassigned CASSANDRA-13058:
--
Assignee: Stefan Podkowinski
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Assignee: Stefan Podkowinski
>Priority: Blocker
> Labels: dtest, test-failure
> Fix For: 3.10
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log,
> node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']]
> exited with non-zero status; exit status: 2;
> stderr: error: Error while decommissioning node: Failed to transfer all hints
> to 59f20b4f-0215-4e18-be1b-7e00f2901629
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in
> hintedhandoff_decom_test
> node1.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}{code}
> java.lang.RuntimeException: Error while decommissioning node: Failed to
> transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
> at
> org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:828)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native
[jira] [Commented] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811534#comment-15811534
]
Stefan Podkowinski commented on CASSANDRA-13058:
This seems to be caused by CASSANDRA-12905, as hints for which the node is not
the final destination are no longer ACKed.
||3.x||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13058-3.x]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13058-3.x-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13058-3.x-testall/]|
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>Priority: Blocker
> Labels: dtest, test-failure
> Fix For: 3.10
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log,
> node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/
> {code}
> Error Message
> Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']]
> exited with non-zero status; exit status: 2;
> stderr: error: Error while decommissioning node: Failed to transfer all hints
> to 59f20b4f-0215-4e18-be1b-7e00f2901629
> {code}{code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in
> hintedhandoff_decom_test
> node1.decommission()
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in
> decommission
> self.nodetool("decommission")
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in
> nodetool
> return handle_external_tool_process(p, ['nodetool', '-h', 'localhost',
> '-p', str(self.jmx_port), cmd.split()])
> File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in
> handle_external_tool_process
> raise ToolError(cmd_args, rc, out, err)
> {code}{code}
> java.lang.RuntimeException: Error while decommissioning node: Failed to
> transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
> at
> org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:828)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Issue Comment Deleted] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13058:
---
Comment: was deleted
(was: Edit: looks like I did something wrong while reverting the mentioned
commit. It does not effect the test result. Sorry for that.
I still want to leave the rest of my comment here as it may indicate that this
could be indeed related to some buggy gossip state/failure detection handling.
-This looks like a regression caused by CASSANDRA-12192 to me-. -The test seems
to work fine without the commit.-
Take a look at this dtest log:
{{./run_dtests.py --vnodes false --nose-options -v
hintedhandoff_test.py:TestHintedHandoff.hintedhandoff_decom_test}}
{noformat}
Node 4:
DEBUG [StorageServiceShutdownHook] 2017-01-04 15:06:16,748
StorageService.java:1437 - DRAINING: starting drain process
DEBUG [StorageServiceShutdownHook] 2017-01-04 15:06:20,011
StorageService.java:1437 - DRAINED
Node 1:
INFO [GossipStage:1] 2017-01-04 15:06:16,756 Gossiper.java:1036 - InetAddress
/127.0.0.4 is now DOWN
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 FailureDetector.java:325 -
Forcing conviction of /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,758 Gossiper.java:370 - /127.0.0.4
marked as alive: false
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:16,758
OutboundTcpConnection.java:374 - Socket to /127.0.0.4 closed
DEBUG [MessagingService-Outgoing-/127.0.0.4-Small] 2017-01-04 15:06:16,759
OutboundTcpConnection.java:374 - Socket to /127.0.0.4 closed
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:17,883
OutboundTcpConnection.java:388 - Attempting to connect to /127.0.0.4
INFO [HANDSHAKE-/127.0.0.4] 2017-01-04 15:06:17,884
OutboundTcpConnection.java:510 - Handshaking version with /127.0.0.4
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:17,884
OutboundTcpConnection.java:482 - Done connecting to /127.0.0.4
DEBUG [Native-Transport-Requests-3] 2017-01-04 15:06:20,825
FailureDetector.java:252 - /127.0.0.4 is ALIVE
INFO [RMI TCP Connection(2)-127.0.0.1] 2017-01-04 15:06:52,930
StorageService.java:1435 - LEAVING: streaming hints to other nodes
INFO [HintsDispatcher:2] 2017-01-04 15:06:52,955
HintsDispatchExecutor.java:152 - Transferring all hints to
054006b5-412c-4d4e-a28a-7610574de79 d
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:53,035
OutboundTcpConnection.java:495 - Unable to connect to /127.0.0.4
java.net.ConnectException: Connection refused
▸ at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_112]
▸ at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_112]
▸ at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_112]
▸ at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
~[na:1.8.0_112]
▸ at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146)
~[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132)
~[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:397)
[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:234)
[main/:na]
{noformat}
I've added some additional log messages, so don't get confused when you can't
find some of them in the code. What's happening here is that the gossiper will
down node1 and will trigger {{OutboundTcpConnectionPool.reset()}} via
{{MessagingService.convict()}}. This will cause a reconnect shortly, as the
message should now always be retried from the backlog again.
Unfortunately I can't see which message is affected here. But my assumption is
that node4 keeps accepting messages even for a short time after gossip change
propagation and will return a response to node1, which will flag node4 alive
again in the process. But I'm not exactly sure, as the node status related code
is not easy to understand with all the side effects and unclear state handling.
:(
/cc [~thobbs]
)
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
>
[jira] [Updated] (CASSANDRA-13023) Add droppable tombstone metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13023:
---
Summary: Add droppable tombstone metrics (was: Add Tombstone Metrics)
> Add droppable tombstone metrics
> ---
>
> Key: CASSANDRA-13023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13023
> Project: Cassandra
> Issue Type: Improvement
> Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 13023-3.0.patch
>
>
> Currently it's only possible to inspect the ratio of droppable tombstones on
> sstable basis using the {{sstablemetadata}} tool. It would be very useful to
> have this information provided as part of Cassandra metrics as well, e.g. to
> get an idea on the effectiveness of tombstone eviction during compactions
> over time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13052:
---
Status: Patch Available (was: In Progress)
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: 13052-2.1.patch, ccm_reproduce-13052.txt,
> system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801665#comment-15801665
]
Stefan Podkowinski commented on CASSANDRA-13052:
I've started tests for my attached patch.
||2.1||2.2||3.0||3.x||trunk||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13052-2.1]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13052-2.2]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13052-3.0]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13052-3.x]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13052-trunk]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-2.1-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-3.x-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-trunk-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-2.1-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-3.x-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13052-trunk-testall/]|
Happy to contribute a dtest as well if someone would step up reviewing the
patch.
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: 13052-2.1.patch, ccm_reproduce-13052.txt,
> system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13052:
---
Attachment: 13052-2.1.patch
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: 13052-2.1.patch, ccm_reproduce-13052.txt,
> system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13052:
---
Attachment: (was: system-dev-debug-13052.log)
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13052:
---
Attachment: system-dev-debug-13052.log
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log,
> system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801192#comment-15801192
]
Stefan Podkowinski commented on CASSANDRA-13052:
Which repairs are affected? The tree depth will effectively calculated as
{{min(log(partitions), 20)}}, ie. the depth will increase logarithmic until
capped at 20. In the worst case scenario, the described bug will happen when
the repaired token range will be less than 2^20. Using the Murmur3Partitioner,
this is still less than individual ranges created by using vnodes in a large
cluster {{(2^64 / (vnodes * nodes))}}. As vnodes are created based on random
values, there's still a chance that two vnodes are close enough together to get
affected by this bug regulary. Although this requires a fairly large amount of
partitions and I'm wondering if it would be practically possible to run repairs
in this setup anyways. More likely this could be an issue for tools that would
work by calculating their own (smaller) ranges instead of running repairs based
on (v)nodes.
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801182#comment-15801182
]
Stefan Podkowinski commented on CASSANDRA-13052:
The issue is not that ranges will be added multiple times, but the fact that
the added range has an equal start and endpoint and is thus invalid by
definition. Things starts to go awry in the normalize method, which will handle
the range as a wrap-around value and turns the specified range into another
range of (MINIMUM, MINIMUM). This range is handled special in some occassions
and seems to cause {{DataTracker.sstablesInBounds()}} return basically all
sstables. Based on those tables, the SSTable section calculation will return
all content from each SSTable as well and so it looks to me that this will
cause streaming of all data.
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13096) Snapshots slow down jmx scrapping
[
https://issues.apache.org/jira/browse/CASSANDRA-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800953#comment-15800953
]
Stefan Podkowinski commented on CASSANDRA-13096:
I think you should at least limit scraping to metric mbeans by having a
whitelist as in [this|https://github.com/prometheus/jmx_exporter#configuration]
example. Constantly polling every attribute of all mbeans sounds like trouble
to me, as even reading non-metric mbeans may have side effects.
> Snapshots slow down jmx scrapping
> -
>
> Key: CASSANDRA-13096
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13096
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxime Fouilleul
> Attachments: CPU Load.png, Clear Snapshots.png, JMX Scrape
> Duration.png
>
>
> Hello,
> We are scraping the jmx metrics through a prometheus exporter and we noticed
> that some nodes became really long to answer (more than 20 seconds). After
> some investigations we do not find any hardware problem or overload issues on
> there "slow" nodes. It happens on different clusters, some with only few giga
> bytes of dataset and it does not seams to be related to a specific version
> neither as it happens on 2.1, 2.2 and 3.0 nodes.
> After some unsuccessful actions, one of our ideas was to clean the snapshots
> staying on one problematic node:
> {code}
> nodetool clearsnapshot
> {code}
> And the magic happens... as you can see in the attached diagrams, the second
> we cleared the snapshots, the CPU activity dropped immediatly and the
> duration to scrape the jmx metrics goes from +20 secs to instantaneous...
> Can you enlighten us on this issue? Once again, it appears on our three 2.1,
> 2.2 and 3.0 versions, on different volumetry and it is not systematically
> linked to the snapshots as we have some nodes with the same snapshots volume
> which are going pretty well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13096) Snapshots slow down jmx scrapping
[
https://issues.apache.org/jira/browse/CASSANDRA-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800761#comment-15800761
]
Stefan Podkowinski commented on CASSANDRA-13096:
Can you provide more details on how the prometheus integration is configured
(metrics-reporter-config push gateway, -servlet, or simple jmx scraping)? Did
you try to limit the number of read metrics to pin down a specific one as the
actual cause? What metrics are you reading?
> Snapshots slow down jmx scrapping
> -
>
> Key: CASSANDRA-13096
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13096
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxime Fouilleul
> Attachments: CPU Load.png, Clear Snapshots.png, JMX Scrape
> Duration.png
>
>
> Hello,
> We are scraping the jmx metrics through a prometheus exporter and we noticed
> that some nodes became really long to answer (more than 20 seconds). After
> some investigations we do not find any hardware problem or overload issues on
> there "slow" nodes. It happens on different clusters, some with only few giga
> bytes of dataset and it does not seams to be related to a specific version
> neither as it happens on 2.1, 2.2 and 3.0 nodes.
> After some unsuccessful actions, one of our ideas was to clean the snapshots
> staying on one problematic node:
> {code}
> nodetool clearsnapshot
> {code}
> And the magic happens... as you can see in the attached diagrams, the second
> we cleared the snapshots, the CPU activity dropped immediatly and the
> duration to scrape the jmx metrics goes from +20 secs to instantaneous...
> Can you enlighten us on this issue? Once again, it appears on our three 2.1,
> 2.2 and 3.0 versions, on different volumetry and it is not systematically
> linked to the snapshots as we have some nodes with the same snapshots volume
> which are going pretty well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798508#comment-15798508
]
Stefan Podkowinski edited comment on CASSANDRA-13058 at 1/4/17 4:22 PM:
Edit: looks like I did something wrong while reverting the mentioned commit. It
does not effect the test result. Sorry for that.
I still want to leave the rest of my comment here as it may indicate that this
could be indeed related to some buggy gossip state/failure detection handling.
-This looks like a regression caused by CASSANDRA-12192 to me-. -The test seems
to work fine without the commit.-
Take a look at this dtest log:
{{./run_dtests.py --vnodes false --nose-options -v
hintedhandoff_test.py:TestHintedHandoff.hintedhandoff_decom_test}}
{noformat}
Node 4:
DEBUG [StorageServiceShutdownHook] 2017-01-04 15:06:16,748
StorageService.java:1437 - DRAINING: starting drain process
DEBUG [StorageServiceShutdownHook] 2017-01-04 15:06:20,011
StorageService.java:1437 - DRAINED
Node 1:
INFO [GossipStage:1] 2017-01-04 15:06:16,756 Gossiper.java:1036 - InetAddress
/127.0.0.4 is now DOWN
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 FailureDetector.java:325 -
Forcing conviction of /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,758 Gossiper.java:370 - /127.0.0.4
marked as alive: false
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:16,758
OutboundTcpConnection.java:374 - Socket to /127.0.0.4 closed
DEBUG [MessagingService-Outgoing-/127.0.0.4-Small] 2017-01-04 15:06:16,759
OutboundTcpConnection.java:374 - Socket to /127.0.0.4 closed
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:17,883
OutboundTcpConnection.java:388 - Attempting to connect to /127.0.0.4
INFO [HANDSHAKE-/127.0.0.4] 2017-01-04 15:06:17,884
OutboundTcpConnection.java:510 - Handshaking version with /127.0.0.4
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:17,884
OutboundTcpConnection.java:482 - Done connecting to /127.0.0.4
DEBUG [Native-Transport-Requests-3] 2017-01-04 15:06:20,825
FailureDetector.java:252 - /127.0.0.4 is ALIVE
INFO [RMI TCP Connection(2)-127.0.0.1] 2017-01-04 15:06:52,930
StorageService.java:1435 - LEAVING: streaming hints to other nodes
INFO [HintsDispatcher:2] 2017-01-04 15:06:52,955
HintsDispatchExecutor.java:152 - Transferring all hints to
054006b5-412c-4d4e-a28a-7610574de79 d
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:53,035
OutboundTcpConnection.java:495 - Unable to connect to /127.0.0.4
java.net.ConnectException: Connection refused
▸ at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_112]
▸ at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_112]
▸ at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_112]
▸ at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
~[na:1.8.0_112]
▸ at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146)
~[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132)
~[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:397)
[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:234)
[main/:na]
{noformat}
I've added some additional log messages, so don't get confused when you can't
find some of them in the code. What's happening here is that the gossiper will
down node1 and will trigger {{OutboundTcpConnectionPool.reset()}} via
{{MessagingService.convict()}}. This will cause a reconnect shortly, as the
message should now always be retried from the backlog again.
Unfortunately I can't see which message is affected here. But my assumption is
that node4 keeps accepting messages even for a short time after gossip change
propagation and will return a response to node1, which will flag node4 alive
again in the process. But I'm not exactly sure, as the node status related code
is not easy to understand with all the side effects and unclear state handling.
:(
/cc [~thobbs]
was (Author: spo...@gmail.com):
This looks like a regression caused by
[jira] [Commented] (CASSANDRA-13058) dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798508#comment-15798508
]
Stefan Podkowinski commented on CASSANDRA-13058:
This looks like a regression caused by CASSANDRA-12192 to me. The test seems to
work fine without the commit.
Take a look at this dtest log:
{{./run_dtests.py --vnodes false --nose-options -v
hintedhandoff_test.py:TestHintedHandoff.hintedhandoff_decom_test}}
{noformat}
Node 4:
DEBUG [StorageServiceShutdownHook] 2017-01-04 15:06:16,748
StorageService.java:1437 - DRAINING: starting drain process
DEBUG [StorageServiceShutdownHook] 2017-01-04 15:06:20,011
StorageService.java:1437 - DRAINED
Node 1:
INFO [GossipStage:1] 2017-01-04 15:06:16,756 Gossiper.java:1036 - InetAddress
/127.0.0.4 is now DOWN
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 OutboundTcpConnection.java:180 -
Enqueuing socket close for /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,757 FailureDetector.java:325 -
Forcing conviction of /127.0.0.4
DEBUG [GossipStage:1] 2017-01-04 15:06:16,758 Gossiper.java:370 - /127.0.0.4
marked as alive: false
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:16,758
OutboundTcpConnection.java:374 - Socket to /127.0.0.4 closed
DEBUG [MessagingService-Outgoing-/127.0.0.4-Small] 2017-01-04 15:06:16,759
OutboundTcpConnection.java:374 - Socket to /127.0.0.4 closed
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:17,883
OutboundTcpConnection.java:388 - Attempting to connect to /127.0.0.4
INFO [HANDSHAKE-/127.0.0.4] 2017-01-04 15:06:17,884
OutboundTcpConnection.java:510 - Handshaking version with /127.0.0.4
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:17,884
OutboundTcpConnection.java:482 - Done connecting to /127.0.0.4
DEBUG [Native-Transport-Requests-3] 2017-01-04 15:06:20,825
FailureDetector.java:252 - /127.0.0.4 is ALIVE
INFO [RMI TCP Connection(2)-127.0.0.1] 2017-01-04 15:06:52,930
StorageService.java:1435 - LEAVING: streaming hints to other nodes
INFO [HintsDispatcher:2] 2017-01-04 15:06:52,955
HintsDispatchExecutor.java:152 - Transferring all hints to
054006b5-412c-4d4e-a28a-7610574de79 d
DEBUG [MessagingService-Outgoing-/127.0.0.4-Gossip] 2017-01-04 15:06:53,035
OutboundTcpConnection.java:495 - Unable to connect to /127.0.0.4
java.net.ConnectException: Connection refused
▸ at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_112]
▸ at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_112]
▸ at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_112]
▸ at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
~[na:1.8.0_112]
▸ at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146)
~[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132)
~[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:397)
[main/:na]
▸ at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:234)
[main/:na]
{noformat}
I've added some additional log messages, so don't get confused when you can't
find some of them in the code. What's happening here is that the gossiper will
down node1 and will trigger {{OutboundTcpConnectionPool.reset()}} via
{{MessagingService.convict()}}. This will cause a reconnect shortly, as the
message should now always be retried from the backlog again.
Unfortunately I can't see which message is affected here. But my assumption is
that node4 keeps accepting messages even for a short time after gossip change
propagation and will return a response to node1, which will flag node4 alive
again in the process. But I'm not exactly sure, as the node status related code
is not easy to understand with all the side effects and unclear state handling.
:(
/cc [~thobbs]
> dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test
> --
>
> Key: CASSANDRA-13058
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13058
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Sean McCarthy
>
[jira] [Commented] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795285#comment-15795285
]
Stefan Podkowinski commented on CASSANDRA-13052:
{quote}
We soon notice heavy streaming and according to the logs the number of ranges
streamed was in thousands.
{quote}
[~cmposto], I'm currently a bit confused while trying to evaluate the actual
effect of this behaviour. I was first a bit concerned about either having
ranges streamed that shouldn't be repaired or to send identical ranges thousand
of times. But I've come to the conclusion that none of this should happen, as
{{StreamSession.addTransferRanges()}} should normalize all redundant ranges
into a singe range, before starting to stream any files. Can you further
describe how this bug resulted into "heavy streaming" on your cluster? What was
is that you noticed before further looking into this issue?
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval
[
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15792783#comment-15792783
]
Stefan Podkowinski commented on CASSANDRA-11349:
I've looked at some metrics today for one of our clusters that has been updated
to 2.1.16 a couple of weeks ago. We used to see tens of thousands of sstables
getting streamed each night during repairs with many GBs.
With 2.1.16 the number of streamed sstables went down to almost none. Thanks
for fixing this to everyone involved! :)
> MerkleTree mismatch when multiple range tombstones exists for the same
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
> Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Branimir Lambov
> Labels: repair
> Fix For: 2.1.16, 2.2.8
>
> Attachments: 11349-2.1-v2.patch, 11349-2.1-v3.patch,
> 11349-2.1-v4.patch, 11349-2.1.patch, 11349-2.2-v4.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters,
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a
> partition for the same range/interval, they're both included in the merkle
> tree computation.
> But, if for some reason, on another node, the two range tombstones were
> already compacted into a single range tombstone, this will result in a merkle
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent
> on compactions (and if a partition is deleted and created multiple times, the
> only way to ensure that repair "works correctly"/"don't overstream data" is
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables
> (up to thousands for a rather short period of time when using VNodes, the
> time for compaction to absorb those small files), but also an increased size
> on disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9625) GraphiteReporter not reporting
[
https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15792585#comment-15792585
]
Stefan Podkowinski commented on CASSANDRA-9625:
---
[~llambiel], I can't see anything that would indicate any deadlocks in your
provided thread dump. In fact {{metrics-graphite-reporter-thread-1}} is just
waiting for the next tick to execute and pull the latest metrics from
Cassandra. Are you sure this isn't a problem with your Graphite installation?
Are there any error messages in Cassandra related to Graphite? Are there any
connections from Cassandra to Graphite (as can be checked using netstat)?
> GraphiteReporter not reporting
> --
>
> Key: CASSANDRA-9625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>Reporter: Eric Evans
>Assignee: Stefan Podkowinski
> Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml,
> thread-dump.log, thread-dump2.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops
> working. The usual startup is logged, and one batch of samples is sent, but
> the reporting interval comes and goes, and no other samples are ever sent.
> The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment
> on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not
> on a 3 node (otherwise identical) staging cluster (maybe it takes a certain
> level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766949#comment-15766949
]
Stefan Podkowinski edited comment on CASSANDRA-13052 at 12/21/16 12:34 PM:
---
My [WIP
branch|https://github.com/apache/cassandra/compare/cassandra-2.1...spodkowinski:WIP-13052]
has now been updated with an exit condition that should avoid running into the
situation to have the midpoint method called with an invalid range.
The current MT code recursively looks for the node closest to the leafs by
continually dividing the range by it's midpoint and returning at a point when
any child is not contained anymore in the range we're looking for.
Unfortunately this condition will not apply when we have a leaf in the MT that
exactly spans the range of a single token we're searching. See
[MerkleTree.findHelper|https://github.com/apache/cassandra/blob/4a2464192e9e69457f5a5ecf26c094f9298bf069/src/java/org/apache/cassandra/utils/MerkleTree.java#L420].
I still want to investigate how likely this going to happen for larger ranges,
e.g. complete vnodes.
was (Author: spo...@gmail.com):
My [WIP
branch|https://github.com/apache/cassandra/compare/cassandra-2.1...spodkowinski:WIP-13052]
has now been updated with an exit condition that should avoid running into the
situation to have the midpoint method called with an invalid range.
The current MT code recursively looks for the node closest to the leafs by
continually dividing the range by it's midpoint and returning at a point when
any child is not contained anymore in the range we're looking for.
Unfortunately this condition will not apply when we have a child in the MT that
exactly spans the range of a single token we're searching. See
[MerkleTree.findHelper|https://github.com/apache/cassandra/blob/4a2464192e9e69457f5a5ecf26c094f9298bf069/src/java/org/apache/cassandra/utils/MerkleTree.java#L420].
I still want to investigate how likely this going to happen for larger ranges,
e.g. complete vnodes.
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski reassigned CASSANDRA-13052:
--
Assignee: Stefan Podkowinski
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Assignee: Stefan Podkowinski
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766949#comment-15766949
]
Stefan Podkowinski commented on CASSANDRA-13052:
My [WIP
branch|https://github.com/apache/cassandra/compare/cassandra-2.1...spodkowinski:WIP-13052]
has now been updated with an exit condition that should avoid running into the
situation to have the midpoint method called with an invalid range.
The current MT code recursively looks for the node closest to the leafs by
continually dividing the range by it's midpoint and returning at a point when
any child is not contained anymore in the range we're looking for.
Unfortunately this condition will not apply when we have a child in the MT that
exactly spans the range of a single token we're searching. See
[MerkleTree.findHelper|https://github.com/apache/cassandra/blob/4a2464192e9e69457f5a5ecf26c094f9298bf069/src/java/org/apache/cassandra/utils/MerkleTree.java#L420].
I still want to investigate how likely this going to happen for larger ranges,
e.g. complete vnodes.
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13052:
---
Priority: Major (was: Minor)
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764262#comment-15764262
]
Stefan Podkowinski commented on CASSANDRA-13052:
Please find instructions how to locally reproduce this issue attached in
{{ccm_reproduce-13052.txt}}. I've also linked my WIP branch, which includes
debug statements for relevant parts of the MerkleTree class that will traverse
and hash two trees based on the midpoint calculation. See
{{system-dev-debug-13052.log}} for example log output.
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Priority: Minor
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13052:
---
Attachment: system-dev-debug-13052.log
ccm_reproduce-13052.txt
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Priority: Minor
> Attachments: ccm_reproduce-13052.txt, system-dev-debug-13052.log
>
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13060) NPE in StorageService.java while bootstrapping
[
https://issues.apache.org/jira/browse/CASSANDRA-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763652#comment-15763652
]
Stefan Podkowinski commented on CASSANDRA-13060:
Possibly related to CASSANDRA-12653.
> NPE in StorageService.java while bootstrapping
> --
>
> Key: CASSANDRA-13060
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13060
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Labels: lhf
> Fix For: 3.0.x
>
>
> Lots of NPE happens when bootstrapping new node:
> {code}
> WARN [SharedPool-Worker-1] 2016-12-19 23:09:09,034
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.service.StorageService.isRpcReady(StorageService.java:1829)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.service.StorageService.notifyUp(StorageService.java:1787)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.service.StorageService.onAlive(StorageService.java:2424)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.gms.Gossiper.realMarkAlive(Gossiper.java:999)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.gms.Gossiper$3.response(Gossiper.java:979)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761524#comment-15761524
]
Stefan Podkowinski commented on CASSANDRA-13052:
I've now reproduced the described situation locally, but I'm not fully sure
about the implications. It's certainly not the intended behavior, so thanks a
lot for reporting!
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Priority: Minor
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13052) Repair process is violating the start/end token limits for small ranges
[
https://issues.apache.org/jira/browse/CASSANDRA-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761019#comment-15761019
]
Stefan Podkowinski commented on CASSANDRA-13052:
Cristian, please be aware that token ranges are start-exclusive and
end-inclusive. The result of a midpoint calculation is probably undefined, in
case an invalid range has been specified.
> Repair process is violating the start/end token limits for small ranges
> ---
>
> Key: CASSANDRA-13052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13052
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: We tried this in 2.0.14 and 3.9, same bug.
>Reporter: Cristian P
>Priority: Minor
>
> We tried to do a single token repair by providing 2 consecutive token values
> for a large column family. We soon notice heavy streaming and according to
> the logs the number of ranges streamed was in thousands.
> After investigation we found a bug in the two partitioner classes we use
> (RandomPartitioner and Murmur3Partitioner).
> The midpoint method used by MerkleTree.differenceHelper method to find ranges
> with differences for streaming returns abnormal values (way out of the
> initial range requested for repair) if the repair requested range is small (I
> expect smaller than 2^15).
> Here is the simple code to reproduce the bug for Murmur3Partitioner:
> Token left = new Murmur3Partitioner.LongToken(123456789L);
> Token right = new Murmur3Partitioner.LongToken(123456789L);
> IPartitioner partitioner = new Murmur3Partitioner();
> Token midpoint = partitioner.midpoint(left, right);
> System.out.println("Murmur3: [ " + left.getToken() + " : " +
> midpoint.getToken() + " : " + right.getToken() + " ]");
> The output is:
> Murmur3: [ 123456789 : -9223372036731319019 : 123456789 ]
> Note that the midpoint token is nowhere near the suggested repair range. This
> will happen if during the parsing of the tree (in
> MerkleTree.differenceHelper) in search for differences there isn't enough
> tokens for the split and the subrange becomes 0 (left.token=right.token) as
> in the above test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9625) GraphiteReporter not reporting
[ https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760774#comment-15760774 ] Stefan Podkowinski commented on CASSANDRA-9625: --- [~llambiel], can you create and attach another thread dump for one of the affected nodes? > GraphiteReporter not reporting > -- > > Key: CASSANDRA-9625 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9625 > Project: Cassandra > Issue Type: Bug > Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3 >Reporter: Eric Evans >Assignee: Stefan Podkowinski > Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml, > thread-dump.log > > > When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops > working. The usual startup is logged, and one batch of samples is sent, but > the reporting interval comes and goes, and no other samples are ever sent. > The logs are free from errors. > Frustratingly, metrics reporting works in our smaller (staging) environment > on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not > on a 3 node (otherwise identical) staging cluster (maybe it takes a certain > level of concurrency?). > Attached is a thread dump, and our metrics.yaml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12859) Column-level permissions
[
https://issues.apache.org/jira/browse/CASSANDRA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751659#comment-15751659
]
Stefan Podkowinski commented on CASSANDRA-12859:
This looks really promising. Just played around with your branch and it worked
as expected for me.
Can you also update the documentation for syntax changes and examples? There's
{{doc/cql3/CQL.textile}} ({{ant generate-cql-html}}) and
{{doc/source/cql/security.rst}} that could use some hints how this works.
> Column-level permissions
>
>
> Key: CASSANDRA-12859
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12859
> Project: Cassandra
> Issue Type: New Feature
> Components: Core, CQL
>Reporter: Boris Melamed
> Labels: doc-impacting
> Attachments: Cassandra Proposal - Column-level permissions v2.docx,
> Cassandra Proposal - Column-level permissions.docx
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> h4. Here is a draft of:
> Cassandra Proposal - Column-level permissions.docx (attached)
> h4. Quoting the 'Overview' section:
> The purpose of this proposal is to add column-level (field-level) permissions
> to Cassandra. It is my intent to soon start implementing this feature in a
> fork, and to submit a pull request once it’s ready.
> h4. Motivation
> Cassandra already supports permissions on keyspace and table (column family)
> level. Sources:
> * http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra
> * https://cassandra.apache.org/doc/latest/cql/security.html#data-control
> At IBM, we have use cases in the area of big data analytics where
> column-level access permissions are also a requirement. All industry RDBMS
> products are supporting this level of permission control, and regulators are
> expecting it from all data-based systems.
> h4. Main day-one requirements
> # Extend CQL (Cassandra Query Language) to be able to optionally specify a
> list of individual columns, in the {{GRANT}} statement. The relevant
> permission types are: {{MODIFY}} (for {{UPDATE}} and {{INSERT}}) and
> {{SELECT}}.
> # Persist the optional information in the appropriate system table
> ‘system_auth.role_permissions’.
> # Enforce the column access restrictions during execution. Details:
> #* Should fit with the existing permission propagation down a role chain.
> #* Proposed message format when a user’s roles give access to the queried
> table but not to all of the selected, inserted, or updated columns:
> "User %s has no %s permission on column %s of table %s"
> #* Error will report only the first checked column.
> Nice to have: list all inaccessible columns.
> #* Error code is the same as for table access denial: 2100.
> h4. Additional day-one requirements
> # Reflect the column-level permissions in statements of type
> {{LIST ALL PERMISSIONS OF someuser;}}
> # When columns are dropped or renamed, trigger purging or adapting of their
> permissions
> # Performance should not degrade in any significant way.
> # Backwards compatibility
> #* Permission enforcement for DBs created before the upgrade should continue
> to work with the same behavior after upgrading to a version that allows
> column-level permissions.
> #* Previous CQL syntax will remain valid, and have the same effect as before.
> h4. Documentation
> *
> https://cassandra.apache.org/doc/latest/cql/security.html#grammar-token-permission
> * Feedback request: any others?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()
[
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745347#comment-15745347
]
Stefan Podkowinski commented on CASSANDRA-13038:
It would be great if you could do some testing again with CASSANDRA-13040.
There still might be performance issues during compaction of sstables with many
distinct deletion times, but it's hard to tell by just looking at the code or
just by doing some quick local testing.
> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Corentin Chary
> Attachments: compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells:
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS',
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in
> StreamingHistogram.update() (which is used to store the estimated tombstone
> drop times).
> This could be caused by a huge number of different deletion times which would
> makes the bin huge but it this histogram should be capped to 100 keys. It's
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the
> fact the this table has a TWCS also gives us an additional hint that sampling
> deletion times would be fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13040) Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
[
https://issues.apache.org/jira/browse/CASSANDRA-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745135#comment-15745135
]
Stefan Podkowinski commented on CASSANDRA-13040:
||3.0||3.x||trunk||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13040-3.0]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13040-3.x]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13040-trunk]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13040-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13040-3.x-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13040-trunk-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13040-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13040-3.x-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13040-trunk-testall/]|
> Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
> ---
>
> Key: CASSANDRA-13040
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13040
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 13040-3.0.patch
>
>
> Starting with 3.0, {{MetadataCollector.updateLocalDeletionTime(int
> newLocalDeletionTime)}} will be called with {{cell.localDeletionTime()}} in
> order to update the min/max deletion times as well as the
> {{estimatedTombstoneDropTime}} histogram. Currently this also happens for
> {{Cell.NO_DELETION_TIME}} ({{Integer.MAX_VALUE}}), which causes the histogram
> to be called for every regular cell.
> This can be easily verified by using {{sstablemetadata}} on any sstable. You
> should be able to find a very high value for 2147483647 in the "Estimated
> tombstone drop times" section.
> At this point, I don't think this is causing serious harm, but could be a
> performance issue (see linked ticket). Calculated droppable TS times should
> not be affected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13040) Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
[
https://issues.apache.org/jira/browse/CASSANDRA-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13040:
---
Attachment: (was: 13040-3.0.patch)
> Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
> ---
>
> Key: CASSANDRA-13040
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13040
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 13040-3.0.patch
>
>
> Starting with 3.0, {{MetadataCollector.updateLocalDeletionTime(int
> newLocalDeletionTime)}} will be called with {{cell.localDeletionTime()}} in
> order to update the min/max deletion times as well as the
> {{estimatedTombstoneDropTime}} histogram. Currently this also happens for
> {{Cell.NO_DELETION_TIME}} ({{Integer.MAX_VALUE}}), which causes the histogram
> to be called for every regular cell.
> This can be easily verified by using {{sstablemetadata}} on any sstable. You
> should be able to find a very high value for 2147483647 in the "Estimated
> tombstone drop times" section.
> At this point, I don't think this is causing serious harm, but could be a
> performance issue (see linked ticket). Calculated droppable TS times should
> not be affected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13040) Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
[
https://issues.apache.org/jira/browse/CASSANDRA-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13040:
---
Attachment: 13040-3.0.patch
> Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
> ---
>
> Key: CASSANDRA-13040
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13040
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 13040-3.0.patch
>
>
> Starting with 3.0, {{MetadataCollector.updateLocalDeletionTime(int
> newLocalDeletionTime)}} will be called with {{cell.localDeletionTime()}} in
> order to update the min/max deletion times as well as the
> {{estimatedTombstoneDropTime}} histogram. Currently this also happens for
> {{Cell.NO_DELETION_TIME}} ({{Integer.MAX_VALUE}}), which causes the histogram
> to be called for every regular cell.
> This can be easily verified by using {{sstablemetadata}} on any sstable. You
> should be able to find a very high value for 2147483647 in the "Estimated
> tombstone drop times" section.
> At this point, I don't think this is causing serious harm, but could be a
> performance issue (see linked ticket). Calculated droppable TS times should
> not be affected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13040) Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
[
https://issues.apache.org/jira/browse/CASSANDRA-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13040:
---
Attachment: 13040-3.0.patch
> Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
> ---
>
> Key: CASSANDRA-13040
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13040
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 13040-3.0.patch
>
>
> Starting with 3.0, {{MetadataCollector.updateLocalDeletionTime(int
> newLocalDeletionTime)}} will be called with {{cell.localDeletionTime()}} in
> order to update the min/max deletion times as well as the
> {{estimatedTombstoneDropTime}} histogram. Currently this also happens for
> {{Cell.NO_DELETION_TIME}} ({{Integer.MAX_VALUE}}), which causes the histogram
> to be called for every regular cell.
> This can be easily verified by using {{sstablemetadata}} on any sstable. You
> should be able to find a very high value for 2147483647 in the "Estimated
> tombstone drop times" section.
> At this point, I don't think this is causing serious harm, but could be a
> performance issue (see linked ticket). Calculated droppable TS times should
> not be affected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13040) Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
[
https://issues.apache.org/jira/browse/CASSANDRA-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13040:
---
Status: Patch Available (was: Open)
> Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
> ---
>
> Key: CASSANDRA-13040
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13040
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>
> Starting with 3.0, {{MetadataCollector.updateLocalDeletionTime(int
> newLocalDeletionTime)}} will be called with {{cell.localDeletionTime()}} in
> order to update the min/max deletion times as well as the
> {{estimatedTombstoneDropTime}} histogram. Currently this also happens for
> {{Cell.NO_DELETION_TIME}} ({{Integer.MAX_VALUE}}), which causes the histogram
> to be called for every regular cell.
> This can be easily verified by using {{sstablemetadata}} on any sstable. You
> should be able to find a very high value for 2147483647 in the "Estimated
> tombstone drop times" section.
> At this point, I don't think this is causing serious harm, but could be a
> performance issue (see linked ticket). Calculated droppable TS times should
> not be affected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-13040) Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME
Stefan Podkowinski created CASSANDRA-13040:
--
Summary: Estimated TS drop-time histogram updated with
Cell.NO_DELETION_TIME
Key: CASSANDRA-13040
URL: https://issues.apache.org/jira/browse/CASSANDRA-13040
Project: Cassandra
Issue Type: Bug
Reporter: Stefan Podkowinski
Assignee: Stefan Podkowinski
Starting with 3.0, {{MetadataCollector.updateLocalDeletionTime(int
newLocalDeletionTime)}} will be called with {{cell.localDeletionTime()}} in
order to update the min/max deletion times as well as the
{{estimatedTombstoneDropTime}} histogram. Currently this also happens for
{{Cell.NO_DELETION_TIME}} ({{Integer.MAX_VALUE}}), which causes the histogram
to be called for every regular cell.
This can be easily verified by using {{sstablemetadata}} on any sstable. You
should be able to find a very high value for 2147483647 in the "Estimated
tombstone drop times" section.
At this point, I don't think this is causing serious harm, but could be a
performance issue (see linked ticket). Calculated droppable TS times should not
be affected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9625) GraphiteReporter not reporting
[ https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-9625: -- Status: Patch Available (was: Reopened) > GraphiteReporter not reporting > -- > > Key: CASSANDRA-9625 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9625 > Project: Cassandra > Issue Type: Bug > Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3 >Reporter: Eric Evans >Assignee: T Jake Luciani > Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml, > thread-dump.log > > > When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops > working. The usual startup is logged, and one batch of samples is sent, but > the reporting interval comes and goes, and no other samples are ever sent. > The logs are free from errors. > Frustratingly, metrics reporting works in our smaller (staging) environment > on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not > on a 3 node (otherwise identical) staging cluster (maybe it takes a certain > level of concurrency?). > Attached is a thread dump, and our metrics.yaml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9625) GraphiteReporter not reporting
[
https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744732#comment-15744732
]
Stefan Podkowinski commented on CASSANDRA-9625:
---
I think [~ruoranwang] is right by addressing the {{getEstimatedRemainingTasks}}
call, as it will delegate to the {{LeveledManifest}} version, which is
synchronized and causes the reporter thread to block. At some point the
reporter must get stuck after waiting too long. I'm not certain about the exact
reasons for this, but having the reporter thread competing for compaction locks
doesn't seem like a good idea in general to me, so I'd suggest to use a cached
value of the remaining tasks count instead. This should also improve
performance a bit by avoiding continuous level size calculation on unchanged
sets of sstables.
||2.1||2.2||3.0||3.x||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-9625-2.1]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-9625-2.2]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-9625-3.0]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-9625-3.x]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-9625-2.1-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-9625-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-9625-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-9625-3.x-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-9625-2.1-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-9625-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-9625-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-9625-3.x-testall/]|
Anyone wants to give this a try by running a patched node? Test results look ok
except for the failing 2.1 {{LeveledCompactionStrategyTest.testMutateLevel}},
which always times out but works fine locally - any idea what can be done about
that?
> GraphiteReporter not reporting
> --
>
> Key: CASSANDRA-9625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>Reporter: Eric Evans
>Assignee: T Jake Luciani
> Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml,
> thread-dump.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops
> working. The usual startup is logged, and one batch of samples is sent, but
> the reporting interval comes and goes, and no other samples are ever sent.
> The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment
> on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not
> on a 3 node (otherwise identical) staging cluster (maybe it takes a certain
> level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13036) Row and Cell Level Authorisation
[ https://issues.apache.org/jira/browse/CASSANDRA-13036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744671#comment-15744671 ] Stefan Podkowinski commented on CASSANDRA-13036: Hi Alex Would you be able to provide feedback to CASSANDRA-12859 and see if this fits your requirements? > Row and Cell Level Authorisation > > > Key: CASSANDRA-13036 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13036 > Project: Cassandra > Issue Type: Wish > Components: Core >Reporter: Alexander McRae > > Hi, > I am wondering if there are plans to implement Row Level Authorisation. > The industry I work in his highly regulated and the security policies within > our company/industry require us to have some strict data access control > requirements which means that we need to do permission checking on a user at > a row level or potentially down to a cell (attribute) level. > I have had a quick look through the documentation, code and jira tickets and > it doesn't appear that level of authorisation is catered for or is currently > being considered. > As such I would like request to see if this feature could possibly listed as > future enhancement. > Regards, > Alex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13024) Droppable Tombstone Ratio Calculation
[
https://issues.apache.org/jira/browse/CASSANDRA-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13024:
---
Component/s: Compaction
> Droppable Tombstone Ratio Calculation
> -
>
> Key: CASSANDRA-13024
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13024
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Stefan Podkowinski
>
> Whenever no sstables can be compacted in a standard way, we currently try to
> compact sstables that would make worthwhile candidates by evaluation
> {{sstable.getEstimatedDroppableTombstoneRatio(gcBefore) <=
> tombstoneThreshold}}. You can find out more on how this is supposed to work
> and the various settings options by reading "[About Deletes and Tombstones in
> Cassandra|http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html#single-sstable-compaction];
> by [~arodrime].
> The bad news is that currently the ratio value will vary to a great degree
> depending on the data model and type of deletes, as the ratio will be created
> based on the number of tombstones and number of _columns_. Any kind of
> tombstone will be counted here, no matter if on column or partition level,
> which will give you very different ratios based on the number of columns in
> your table and how you delete the data.
> Considering a 0.20 default threshold for finding sstables with enough
> droppable tombstones, it only takes 3 columns in your table to never hit the
> threshold at all when using partition tombstones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-13024) Droppable Tombstone Ratio Calculation
Stefan Podkowinski created CASSANDRA-13024:
--
Summary: Droppable Tombstone Ratio Calculation
Key: CASSANDRA-13024
URL: https://issues.apache.org/jira/browse/CASSANDRA-13024
Project: Cassandra
Issue Type: Bug
Reporter: Stefan Podkowinski
Whenever no sstables can be compacted in a standard way, we currently try to
compact sstables that would make worthwhile candidates by evaluation
{{sstable.getEstimatedDroppableTombstoneRatio(gcBefore) <=
tombstoneThreshold}}. You can find out more on how this is supposed to work and
the various settings options by reading "[About Deletes and Tombstones in
Cassandra|http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html#single-sstable-compaction];
by [~arodrime].
The bad news is that currently the ratio value will vary to a great degree
depending on the data model and type of deletes, as the ratio will be created
based on the number of tombstones and number of _columns_. Any kind of
tombstone will be counted here, no matter if on column or partition level,
which will give you very different ratios based on the number of columns in
your table and how you delete the data.
Considering a 0.20 default threshold for finding sstables with enough droppable
tombstones, it only takes 3 columns in your table to never hit the threshold at
all when using partition tombstones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13023) Add Tombstone Metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13023:
---
Status: Patch Available (was: Open)
> Add Tombstone Metrics
> -
>
> Key: CASSANDRA-13023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13023
> Project: Cassandra
> Issue Type: Improvement
> Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 13023-3.0.patch
>
>
> Currently it's only possible to inspect the ratio of droppable tombstones on
> sstable basis using the {{sstablemetadata}} tool. It would be very useful to
> have this information provided as part of Cassandra metrics as well, e.g. to
> get an idea on the effectiveness of tombstone eviction during compactions
> over time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13023) Add Tombstone Metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13023:
---
Attachment: 13023-3.0.patch
> Add Tombstone Metrics
> -
>
> Key: CASSANDRA-13023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13023
> Project: Cassandra
> Issue Type: Improvement
> Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 13023-3.0.patch
>
>
> Currently it's only possible to inspect the ratio of droppable tombstones on
> sstable basis using the {{sstablemetadata}} tool. It would be very useful to
> have this information provided as part of Cassandra metrics as well, e.g. to
> get an idea on the effectiveness of tombstone eviction during compactions
> over time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-13023) Add Tombstone Metrics
Stefan Podkowinski created CASSANDRA-13023:
--
Summary: Add Tombstone Metrics
Key: CASSANDRA-13023
URL: https://issues.apache.org/jira/browse/CASSANDRA-13023
Project: Cassandra
Issue Type: Improvement
Components: Observability
Reporter: Stefan Podkowinski
Assignee: Stefan Podkowinski
Currently it's only possible to inspect the ratio of droppable tombstones on
sstable basis using the {{sstablemetadata}} tool. It would be very useful to
have this information provided as part of Cassandra metrics as well, e.g. to
get an idea on the effectiveness of tombstone eviction during compactions over
time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730053#comment-15730053 ] Stefan Podkowinski commented on CASSANDRA-12991: Please keep in mind that there can be concurrent validation compactions at the same time (for different token ranges and tables). See CASSANDRA-9491 for an overview on how repair units are created and how bad this behaviour will effect sequential repairs. Coordinating flushes based on a timestamp would also require to only flush certain token ranges and in return would leave you with lots of tiny sstables in the beginning. I'd also expect ValidationRequest messages to be handled within at most a few milliseconds in normal cases. A single replica node set would have to ingest a substantial amount of writes to make the scenario described in the ticket happen within this small timeframe. Even so, how bad would it be to repair a couple of keys that are affected by this? We really should get some data here and do the math before making the repair process even more complex, while applying minor optimizations that will not make a difference for 90% of clusters anyways. Maybe as a start we should simply add a timestamp for the ValidationRequest as suggested, but only log the time difference upon validation compaction on the remote node. We could also log a warning if the interval gets to larger than a certain threshold value. > Inter-node race condition in validation compaction > -- > > Key: CASSANDRA-12991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12991 > Project: Cassandra > Issue Type: Improvement >Reporter: Benjamin Roth >Priority: Minor > > Problem: > When a validation compaction is triggered by a repair it may happen that due > to flying in mutations the merkle trees differ but the data is consistent > however. > Example: > t = 1: > Repair starts, triggers validations > Node A starts validation > t = 10001: > Mutation arrives at Node A > t = 10002: > Mutation arrives at Node B > t = 10003: > Node B starts validation > Hashes of node A+B will differ but data is consistent from a view (think of > it like a snapshot) t = 1. > Impact: > Unnecessary streaming happens. This may not a big impact on low traffic CFs, > partitions but on high traffic CFs and maybe very big partitions, this may > have a bigger impact and is a waste of resources. > Possible solution: > Build hashes based upon a snapshot timestamp. > This requires SSTables created after that timestamp to be filtered when doing > a validation compaction: > - Cells with timestamp > snapshot time have to be removed > - Tombstone range markers have to be handled > - Bounds have to be removed if delete timestamp > snapshot time > - Boundary markers have to be either changed to a bound or completely > removed, depending if start and/or end are both affected or not > Probably this is a known behaviour. Have there been any discussions about > this in the past? Did not find an matching issue, so I created this one. > I am happy about any feedback, whatsoever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722046#comment-15722046 ] Stefan Podkowinski commented on CASSANDRA-12991: Let's assume it would be possible to do validation of sstables based on a provided timestamp, there's still the issue that data could be reconciled in memtables in a way that would still result in a mismatch. t = 1 NodeA - Mutation(k=1, ts=1) NodeA - Flush NodeB - Mutation(k=1, ts=1) t = 10001 NodeA - Mutation(k=1, ts=2) NodeB - Mutation(k=1, ts=2) NodeB - Flush If you start validation based on t(1) you'd still end up with a mismatch, as only Mutation(k=1, ts=2) would have been flushed to disk on NodeB, while ts=1 (which would be subject to the validation timestamp) would not. > Inter-node race condition in validation compaction > -- > > Key: CASSANDRA-12991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12991 > Project: Cassandra > Issue Type: Improvement >Reporter: Benjamin Roth >Priority: Minor > > Problem: > When a validation compaction is triggered by a repair it may happen that due > to flying in mutations the merkle trees differ but the data is consistent > however. > Example: > t = 1: > Repair starts, triggers validations > Node A starts validation > t = 10001: > Mutation arrives at Node A > t = 10002: > Mutation arrives at Node B > t = 10003: > Node B starts validation > Hashes of node A+B will differ but data is consistent from a view (think of > it like a snapshot) t = 1. > Impact: > Unnecessary streaming happens. This may not a big impact on low traffic CFs, > partitions but on high traffic CFs and maybe very big partitions, this may > have a bigger impact and is a waste of resources. > Possible solution: > Build hashes based upon a snapshot timestamp. > This requires SSTables created after that timestamp to be filtered when doing > a validation compaction: > - Cells with timestamp > snapshot time have to be removed > - Tombstone range markers have to be handled > - Bounds have to be removed if delete timestamp > snapshot time > - Boundary markers have to be either changed to a bound or completely > removed, depending if start and/or end are both affected or not > Probably this is a known behaviour. Have there been any discussions about > this in the past? Did not find an matching issue, so I created this one. > I am happy about any feedback, whatsoever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721760#comment-15721760 ] Stefan Podkowinski commented on CASSANDRA-12991: My assumption is that validation compaction works as follows: * involved nodes receive a ValidationRequest message * affected keyspace is being flushed * validation is started using sstables candidates determined right after the flush I don't see why you'd have to "SSTables created after that timestamp to be filtered when doing a validation compaction". Any SSTable created after the validation compaction was started should not be involved in the validation process anyways. > Inter-node race condition in validation compaction > -- > > Key: CASSANDRA-12991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12991 > Project: Cassandra > Issue Type: Improvement >Reporter: Benjamin Roth >Priority: Minor > > Problem: > When a validation compaction is triggered by a repair it may happen that due > to flying in mutations the merkle trees differ but the data is consistent > however. > Example: > t = 1: > Repair starts, triggers validations > Node A starts validation > t = 10001: > Mutation arrives at Node A > t = 10002: > Mutation arrives at Node B > t = 10003: > Node B starts validation > Hashes of node A+B will differ but data is consistent from a view (think of > it like a snapshot) t = 1. > Impact: > Unnecessary streaming happens. This may not a big impact on low traffic CFs, > partitions but on high traffic CFs and maybe very big partitions, this may > have a bigger impact and is a waste of resources. > Possible solution: > Build hashes based upon a snapshot timestamp. > This requires SSTables created after that timestamp to be filtered when doing > a validation compaction: > - Cells with timestamp > snapshot time have to be removed > - Tombstone range markers have to be handled > - Bounds have to be removed if delete timestamp > snapshot time > - Boundary markers have to be either changed to a bound or completely > removed, depending if start and/or end are both affected or not > Probably this is a known behaviour. Have there been any discussions about > this in the past? Did not find an matching issue, so I created this one. > I am happy about any feedback, whatsoever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-12991: --- Comment: was deleted (was: My assumption is that validation compaction works as follows: * involved nodes receive a ValidationRequest message * affected keyspace is being flushed * validation is started using sstables candidates determined right after the flush I don't see why you'd have to "SSTables created after that timestamp to be filtered when doing a validation compaction". Any SSTable created after the validation compaction was started should not be involved in the validation process anyways. ) > Inter-node race condition in validation compaction > -- > > Key: CASSANDRA-12991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12991 > Project: Cassandra > Issue Type: Improvement >Reporter: Benjamin Roth >Priority: Minor > > Problem: > When a validation compaction is triggered by a repair it may happen that due > to flying in mutations the merkle trees differ but the data is consistent > however. > Example: > t = 1: > Repair starts, triggers validations > Node A starts validation > t = 10001: > Mutation arrives at Node A > t = 10002: > Mutation arrives at Node B > t = 10003: > Node B starts validation > Hashes of node A+B will differ but data is consistent from a view (think of > it like a snapshot) t = 1. > Impact: > Unnecessary streaming happens. This may not a big impact on low traffic CFs, > partitions but on high traffic CFs and maybe very big partitions, this may > have a bigger impact and is a waste of resources. > Possible solution: > Build hashes based upon a snapshot timestamp. > This requires SSTables created after that timestamp to be filtered when doing > a validation compaction: > - Cells with timestamp > snapshot time have to be removed > - Tombstone range markers have to be handled > - Bounds have to be removed if delete timestamp > snapshot time > - Boundary markers have to be either changed to a bound or completely > removed, depending if start and/or end are both affected or not > Probably this is a known behaviour. Have there been any discussions about > this in the past? Did not find an matching issue, so I created this one. > I am happy about any feedback, whatsoever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721759#comment-15721759 ] Stefan Podkowinski commented on CASSANDRA-12991: My assumption is that validation compaction works as follows: * involved nodes receive a ValidationRequest message * affected keyspace is being flushed * validation is started using sstables candidates determined right after the flush I don't see why you'd have to "SSTables created after that timestamp to be filtered when doing a validation compaction". Any SSTable created after the validation compaction was started should not be involved in the validation process anyways. > Inter-node race condition in validation compaction > -- > > Key: CASSANDRA-12991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12991 > Project: Cassandra > Issue Type: Improvement >Reporter: Benjamin Roth >Priority: Minor > > Problem: > When a validation compaction is triggered by a repair it may happen that due > to flying in mutations the merkle trees differ but the data is consistent > however. > Example: > t = 1: > Repair starts, triggers validations > Node A starts validation > t = 10001: > Mutation arrives at Node A > t = 10002: > Mutation arrives at Node B > t = 10003: > Node B starts validation > Hashes of node A+B will differ but data is consistent from a view (think of > it like a snapshot) t = 1. > Impact: > Unnecessary streaming happens. This may not a big impact on low traffic CFs, > partitions but on high traffic CFs and maybe very big partitions, this may > have a bigger impact and is a waste of resources. > Possible solution: > Build hashes based upon a snapshot timestamp. > This requires SSTables created after that timestamp to be filtered when doing > a validation compaction: > - Cells with timestamp > snapshot time have to be removed > - Tombstone range markers have to be handled > - Bounds have to be removed if delete timestamp > snapshot time > - Boundary markers have to be either changed to a bound or completely > removed, depending if start and/or end are both affected or not > Probably this is a known behaviour. Have there been any discussions about > this in the past? Did not find an matching issue, so I created this one. > I am happy about any feedback, whatsoever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12966) Gossip thread slows down when using batch commit log
[
https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721705#comment-15721705
]
Stefan Podkowinski commented on CASSANDRA-12966:
My personally preference for this kind of API design is to provide the executor
as a method parameter. This makes it obvious (while reading the code on the
caller side) to recognize the asynchronous nature of the function, as well as
in what executor context will be used to run the code. In case of the discussed
{{updateTokens}} method, I'd probably just add the {{Stage}} enum as another
parameter, so you'd call {{SystemKeyspace.updateTokens(endpoint,
tokensToUpdateInSystemKeyspace, Stage.MUTATION)}}.
> Gossip thread slows down when using batch commit log
>
>
> Key: CASSANDRA-12966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12966
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> When using batch commit log mode, the Gossip thread slows down when peers
> after a node bounces. This is because we perform a bunch of updates to the
> peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized
> method. How quickly each one of those individual updates takes depends on how
> busy the system is at the time wrt write traffic. If the system is largely
> quiescent, each update will be relatively quick (just waiting for the fsync).
> If the system is getting a lot of writes, and depending on the
> commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can
> get stuck in the backlog, which causes the Gossip thread to stop processing.
> We have observed in large clusters that a rolling restart causes triggers and
> exacerbates this behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12966) Gossip thread slows down when using batch commit log
[
https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709058#comment-15709058
]
Stefan Podkowinski commented on CASSANDRA-12966:
Seems like the gossip single thread execution is a bit problematic, as this
also caused some pain in CASSANDRA-12281. Looks like CASSANDRA-8398 will be a
good thing to have here.
Some comments regarding your patch:
My thoughts on concurrency aspects:
StorageService.handleStateNormal will update tokens for both TokenMetadata and
SystemKeyspace. The
previous blocking behavior would ensure both would be in-sync. Offloading the
system table update to the mutation stage would allow to have the table lag
behind, but I would not expect any races between mutations, as the execution
order hasn't changed, just the executor.
Uncoupling the mutations this way without waiting for the write result
shouldn't be a problem, as the system table is only used during initialization
and there's no guarantees that the gossip state for a node is always recent
anyways.
The synchronized keywords for removeEndpoints looks like a leftover from when
the code would read and write back the modified token set and it should be safe
to remove it.
As for API modifications, there are now two updateToken versions, one blocking
and one asynchronous. Maybe async methods should be named differently, as the
Future return value will not be checked in the code and you wouldn't be able to
tell which version is called by reading code on the caller side.
> Gossip thread slows down when using batch commit log
>
>
> Key: CASSANDRA-12966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12966
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> When using batch commit log mode, the Gossip thread slows down when peers
> after a node bounces. This is because we perform a bunch of updates to the
> peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized
> method. How quickly each one of those individual updates takes depends on how
> busy the system is at the time wrt write traffic. If the system is largely
> quiescent, each update will be relatively quick (just waiting for the fsync).
> If the system is getting a lot of writes, and depending on the
> commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can
> get stuck in the backlog, which causes the Gossip thread to stop processing.
> We have observed in large clusters that a rolling restart causes triggers and
> exacerbates this behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9633) Add ability to encrypt sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696103#comment-15696103
]
Stefan Podkowinski commented on CASSANDRA-9633:
---
I haven't looked specifically into how an external key management system such
as vault could be integrated by implementing {{KeyProvider}}. But your're
right, it could be an option.
What I'm wondering though is if it wouldn't be worth it to implement a two tier
encryption architecture right from the start. To me support for key rotation
and table based encryption would be a key feature that makes Cassandra at-rest
encryption really worthwhile compared to filesystem/block device based
encryption. Therefor my suggestion would be to encrypt sstables using a
intermediate, sstable specific key (derived from a master key) that would be
stored along with the sstable (in an extra component). This would only require
to re-encrypt the associated keys upon key rotation, instead of rewriting all
sstables.
> Add ability to encrypt sstables
> ---
>
> Key: CASSANDRA-9633
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9633
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jason Brown
>Assignee: Jason Brown
> Labels: encryption, security, sstable
> Fix For: 3.x
>
>
> Add option to allow encrypting of sstables.
> I have a version of this functionality built on cassandra 2.0 that
> piggy-backs on the existing sstable compression functionality and ICompressor
> interface (similar in nature to what DataStax Enterprise does). However, if
> we're adding the feature to the main OSS product, I'm not sure if we want to
> use the pluggable compression framework or if it's worth investigating a
> different path. I think there's a lot of upside in reusing the sstable
> compression scheme, but perhaps add a new component in cqlsh for table
> encryption and a corresponding field in CFMD.
> Encryption configuration in the yaml can use the same mechanism as
> CASSANDRA-6018 (which is currently pending internal review).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12945) Resolve unit testing without JCE security libraries installed
[ https://issues.apache.org/jira/browse/CASSANDRA-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-12945: --- Attachment: 12945-3.X.patch > Resolve unit testing without JCE security libraries installed > - > > Key: CASSANDRA-12945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12945 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jason Brown >Assignee: Stefan Podkowinski > Attachments: 12945-3.X.patch > > > Running unit tests can fail on encryption-related tests if you don't have > something like the Oracle JCE libraries installed in your jdk. We can't > redistribute the Oracle JCE due to export laws, then we'd need to somehow get > it into the /jre/lib/security. > One possibility is to ignore encryption-related tests if there is no > encryption lib available. Another is to ship something like bouncycastle.jar > in the test directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12945) Resolve unit testing without JCE security libraries installed
[ https://issues.apache.org/jira/browse/CASSANDRA-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-12945: --- Assignee: Stefan Podkowinski (was: Jason Brown) Reviewer: Jason Brown Status: Patch Available (was: Open) > Resolve unit testing without JCE security libraries installed > - > > Key: CASSANDRA-12945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12945 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jason Brown >Assignee: Stefan Podkowinski > > Running unit tests can fail on encryption-related tests if you don't have > something like the Oracle JCE libraries installed in your jdk. We can't > redistribute the Oracle JCE due to export laws, then we'd need to somehow get > it into the /jre/lib/security. > One possibility is to ignore encryption-related tests if there is no > encryption lib available. Another is to ship something like bouncycastle.jar > in the test directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12945) Resolve unit testing without JCE security libraries installed
[ https://issues.apache.org/jira/browse/CASSANDRA-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695794#comment-15695794 ] Stefan Podkowinski commented on CASSANDRA-12945: Tests after recreating the keystore with 128bit keys seems to work fine with the stock JDK without JCE strong ciphers. Keys have been generated as described in my comment above. I just had to recreate the legacy commit log files, as they seem to have been encrypted with the replaced keys. ||3.X||trunk|| |[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-12945-3.X]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-12945-trunk]| |[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-12945-3.X-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-12945-trunk-dtest/]| |[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-12945-3.X-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-12945-trunk-testall/]*| *see CASSANDRA-12875 for failed test > Resolve unit testing without JCE security libraries installed > - > > Key: CASSANDRA-12945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12945 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jason Brown >Assignee: Jason Brown > > Running unit tests can fail on encryption-related tests if you don't have > something like the Oracle JCE libraries installed in your jdk. We can't > redistribute the Oracle JCE due to export laws, then we'd need to somehow get > it into the /jre/lib/security. > One possibility is to ignore encryption-related tests if there is no > encryption lib available. Another is to ship something like bouncycastle.jar > in the test directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12875) testall failure in org.apache.cassandra.net.MessagingServiceTest.testDCLatency-compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695723#comment-15695723
]
Stefan Podkowinski commented on CASSANDRA-12875:
+1
I had two testall runs affected by this issue. Applying the proposed patch made
the tests pass.
* [trunk w/ 2
failures|http://cassci.datastax.com/job/spodkowinski-CASSANDRA-12945-trunk-testall/lastCompletedBuild/testReport/org.apache.cassandra.net/MessagingServiceTest/testDCLatency_compression/]
* [12875.patch
results|http://cassci.datastax.com/job/spodkowinski-CASSANDRA-12945-trunk-12875-testall/]
> testall failure in
> org.apache.cassandra.net.MessagingServiceTest.testDCLatency-compression
> --
>
> Key: CASSANDRA-12875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12875
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Chris Lohfink
> Labels: test-failure, testall
> Attachments: 12875.patch
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.X_testall/54/testReport/org.apache.cassandra.net/MessagingServiceTest/testDCLatency_compression/
> {code}
> Error Message
> expected:<107964792> but was:<129557750>
> {code}{code}
> Stacktrace
> junit.framework.AssertionFailedError: expected:<107964792> but was:<129557750>
> at
> org.apache.cassandra.net.MessagingServiceTest.testDCLatency(MessagingServiceTest.java:115)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12945) Resolve unit testing without JCE security libraries installed
[
https://issues.apache.org/jira/browse/CASSANDRA-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690319#comment-15690319
]
Stefan Podkowinski commented on CASSANDRA-12945:
What I did was
* Replace export restricted JCE files with standard, weak policy files
* Clone your 9633 branch and run e.g, {{EncryptingCompressorTest}} just to get
an {{java.security.InvalidKeyException: Illegal key size}} error
* Create a new keystore with 128 bit key sizes: {{keytool -genseckey -alias
'testing:1' -keyalg AES -keysize 128 -storetype jceks -keystore mykeystore.jks}}
* Replace {{cassandra.keystore}} with new keystore and verify that tests pass
>From my understanding, the export restricted JCE policies will only unlock
>stronger cipher versions. I'd be curious to know how this would break unit
>tests.
> Resolve unit testing without JCE security libraries installed
> -
>
> Key: CASSANDRA-12945
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12945
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> Running unit tests can fail on encryption-related tests if you don't have
> something like the Oracle JCE libraries installed in your jdk. We can't
> redistribute the Oracle JCE due to export laws, then we'd need to somehow get
> it into the /jre/lib/security.
> One possibility is to ignore encryption-related tests if there is no
> encryption lib available. Another is to ship something like bouncycastle.jar
> in the test directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9633) Add ability to encrypt sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690122#comment-15690122 ] Stefan Podkowinski commented on CASSANDRA-9633: --- I wonder if it would make sense to use a ParameterizedClass for the cassandra.yaml configuration. The keystore based implementation is a great start, but there could be other ways to do this and maybe we'll have someone contribute say a [Vault|https://www.vaultproject.io] based implementation one day. Being able to configure a different class_name and parameters would make it easier to integrate other solutions. > Add ability to encrypt sstables > --- > > Key: CASSANDRA-9633 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9633 > Project: Cassandra > Issue Type: New Feature >Reporter: Jason Brown >Assignee: Jason Brown > Labels: encryption, security, sstable > Fix For: 3.x > > > Add option to allow encrypting of sstables. > I have a version of this functionality built on cassandra 2.0 that > piggy-backs on the existing sstable compression functionality and ICompressor > interface (similar in nature to what DataStax Enterprise does). However, if > we're adding the feature to the main OSS product, I'm not sure if we want to > use the pluggable compression framework or if it's worth investigating a > different path. I think there's a lot of upside in reusing the sstable > compression scheme, but perhaps add a new component in cqlsh for table > encryption and a corresponding field in CFMD. > Encryption configuration in the yaml can use the same mechanism as > CASSANDRA-6018 (which is currently pending internal review). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12945) Resolve unit testing without JCE security libraries installed
[
https://issues.apache.org/jira/browse/CASSANDRA-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689528#comment-15689528
]
Stefan Podkowinski commented on CASSANDRA-12945:
Are there any reasons not to use shorter keys in
{{test/conf/cassandra.keystore}} for the sake of compatibility? I've tried some
of the unit tests that would fail without mentioned JCE using 128 bit keys
instead and they seem to work fine.
> Resolve unit testing without JCE security libraries installed
> -
>
> Key: CASSANDRA-12945
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12945
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> Running unit tests can fail on encryption-related tests if you don't have
> something like the Oracle JCE libraries installed in your jdk. We can't
> redistribute the Oracle JCE due to export laws, then we'd need to somehow get
> it into the /jre/lib/security.
> One possibility is to ignore encryption-related tests if there is no
> encryption lib available. Another is to ship something like bouncycastle.jar
> in the test directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-12944) Diagnostic Events
Stefan Podkowinski created CASSANDRA-12944: -- Summary: Diagnostic Events Key: CASSANDRA-12944 URL: https://issues.apache.org/jira/browse/CASSANDRA-12944 Project: Cassandra Issue Type: Improvement Components: Observability Reporter: Stefan Podkowinski I'd like to propose a new "diagnostic events" feature that would allow to observe internal Cassandra events in unit tests and from external tools via native transport. The motivation is to improve testing as well as operational monitoring and troubleshooting beyond logs and metrics. Please find more details in the linked proposal and give it some thoughts :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12906) Update doco with new getting started with contribution section
[ https://issues.apache.org/jira/browse/CASSANDRA-12906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673568#comment-15673568 ] Stefan Podkowinski commented on CASSANDRA-12906: Looks like to attached the wrong patch. > Update doco with new getting started with contribution section > -- > > Key: CASSANDRA-12906 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12906 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Ben Slater >Assignee: Ben Slater >Priority: Minor > Attachments: 12906-trunk.patch > > > Following discussion on the mailing list about how to get more community > input it seemed to be agreed that adding some doco emphasising contributions > other than creating new features would be a good idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12281) Gossip blocks on startup when another node is bootstrapping
[
https://issues.apache.org/jira/browse/CASSANDRA-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-12281:
---
Attachment: 12281-3.0.patch
12281-3.X.patch
12281-trunk.patch
12281-2.2.patch
> Gossip blocks on startup when another node is bootstrapping
> ---
>
> Key: CASSANDRA-12281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12281
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Eric Evans
>Assignee: Stefan Podkowinski
> Attachments: 12281-2.2.patch, 12281-3.0.patch, 12281-3.X.patch,
> 12281-trunk.patch, restbase1015-a_jstack.txt
>
>
> In our cluster, normal node startup times (after a drain on shutdown) are
> less than 1 minute. However, when another node in the cluster is
> bootstrapping, the same node startup takes nearly 30 minutes to complete, the
> apparent result of gossip blocking on pending range calculations.
> {noformat}
> $ nodetool-a tpstats
> Pool NameActive Pending Completed Blocked All
> time blocked
> MutationStage 0 0 1840 0
> 0
> ReadStage 0 0 2350 0
> 0
> RequestResponseStage 0 0 53 0
> 0
> ReadRepairStage 0 0 1 0
> 0
> CounterMutationStage 0 0 0 0
> 0
> HintedHandoff 0 0 44 0
> 0
> MiscStage 0 0 0 0
> 0
> CompactionExecutor3 3395 0
> 0
> MemtableReclaimMemory 0 0 30 0
> 0
> PendingRangeCalculator1 2 29 0
> 0
> GossipStage 1 5602164 0
> 0
> MigrationStage0 0 0 0
> 0
> MemtablePostFlush 0 0111 0
> 0
> ValidationExecutor0 0 0 0
> 0
> Sampler 0 0 0 0
> 0
> MemtableFlushWriter 0 0 30 0
> 0
> InternalResponseStage 0 0 0 0
> 0
> AntiEntropyStage 0 0 0 0
> 0
> CacheCleanupExecutor 0 0 0 0
> 0
> Message type Dropped
> READ 0
> RANGE_SLICE 0
> _TRACE 0
> MUTATION 0
> COUNTER_MUTATION 0
> REQUEST_RESPONSE 0
> PAGED_RANGE 0
> READ_REPAIR 0
> {noformat}
> A full thread dump is attached, but the relevant bit seems to be here:
> {noformat}
> [ ... ]
> "GossipStage:1" #1801 daemon prio=5 os_prio=0 tid=0x7fe4cd54b000
> nid=0xea9 waiting on condition [0x7fddcf883000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0004c1e922c0> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
> at
> org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:174)
> at
> org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:160)
> at
> org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2023)
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1682)
> at
> org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1182)
> at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1165)
> at
>
[jira] [Updated] (CASSANDRA-12281) Gossip blocks on startup when another node is bootstrapping
[
https://issues.apache.org/jira/browse/CASSANDRA-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-12281:
---
Attachment: (was: 12281-2.2.patch)
> Gossip blocks on startup when another node is bootstrapping
> ---
>
> Key: CASSANDRA-12281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12281
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Eric Evans
>Assignee: Stefan Podkowinski
> Attachments: restbase1015-a_jstack.txt
>
>
> In our cluster, normal node startup times (after a drain on shutdown) are
> less than 1 minute. However, when another node in the cluster is
> bootstrapping, the same node startup takes nearly 30 minutes to complete, the
> apparent result of gossip blocking on pending range calculations.
> {noformat}
> $ nodetool-a tpstats
> Pool NameActive Pending Completed Blocked All
> time blocked
> MutationStage 0 0 1840 0
> 0
> ReadStage 0 0 2350 0
> 0
> RequestResponseStage 0 0 53 0
> 0
> ReadRepairStage 0 0 1 0
> 0
> CounterMutationStage 0 0 0 0
> 0
> HintedHandoff 0 0 44 0
> 0
> MiscStage 0 0 0 0
> 0
> CompactionExecutor3 3395 0
> 0
> MemtableReclaimMemory 0 0 30 0
> 0
> PendingRangeCalculator1 2 29 0
> 0
> GossipStage 1 5602164 0
> 0
> MigrationStage0 0 0 0
> 0
> MemtablePostFlush 0 0111 0
> 0
> ValidationExecutor0 0 0 0
> 0
> Sampler 0 0 0 0
> 0
> MemtableFlushWriter 0 0 30 0
> 0
> InternalResponseStage 0 0 0 0
> 0
> AntiEntropyStage 0 0 0 0
> 0
> CacheCleanupExecutor 0 0 0 0
> 0
> Message type Dropped
> READ 0
> RANGE_SLICE 0
> _TRACE 0
> MUTATION 0
> COUNTER_MUTATION 0
> REQUEST_RESPONSE 0
> PAGED_RANGE 0
> READ_REPAIR 0
> {noformat}
> A full thread dump is attached, but the relevant bit seems to be here:
> {noformat}
> [ ... ]
> "GossipStage:1" #1801 daemon prio=5 os_prio=0 tid=0x7fe4cd54b000
> nid=0xea9 waiting on condition [0x7fddcf883000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0004c1e922c0> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
> at
> org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:174)
> at
> org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:160)
> at
> org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2023)
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1682)
> at
> org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1182)
> at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1165)
> at
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1128)
> at
> org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:58)
>
