[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794949#comment-15794949
]
Tomasz Grabiec commented on CASSANDRA-5483:
---
This change seems to have changed the behavior of "nodetool flush" to not flush
the system keyspace (see CASSANDRA-11830).
{code}
commit f5866ca2bac8ce530bb4e20832ff61e622206815
Author: Ben Chan
Date: Wed Dec 3 13:15:10 2014 -0600
Add repair tracing
Patch by Ben Chan; reviewed by jmckenzie for CASSANDRA-5483
{code}
Relevant hunk:
{code}
--- a/src/java/org/apache/cassandra/tools/NodeTool.java
+++ b/src/java/org/apache/cassandra/tools/NodeTool.java
@@ -318,10 +318,15 @@ public class NodeTool
protected List parseOptionalKeyspace(List cmdArgs,
NodeProbe nodeProbe)
{
+return parseOptionalKeyspace(cmdArgs, nodeProbe, false);
+}
+
+protected List parseOptionalKeyspace(List cmdArgs,
NodeProbe nodeProbe, boolean includeSystemKS)
+{
List keyspaces = new ArrayList<>();
if (cmdArgs == null || cmdArgs.isEmpty())
-keyspaces.addAll(nodeProbe.getKeyspaces());
+keyspaces.addAll(includeSystemKS ? nodeProbe.getKeyspaces() :
nodeProbe.getNonSystemKeyspaces());
else
keyspaces.add(cmdArgs.get(0));
{code}
I wonder what is the rationale for this.
> Repair tracing
> --
>
> Key: CASSANDRA-5483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Yuki Morishita
>Assignee: Ben Chan
>Priority: Minor
> Labels: repair
> Fix For: 2.2.0 beta 1
>
> Attachments: 5483-full-trunk.txt,
> 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
> 5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
> 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
> 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
> 5483-v07-08-Fix-brace-style.patch,
> 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
> 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
> 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
> 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
> 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
> 5483-v08-14-Poll-system_traces.events.patch,
> 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
> 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
> 5483-v10-17-minor-bugfixes-and-changes.patch,
> 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
> 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
> 5483-v13-608fb03-May-14-trace-formatting-changes.patch,
> 5483-v14-01-squashed.patch,
> 5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
> 5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
> 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
> 5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
> 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch,
> ccm-repair-test, cqlsh-left-justify-text-columns.patch,
> prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
> trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
> trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
> tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
> tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
> v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
> v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
> v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
>
>
> I think it would be nice to log repair stats and results like query tracing
> stores traces to system keyspace. With it, you don't have to lookup each log
> file to see what was the status and how it performed the repair you invoked.
> Instead, you can query the repair log with session ID to see the state and
> stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14237144#comment-14237144 ] Ben Chan commented on CASSANDRA-5483: - Thanks for helping get this through the final (?) set of speed bumps. As fast as this codebase moves, I'm surprised that last rebase went so smoothly. (NRN) Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231840#comment-14231840 ] Ben Chan commented on CASSANDRA-5483: - Sorry; limited computer access last week, zero access to dev box (network setup issues), turkey. Branch 5483_squashed (currently at commit 78686c61e38e, merges cleanly with trunk at 06f626acd27b) looks fine to me; looks like there were only mechanical changes needed to resolve the merge conflicts. Ran my standard simplistic test, and no obvious problems (for both 78686c61e38e and again when merged with 06f626acd27b). Any problem is likely to be something I did, or some unforseen interaction with code in the updated trunk. Hopefully neither of those possibilities is true. I'll keep watching this thread/issue just in case. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225291#comment-14225291 ] Joshua McKenzie commented on CASSANDRA-5483: Sorry for the delay - was at a conference last week. I went ahead and rebased to current trunk as of today as there were some more changes around repair parallelism; your most recent changes look good and I'm not too worried about us having some public final strings in TraceKeyspace. If I could get a once-over reverse-review from you to make sure I didn't miss anything in that final rebase, I'll get this committed. Rebased and squashed branch available [*Here*|https://github.com/josh-mckenzie/cassandra/compare/5483_squashed] Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212771#comment-14212771
]
Ben Chan commented on CASSANDRA-5483:
-
Updated https://github.com/usrbincc/cassandra/tree/5483-review (currently at
commit 85d974753dc7f).
So a few changes to tracing got merged in on Nov 12, making a cut-and-dried
manual rebase impossible.
- Use a default ttl on the system_traces.* CFs. Use that to clean up the code
in Tracing, since you no longer have to specify ttls.
- Move a lot of row-insertion code out of Tracing into a new TraceKeyspace
class.
- Since there is no more need to specify ttl, use CFRowAdder (which doesn't do
ttl) for convenience.
Since the repair tracing patch requires a user-configurable ttl (or at the very
least, a different ttl for repair tracing and query tracing), I needed to
re-add ttl specification. Since it wasn't much more code, I decided to only use
explicitly-expiring cells if the ttl didn't match the default ttl.
This appears to save two native ints per column from a quick skim of the source
code. Not sure if that's really enough to care about (especially since repair
tracing, which is likely to insert more rows, has a ttl different from the
default), but it was a simple enough change.
Merge conflicts are a real pain point with git. I attempted to make reviewing
the merge conflict resolution changes easier by inserting an intermediate
commit that includes the conflict markers from git, unmodified. Feel free to
hide all the messiness with a {{git merge --squash}}.
- Fixed a race condition with repair trace sessions sometimes not being
properly stopped (see commit 801b6fbf56771).
- Possible code no-no to note: I made two private fields (SESSIONS_TABLE and
EVENTS_TABLE) in TraceKeyspace public. I only use EVENTS_TABLE, but made them
both public for symmetry (good? bad? don't know). They're final fields, if
that's any consolation.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Fix For: 3.0
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208811#comment-14208811 ] Joshua McKenzie commented on CASSANDRA-5483: Linked branch looks good, renames are solid, tests out fine. 300MB of repairs at RF=3 generated around 300k of event trace data so I'm comfortable with the current level; we can revisit paring that down in the future as appropriate. Patch doesn't apply/rebase to current trunk cleanly - if I could get you to do a quick rebase we'll be ready to wrap this up. Thanks for all the hard work on this! Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205451#comment-14205451 ] Ben Chan commented on CASSANDRA-5483: - Sorry; SMART and reallocated sectors. Which only really accounts for about 2-3 days of the delay. But just to say that it wasn't merely a case of hangover from too much Halloween candy. Updated https://github.com/usrbincc/cassandra/tree/5483-review (currently at commit 202a2e2e5e602). Merges cleanly with trunk (at commit d286ac7d072fe), building and testing cleanly. I decided to be a little opinionated and did some refactoring along the lines of my Oct 23 message. - Used a TraceState#waitActivity function instead of TraceState#isDone (waitActivity gets closer to doing only what it says on the tin; makes it less hairy to comment). - Moved all exponential backoff timeout code to StorageService#createQueryThread In addition, I renamed TraceState#enableNotifications to TraceState#enableActivityNotification to attempt (naming is hard) to avoid confusion with TraceState#setNotificationHandle, which is entirely unrelated. Note: beyond having made this opinionated edit, I'm not planning to be particularly opinionated about advocating for it. All of that code should eventually go away once there is some way to get notified about table updates instead of having to do all that messy polling. Extra note: Cassandra triggers seem to be very close to what is needed, if only they could be specified to run on a given node (i.e. the node that is being repaired). The last time I checked on this, this wasn't possible. Unfiltered traces: - The extra traces are generic message send-receive traces that existed prior to this patch. They were originally there for query tracing, which benefits from more detailed tracing. - These extra traces were filtered out for repair up until v16 of this patch. This means that any discussions of trace messages prior to that point are referring to the filtered traces. But I can't say that they're doing any real harm. I mean, it's only 3x the traces (estimated), and not an order of magnitude or more. It's probably fine as it is. I certainly can't unequivocally state that there's no use for those extra traces. Besides, extra information can always be filtered out at a higher level (assuming it's tagged appropriately). Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188766#comment-14188766 ] Joshua McKenzie commented on CASSANDRA-5483: Think that was just me not refreshing the tab I had open in Chrome w/this ticket in it - missed the update concerning local build problems. Re: isDone: miscommunication in ticket. Changing from bit-magic to named variables w/flow changed isDone into a self-commenting method and addressed my concerns with it. I appreciate the javadoc @ the top to give just general overview as well. The method is simple enough that switching to BlockingQueue isn't really necessary at this time imo. Re: unfiltered traces: are these extraneous messages or just more verbosity surrounding the activity of the repair? It does generate a lot of output but this is for tracing and I'd be comfortable with us filtering down as a future effort if we decide it's necessary. Last nit: isDone should be renamed since it does more than just check if things are done. We could either split out the isDone check from waitAndDouble() or just rename to waitWithDoubleTimeout() or something like that? Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187618#comment-14187618
]
Ben Chan commented on CASSANDRA-5483:
-
Communication seems to be off by a phase. My Oct 23 message may not have made
it across. Either that or I wasn't writing clearly enough (But in my defense,
natural language has a fiddly REPL and almost-useless error messages. And
there's no source code.).
Summary:
- Forget about the removal of the addCallback + sameThreadExecutor code before.
It turns out I didn't need to do that. The problem (I think) was some
corruption in my build.
- Commenting isDone results in a convoluted comment. I was considering whether
or not to do a refactor in order to make the comment require less complication.
For now I've gone ahead and commented isDone as best I can and fixed the naming
that resulted in {{wait(wait);}}, along with various other fixes.
See https://github.com/usrbincc/cassandra/tree/5483-review (currently at commit
15d8ec0a9fbbf).
Closing comments:
- For future reference: If you don't care about handling multiple consumers (as
the current isDone does not), a large chunk of isDone (and its supporting code)
could easily be replaced with a BlockingQueue and its {{offer}} and {{poll}}
methods.
- Can I get a confirmation that the unfiltered repair traces (see my Oct 22
message) in the current patch is acceptable?
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Fix For: 3.0
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14185252#comment-14185252
]
Joshua McKenzie commented on CASSANDRA-5483:
DebuggableThreadPoolExecutor:
bq. Just to confirm (though I'm mostly sure this is what you mean), Do you mean
that the function name should be changed to something like
createCachedThreadpoolWithMaximumPoolSize?
Yep - just denote that it's max pool size
bq. If an older server version ever initiates a trace, then traceType could be
null. And if it is null, that means that it's a QUERY traceType (since that's
the only kind of trace available in previous versions). If a newer server
version ever sends a null traceType, that should probably be an error, though
nothing like that is being done yet.
Totally legit. I'm fine with that.
bq. I touched on this code change earlier in the paragraph that begins Note
that I switched some code.
You did, and that's my bad. There's a lot of history on this ticket that I
needed to re-read and that was one of those things. :) I believe the changed
code will behave identically to the original implementation with the caveats
you listed, and so long as we can hand off that swallowed exception to the
JVMStabilityInspector and document that we intend to swallow if things explode,
I have no complaints.
bq. Futures#addCallback uses Uninterruptibles#getUninterruptibly instead of a
plain Future#get, the main difference being that it will keep retrying
Future#get even in the face of InterruptedException.
Not sure whether this would be a problem or not, but replicating previous
behavior would be the safest route.
bq. In the interests of not gratuitously rearranging code, I'll merely see
about adding better comments and making the code less magical.
Less magical is almost always a good thing.
{code} wait(wait); {code}
Reads oddly - might want to tweak naming there. Other than that, isDone on the
review branch comments itself.
nit: a more helpful description for trace_repair would be good. Currently
reads Use -tr to trace repair
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Fix For: 3.0
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181643#comment-14181643
]
Ben Chan commented on CASSANDRA-5483:
-
Review branch up at:
https://github.com/usrbincc/cassandra/tree/5483-review
StorageService#createRepairTask:
So it turns out that my mistaken assumption was this:
Tracing.instance.stopSession() needs to run in the main repair thread. (This
was not one of the assumptions that I listed explicitly above).
Why did I believe this? Because repair kept on hanging at the end until I made
this change as a sanity check.
Why does it not hang currently? I don't know, but I suspect there was something
odd in the build that got cleaned out when I did an ant clean. Or when I
manually cleaned out the build/lib/jars directory (there were multiple versions
of the same library in there; ant clean was not taking care of this).
So I put back the original addCallback code structure.
Commenting isDone:
I started trying to do this, but it quickly devolved into just badly parroting
the code. It would simplify commenting if I separated this function into
multiple parts, maybe something like this:
{code:java}
// TraceState
public enum Status { IDLE, ACTIVE, STOPPED; }
private Status status;
/* tentative doc comment
* Wait (with timeout) until status is ACTIVE or STOPPED. Reading an ACTIVE
* status resets it to IDLE. Due to the status reset logic, this function only
* works properly with a single consumer.
* @param timeout timeout in milliseconds
* @return activity status
*/
public synchronized Status getActivity(long timeout)
{
// Note: This is just a placeholder for the general gist.
while (status == Status.IDLE)
wait(timeout); // TODO: we need to update timeout if this actually
loops.
Status retStatus = status;
status = status == Status.ACTIVE ? Status.IDLE : status;
return retStatus;
}
// createQueryThread's returned Runnable#run
while ((r = getActivity(timeout)) != TraceState.Status.STOPPED)
{
if (r == TraceState.Status.IDLE) // double timeout if second time through
with this timeout
{ ... }
else // reset timeout to minWait
{ ... }
// ...
}
{code}
Note that this would end up being structured like a cleaner version of
[^5483-v14-01-squashed.patch], with further simplifications due to not trying
to handle multiple consumers.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Fix For: 3.0
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180004#comment-14180004
]
Ben Chan commented on CASSANDRA-5483:
-
Thanks; node repair -full (in conjunction with ccm clear) does indeed force
streaming repair.
[^5483-v17-00.patch] fixes a few trace calls that got lost/misplaced in the
rebase.
[^5483-v17-01.patch] reorganizes some code; functionally identical.
{noformat}
# This bash code should work out-of-the-box with a stock setup.
JMXGET='/jmx_port/{p=$2;} /binary/{split($2,a,/\047/);h=a[2];}
END{printf(bin/nodetool -h %s -p %s\n,h,p,cmd);}'
ccm_nodetool() { local N=$1; shift; $(ccm $N show | awk -F= $JMXGET) $@; }
dl_apply_maybe() { for url; do { [ -e $(basename $url) ] || curl -sO $url; }
! [ ${url%.patch} = $url ] git apply $(basename $url); done; }
NEW_BRANCH=$(date +5483-17--%Y%m%d-%H%M%S)
W=https://issues.apache.org/jira/secure/attachment
git checkout -b $NEW_BRANCH 49833b9
dl_apply_maybe \
$W/12633156/ccm-repair-test \
$W/12675963/5483-v17.patch \
$W/12675963/5483-v17-00.patch \
$W/12676340/5483-v17-01.patch
ant clean ant
chmod +x ./ccm-repair-test ./ccm-repair-test -kR
ccm node1 stop ccm node1 clear ccm node1 start
ccm_nodetool node1 repair -tr -full
ccm node1 showlog | grep Performing streaming repair
{noformat}
Note that I switched some code to a different thread in order to facilitate
trace handling -- see the diff hunk near the end of
StorageService#createRepairTask. I needed certain calls to happen in the parent
repair thread, and this seemed to be the simplest way.
As far as I can tell, there shouldn't be any differences in functionality or
concurrency level (i.e. number of tasks that are executing concurrently), but
someone should examine that section just to make sure.
The issue with unfiltered traces (see my previous message) still remains, but
if push comes to shove, you can consider this as just a special case of having
a trace call where it's not needed, or missing where it *is* needed.
In other words, a cosmetic problem. Fixing this should not require any changes
or additions to the over-wire protocol or the JMX interface.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Fix For: 3.0
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180122#comment-14180122 ] Joshua McKenzie commented on CASSANDRA-5483: I've put together a feature branch w/v17 and the 2 patches [here|https://github.com/josh-mckenzie/cassandra/compare/5483]. Are we stable and ready for review at this point [~usrbincc]? Also, for subsequent revisions please clone this feature branch, make modifications, and upload to github and link the branch here in the comments. This ticket is already far too cluttered w/attachments and it'll help keep things more clear. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180236#comment-14180236 ] Ben Chan commented on CASSANDRA-5483: - I'm not planning on making any more changes pre-review. Feel free to go ahead. (Though it would have been fine even if you had started prior to 5483-v17-00 and 5483-v17-01; the differences are not substantial) Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180242#comment-14180242 ] Joshua McKenzie commented on CASSANDRA-5483: I actually started looking at this yesterday, hence my question. :) Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180257#comment-14180257 ] Ben Chan commented on CASSANDRA-5483: - Hm. Does that mean you've already got some feedback ready? I'll be here on and off for a little while. Maybe we can get in one full iteration (or n?) today. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180384#comment-14180384
]
Joshua McKenzie commented on CASSANDRA-5483:
Initial observations / thoughts:
90 days on default repair trace TTL seems long to me, especially w/incremental
being new default.
DebuggableThreadPoolExecutor:
- 'createWithCachedTheadpool' is misspelled - needs an r in Thread
- Do we intend to create w/Integer.MAX_VALUE as 'size' on this method? If so,
name should reflect it like other methods in file.
MessageOut:
- traceParameters isn't used elsewhere so logic can be nested inside
constructor where it's called and remove unnecessary method call indirection.
OutboundTcpConnection:
- Why do we assume that if TraceType is null the type is QUERY? We should
probably be explicitly setting that for all tracing types going forward if we
plan on expanding the use of the enum
StorageService:
- nit: declaring queryThread in createRepairTask but aren't doing anything with
it - we should just call the method w/out declaration if we don't intend to
use it
- usage of bitwise ops to toggle si for indexing into seen array is
unnecessarily complex. A distinct 0/1 and toggle is more clear - there's no
need for inline index flipping w/bitwise ops as it's not gaining us any
performance or code clarity.
- nit: If not intending to use Object result from allSessions.get(), just
remove the declaration and call the method
- Swallowing Throwable on line 2828 w/out any logging or comment as to why is
unclear. Using exception handling as control flow is step back from the
sameThreadExecutor error handling approach taken prior as we lose naming and
intention information. Does this gain us something that I'm not aware of?
TraceState
- Right now isDone is only called by the query thread but there's nothing in
the method signature or comments of that method documenting the
doubling-every-second-run algorithm it's using.
- the implementation of isDone looks like a premature optimization. I would
prefer a more clear implementation of 'double every second run' than:
{code}
else if (haveWaited)
{
wait = Math.min((wait ^ 1) (wait 1), maxWaitMillis);
return false;
}
else
{
haveWaited = true;
try
{
wait(wait ~1);
}
{code}
Now that I'm more familiar with the changes here I'm going to need to re-read
the notes on this ticket and test out / inspect output. The above is just from
my first read-through of the code yesterday.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Fix For: 3.0
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
5483-v15.patch, 5483-v17-00.patch, 5483-v17-01.patch, 5483-v17.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180613#comment-14180613
]
Ben Chan commented on CASSANDRA-5483:
-
I'm going to dissapoint you and say that a lot of the reasons for how things
are is historical, or due to code transformations intended to precisely
maintain some aspect of the old behavior, without trying to question the logic
behind that particular behavior.
In other words, I've just been trying to add this one particular behavior
(repair tracing) without otherwise changing program behavior. This is probably
most obvious around the trace calls, where (hopefully) the log messages are
unchanged, even when they're different from the trace messages. I really need
to go back through that with a fine-tooth comb, because I think there were some
places where I may have changed capitalization. Or worse.
Most of your comments were straightforward. Just need some clarification on
these:
{quote}
DebuggableThreadPoolExecutor:
Do we intend to create w/Integer.MAX_VALUE as 'size' on this method?
{quote}
Yes, in order to preserve the behavior of the previous code (see my 2014-03-08
comment and the discussion surrounding it).
{quote}
If so, name should reflect it like other methods in file.
{quote}
Just to confirm (though I'm mostly sure this is what you mean), Do you mean
that the function name should be changed to something like
{{createCachedThreadpoolWithMaximumPoolSize}}?
{quote}
OutboundTcpConnection:
Why do we assume that if TraceType is null the type is QUERY?
{quote}
If an older server version ever initiates a trace, then traceType could be
null. And if it *is* null, that means that it's a QUERY traceType (since that's
the only kind of trace available in previous versions). If a newer server
version ever sends a null traceType, that should probably be an error, though
nothing like that is being done yet.
{quote}
StorageService:
Swallowing Throwable on line 2828 w/out any logging or comment as to why is
unclear. Using exception handling as control flow is step back from the
sameThreadExecutor error handling approach taken prior as we lose naming and
intention information. Does this gain us something that I'm not aware of?
{quote}
Before saying anything else: yes, I probably should have commented the
exception-swallowing.
I touched on this code change earlier in the paragraph that begins Note that I
switched some code. In a nutshell: preserving behavior of previous code.
Whether I succeeded or not is a different issue, but at least let me run
through my thought process so that you can help me find the holes in it. Here
is a redacted version of the code:
{noformat}
// Previous version:
Futures.addCallback(allSessions, new FutureCallbackObject()
{
public void onSuccess(@Nullable Object result)
{
// onSuccess stuff ...
repairComplete();
}
public void onFailure(Throwable t)
{
// onFailure stuff (empty, except for repairComplete) ...
repairComplete();
}
private void repairComplete()
{
// repairComplete stuff ...
}
}, MoreExecutors.sameThreadExecutor());
{noformat}
Here are my assumptions:
- {{// onSuccess stuff ...}} does not throw any exceptions out of its code
block.
- {{// repairComplete stuff ...}} works the same no matter which thread it runs
in (assuming it's called at the proper time, of course).
- The guava library used corresponds to the source code found at
{{build/lib/sources/guava-16.0.1-sources.jar}}.
{noformat}
// Review version:
try
{
allSessions.get();
// onSuccess stuff ...
}
catch (Throwable t)
{
// onFailure stuff (empty, except for repairComplete) ...
}
// repairComplete stuff ...
{noformat}
Since {{allSessions.get()}} is the only thing that can throw to the catch
block, then its success or failure should (almost, see below) be the only way
to trigger the onSuccess and onFailure branches that correspond to the
original code.
Intended difference:
- {{// repairComplete stuff ...}} should now run in the main repair thread,
instead of in the thread of the last exiting subtask in allSessions.
Unintended differences:
- Futures#addCallback uses Uninterruptibles#getUninterruptibly instead of a
plain Future#get, the main difference being that it will keep retrying
Future#get even in the face of InterruptedException. This may be important, so
I could do this also.
{quote}
TraceState
Right now isDone is only called by the query thread but there's nothing in the
method signature or comments of that method documenting the
doubling-every-second-run algorithm it's using.
{quote}
The doubling behavior used to be contained in the query thread; isDone used
to be a (more or less) simple wait with timeout.
Agree on needing comments. In the interests of not gratuitously rearranging
code, I'll merely see about
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177629#comment-14177629
]
Ben Chan commented on CASSANDRA-5483:
-
Some nice changes to the JMX interface in the last 3 months; I no longer have
to add lots of boilerplate function overloads.
{noformat}
read -r JMXGET E
/jmx_port/{p=\$2;} /binary/{split(\$2,a,/\047/);h=a[2];} \
END{printf(bin/nodetool -h %s -p %s\n,h,p,cmd);}
E
ccm_nodetool() { local N=$1; shift; $(ccm $N show | awk -F= $JMXGET) $@; }
# git checkout 49833b9
W=https://issues.apache.org/jira/secure/attachment
for url in $W/12675963/5483-v17.patch; do
{ [ -e $(basename $url) ] || curl -sO $url; } git apply $(basename $url)
done
ant clean ant
./ccm-repair-test -kR
ccm node1 stop
ccm node1 clear
ccm node1 start
ccm_nodetool node1 repair -tr
{noformat}
Issues encountered during testing:
- My old method for forcing a streaming repair (yukim's ccm clear suggestion)
no longer seems to work. Which means I can no longer test tracing of streaming.
Is there something else I have to do now to force a streaming repair?
- 5483-16 and 5483-17 no longer filter traces based on traceType. Using my
standard simple test, I count 182 traces, versus 61 from a test run fom
2014-06, about 200% more traces (from the query-tracing code paths), though I
don't know how this percentage will change with longer repairs. Some
representative extraneous traces:
{noformat}
Sending message to /127.0.0.2
Sending message to /127.0.0.3
Message received from /127.0.0.3
Processing response from /127.0.0.3
Message received from /127.0.0.2
Processing response from /127.0.0.2
Parsing SELECT * FROM system.sstable_activity WHERE keyspace_name=? and
columnfamily_name=? and generation=?
Preparing statement
Executing single-partition query on sstable_activity
Acquiring sstable references
Merging memtable tombstones
Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones
{noformat}
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Fix For: 3.0
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
5483-v15.patch, 5483-v17.patch, ccm-repair-test,
cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177804#comment-14177804
]
Michael Shuler commented on CASSANDRA-5483:
---
bq. My old method for forcing a streaming repair (yukim's ccm clear
suggestion) no longer seems to work.
I peeked at the ccm git repo and I don't see any recent changes around clear.
I did not go through your whole process, but a quick test of clear works out
for me with the latest ccm HEAD:
{noformat}
mshuler@hana:~$ ccm create test -v git:cassandra-2.1.0 -n1 -s
Fetching Cassandra updates...
Current cluster is now: test
mshuler@hana:~$ ls -la ~/.ccm/test/node1/{commitlogs,data/system}/
/home/mshuler/.ccm/test/node1/commitlogs/:
total 108
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34 .
drwxr-xr-x 8 mshuler mshuler 4096 Oct 20 20:34 ..
-rw-r--r-- 1 mshuler mshuler 33554432 Oct 20 20:35 CommitLog-4-1413855299422.log
-rw-r--r-- 1 mshuler mshuler 33554432 Oct 20 20:34 CommitLog-4-1413855299423.log
/home/mshuler/.ccm/test/node1/data/system/:
total 72
drwxr-xr-x 18 mshuler mshuler 4096 Oct 20 20:34 .
drwxr-xr-x 4 mshuler mshuler 4096 Oct 20 20:35 ..
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
IndexInfo-9f5c6374d48532299a0a5094af9ad1e3
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
batchlog-0290003c977e397cac3efdfdc01d626b
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca
drwxr-xr-x 3 mshuler mshuler 4096 Oct 20 20:34
compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
hints-2666e20573ef38b390fefecf96e8f0c7
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:35
local-7ad54392bcdd35a684174e047860b377
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
paxos-b7b7f0c2fd0a34108c053ef614bb7c2d
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
peer_events-59dfeaea8db2334191ef109974d81484
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
peers-37f71aca7dc2383ba70672528af04d4f
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
range_xfers-55d764384e553f8b9f6e676d4af3976d
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:35
schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:35
schema_columns-296e9c049bec3085827dc17d3df2122a
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:35
schema_keyspaces-b0f2235744583cdb9631c43e59ce3676
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
schema_triggers-0359bc7171233ee19a4ab9dfb11fc125
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
schema_usertypes-3aa752254f82350b8d5c430fa221fa0a
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:34
sstable_activity-5a1ff267ace03f128563cfae6103c65e
mshuler@hana:~$ ccm node1 clear
mshuler@hana:~$ ls -la ~/.ccm/test/node1/{commitlogs,data/system}/
ls: cannot access /home/mshuler/.ccm/test/node1/data/system/: No such file or
directory
/home/mshuler/.ccm/test/node1/commitlogs/:
total 8
drwxr-xr-x 2 mshuler mshuler 4096 Oct 20 20:35 .
drwxr-xr-x 8 mshuler mshuler 4096 Oct 20 20:35 ..
{noformat}
[~philipthompson] do you know of any recent changes around clear?
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Fix For: 3.0
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177816#comment-14177816 ] Philip Thompson commented on CASSANDRA-5483: No, I know of no changes to that command. I was not even aware of its existence. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177945#comment-14177945 ] Marcus Eriksson commented on CASSANDRA-5483: you probably need to do nodetool repair -full, incremental repairs are default in trunk Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, 5483-v17.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171479#comment-14171479 ] Jonathan Ellis commented on CASSANDRA-5483: --- That should be it! Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168742#comment-14168742 ] Ben Chan commented on CASSANDRA-5483: - Okay, that covers all the questions that I had. TODO list: - Rebase to trunk (hopefully the last 3 months haven't made this too painful). - Miscellaneous remaining cruft from the 5483-16 cleanup. Possibly some code simplification (low-hanging fruit only). - Guard out-of-bounds-array access in TraceState#deserialize. The simplest thing I can think of is to have a NONE tracetype and return that for any unknown tracetype. - Fix format string Sending completed merkle tree to %s for %s.%s being traced verbatim. Barring anything unforseen, is that about it before it's good enough for a merge? It may be a few days before I actually get around to this. I had some computer trouble recently, so I'm going to have to set up my build environment from scratch. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162149#comment-14162149 ] Jonathan Ellis commented on CASSANDRA-5483: --- bq. If remote nodes have different config values for tracetype_repair_ttl, then remote traces will end up with a different ttl I'm fine with the limitation that weird things happen if you configure it wrong. bq. If server versions are ever mixed, then an unknown trace type could be sent over the wire, causing an out-of-bounds array access during deserialization. We definitely need to support mixed-version repairs, so it sounds like this needs to be fixed. bq. Activity notification for every local repair trace starts a new exponential backoff polling cycle more often than necessary... I was thinking forward to with bitmasks was flexible subsetting of traces and notifications. I'm a big fan of not oversolving the problem. Let's start with a simple solution and evolve from there. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Fix For: 3.0 Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057548#comment-14057548
]
Ben Chan commented on CASSANDRA-5483:
-
A few things to note about 5483-16:
Just to emphasize: these are corner-cases and deoptimizations that may not ever
matter in practice. Even if the decision is to not fix them, I wanted to note
their existence.
*(1)*
If remote nodes have different config values for {{tracetype_repair_ttl}}, then
remote traces will end up with a different ttl. Having the trace for a given
session partially disappear may be surprising to the user.
The approach 5483-15 took was to use the ttl configured on the node being
repaired.
*(2)*
I don't know whether this will ever arise in practice, but I thought I'd note:
If server versions are ever mixed, then an unknown trace type could be sent
over the wire, causing an out-of-bounds array access during deserialization.
The approach 5483-15 took was almost a no-op on the remote end (the tracing
session was created, but nothing was traced). This is still not quite right.
Better would be to log an error and not create the session at all.
It might still possibly make sense to create the session anyway in the case of
A-B-C tracestate propagation, but that might be stretching things.
*(3)*
Activity notification for every local repair trace starts a new exponential
backoff polling cycle more often than necessary.
Of course, I can't say whether this extra resource usage will actually matter
in practice (only practice can tell).
Okay, so what you meant was to change tracetype to be an actual Enum type (I
can be a little dense sometimes, and code leaves far less room for ambiguity).
One thing I was thinking forward to with bitmasks was flexible subsetting of
traces and notifications. Tracing may not ever need trace levels or flexible
subsetting, but I thought I'd note the idea for future reference.
{noformat}
// pseudocode; please excuse the bad naming sense.
REPAIR_0 = 0b001 MIN_SHIFT;
REPAIR_1 = 0b010 MIN_SHIFT;
REPAIR_2 = 0b100 MIN_SHIFT;
// read zero and up, etc
REPAIR_0U = REPAIR_0 | REPAIR_1 | REPAIR_2;
REPAIR_1U =REPAIR_1 | REPAIR_2;
NOTIFY_0 = 0b001;
NOTIFY_1 = 0b010;
NOTIFY_2 = 0b100;
NOTIFY_0U = ...;
NOTIFY_1U = ...;
val = config.get_val();
trace(REPAIR_0, only traced for REPAIR_0; superseded in the more verbose
traces);
trace(REPAIR_0U, always traced for repair);
trace(REPAIR_1U, traced at REPAIR_1 and up);
trace(REPAIR_2 | val, only at REPAIR_2, or during the tracetypes specified in
val);
trace(REPAIR_1U ~val, only at REPAIR_1 and up, excluding the tracetypes
specified in val);
trace(REPAIR_2 | REPAIR_0, for some reason, we don't want to trace this at
REPAIR_1);
// you could even have semi-orthogonal subsetting and levels
trace(REPAIR_0U | NOTIFY_1, notify at NOTIFY_1 only);
trace(REPAIR_1U | NOTIFY_0U, notify at all NOTIFY levels, but only when
tracing REPAIR_1 and up);
awaitNotification(NOTIFY_1U); // notify me for NOTIFY_1 and up.
{noformat}
I believe log4j does something similar with what they call markers.
Of course, you can do this with Enum and EnumSet too (though more verbosely).
Aside from lyubent's nit (and possibly the issues in section 1), are there any
other issues that prevent this patch from being minimally useful?
Or even better, is there at least an alpha period where {{trunk}} is still
allowed to make breaking changes to the api, protocol, etc? It's hard to forsee
everything.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056727#comment-14056727 ] Jonathan Ellis commented on CASSANDRA-5483: --- bq. The code actually does use an enum (of sorts; not an Enum proper), the traceType. The traceType is passed to Tracing#getCommandName to look up the String for command name. Right. Any reason to not just use a real enum? Or maybe even better -- bq. the server isn't going to work with old versions of nodetool I get that, but C* is popular enough now that there's a reasonable number of out-of-tree C* JMX users. I'm okay with breaking compatibility for a good cause but here it seems pretty easy to avoid. bq. Repair can take a long time, so 24 hours may be too short of a ttl. Good point. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053222#comment-14053222
]
Lyuben Todorov commented on CASSANDRA-5483:
---
+1 but with a nit:
*Sending completed merkle tree to %s for %s.%s* appears in the logs because of:
{{Tracing.traceRepair(Sending completed merkle tree to %s for %s.%s,
initiator, desc.keyspace, desc.columnFamily);}} in {{Validator#run}}.
A switch can be made from _%s_ to _{}_ instead of using String.format for the
sake of minimalism.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch,
5483-v15-02-Hook-up-exponential-backoff-functionality.patch,
5483-v15-03-Exact-doubling-for-exponential-backoff.patch,
5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch,
5483-v15-05-Move-command-column-to-system_traces.sessions.patch,
5483-v15.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046316#comment-14046316 ] Jonathan Ellis commented on CASSANDRA-5483: --- Posted v16 to https://github.com/jbellis/cassandra/commits/5483-16. Simplifies TraceState as follows: - TraceType is an enum - always use the tracetype the state was created for - don't try to overgeneralize the notification handle - encapsulate notification fields - ttl is uniquely determined by trace type Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15-02-Hook-up-exponential-backoff-functionality.patch, 5483-v15-03-Exact-doubling-for-exponential-backoff.patch, 5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch, 5483-v15-05-Move-command-column-to-system_traces.sessions.patch, 5483-v15.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030076#comment-14030076
]
Ben Chan commented on CASSANDRA-5483:
-
{quote}
the exponential backoff is maintained, it's only defined more simply (as far as
I could read you doubled the wait time every second timeout, which is what
happens now also).
{quote}
Sorry, I was speed-reading the code; I saw {{wait();}} (i.e. with no timeout),
so I immediately knew there couldn't be a timeout, exponential or otherwise.
I mostly ignored what wouldn't affect execution (stupid mental code optimizer).
If you have {{wait(wait);}} and reset {{wait}} (the {{long}}) to
{{minWaitMillis}} in {{if (hasNotifications)}} then that should be about right.
I've attached a patch that does this
([^5483-v15-02-Hook-up-exponential-backoff-functionality.patch]).
I apologize for reading-code-while-distracted, thus causing me to go on about
the exponential backoff. I'll try to learn from this experience.
---
I liked your exponential backoff calculation code; I took the liberty of
tweaking it to double exactly in
[^5483-v15-03-Exact-doubling-for-exponential-backoff.patch].
---
{quote}
As the code is defined now we explicitly create a repair session and spawn a
specific thread to process it, so we have a guarantee there's only one thread.
It doesn't matter if the consumer is in its method when notifyAll is called; if
it isn't it will receive the notification as soon as it next enters the method.
{quote}
I agree with you.
When I said only ever before, I was talking about future code, not the code
in its current state. Had I considered the alternate interpretation of edge
case in the current code, I would have clarified better, but there we are.
Just to reiterate, I'm reasonably cognizant that for the code in its current
state, only a single thread is accessing the notification for a given
TraceState, and how this allows for a simpler implementation.
All I was trying to say in my previous comment was that notification upon
activity is a general capability, and in future code, should there ever be a
use case for multiple threads accessing the same activity notification, then
the implementation will have to be changed. (This also holds, should there ever
be a use case that requires something other than exponential backoff.)
Our differences turn out to be philosophical only, and I'm not all that
attached to mine.
I've already accepted [^5483-v15.patch] and have built my fixes for (some of)
the Jun 10 comments on top of it. I need some more clarification before I can
be sure what is required in order to resolve the remaining issues.
---
Test code, modified and simplified because of the updated system_traces.*
schemas (I may upload an updated {{ccm-repair-test}} if we have many more patch
iterations).
{noformat}
read -r JMXGET E
/jmx_port/{p=\$2;} \
/binary/{split(\$2,a,/\047/);h=a[2];} \
END{printf(bin/nodetool -h %s -p %s\n,h,p,cmd);}
E
ccm_nodetool() {
local N=$1
shift
$(ccm $N show | awk -F= $JMXGET) $@
}
# git checkout 85956ae
W=https://issues.apache.org/jira/secure/attachment
for url in \
$W/12649785/5483-v15.patch \
$W/12650175/5483-v15-02-Hook-up-exponential-backoff-functionality.patch \
$W/12650174/5483-v15-03-Exact-doubling-for-exponential-backoff.patch \
$W/12650173/5483-v15-04-Re-add-old-StorageService-JMX-signatures.patch \
$W/12650172/5483-v15-05-Move-command-column-to-system_traces.sessions.patch
do
{ [ -e $(basename $url) ] || curl -sO $url; } git apply $(basename $url)
done
ant clean ant
./ccm-repair-test -kR
ccm node1 stop
ccm node1 clear
ccm node1 start
ccm_nodetool node1 repair -tr
{noformat}
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027905#comment-14027905 ] Ben Chan commented on CASSANDRA-5483: - I'm sorry I can't think this through very deeply at the moment, so please allow a little slack if I say something incorrect. I'm writing this during a small window of time (where I can't do anything else) in a crisis I'm dealing with. bq. Why is command part of events instead of sessions? It was a somewhat arbitrary design decision. I can move it over to sessions. The only real reason was historical (see the Mar 3 comment); it was a proof-of-concept that never got followed up upon until just now. bq. Also: should use an enum internally. Logging as string representation is fine. Just to be clear, you mean Java code should work with an enum, and the actual cql table column is fine as a string? The code actually does use an enum (of sorts; not an Enum proper), the traceType. The traceType is passed to Tracing#getCommandName to look up the String for command name. bq. It makes people grumpy when we break JMX signatures. Can we add a new overload instead, preserving the old? This should cut down on some of the code churn in StorageService as well. I will admit that I didn't really consider the entire ecosystem of tools that use JMX to control Cassandra (though note that I did mention the JMX api change in a Mar 8 comment ... the server isn't going to work with old versions of nodetool. And a newer nodetool still won't work with older server versions.). bq. It's a minor thing to get hung up on, but I'm not wild about all the work needed to propagate TTLs around. Is it really super important to persist repair traces much longer than queries? If so, what if we used a separate table and just allowed users to modify the default ttl? (The original trace code predates default TTLs, I think, or we would have made use of it there.) I guess the question is how many different types of things (bootstrap, repair, query, decommission, ... anything else?) might eventually end up being traced. If n is small, then having n tables may be fine. The logic was this (see Mar 1-3 discussion): Repair can take a long time, so 24 hours may be too short of a ttl. I recall reading about problematic repairs taking several days, which wouldn't mix well with a 24 hour ttl. bq. Also have a nagging feeling that the notify/wait logic in StorageService + TraceState is more complex than necessary. If there is guaranteed to only be one consumer of notifications at a time, then the updated v15 logic seems fine. But if there are ever two traces going on (either of different or the same type; are you allowed to have two simultaneous repairs of different keyspaces?) which require update notifications, then there could be dropped notifications. The problem (I believe) is that all consumers may not be in a wait state at the moment when notifyAll is signalled. This means a notification could be missed, right? I'm not experienced in Java concurrency, and this isn't the best time for me to slowly think things through, so it's quite possible I'm wrong here. However, if you can be completely sure there will never be concurrent repair traces happening on the same node, or any other trace types (whatever types are added in the future) that require update notifications in order to implement on-the-fly reporting, then that issue is moot, and v15 should work fine, as far as my cursory inspection goes. bq. I should note I've made no attempt to corroborate this behaviour is sensible; I've only simplified it. Any feedback would be welcome. As I've said before, heuristics are messy. I talked about the reasoning behind my design decisions, and a possibility for an alternate implementation (with attendant tradeoffs) in a Mar 17 comment. I honestly thought I'd get more comments on it at the time, but it's possible the patch had already gotten into TL; DR territory even then. --- Okay my short break from reality is over. Time to panic. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028104#comment-14028104 ] Benedict commented on CASSANDRA-5483: - Like I said, I only replicated functionality - the exponential backoff is maintained, it's only defined more simply (as far as I could read you doubled the wait time every second timeout, which is what happens now also). As the code is defined now we explicitly create a repair session and spawn a specific thread to process it, so we have a guarantee there's only one thread. It doesn't matter if the consumer is in its method when notifyAll is called; if it isn't it will receive the notification as soon as it next enters the method. When I have some time I'll take a look and see if there's an alternative approach. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, 5483-v15.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027102#comment-14027102 ] Jonathan Ellis commented on CASSANDRA-5483: --- Some comments having applied the patch: # Why is command part of events instead of sessions? Also: should use an enum internally. Logging as string representation is fine. # It makes people grumpy when we break JMX signatures. Can we add a new overload instead, preserving the old? This should cut down on some of the code churn in StorageService as well. # It's a minor thing to get hung up on, but I'm not wild about all the work needed to propagate TTLs around. Is it really super important to persist repair traces much longer than queries? If so, what if we used a separate table and just allowed users to modify the default ttl? (The original trace code predates default TTLs, I think, or we would have made use of it there.) Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027108#comment-14027108 ] Jonathan Ellis commented on CASSANDRA-5483: --- Also have a nagging feeling that the notify/wait logic in StorageService + TraceState is more complex than necessary. WDYT [~benedict]? Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, 5483-v14-01-squashed.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020510#comment-14020510
]
Ben Chan commented on CASSANDRA-5483:
-
{noformat}
# git checkout a61ef51
W=https://issues.apache.org/jira/secure/attachment
for url in \
$W/12648746/5483-v14-01-squashed.patch
do
{ [ -e $(basename $url) ] || curl -sO $url; } git apply $(basename $url)
done
ant clean ant
./ccm-repair-test -kR
ccm node1 stop
ccm node1 clear
ccm node1 start
./ccm-repair-test -rt
{noformat}
No changes, just a (mostly) straightforward rebase onto a61ef51 (recent trunk).
Some issues only show up under actual use, so I fully expect this will need
some followup.
An idea for the timely tracing of exceptions: you might be able to kludge most
of it by modifying WrappedRunnable to trace exceptions. I haven't looked too
closely at it, though.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch,
5483-v14-01-squashed.patch, ccm-repair-test,
cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017950#comment-14017950 ] Jonathan Ellis commented on CASSANDRA-5483: --- All right, I think we're at a point where this is certainly more useful than what we have now. The perfect is the enemy of the good and all that. Ben, can you post a branch or a squashed patch? Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005853#comment-14005853
]
Ben Chan commented on CASSANDRA-5483:
-
{quote}
Is that because the traces are asynchronous? Because I think session 2 only
starts after session 1 finishes.
{quote}
Here's my high level understanding.
* Repair command #1 and #2, etc are serial.
* Each repair session Syncing range ... is technically concurrent, since
each is submitted to a ThreadPoolExecutor.
** However, differencing is serialized, so if there is no streaming going on,
you won't see very much overlap between the sessions, except at the beginning
and end (which is exactly what we see with these simple tests).
** Conversely, this means you will see much more interleaving when heavy
streaming is going on.
So at the very least, it might be good to eventually disambiguate the streaming
portion.
{quote}
The easiest thing would be to make them non-redundant. Can we make the tracing
extra detail on top of the normal ones instead of competing with them?
{quote}
I think it may be a conceptual block on my part. I tend to think of traces as a
kind of profiling mechanism.
* Most of the sendNotification calls in StorageService#createRepairTask consist
of reporting any errors from the results of RepairFuture objects. So the timing
on those is not really that useful for profiling. They're not really what I'd
usually think of as a trace.
* Some are request validation reporting before the repair proper even starts.
* The rest are informational sendNotification messages which are redundant when
tracing is active (this is the easy case).
In pseudocode:
{noformat}
if (some error #1 in repair request)
sendNotification(NO #1!);
if (some error #2 in repair request)
sendNotification(NO #2!);
for (r : ranges)
{
f = something.submitRepairSession(new RepairSession(r));
futures.add(f);
try
{
// this serializes the differencing part.
f.waitForDifferencing()
}
catch (SomeException)
{
// handle, sendNotification
}
}
try
{
for (f : futures)
{
r = f.get();
sendNotification(done: %s, r);
}
}
catch (ExecutionException ee)
{
// handle, sendNotification
}
catch (Exception e)
{
// handle, sendNotification
}
{noformat}
The main point being that I can't be sure that every single interesting
exception is caught and traced in the thread where it's thrown, then rethrown.
Most likely, this is not the case, and some exceptions are only reported at the
StorageService#createRepairTask level. I believe most \(?) cases are already
caught and traced, though.
So after going through all that, I'm thinking that the easiest thing is to just
accept the possibility of redundancy and delayed reporting, and just trace all
sendNotification in StorageService#createRepairTask (unless it's demonstrably
redundant, or already being traced through some other mechanism).
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
5483-v13-608fb03-May-14-trace-formatting-changes.patch, ccm-repair-test,
cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004285#comment-14004285 ] Jonathan Ellis commented on CASSANDRA-5483: --- bq. To make the point clearer, I've marked each sub-task graphically in the nodetool trace output below You're right that it's a bit tough to keep the sessions straight. Is that because the traces are asynchronous? Because I think session 2 only starts after session 1 finishes. Either way I think we can live with it. bq. not all nodetool messages are rendered redundant by trace output. Since we can't just suppress all non-trace sendNotification, how can we unambiguously tell nodetool trace output from normal sendNotification messages? The easiest thing would be to make them non-redundant. :) Can we make the tracing extra detail on top of the normal ones instead of competing with them? Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, 5483-v13-608fb03-May-14-trace-formatting-changes.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997265#comment-13997265
]
Ben Chan commented on CASSANDRA-5483:
-
I seem to be having problems with JIRA email notifications. May 12 arrived
fine, May 7 never arrived, May 8 arrived on May 13, but was dated May 10.
Moving on...
{quote}
Just skip system keyspace entirely and save the logspam (use Keyspace.nonSystem
instead of Keyspace.all)
{quote}
This patch has grown enough parts to be a little unwieldy. Just to be clear,
the output from [https://gist.github.com/lyubent/bfc133fe92ef1afb9dd4] is the
verbose output from {{nodetool}}, which means there is some extra output aside
from the traces themselves. (TODO to self, I need to make the nodetool verbose
output optional.). That particular message comes from
{{src/java/org/apache/cassandra/tools/NodeProbe.java}}, in a part of the code
untouched by this patch. I can go ahead and nuke that particular message for
the system keyspace.
{quote}
How does {{Endpoints /127.0.0.2 and /127.0.0.1 are consistent for events}}
scale up to more replicas? Should we switch to using {{\[..]}} notation instead?
{quote}
{{n * (n - 1)}} differences calculated for {{n}} replicas, so {{n * (n - 1)}}
are consistent messages. I haven't dug deep enough into the code to be
certain, but on the face of it, it seems like there should be some (possibly
not-simple) way to reduce this to {{O(n * log\(n))}}. Enough speculation,
though.
One edge case for the proposed notation would be a consistency partition:
{noformat}
A == B == C
A != D
D == E == F
=
# We need a separate message for each partition.
Endpoints [A, B, C] are consistent for events
Endpoints [D, E, F] are consistent for events
{noformat}
Even with the edge case, it seems messy, but doable. You do lose trace timing
information on the calculation of individual differences (the consistent ones,
at least). On the other hand, comparing matching merkle trees should be a
consistently fast operation, so you're probably not missing out on too much
information.
{quote}
I'm a little lost in the commands and sessions, e.g. does {{\[2014-05-08
23:27:45,368] Session completed successfully}} refer to session
3617e3f0-d6ef-11e3-a493-7d438369d7fc or 36a49390-d6ef-11e3-a493-7d438369d7fc?
Is there exactly one session per command? If so let's merge the starting
repair command + new session output, and the completed + finished.
{quote}
Each repair command seems to consist of multiple repair sessions (one per
range). The sessions go semi-sequentially; there's a semi-random overlap
between the end of one session and the start of another, like so (using small
integers instead of UUIDs, and some labels on the left for readability):
{noformat}
[command #1 ] Starting repair command #1
[command #1, session 1] New session 1 will sync range ...
[command #1, session 1] Requesting merkle tree for ...
[command #1, session 1] Received merkle tree for ...
[command #1, session 2] New session 2 will sync range ...
[command #1, session 2] Requesting merkle tree for ...
[command #1, session 1] Endpoints ... consistent.
[command #1, session 1] Session 1 completed successfully
[command #1, session 2] Received merkle tree for ...
[command #1, session 2] Endpoints ... consistent.
[command #1, session 3] New session 3 will sync range ...
[command #1, session 2] Session 2 completed successfully
[command #1, session 3] Requesting merkle tree for ...
[command #1, session 3] Received merkle tree for ...
[command #1, session 3] Endpoints ... consistent.
[command #1, session 3] Session 3 completed successfully
[command #1 ] Repair command #1 finished
{noformat}
Most of the time it's obvious from context, but during that overlap, having the
repair session UUID helps to disambiguate. I suspect the overlap is even
greater (and more confusing) when you have heavy streaming.
{quote}
Why do we log Repair command #1 finished with no merkle trees requested for
db.tweet? Is it because all sstables are already repaired? If so we should log
that.
{quote}
I've never encountered a trace like that in my testing. I always seem to get
merkle trees exchanged (see the log below), even if no streaming is needed. I'm
hoping lyubent can provide enough information for me to be able to recreate
this situation locally.
{quote}
Does this actually show any streaming? If so I'm missing it.
{quote}
lyubent's sample run didn't need streaming, so no streaming to trace. Here's
how I usually test streaming (using [^ccm-repair-test] and yukim's method.):
{noformat}
./ccm-repair-test -kR
ccm node1 stop
ccm node1 clear
ccm node1 start
./ccm-repair-test -rt
{noformat}
Note that this sample run uses the codebase as of
[^5483-v12-02-cassandra-yaml-ttl-doc.patch]; I haven't got around to doing the
May 12 changes yet.
I should also warn you (if recent
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997706#comment-13997706 ] Jonathan Ellis commented on CASSANDRA-5483: --- bq. Pretty sure we always do this comparison pair wise? So there will always only be two ip's in it. Ah, right. Let's leave that alone then. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993089#comment-13993089 ] Lyuben Todorov commented on CASSANDRA-5483: --- [~jbellis] Output available [here|https://gist.github.com/lyubent/bfc133fe92ef1afb9dd4] with vnode output at the top and non-vnode output at the bottom. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995530#comment-13995530
]
Jonathan Ellis commented on CASSANDRA-5483:
---
Suggestions based on the rest:
- Just skip system keyspace entirely and save the logspam (use
Keyspace.nonSystem instead of Keyspace.all)
- Be consistent about using ks.cf notation (instead of ks/cf)
- Word keyspace is redundant in repair command log: {{Starting repair command
#1, repairing 1 ranges for keyspace db.{tweet} (seq=true, full=true)}}
- Reformat {{Requesting merkle trees for sessions (to [/127.0.0.3,
/127.0.0.1])}} to {{Requesting merkle trees for sessions from [/127.0.0.3,
/127.0.0.1]}}
- How does {{Endpoints /127.0.0.2 and /127.0.0.1 are consistent for events}}
scale up to more replicas? Should we switch to using {{[..]}} notation instead?
- I'm a little lost in the commands and sessions, e.g. does {{[2014-05-08
23:27:45,368] Session completed successfully}} refer to session
3617e3f0-d6ef-11e3-a493-7d438369d7fc or 36a49390-d6ef-11e3-a493-7d438369d7fc?
Is there exactly one session per command? If so let's merge the starting
repair command + new session output, and the completed + finished.
- Why do we log {{Repair command #1 finished}} with no merkle trees requested
for db.tweet? Is it because all sstables are already repaired? If so we
should log that.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997607#comment-13997607 ] Jeremiah Jordan commented on CASSANDRA-5483: bq. How does Endpoints /127.0.0.2 and /127.0.0.1 are consistent for events scale up to more replicas? Should we switch to using [..] notation instead? Pretty sure we always do this comparison pair wise? So there will always only be two ip's in it. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997375#comment-13997375
]
Lyuben Todorov commented on CASSANDRA-5483:
---
bq. How does Endpoints /127.0.0.2 and /127.0.0.1 are consistent for events
scale up to more replicas? Should we switch to using [..] notation instead?
We can store the message in the {{Differencer}}, wait until all the
differencers complete and then retrieve their messages and build the list of
consistent / non-consistent replicas, but if this is with pre 5351 repairs,
trees will still get very large and take a long time to build thus their
comparisons might take long too so my only reason against doing this is it
might take longer to give progress information back to users.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997632#comment-13997632
]
Lyuben Todorov commented on CASSANDRA-5483:
---
bq. Why do we log Repair command #1 finished with no merkle trees requested for
db.tweet? Is it because all sstables are already repaired? If so we should log
that.
Ah, I used a Keyspace with RF 1, Here's
[output|https://gist.github.com/lyubent/b4718a132ecb7fe68a86] from a 3 node
cluster with the db keyspace having RF = 3 after node 1 was nuked and bought
back. The output has a few changes:
# System ks is no longer displayed as its not part of the repair.
# Removed _keyspace_ from Starting repair command #1, repairing 1 ranges for
keyspace db.\{tweet\} (seq=true, full=true)\}
# Reformated Requesting merkle trees for sessions (to \[/127.0.0.3,
/127.0.0.1\])
Changes for the above are on
[43eaaae..fa4089e|https://github.com/lyubent/cassandra/commit/fa4089e8e51b6d647a8f616d33a64fc13b578b14]
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995518#comment-13995518 ] Jonathan Ellis commented on CASSANDRA-5483: --- Does this actually show any streaming? If so I'm missing it. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, 5483-v10-17-minor-bugfixes-and-changes.patch, 5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch, 5483-v11-squashed-nits.patch, 5483-v12-02-cassandra-yaml-ttl-doc.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987979#comment-13987979
]
Ben Chan commented on CASSANDRA-5483:
-
Patch updated to work with current trunk (08d5f26).
{noformat}
# git checkout 08d5f26
W=https://issues.apache.org/jira/secure/attachment
for url in \
$W/12643077/5483-v11-01-squashed.patch
do
{ [ -e $(basename $url) ] || curl -sO $url; } git apply $(basename $url)
done
ant clean ant
./ccm-repair-test -kR
ccm node1 stop
ccm node1 clear
ccm node1 start
./ccm-repair-test -rt
{noformat}
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, 5483-v11-01-squashed.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966778#comment-13966778
]
Ben Chan commented on CASSANDRA-5483:
-
I made some additional changes. Everything is included in
[^5483-v10-rebased-and-squashed-471f5cc.patch], but I attached
[^5483-v10-17-minor-bugfixes-and-changes.patch] to make it more more convenient
to review.
Repair fails without
[7000-2.1-v2.txt|https://issues.apache.org/jira/secure/attachment/12639633/7000-2.1-v2.txt]
so I've included that patch in the test code.
Overview:
* Limit exponential backoff.
* Handle the case where traceType is negative.
* Reimplement log2 in BitUtil style.
* Forgot to add a trace parameter.
{noformat}
# rebased against trunk @ 471f5cc
# git checkout 471f5cc
W=https://issues.apache.org/jira/secure/attachment
for url in \
$W/12639821/5483-v10-rebased-and-squashed-471f5cc.patch \
$W/12639633/7000-2.1-v2.txt
do
{ [ -e $(basename $url) ] || curl -sO $url; } git apply $(basename $url)
done
ant clean ant
./ccm-repair-test -kR
ccm node1 stop
ccm node1 clear
ccm node1 start
./ccm-repair-test -rt
{noformat}
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch,
5483-v10-17-minor-bugfixes-and-changes.patch,
5483-v10-rebased-and-squashed-471f5cc.patch, ccm-repair-test,
cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965957#comment-13965957 ] Lyuben Todorov commented on CASSANDRA-5483: --- [~usrbincc] Can you post the squashed rebase? Would really appreciate it! Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961081#comment-13961081
]
Ben Chan commented on CASSANDRA-5483:
-
Rebased, but can't test properly, because the trunk as of 64bc45849fd2 (my
{{git rebase}} base) is throwing an error:
{noformat}
java.lang.ExceptionInInitializerError: null
at
org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:92)
~[main/:na]
at
org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:545)
~[main/:na]
at
org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:128)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:109)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:454)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:543)
[main/:na]
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.SyntaxException: line 1:127 mismatched
character 'T' expecting set null
at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:518)
~[main/:na]
at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:501)
~[main/:na]
at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:99)
~[main/:na]
... 6 common frames omitted
Caused by: org.apache.cassandra.exceptions.SyntaxException: line 1:127
mismatched character 'T' expecting set null
at
org.apache.cassandra.cql3.CqlLexer.throwLastRecognitionError(CqlLexer.java:201)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:348)
~[main/:na]
at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:509)
~[main/:na]
... 8 common frames omitted
{noformat}
Things are currently hectic, so I don't know when I'll have the chance to have
a proper look at it. I'm currently doing a {{git bisect run}} so maybe
something obvious will turn up.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, ccm-repair-test,
cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959183#comment-13959183 ] Jonathan Ellis commented on CASSANDRA-5483: --- Can you both rebase and squash? Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, 5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch, 5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch, 5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch, 5483-v08-14-Poll-system_traces.events.patch, 5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch, 5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958055#comment-13958055
]
Lyuben Todorov commented on CASSANDRA-5483:
---
v9 has indeed fixed the high cpu usage. I pushed v8 and v9 to
[d9abbfc62319935ef42d4685037f80b292e03e0f|https://github.com/lyubent/cassandra/commit/d9abbfc62319935ef42d4685037f80b292e03e0f]
with only the minor change of giving a name to the thread that [carries out
polling|https://github.com/lyubent/cassandra/commit/d9abbfc62319935ef42d4685037f80b292e03e0f#diff-b76a607445d53f18a98c9df14323c7ddR2649]
for nodetool output. The thread name is currently {{RepairTracePolling}} but
I'm open to suggestions.
[~usrbincc] I think we should rebase to the latest trunk but the patch is
looking good.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, ccm-repair-test,
cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13954746#comment-13954746
]
Ben Chan commented on CASSANDRA-5483:
-
Ouch. After running a before and after test, I'm 99% sure this was the problem.
There was some obviously wrong code in {{waitActivity}} (an older version used
0 instead of -1 to signify done; I apparently forgot to update everything
when I changed this).
Sorry about removing the previous patch. It didn't have the correct {{git diff
-p}} parameters.
For convenience:
{noformat}
W=https://issues.apache.org/jira/secure/attachment
for url in \
$W/12637720/5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch
do [ -e $(basename $url) ] || curl -sO $url; done
git apply 5483-v09-*.patch
ant clean ant
{noformat}
Here's what I used to test with; I get slower and slower repairs, with a hang
on the 5th repair with the before code, and consistent 10-second repairs with
the after code.
{noformat}
cat ccm-nodetool E
#!/bin/sh
# ccm doesn't let us call nodetool with options, but we still need to get the
# host and port config from it.
read -r JMXGET E
/jmx_port/{p=\$2;} \
/binary/{split(\$2,a,/\047/);h=a[2];} \
END{printf(bin/nodetool -h %s -p %s\n,h,p,cmd);}
E
NODETOOL=$(ccm $1 show | awk -F= $JMXGET)
shift
$NODETOOL $@
E
chmod +x ccm-nodetool
for x in $(seq 3); do
for y in $(seq 2); do
echo repair node$x \#$y
./ccm-nodetool node$x repair -tr
done
done
{noformat}
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
5483-v09-16-Fix-hang-caused-by-incorrect-exit-code.patch, ccm-repair-test,
cqlsh-left-justify-text-columns.patch, prerepair-vs-postbuggedrepair.diff,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937976#comment-13937976
]
Ben Chan commented on CASSANDRA-5483:
-
{noformat}
# tested with branch 5483 @ bce0c2c555a3; should also work following successful
#git apply 5483-full-trunk.txt
W=https://issues.apache.org/jira/secure/attachment
for url in \
$W/12635094/5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch
\
$W/12635095/5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch
\
$W/12635096/5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch
\
$W/12635097/5483-v08-14-Poll-system_traces.events.patch
\
$W/12635098/5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch
do [ -e $(basename $url) ] || curl -sO $url; done
git apply 5483-v08-*.patch
ant clean ant
./ccm-repair-test -kR
ccm node1 stop
ccm node1 clear
ccm node1 start
./ccm-repair-test -rt
{noformat}
* {{v08-11}} There was an error in one of the log formats in Differencer, which
made my grep for out of sync in the logs fruitless.
* {{v08-12}} I ended up using the handleStreamEvent of StreamingRepairTask
instead of implementing and registering my own StreamEventHandler. The new
trace messages may need adjusting, especially for ProgressEvent, which is
essentially just a toString currently.
* {{v08-13}} This works by adding a guarded sendNotification to
TraceState#trace.
* {{v08-14}} This works by starting a thread to poll {{system_traces.events}},
and by adding notify functionality to TraceState. There is some jitter in the
ordering between local and remote traces. An easy fix would be to have the
query thread handle all sendNotification of traces. You have to accept latency
in sendNotification of local traces in order to get better ordering. It might
be necessary to delay all trace sendNotification by a few seconds to make it
more likely that remote traces have arrived.
* {{v08-15}} Even more added TraceState functionality. All to try to reduce the
amount of polling without hurting latency too much. There are only a few local
traces that you would expect to be followed by a remote trace, so only wake up
for those. Poll with an exponential backoff after each notification.
---
Heuristics are messy, and I expect plenty of opinions on {{v08-14}} and
{{v08-15}}. I'm not especially proud of that code, but I can't think of
anything better at the moment, given the (self-imposed?) constraints.
I may have reinvented the wheel with synchronization primitives. I checked
{{java.util.concurrent.*}} and {{SimpleCondition}}, but not much beyond that. I
could have missed something; I don't fully understand some of the classes. What
I wanted was to be woken up (with a timeout) if anything has changed since the
last time I checked. Theoretically, it should work for multiple consumers (As
long as no one waits for longer than {{Integer.MAX_VALUE}} updates), though
that's not really necessary here, if that would simplify the code.
The code seems to work reasonably well for small-scale tests. I can convince
myself that it won't blow up for long repairs, but haven't done a full test yet.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
5483-v08-11-Shorten-trace-messages.-Use-Tracing-begin.patch,
5483-v08-12-Trace-streaming-in-Differencer-StreamingRepairTask.patch,
5483-v08-13-sendNotification-of-local-traces-back-to-nodetool.patch,
5483-v08-14-Poll-system_traces.events.patch,
5483-v08-15-Limit-trace-notifications.-Add-exponential-backoff.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13936173#comment-13936173
]
Ben Chan commented on CASSANDRA-5483:
-
About the {{system_traces}} feedback loop during repair. I don't have a way to
consistently reproduce it. Normally these test repairs take no longer than 1-2
minutes, but occasionally, it gets into the 10-20 minute range, with lots of
merkle trees for {{system_traces}} getting sent ({{system_traces.events}} at
that point is in the tens of thousands of rows.), at which point I usually just
stop the repair, and start fresh with a new cluster. I'll pay closer attention
next time to see if I can get a good repro case.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
ccm-repair-test, cqlsh-left-justify-text-columns.patch,
test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935288#comment-13935288 ] Jonathan Ellis commented on CASSANDRA-5483: --- bq. Maybe exclude system_traces from repair if a repair trace is going on. I agree that something fishy is going on, but I'd like to understand the cause before adding a workaround. Can you shed any light [~yukim]? bq. Maybe add a placeholder row with a null duration for ongoing repair sessions This is what query tracing does (Tracing.begin, then stopSession writes the duration). bq. Verbose option; dump all traces to nodetool. I'd be okay pushing this to another ticket, FWIW. bq. Simplify trace messages The main principle you want to be careful to observe is to 1) log everything the user needs to know but 2) without logging extra noise. Redundant information is noise. So logging the repairsession once in the New session line is adequate, you don't need to repeat it with every message afterwards. Similarly, Requesting merkle tree line shows what peers are involved, we don't need to repeat that elsewhere either. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934196#comment-13934196 ] Yuki Morishita commented on CASSANDRA-5483: --- bq. Is there a simple way to create a situation where a repair requires streaming? The easiest way is to nuke one node after writing data to all nodes. So, for example, you create 3 nodes cluster, create keyspace with RF=3, fill data in, clear one node(ccm node1 clear), bring it up again, then repair. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, ccm-repair-test, cqlsh-left-justify-text-columns.patch, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13931858#comment-13931858 ] Jonathan Ellis commented on CASSANDRA-5483: --- Can you post an example trace session? Does this add a verbose flag to trace to show the trace as-it-happens or do I have to manually check the trace tables? Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-full-trunk.txt, 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch, 5483-v07-08-Fix-brace-style.patch, 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch, 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch, ccm-repair-test, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13931875#comment-13931875
]
Lyuben Todorov commented on CASSANDRA-5483:
---
It's a good idea to add a verbose option to show tracing as it happens.
Currently to enable logging to the system_traces.events table the {{-tr}} flag
has to be supplied {{./nodetool repair -tr}}, and then to view the traces
{{SELECT * FROM system_traces.events}}
{noformat}
cqlsh SELECT * FROM system_traces.events ;
session_id | event_id |
activity
| command | source| source_elapsed | thread
--+--+---+-+---++---
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fedbe971-a9fa-11e3-ab61-2387bbb38302 |
Starting repair command #3, repairing 2 ranges for keyspace system_traces
(seq=true, full=true) | REPAIR | 127.0.0.1 | 11 |
Thread-16
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fedc5ea0-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] new session: will sync
/127.0.0.1, /127.0.0.3 on range (-3074457345618258603,3074457345618258602] for
system_traces.[sessions, events] | REPAIR | 127.0.0.1 | 2493 |
AntiEntropySessions:7
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fedef6b0-a9fa-11e3-ab61-2387bbb38302 |
[repair
#fedc3790-a9fa-11e3-ab61-2387bbb38302] requesting merkle trees for sessions (to
[/127.0.0.3, /127.0.0.1]) | REPAIR | 127.0.0.1 | 19687 |
RepairJobTask:2
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fee167b0-a9fa-11e3-b120-2387bbb38302 |
[repair
#fedc3790-a9fa-11e3-ab61-2387bbb38302] Sending completed merkle tree to
/127.0.0.1 for system_traces/sessions | REPAIR | 127.0.0.3 | 31937 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fee3b1a0-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
sessions from /127.0.0.3 | REPAIR | 127.0.0.1 | 50996 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fee7d050-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
sessions from /127.0.0.1 | REPAIR | 127.0.0.1 | 77907 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | feea1a40-a9fa-11e3-ab61-2387bbb38302 |
[repair
#fedc3790-a9fa-11e3-ab61-2387bbb38302] requesting merkle trees for events (to
[/127.0.0.3, /127.0.0.1]) | REPAIR | 127.0.0.1 | 92358 |
RepairJobTask:2
fedbe970-a9fa-11e3-ab61-2387bbb38302 | feecd960-a9fa-11e3-b120-2387bbb38302 |
[repair
#fedc3790-a9fa-11e3-ab61-2387bbb38302] Sending completed merkle tree to
/127.0.0.1 for system_traces/events | REPAIR | 127.0.0.3 | 107062 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | feef2350-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
events from /127.0.0.3 | REPAIR | 127.0.0.1 | 126230 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fef3de40-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
events from /127.0.0.1 | REPAIR | 127.0.0.1 | 157175 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fef40551-a9fa-11e3-ab61-2387bbb38302 |
[repair #fef40550-a9fa-11e3-ab61-2387bbb38302] new session: will sync
/127.0.0.1, /127.0.0.2 on range (3074457345618258602,-9223372036854775808] for
system_traces.[sessions, events] | REPAIR | 127.0.0.1 | 158225 |
AntiEntropySessions:8
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fef4c8a0-a9fa-11e3-ab61-2387bbb38302 |
[repair
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13932510#comment-13932510
]
Jonathan Ellis commented on CASSANDRA-5483:
---
Thanks. I'm going to bikeshed the messages a bit:
- {{Starting repair command #3, repairing 2 ranges for keyspace system_traces
(seq=true, full=true)}} - {{Repairing 2 ranges for system_traces.{sessions,
events} (seq=true, full=true)}}
- {{[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] new session: will sync
/127.0.0.1, /127.0.0.3 on range (-3074457345618258603,3074457345618258602] for
system_traces.[sessions, events]}} - {{New session
#fedc3790-a9fa-11e3-ab61-2387bbb38302 will sync range
(-3074457345618258603,3074457345618258602]}}
- All the rest: omit the session id since that's already traced, and capitalize
the first word
- Add a message if there's nothing to stream
What does it log when there *is* something to stream?
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-full-trunk.txt,
5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
5483-v07-08-Fix-brace-style.patch,
5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
ccm-repair-test, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13930004#comment-13930004 ] Lyuben Todorov commented on CASSANDRA-5483: --- [~usrbincc] Can you please attach the last patch so we can complete this ticket? I'll apply it into my git branch :) Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, ccm-repair-test, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924874#comment-13924874
]
Ben Chan commented on CASSANDRA-5483:
-
Thanks for the catch. Hopefully the fix is as easy as:
{noformat}
// using OpenJDK's source code for Executors.newCachedThreadPool() as a
reference
new DebuggableThreadPoolExecutor(0, Integer.MAX_VALUE, 60L, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
new NamedThreadFactory(RepairJobTask));
{noformat}
Also note that {{v02-01}} added {{CompactionExecutor extends
DebuggableThreadPoolExecutor}}, so that is something else to watch out for.
---
Re: {
Oops (I'm normally KR).
Re: github
Move to a github workflow, or some sort of hybrid github/JIRA?
---
I guess the last caveat here is that most (I may have left in a few) of the old
non-{{trace}} function overloads are gone, so the server isn't going to work
with old versions of {{nodetool}}.
And a newer {{nodetool}} still won't work with older server versions. There may
be some tricks you can do with JMX, but I don't have deep knowledge there.
Maybe adding function overloads from the future (I noticed a similar trick in
{{MessagingService.java}}).
And there is a lot of repetitious parallel code with the tracing and logging
together. But that shouldn't affect the external functionality.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
ccm-repair-test, test-5483-system_traces-events.txt,
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt,
v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924903#comment-13924903 ] Lyuben Todorov commented on CASSANDRA-5483: --- bq. Move to a github workflow, or some sort of hybrid github/JIRA? [~usrbincc] Whichever you prefer, attaching a patch will be simpler, I can merge it into the existing github branch and post the update, but if you want feel free to create a branch yourself. bq. the server isn't going to work with old versions of nodetool I think that is ok since if someone upgrades it would kind of be expected for them to upgrade to the new nodetool as well. bq. there is a lot of repetitious parallel code with the tracing and logging together Maybe worth refactoring on a separate ticket, but lets not cause regression by trying to minimise code. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch, 5483-v06-05-Add-a-command-column-to-system_traces.events.patch, 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch, ccm-repair-test, test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt, v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch, v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch, v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13922721#comment-13922721
]
Ben Chan commented on CASSANDRA-5483:
-
It was more involved than I thought, partly because of heisenbugs and the trace
state mysteriously not propagating (see {{v06-05}}).
Note: changing JMX can cause mysterious errors if you don't {{ant clean
ant}}. I ran into the same kinds of stack traces as you did. It's not
consistent. Sometimes I can make a JMX change and {{ant}} with no problem.
To make patches simpler, I'm posting full repro code. I also tried to simplify
the naming. Unfortunately, all the previous patches are in jumbled order due to
a naming convention that doesn't sort. Fortunately, JIRA seems to have an
easter egg where you can choose the attachment name by changing the url.
{noformat}
# Uncomment to exactly reproduce state.
#git checkout -b 5483-e30d6dc e30d6dc
# Download all needed patches with consistent names, apply patches, build.
W=https://issues.apache.org/jira/secure/attachment
for url in \
$W/12630490/5483-v02-01-Trace-filtering-and-tracestate-propagation.patch \
$W/12630491/5483-v02-02-Put-a-few-traces-parallel-to-the-repair-logging.patch
\
$W/12631967/5483-v03-03-Make-repair-tracing-controllable-via-nodetool.patch \
$W/12633153/5483-v06-04-Allow-tracing-ttl-to-be-configured.patch \
$W/12633154/5483-v06-05-Add-a-command-column-to-system_traces.events.patch \
$W/12633155/5483-v06-06-Fix-interruption-in-tracestate-propagation.patch \
$W/12633156/ccm-repair-test
do [ -e $(basename $url) ] || curl -sO $url; done
git apply 5483-v0[236]-*.patch
ant clean ant
# put on a separate line because you should at least minimally inspect
# arbitrary code before running.
chmod +x ./ccm-repair-test ./ccm-repair-test
{noformat}
{{ccm-repair-test}} has some options for convenience:
{noformat}
-k keep (don't delete) the created cluster after successful exit.
-r repair only
-R don't repair
-t do traced repair only
-T don't do traced repair (if neither, then do both traced and untraced repair)
{noformat}
The output of a test run:
{noformat}
Current cluster is now: test-5483-QiR
[2014-03-06 10:46:13,617] Nothing to repair for keyspace 'system'
[2014-03-06 10:46:13,646] Starting repair command #1, repairing 2 ranges for
keyspace s1 (seq=true, full=true)
[2014-03-06 10:46:16,999] Repair session 72648190-a546-11e3-a5f4-f94811c7b860
for range (-3074457345618258603,3074457345618258602] finished
[2014-03-06 10:46:17,465] Repair session 73ee2ed0-a546-11e3-a5f4-f94811c7b860
for range (3074457345618258602,-9223372036854775808] finished
[2014-03-06 10:46:17,465] Repair command #1 finished
[2014-03-06 10:46:17,485] Starting repair command #2, repairing 2 ranges for
keyspace system_traces (seq=true, full=true)
[2014-03-06 10:46:18,782] Repair session 74aaef20-a546-11e3-a5f4-f94811c7b860
for range (-3074457345618258603,3074457345618258602] finished
[2014-03-06 10:46:18,816] Repair session 74ff0290-a546-11e3-a5f4-f94811c7b860
for range (3074457345618258602,-9223372036854775808] finished
[2014-03-06 10:46:18,816] Repair command #2 finished
0 rows exported in 0.015 seconds.
test-5483-QiR-system_traces-events.txt
ok
[2014-03-06 10:46:24,128] Nothing to repair for keyspace 'system'
[2014-03-06 10:46:24,166] Starting repair command #3, repairing 2 ranges for
keyspace s1 (seq=true, full=true)
[2014-03-06 10:46:25,366] Repair session 78a6d4e0-a546-11e3-a5f4-f94811c7b860
for range (-3074457345618258603,3074457345618258602] finished
[2014-03-06 10:46:25,415] Repair session 79263e10-a546-11e3-a5f4-f94811c7b860
for range (3074457345618258602,-9223372036854775808] finished
[2014-03-06 10:46:25,415] Repair command #3 finished
[2014-03-06 10:46:25,485] Starting repair command #4, repairing 2 ranges for
keyspace system_traces (seq=true, full=true)
[2014-03-06 10:46:27,077] Repair session 796f7c10-a546-11e3-a5f4-f94811c7b860
for range (-3074457345618258603,3074457345618258602] finished
[2014-03-06 10:46:27,120] Repair session 79f240a0-a546-11e3-a5f4-f94811c7b860
for range (3074457345618258602,-9223372036854775808] finished
[2014-03-06 10:46:27,120] Repair command #4 finished
48 rows exported in 0.104 seconds.
test-5483-QiR-system_traces-events-tr.txt
found source: 127.0.0.1
found thread: Thread-15
found thread: AntiEntropySessions:1
found thread: RepairJobTask:1
found source: 127.0.0.2
found thread: AntiEntropyStage:1
found source: 127.0.0.3
found thread: AntiEntropySessions:2
found thread: Thread-16
found thread: AntiEntropySessions:3
found thread: AntiEntropySessions:4
unique sources traced: 3
unique threads traced: 8
All thread categories accounted for
ok
{noformat}
---
Patch comments:
- {{v06-04}} I did something similar to {{v03-03}}, (almost) no refactoring.
The implementation is a little messy architecturally.
- {{v06-05}} This is the suggestion you had to add a command
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918251#comment-13918251
]
Lyuben Todorov commented on CASSANDRA-5483:
---
Are the latest 3 patches supposed to be incrementally added onto
{{trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch}}
and
{{trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch}}?
As in
{noformat}
1 - apply
trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch
2 - apply
trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch
3 - apply one of the three latest patches (v3, v4 or v5)
{noformat}
v5 Does a lot of refactoring that I think is outside the scope of this ticket
(but might be worth it's own ticket as the idea is good), so my vote is for v3,
but I'm getting a NoSuchMethod exception, can you post a branch with all the
patches added onto trunk (for v3)?
The exception:
{noformat}
java.lang.NoSuchMethodException: forceRepairAsync(java.lang.String, boolean,
java.util.Collection, java.util.Collection, boolean, boolean, boolean,
[Ljava.lang.String;)
at
com.sun.jmx.mbeanserver.PerInterface.noSuchMethod(PerInterface.java:168)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:135)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{noformat}
bq. I am thinking of calling the new table something generic like
system_traces.trace_logs. I also assume, that like system_traces.events
I'd say events is pretty generic, the new table should show that the traces
aren't query related like in events. If we are going to add new tables to the
trace CF it's worth thinking about refactoring events into something more
specific and adding new tables with names that carry meaning. Another possible
solution is to add a command field to system_traces.events where it can allow
users to retrieve data about specific events, e.g. [~jbellis] WDYT?
{noformat}
SELECT * FROM system_traces.events;
session_id | ... | thread| command
--+ ... +---+-
09d48eb0-a2f1-11e3-9f04-7d9e3709bf93 | ... | Thrift:1 | REPAIR
29084f90-a2f3-11e3-9f04-7d9e3709bf93 | ... | Thrift:1 | QUERY
(2 rows)
SELECT * FROM system_traces.events WHERE command='REPAIR';
session_id | ... | thread| command
--+ ... +---+-
09d48eb0-a2f1-11e3-9f04-7d9e3709bf93 | ... | Thrift:1 | REPAIR
(1 rows)
{noformat}
bq. the rows in this table should expire, though perhaps not as fast as 24
hours.
+1, repairs can take a very long time so this should be configurable with the
default perhaps being around 30 days, but with incremental repairs (in 2.1) it
will end up logging a lot of data, still a better choice than users doing
regular repairs missing out on information.
bq. One last thing I wanted to ask is about the
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912754#comment-13912754 ] Lyuben Todorov commented on CASSANDRA-5483: --- [~usrbincc] Join the club :D. The patch makes it so repair is traced regardless of whether tracing is on or not. Is this on purpose? Also as you said the table does fill up rather quickly and this tracing should be moved to a separate table as [~ob] suggested. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: test-5483-system_traces-events.txt, trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch, trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902431#comment-13902431 ] Lyuben Todorov commented on CASSANDRA-5483: --- [~usrbincc] Can you please rebase to trunk? Patches don't apply cleanly anymore. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: test-5483-system_traces-events.txt, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13899275#comment-13899275 ] Jonathan Ellis commented on CASSANDRA-5483: --- Can you review, Lyuben? Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: test-5483-system_traces-events.txt, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870131#comment-13870131
]
Ben Chan commented on CASSANDRA-5483:
-
Thus far, I've been going under the assumption that it's mostly meant to be
used as a combination of performance profiling and some sort of globally
accessible error log (for that particular repair session), though there will
probably easily be enough information there to be able to extract some stats.
I could use some suggestions on what things (and what level of detail) would be
good to trace. In addition to what I already have, I can only think of a few
places in {{Differencer}} and {{StreamingRepairTask}}, along with a more
complete covering of the repair-related error logs.
Repair tracing
--
Key: CASSANDRA-5483
URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Yuki Morishita
Assignee: Ben Chan
Priority: Minor
Labels: repair
Attachments: test-5483-system_traces-events.txt,
tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt
I think it would be nice to log repair stats and results like query tracing
stores traces to system keyspace. With it, you don't have to lookup each log
file to see what was the status and how it performed the repair you invoked.
Instead, you can query the repair log with session ID to see the state and
stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869151#comment-13869151 ] Oscar Bonilla commented on CASSANDRA-5483: -- I was looking at getting some stats from repair and was pointed here. Looking at the code makes me think that repair is more like compaction than query tracing. I.e. it would make more sense to keep a system.repair_history that mimics system.compaction_history. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Yuki Morishita Assignee: Ben Chan Priority: Minor Labels: repair Attachments: test-5483-system_traces-events.txt, tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt, tr...@8ebeee1-5483-v01-002-simple-repair-tracing.txt I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637887#comment-13637887 ] Sylvain Lebresne commented on CASSANDRA-5483: - We should even probably don't stop at repair (but include bootstrap/decommission etc...). Though not suggesting by that that we should all do in one shot. Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Reporter: Yuki Morishita Assignee: Yuki Morishita Priority: Minor Labels: repair I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5483) Repair tracing
[ https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633321#comment-13633321 ] Jonathan Ellis commented on CASSANDRA-5483: --- Good idea! Repair tracing -- Key: CASSANDRA-5483 URL: https://issues.apache.org/jira/browse/CASSANDRA-5483 Project: Cassandra Issue Type: Improvement Reporter: Yuki Morishita Assignee: Yuki Morishita Priority: Minor Labels: repair I think it would be nice to log repair stats and results like query tracing stores traces to system keyspace. With it, you don't have to lookup each log file to see what was the status and how it performed the repair you invoked. Instead, you can query the repair log with session ID to see the state and stats of all nodes involved in that repair session. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
