[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14790547#comment-14790547
]
Alexey edited comment on CASSANDRA-8696 at 9/16/15 3:27 PM:
Experiencing both this and CASSANDRA-9624 on 2.1.9
{code}
$ dpkg -s cassandra
...
Architecture: all
Version: 2.1.9
$ uname -a
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux
{code}
Here is how nodetool repair finishing
{code}
Repair command #17 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}
Tried to:
{code}
$ nodetool flush
$ nodetool compact
$ nodetool cleanup
{code}
compactionstats show 0 pending tasks for all nodes.
Attached Logs.zip:
There a 2 system.log files since first was rolled to second due to size.
There is also stdout/stderr output of nodetool repair.
was (Author: folex):
Experiencing both this and CASSANDRA-9624 on 2.1.9
{code}
$ dpkg -s cassandra
...
Architecture: all
Version: 2.1.9
$ uname -a
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux
{code}
Here is how nodetool repair finishing
{code}
Repair command #17 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}
Tried to:
{code}
$ nodetool flush
$ nodetool compact
$ nodetool cleanup
{code}
compactionstats show 0 pending tasks for all nodes.
There a 2 system.log files since first was rolled to second due to size.
There is also stdout/stderr output of nodetool repair.
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>Assignee: Yuki Morishita
> Fix For: 2.1.x
>
> Attachments: Logs.zip
>
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14790547#comment-14790547
]
Alexey edited comment on CASSANDRA-8696 at 9/16/15 3:25 PM:
Experiencing both this and CASSANDRA-9624 on 2.1.9
{code}
$ dpkg -s cassandra
...
Architecture: all
Version: 2.1.9
$ uname -a
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux
{code}
Here is how nodetool repair finishing
{code}
Repair command #17 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}
There a 2 system.log files since first was rolled to seconds due to size.
There is also stdout/stderr output of nodetool repair.
was (Author: folex):
There a 2 system.log files since first was rolled to seconds due to size.
There is also stdout/stderr output of nodetool repair.
- by folex
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>Assignee: Yuki Morishita
> Fix For: 2.1.x
>
> Attachments: Logs.zip
>
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> -
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14790547#comment-14790547
]
Alexey edited comment on CASSANDRA-8696 at 9/16/15 3:25 PM:
Experiencing both this and CASSANDRA-9624 on 2.1.9
{code}
$ dpkg -s cassandra
...
Architecture: all
Version: 2.1.9
$ uname -a
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux
{code}
Here is how nodetool repair finishing
{code}
Repair command #17 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}
There a 2 system.log files since first was rolled to second due to size.
There is also stdout/stderr output of nodetool repair.
was (Author: folex):
Experiencing both this and CASSANDRA-9624 on 2.1.9
{code}
$ dpkg -s cassandra
...
Architecture: all
Version: 2.1.9
$ uname -a
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux
{code}
Here is how nodetool repair finishing
{code}
Repair command #17 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}
There a 2 system.log files since first was rolled to seconds due to size.
There is also stdout/stderr output of nodetool repair.
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>Assignee: Yuki Morishita
> Fix For: 2.1.x
>
> Attachments: Logs.zip
>
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
>
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14790547#comment-14790547
]
Alexey edited comment on CASSANDRA-8696 at 9/16/15 3:26 PM:
Experiencing both this and CASSANDRA-9624 on 2.1.9
{code}
$ dpkg -s cassandra
...
Architecture: all
Version: 2.1.9
$ uname -a
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux
{code}
Here is how nodetool repair finishing
{code}
Repair command #17 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}
Tried to:
{code}
$ nodetool flush
$ nodetool compact
$ nodetool cleanup
{code}
compactionstats show 0 pending tasks for all nodes.
There a 2 system.log files since first was rolled to second due to size.
There is also stdout/stderr output of nodetool repair.
was (Author: folex):
Experiencing both this and CASSANDRA-9624 on 2.1.9
{code}
$ dpkg -s cassandra
...
Architecture: all
Version: 2.1.9
$ uname -a
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux
{code}
Here is how nodetool repair finishing
{code}
Repair command #17 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}
There a 2 system.log files since first was rolled to second due to size.
There is also stdout/stderr output of nodetool repair.
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>Assignee: Yuki Morishita
> Fix For: 2.1.x
>
> Attachments: Logs.zip
>
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
>
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484169#comment-14484169
]
kris edited comment on CASSANDRA-8696 at 4/7/15 10:00 PM:
--
faced the same issue with dse 4.6 and cassandra version 2.0.11.83. Have been
running repair with opscenter
was (Author: kris88):
faced the same issue with dse 4.6 and cassandra version 2.0.11.83
nodetool repair on cassandra 2.1.2 keyspaces return
java.lang.RuntimeException: Could not create snapshot
-
Key: CASSANDRA-8696
URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Yuki Morishita
Fix For: 2.1.5
When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
throw java exceptions: cannot create snapshot.
the error log from system.log:
{noformat}
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
files(632105 bytes)
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
Session with /10.97.9.110 is complete
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
All sessions completed
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
streaming task succeed, returning response to /10.98.194.68
INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
[Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
Starting streaming to /10.66.187.201
INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
ID#0] Beginning stream session with /10.66.187.201
INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
files(632105 bytes)
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
Session with /10.66.187.201 is complete
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
All sessions completed
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
streaming task succeed, returning response to /10.98.194.68
ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
- [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
/10.98.194.68, /10.66.187.201, /10.226.218.135 on range
(12817179804668051873746972069086
2638799,12863540308359254031520865977436165] for events.[bigint0text,
bigint0boolean, bigint0int, dataset_catalog, column_categories,
bigint0double, bigint0bigint]
ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
- [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
following error
java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391477#comment-14391477
]
Stefano Ortolani edited comment on CASSANDRA-8696 at 4/1/15 9:24 PM:
-
It eventually finished the repair (after 12 hours) and the node is now silent.
UPDATE: started again on another node, again, same error
was (Author: ostefano):
It eventually finished the repair (after 12 hours) and the node is now silent.
nodetool repair on cassandra 2.1.2 keyspaces return
java.lang.RuntimeException: Could not create snapshot
-
Key: CASSANDRA-8696
URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Yuki Morishita
Fix For: 2.1.5
When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
throw java exceptions: cannot create snapshot.
the error log from system.log:
{noformat}
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
files(632105 bytes)
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
Session with /10.97.9.110 is complete
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
All sessions completed
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
streaming task succeed, returning response to /10.98.194.68
INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
[Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
Starting streaming to /10.66.187.201
INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
ID#0] Beginning stream session with /10.66.187.201
INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
files(632105 bytes)
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
Session with /10.66.187.201 is complete
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
All sessions completed
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
streaming task succeed, returning response to /10.98.194.68
ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
- [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
/10.98.194.68, /10.66.187.201, /10.226.218.135 on range
(12817179804668051873746972069086
2638799,12863540308359254031520865977436165] for events.[bigint0text,
bigint0boolean, bigint0int, dataset_catalog, column_categories,
bigint0double, bigint0bigint]
ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
- [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
following error
java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391357#comment-14391357
]
Stefano Ortolalani edited comment on CASSANDRA-8696 at 4/1/15 8:15 PM:
---
It seems it hit me as well.
Here too after changing the replication factor.
As a result the logs show that exception plus a repair job is still going on
(looping I presume) for the last 12 hours.
I enabled the additional logs. Below an excerpt:
{noformat}
INFO [RepairJobTask:2] 2015-04-01 20:10:38,515 StreamingRepairTask.java:68 -
[streaming task #0063b860-d8ab-11e4-9931-47110801b395] Performing streaming
repair of 5 ranges with /XX.YY.81.6
INFO [RepairJobTask:1] 2015-04-01 20:10:38,515 ColumnFamilyStore.java:877 -
Enqueuing flush of rollups300: 474175 (0%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:474] 2015-04-01 20:10:38,516 Memtable.java:339 -
Writing Memtable-rollups300@1473895455(40384 serialized bytes, 1262 ops, 0%/0%
of on/off-heap limit)
INFO [MemtableFlushWriter:474] 2015-04-01 20:10:38,788 Memtable.java:378 -
Completed flushing
/data/cassandra/data/OpsCenter/rollups300-733a31d374af11e48baa6776bf484396/OpsCenter-rollups300-ka-63048-Data.db
(41977 bytes) for commitlog position ReplayPosition(segmentId=1427807498701,
position=22965644)
INFO [AntiEntropyStage:1] 2015-04-01 20:10:39,718 RepairSession.java:237 -
[repair #0063b860-d8ab-11e4-9931-47110801b395] rollups60 is fully synced (1
remaining column family to sync for this session)
INFO [AntiEntropyStage:1] 2015-04-01 20:10:42,741 RepairSession.java:171 -
[repair #0063b860-d8ab-11e4-9931-47110801b395] Received merkle tree for
rollups7200 from /XX.YY.81.3
INFO [AntiEntropyStage:1] 2015-04-01 20:10:42,765 RepairSession.java:171 -
[repair #0063b860-d8ab-11e4-9931-47110801b395] Received merkle tree for
rollups7200 from /XX.YY.81.6
INFO [AntiEntropyStage:1] 2015-04-01 20:10:42,767 RepairSession.java:171 -
[repair #0063b860-d8ab-11e4-9931-47110801b395] Received merkle tree for
rollups7200 from /XX.YY.33.6
INFO [RepairJobTask:6] 2015-04-01 20:10:42,767 Differencer.java:67 - [repair
#0063b860-d8ab-11e4-9931-47110801b395] Endpoints /XX.YY.81.3 and /XX.YY.81.6
are consistent for rollups7200
INFO [RepairJobTask:5] 2015-04-01 20:10:42,767 Differencer.java:74 - [repair
#0063b860-d8ab-11e4-9931-47110801b395] Endpoints /XX.YY.81.3 and /XX.YY.33.6
have 1 range(s) out of sync for rollups7200
INFO [RepairJobTask:4] 2015-04-01 20:10:42,767 Differencer.java:74 - [repair
#0063b860-d8ab-11e4-9931-47110801b395] Endpoints /XX.YY.81.6 and /XX.YY.33.6
have 1 range(s) out of sync for rollups7200
INFO [RepairJobTask:5] 2015-04-01 20:10:42,769 StreamingRepairTask.java:68 -
[streaming task #0063b860-d8ab-11e4-9931-47110801b395] Performing streaming
repair of 1 ranges with /XX.YY.81.3
INFO [RepairJobTask:4] 2015-04-01 20:10:42,769 StreamingRepairTask.java:68 -
[streaming task #0063b860-d8ab-11e4-9931-47110801b395] Performing streaming
repair of 1 ranges with /XX.YY.81.6
ERROR [RepairJobTask:6] 2015-04-01 20:10:53,557 RepairJob.java:145 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /XX.YY.81.6
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:349)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_67]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_67]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_67]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
ERROR [AntiEntropySessions:353] 2015-04-01 20:10:53,558 RepairSession.java:303
- [repair #0063b860-d8ab-11e4-9931-47110801b395] session completed with the
following error
java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146)
~[apache-cassandra-2.1.3.jar:2.1.3]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_67]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
INFO [AntiEntropySessions:354] 2015-04-01 20:10:53,558 RepairSession.java:260
- [repair
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301157#comment-14301157
]
Jan Karlsson edited comment on CASSANDRA-8696 at 2/2/15 11:08 AM:
--
We stumbled upon this issue aswell. I was only able to reproduce this when the
amount of data on disk was over 12G.
was (Author: jan karlsson):
I was only able to reproduce this when the amount of data on disk was over 12G.
From taking a quick glance at the code, this is caused by the snapshot process
throwing a timeout.
nodetool repair on cassandra 2.1.2 keyspaces return
java.lang.RuntimeException: Could not create snapshot
-
Key: CASSANDRA-8696
URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
throw java exceptions: cannot create snapshot.
the error log from system.log:
{noformat}
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
files(632105 bytes)
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
Session with /10.97.9.110 is complete
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
All sessions completed
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
streaming task succeed, returning response to /10.98.194.68
INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
[Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
Starting streaming to /10.66.187.201
INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
ID#0] Beginning stream session with /10.66.187.201
INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
files(632105 bytes)
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
Session with /10.66.187.201 is complete
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
All sessions completed
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
streaming task succeed, returning response to /10.98.194.68
ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
- [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
/10.98.194.68, /10.66.187.201, /10.226.218.135 on range
(12817179804668051873746972069086
2638799,12863540308359254031520865977436165] for events.[bigint0text,
bigint0boolean, bigint0int, dataset_catalog, column_categories,
bigint0double, bigint0bigint]
ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
- [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
following error
java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295881#comment-14295881
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/29/15 6:19 PM:
--
The replica nodes are not busy at all. Actually I have stopped all client
connection while trying to run a nodetool repair.
nodetool tpstats also shows that there isn't much activities going on.
{noformat}
Pool NameActive Pending Completed Blocked All
time blocked
CounterMutationStage 0 0 0 0
0
ReadStage 0 0 35441 0
0
RequestResponseStage 0 0 26940702 0
0
MutationStage 1 0 42298652 0
0
ReadRepairStage 0 0101 0
0
GossipStage 0 0 77379 0
0
CacheCleanupExecutor 0 0 0 0
0
AntiEntropyStage 0 0333 0
0
MigrationStage0 0 1664 0
0
ValidationExecutor0 0 51 0
0
CommitLogArchiver 0 0 0 0
0
MiscStage 0 0 0 0
0
MemtableFlushWriter 1 1853 0
0
MemtableReclaimMemory 0 0853 0
0
PendingRangeCalculator0 0 12 0
0
MemtablePostFlush 0 0 1365 0
0
CompactionExecutor2 9244 0
0
InternalResponseStage 0 0 1679 0
0
HintedHandoff 0 0 13 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 6067
MUTATION 8
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{noformat}
tail -30 /var/log/cassandra/system.log where you can see the log suddenly
stopped.
{noformat}
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [AntiEntropySessions:38] 2015-01-28 19:47:39,746 CassandraDaemon.java:153
- Exception in thread Thread[AntiEntropySessions:38,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot
creation.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.jar:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
... 3 common frames omitted
WARN [HintedHandoff:2] 2015-01-28 19:47:39,749 SliceQueryFilter.java:236 -
Read 0 live and 4525 tombstoned cells in system.hints (see
tombstone_warn_threshold). 128 columns was requested, slices=[-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
INFO
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14297272#comment-14297272
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/29/15 6:21 PM:
--
It's getting worse today after having the failed to create snapshot error. 4
nodes of 6 are showing UN in nodetool status. I checked the system.log and
syslog and couldn't find anything that tells why the node was down. The java
cassandra process is also gone.
I tried to restart cassandra service on one node and watch system.log. Shortly
after the service start running, cassandra starts to throw failed to create
snapshot error for a while before it hung with no log output, nodetool status
show error:error: No nodes present in the cluster. Has this node finished
starting up?. However the cassandra java process is still running in the
system.
{noformat}
ps -ef | grep cass
110 18634 1 19 18:01 ?00:01:39 java -ea
-javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M -Xmn400M
-XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
-XX:+UseParNewGC -XX
:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX
:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1422554504.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=719
9 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
-Dlogback.configurationFile=logback.xml -Dca
ssandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
-Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
/etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cas
sandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.ja
r:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar
:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassan
dra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassan
dra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cass
andra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/u
sr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
-XX:HeapDumpPath=/var/lib/cassandra/java_1422554504.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1422554504.log
org.apache.cassandra.service.CassandraDaemon
root 19317 19074 0 18:10 pts/100:00:00 grep --color=auto cass
#:/home/jliu# less /var/log/cassandra/system.log
{noformat}
{noformat}
#:/home/jliu# nodetool status
error: No nodes present in the cluster. Has this node finished starting up?
-- StackTrace --
java.lang.RuntimeException: No nodes present in the cluster. Has this node
finished starting up?
at
org.apache.cassandra.dht.RandomPartitioner.describeOwnership(RandomPartitioner.java:143)
at
org.apache.cassandra.service.StorageService.getOwnership(StorageService.java:3702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295881#comment-14295881
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/29/15 6:20 PM:
--
The replica nodes are not busy at all. Actually I have stopped all client
connection while trying to run a nodetool repair.
nodetool tpstats also shows that there isn't much activities going on.
{noformat}
Pool NameActive Pending Completed Blocked All
time blocked
CounterMutationStage 0 0 0 0
0
ReadStage 0 0 35441 0
0
RequestResponseStage 0 0 26940702 0
0
MutationStage 1 0 42298652 0
0
ReadRepairStage 0 0101 0
0
GossipStage 0 0 77379 0
0
CacheCleanupExecutor 0 0 0 0
0
AntiEntropyStage 0 0333 0
0
MigrationStage0 0 1664 0
0
ValidationExecutor0 0 51 0
0
CommitLogArchiver 0 0 0 0
0
MiscStage 0 0 0 0
0
MemtableFlushWriter 1 1853 0
0
MemtableReclaimMemory 0 0853 0
0
PendingRangeCalculator0 0 12 0
0
MemtablePostFlush 0 0 1365 0
0
CompactionExecutor2 9244 0
0
InternalResponseStage 0 0 1679 0
0
HintedHandoff 0 0 13 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 6067
MUTATION 8
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{noformat}
was (Author: jeffl):
The replica nodes are not busy at all. Actually I have stopped all client
connection while trying to run a nodetool repair.
nodetool tpstats also shows that there isn't much activities going on.
{noformat}
Pool NameActive Pending Completed Blocked All
time blocked
CounterMutationStage 0 0 0 0
0
ReadStage 0 0 35441 0
0
RequestResponseStage 0 0 26940702 0
0
MutationStage 1 0 42298652 0
0
ReadRepairStage 0 0101 0
0
GossipStage 0 0 77379 0
0
CacheCleanupExecutor 0 0 0 0
0
AntiEntropyStage 0 0333 0
0
MigrationStage0 0 1664 0
0
ValidationExecutor0 0 51 0
0
CommitLogArchiver 0 0 0 0
0
MiscStage 0 0 0 0
0
MemtableFlushWriter 1 1853 0
0
MemtableReclaimMemory 0 0853 0
0
PendingRangeCalculator0 0 12 0
0
MemtablePostFlush 0 0 1365 0
0
CompactionExecutor2 9244 0
0
InternalResponseStage 0 0 1679 0
0
HintedHandoff 0 0 13 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 6067
MUTATION 8
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{noformat}
