[jira] [Comment Edited] (CASSANDRA-12100) Compactions are stuck after TRUNCATE
[
https://issues.apache.org/jira/browse/CASSANDRA-12100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880149#comment-15880149
]
Stefania edited comment on CASSANDRA-12100 at 2/23/17 9:07 AM:
---
Thanks, committed to 2.2 as {{dffb1a6da8e2c3a9c08bb94bfd12130b9ddded74}} and
merged upwards with {{-s ours}}.
was (Author: stefania):
Thanks, committed to 2.2 as dffb1a6da8e2c3a9c08bb94bfd12130b9ddded74.
> Compactions are stuck after TRUNCATE
>
>
> Key: CASSANDRA-12100
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12100
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Stefano Ortolani
>Assignee: Stefania
> Fix For: 2.2.10, 3.0.9, 3.8
>
> Attachments: node3_jstack.log
>
>
> Hi,
> since the upgrade to C* 3.0.7 I see compaction tasks getting stuck when
> truncating the column family. I verified this on all nodes of the cluster.
> Pending compactions seem to disappear after restarting the node.
> {noformat}
> root@node10:~# nodetool -h localhost compactionstats
> pending tasks: 6
> id compaction type
> keyspacetable completed totalunit progress
>24e1ad30-3cac-11e6-870d-5de740693258Compaction
> schema table_1 0 57558382 bytes 0.00%
>2be2e3b0-3cac-11e6-870d-5de740693258Compaction
> schema table_2 0 65063705 bytes 0.00%
>54de38f0-3cac-11e6-870d-5de740693258Compaction
> schema table_3 0 187031 bytes 0.00%
>31926ce0-3cac-11e6-870d-5de740693258Compaction
> schema table_4 0 42951119 bytes 0.00%
>3911ad00-3cac-11e6-870d-5de740693258Compaction
> schema table_5 0 25918949 bytes 0.00%
>3e6a8ab0-3cac-11e6-870d-5de740693258Compaction
> schema table_6 0 65466210 bytes 0.00%
> Active compaction remaining time : 0h00m15s
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (CASSANDRA-12100) Compactions are stuck after TRUNCATE
[
https://issues.apache.org/jira/browse/CASSANDRA-12100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832261#comment-15832261
]
Sotirios Delimanolis edited comment on CASSANDRA-12100 at 1/20/17 7:13 PM:
---
Any chance you can backport this to 2.2.9? (Referring to [this
commit|https://github.com/apache/cassandra/commit/05483a962c64c350315fc738c697980b22361cc3].)
was (Author: s_delima):
Any chance you can backport this to 2.2.9?
> Compactions are stuck after TRUNCATE
>
>
> Key: CASSANDRA-12100
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12100
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Stefano Ortolani
>Assignee: Stefania
> Fix For: 3.0.9, 3.8
>
> Attachments: node3_jstack.log
>
>
> Hi,
> since the upgrade to C* 3.0.7 I see compaction tasks getting stuck when
> truncating the column family. I verified this on all nodes of the cluster.
> Pending compactions seem to disappear after restarting the node.
> {noformat}
> root@node10:~# nodetool -h localhost compactionstats
> pending tasks: 6
> id compaction type
> keyspacetable completed totalunit progress
>24e1ad30-3cac-11e6-870d-5de740693258Compaction
> schema table_1 0 57558382 bytes 0.00%
>2be2e3b0-3cac-11e6-870d-5de740693258Compaction
> schema table_2 0 65063705 bytes 0.00%
>54de38f0-3cac-11e6-870d-5de740693258Compaction
> schema table_3 0 187031 bytes 0.00%
>31926ce0-3cac-11e6-870d-5de740693258Compaction
> schema table_4 0 42951119 bytes 0.00%
>3911ad00-3cac-11e6-870d-5de740693258Compaction
> schema table_5 0 25918949 bytes 0.00%
>3e6a8ab0-3cac-11e6-870d-5de740693258Compaction
> schema table_6 0 65466210 bytes 0.00%
> Active compaction remaining time : 0h00m15s
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12100) Compactions are stuck after TRUNCATE
[
https://issues.apache.org/jira/browse/CASSANDRA-12100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355799#comment-15355799
]
Stefano Ortolani edited comment on CASSANDRA-12100 at 6/29/16 9:24 PM:
---
Also, in the logs this time I can see the following.
{noformat}
INFO [CompactionExecutor:529] 2016-06-29 19:17:49,255
CompactionManager.java:1518 - Compaction interrupted:
Compaction@685a9350-14a3-11e6-9af3-cdff32bdcb0d(schema, table_1,
0/97492039)bytes
INFO [CompactionExecutor:529] 2016-06-29 19:17:51,050
CompactionManager.java:1518 - Compaction interrupted:
Compaction@7d2f3790-14a3-11e6-9af3-cdff32bdcb0d(schema, table_2, 0/1871440)bytes
INFO [CompactionExecutor:529] 2016-06-29 19:17:52,481
CompactionManager.java:1518 - Compaction interrupted:
Compaction@e06f37a0-10bc-11e6-bb23-6776bf484396(schema, table_3, 0/2251297)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:53,547
CompactionManager.java:1518 - Compaction interrupted:
Compaction@e1ede860-10bc-11e6-bb23-6776bf484396(schema, table_4, 0/1373193)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:54,586
CompactionManager.java:1518 - Compaction interrupted:
Compaction@8d398280-14a3-11e6-9af3-cdff32bdcb0d(schema, table_5, 0/1646136)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:57,093
CompactionManager.java:1518 - Compaction interrupted:
Compaction@432af280-29e0-11e6-80c2-e524daec2815(schema, table_6, 0/7144)bytes
{noformat}
was (Author: ostefano):
Also, in the logs this time I can see the following (no traces of the
exceptions I posted before, so they are likely to be unrelated)
{noformat}
INFO [CompactionExecutor:529] 2016-06-29 19:17:49,255
CompactionManager.java:1518 - Compaction interrupted:
Compaction@685a9350-14a3-11e6-9af3-cdff32bdcb0d(schema, table_1,
0/97492039)bytes
INFO [CompactionExecutor:529] 2016-06-29 19:17:51,050
CompactionManager.java:1518 - Compaction interrupted:
Compaction@7d2f3790-14a3-11e6-9af3-cdff32bdcb0d(schema, table_2, 0/1871440)bytes
INFO [CompactionExecutor:529] 2016-06-29 19:17:52,481
CompactionManager.java:1518 - Compaction interrupted:
Compaction@e06f37a0-10bc-11e6-bb23-6776bf484396(schema, table_3, 0/2251297)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:53,547
CompactionManager.java:1518 - Compaction interrupted:
Compaction@e1ede860-10bc-11e6-bb23-6776bf484396(schema, table_4, 0/1373193)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:54,586
CompactionManager.java:1518 - Compaction interrupted:
Compaction@8d398280-14a3-11e6-9af3-cdff32bdcb0d(schema, table_5, 0/1646136)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:57,093
CompactionManager.java:1518 - Compaction interrupted:
Compaction@432af280-29e0-11e6-80c2-e524daec2815(schema, table_6, 0/7144)bytes
{noformat}
> Compactions are stuck after TRUNCATE
>
>
> Key: CASSANDRA-12100
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12100
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Stefano Ortolani
> Fix For: 3.0.x
>
> Attachments: node3_jstack.log
>
>
> Hi,
> since the upgrade to C* 3.0.7 I see compaction tasks getting stuck when
> truncating the column family. I verified this on all nodes of the cluster.
> Pending compactions seem to disappear after restarting the node.
> {noformat}
> root@node10:~# nodetool -h localhost compactionstats
> pending tasks: 6
> id compaction type
> keyspacetable completed totalunit progress
>24e1ad30-3cac-11e6-870d-5de740693258Compaction
> schema table_1 0 57558382 bytes 0.00%
>2be2e3b0-3cac-11e6-870d-5de740693258Compaction
> schema table_2 0 65063705 bytes 0.00%
>54de38f0-3cac-11e6-870d-5de740693258Compaction
> schema table_3 0 187031 bytes 0.00%
>31926ce0-3cac-11e6-870d-5de740693258Compaction
> schema table_4 0 42951119 bytes 0.00%
>3911ad00-3cac-11e6-870d-5de740693258Compaction
> schema table_5 0 25918949 bytes 0.00%
>3e6a8ab0-3cac-11e6-870d-5de740693258Compaction
> schema table_6 0 65466210 bytes 0.00%
> Active compaction remaining time : 0h00m15s
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12100) Compactions are stuck after TRUNCATE
[
https://issues.apache.org/jira/browse/CASSANDRA-12100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355799#comment-15355799
]
Stefano Ortolani edited comment on CASSANDRA-12100 at 6/29/16 9:23 PM:
---
Also, in the logs this time I can see the following (no traces of the
exceptions I posted before, so they are likely to be unrelated)
{noformat}
INFO [CompactionExecutor:529] 2016-06-29 19:17:49,255
CompactionManager.java:1518 - Compaction interrupted:
Compaction@685a9350-14a3-11e6-9af3-cdff32bdcb0d(schema, table_1,
0/97492039)bytes
INFO [CompactionExecutor:529] 2016-06-29 19:17:51,050
CompactionManager.java:1518 - Compaction interrupted:
Compaction@7d2f3790-14a3-11e6-9af3-cdff32bdcb0d(schema, table_2, 0/1871440)bytes
INFO [CompactionExecutor:529] 2016-06-29 19:17:52,481
CompactionManager.java:1518 - Compaction interrupted:
Compaction@e06f37a0-10bc-11e6-bb23-6776bf484396(schema, table_3, 0/2251297)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:53,547
CompactionManager.java:1518 - Compaction interrupted:
Compaction@e1ede860-10bc-11e6-bb23-6776bf484396(schema, table_4, 0/1373193)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:54,586
CompactionManager.java:1518 - Compaction interrupted:
Compaction@8d398280-14a3-11e6-9af3-cdff32bdcb0d(schema, table_5, 0/1646136)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:57,093
CompactionManager.java:1518 - Compaction interrupted:
Compaction@432af280-29e0-11e6-80c2-e524daec2815(schema, table_6, 0/7144)bytes
{noformat}
was (Author: ostefano):
Also, in the logs this time I can see the following
{noformat}
INFO [CompactionExecutor:529] 2016-06-29 19:17:49,255
CompactionManager.java:1518 - Compaction interrupted:
Compaction@685a9350-14a3-11e6-9af3-cdff32bdcb0d(schema, table_1,
0/97492039)bytes
INFO [CompactionExecutor:529] 2016-06-29 19:17:51,050
CompactionManager.java:1518 - Compaction interrupted:
Compaction@7d2f3790-14a3-11e6-9af3-cdff32bdcb0d(schema, table_2, 0/1871440)bytes
INFO [CompactionExecutor:529] 2016-06-29 19:17:52,481
CompactionManager.java:1518 - Compaction interrupted:
Compaction@e06f37a0-10bc-11e6-bb23-6776bf484396(schema, table_3, 0/2251297)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:53,547
CompactionManager.java:1518 - Compaction interrupted:
Compaction@e1ede860-10bc-11e6-bb23-6776bf484396(schema, table_4, 0/1373193)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:54,586
CompactionManager.java:1518 - Compaction interrupted:
Compaction@8d398280-14a3-11e6-9af3-cdff32bdcb0d(schema, table_5, 0/1646136)bytes
INFO [CompactionExecutor:530] 2016-06-29 19:17:57,093
CompactionManager.java:1518 - Compaction interrupted:
Compaction@432af280-29e0-11e6-80c2-e524daec2815(schema, table_6, 0/7144)bytes
{noformat}
> Compactions are stuck after TRUNCATE
>
>
> Key: CASSANDRA-12100
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12100
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Stefano Ortolani
> Fix For: 3.0.x
>
> Attachments: node3_jstack.log
>
>
> Hi,
> since the upgrade to C* 3.0.7 I see compaction tasks getting stuck when
> truncating the column family. I verified this on all nodes of the cluster.
> Pending compactions seem to disappear after restarting the node.
> {noformat}
> root@node10:~# nodetool -h localhost compactionstats
> pending tasks: 6
> id compaction type
> keyspacetable completed totalunit progress
>24e1ad30-3cac-11e6-870d-5de740693258Compaction
> schema table_1 0 57558382 bytes 0.00%
>2be2e3b0-3cac-11e6-870d-5de740693258Compaction
> schema table_2 0 65063705 bytes 0.00%
>54de38f0-3cac-11e6-870d-5de740693258Compaction
> schema table_3 0 187031 bytes 0.00%
>31926ce0-3cac-11e6-870d-5de740693258Compaction
> schema table_4 0 42951119 bytes 0.00%
>3911ad00-3cac-11e6-870d-5de740693258Compaction
> schema table_5 0 25918949 bytes 0.00%
>3e6a8ab0-3cac-11e6-870d-5de740693258Compaction
> schema table_6 0 65466210 bytes 0.00%
> Active compaction remaining time : 0h00m15s
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12100) Compactions are stuck after TRUNCATE
[
https://issues.apache.org/jira/browse/CASSANDRA-12100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352614#comment-15352614
]
Stefano Ortolani edited comment on CASSANDRA-12100 at 6/28/16 8:57 AM:
---
I see nothing weird in the logs but the following two things:
{code:title=system.log|borderStyle=solid}
INFO [SharedPool-Worker-2] 2016-06-27 22:12:06,323 Message.java:605 -
Unexpected exception during request; channel = [id: 0x6f0c2f56, /10.0.0.8:47592
:> /10.0.0.5:9042]
java.io.IOException: Error while read(...): Connection reset by peer
at io.netty.channel.epoll.Native.readAddress(Native Method)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
INFO [CompactionExecutor:3] 2016-06-27 22:45:47,213 AutoSavingCache.java:386 -
Saved CounterCache (1 items) in 169 ms
INFO [CompactionExecutor:4] 2016-06-27 22:45:47,713 AutoSavingCache.java:386 -
Saved KeyCache (15158 items) in 671 ms
INFO [IndexSummaryManager:1] 2016-06-27 22:45:58,397
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2016-06-27 23:45:58,440
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [SharedPool-Worker-17] 2016-06-28 00:33:57,301 Message.java:605 -
Unexpected exception during request; channel = [id: 0x52e03a04, /10.0.0.8:53218
:> /10.0.0.5:9042]
java.io.IOException: Error while read(...): Connection reset by peer
at io.netty.channel.epoll.Native.readAddress(Native Method)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
INFO [SharedPool-Worker-4] 2016-06-28 00:33:57,687 Message.java:605 -
Unexpected exception during request; channel = [id: 0x21eeb0b8, /10.0.0.8:53251
:> /10.0.0.5:9042]
{code}
{code:title=debug.log|borderStyle=solid}
DEBUG [GossipStage:1] 2016-06-27 22:12:06,112 FailureDetector.java:456 -
Ignoring interval time of 2713433121 for /10.0.0.7
INFO [SharedPool-Worker-2] 2016-06-27 22:12:06,323 Message.java:605 -
Unexpected exception during request; channel = [id: 0x6f0c2f56, /10.0.0.8:47592
:> /10.0.0.5:9042]
{code}
was (Author: ostefano):
I see nothing weird in the logs but the following two things:
system.log
{noformat}
INFO [SharedPool-Worker-2] 2016-06-27 22:12:06,323 Message.java:605 -
Unexpected exception during request; channel = [id: 0x6f0c2f56, /10.0.0.8:47592
:> /10.0.0.5:9042]
java.io.IOException: Error while read(...): Connection reset by peer
at io.netty.channel.epoll.Native.readAddress(Native Method)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.epoll.EpollEventLoop.run(E
