[
https://issues.apache.org/jira/browse/CASSANDRA-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578124#comment-13578124
]
Michael Kjellman edited comment on CASSANDRA-5129 at 2/14/13 3:39 AM:
----------------------------------------------------------------------
it appears this is related to secondary indexes. after the bootstrapping node
finishes streaming it submits an index build. This gets submitted but never
makes any progress and hangs indefinitely.
{code}
INFO [Thread-382] 2013-02-13 18:02:57,205 StreamInSession.java (line 199)
Finished streaming session 4ae0be23-75fb-11e2-ba65-8f73c0b9d93d from
/10.138.12.10
INFO [Thread-540] 2013-02-13 18:17:42,526 SecondaryIndexManager.java (line
137) Submitting index build of [domain_metadata.classificationIdx,
domain_metadata.domaintypeIdx] for data in
SSTableReader(path='/data/cassandra/brts/domain_metadata/brts-domain_metadata-ib-1-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-2-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-3-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-4-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-5-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-6-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-7-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-8-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-9-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-10-Data.db')
{code}
{code}
#nodetool compactionstats
pending tasks: 23
Active compaction remaining time : n/a
{code}
also when C* is killed, the node hung with nothing streaming logs:
{code}
ERROR 19:37:42,274 Exception in thread Thread[Streaming to
/10.138.12.11:1,5,main]
java.lang.RuntimeException: java.io.EOFException
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:101)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 3 more
{code}
was (Author: mkjellman):
it appears this is related to secondary indexes. after the bootstrapping
node finishes streaming it submits an index build. This gets submitted but
never makes any progress and hangs indefinitely.
{code}
INFO [Thread-382] 2013-02-13 18:02:57,205 StreamInSession.java (line 199)
Finished streaming session 4ae0be23-75fb-11e2-ba65-8f73c0b9d93d from
/10.138.12.10
INFO [Thread-540] 2013-02-13 18:17:42,526 SecondaryIndexManager.java (line
137) Submitting index build of [domain_metadata.classificationIdx,
domain_metadata.domaintypeIdx] for data in
SSTableReader(path='/data/cassandra/brts/domain_metadata/brts-domain_metadata-ib-1-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-2-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-3-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-4-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-5-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-6-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-7-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-8-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-9-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-10-Data.db')
{code}
{code}
#nodetool compactionstats
pending tasks: 23
Active compaction remaining time : n/a
{code}
> newly bootstrapping nodes hang indefinitely in STATUS:BOOT while JOINING
> cluster
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-5129
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5129
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0
> Environment: Ubuntu 12.04
> Reporter: Michael Kjellman
> Assignee: Yuki Morishita
>
> bootstrapping a new node causes it to hang indefinitely in STATUS:BOOT
> Nodes streaming to the new node report
> {code}
> Mode: NORMAL
> Nothing streaming to /10.8.30.16
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 0 1843990
> Responses n/a 2 661750
> {code}
> the node being streamed to stuck in the JOINING state reports:
> {code}
> Mode: JOINING
> Not sending any streams.
> Nothing streaming from /10.8.30.103
> Nothing streaming from /10.8.30.102
> Pool Name Active Pending Completed
> Commands n/a 0 10
> Responses n/a 0 613577
> {code}
> it appears that the nodes in the "nothing streaming" state never sends a
> "finished streaming" to the joining node.
> no exceptions are thrown during the streaming on either node while the node
> is in this state.
> {code:name="full gossip state of bootstrapping node"}
> /10.8.30.16
> NET_VERSION:6
> RELEASE_VERSION:1.2.0
> STATUS:BOOT,127605887595351923798765477786913079289
> RACK:RAC1
> RPC_ADDRESS:0.0.0.0
> DC:DC1
> SCHEMA:5cd8420d-ce3c-3625-8293-67558a24816b
> HOST_ID:e20817ce-7454-4dc4-a1c6-b1dec35c4491
> LOAD:1.11824041581E11
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
