Zephyr Guo created CASSANDRA-15295: -------------------------------------- Summary: Running into deadlock when do CommitLog initialization Key: CASSANDRA-15295 URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 Project: Cassandra Issue Type: Bug Components: Local/Commit Log Reporter: Zephyr Guo Assignee: Zephyr Guo Attachments: image-2019-08-30-21-25-17-638.png, image-2019-08-30-21-29-48-623.png, image-2019-08-30-21-30-11-683.png, image-2019-08-30-21-40-43-748.png
Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time. I used jstack to saw what happened. The main thread stuck in *AbstractCommitLogSegmentManager.awaitAvailableSegment* !image-2019-08-30-21-40-43-748.png|thumbnail! The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. !image-2019-08-30-21-30-11-683.png|thumbnail! And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization. !image-2019-08-30-21-25-17-638.png|thumbnail! This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call *CommitLog.handleCommitError*(static method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog class is still initializing. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org