[jira] [Comment Edited] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958601#comment-16958601 ] Zephyr Guo edited comment on CASSANDRA-15295 at 10/24/19 7:04 AM: -- LGTM (y) It gets rid of the potential for the bug to be reintroduced completely. Thanks for your work [~djoshi] . was (Author: gzh1992n): LGTM (y) It gets rid of the potential for the bug to be reintroduced completely. Thanks for your work. > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: image.png, jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958601#comment-16958601 ] Zephyr Guo commented on CASSANDRA-15295: LGTM (y) It gets rid of the potential for the bug to be reintroduced completely. Thanks for your work. > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: image.png, jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948268#comment-16948268 ] Zephyr Guo edited comment on CASSANDRA-15295 at 10/10/19 7:49 AM: -- Hi, [~djoshi] thanks for the review. I agree with most of what you said. Your change is very minimal for this issue. But something else I have to remind you. 1. Most of the changes in my patch are to build a UT to check the exception case for CommitLog. It's worth it for Cassandra. Not only we have to fix this problem but we also need to understand the root cause of the problem (lack of exception tests). 2. Your change introduces the risk of starting twice. CommitLog was designed as a singleton and It manages lifecycle by itself. When other modules call CommitLog.instance, they expect an initialized CommitLog. You change the original initialization process. 3. The major change (move to a different class) is very simple. The change DOES NOT change any original initialization process. 4. I agree with that "I think it is important to get the correctness issue resolved first". Don't you think that moving the code to another class is the easiest? I respect your decision to incorporate my patch to get a better one. What's next? was (Author: gzh1992n): Hi, [~djoshi] thanks for the review. I agree with most of what you said. Your change is very minimal for this issue. But something else I have to remind you. 1. Most of the changes in my patch are to build a UT to check the exception case for CommitLog. It's worth it for Cassandra. Not only we have to fix this problem but we also need to understand the root cause of the problem (lack of exception tests). 2. Your change introduces the risk of starting twice. CommitLog was designed as a singleton and It manages lifecycle by itself. When other modules call CommitLog.instance, they expect an initialized CommitLog. You change the original initialization process. 3. The major change (move to a different class) is very simple. The change DOES NOT change any original initialization process. 4. I agree with that "I think it is important to get the correctness issue resolved first". Don't you think that moving the code to another class is the easiest? I respect your decision, incorporate my patch to get a better one. What's next? > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948268#comment-16948268 ] Zephyr Guo edited comment on CASSANDRA-15295 at 10/10/19 7:20 AM: -- Hi, [~djoshi] thanks for the review. I agree with most of what you said. Your change is very minimal for this issue. But something else I have to remind you. 1. Most of the changes in my patch are to build a UT to check the exception case for CommitLog. It's worth it for Cassandra. Not only we have to fix this problem but we also need to understand the root cause of the problem (lack of exception tests). 2. Your change introduces the risk of starting twice. CommitLog was designed as a singleton and It manages lifecycle by itself. When other modules call CommitLog.instance, they expect an initialized CommitLog. You change the original initialization process. 3. The major change (move to a different class) is very simple. The change DOES NOT change any original initialization process. 4. I agree with that "I think it is important to get the correctness issue resolved first". Don't you think that moving the code to another class is the easiest? I respect your decision, incorporate my patch to get a better one. What's next? was (Author: gzh1992n): Hi, [~djoshi] thanks for the review. I agree with most of what you said. Your change is very minimal for this issue. But something else I have to remind you. 1. Most of the changes in my patch are to build a UT to check the exception case for CommitLog. It's worth it for Cassandra. Not only we have to fix this problem but we also need to understand the root cause of the problem (lack of exception tests). 2. Your change introduces the risk of starting twice. CommitLog was designed to a singleton and It manages lifecycle by itself. When other modules call CommitLog.instance, they expect an initialized CommitLog. You change the original initialization process. 3. The major change (move to a different class) is very simple. The change DOES NOT change any original initialization process. 4. I agree with that "I think it is important to get the correctness issue resolved first". Don't you think that moving the code to another class is the easiest? I respect your decision, incorporate my patch to get a better one. What's next? > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948268#comment-16948268 ] Zephyr Guo edited comment on CASSANDRA-15295 at 10/10/19 7:18 AM: -- Hi, [~djoshi] thanks for the review. I agree with most of what you said. Your change is very minimal for this issue. But something else I have to remind you. 1. Most of the changes in my patch are to build a UT to check the exception case for CommitLog. It's worth it for Cassandra. Not only we have to fix this problem but we also need to understand the root cause of the problem (lack of exception tests). 2. Your change introduces the risk of starting twice. CommitLog was designed to a singleton and It manages lifecycle by itself. When other modules call CommitLog.instance, they expect an initialized CommitLog. You change the original initialization process. 3. The major change (move to a different class) is very simple. The change DOES NOT change any original initialization process. 4. I agree with that "I think it is important to get the correctness issue resolved first". Don't you think that moving the code to another class is the easiest? I respect your decision, incorporate my patch to get a better one. What's next? was (Author: gzh1992n): Hi, [~djoshi] thanks for the review. I agree with most of what you said. Your change is very minimal for this issue. But something else I have to remind you. 1. Most of the changes in my patch are going to build a UT that ensures the exception case for CommitLog. It's worth it for Cassandra. Not only we have to fix this problem but we also need to understand the root cause of the problem (lack of exception tests). 2. Your change introduces the risk of starting twice. CommitLog was designed to a singleton and It manages lifecycle by itself. When other modules call CommitLog.instance, they expect an initialized CommitLog. You change the original initialization process. 3. The major change (move to a different class) is very simple. The change DOES NOT change any original initialization process. 4. I agree with that "I think it is important to get the correctness issue resolved first". Don't you think that moving the code to another class is the easiest? I respect your decision, incorporate my patch to get a better one. What's next? > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948268#comment-16948268 ] Zephyr Guo commented on CASSANDRA-15295: Hi, [~djoshi] thanks for the review. I agree with most of what you said. Your change is very minimal for this issue. But something else I have to remind you. 1. Most of the changes in my patch are going to build a UT that ensures the exception case for CommitLog. It's worth it for Cassandra. Not only we have to fix this problem but we also need to understand the root cause of the problem (lack of exception tests). 2. Your change introduces the risk of starting twice. CommitLog was designed to a singleton and It manages lifecycle by itself. When other modules call CommitLog.instance, they expect an initialized CommitLog. You change the original initialization process. 3. The major change (move to a different class) is very simple. The change DOES NOT change any original initialization process. 4. I agree with that "I think it is important to get the correctness issue resolved first". Don't you think that moving the code to another class is the easiest? I respect your decision, incorporate my patch to get a better one. What's next? > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936351#comment-16936351 ] Zephyr Guo commented on CASSANDRA-15295: I am sorry to reply you so late [~jrwest]. I had updated the PR. Please see: https://github.com/apache/cassandra/pull/347 > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932337#comment-16932337 ] Zephyr Guo commented on CASSANDRA-15295: Thanks for the comments [~jrwest]. I will update the patch as soon as possible. > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924245#comment-16924245 ] Zephyr Guo commented on CASSANDRA-15295: Thanks for review [~jrwest]. > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921193#comment-16921193 ] Zephyr Guo commented on CASSANDRA-15295: Thank you [~jjirsa] for your review and I look forward to your feedback. > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Description: Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time. I used jstack to saw what happened. The main thread stuck in *AbstractCommitLogSegmentManager.awaitAvailableSegment* !screenshot-1.png! The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. !screenshot-2.png! And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization. !screenshot-3.png! This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call *CommitLog.handleCommitError*(static method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog class is still initializing. was: Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time. I used jstack to saw what happened. The main thread stuck in *AbstractCommitLogSegmentManager.awaitAvailableSegment* !screenshot-1.png! The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. !screenshot-2.png! And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization. !screenshot-3.png! This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call *CommitLog.handleCommitError*(static method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog class is still initializing. > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: screenshot-3.png > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Description: Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time. I used jstack to saw what happened. The main thread stuck in *AbstractCommitLogSegmentManager.awaitAvailableSegment* !screenshot-1.png! The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. !screenshot-2.png! And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization. !screenshot-3.png! This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call *CommitLog.handleCommitError*(static method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog class is still initializing. was: Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time. I used jstack to saw what happened. The main thread stuck in *AbstractCommitLogSegmentManager.awaitAvailableSegment* !image-2019-08-30-21-40-43-748.png|thumbnail! The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. !image-2019-08-30-21-30-11-683.png|thumbnail! And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization. !image-2019-08-30-21-25-17-638.png|thumbnail! This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call *CommitLog.handleCommitError*(static method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog class is still initializing. > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: screenshot-2.png > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: (was: image-2019-08-30-21-30-11-683.png) > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: (was: image-2019-08-30-21-40-43-748.png) > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: screenshot-1.png > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: (was: image-2019-08-30-21-29-48-623.png) > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: image-2019-08-30-21-40-43-748.png, jstack.log, pstack.log > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: (was: image-2019-08-30-21-25-17-638.png) > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: image-2019-08-30-21-40-43-748.png, jstack.log, pstack.log > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: pstack.log > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: image-2019-08-30-21-25-17-638.png, > image-2019-08-30-21-29-48-623.png, image-2019-08-30-21-30-11-683.png, > image-2019-08-30-21-40-43-748.png, jstack.log, pstack.log > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15295: --- Attachment: jstack.log > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: image-2019-08-30-21-25-17-638.png, > image-2019-08-30-21-29-48-623.png, image-2019-08-30-21-30-11-683.png, > image-2019-08-30-21-40-43-748.png, jstack.log > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !image-2019-08-30-21-40-43-748.png|thumbnail! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !image-2019-08-30-21-30-11-683.png|thumbnail! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !image-2019-08-30-21-25-17-638.png|thumbnail! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
Zephyr Guo created CASSANDRA-15295: -- Summary: Running into deadlock when do CommitLog initialization Key: CASSANDRA-15295 URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 Project: Cassandra Issue Type: Bug Components: Local/Commit Log Reporter: Zephyr Guo Assignee: Zephyr Guo Attachments: image-2019-08-30-21-25-17-638.png, image-2019-08-30-21-29-48-623.png, image-2019-08-30-21-30-11-683.png, image-2019-08-30-21-40-43-748.png Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time. I used jstack to saw what happened. The main thread stuck in *AbstractCommitLogSegmentManager.awaitAvailableSegment* !image-2019-08-30-21-40-43-748.png|thumbnail! The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. !image-2019-08-30-21-30-11-683.png|thumbnail! And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization. !image-2019-08-30-21-25-17-638.png|thumbnail! This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call *CommitLog.handleCommitError*(static method). COMMIT-LOG-ALLOCATOR will stick on this line because CommitLog class is still initializing. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15198) Preventing RuntimeException when the username or password is empty
[ https://issues.apache.org/jira/browse/CASSANDRA-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879104#comment-16879104 ] Zephyr Guo commented on CASSANDRA-15198: Thanks [~bdeggleston] for review. > Preventing RuntimeException when the username or password is empty > -- > > Key: CASSANDRA-15198 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15198 > Project: Cassandra > Issue Type: Bug > Components: Feature/Authorization >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 3.0.x, 3.11.x > > Attachments: CASSANDRA-15198-v1.patch, empty_username_error.jpg > > Time Spent: 10m > Remaining Estimate: 0h > > !empty_username_error.jpg! > Although this does not affect the service, it's necessary to improve code > robustness. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15198) Preventing RuntimeException when the username or password is empty
[ https://issues.apache.org/jira/browse/CASSANDRA-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15198: --- Description: !empty_username_error.jpg! Although this does not affect the service, it's necessary to improve code robustness. was: !empty_username_error.jpg! Although this does not affect the service, but it's necessary to improve code robustness. > Preventing RuntimeException when the username or password is empty > -- > > Key: CASSANDRA-15198 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15198 > Project: Cassandra > Issue Type: Bug > Components: Feature/Authorization >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Labels: pull-request-available > Attachments: CASSANDRA-15198-v1.patch, empty_username_error.jpg > > Time Spent: 10m > Remaining Estimate: 0h > > !empty_username_error.jpg! > Although this does not affect the service, it's necessary to improve code > robustness. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15198) Preventing RuntimeException when the username or password is empty
[ https://issues.apache.org/jira/browse/CASSANDRA-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15198: --- Severity: Low Complexity: Low Hanging Fruit Discovered By: User Report Bug Category: Parent values: Security(12985)Level 1 values: Remote Code Execution(13002) Component/s: Feature/Authorization Status: Open (was: Triage Needed) > Preventing RuntimeException when the username or password is empty > -- > > Key: CASSANDRA-15198 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15198 > Project: Cassandra > Issue Type: Bug > Components: Feature/Authorization >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: CASSANDRA-15198-v1.patch, empty_username_error.jpg > > > !empty_username_error.jpg! > Although this does not affect the service, but it's necessary to improve code > robustness. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15198) Preventing RuntimeException when the username or password is empty
[ https://issues.apache.org/jira/browse/CASSANDRA-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15198: --- Attachment: CASSANDRA-15198-v1.patch > Preventing RuntimeException when the username or password is empty > -- > > Key: CASSANDRA-15198 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15198 > Project: Cassandra > Issue Type: Bug >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: CASSANDRA-15198-v1.patch, empty_username_error.jpg > > > !empty_username_error.jpg! > Although this does not affect the service, but it's necessary to improve code > robustness. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15198) Preventing RuntimeException when the username or password is empty
Zephyr Guo created CASSANDRA-15198: -- Summary: Preventing RuntimeException when the username or password is empty Key: CASSANDRA-15198 URL: https://issues.apache.org/jira/browse/CASSANDRA-15198 Project: Cassandra Issue Type: Bug Reporter: Zephyr Guo Assignee: Zephyr Guo Attachments: empty_username_error.jpg !empty_username_error.jpg! Although this does not affect the service, but it's necessary to improve code robustness. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15090) Customize cassandra log directory
[ https://issues.apache.org/jira/browse/CASSANDRA-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15090: --- Test and Documentation Plan: Documentation could be found in annotation. Status: Patch Available (was: Open) > Customize cassandra log directory > - > > Key: CASSANDRA-15090 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15090 > Project: Cassandra > Issue Type: New Feature > Components: Local/Config >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: CASSANDRA-15090-v1.patch > > > Add a new variable CASSANDRA_LOG_DIR (default: $CASSANDRA_HOME/logs) so that > we could customize log directory such as ‘/var/log/cassandra’ . > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15090) Customize cassandra log directory
[ https://issues.apache.org/jira/browse/CASSANDRA-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15090: --- Complexity: Low Hanging Fruit Change Category: Operability Component/s: Local/Config Status: Open (was: Triage Needed) > Customize cassandra log directory > - > > Key: CASSANDRA-15090 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15090 > Project: Cassandra > Issue Type: New Feature > Components: Local/Config >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: CASSANDRA-15090-v1.patch > > > Add a new variable CASSANDRA_LOG_DIR (default: $CASSANDRA_HOME/logs) so that > we could customize log directory such as ‘/var/log/cassandra’ . > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15090) Customize cassandra log directory
[
https://issues.apache.org/jira/browse/CASSANDRA-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836409#comment-16836409
]
Zephyr Guo edited comment on CASSANDRA-15090 at 5/10/19 4:00 AM:
-
Thank you for your review. [~michaelsembwever]
1.Take a look on startup command, we put cassandra_parms after JVM_OPTS, so
cassandra.logdir would override by cassandra_parms. I mean that we cannot set
cassandra.logdir by adding it to jvm.options.
{code:sh}
exec $NUMACTL "$JAVA" $JVM_OPTS $cassandra_parms -cp "$CLASSPATH" $props
"$class"
{code}
2.We could put CASSANDRA_CONF CASSANDRA_LOG_DIR to /etc/profile.d, so that
others(script, daemon, tool) could find cassandra log dir conveniently such as
collectting log events agent.
3.All of cassandra‘s tools could take advantage of CASSANDRA_LOG_DIR,
outputting log of tool to uniform place if necessary.
was (Author: gzh1992n):
Thank you for your review. [~michaelsembwever]
1.Take a look on Startup command, we put cassandra_parms after JVM_OPTS, so
cassandra.logdir would override by cassandra_parms. I mean that we cannot set
cassandra.logdir by adding it to jvm.options.
{code:sh}
exec $NUMACTL "$JAVA" $JVM_OPTS $cassandra_parms -cp "$CLASSPATH" $props
"$class"
{code}
2.We could put CASSANDRA_CONF CASSANDRA_LOG_DIR to /etc/profile.d, so that
others(script, daemon, tool) could find cassandra log dir conveniently such as
collectting log events agent.
3.All of cassandra‘s tools could take advantage of CASSANDRA_LOG_DIR,
outputting log of tool to uniform place if necessary.
> Customize cassandra log directory
> -
>
> Key: CASSANDRA-15090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15090
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Normal
> Attachments: CASSANDRA-15090-v1.patch
>
>
> Add a new variable CASSANDRA_LOG_DIR (default: $CASSANDRA_HOME/logs) so that
> we could customize log directory such as ‘/var/log/cassandra’ .
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15090) Customize cassandra log directory
[
https://issues.apache.org/jira/browse/CASSANDRA-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836409#comment-16836409
]
Zephyr Guo edited comment on CASSANDRA-15090 at 5/9/19 2:03 PM:
Thank you for your review. [~michaelsembwever]
1.Take a look on Startup command, we put cassandra_parms after JVM_OPTS, so
cassandra.logdir would override by cassandra_parms. I mean that we cannot set
cassandra.logdir by adding it to jvm.options.
{code:sh}
exec $NUMACTL "$JAVA" $JVM_OPTS $cassandra_parms -cp "$CLASSPATH" $props
"$class"
{code}
2.We could put CASSANDRA_CONF CASSANDRA_LOG_DIR to /etc/profile.d, so that
others(script, daemon, tool) could find cassandra log dir conveniently such as
collectting log events agent.
3.All of cassandra‘s tools could take advantage of CASSANDRA_LOG_DIR,
outputting log of tool to uniform place if necessary.
was (Author: gzh1992n):
Thank you for your review. [~michaelsembwever]
1.Take a look on Startup command, we put cassandra_parms after JVM_OPTS, so
cassandra.logdir would override by cassandra_parms. I mean that we cannot set
cassandra.logdir by adding it to jvm.options.
{code: sh}
exec $NUMACTL "$JAVA" $JVM_OPTS $cassandra_parms -cp "$CLASSPATH" $props
"$class"
{code}
2.We could put CASSANDRA_CONF CASSANDRA_LOG_DIR to /etc/profile.d, so that
others(script, daemon, tool) could find cassandra log dir conveniently such as
collectting log events agent.
3.All of cassandra‘s tools could take advantage of CASSANDRA_LOG_DIR,
outputting log of tool to uniform place if necessary.
> Customize cassandra log directory
> -
>
> Key: CASSANDRA-15090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15090
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Normal
> Attachments: CASSANDRA-15090-v1.patch
>
>
> Add a new variable CASSANDRA_LOG_DIR (default: $CASSANDRA_HOME/logs) so that
> we could customize log directory such as ‘/var/log/cassandra’ .
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15090) Customize cassandra log directory
[
https://issues.apache.org/jira/browse/CASSANDRA-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836409#comment-16836409
]
Zephyr Guo edited comment on CASSANDRA-15090 at 5/9/19 2:02 PM:
Thank you for your review. [~michaelsembwever]
1.Take a look on Startup command, we put cassandra_parms after JVM_OPTS, so
cassandra.logdir would override by cassandra_parms. I mean that we cannot set
cassandra.logdir by adding it to jvm.options.
{code: sh}
exec $NUMACTL "$JAVA" $JVM_OPTS $cassandra_parms -cp "$CLASSPATH" $props
"$class"
{code}
2.We could put CASSANDRA_CONF CASSANDRA_LOG_DIR to /etc/profile.d, so that
others(script, daemon, tool) could find cassandra log dir conveniently such as
collectting log events agent.
3.All of cassandra‘s tools could take advantage of CASSANDRA_LOG_DIR,
outputting log of tool to uniform place if necessary.
was (Author: gzh1992n):
Thank you for your review. [~michaelsembwever]
1.Take a look on Startup command, we put cassandra_parms after JVM_OPTS, so
cassandra.logdir would override by cassandra_parms. I mean that we cannot set
cassandra.logdir by adding it to jvm.options.
{code:shell}
exec $NUMACTL "$JAVA" $JVM_OPTS $cassandra_parms -cp "$CLASSPATH" $props
"$class"
{code}
2.We could put CASSANDRA_CONF CASSANDRA_LOG_DIR to /etc/profile.d, so that
others(script, daemon, tool) could find cassandra log dir conveniently such as
collectting log events agent.
3.All of cassandra‘s tools could take advantage of CASSANDRA_LOG_DIR,
outputting log of tool to uniform place if necessary.
> Customize cassandra log directory
> -
>
> Key: CASSANDRA-15090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15090
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Normal
> Attachments: CASSANDRA-15090-v1.patch
>
>
> Add a new variable CASSANDRA_LOG_DIR (default: $CASSANDRA_HOME/logs) so that
> we could customize log directory such as ‘/var/log/cassandra’ .
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15090) Customize cassandra log directory
[
https://issues.apache.org/jira/browse/CASSANDRA-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836409#comment-16836409
]
Zephyr Guo commented on CASSANDRA-15090:
Thank you for your review. [~michaelsembwever]
1.Take a look on Startup command, we put cassandra_parms after JVM_OPTS, so
cassandra.logdir would override by cassandra_parms. I mean that we cannot set
cassandra.logdir by adding it to jvm.options.
{code:shell}
exec $NUMACTL "$JAVA" $JVM_OPTS $cassandra_parms -cp "$CLASSPATH" $props
"$class"
{code}
2.We could put CASSANDRA_CONF CASSANDRA_LOG_DIR to /etc/profile.d, so that
others(script, daemon, tool) could find cassandra log dir conveniently such as
collectting log events agent.
3.All of cassandra‘s tools could take advantage of CASSANDRA_LOG_DIR,
outputting log of tool to uniform place if necessary.
> Customize cassandra log directory
> -
>
> Key: CASSANDRA-15090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15090
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Normal
> Attachments: CASSANDRA-15090-v1.patch
>
>
> Add a new variable CASSANDRA_LOG_DIR (default: $CASSANDRA_HOME/logs) so that
> we could customize log directory such as ‘/var/log/cassandra’ .
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15090) Customize cassandra log directory
[ https://issues.apache.org/jira/browse/CASSANDRA-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated CASSANDRA-15090: --- Attachment: CASSANDRA-15090-v1.patch > Customize cassandra log directory > - > > Key: CASSANDRA-15090 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15090 > Project: Cassandra > Issue Type: New Feature >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: CASSANDRA-15090-v1.patch > > > Add a new variable CASSANDRA_LOG_DIR (default: $CASSANDRA_HOME/logs) so that > we could customize log directory such as ‘/var/log/cassandra’ . > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15090) Customize cassandra log directory
Zephyr Guo created CASSANDRA-15090: -- Summary: Customize cassandra log directory Key: CASSANDRA-15090 URL: https://issues.apache.org/jira/browse/CASSANDRA-15090 Project: Cassandra Issue Type: New Feature Reporter: Zephyr Guo Assignee: Zephyr Guo Add a new variable CASSANDRA_LOG_DIR (default: $CASSANDRA_HOME/logs) so that we could customize log directory such as ‘/var/log/cassandra’ . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
