[jira] Commented: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12922539#action_12922539 ] Thijs Terlouw commented on ZOOKEEPER-893: - Thanks Flavio! I have been too busy to add a testcase and yours looks great! ZooKeeper high cpu usage when invalid requests -- Key: ZOOKEEPER-893 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 Project: Zookeeper Issue Type: Bug Components: server Affects Versions: 3.3.1 Environment: Linux 2.6.16 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) Reporter: Thijs Terlouw Assignee: Thijs Terlouw Priority: Critical Fix For: 3.3.2, 3.4.0 Attachments: ZOOKEEPER-893.patch, ZOOKEEPER-893.patch Original Estimate: 1h Remaining Estimate: 1h When ZooKeeper receives certain illegally formed messages on the internal communication port (:4181 by default), it's possible for ZooKeeper to enter an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, but that patch does not resolve all issues. from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java the two affected parts: === int length = msgLength.getInt(); if(length = 0) { throw new IOException(Invalid packet length: + length); } === === while (message.hasRemaining()) { temp_numbytes = channel.read(message); if(temp_numbytes 0) { throw new IOException(Channel eof before end); } numbytes += temp_numbytes; } === how to replicate this bug: perform an nmap portscan against your zookeeper server: nmap -sV -n your.ip.here -p4181 wait for a while untill you see some messages in the logfile and then you will see 100% cpu usage. It does not recover from this situation. With my patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12921286#action_12921286 ] Thijs Terlouw commented on ZOOKEEPER-893: - I'll try to write a test case, but not exactly sure yet how to integrate. Need to look at some example unit tests first :) ZooKeeper high cpu usage when invalid requests -- Key: ZOOKEEPER-893 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 Project: Zookeeper Issue Type: Bug Components: server Affects Versions: 3.3.1 Environment: Linux 2.6.16 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) Reporter: Thijs Terlouw Assignee: Thijs Terlouw Priority: Critical Fix For: 3.3.2, 3.4.0 Attachments: ZOOKEEPER-893.patch Original Estimate: 1h Remaining Estimate: 1h When ZooKeeper receives certain illegally formed messages on the internal communication port (:4181 by default), it's possible for ZooKeeper to enter an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, but that patch does not resolve all issues. from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java the two affected parts: === int length = msgLength.getInt(); if(length = 0) { throw new IOException(Invalid packet length: + length); } === === while (message.hasRemaining()) { temp_numbytes = channel.read(message); if(temp_numbytes 0) { throw new IOException(Channel eof before end); } numbytes += temp_numbytes; } === how to replicate this bug: perform an nmap portscan against your zookeeper server: nmap -sV -n your.ip.here -p4181 wait for a while untill you see some messages in the logfile and then you will see 100% cpu usage. It does not recover from this situation. With my patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thijs Terlouw updated ZOOKEEPER-893: Attachment: ZOOKEEPER-893.patch ZooKeeper high cpu usage when invalid requests -- Key: ZOOKEEPER-893 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 Project: Zookeeper Issue Type: Bug Components: server Affects Versions: 3.3.1 Environment: Linux 2.6.16 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) Reporter: Thijs Terlouw Priority: Critical Attachments: ZOOKEEPER-893.patch When ZooKeeper receives certain illegally formed messages on the internal communication port (:4181 by default), it's possible for ZooKeeper to enter an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, but that patch does not resolve all issues. from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java the two affected parts: === int length = msgLength.getInt(); if(length = 0) { throw new IOException(Invalid packet length: + length); } === === while (message.hasRemaining()) { temp_numbytes = channel.read(message); if(temp_numbytes 0) { throw new IOException(Channel eof before end); } numbytes += temp_numbytes; } === how to replicate this bug: perform an nmap portscan against your zookeeper server: nmap -sV -n your.ip.here -p4181 wait for a while untill you see some messages in the logfile and then you will see 100% cpu usage. It does not recover from this situation. With my patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thijs Terlouw updated ZOOKEEPER-893: Remaining Estimate: 1h Original Estimate: 1h ZooKeeper high cpu usage when invalid requests -- Key: ZOOKEEPER-893 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 Project: Zookeeper Issue Type: Bug Components: server Affects Versions: 3.3.1 Environment: Linux 2.6.16 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) Reporter: Thijs Terlouw Priority: Critical Attachments: ZOOKEEPER-893.patch Original Estimate: 1h Remaining Estimate: 1h When ZooKeeper receives certain illegally formed messages on the internal communication port (:4181 by default), it's possible for ZooKeeper to enter an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, but that patch does not resolve all issues. from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java the two affected parts: === int length = msgLength.getInt(); if(length = 0) { throw new IOException(Invalid packet length: + length); } === === while (message.hasRemaining()) { temp_numbytes = channel.read(message); if(temp_numbytes 0) { throw new IOException(Channel eof before end); } numbytes += temp_numbytes; } === how to replicate this bug: perform an nmap portscan against your zookeeper server: nmap -sV -n your.ip.here -p4181 wait for a while untill you see some messages in the logfile and then you will see 100% cpu usage. It does not recover from this situation. With my patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thijs Terlouw updated ZOOKEEPER-893: Attachment: (was: ZOOKEEPER-893.patch) ZooKeeper high cpu usage when invalid requests -- Key: ZOOKEEPER-893 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 Project: Zookeeper Issue Type: Bug Components: server Affects Versions: 3.3.1 Environment: Linux 2.6.16 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) Reporter: Thijs Terlouw Priority: Critical Attachments: ZOOKEEPER-893.patch Original Estimate: 1h Remaining Estimate: 1h When ZooKeeper receives certain illegally formed messages on the internal communication port (:4181 by default), it's possible for ZooKeeper to enter an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, but that patch does not resolve all issues. from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java the two affected parts: === int length = msgLength.getInt(); if(length = 0) { throw new IOException(Invalid packet length: + length); } === === while (message.hasRemaining()) { temp_numbytes = channel.read(message); if(temp_numbytes 0) { throw new IOException(Channel eof before end); } numbytes += temp_numbytes; } === how to replicate this bug: perform an nmap portscan against your zookeeper server: nmap -sV -n your.ip.here -p4181 wait for a while untill you see some messages in the logfile and then you will see 100% cpu usage. It does not recover from this situation. With my patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thijs Terlouw updated ZOOKEEPER-893: Attachment: ZOOKEEPER-893.patch ZooKeeper high cpu usage when invalid requests -- Key: ZOOKEEPER-893 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 Project: Zookeeper Issue Type: Bug Components: server Affects Versions: 3.3.1 Environment: Linux 2.6.16 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) Reporter: Thijs Terlouw Priority: Critical Attachments: ZOOKEEPER-893.patch Original Estimate: 1h Remaining Estimate: 1h When ZooKeeper receives certain illegally formed messages on the internal communication port (:4181 by default), it's possible for ZooKeeper to enter an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, but that patch does not resolve all issues. from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java the two affected parts: === int length = msgLength.getInt(); if(length = 0) { throw new IOException(Invalid packet length: + length); } === === while (message.hasRemaining()) { temp_numbytes = channel.read(message); if(temp_numbytes 0) { throw new IOException(Channel eof before end); } numbytes += temp_numbytes; } === how to replicate this bug: perform an nmap portscan against your zookeeper server: nmap -sV -n your.ip.here -p4181 wait for a while untill you see some messages in the logfile and then you will see 100% cpu usage. It does not recover from this situation. With my patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.