[jira] Commented: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[
https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922910#action_12922910
]
Hudson commented on ZOOKEEPER-893:
--
Integrated in ZooKeeper-trunk #972 (See
[https://hudson.apache.org/hudson/job/ZooKeeper-trunk/972/])
ZOOKEEPER-893. ZooKeeper high cpu usage when invalid requests
> ZooKeeper high cpu usage when invalid requests
> --
>
> Key: ZOOKEEPER-893
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893
> Project: Zookeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.3.1
> Environment: Linux 2.6.16
> 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz
> java version "1.6.0_17"
> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode)
>Reporter: Thijs Terlouw
>Assignee: Thijs Terlouw
>Priority: Critical
> Fix For: 3.3.2, 3.4.0
>
> Attachments: ZOOKEEPER-893-3.3.patch, ZOOKEEPER-893.patch,
> ZOOKEEPER-893.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> When ZooKeeper receives certain illegally formed messages on the internal
> communication port (:4181 by default), it's possible for ZooKeeper to enter
> an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427,
> but that patch does not resolve all issues.
> from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java
> the two affected parts:
> ===
> int length = msgLength.getInt();
>
> if(length <= 0) {
>
> throw new IOException("Invalid packet length:" + length);
>
> }
> ===
> ===
> while (message.hasRemaining()) {
>
> temp_numbytes = channel.read(message);
>
> if(temp_numbytes < 0) {
>
> throw new IOException("Channel eof before end");
>
> }
>
> numbytes += temp_numbytes;
>
> }
> ===
> how to replicate this bug:
> perform an nmap portscan against your zookeeper server: "nmap -sV -n
> your.ip.here -p4181"
> wait for a while untill you see some messages in the logfile and then you
> will see 100% cpu usage. It does not recover from this situation. With my
> patch, it does not occur anymore
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[
https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922539#action_12922539
]
Thijs Terlouw commented on ZOOKEEPER-893:
-
Thanks Flavio! I have been too busy to add a testcase and yours looks great!
> ZooKeeper high cpu usage when invalid requests
> --
>
> Key: ZOOKEEPER-893
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893
> Project: Zookeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.3.1
> Environment: Linux 2.6.16
> 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz
> java version "1.6.0_17"
> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode)
>Reporter: Thijs Terlouw
>Assignee: Thijs Terlouw
>Priority: Critical
> Fix For: 3.3.2, 3.4.0
>
> Attachments: ZOOKEEPER-893.patch, ZOOKEEPER-893.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> When ZooKeeper receives certain illegally formed messages on the internal
> communication port (:4181 by default), it's possible for ZooKeeper to enter
> an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427,
> but that patch does not resolve all issues.
> from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java
> the two affected parts:
> ===
> int length = msgLength.getInt();
>
> if(length <= 0) {
>
> throw new IOException("Invalid packet length:" + length);
>
> }
> ===
> ===
> while (message.hasRemaining()) {
>
> temp_numbytes = channel.read(message);
>
> if(temp_numbytes < 0) {
>
> throw new IOException("Channel eof before end");
>
> }
>
> numbytes += temp_numbytes;
>
> }
> ===
> how to replicate this bug:
> perform an nmap portscan against your zookeeper server: "nmap -sV -n
> your.ip.here -p4181"
> wait for a while untill you see some messages in the logfile and then you
> will see 100% cpu usage. It does not recover from this situation. With my
> patch, it does not occur anymore
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[
https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921286#action_12921286
]
Thijs Terlouw commented on ZOOKEEPER-893:
-
I'll try to write a test case, but not exactly sure yet how to integrate. Need
to look at some example unit tests first :)
> ZooKeeper high cpu usage when invalid requests
> --
>
> Key: ZOOKEEPER-893
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893
> Project: Zookeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.3.1
> Environment: Linux 2.6.16
> 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz
> java version "1.6.0_17"
> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode)
>Reporter: Thijs Terlouw
>Assignee: Thijs Terlouw
>Priority: Critical
> Fix For: 3.3.2, 3.4.0
>
> Attachments: ZOOKEEPER-893.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> When ZooKeeper receives certain illegally formed messages on the internal
> communication port (:4181 by default), it's possible for ZooKeeper to enter
> an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427,
> but that patch does not resolve all issues.
> from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java
> the two affected parts:
> ===
> int length = msgLength.getInt();
>
> if(length <= 0) {
>
> throw new IOException("Invalid packet length:" + length);
>
> }
> ===
> ===
> while (message.hasRemaining()) {
>
> temp_numbytes = channel.read(message);
>
> if(temp_numbytes < 0) {
>
> throw new IOException("Channel eof before end");
>
> }
>
> numbytes += temp_numbytes;
>
> }
> ===
> how to replicate this bug:
> perform an nmap portscan against your zookeeper server: "nmap -sV -n
> your.ip.here -p4181"
> wait for a while untill you see some messages in the logfile and then you
> will see 100% cpu usage. It does not recover from this situation. With my
> patch, it does not occur anymore
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[
https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921103#action_12921103
]
Henry Robinson commented on ZOOKEEPER-893:
--
Thanks for the patch Thijs! It looks pretty good to me - good catch.
Do you think you might be able to write a test case that verifies correct
behaviour when you send malformed messages to the control port?
> ZooKeeper high cpu usage when invalid requests
> --
>
> Key: ZOOKEEPER-893
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893
> Project: Zookeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.3.1
> Environment: Linux 2.6.16
> 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz
> java version "1.6.0_17"
> Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode)
>Reporter: Thijs Terlouw
>Assignee: Thijs Terlouw
>Priority: Critical
> Fix For: 3.3.2, 3.4.0
>
> Attachments: ZOOKEEPER-893.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> When ZooKeeper receives certain illegally formed messages on the internal
> communication port (:4181 by default), it's possible for ZooKeeper to enter
> an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427,
> but that patch does not resolve all issues.
> from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java
> the two affected parts:
> ===
> int length = msgLength.getInt();
>
> if(length <= 0) {
>
> throw new IOException("Invalid packet length:" + length);
>
> }
> ===
> ===
> while (message.hasRemaining()) {
>
> temp_numbytes = channel.read(message);
>
> if(temp_numbytes < 0) {
>
> throw new IOException("Channel eof before end");
>
> }
>
> numbytes += temp_numbytes;
>
> }
> ===
> how to replicate this bug:
> perform an nmap portscan against your zookeeper server: "nmap -sV -n
> your.ip.here -p4181"
> wait for a while untill you see some messages in the logfile and then you
> will see 100% cpu usage. It does not recover from this situation. With my
> patch, it does not occur anymore
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
