[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292190#comment-17292190
]
Hudson commented on HBASE-22492:
Results for branch branch-2.3
[build #173 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/173/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/173/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/173/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK contro
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292183#comment-17292183
]
Hudson commented on HBASE-22492:
Results for branch branch-2
[build #186 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/186/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/186/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/186/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before t
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292089#comment-17292089
]
Hudson commented on HBASE-22492:
Results for branch master
[build #222 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/222/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/222/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/222/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/222/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [htt
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292003#comment-17292003
]
Hudson commented on HBASE-22492:
Results for branch branch-2.4
[build #59 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/59/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/59/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/59/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/59/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/59/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac be
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291923#comment-17291923
]
Hudson commented on HBASE-22492:
Results for branch branch-2.2
[build #177 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/177/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/177//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/177//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/177//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/177//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874090#comment-16874090
]
Duo Zhang commented on HBASE-22492:
---
There is no big difference on the rpc level between branch-1 and branch-2 so I
think branch-2 should also be effected...
Will take a look if I have time...
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873328#comment-16873328
]
Sebastien Barnoud commented on HBASE-22492:
---
[~elserj] I just created https://issues.apache.org/jira/browse/HBASE-22634 that
contains the performance patch.
Regards
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871775#comment-16871775
]
Josh Elser commented on HBASE-22492:
bq. I have done several performance patches of HBase Client in my environment
(I will probably contribute them when i have enough time) that makes this issue
much more likely on heavy load condition (more than 10 mutation/second)
Super. I'm sure you'll have lots of folks interested in seeing what you've
found!
bq. I haven't check if branch-2 is affected or not
branch-2 stuff is using netty exclusively which lead me to think that it isn't
an issue there, but worth checking out. I'll see if I can find the time to do
that.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870900#comment-16870900
]
Sébastien BARNOUD commented on HBASE-22492:
---
[~elserj] many thanks to you for finishing the job.
Notice:
* I have done several performance patches of HBase Client in my environment (I
will probably contribute them when i have enough time) that makes this issue
much more likely on heavy load condition (more than 10 mutation/second)
* I haven't check if branch-2 is affected or not
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870050#comment-16870050
]
Hudson commented on HBASE-22492:
Results for branch branch-1
[build #912 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/912/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/912//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/912//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/912//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ign
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870049#comment-16870049
]
Hudson commented on HBASE-22492:
Results for branch branch-1.4
[build #868 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/868/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/868//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/868//JDK7_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/868//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870035#comment-16870035
]
Hudson commented on HBASE-22492:
Results for branch branch-1.3
[build #838 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/838/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/838//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/838//JDK7_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/838//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of havi
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869996#comment-16869996
]
Hudson commented on HBASE-22492:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #590 (See
[https://builds.apache.org/job/HBase-1.3-IT/590/])
HBASE-22492 Wrap RPC responses with SASL after queueing for response (elserj:
rev 0d5baffd1beb2acdcd989c5c611bab8ce78b0f09)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869991#comment-16869991
]
Josh Elser commented on HBASE-22492:
And, thanks, Sebastien, for the bug report and the patch to fix it.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869984#comment-16869984
]
Josh Elser commented on HBASE-22492:
Will fix the checkstyle on commit.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869957#comment-16869957
]
HBase QA commented on HBASE-22492:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
18s{color} | {color:red} hbase-server: The patch generated 1 new + 76 unchanged
- 0 fixed = 77 total (was 76) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
4m 40s{color} | {color:green} Patch does not cause any errors with Hadoop 2.8.5
2.9.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}111m
5s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HBASE-Build/568/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22492 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12972477/HBASE-22492.003.branch-1.patch
|
| Optional
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869870#comment-16869870
]
Josh Elser commented on HBASE-22492:
Took the liberty of cleaning up this patch, and buffed up some comments around
the changes. Plan on committing after QA, assuming all is good.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Assignee: Sébastien BARNOUD
>Priority: Major
> Fix For: 1.5.0, 1.4.11
>
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch, HBASE-22492.003.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869828#comment-16869828
]
Josh Elser commented on HBASE-22492:
bq. This change doesn't make sense to me. Is there some edge case you saw which
made you change this method? It seems identical in execution at a glance.
Ah, I misread this in the patch -- thought it was inside the for-loop which is
what confused me.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869825#comment-16869825
]
Josh Elser commented on HBASE-22492:
I'm throwing in the towel. I can't reproduce this in a unit test without really
hacking up RpcServer (which kind of invalidate the point of it). I'll attach my
driver program in case anyone wants to reproduce it that way.
Let me amend Sebastien's patch to get QA and then commit it.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869769#comment-16869769
]
Josh Elser commented on HBASE-22492:
I'm looking at this some more, and I actually think the problem is more that we
may mis-order responses in {{doRespond()}}.
Let's take responses: R1 and R2, which we should send back in that order (1,
then 2).
If we fail to write R1 (client socket not ready to read the entire response),
we'll re-queue that Call back to the response queue, but at the _front_ of the
queue. We only add elements to the back of the queue when the queue is empty
and there is no other write happening ({{responseWriteLock}}).
However, we don't hold the responseWriteLock when queueing the element to the
back of the queue. Thus, if both R1 and R2 are trying to be queued at the same
time and the queue is not empty, we have a race condition which could allow for
R2 to be queue before R1.
I don't think this invalidates Sebastien's fix (the same one that Hadoop made),
just wanted to share as it makes sense to me now how we might mis-queue these
elements.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869724#comment-16869724
]
Josh Elser commented on HBASE-22492:
{code:java}
+ if (saslWrapDone == true) return;
{code}
Just {{if (saslWrapDone)}}, please. Also, include braces please. The book
covers common practices that we'd like all contributions to follow
https://hbase.apache.org/book.html#common.patch.feedback
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869718#comment-16869718
]
Josh Elser commented on HBASE-22492:
{code:java}
@@ -1220,6 +1237,7 @@
*/
private boolean processAllResponses(final Connection connection) throws
IOException {
// We want only one writer on the channel for a connection at a time.
+ boolean isEmpty = false;
connection.responseWriteLock.lock();
try {
for (int i = 0; i < 20; i++) {
@@ -1233,11 +1251,13 @@
return false;
}
}
+// Check that state within the lock to be consistent
+isEmpty = connection.responseQueue.isEmpty();
} finally {
connection.responseWriteLock.unlock();
}
- return connection.responseQueue.isEmpty();
+ return isEmpty;
}
{code}
This change doesn't make sense to me. Is there some edge case you saw which
made you change this method? It seems identical in execution at a glance.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869064#comment-16869064
]
Josh Elser commented on HBASE-22492:
I've been working on a unit test to reproduce this (delay the first RPC to let
in the second), but need some more time on it.
I'd feel more comfortable waiting for a test around this.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868958#comment-16868958
]
Andrew Purtell commented on HBASE-22492:
fwiw patch lgtm, except this nit:
{code:java}
+/* to make sure that messages are sent in the same order than Sasl
sequence number, we must wrap the message when we put it in the response queue
if (connection.useWrap) {
bc = wrapWithSasl(bc);
}
+*/
{code}
Remove code, don't comment it out.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864412#comment-16864412
]
Sebastien Barnoud commented on HBASE-22492:
---
I put the patch on one of our cluster today. It solves the issue and no
regression observed.
Thanks for your feed-back.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864394#comment-16864394
]
Josh Elser commented on HBASE-22492:
FYI, I've been able to reproduce this on a standalone test harness. Not
suitable for a unit test, but at I can see the error now.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863488#comment-16863488
]
HBase QA commented on HBASE-22492:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
48s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
46s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
3m 28s{color} | {color:green} Patch does not cause any errors with Hadoop 2.8.5
2.9.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
45s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}182m 8s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}246m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestCompaction |
| | hadoop.hbase.regionserver.TestEncryptionKeyRotation |
| | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| | hadoop.hbase.regionserver.TestRegionServerHostname |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863401#comment-16863401
]
HBase QA commented on HBASE-22492:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
48s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
40s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
52s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
3m 28s{color} | {color:green} Patch does not cause any errors with Hadoop 2.8.5
2.9.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
38s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}158m
10s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HBASE-Build/534/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22492 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12970193/HBASE-22492.001.branch-1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
sha
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863349#comment-16863349
]
Josh Elser commented on HBASE-22492:
{quote}The tested RpcServer should override RpcServer.doRespond and sleep
before adding in the queue so that the first message sleeps more time than the
following
{quote}
Yup, that was along the ideas of what I had. We have lots of test
instrumentation in the codebase that can help here. End-to-end would be tricky
to do.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863336#comment-16863336
]
Sébastien BARNOUD commented on HBASE-22492:
---
I have an idea. The tested RpcServer should override RpcServer.doRespond and
sleep before adding in the queue so that the first message sleeps more time
than the following.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863329#comment-16863329
]
Sébastien BARNOUD commented on HBASE-22492:
---
It's not so easy to have a unit test with the full Sasl context, and having a
handler thread that interleaves ...
In +-HADOOP-12483-+ they have one unit test, but they have the Postponed API
for that.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863325#comment-16863325
]
Josh Elser commented on HBASE-22492:
{quote}I fact no need to "bypass" the optimization. I attach a new version of
the patch
{quote}
Great, thanks for providing a new patch. I've been trying to recreate the issue
you are describing.
I can understand how the problem could occur given your explanation, but we
should have more confidence in the form of a test which exhibits this (if not
intermittently, every time). When we have this, we can validate that your fix
works as you expect.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863323#comment-16863323
]
Sébastien BARNOUD commented on HBASE-22492:
---
I fact no need to "bypass" the optimization. I attach a new version of the patch
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch,
> HBASE-22492.002.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850970#comment-16850970
]
Sébastien BARNOUD commented on HBASE-22492:
---
[~elserj]: patch uploaded.
Notice: the patch has been done and tested on an HortonWorks (2.6.5.108-1), and
submitted for rework to HortonWorks support team. The Apache version differs,
and i port it on the Apache version before the upload.
I let a "TODO" in it, because when SASL is used, i "bypass" an optimization
(direct doResponse from the handler thread when the queue is empty). If this
optimization is really useful, then it has to added again, taking care on the
SASL sequence number order in a appropriate way (like HADOOP-12483?)
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
> Attachments: HBASE-22492.001.branch-1.patch
>
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850939#comment-16850939
]
Sébastien BARNOUD commented on HBASE-22492:
---
[~Apache9] : Yes that's it. The client I'm using is OK (1.1.2 or 2.1.4, but
with some performance patch i did), because a single thread encrypt/send on a
given connection via the IPC pool.
It's KO only in the server, as far as i tested.
It may be a good idea to check the replication client used by the server. I
didn't do that.
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850869#comment-16850869
]
Josh Elser commented on HBASE-22492:
[~sbarnoud], can you please attach your change as a patch to this Jira issue?
Without digging into exactly why we need this change yet (looks like a
deep-dive), I'd be curious to see what the tests think of your change.
Assuming this is your first patch, you can try to use the script
{{dev-support/make-patch.sh}}. I usually just run something like {{git
format-patch --stdout origin/branch-1 >
~/patches/HBASE-22492.001.branch-1.patch}}. Please note that the patch naming
is important:
* The numbering helps keep a history, assuming that there is iteration on the
patch
* The "branch-1" makes sure that the robot applies your patch to the correct
branch to run the tests
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850833#comment-16850833
]
Duo Zhang commented on HBASE-22492:
---
So the problem here is that, if we have two rpc responses at the same time,
let's say respA and respB, if we encrypt respA first and then respB, then we
must send respA to client first and then respB? Do we have the same problem at
client side?
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22492) HBase server doesn't preserve SASL sequence number on the network
[
https://issues.apache.org/jira/browse/HBASE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850749#comment-16850749
]
Sébastien BARNOUD commented on HBASE-22492:
---
I just discover
https://issues.apache.org/jira/browse/HADOOP-12483?attachmentSortBy=fileName
Exactly the same thing already patched in hadoop-common
> HBase server doesn't preserve SASL sequence number on the network
> -
>
> Key: HBASE-22492
> URL: https://issues.apache.org/jira/browse/HBASE-22492
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: HDP 2.6.5.108-1
>
>Reporter: Sébastien BARNOUD
>Priority: Major
>
> When auth-conf is enabled on RPC, the server encrypt response in setReponse()
> using saslServer. The generated cryptogram included a sequence number manage
> by saslServer. But then, when the response is sent over the network, the
> sequence number order is not preserved.
> The client receives reply in the wrong order, leading to a log message from
> DigestMD5Base:
> {code:java}
> sasl:1481 - DIGEST41:Unmatched MACs
> {code}
> Then the message is discarded, leading the client to a timeout.
> I propose a fix here:
> [https://github.com/sbarnoud/hbase-release/commit/ce9894ffe0e4039deecd1ed51fa135f64b311d41]
> It seems that any HBase 1.x is affected.
> This part of code has been fully rewritten in HBase 2.x, and i haven't do the
> analysis on HBase 2.x which may be affected.
>
> Here, an extract of client log that i added to help me to understand:
> {code:java}
> …
> 2019-05-28 12:53:48,644 DEBUG [Default-IPC-NioEventLoopGroup-1-32]
> NettyRpcDuplexHandler:80 - callId: 5846 /192.163.201.65:58870 ->
> dtltstap004.fr.world.socgen/192.163.201.72:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> NioEventLoop:101 - SG: Channel ready to read 1315913615 unsafe 1493023957
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,651 INFO [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:78 - SG: after unwrap:46 -> 29 for /192.163.201.65:44236
> -> dtltstap008.fr.world.socgen/192.163.201.109:16020 seqNum 150
> 2019-05-28 12:53:48,652 DEBUG [Default-IPC-NioEventLoopGroup-1-18]
> NettyRpcDuplexHandler:192 - callId: 5801 received totalSize:25 Message:20
> scannerSize:(null)/192.163.201.65:44236 ->
> dtltstap008.fr.world.socgen/192.163.201.109:16020
> 2019-05-28 12:53:48,652 INFO [Default-IPC-NioEventLoopGroup-1-18] sasl:1481
> - DIGEST41:Unmatched MACs
> 2019-05-28 12:53:48,652 WARN [Default-IPC-NioEventLoopGroup-1-18]
> SaslUnwrapHandler:70 - Sasl error (probably invalid MAC) detected for
> /192.163.201.65:44236 -> dtltstap008.fr.world.socgen/192.163.201.109:16020
> saslClient @4ac31121 ctx @14fb001d msg @140313192718406 len 118
> data:1c^G?^P?3??h?k??"??x?$^_??^D;^]7^Es??Em?c?w^R^BL?x??omG?z?I???45}???dE?^\^S>D?^/4f?^^??
> ?^Ed?D?kM^@^A^@^@^@? readerIndex 118 writerIndex 118 seqNum
> 152{code}
> We can see that the client unwraps the Sasl message with sequence number 152
> before sequence number 151 and fails with the unmatched MAC.
>
> I opened a case to Oracle because we should had an error (and not the message
> ignored). That's because the JDK doesn't controls integrity in the right way.
> [https://github.com/openjdk/jdk/blob/master/src/java.security.sasl/share/classes/com/sun/security/sasl/digest/DigestMD5Base.java]
> The actual JDK controls the HMac before the sequence number and hides the
> real error (bad sequence number) because SASL is stateful. The JDK should
> check FIRST the sequence number and THEN the HMac.
> When (and if) the JDK will be patched, and accordingly to
> [https://www.ietf.org/rfc/rfc2831.txt|https://www.ietf.org/rfc/rfc2831.txt,]
> , we will get an exception in that case instead of having the message ignored.
> h3.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
