[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[ https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236283#comment-13236283 ] Joey Echeverria commented on HDFS-2386: --- [HDFS-2617] is definitely the right solution for Hadoop. I still plan on filing the JDK bug to make the world a little less broken. with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs - Key: HDFS-2386 URL: https://issues.apache.org/jira/browse/HDFS-2386 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[ https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232875#comment-13232875 ] Joey Echeverria commented on HDFS-2386: --- From testing I've been doing it looks like KSSL won't work without at least one of the DES encryption types enabled (e.g. DES_CBC_CRC). This looks like it's caused by a bug in the JDK. Basically, AES and RC4 don't pad unless they encrypt a message which is not a multiple of a block. However, the JDK is assuming that the PreMasterSecret will be padded and assumes that the last byte in the decrypted secret is the length of the padding. When using AES or RC4, this ends up being a random byte and usually will cause the JDK to end up with an invalid PreMasterSecret. In defense of this, the JDK generates a random secret that then caused the handshake to fail later on. I need to do some more testing with another version of Kerberos, but I plan on filing a JDK bug. with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs - Key: HDFS-2386 URL: https://issues.apache.org/jira/browse/HDFS-2386 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[
https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13177534#comment-13177534
]
Rajesh Balamohan commented on HDFS-2386:
we are actively hitting this issue with the secondary namenode and fsck with
the 204. JDK 1.6.0_29, RHEL 6.1, MIT 1.8.x, AES-256, AES-128, and RC4 enc types
are enabled. JCE is installed.
+1, We are facing this issue as well and get the following exception in
NameNode.
11/12/29 18:47:02 WARN mortbay.log: EXCEPTION
javax.net.ssl.SSLHandshakeException: Invalid padding
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1699)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:852)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1165)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1149)
at
org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 238
at
com.sun.net.ssl.internal.ssl.CipherBox.removePadding(CipherBox.java:399)
at com.sun.net.ssl.internal.ssl.CipherBox.decrypt(CipherBox.java:247)
at
com.sun.net.ssl.internal.ssl.InputRecord.decrypt(InputRecord.java:153)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:840)
... 5 more
Pasting the javax.net.debug output from secondary namenode (if this would be of
help)
Enabled javax.net.debug=all in secondary namenode and got the following output
Cipher Suite: TLS_KRB5_WITH_3DES_EDE_CBC_SHA
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: empty
***
%% Created: [Session-1, TLS_KRB5_WITH_3DES_EDE_CBC_SHA]
** TLS_KRB5_WITH_3DES_EDE_CBC_SHA
*** ServerHelloDone
*** ClientKeyExchange, Kerberos
...
...
..
*** Finished
verify_data: { 190, 127, 20, 131, 10, 136, 84, 207, 172, 130, 31, 53 }
***
main, WRITE: TLSv1 Handshake, length = 40
main, READ: TLSv1 Alert, length = 2
main, RECV TLSv1 ALERT: fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal
alert: handshake_failure
11/12/29 18:47:02 ERROR namenode.SecondaryNameNode: checkpoint: Content-Length
header is not provided by the namenode when trying to fetch
https://NN:50475/getimage?getimage=1
with security enabled fsck calls lead to handshake_failure and hftp fails
throwing the same exception in the logs
-
Key: HDFS-2386
URL: https://issues.apache.org/jira/browse/HDFS-2386
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.20.205.0
Reporter: Arpit Gupta
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[ https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160492#comment-13160492 ] Allen Wittenauer commented on HDFS-2386: FWIW, we are actively hitting this issue with the secondary namenode and fsck with the 204. JDK 1.6.0_29, RHEL 6.1, MIT 1.8.x, AES-256, AES-128, and RC4 enc types are enabled. JCE is installed. We see on the NN side that we throw an invalid_padding error while the 2nd NN and fsck throw the handshake_failure message. At this point, I'm leaning towards ripping out the SSL code from the namenode and running at least that portion unsecured. with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs - Key: HDFS-2386 URL: https://issues.apache.org/jira/browse/HDFS-2386 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[ https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160534#comment-13160534 ] Jitendra Nath Pandey commented on HDFS-2386: Can you try with DES_CBC_CRC? It is also available in Java 6 by default. with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs - Key: HDFS-2386 URL: https://issues.apache.org/jira/browse/HDFS-2386 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[ https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160542#comment-13160542 ] Jakob Homan commented on HDFS-2386: --- bq. At this point, I'm leaning towards ripping out the SSL code from the namenode and running at least that portion unsecured. Now that we have a SPNEGO filter, we no longer need to use Kerberized SSL. It would be good to remove that entirely. It was added as a work-around to having no suitable SPNEGO solution and is rather unique to Hadoop (although apparently ActiveMQ took the code as well). This would save us from having to use the host principal (and instead use the more standard http principal) and simplify a lot of the config. with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs - Key: HDFS-2386 URL: https://issues.apache.org/jira/browse/HDFS-2386 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[ https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160546#comment-13160546 ] Aaron T. Myers commented on HDFS-2386: -- +1 to Jakob's recommendation. In practice, I've found setting up a 2NN to successfully checkpoint to be the most annoying and difficult-to-debug part of configuring secure Hadoop. with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs - Key: HDFS-2386 URL: https://issues.apache.org/jira/browse/HDFS-2386 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[ https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160582#comment-13160582 ] Allen Wittenauer commented on HDFS-2386: Can you try with DES_CBC_CRC? No, because at that level you might as well send it across the wire unencrypted. with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs - Key: HDFS-2386 URL: https://issues.apache.org/jira/browse/HDFS-2386 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2386) with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs
[ https://issues.apache.org/jira/browse/HDFS-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117794#comment-13117794 ] Jitendra Nath Pandey commented on HDFS-2386: @Aaron From kerberos debug logs the encryption used between nodes in my cluster seems to be DES_CBC_CRC. with security enabled fsck calls lead to handshake_failure and hftp fails throwing the same exception in the logs - Key: HDFS-2386 URL: https://issues.apache.org/jira/browse/HDFS-2386 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
