[
https://issues.apache.org/jira/browse/HDFS-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683033#comment-17683033
]
Yiqun Lin edited comment on HDFS-16644 at 2/1/23 2:19 PM:
----------------------------------------------------------
We also meet this issue in our Hadoop3 cluster. Our DN server is Hadoop 3.3
version, and client version is 2.10.2.
We find that there is one chance that abnormal QOP value(e.g. DI) can be passed
and overwrite for DataNode sasl props.
But by default case(HDFS-13541 feature not enabled), the secret should not be
passed here. Somehow that there maybe some bug on 2.10.2 version that still
pass the secret here.
[~vagarychen], could you please check for this code on branch-2.10. It's very
dangerous that once DN sasl props is overwrite with an invalid value, all the
data read/write could be impacted. And also here we don't do any validation
check for QOP value.
SaslDataTransferServer#doSaslHandshake
{noformat}
private IOStreamPair doSaslHandshake(Peer peer, OutputStream underlyingOut,
InputStream underlyingIn, Map<String, String> saslProps,
CallbackHandler callbackHandler) throws IOException {
DataInputStream in = new DataInputStream(underlyingIn);
DataOutputStream out = new DataOutputStream(underlyingOut);
int magicNumber = in.readInt();
if (magicNumber != SASL_TRANSFER_MAGIC_NUMBER) {
throw new InvalidMagicNumberException(magicNumber,
dnConf.getEncryptDataTransfer());
}
try {
// step 1
SaslMessageWithHandshake message = readSaslMessageWithHandshakeSecret(in);
byte[] secret = message.getSecret();
String bpid = message.getBpid();
if (secret != null || bpid != null) {
// sanity check, if one is null, the other must also not be null
assert(secret != null && bpid != null);
String qop = new String(secret, Charsets.UTF_8);
saslProps.put(Sasl.QOP, qop); <===== here any QOP value could be set
here
}
...
{noformat}
was (Author: linyiqun):
We also meet this issue in our Hadoop3 cluster. Our DN server is Hadoop 3.3
version, and client version is 2.10.2.
We find that there is one chance that abnormal QOP value(e.g. DI) can be passed
and overwrite for DataNode sasl props.
But by default case(HDFS-13541 feature not enabled), the secret should not be
passed here. Somehow that there maybe some bug on 2.10.2 version that still
pass the secret here.
[~vagarychen], could you please check for this code on branch-2.10. It's very
dangerous that once DN sasl props is overwrite with an invalid value. All the
data read/write could be impacted. And also here we don't do any validation
check for QOP value.
SaslDataTransferServer#doSaslHandshake
{noformat}
private IOStreamPair doSaslHandshake(Peer peer, OutputStream underlyingOut,
InputStream underlyingIn, Map<String, String> saslProps,
CallbackHandler callbackHandler) throws IOException {
DataInputStream in = new DataInputStream(underlyingIn);
DataOutputStream out = new DataOutputStream(underlyingOut);
int magicNumber = in.readInt();
if (magicNumber != SASL_TRANSFER_MAGIC_NUMBER) {
throw new InvalidMagicNumberException(magicNumber,
dnConf.getEncryptDataTransfer());
}
try {
// step 1
SaslMessageWithHandshake message = readSaslMessageWithHandshakeSecret(in);
byte[] secret = message.getSecret();
String bpid = message.getBpid();
if (secret != null || bpid != null) {
// sanity check, if one is null, the other must also not be null
assert(secret != null && bpid != null);
String qop = new String(secret, Charsets.UTF_8);
saslProps.put(Sasl.QOP, qop); <===== here any QOP value could be set
here
}
...
{noformat}
> java.io.IOException Invalid token in javax.security.sasl.qop
> ------------------------------------------------------------
>
> Key: HDFS-16644
> URL: https://issues.apache.org/jira/browse/HDFS-16644
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.2.1
> Reporter: Walter Su
> Priority: Major
>
> deployment:
> server side: kerberos enabled cluster with jdk 1.8 and hdfs-server 3.2.1
> client side:
> I run command hadoop fs -put a test file, with kerberos ticket inited first,
> and use identical core-site.xml & hdfs-site.xml configuration.
> using client ver 3.2.1, it succeeds.
> using client ver 2.8.5, it succeeds.
> using client ver 2.10.1, it fails. The client side error info is:
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient:
> SASL encryption trust check: localHostTrusted = false, remoteHostTrusted =
> false
> 2022-06-27 01:06:15,781 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> DataNode{data=FSDataset{dirpath='[/mnt/disk1/hdfs, /mnt/***/hdfs,
> /mnt/***/hdfs, /mnt/***/hdfs]'}, localName='emr-worker-***.***:9866',
> datanodeUuid='b1c7f64a-6389-4739-bddf-***', xmitsInProgress=0}:Exception
> transfering block BP-1187699012-10.****-***:blk_1119803380_46080919 to mirror
> 10.*****:9866
> java.io.IOException: Invalid token in javax.security.sasl.qop: D
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessage(DataTransferSaslUtil.java:220)
> Once any client ver 2.10.1 connect to hdfs server, the DataNode no longer
> accepts any client connection, even client ver 3.2.1 cannot connects to hdfs
> server. The DataNode rejects any client connection. For a short time, all
> DataNodes rejects client connections.
> The problem exists even if I replace DataNode with ver 3.3.0 or replace java
> with jdk 11.
> The problem is fixed if I replace DataNode with ver 3.2.0. I guess the
> problem is related to HDFS-13541
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
