[
https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322640#comment-17322640
]
zhangxiping edited comment on HDFS-15756 at 4/16/21, 7:05 AM:
--------------------------------------------------------------
[~hexiaoqiao] Thank you for your reply. Yes, I have the same question. I
think Spark should not renew token so soon.To avoid the current problem, we
implement that the Router waits for 100ms to retry to get the token(recently
generated) from ZK.Since the probability of this problem is relatively small,
it occurs dozens of times a day and does not affect the performance of the
Router, it can also be obtained from ZK.But I think it would be better to try
again on the client side, which is not a perfect solution and can only
circumvent the problem temporarily. Expect a better solution.
was (Author: zhangxiping):
[~hexiaoqiao] Thank you for your reply. Yes, I have the same question. I
think Spark should not renew token so soon.To avoid the current problem, we
implement that the Router waits for 100ms to retry to get the token(recently
generated) from ZK.Since the probability of this problem is relatively small,
it occurs dozens of times a day and does not affect the performance of the
Router, it can also be obtained from ZK.But I think it would be better to try
again on the client side, which is not a perfect solution and can only
circumvent the problem temporarily. Expect a better solution.
{code:java}
//代码占位符 AbstractDelegationTokenSecretManager
protected synchronized byte[] createPassword(TokenIdent identifier) {
int sequenceNum;
long now = Time.now();
sequenceNum = incrementDelegationTokenSeqNum();
identifier.setIssueDate(now);
identifier.setMaxDate(now + tokenMaxLifetime);
identifier.setMasterKeyId(currentKey.getKeyId());
identifier.setSequenceNumber(sequenceNum);
LOG.info("Creating password for identifier: " + formatTokenId(identifier)
+ ", currentKey: " + currentKey.getKeyId());
byte[] password = createPassword(identifier.getBytes(), currentKey.getKey());
DelegationTokenInformation tokenInfo = new DelegationTokenInformation(now
+ tokenRenewInterval, password, getTrackingIdIfEnabled(identifier));
try {
storeToken(identifier, tokenInfo);
} catch (IOException ioe) {
LOG.error("Could not store token " + formatTokenId(identifier) + "!!",
ioe);
}
return password;
}
{code}
> RBF: Cannot get updated delegation token from zookeeper
> -------------------------------------------------------
>
> Key: HDFS-15756
> URL: https://issues.apache.org/jira/browse/HDFS-15756
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
> Affects Versions: 3.0.0
> Reporter: hbprotoss
> Priority: Major
>
> Affected version: all version with rbf
> When RBF work with spark 2.4 client mode, there will be a chance that token
> is missing across different nodes in RBF cluster. The root cause is that
> spark renew the token(via resource manager) immediately after got one, as
> zookeeper don't have a strong consistency guarantee after an update in
> cluster, zookeeper client may read a stale value in some followers not synced
> with other nodes.
>
> We apply a patch in spark, but it is still the problem of RBF. Is it possible
> for RBF to replace the delegation token store using some other
> datasource(redis for example)?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
