[
https://issues.apache.org/jira/browse/HDFS-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589050#comment-15589050
]
Ewan Higgs commented on HDFS-11026:
-----------------------------------
Hi Daryn,
Thanks for taking a look. The patch targets Hadoop 3.0 which hasn't been
released yet and I was under the impression that rolling from 2.x to 3.0 isn't
supported (hence the major version number change). Also, the change to make
Token Identifiers in Yarn
use protobuf instead of {{WritableUtils}} (YARN-668) was done with no gating.
Those changes were done in order to support rolling upgrades in the first place
(YARN-666):
https://github.com/apache/hadoop/commit/5391919b09ce9549d13c897aa89bb0a0536760fe
That aside, if we want to support both payload formats then I propose a config
option ({{DFS_ACCESS_TOKEN_ENABLE_PROTOBUF =
"dfs.block.access.token.protobuf.enable"}}) which is turned off on default.
Then the NN sends old or new style payloads based on this. It should be turned
off until all datanodes are updated, as which point the config can be changed
and the NN starts sending new protobuf style payloads.
I'm not confident that the datanode will be able to 100% detect whether it's
looking at a protobuf message or an old style message (which is overly
flexible). So, we can put a boolean in {{BlockKeyProto}} that describes the
version information for the payload.
> Convert BlockTokenIdentifier to use Protobuf
> --------------------------------------------
>
> Key: HDFS-11026
> URL: https://issues.apache.org/jira/browse/HDFS-11026
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs, hdfs-client
> Affects Versions: 2.9.0, 3.0.0-alpha1
> Reporter: Ewan Higgs
> Fix For: 3.0.0-alpha2
>
> Attachments: blocktokenidentifier-protobuf.patch
>
>
> {{BlockTokenIdentifier}} currently uses a {{DataInput}}/{{DataOutput}}
> (basically a {{byte[]}}) and manual serialization to get data into and out of
> the encrypted buffer (in {{BlockKeyProto}}). Other TokenIdentifiers (e.g.
> {{ContainerTokenIdentifier}}, {{AMRMTokenIdentifier}}) use Protobuf. The
> {{BlockTokenIdenfitier}} should use Protobuf as well so it can be expanded
> more easily and will be consistent with the rest of the system.
> NB: Release of this will require a version update since 2.8.x won't be able
> to decipher {{BlockKeyProto.keyBytes}} from 2.8.y.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
