[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: HDFS-8433-HDFS-7285.02.patch bq. can we let NN return multiple block tokens for a striped block group? Sure. It's easy. I'm just a little worry about performance. (processing lots of token protos) 03 patch implements the approach. 4/5 of the patch is to refactor TestBlockTokenWithDFS, so TestBlockTokenWithDFSStriped can be extended. Sorry about that. Maybe I should just copy the code and leave TestBlockTokenWithDFS unchanged? So review and merging can be easier. > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433-HDFS-7285.02.patch, HDFS-8433.00.patch, > HDFS-8433.01.patch > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[
https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8433:
Attachment: HDFS-8433.03.PoC.patch
Uploaded 03 patch.
02 patch uses multiple tokens approach.
03 patch is a PoC, uses block id range approach.
bq. A token generated by old DN can't be parsed by new DN.
03 PoC patch add {{BlockTokenIdentifierV2}} to address this issue.
I think 03 patch is doable, and can solve the performance issue of 02 patch,
and the rolling upgrade issue of 01 patch.
> blockToken is not set in constructInternalBlock and parseStripedBlockGroup in
> StripedBlockUtil
> --
>
> Key: HDFS-8433
> URL: https://issues.apache.org/jira/browse/HDFS-8433
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Walter Su
> Attachments: HDFS-8433-HDFS-7285.02.patch, HDFS-8433.00.patch,
> HDFS-8433.01.patch, HDFS-8433.03.PoC.patch
>
>
> The blockToken provided in LocatedStripedBlock is not used to create
> LocatedBlock in constructInternalBlock and parseStripedBlockGroup in
> StripedBlockUtil.
> We should also add ec tests with security on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: (was: HDFS-8433.03.PoC.patch) > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433-HDFS-7285.02.patch, HDFS-8433.00.patch, > HDFS-8433.01.patch > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: HDFS-8433.03.txt 03 patch is impossible to be implemented. Please ignore 03 patch. I'll keep the file as a history file. > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433-HDFS-7285.02.patch, HDFS-8433.00.patch, > HDFS-8433.01.patch, HDFS-8433.03.txt > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: HDFS-8433.04.patch I have another idea. Uploaded 04 patch. I think it's better than 01~03 patch. > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433-HDFS-7285.02.patch, HDFS-8433.00.patch, > HDFS-8433.01.patch, HDFS-8433.03.txt, HDFS-8433.04.patch > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: HDFS-8433-HDFS-7285.05.patch >...The 04 patch still has issue: the old DN cannot parse the token ID created >by new NN/DN Only if old DN has striped block. >...semantically GROUP is not an access mode. That's true. Uploaded 05 patch. A little clean up based on 02 patch. (Repeat what I said before, testRead() fails in the middle, fails at "verify blockSeekTo()". That's expected. I'll add the retry logic of InputStream in another jira.) And Thanks [~jingzhao] for pointing out the defects of 01,03,04 patch. > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433-HDFS-7285.02.patch, > HDFS-8433-HDFS-7285.05.patch, HDFS-8433.00.patch, HDFS-8433.01.patch, > HDFS-8433.03.txt, HDFS-8433.04.patch > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: HDFS-8433-HDFS-7285.06.patch uploaded 06 patch address Jing's comments. > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433-HDFS-7285.02.patch, > HDFS-8433-HDFS-7285.05.patch, HDFS-8433-HDFS-7285.06.patch, > HDFS-8433.00.patch, HDFS-8433.01.patch, HDFS-8433.03.txt, HDFS-8433.04.patch > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Status: Patch Available (was: Open) > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433-HDFS-7285.02.patch, > HDFS-8433-HDFS-7285.05.patch, HDFS-8433-HDFS-7285.06.patch, > HDFS-8433.00.patch, HDFS-8433.01.patch, HDFS-8433.03.txt, HDFS-8433.04.patch > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: HDFS-8433.00.patch 00 initial patch: 1. Token support block Id range. So inner blocks of a block group share a token. 2. set accessToken for inner blocks in StripedBlockUtil. 3. Fix error handling in InputStream I'll fix error handling in OutputStream in next patch. > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433.00.patch > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: HDFS-8433.01.patch Uploaded 01 patch. Delete retry logic of Inputstream. testRead() pass half part. Fully read case passes. Retry case fails. Please run the test you'll know. > blockToken is not set in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil > -- > > Key: HDFS-8433 > URL: https://issues.apache.org/jira/browse/HDFS-8433 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Walter Su > Attachments: HDFS-8433.00.patch, HDFS-8433.01.patch > > > The blockToken provided in LocatedStripedBlock is not used to create > LocatedBlock in constructInternalBlock and parseStripedBlockGroup in > StripedBlockUtil. > We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
