[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-3689: --- Fix Version/s: (was: 3.0.0) > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Fix For: 2.7.0 > > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, > HDFS-3689.branch-2.patch, editsStored > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Fix Version/s: 2.7.0 > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Fix For: 3.0.0, 2.7.0 > > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, > HDFS-3689.branch-2.patch, editsStored > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Release Note: 1. HDFS now can choose to append data to a new block instead of end of the last partial block. Users can pass {{CreateFlag.APPEND}} and {{CreateFlag.NEW_BLOCK}} to the {{append}} API to indicate this requirement. 2. HDFS now allows users to pass {{SyncFlag.END_BLOCK}} to the {{hsync}} API to finish the current block and write remaining data to a new block. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Fix For: 3.0.0 > > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, > HDFS-3689.branch-2.patch, editsStored > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.branch-2.patch Post the patch for branch-2. So far I have not found any place where the functionality is broken by the variable length block. Maybe we should merge this to branch-2 this week? Note that the variable length block will not be generated unless the user explicitly passes in the {{CreateFlag#NEW_BLOCK}} while creating the file. Also if we find anything is broken by this feature we can fix them in separate jiras. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Fix For: 3.0.0 > > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, > HDFS-3689.branch-2.patch, editsStored > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for the review, Nicholas! I've committed this to trunk. And thanks to all for the discussion. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Fix For: 3.0.0 > > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, editsStored > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-3689: -- Hadoop Flags: Reviewed +1 patch looks good. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, editsStored > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.010.patch Thanks Nicholas! Update the patch to address the comments. bq. The if-condition below should This if-condition has actually been covered by {{verifyQuota}}. Thus I only update the comment here. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, editsStored > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.009.patch editsStored > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch, editsStored > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: (was: HDFS-3689.009.patch) > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.009.patch > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch, HDFS-3689.009.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.009.patch Thanks again Nicholas! Update the patch to add quota verification. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, > HDFS-3689.009.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.008.patch > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.008.patch Thanks for the comments, Colin. To use CreateFlag is a good suggestion. Update the patch to address your comments. Currently the patch targets for 3.0. We can merge this into 2.x after making sure this feature does not break existing applications' functionalities. So far I only checked {{FileInputFormat}} and looks like variable length block may only affect its performance but will not break its functionality. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch, HDFS-3689.008.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.007.patch Thanks for the review, Nicholas! Update the patch to address the comments. bq. We also need to enforce the same replication in concat if we don't want to update disk quota. The new patch just updates the diskspace quota usage after the concat. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, > HDFS-3689.007.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.006.patch Thanks for the review, Nicholas! Update the patch to address the comments. Here's a summary of the changes: # remove the new Append2 API and add a boolean {{newBlock}} to {{ClientProtocol#append}} to indicate whether the data should be appended to a new block # still add a new OP_APPEND editlog and use it also for the old append operation (instead of using OP_ADD) # update the APPEND inotify event # Fix the bug in flushOrSync as pointed by Nicholas > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.005.patch After an offline discussion with [~sanjay.radia] and [~szetszwo], the 005 patch still keeps the restriction that the source files and the target file should be in the same directory. In this way we do not need to update the quota. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch, HDFS-3689.005.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.004.patch > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, > HDFS-3689.004.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.003.patch > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.003.patch Update the patch with concat support. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch, HDFS-3689.003.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.002.patch Rebase the patch. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Assignee: Jing Zhao (was: Suresh Srinivas) Status: Patch Available (was: Open) > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, > HDFS-3689.002.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.001.patch Update the patch a little bit to demonstrate #3: the support in DFSOutputStream to let clients specify when to allocate a new block. Currently I simply add a new SyncFlag named END_BLOCK. Clients can call {{hsync(END_BLOCK)}} to force the completion of the current block. Some unit tests are also included to make sure the basic functionality works. > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.000.patch Upload a preliminary patch that addresses the above #3 and #4. Specifically a new append RPC is added which always appends new data to a new block, and thus the previous last block becomes a block with variable length. Some unit tests are added to make sure the appended file can still be read/pread by the current DFSInputStream. If this is a correct direction towards variable block length support, the remaining work can be: # bug fix and code cleanup # loose the restriction of concat # add support in DFSOutputStream to let clients specify when to allocate a new block > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-3689.000.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)