[jira] [Updated] (HDFS-9829) Erasure Coding: Improve few exception handling logic of ErasureCodingWorker

[ https://issues.apache.org/jira/browse/HDFS-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9829: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: P

[jira] [Updated] (HDFS-9818) Correctly handle EC reconstruction work caused by not enough racks

[ https://issues.apache.org/jira/browse/HDFS-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9818: Attachment: HDFS-9818.004.patch Thanks for the review, [~rakeshr]! Update the patch to address your comments.

[jira] [Created] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

Jing Zhao created HDFS-9837: --- Summary: BlockManager#countNodes should be able to detect duplicated internal blocks Key: HDFS-9837 URL: https://issues.apache.org/jira/browse/HDFS-9837 Project: Hadoop HDFS

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Status: Patch Available (was: Open) > BlockManager#countNodes should be able to detect duplicated internal b

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Attachment: HDFS-9837.000.patch Upload a patch to fix. > BlockManager#countNodes should be able to detect du

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Attachment: HDFS-9837.001.patch The 001 patch does some code refactoring for {{countNodes}} and also avoids

[jira] [Updated] (HDFS-9818) Correctly handle EC reconstruction work caused by not enough racks

[ https://issues.apache.org/jira/browse/HDFS-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9818: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) The failed

[jira] [Commented] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155346#comment-15155346 ] Jing Zhao commented on HDFS-9837: - The current patch has an bug that it causes NN to ignore

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Attachment: HDFS-9837.002.patch Update the patch. Add a new state "REDUNDANT" in NumberReplicas to record the

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Status: Patch Available (was: Open) > BlockManager#countNodes should be able to detect duplicated internal b

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Status: Open (was: Patch Available) > BlockManager#countNodes should be able to detect duplicated internal b

[jira] [Commented] (HDFS-9838) Refactor the excessReplicateMap to a class

[ https://issues.apache.org/jira/browse/HDFS-9838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158115#comment-15158115 ] Jing Zhao commented on HDFS-9838: - Thanks for the refactoring, Nicholas! The patch looks go

[jira] [Commented] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159485#comment-15159485 ] Jing Zhao commented on HDFS-9837: - Thanks for the comments, [~rakeshr]. bq. Should be Sto

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Attachment: HDFS-9837.003.patch Update the patch to address comments. > BlockManager#countNodes should be ab

[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned

[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159615#comment-15159615 ] Jing Zhao commented on HDFS-8786: - Thanks for working on this, Rakesh. Some quick comments:

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Attachment: HDFS-9837.004.patch Thanks for the review, Nicholas! Update the patch to incorporate your change

[jira] [Commented] (HDFS-9710) Change DN to send block receipt IBRs in batches

[ https://issues.apache.org/jira/browse/HDFS-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15165669#comment-15165669 ] Jing Zhao commented on HDFS-9710: - Thanks for working on this, Nicholas! The patch looks go

[jira] [Updated] (HDFS-9837) BlockManager#countNodes should be able to detect duplicated internal blocks

[ https://issues.apache.org/jira/browse/HDFS-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9837: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) The failed

[jira] [Commented] (HDFS-9838) Refactor the excessReplicateMap to a class

[ https://issues.apache.org/jira/browse/HDFS-9838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15166539#comment-15166539 ] Jing Zhao commented on HDFS-9838: - Thanks for updating the patch, Nicholas! The new patch l

[jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned

[ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167762#comment-15167762 ] Jing Zhao commented on HDFS-8786: - bq. I will add an extra check against the BlockManager#c

[jira] [Created] (HDFS-9866) BlockManager#chooseExcessReplicasStriped may weaken rack fault tolerance

Jing Zhao created HDFS-9866: --- Summary: BlockManager#chooseExcessReplicasStriped may weaken rack fault tolerance Key: HDFS-9866 URL: https://issues.apache.org/jira/browse/HDFS-9866 Project: Hadoop HDFS

[jira] [Commented] (HDFS-9710) Change DN to send block receipt IBRs in batches

[ https://issues.apache.org/jira/browse/HDFS-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170080#comment-15170080 ] Jing Zhao commented on HDFS-9710: - The latest patch looks good to me. +1. Thanks for the wo

[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones

[ https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723837#comment-14723837 ] Jing Zhao commented on HDFS-8833: - Thanks for all the discussion! Currently I think it's fi

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723852#comment-14723852 ] Jing Zhao commented on HDFS-8980: - Sure. I will take a look at HDFS-8763. > Remove unneces

[jira] [Updated] (HDFS-8980) Remove unnecessary block replacement in INodeFile

[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8980: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: P

[jira] [Commented] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

[ https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724094#comment-14724094 ] Jing Zhao commented on HDFS-8982: - Actually I think here it's better to use {{getFileReplic

[jira] [Comment Edited] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

[ https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724094#comment-14724094 ] Jing Zhao edited comment on HDFS-8982 at 8/31/15 9:44 PM: -- Actuall

[jira] [Commented] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

[ https://issues.apache.org/jira/browse/HDFS-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724151#comment-14724151 ] Jing Zhao commented on HDFS-8936: - Maybe we can keep this jira open until we fix the known

[jira] [Commented] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

[ https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724372#comment-14724372 ] Jing Zhao commented on HDFS-8982: - After HDFS-8823, the replication factor is recorded insi

[jira] [Commented] (HDFS-8763) After file closed, a race condition between IBR of 3rd replica of lastBlock and ReplicationMonitor

[ https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724484#comment-14724484 ] Jing Zhao commented on HDFS-8763: - So one question I always have is whether we need to let

[jira] [Comment Edited] (HDFS-8763) After file closed, a race condition between IBR of 3rd replica of lastBlock and ReplicationMonitor

[ https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724484#comment-14724484 ] Jing Zhao edited comment on HDFS-8763 at 9/1/15 12:40 AM: -- So one

[jira] [Commented] (HDFS-8937) Fix the exception when set replication to Erasure Coding files

[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726169#comment-14726169 ] Jing Zhao commented on HDFS-8937: - Thanks for working on this, [~demongaorui]! Here are the

[jira] [Commented] (HDFS-8999) Namenode need not wait for {{blockReceived}} for the last block before completing a file.

[ https://issues.apache.org/jira/browse/HDFS-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726300#comment-14726300 ] Jing Zhao commented on HDFS-8999: - bq. if we are going to trust the client, do we even need

[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity

[ https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726497#comment-14726497 ] Jing Zhao commented on HDFS-8287: - Thanks for working on this, Kai! I agree with Zhe's comm

[jira] [Commented] (HDFS-8937) Fix the exception when set replication to Erasure Coding files

[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727811#comment-14727811 ] Jing Zhao commented on HDFS-8937: - +1. The failed tests are unrelated. I will commit the pa

[jira] [Updated] (HDFS-8937) Erasure coding: do not throw exception when setting replication factor on EC files

[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8937: Summary: Erasure coding: do not throw exception when setting replication factor on EC files (was: Fix the ex

[jira] [Updated] (HDFS-8937) Fix the exception when set replication to Erasure Coding files

[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8937: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7285 Status: Resolved (wa

[jira] [Commented] (HDFS-8999) Namenode need not wait for {{blockReceived}} for the last block before completing a file.

[ https://issues.apache.org/jira/browse/HDFS-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727883#comment-14727883 ] Jing Zhao commented on HDFS-8999: - Thanks for the comments, [~shv]. So one of the motivatio

[jira] [Updated] (HDFS-9003) ForkJoin thread pool leaks

[ https://issues.apache.org/jira/browse/HDFS-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9003: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: P

[jira] [Created] (HDFS-9011) Support splitting BlockReport of a storage into multiple RPC

Jing Zhao created HDFS-9011: --- Summary: Support splitting BlockReport of a storage into multiple RPC Key: HDFS-9011 URL: https://issues.apache.org/jira/browse/HDFS-9011 Project: Hadoop HDFS Issue T

[jira] [Updated] (HDFS-9011) Support splitting BlockReport of a storage into multiple RPC

[ https://issues.apache.org/jira/browse/HDFS-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9011: Attachment: HDFS-9011.000.patch Initial patch to demo the idea. Still need to add more tests and fix possible

[jira] [Updated] (HDFS-8978) Erasure coding: fix 2 failed tests of DFSStripedOutputStream

[ https://issues.apache.org/jira/browse/HDFS-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8978: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7285 Status: Resolved (wa

[jira] [Updated] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8384: Attachment: HDFS-8384.000.patch Post a simple patch: # Log a warning msg instead of throwing exception in {

[jira] [Updated] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8384: Assignee: Jing Zhao (was: Tsz Wo Nicholas Sze) Target Version/s: 2.8.0 Status: Patc

[jira] [Updated] (HDFS-9011) Support splitting BlockReport of a storage into multiple RPC

[ https://issues.apache.org/jira/browse/HDFS-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9011: Status: Patch Available (was: Open) > Support splitting BlockReport of a storage into multiple RPC > ---

[jira] [Commented] (HDFS-8966) Separate the lock used in namespace and block management layer

[ https://issues.apache.org/jira/browse/HDFS-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729826#comment-14729826 ] Jing Zhao commented on HDFS-8966: - Yeah, I agree more design details are necessary. In the

[jira] [Commented] (HDFS-8984) Move replication queues related methods in FSNamesystem to BlockManager

[ https://issues.apache.org/jira/browse/HDFS-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730035#comment-14730035 ] Jing Zhao commented on HDFS-8984: - The 04 patch looks good to me. One nit is that the chang

[jira] [Updated] (HDFS-9011) Support splitting BlockReport of a storage into multiple RPC

[ https://issues.apache.org/jira/browse/HDFS-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9011: Attachment: HDFS-9011.001.patch Add unit tests and fix some bugs. > Support splitting BlockReport of a stora

[jira] [Commented] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731224#comment-14731224 ] Jing Zhao commented on HDFS-8384: - Thanks for the review, Nicholas! Looks like the failed t

[jira] [Updated] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8384: Fix Version/s: 2.8.0 > Allow NN to startup if there are files having a lease but are not under > constructio

[jira] [Updated] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8384: Attachment: HDFS-8384-branch-2.7.patch HDFS-8384-branch-2.6.patch I've committed the patch to

[jira] [Updated] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8384: Target Version/s: 2.6.1, 2.7.2 (was: 2.8.0) > Allow NN to startup if there are files having a lease but are

[jira] [Commented] (HDFS-8967) Create a BlockManagerLock class to represent the lock used in the BlockManager

[ https://issues.apache.org/jira/browse/HDFS-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731369#comment-14731369 ] Jing Zhao commented on HDFS-8967: - I think the patch should work since we only provide a wr

[jira] [Updated] (HDFS-9011) Support splitting BlockReport of a storage into multiple RPC

[ https://issues.apache.org/jira/browse/HDFS-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9011: Attachment: HDFS-9011.002.patch Thanks for the review, Nicholas! Update the patch to fix the failed unit test

[jira] [Commented] (HDFS-8383) Tolerate multiple failures in DFSStripedOutputStream

[ https://issues.apache.org/jira/browse/HDFS-8383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731630#comment-14731630 ] Jing Zhao commented on HDFS-8383: - Thanks for working on this, Walter! So could you please

[jira] [Commented] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

[ https://issues.apache.org/jira/browse/HDFS-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735229#comment-14735229 ] Jing Zhao commented on HDFS-8936: - bq, Maybe we need to notify the other streamers this rem

[jira] [Comment Edited] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

[ https://issues.apache.org/jira/browse/HDFS-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735229#comment-14735229 ] Jing Zhao edited comment on HDFS-8936 at 9/8/15 5:35 PM: - bq. Maybe

[jira] [Commented] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

[ https://issues.apache.org/jira/browse/HDFS-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735263#comment-14735263 ] Jing Zhao commented on HDFS-8936: - Yeah, totally agree. But to still reuse the current Data

[jira] [Commented] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735274#comment-14735274 ] Jing Zhao commented on HDFS-8384: - Thanks again for the review, Nicholas! I've committed th

[jira] [Updated] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8384: Fix Version/s: 2.7.2 > Allow NN to startup if there are files having a lease but are not under > constructio

[jira] [Updated] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8384: Target Version/s: (was: 2.6.1, 2.7.2) > Allow NN to startup if there are files having a lease but are not u

[jira] [Updated] (HDFS-8384) Allow NN to startup if there are files having a lease but are not under construction

[ https://issues.apache.org/jira/browse/HDFS-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8384: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Allow NN

[jira] [Commented] (HDFS-9011) Support splitting BlockReport of a storage into multiple RPC

[ https://issues.apache.org/jira/browse/HDFS-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735775#comment-14735775 ] Jing Zhao commented on HDFS-9011: - Thanks for the review, Nicholas and Yi! bq. for each pa

[jira] [Commented] (HDFS-9052) deleteSnapshot runs into AssertionError

[ https://issues.apache.org/jira/browse/HDFS-9052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739684#comment-14739684 ] Jing Zhao commented on HDFS-9052: - Thanks for the report, Alex. I'm thinking whether the e

[jira] [Commented] (HDFS-9050) updatePipeline RPC call should only update the GS of the block

[ https://issues.apache.org/jira/browse/HDFS-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739694#comment-14739694 ] Jing Zhao commented on HDFS-9050: - Thanks for working on this, Zhe. But looks like the patc

[jira] [Commented] (HDFS-9047) deprecate libwebhdfs in branch-2; remove from trunk

[ https://issues.apache.org/jira/browse/HDFS-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739695#comment-14739695 ] Jing Zhao commented on HDFS-9047: - I also agree the current implementation is not good. So

[jira] [Commented] (HDFS-9052) deleteSnapshot runs into AssertionError

[ https://issues.apache.org/jira/browse/HDFS-9052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739734#comment-14739734 ] Jing Zhao commented on HDFS-9052: - A possible scenario is like this: 1. We deleted a file "

[jira] [Comment Edited] (HDFS-9052) deleteSnapshot runs into AssertionError

[ https://issues.apache.org/jira/browse/HDFS-9052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739734#comment-14739734 ] Jing Zhao edited comment on HDFS-9052 at 9/10/15 10:27 PM: --- A pos

[jira] [Created] (HDFS-9063) Correctly handle snapshot path for getContentSummary

Jing Zhao created HDFS-9063: --- Summary: Correctly handle snapshot path for getContentSummary Key: HDFS-9063 URL: https://issues.apache.org/jira/browse/HDFS-9063 Project: Hadoop HDFS Issue Type: Bug

[jira] [Commented] (HDFS-9040) Erasure coding: A BlockGroupDataStreamer to rule all internal blocks streamers

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741388#comment-14741388 ] Jing Zhao commented on HDFS-9040: - Thanks for the patch, Walter! I think this looks much cl

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741620#comment-14741620 ] Jing Zhao commented on HDFS-7285: - Thanks Zhe! Also IMO the write pipeline error handling m

[jira] [Updated] (HDFS-9040) Erasure coding: A BlockGroupDataStreamer to rule all internal blocks streamers

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040.001.wip.patch Upload a wip patch (based on Walter's great job in HDFS-8383) to demo the

[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity

[ https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744015#comment-14744015 ] Jing Zhao commented on HDFS-8287: - Thanks for working on this, [~kaisasak] and [~szetszwo]!

[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744281#comment-14744281 ] Jing Zhao commented on HDFS-9040: - Thanks for the comments, Zhe, Walter and Bo! bq. With B

[jira] [Commented] (HDFS-9011) Support splitting BlockReport of a storage into multiple RPC

[ https://issues.apache.org/jira/browse/HDFS-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744491#comment-14744491 ] Jing Zhao commented on HDFS-9011: - Thanks for the comments, Colin. I'm also leaning towards

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Status: Patch Available (was: Open) > Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Req

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040-HDFS-7285.002.patch Update the patch to fix some current unit tests. > Erasure coding:

[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746443#comment-14746443 ] Jing Zhao commented on HDFS-9040: - Thanks for the great review, Walter and Zhe! bq. Speaki

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040-HDFS-7285.003.patch Upload a patch trying to fix race conditions. Still need to fix the

[jira] [Commented] (HDFS-9052) deleteSnapshot runs into AssertionError

[ https://issues.apache.org/jira/browse/HDFS-9052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746534#comment-14746534 ] Jing Zhao commented on HDFS-9052: - Hi Alex, so the issue here is not about {{computeDiffBet

[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746541#comment-14746541 ] Jing Zhao commented on HDFS-9040: - I agree. Let's see if we can be confident about this fix

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: (was: HDFS-9040-HDFS-7285.003.patch) > Erasure coding: Refactor DFSStripedOutputStream (Move

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040-HDFS-7285.003.patch > Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RP

[jira] [Commented] (HDFS-9052) deleteSnapshot runs into AssertionError

[ https://issues.apache.org/jira/browse/HDFS-9052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790779#comment-14790779 ] Jing Zhao commented on HDFS-9052: - Yeah, we can resolve this jira as duplicated with HDFS-6

[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790913#comment-14790913 ] Jing Zhao commented on HDFS-1172: - Thanks for keeping working on the issue, [~walter.k.su]

[jira] [Updated] (HDFS-9063) Correctly handle snapshot path for getContentSummary

[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9063: Status: Patch Available (was: Open) > Correctly handle snapshot path for getContentSummary > ---

[jira] [Updated] (HDFS-9063) Correctly handle snapshot path for getContentSummary

[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9063: Attachment: HDFS-9063.000.patch Upload a patch to fix the issue. In the patch I add an extra parameter for {

[jira] [Commented] (HDFS-9070) Erasure coding: allow fsck display the pending replica information(including datanode information) of being-written EC block group

[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791391#comment-14791391 ] Jing Zhao commented on HDFS-9070: - [~demongaorui], looks like your patch contains a lot of

[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary

[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803241#comment-14803241 ] Jing Zhao commented on HDFS-9063: - Thanks Yi! {{getContentSummary(dir's current path)}} sho

[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary

[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803250#comment-14803250 ] Jing Zhao commented on HDFS-9063: - The test failures look suspicious. I just triggered the

[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803339#comment-14803339 ] Jing Zhao commented on HDFS-9040: - I think the key point here is to bump the GS, which is n

[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary

[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875883#comment-14875883 ] Jing Zhao commented on HDFS-9063: - Thanks for the review, Yi! I will commit the patch short

[jira] [Updated] (HDFS-9063) Correctly handle snapshot path for getContentSummary

[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9063: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: P

[jira] [Commented] (HDFS-9070) Erasure coding: allow fsck display the pending replica information(including datanode information) of being-written EC block group

[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875944#comment-14875944 ] Jing Zhao commented on HDFS-9070: - Thanks for working on this, Rui! The patch looks good to

[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9070: Summary: Allow fsck display pending replica location information for being-written blocks (was: Erasure codi

[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9070: Issue Type: Improvement (was: Sub-task) Parent: (was: HDFS-8031) > Allow fsck display pending re

[jira] [Commented] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875947#comment-14875947 ] Jing Zhao commented on HDFS-9070: - For #1, I've moved the issue out of the HDFS-8031 and al

[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9070: Affects Version/s: (was: HDFS-7285) > Allow fsck display pending replica location information for being-w

[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9070: Target Version/s: 2.8.0 (was: HDFS-7285) > Allow fsck display pending replica location information for being

[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876807#comment-14876807 ] Jing Zhao commented on HDFS-9040: - Good point about the GS bump! Some thinking along the li

<    1   2   3   4   5   6   7   8   9   10   >