[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp

2015-04-15 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8151: Reporter: Sushmitha Sreenivasan (was: Jing Zhao) > Always use snapshot path as source when invalid snapshot

[jira] [Updated] (HDFS-8151) Always use snapshot path as source when invalid snapshot names are used for diff based distcp

2015-04-15 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8151: Description: This is a bug reported by [~ssreenivasan]: HDFS-8036 makes the diff-based distcp use snapshot p

[jira] [Updated] (HDFS-8036) Use snapshot path as source when using snapshot diff report in DistCp

2015-04-15 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8036: Reporter: Sushmitha Sreenivasan (was: Jing Zhao) > Use snapshot path as source when using snapshot diff repo

[jira] [Commented] (HDFS-7369) Erasure coding: distribute recovery work for striped blocks to DataNode

2015-04-15 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497526#comment-14497526 ] Jing Zhao commented on HDFS-7369: - You need {{liveBlockIndices}} to reconstruct the ID of e

[jira] [Commented] (HDFS-7545) Data striping support in HDFS client

2015-04-16 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498871#comment-14498871 ] Jing Zhao commented on HDFS-7545: - Thanks Zhe! I will review the patch in HDFS-8033. > Dat

[jira] [Resolved] (HDFS-8166) DFSStripedOutputStream should not create empty blocks

2015-04-16 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-8166. - Resolution: Duplicate I confirmed with Nicholas. Closed this as dup. > DFSStripedOutputStream should not c

[jira] [Comment Edited] (HDFS-8145) Fix the editlog corruption exposed by failed TestAddStripedBlocks

2015-04-16 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499075#comment-14499075 ] Jing Zhao edited comment on HDFS-8145 at 4/17/15 1:02 AM: -- Initial

[jira] [Updated] (HDFS-8145) Fix the editlog corruption exposed by failed TestAddStripedBlocks

2015-04-16 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8145: Attachment: HDFS-8145.000.patch Initial patch to fix the above two issues. > Fix the editlog corruption expo

[jira] [Commented] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500889#comment-14500889 ] Jing Zhao commented on HDFS-7937: - Thanks Kai for updating the patch! I will review the pat

[jira] [Updated] (HDFS-8145) Fix the editlog corruption exposed by failed TestAddStripedBlocks

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8145: Attachment: HDFS-8145.001.patch Thanks for the review, Nicholas! Update the patch to address the comments. Th

[jira] [Reopened] (HDFS-8166) DFSStripedOutputStream should not create empty blocks

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao reopened HDFS-8166: - Assignee: Jing Zhao (was: Tsz Wo Nicholas Sze) Reopen the jira. We will separate out the fix for DFSStri

[jira] [Updated] (HDFS-8145) Fix the editlog corruption exposed by failed TestAddStripedBlocks

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8145: Description: {{TestAddStripedBlocks}} failed with some editlog corruption. Did some debugging, I can see at

[jira] [Updated] (HDFS-8166) DFSStripedOutputStream should not create empty blocks

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8166: Attachment: HDFS-8166.000.patch The 000 patch includes: # Fix {{DFSStripedOutputStream}} to avoid creating em

[jira] [Updated] (HDFS-8145) Fix the editlog corruption exposed by failed TestAddStripedBlocks

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8145: Attachment: HDFS-8145.002.patch Thanks for the review, Zhe! Update the patch to address your comments: # I ju

[jira] [Resolved] (HDFS-8166) DFSStripedOutputStream should not create empty blocks

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-8166. - Resolution: Fixed Fix Version/s: HDFS-7285 Thanks Nicholas for the review! I've committed this to th

[jira] [Updated] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7937: Attachment: HDFS-7937.9.patch The .8 patch looks pretty good to me. +1 Only nits is that maybe we can rename

[jira] [Resolved] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-7937. - Resolution: Fixed Fix Version/s: HDFS-7285 Hadoop Flags: Reviewed I've committed this to the fe

[jira] [Resolved] (HDFS-8145) Fix the editlog corruption exposed by failed TestAddStripedBlocks

2015-04-17 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-8145. - Resolution: Fixed Fix Version/s: HDFS-7285 I've committed this to the feature branch. Thanks for the

[jira] [Commented] (HDFS-8190) StripedBlockUtil.getInternalBlockLength may have overflow error

2015-04-20 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504026#comment-14504026 ] Jing Zhao commented on HDFS-8190: - Thanks for the fix, Nicholas. The patch looks pretty goo

[jira] [Commented] (HDFS-8190) StripedBlockUtil.getInternalBlockLength may have overflow error

2015-04-20 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504052#comment-14504052 ] Jing Zhao commented on HDFS-8190: - +1 for the new patch as well. > StripedBlockUtil.getInt

[jira] [Commented] (HDFS-8163) Using monotonicNow for block report scheduling causes test failures on recently restarted systems

2015-04-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505350#comment-14505350 ] Jing Zhao commented on HDFS-8163: - Thanks for working on this, [~arpitagarwal]! The patch l

[jira] [Commented] (HDFS-8200) Refactor FSDirStatAndListingOp

2015-04-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505416#comment-14505416 ] Jing Zhao commented on HDFS-8200: - The patch looks good to me. One minor is that we can als

[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout

2015-04-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505735#comment-14505735 ] Jing Zhao commented on HDFS-8033: - Thanks for working on this, Zhe! And thanks for the revi

[jira] [Commented] (HDFS-8176) Provide information about the snapshots compared in audit log

2015-04-22 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507562#comment-14507562 ] Jing Zhao commented on HDFS-8176: - Thanks for working on this, [~andreina]! The patch looks

[jira] [Commented] (HDFS-8218) Move classes that used by ClientProtocol into hdfs-client

2015-04-22 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507592#comment-14507592 ] Jing Zhao commented on HDFS-8218: - Looks like the patch only moves the classes, no internal

[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout

2015-04-22 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508177#comment-14508177 ] Jing Zhao commented on HDFS-8033: - Thanks for updating the patch, Zhe! Some comments on the

[jira] [Updated] (HDFS-8228) TestDFSStripedOutputStream#testFileSmallerThanOneStripe2 failed

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8228: Summary: TestDFSStripedOutputStream#testFileSmallerThanOneStripe2 failed (was: TestDFSStripedInputStream#tes

[jira] [Updated] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8228: Issue Type: Sub-task (was: Bug) Parent: HDFS-7285 > TestDFSStripedInputStream#testFileSmallerThanOne

[jira] [Created] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary

2015-04-23 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8233: --- Summary: Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary Key: HDFS-8233 URL: https://issues.apache.org/jira/browse/HDFS-8233

[jira] [Updated] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8233: Attachment: hdfs8233-HDFS-7285.000.patch When we call {{getCurrentBlockGroupBytes}}, the block object in each

[jira] [Updated] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8233: Description: Currently {{DFSStripedOutputStream#getCurrentBlockGroupBytes}} simply uses {{getBytesCurBlock}}

[jira] [Created] (HDFS-8235) Create DFSStripedInputStream in DFSClient#open

2015-04-23 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8235: --- Summary: Create DFSStripedInputStream in DFSClient#open Key: HDFS-8235 URL: https://issues.apache.org/jira/browse/HDFS-8235 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Commented] (HDFS-8223) Should calculate checksum for parity blocks in DFSStripedOutputStream

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509985#comment-14509985 ] Jing Zhao commented on HDFS-8223: - Thanks for fixing this, Yi! The patch looks good to me.

[jira] [Resolved] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-8233. - Resolution: Fixed Fix Version/s: HDFS-7285 Thanks for the review, Nicholas! I've committed this. >

[jira] [Updated] (HDFS-8223) Should calculate checksum for parity blocks in DFSStripedOutputStream

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8223: Attachment: HDFS-8223.rebase.patch Upload the rebased patch. > Should calculate checksum for parity blocks i

[jira] [Resolved] (HDFS-8223) Should calculate checksum for parity blocks in DFSStripedOutputStream

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-8223. - Resolution: Fixed Fix Version/s: HDFS-7285 Hadoop Flags: Reviewed I've committed this to the fe

[jira] [Updated] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8228: Summary: TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed (was: TestDFSStripedOutputStream#tes

[jira] [Commented] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510063#comment-14510063 ] Jing Zhao commented on HDFS-8228: - bq. It did not fail in my machine. My bad. I thought th

[jira] [Commented] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510123#comment-14510123 ] Jing Zhao commented on HDFS-8228: - Spent some time debugging this. Looks like the cause of

[jira] [Updated] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8228: Attachment: HDFS-8228.000.patch Upload a patch to fix. A new test case that can reproduce the issue is also

[jira] [Commented] (HDFS-8239) Erasure coding: [bug] should always allocate unique striped block group IDs

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510498#comment-14510498 ] Jing Zhao commented on HDFS-8239: - duplicated with HDFS-8228? > Erasure coding: [bug] shou

[jira] [Updated] (HDFS-8228) Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause block id conflicts

2015-04-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8228: Summary: Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause block id conflicts (was: TestD

[jira] [Updated] (HDFS-8228) Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause block id conflicts

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8228: Attachment: HDFS-8228.001.patch Thanks for the review, Nicholas and Zhe! Update the patch to refactor the cod

[jira] [Updated] (HDFS-8228) Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause block id conflicts

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8228: Attachment: HDFS-8228.001.patch Remove an used import. > Erasure Coding: SequentialBlockGroupIdGenerator#nex

[jira] [Commented] (HDFS-8235) Create DFSStripedInputStream in DFSClient#open

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510612#comment-14510612 ] Jing Zhao commented on HDFS-8235: - Sure, thanks for working on this, [~kaisasak]! > Create

[jira] [Updated] (HDFS-8235) Create DFSStripedInputStream in DFSClient#open

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8235: Assignee: Kai Sasaki (was: Jing Zhao) > Create DFSStripedInputStream in DFSClient#open > ---

[jira] [Commented] (HDFS-8244) HDFS Custom Storage Tier Policies

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511285#comment-14511285 ] Jing Zhao commented on HDFS-8244: - Thanks for reporting this, [~harisekhon]! Actually in HD

[jira] [Updated] (HDFS-7076) Allow users to define custom storage policies

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7076: Summary: Allow users to define custom storage policies (was: Persist Block Storage Policy in FSImage and Edi

[jira] [Updated] (HDFS-8176) Record from/to snapshots in audit log for snapshot diff report

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8176: Summary: Record from/to snapshots in audit log for snapshot diff report (was: Provide information about the

[jira] [Updated] (HDFS-8176) Record from/to snapshots in audit log for snapshot diff report

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8176: Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: P

[jira] [Updated] (HDFS-8176) Record from/to snapshots in audit log for snapshot diff report

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8176: Component/s: snapshots > Record from/to snapshots in audit log for snapshot diff report > ---

[jira] [Updated] (HDFS-8176) Record from/to snapshots in audit log for snapshot diff report

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8176: Priority: Minor (was: Major) > Record from/to snapshots in audit log for snapshot diff report >

[jira] [Commented] (HDFS-8235) Create DFSStripedInputStream in DFSClient#open

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511401#comment-14511401 ] Jing Zhao commented on HDFS-8235: - The patch looks good to me overall. Some further optimiz

[jira] [Updated] (HDFS-7076) Allow users to define custom storage policies

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7076: Attachment: HDFS-7076.008.patch Rebase the patch. > Allow users to define custom storage policies >

[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511881#comment-14511881 ] Jing Zhao commented on HDFS-8033: - Thanks for updating the patch, Zhe! A very quick comment

[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511973#comment-14511973 ] Jing Zhao commented on HDFS-8033: - Thanks Zhe! The 006 patch looks pretty good to me. +1 >

[jira] [Comment Edited] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511973#comment-14511973 ] Jing Zhao edited comment on HDFS-8033 at 4/24/15 11:11 PM: --- Thank

[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512010#comment-14512010 ] Jing Zhao commented on HDFS-8033: - Yeah, in fact I think we do not need to extend TestDFSSt

[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout

2015-04-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512254#comment-14512254 ] Jing Zhao commented on HDFS-8033: - Thanks Zhe! +1 for the 007 patch. > Erasure coding: sta

[jira] [Created] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream

2015-04-27 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8272: --- Summary: Erasure Coding: simplify the retry logic in DFSStripedInputStream Key: HDFS-8272 URL: https://issues.apache.org/jira/browse/HDFS-8272 Project: Hadoop HDFS Is

[jira] [Updated] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream

2015-04-27 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8272: Attachment: h8272-HDFS-7285.000.patch The 000 patch made the following changes: # {{DFSStripedInputStream}} n

[jira] [Comment Edited] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream

2015-04-27 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515178#comment-14515178 ] Jing Zhao edited comment on HDFS-8272 at 4/27/15 10:58 PM: --- The 0

[jira] [Commented] (HDFS-8235) Create DFSStripedInputStream in DFSClient#open

2015-04-27 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515833#comment-14515833 ] Jing Zhao commented on HDFS-8235: - The 002 patch looks good to me. Some minors: # We can ad

[jira] [Commented] (HDFS-8269) getBlockLocations() does not resolve the .reserved path and generates incorrect edit logs when updating the atime

2015-04-27 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515870#comment-14515870 ] Jing Zhao commented on HDFS-8269: - Thanks for the fix, Haohui. The patch looks good to me.

[jira] [Created] (HDFS-8273) logSync() is called inside of write lock for delete op

2015-04-27 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8273: --- Summary: logSync() is called inside of write lock for delete op Key: HDFS-8273 URL: https://issues.apache.org/jira/browse/HDFS-8273 Project: Hadoop HDFS Issue Type: Bu

[jira] [Commented] (HDFS-8269) getBlockLocations() does not resolve the .reserved path and generates incorrect edit logs when updating the atime

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517555#comment-14517555 ] Jing Zhao commented on HDFS-8269: - The 001 patch looks good to me. But among the failed tes

[jira] [Commented] (HDFS-8273) logSync() is called inside of write lock for delete op

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517631#comment-14517631 ] Jing Zhao commented on HDFS-8273: - # We should also move the {{removeBlocks}} call out of t

[jira] [Updated] (HDFS-8235) Erasure Coding: Create DFSStripedInputStream in DFSClient#open

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8235: Summary: Erasure Coding: Create DFSStripedInputStream in DFSClient#open (was: Create DFSStripedInputStream i

[jira] [Resolved] (HDFS-8235) Erasure Coding: Create DFSStripedInputStream in DFSClient#open

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-8235. - Resolution: Fixed Fix Version/s: HDFS-7285 Hadoop Flags: Reviewed The .3 patch looks good to me

[jira] [Created] (HDFS-8280) Code Cleanup in DFSInputStream

2015-04-28 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8280: --- Summary: Code Cleanup in DFSInputStream Key: HDFS-8280 URL: https://issues.apache.org/jira/browse/HDFS-8280 Project: Hadoop HDFS Issue Type: Improvement Re

[jira] [Updated] (HDFS-8280) Code Cleanup in DFSInputStream

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8280: Attachment: HDFS-8280.000.patch > Code Cleanup in DFSInputStream > -- > >

[jira] [Commented] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518097#comment-14518097 ] Jing Zhao commented on HDFS-8272: - Thanks for the review, Zhe! Sure, I just separate the re

[jira] [Updated] (HDFS-8280) Code Cleanup in DFSInputStream

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8280: Status: Patch Available (was: Open) > Code Cleanup in DFSInputStream > -- > >

[jira] [Created] (HDFS-8281) Erasure Coding: implement parallel stateful reading for striped layout

2015-04-28 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8281: --- Summary: Erasure Coding: implement parallel stateful reading for striped layout Key: HDFS-8281 URL: https://issues.apache.org/jira/browse/HDFS-8281 Project: Hadoop HDFS

[jira] [Updated] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8272: Attachment: h8272-HDFS-7285.001.patch Thanks again for the review, Zhe! Update the patch to address your comm

[jira] [Commented] (HDFS-8273) logSync() is called inside of write lock for delete op

2015-04-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518492#comment-14518492 ] Jing Zhao commented on HDFS-8273: - +1 for the latest patch. Thanks for the fix, Haohui! >

[jira] [Commented] (HDFS-8270) create() always retried with hardcoded timeout when file already exists

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519769#comment-14519769 ] Jing Zhao commented on HDFS-8270: - I think we can use HDFS-6697 to make the lease soft and

[jira] [Commented] (HDFS-8283) DataStreamer cleanup and some minor improvement

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519802#comment-14519802 ] Jing Zhao commented on HDFS-8283: - Thanks for working on this, Nicholas! The patch looks pr

[jira] [Updated] (HDFS-8283) DataStreamer cleanup and some minor improvement

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8283: Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: P

[jira] [Commented] (HDFS-8269) getBlockLocations() does not resolve the .reserved path and generates incorrect edit logs when updating the atime

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519834#comment-14519834 ] Jing Zhao commented on HDFS-8269: - The 003 patch looks pretty good to me. +1. The failed te

[jira] [Updated] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream (stateful read)

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8272: Attachment: HDFS-8272.002.patch Thanks for the review, Zhe and Yi! I just merged HDFS-8280 into HDFS-7285 br

[jira] [Comment Edited] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream (stateful read)

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520095#comment-14520095 ] Jing Zhao edited comment on HDFS-8272 at 4/29/15 8:03 PM: -- Thanks

[jira] [Commented] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream (stateful read)

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520607#comment-14520607 ] Jing Zhao commented on HDFS-8272: - So far I do not know any specific tests for the retry lo

[jira] [Updated] (HDFS-8281) Erasure Coding: implement parallel stateful reading for striped layout

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8281: Attachment: HDFS-8281.000.patch Upload the initial patch for review. Still need to add more tests. The basic

[jira] [Created] (HDFS-8293) Erasure Coding: test the retry logic of DFSStripedInputStream

2015-04-29 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8293: --- Summary: Erasure Coding: test the retry logic of DFSStripedInputStream Key: HDFS-8293 URL: https://issues.apache.org/jira/browse/HDFS-8293 Project: Hadoop HDFS Issue

[jira] [Commented] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream (stateful read)

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520633#comment-14520633 ] Jing Zhao commented on HDFS-8272: - Yeah, I just created HDFS-8293 to track the test. > Era

[jira] [Commented] (HDFS-8281) Erasure Coding: implement parallel stateful reading for striped layout

2015-04-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520661#comment-14520661 ] Jing Zhao commented on HDFS-8281: - Thanks for the comment, Zhe! One scenario this stripe bu

[jira] [Created] (HDFS-8300) Fix unit test failures and findbugs warning caused by HDFS-8283

2015-04-30 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8300: --- Summary: Fix unit test failures and findbugs warning caused by HDFS-8283 Key: HDFS-8300 URL: https://issues.apache.org/jira/browse/HDFS-8300 Project: Hadoop HDFS Issu

[jira] [Commented] (HDFS-8283) DataStreamer cleanup and some minor improvement

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521891#comment-14521891 ] Jing Zhao commented on HDFS-8283: - bq. It seems this patch introduced a findbugs warning. I

[jira] [Commented] (HDFS-8298) HA: NameNode should not shut down completely without quorum, doesn't recover from temporary network outages

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521912#comment-14521912 ] Jing Zhao commented on HDFS-8298: - Thanks for reporting this, Hari. Currently the active NN

[jira] [Assigned] (HDFS-8300) Fix unit test failures and findbugs warning caused by HDFS-8283

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao reassigned HDFS-8300: --- Assignee: Jing Zhao > Fix unit test failures and findbugs warning caused by HDFS-8283 > --

[jira] [Updated] (HDFS-8300) Fix unit test failures and findbugs warning caused by HDFS-8283

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8300: Attachment: HDFS-8300.000.patch > Fix unit test failures and findbugs warning caused by HDFS-8283 > -

[jira] [Updated] (HDFS-8300) Fix unit test failures and findbugs warning caused by HDFS-8283

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8300: Affects Version/s: 2.8.0 > Fix unit test failures and findbugs warning caused by HDFS-8283 >

[jira] [Updated] (HDFS-8300) Fix unit test failures and findbugs warning caused by HDFS-8283

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8300: Status: Patch Available (was: Open) > Fix unit test failures and findbugs warning caused by HDFS-8283 >

[jira] [Updated] (HDFS-8300) Fix unit test failures and findbugs warning caused by HDFS-8283

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8300: Target Version/s: 2.8.0 > Fix unit test failures and findbugs warning caused by HDFS-8283 > -

[jira] [Updated] (HDFS-8281) Erasure Coding: implement parallel stateful reading for striped layout

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8281: Attachment: HDFS-8281-HDFS-7285.001.patch Thanks for the review, Nicholas! Update the patch to address your c

[jira] [Updated] (HDFS-8281) Erasure Coding: implement parallel stateful reading for striped layout

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8281: Attachment: HDFS-8281-HDFS-7285.001.patch Thanks Zhe! Rebase the patch. > Erasure Coding: implement parallel

[jira] [Created] (HDFS-8308) Erasure Coding: NameNode may get blocked in waitForLoadingFSImage() when loading editlog

2015-04-30 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-8308: --- Summary: Erasure Coding: NameNode may get blocked in waitForLoadingFSImage() when loading editlog Key: HDFS-8308 URL: https://issues.apache.org/jira/browse/HDFS-8308 Project: H

[jira] [Assigned] (HDFS-8289) DFSStripedOutputStream uses an additional rpc all to getErasureCodingInfo

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao reassigned HDFS-8289: --- Assignee: Jing Zhao (was: Tsz Wo Nicholas Sze) > DFSStripedOutputStream uses an additional rpc all to

[jira] [Updated] (HDFS-8308) Erasure Coding: NameNode may get blocked in waitForLoadingFSImage() when loading editlog

2015-04-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8308: Attachment: HDFS-8308.000.patch Upload a patch to fix. The main fix is actually just one line: {code} -

<    9   10   11   12   13   14   15   16   17   18   >