[jira] [Commented] (HDFS-7117) Not all datanodes are displayed on the namenode http tab

2015-01-21 Thread Vinayakumar B (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287063#comment-14287063 ] Vinayakumar B commented on HDFS-7117: - There could be one possibility. Are you seeing

[jira] [Commented] (HDFS-3107) HDFS truncate

2015-01-21 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287050#comment-14287050 ] Yi Liu commented on HDFS-3107: -- I just noticed we have not exposed truncate API for Web HDFS a

[jira] [Resolved] (HDFS-7117) Not all datanodes are displayed on the namenode http tab

2015-01-21 Thread Vinayakumar B (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B resolved HDFS-7117. - Resolution: Invalid Fix Version/s: (was: 2.4.0) This issue is not present in current tru

[jira] [Created] (HDFS-7656) Expose truncate API for HDFS httpfs

2015-01-21 Thread Yi Liu (JIRA)
Yi Liu created HDFS-7656: Summary: Expose truncate API for HDFS httpfs Key: HDFS-7656 URL: https://issues.apache.org/jira/browse/HDFS-7656 Project: Hadoop HDFS Issue Type: Sub-task Report

[jira] [Created] (HDFS-7655) Expose truncate API for Web HDFS

2015-01-21 Thread Yi Liu (JIRA)
Yi Liu created HDFS-7655: Summary: Expose truncate API for Web HDFS Key: HDFS-7655 URL: https://issues.apache.org/jira/browse/HDFS-7655 Project: Hadoop HDFS Issue Type: Sub-task Reporter:

[jira] [Updated] (HDFS-7447) Number of maximum Acl entries on a File/Folder should be made user configurable than hardcoding .

2015-01-21 Thread Vinayakumar B (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-7447: Description: By default on creating a folder1 will have 6 acl entries . On top of that assigning acl

[jira] [Commented] (HDFS-7633) When Datanode has too many blocks, BlockPoolSliceScanner.getNewBlockScanTime throws IllegalArgumentException

2015-01-21 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287002#comment-14287002 ] Hadoop QA commented on HDFS-7633: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)

2015-01-21 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287001#comment-14287001 ] Xiaoyu Yao commented on HDFS-7584: -- Thanks [~zhz] for the feedback. bq. 1. This is an in

[jira] [Comment Edited] (HDFS-7392) org.apache.hadoop.hdfs.DistributedFileSystem open invalid URI forever

2015-01-21 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286987#comment-14286987 ] Yi Liu edited comment on HDFS-7392 at 1/22/15 5:47 AM: --- [~peschd], I

[jira] [Commented] (HDFS-7392) org.apache.hadoop.hdfs.DistributedFileSystem open invalid URI forever

2015-01-21 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286987#comment-14286987 ] Yi Liu commented on HDFS-7392: -- [~peschd], I think this issue is not important, the reasons ar

[jira] [Updated] (HDFS-7056) Snapshot support for truncate

2015-01-21 Thread Plamen Jeliazkov (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-7056: --- Attachment: HDFS-7056.15_branch2.patch Attaching patch for branch-2 port. Tests passed locally

[jira] [Updated] (HDFS-3107) HDFS truncate

2015-01-21 Thread Plamen Jeliazkov (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3107: --- Attachment: HDFS-3107.15_branch2.patch > HDFS truncate > - > > Key:

[jira] [Updated] (HDFS-3107) HDFS truncate

2015-01-21 Thread Plamen Jeliazkov (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3107: --- Attachment: (was: HDFS-3107.15_branch2.patch) > HDFS truncate > - > >

[jira] [Commented] (HDFS-3443) Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer

2015-01-21 Thread Vinayakumar B (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286913#comment-14286913 ] Vinayakumar B commented on HDFS-3443: - Updated the title as per CHANGES.txt > Fix NPE

[jira] [Updated] (HDFS-3443) Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer

2015-01-21 Thread Vinayakumar B (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-3443: Summary: Fix NPE when namenode transition to active during startup by adding checkNNStartup() in Name

[jira] [Commented] (HDFS-3443) Unable to catch up edits during standby to active switch due to NPE

2015-01-21 Thread Vinayakumar B (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286911#comment-14286911 ] Vinayakumar B commented on HDFS-3443: - Thanks [~szetszwo] for reviews and commit > Una

[jira] [Updated] (HDFS-3107) HDFS truncate

2015-01-21 Thread Plamen Jeliazkov (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3107: --- Attachment: HDFS-3107.15_branch2.patch Attaching patch for branch-2 port. Ran tests locally wit

[jira] [Commented] (HDFS-7633) When Datanode has too many blocks, BlockPoolSliceScanner.getNewBlockScanTime throws IllegalArgumentException

2015-01-21 Thread Walter Su (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286901#comment-14286901 ] Walter Su commented on HDFS-7633: - {color:red}-1 overall{color}. {color:green}+1 @a

[jira] [Commented] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286896#comment-14286896 ] Yi Liu commented on HDFS-7430: -- {quote} As per Nicholas' suggestion, let's let this soak for a

[jira] [Updated] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Milan Desai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Desai updated HDFS-7647: -- Attachment: HDFS-7647-2.patch Attaching a patch that creates a class, DatanodeInfosWithStorage, that exte

[jira] [Updated] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Milan Desai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Desai updated HDFS-7647: -- Status: Open (was: Patch Available) > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not St

[jira] [Work started] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Milan Desai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-7647 started by Milan Desai. - > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -

[jira] [Commented] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286885#comment-14286885 ] Hudson commented on HDFS-7430: -- FAILURE: Integrated in Hadoop-trunk-Commit #6911 (See [https:

[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286883#comment-14286883 ] Yi Liu commented on HDFS-7647: -- {quote} We should fix LocatedBlock to group the three fields i

[jira] [Updated] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7430: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch A

[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286858#comment-14286858 ] Hadoop QA commented on HDFS-7647: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Commented] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286857#comment-14286857 ] Colin Patrick McCabe commented on HDFS-7430: TestBalancer#testUnknownDatanode f

[jira] [Created] (HDFS-7654) TestFileTruncate#testTruncateEditLogLoad fails intermittently

2015-01-21 Thread Colin Patrick McCabe (JIRA)
Colin Patrick McCabe created HDFS-7654: -- Summary: TestFileTruncate#testTruncateEditLogLoad fails intermittently Key: HDFS-7654 URL: https://issues.apache.org/jira/browse/HDFS-7654 Project: Hadoop

[jira] [Commented] (HDFS-7653) Block Readers and Writers used in both client side and datanode side

2015-01-21 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286851#comment-14286851 ] Li Bo commented on HDFS-7653: - This patch includes block readers and writers that will be used

[jira] [Updated] (HDFS-7653) Block Readers and Writers used in both client side and datanode side

2015-01-21 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-7653: Attachment: BlockReadersWriters.patch This patch includes block readers and writers that will be used by client and

[jira] [Updated] (HDFS-7633) When Datanode has too many blocks, BlockPoolSliceScanner.getNewBlockScanTime throws IllegalArgumentException

2015-01-21 Thread Walter Su (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-7633: Attachment: HDFS-7633.patch regenerating patch with 'git diff' > When Datanode has too many blocks, BlockPoo

[jira] [Updated] (HDFS-7633) When Datanode has too many blocks, BlockPoolSliceScanner.getNewBlockScanTime throws IllegalArgumentException

2015-01-21 Thread Walter Su (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-7633: Attachment: (was: h7633_20150116.patch) > When Datanode has too many blocks, BlockPoolSliceScanner.getNew

[jira] [Commented] (HDFS-3519) Checkpoint upload may interfere with a concurrent saveNamespace

2015-01-21 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286839#comment-14286839 ] Hadoop QA commented on HDFS-3519: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Commented] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286837#comment-14286837 ] Andrew Wang commented on HDFS-7430: --- +1 again if you need it, this patch has been through

[jira] [Updated] (HDFS-3519) Checkpoint upload may interfere with a concurrent saveNamespace

2015-01-21 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-3519: -- Attachment: HDFS-3519-branch-2.patch Thanks, Chris. Here is the patch for branch-2. > Checkpoint upload may inte

[jira] [Updated] (HDFS-7653) Block Readers and Writers used in both client side and datanode side

2015-01-21 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-7653: Remaining Estimate: (was: 336h) Original Estimate: (was: 336h) > Block Readers and Writers used in both

[jira] [Commented] (HDFS-7353) Raw Erasure Coder API for concrete encoding and decoding

2015-01-21 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286827#comment-14286827 ] Kai Zheng commented on HDFS-7353: - Sure. Let me update my patch. > Raw Erasure Coder API f

[jira] [Created] (HDFS-7653) Block Readers and Writers used in both client side and datanode side

2015-01-21 Thread Li Bo (JIRA)
Li Bo created HDFS-7653: --- Summary: Block Readers and Writers used in both client side and datanode side Key: HDFS-7653 URL: https://issues.apache.org/jira/browse/HDFS-7653 Project: Hadoop HDFS Issue T

[jira] [Commented] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286804#comment-14286804 ] Hadoop QA commented on HDFS-7430: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Commented] (HDFS-3689) Add support for variable length block

2015-01-21 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286800#comment-14286800 ] Hadoop QA commented on HDFS-3689: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Commented] (HDFS-7339) Allocating and persisting block groups in NameNode

2015-01-21 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286776#comment-14286776 ] Hadoop QA commented on HDFS-7339: - {color:green}+1 overall{color}. Here are the results of

[jira] [Updated] (HDFS-4173) If the NameNode has already been formatted, but a QuroumJournal has not, auto-format it on startup

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-4173: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) We currently req

[jira] [Assigned] (HDFS-147) Stack trace on spaceQuota excced .

2015-01-21 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDFS-147: --- Assignee: Xiaoyu Yao > Stack trace on spaceQuota excced . > -- > >

[jira] [Commented] (HDFS-147) Stack trace on spaceQuota excced .

2015-01-21 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286711#comment-14286711 ] Xiaoyu Yao commented on HDFS-147: - I just checked and it still can repro with space quota bu

[jira] [Updated] (HDFS-7611) deleteSnapshot and delete of a file can leave orphaned blocks in the blocksMap on NameNode restart.

2015-01-21 Thread Konstantin Shvachko (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-7611: -- Component/s: namenode Description: If quotas are enabled a combination of

[jira] [Commented] (HDFS-7611) TestFileTruncate.testTruncateEditLogLoad times out waiting for Mini HDFS Cluster to start

2015-01-21 Thread Konstantin Shvachko (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286697#comment-14286697 ] Konstantin Shvachko commented on HDFS-7611: --- Byron, good job investigating and re

[jira] [Resolved] (HDFS-3458) Convert Forrest docs to APT

2015-01-21 Thread Allen Wittenauer (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HDFS-3458. Resolution: Fixed Fix Version/s: 2.0.0-alpha Target Version/s: (was: ) Clo

[jira] [Updated] (HDFS-7611) TestFileTruncate.testTruncateEditLogLoad times out waiting for Mini HDFS Cluster to start

2015-01-21 Thread Byron Wong (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Byron Wong updated HDFS-7611: - Attachment: blocksNotDeletedTest.patch Attached blocksNotDeletedTest.patch. Includes a test that reproduces

[jira] [Commented] (HDFS-7575) NameNode not handling heartbeats properly after HDFS-2832

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286637#comment-14286637 ] Colin Patrick McCabe commented on HDFS-7575: bq. I disagree. The implementation

[jira] [Commented] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286617#comment-14286617 ] Colin Patrick McCabe commented on HDFS-7430: bq. Then, could you reuse the old

[jira] [Commented] (HDFS-6874) Add GET_BLOCK_LOCATIONS operation to HttpFS

2015-01-21 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286613#comment-14286613 ] Charles Lamb commented on HDFS-6874: [~lianggz], Thanks for working on this. In gener

[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Milan Desai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286594#comment-14286594 ] Milan Desai commented on HDFS-7647: --- Hi [~arpitagarwal], sounds good - I will give it a s

[jira] [Commented] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286588#comment-14286588 ] Tsz Wo Nicholas Sze commented on HDFS-7430: --- > It is fair to call this a rewrite

[jira] [Updated] (HDFS-7430) Rewrite the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-7430: -- Summary: Rewrite the BlockScanner to use O(1) memory and use multiple threads (was: Refa

[jira] [Commented] (HDFS-7575) NameNode not handling heartbeats properly after HDFS-2832

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286583#comment-14286583 ] Tsz Wo Nicholas Sze commented on HDFS-7575: --- For the so called practical points y

[jira] [Commented] (HDFS-7575) NameNode not handling heartbeats properly after HDFS-2832

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286574#comment-14286574 ] Tsz Wo Nicholas Sze commented on HDFS-7575: --- > What's important is what was imple

[jira] [Commented] (HDFS-7645) Rolling upgrade is restoring blocks from trash multiple times

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286576#comment-14286576 ] Colin Patrick McCabe commented on HDFS-7645: bq. DN layout changes will be rare

[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286567#comment-14286567 ] Arpit Agarwal commented on HDFS-7647: - Hi Milan, while this solution would work I'd pre

[jira] [Updated] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Milan Desai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Desai updated HDFS-7647: -- Status: Patch Available (was: Open) > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not St

[jira] [Updated] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Milan Desai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Desai updated HDFS-7647: -- Description: DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBloc

[jira] [Updated] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs

2015-01-21 Thread Milan Desai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Desai updated HDFS-7647: -- Summary: DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs (was: DatanodeManage

[jira] [Updated] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeIDs but not StorageIDs

2015-01-21 Thread Milan Desai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Desai updated HDFS-7647: -- Attachment: HDFS-7647.patch Thanks [~shv] and [~arpitagarwal]. I was unable to use the index arrays metho

[jira] [Updated] (HDFS-3519) Checkpoint upload may interfere with a concurrent saveNamespace

2015-01-21 Thread Chris Nauroth (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-3519: Hadoop Flags: Reviewed +1 for the patch. Ming, thank you for incorporating the feedback. Could you

[jira] [Updated] (HDFS-3519) Checkpoint upload may interfere with a concurrent saveNamespace

2015-01-21 Thread Chris Nauroth (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-3519: Target Version/s: 2.7.0 > Checkpoint upload may interfere with a concurrent saveNamespace > -

[jira] [Updated] (HDFS-7584) Enable Quota Support for Storage Types (SSD)

2015-01-21 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-7584: - Attachment: HDFS-7584.0.patch > Enable Quota Support for Storage Types (SSD) > ---

[jira] [Commented] (HDFS-7575) NameNode not handling heartbeats properly after HDFS-2832

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286527#comment-14286527 ] Colin Patrick McCabe commented on HDFS-7575: bq. So, you claim that the current

[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)

2015-01-21 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286520#comment-14286520 ] Zhe Zhang commented on HDFS-7584: - This is a very useful feature and thanks [~xyao] for ini

[jira] [Commented] (HDFS-7430) Refactor the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286495#comment-14286495 ] Colin Patrick McCabe commented on HDFS-7430: It is fair to call this a rewrite

[jira] [Updated] (HDFS-7430) Refactor the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7430: --- Attachment: HDFS-7430.012.patch > Refactor the BlockScanner to use O(1) memory and use

[jira] [Commented] (HDFS-7339) Allocating and persisting block groups in NameNode

2015-01-21 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286473#comment-14286473 ] Zhe Zhang commented on HDFS-7339: - Thanks for review [~vinayrpet], it was very helpful. The

[jira] [Updated] (HDFS-7339) Allocating and persisting block groups in NameNode

2015-01-21 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7339: Attachment: HDFS-7339-006.patch Addressing Vinay's comments. > Allocating and persisting block groups in Nam

[jira] [Commented] (HDFS-7611) TestFileTruncate.testTruncateEditLogLoad times out waiting for Mini HDFS Cluster to start

2015-01-21 Thread Byron Wong (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286458#comment-14286458 ] Byron Wong commented on HDFS-7611: -- Found it. The problem occurs in how we do {{FSImage$l

[jira] [Updated] (HDFS-3689) Add support for variable length block

2015-01-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.008.patch > Add support for variable length block > ---

[jira] [Commented] (HDFS-3689) Add support for variable length block

2015-01-21 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286436#comment-14286436 ] Hadoop QA commented on HDFS-3689: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Updated] (HDFS-3689) Add support for variable length block

2015-01-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.008.patch Thanks for the comments, Colin. To use CreateFlag is a good suggestion. Updat

[jira] [Commented] (HDFS-7430) Refactor the BlockScanner to use O(1) memory and use multiple threads

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286408#comment-14286408 ] Tsz Wo Nicholas Sze commented on HDFS-7430: --- It seems that the entire BlockScanne

[jira] [Commented] (HDFS-6833) DirectoryScanner should not register a deleting block with memory of DataNode

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286390#comment-14286390 ] Tsz Wo Nicholas Sze commented on HDFS-6833: --- Patch looks good in general. Some c

[jira] [Commented] (HDFS-7566) Remove obsolete entries from hdfs-default.xml

2015-01-21 Thread Ray Chiang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286284#comment-14286284 ] Ray Chiang commented on HDFS-7566: -- RE: findbugs No code changes in this patch. RE: Fail

[jira] [Commented] (HDFS-7353) Raw Erasure Coder API for concrete encoding and decoding

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286264#comment-14286264 ] Tsz Wo Nicholas Sze commented on HDFS-7353: --- > ... has potential to contain more

[jira] [Commented] (HDFS-7548) Corrupt block reporting delayed until datablock scanner thread detects it

2015-01-21 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286238#comment-14286238 ] Hudson commented on HDFS-7548: -- FAILURE: Integrated in Hadoop-trunk-Commit #6906 (See [https:

[jira] [Commented] (HDFS-7575) NameNode not handling heartbeats properly after HDFS-2832

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286230#comment-14286230 ] Tsz Wo Nicholas Sze commented on HDFS-7575: --- > Can you response to the practical

[jira] [Updated] (HDFS-7548) Corrupt block reporting delayed until datablock scanner thread detects it

2015-01-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-7548: - Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was:

[jira] [Commented] (HDFS-7548) Corrupt block reporting delayed until datablock scanner thread detects it

2015-01-21 Thread Rushabh S Shah (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286224#comment-14286224 ] Rushabh S Shah commented on HDFS-7548: -- [~daryn] and [~kihwal] Thanks for the comment

[jira] [Commented] (HDFS-7548) Corrupt block reporting delayed until datablock scanner thread detects it

2015-01-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286221#comment-14286221 ] Kihwal Lee commented on HDFS-7548: -- +1 the patch looks good. The test case is failing even

[jira] [Commented] (HDFS-7575) NameNode not handling heartbeats properly after HDFS-2832

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286217#comment-14286217 ] Tsz Wo Nicholas Sze commented on HDFS-7575: --- > This patch does change the layout

[jira] [Commented] (HDFS-5183) Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a global view in choosing DN storage for replica.

2015-01-21 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286183#comment-14286183 ] Arpit Agarwal commented on HDFS-5183: - Also thanks for filing this and the suggested so

[jira] [Created] (HDFS-7652) Process block reports for erasure coded blocks

2015-01-21 Thread Zhe Zhang (JIRA)
Zhe Zhang created HDFS-7652: --- Summary: Process block reports for erasure coded blocks Key: HDFS-7652 URL: https://issues.apache.org/jira/browse/HDFS-7652 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Resolved] (HDFS-5183) Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a global view in choosing DN storage for replica.

2015-01-21 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5183. - Resolution: Implemented Hadoop Flags: (was: Incompatible change) Resolving this as Implemen

[jira] [Commented] (HDFS-7575) NameNode not handling heartbeats properly after HDFS-2832

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286169#comment-14286169 ] Colin Patrick McCabe commented on HDFS-7575: This patch does change the layout

[jira] [Commented] (HDFS-7548) Corrupt block reporting delayed until datablock scanner thread detects it

2015-01-21 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286164#comment-14286164 ] Hadoop QA commented on HDFS-7548: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Updated] (HDFS-3443) Unable to catch up edits during standby to active switch due to NPE

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-3443: -- Resolution: Fixed Fix Version/s: 2.6.1 Status: Resolved (was: Patch Ava

[jira] [Resolved] (HDFS-5423) Verify initializations of LocatedBlock/RecoveringBlock

2015-01-21 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5423. - Resolution: Invalid Resolving as Invalid. I don't believe any work is remaining here. > Verify ini

[jira] [Updated] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances

2015-01-21 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7496: --- Fix Version/s: (was: 3.0.0) 2.7.0 committed to 2.7. Thanks. >

[jira] [Commented] (HDFS-3443) Unable to catch up edits during standby to active switch due to NPE

2015-01-21 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286148#comment-14286148 ] Hudson commented on HDFS-3443: -- FAILURE: Integrated in Hadoop-trunk-Commit #6904 (See [https:

[jira] [Comment Edited] (HDFS-7645) Rolling upgrade is restoring blocks from trash multiple times

2015-01-21 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286128#comment-14286128 ] Arpit Agarwal edited comment on HDFS-7645 at 1/21/15 7:34 PM: --

[jira] [Commented] (HDFS-7645) Rolling upgrade is restoring blocks from trash multiple times

2015-01-21 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286128#comment-14286128 ] Arpit Agarwal commented on HDFS-7645: - The first restore was by design when the rolling

[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)

2015-01-21 Thread Haohui Mai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286085#comment-14286085 ] Haohui Mai commented on HDFS-7037: -- Note that distcp over webhdfs has the same issue as it

[jira] [Updated] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances

2015-01-21 Thread Lei (Eddy) Xu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-7496: Attachment: HDFS-7496-branch-2.000.patch Uploaded for branch-2 > Fix FsVolume removal race condition

[jira] [Updated] (HDFS-3443) Unable to catch up edits during standby to active switch due to NPE

2015-01-21 Thread Tsz Wo Nicholas Sze (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-3443: -- Hadoop Flags: Reviewed +1 patch looks good > Unable to catch up edits during standby to

[jira] [Commented] (HDFS-7406) SimpleHttpProxyHandler puts incorrect "Connection: Close" header

2015-01-21 Thread Haohui Mai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286074#comment-14286074 ] Haohui Mai commented on HDFS-7406: -- This is not an incompatible change since it has not be

  1   2   >