[jira] [Work started] (HDFS-8968) New benchmark throughput tool for striping erasure coding

2015-08-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-8968 started by Rui Li. > New benchmark throughput tool for striping erasure coding >

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2015-08-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723022#comment-14723022 ] Rui Li commented on HDFS-7285: -- [~brahmareddy] - Yeah I'm working on the two issues to support parallel

[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl

2015-08-31 Thread kanaka kumar avvaru (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723045#comment-14723045 ] kanaka kumar avvaru commented on HDFS-8836: --- Thanks [~jfilipiak] I missed to observe stats in

[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones

2015-08-31 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723063#comment-14723063 ] Zhe Zhang commented on HDFS-8833: - Thanks Walter for the feedback. Based on the above I think we have

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2015-08-31 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723066#comment-14723066 ] Zhe Zhang commented on HDFS-7285: - Thanks Walter! I updated the {{getBlockCollection}} in my personal

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2015-08-31 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723068#comment-14723068 ] Zhe Zhang commented on HDFS-7285: - Thanks Rui! I guess you are referring to HDFS-8968? > Erasure Coding

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2015-08-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723077#comment-14723077 ] Rui Li commented on HDFS-7285: -- Yep. We intend to create a throughput benchmark tool in HDFS-8968.

[jira] [Commented] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724610#comment-14724610 ] Hudson commented on HDFS-8990: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2254 (See

[jira] [Commented] (HDFS-8984) Move replication queues related methods in FSNamesystem to BlockManager

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724641#comment-14724641 ] Hadoop QA commented on HDFS-8984: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HDFS-8946) Improve choosing datanode storage for block placement

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724602#comment-14724602 ] Hudson commented on HDFS-8946: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1058 (See

[jira] [Commented] (HDFS-8808) dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724603#comment-14724603 ] Hadoop QA commented on HDFS-8808: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Updated] (HDFS-8937) Fix the exception when set replication to Erasure Coding files

2015-08-31 Thread GAO Rui (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-8937: -- Attachment: HDFS-8937-HDFS-7285.00.patch Thanks [~jingzhao], [~zhz]. I have attached an patch to add a TODO

[jira] [Updated] (HDFS-8937) Fix the exception when set replication to Erasure Coding files

2015-08-31 Thread GAO Rui (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-8937: -- Status: Patch Available (was: Open) > Fix the exception when set replication to Erasure Coding files >

[jira] [Commented] (HDFS-8995) Flaw in registration bookeeping can make DN die on reconnect

2015-08-31 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723497#comment-14723497 ] Kihwal Lee commented on HDFS-8995: -- {noformat} 2018-09-26 12:15:08,497 WARN datanode.DataNode: Block pool

[jira] [Commented] (HDFS-8995) Flaw in registration bookeeping can make DN die on reconnect

2015-08-31 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723500#comment-14723500 ] Kihwal Lee commented on HDFS-8995: -- [~daryn] did further analysis: {panel} It's a bug during

[jira] [Updated] (HDFS-8995) Flaw in registration bookeeping can make DN die on reconnect

2015-08-31 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8995: - Attachment: HDFS-8995.patch > Flaw in registration bookeeping can make DN die on reconnect >

[jira] [Commented] (HDFS-8946) Improve choosing datanode storage for block placement

2015-08-31 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723589#comment-14723589 ] Kihwal Lee commented on HDFS-8946: -- +1 the patch looks good. > Improve choosing datanode storage for

[jira] [Updated] (HDFS-4957) NameNode failover should not fail because a DNS entry for a quorum node cannot be resolved

2015-08-31 Thread Eric Walk (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Walk updated HDFS-4957: Affects Version/s: 2.6.0 > NameNode failover should not fail because a DNS entry for a quorum node > cannot

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723831#comment-14723831 ] Colin Patrick McCabe commented on HDFS-8965: bq. I think it will be easy to understand code

[jira] [Updated] (HDFS-7596) NameNode should prune dead storages from storageMap

2015-08-31 Thread Vinod Kumar Vavilapalli (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7596: -- Fix Version/s: 2.6.1 [~sjlee0] backported this to 2.6.1. I just pushed the

[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones

2015-08-31 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723837#comment-14723837 ] Jing Zhao commented on HDFS-8833: - Thanks for all the discussion! Currently I think it's fine to set an EC

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723839#comment-14723839 ] Colin Patrick McCabe commented on HDFS-8965: bq. Currently jenkins report will not show this

[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets

2015-08-31 Thread kanaka kumar avvaru (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Attachment: HDFS-8894-02.patch Updated patch to fix white space errors > Set

[jira] [Updated] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8965: --- Attachment: HDFS-8965.006.patch > Harden edit log reading code against out of memory

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723852#comment-14723852 ] Jing Zhao commented on HDFS-8980: - Sure. I will take a look at HDFS-8763. > Remove unnecessary block

[jira] [Updated] (HDFS-8477) Missing ZKFC config keys can be added to hdfs-default.xml and HDFSHighAvailabilityWithQJM.md

2015-08-31 Thread kanaka kumar avvaru (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8477: -- Attachment: HDFS-8477-01.patch > Missing ZKFC config keys can be added to

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723860#comment-14723860 ] Hudson commented on HDFS-8980: -- FAILURE: Integrated in Hadoop-trunk-Commit #8372 (See

[jira] [Updated] (HDFS-8477) ZKFC port config key can be described in hdfs-default.xml

2015-08-31 Thread kanaka kumar avvaru (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8477: -- Summary: ZKFC port config key can be described in hdfs-default.xml (was: Missing ZKFC

[jira] [Updated] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8980: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was:

[jira] [Updated] (HDFS-8477) ZKFC port config key can be described in hdfs-default.xml

2015-08-31 Thread kanaka kumar avvaru (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8477: -- Description: Right now ZKFC port is not documented anywhere. So, it is difficult to

[jira] [Updated] (HDFS-8836) Skip newline on empty files with getMerge -nl

2015-08-31 Thread kanaka kumar avvaru (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8836: -- Attachment: HDFS-8836-03.patch > Skip newline on empty files with getMerge -nl >

[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl

2015-08-31 Thread kanaka kumar avvaru (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723879#comment-14723879 ] kanaka kumar avvaru commented on HDFS-8836: --- Updated patch for above review comment and check

[jira] [Updated] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files

2015-08-31 Thread kanaka kumar avvaru (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-6955: -- Attachment: HDFS-6955-04.patch Updated patch for the check style issue reported > DN

[jira] [Updated] (HDFS-7533) Datanode sometimes does not shutdown on receiving upgrade shutdown command

2015-08-31 Thread Vinod Kumar Vavilapalli (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7533: -- Fix Version/s: 2.6.1 [~sjlee0] backported this to 2.6.1. I just pushed the

[jira] [Commented] (HDFS-8964) Provide max TxId when validating in-progress edit log files

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723950#comment-14723950 ] Colin Patrick McCabe commented on HDFS-8964: Thank you, [~zhz]. JNStorage: {{import

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723966#comment-14723966 ] Hudson commented on HDFS-8980: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1056 (See

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723992#comment-14723992 ] Hudson commented on HDFS-8980: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2272 (See

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723998#comment-14723998 ] Hadoop QA commented on HDFS-8965: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Updated] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8965: --- Attachment: HDFS-8965.007.patch missed a skip() > Harden edit log reading code

[jira] [Commented] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Haohui Mai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724014#comment-14724014 ] Haohui Mai commented on HDFS-8990: -- +1. I'll commit it shortly. > Move RemoteBlockReader to hdfs-client

[jira] [Commented] (HDFS-8985) Restarted namenode suffer from block report storm

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724012#comment-14724012 ] Colin Patrick McCabe commented on HDFS-8985: Seems like a duplicate of HDFS-7923 > Restarted

[jira] [Commented] (HDFS-5995) TestFSEditLogLoader#testValidateEditLogWithCorruptBody gets OutOfMemoryError and dumps heap.

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724022#comment-14724022 ] Colin Patrick McCabe commented on HDFS-5995: Update: So, it turns out that the NameNode or

[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724027#comment-14724027 ] Hadoop QA commented on HDFS-8836: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Updated] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Haohui Mai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-8990: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved

[jira] [Commented] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724059#comment-14724059 ] Colin Patrick McCabe commented on HDFS-8982: [~wheat9]: it seems to me that it should be

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724066#comment-14724066 ] Hudson commented on HDFS-8980: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #329 (See

[jira] [Commented] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724080#comment-14724080 ] Hudson commented on HDFS-8990: -- FAILURE: Integrated in Hadoop-trunk-Commit #8373 (See

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724083#comment-14724083 ] Hudson commented on HDFS-8980: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #323 (See

[jira] [Commented] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

2015-08-31 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724094#comment-14724094 ] Jing Zhao commented on HDFS-8982: - Actually I think here it's better to use {{getFileReplication}} instead

[jira] [Comment Edited] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

2015-08-31 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724094#comment-14724094 ] Jing Zhao edited comment on HDFS-8982 at 8/31/15 9:44 PM: -- Actually I think for

[jira] [Commented] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

2015-08-31 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724102#comment-14724102 ] Zhe Zhang commented on HDFS-8982: - Thanks Colin and Jing for the discussion. Yes my intention was for us

[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724106#comment-14724106 ] Hadoop QA commented on HDFS-6955: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Created] (HDFS-8996) Remove {{scanEditLog}}

2015-08-31 Thread Zhe Zhang (JIRA)
Zhe Zhang created HDFS-8996: --- Summary: Remove {{scanEditLog}} Key: HDFS-8996 URL: https://issues.apache.org/jira/browse/HDFS-8996 Project: Hadoop HDFS Issue Type: Bug Components:

[jira] [Updated] (HDFS-8996) Remove redundant scanEditLog code

2015-08-31 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8996: Summary: Remove redundant scanEditLog code (was: Remove {{scanEditLog}}) > Remove redundant scanEditLog

[jira] [Updated] (HDFS-7575) Upgrade should generate a unique storage ID for each volume

2015-08-31 Thread Vinod Kumar Vavilapalli (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7575: -- Fix Version/s: 2.6.1 [~sjlee0] backported this to 2.6.1. I just pushed the

[jira] [Updated] (HDFS-8964) Provide max TxId when validating in-progress edit log files

2015-08-31 Thread Zhe Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8964: Attachment: HDFS-8964.03.patch Thanks Colin for the review! Updating the patch to add {{lastSyncTxId}} to

[jira] [Commented] (HDFS-8477) ZKFC port config key can be described in hdfs-default.xml

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724139#comment-14724139 ] Hadoop QA commented on HDFS-8477: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724146#comment-14724146 ] Hadoop QA commented on HDFS-8894: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724147#comment-14724147 ] Hudson commented on HDFS-8990: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1057 (See

[jira] [Commented] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

2015-08-31 Thread Jing Zhao (JIRA)
age > - > > Key: HDFS-8936 > URL: https://issues.apache.org/jira/browse/HDFS-8936 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: EC space quota.log, E

[jira] [Updated] (HDFS-7676) Fix TestFileTruncate to avoid bug of HDFS-7611

2015-08-31 Thread Vinod Kumar Vavilapalli (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7676: -- Labels: 2.6.1-candidate (was: ) > Fix TestFileTruncate to avoid bug of

[jira] [Commented] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724229#comment-14724229 ] Hudson commented on HDFS-8990: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #330 (See

[jira] [Updated] (HDFS-8984) Move replication queues related methods in FSNamesystem to BlockManager

2015-08-31 Thread Haohui Mai (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-8984: - Attachment: HDFS-8984.001.patch > Move replication queues related methods in FSNamesystem to BlockManager

[jira] [Updated] (HDFS-7707) Edit log corruption due to delayed block removal again

2015-08-31 Thread Vinod Kumar Vavilapalli (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7707: -- Fix Version/s: 2.6.1 [~sjlee0] backported this to 2.6.1. I just pushed the

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724350#comment-14724350 ] Hudson commented on HDFS-8980: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2253 (See

[jira] [Commented] (HDFS-8980) Remove unnecessary block replacement in INodeFile

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724355#comment-14724355 ] Hudson commented on HDFS-8980: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #314 (See

[jira] [Commented] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

2015-08-31 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724372#comment-14724372 ] Jing Zhao commented on HDFS-8982: - After HDFS-8823, the replication factor is recorded inside of BlockInfo,

[jira] [Created] (HDFS-8997) Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract

2015-08-31 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-8997: - Summary: Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract Key: HDFS-8997 URL: https://issues.apache.org/jira/browse/HDFS-8997 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-8997) Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract

2015-08-31 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8997: -- Attachment: hdfs-8997.001.patch Trivial patch attached > Mark ViewFileSystemBaseTest and ViewFsBaseTest

[jira] [Updated] (HDFS-8997) Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract

2015-08-31 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8997: -- Affects Version/s: 2.7.1 > Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract >

[jira] [Updated] (HDFS-8997) Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract

2015-08-31 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8997: -- Target Version/s: 2.8.0 Status: Patch Available (was: Open) > Mark ViewFileSystemBaseTest

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724424#comment-14724424 ] Andrew Wang commented on HDFS-8965: --- LGTM +1 > Harden edit log reading code against out of memory errors

[jira] [Commented] (HDFS-8763) After file closed, a race condition between IBR of 3rd replica of lastBlock and ReplicationMonitor

2015-08-31 Thread Konstantin Shvachko (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724432#comment-14724432 ] Konstantin Shvachko commented on HDFS-8763: --- Hey guys, it looks to me that this works as

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724457#comment-14724457 ] Hadoop QA commented on HDFS-8965: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724462#comment-14724462 ] Hudson commented on HDFS-8990: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #324 (See

[jira] [Commented] (HDFS-8984) Move replication queues related methods in FSNamesystem to BlockManager

2015-08-31 Thread Konstantin Shvachko (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724466#comment-14724466 ] Konstantin Shvachko commented on HDFS-8984: --- Not sure it is a good idea to introduce another

[jira] [Updated] (HDFS-7714) Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.

2015-08-31 Thread Vinod Kumar Vavilapalli (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7714: -- Fix Version/s: 2.6.1 [~sjlee0] backported this to 2.6.1. I just pushed the

[jira] [Commented] (HDFS-8763) After file closed, a race condition between IBR of 3rd replica of lastBlock and ReplicationMonitor

2015-08-31 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724484#comment-14724484 ] Jing Zhao commented on HDFS-8763: - So one question I always have is whether we need to let NameNode wait

[jira] [Comment Edited] (HDFS-8763) After file closed, a race condition between IBR of 3rd replica of lastBlock and ReplicationMonitor

2015-08-31 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724484#comment-14724484 ] Jing Zhao edited comment on HDFS-8763 at 9/1/15 12:40 AM: -- So one question I

[jira] [Commented] (HDFS-8946) Improve choosing datanode storage for block placement

2015-08-31 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724483#comment-14724483 ] Yi Liu commented on HDFS-8946: -- Thanks a lot for the view, [~kihwal]! Will commit shortly. > Improve choosing

[jira] [Commented] (HDFS-8997) Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract

2015-08-31 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724485#comment-14724485 ] Yi Liu commented on HDFS-8997: -- +1, pending Jenkins. Thanks Andrew. > Mark ViewFileSystemBaseTest and

[jira] [Updated] (HDFS-8995) Flaw in registration bookeeping can make DN die on reconnect

2015-08-31 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-8995: - Assignee: Kihwal Lee > Flaw in registration bookeeping can make DN die on reconnect >

[jira] [Commented] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724505#comment-14724505 ] Hudson commented on HDFS-8990: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2273 (See

[jira] [Updated] (HDFS-8946) Improve choosing datanode storage for block placement

2015-08-31 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-8946: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724514#comment-14724514 ] Colin Patrick McCabe commented on HDFS-8965: TestDNFencingWithReplication and bkjm test

[jira] [Updated] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8965: --- Resolution: Fixed Fix Version/s: 2.8 Status: Resolved (was: Patch

[jira] [Commented] (HDFS-8997) Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724517#comment-14724517 ] Hadoop QA commented on HDFS-8997: - \\ \\ | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote ||

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724529#comment-14724529 ] Hudson commented on HDFS-8965: -- FAILURE: Integrated in Hadoop-trunk-Commit #8375 (See

[jira] [Commented] (HDFS-8946) Improve choosing datanode storage for block placement

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724528#comment-14724528 ] Hudson commented on HDFS-8946: -- FAILURE: Integrated in Hadoop-trunk-Commit #8375 (See

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Brahma Reddy Battula (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724530#comment-14724530 ] Brahma Reddy Battula commented on HDFS-8965: [~cmccabe]Thanks for addressing all the comments

[jira] [Commented] (HDFS-8964) Provide max TxId when validating in-progress edit log files

2015-08-31 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724543#comment-14724543 ] Hadoop QA commented on HDFS-8964: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Updated] (HDFS-7788) Post-2.6 namenode may not start up with an image containing inodes created with an old release.

2015-08-31 Thread Vinod Kumar Vavilapalli (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-7788: -- Fix Version/s: 2.6.1 Thanks [~sjlee0]. Pushed the fix into 2.6.1 after running

[jira] [Commented] (HDFS-8990) Move RemoteBlockReader to hdfs-client module

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724557#comment-14724557 ] Hudson commented on HDFS-8990: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #315 (See

[jira] [Commented] (HDFS-8977) NNBench result wrong when number of reducers greater than 1

2015-08-31 Thread Bibin A Chundatt (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724570#comment-14724570 ] Bibin A Chundatt commented on HDFS-8977: Updated to take all part files during

[jira] [Commented] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

2015-08-31 Thread Walter Su (JIRA)
ira/browse/HDFS-8936 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: EC space quota.log, ECSpcaeQuota-20150831.log, None > EC(Replication) space quota.log > > >

[jira] [Commented] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

2015-08-31 Thread GAO Rui (JIRA)
DiskSpace quota exceeded issue in the latest version of HDFS-7285 today. Exceptions messages keep coming out in terminal, it will not stop until the java process was killed. Segment of terminal output [^ECSpcaeQuota-20150831.log] was attached. The improvement in [HDFS-8909] and [HDFS-8838] may

[jira] [Updated] (HDFS-8936) Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

2015-08-31 Thread GAO Rui (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-8936: -- Attachment: ECSpcaeQuota-20150831.log > Simplify Erasure Coding Zone DiskSpace quota exceeded exception er

[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors

2015-08-31 Thread Brahma Reddy Battula (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723320#comment-14723320 ] Brahma Reddy Battula commented on HDFS-8965: Thanks [~cmccabe] for working on this issue..Great

[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl

2015-08-31 Thread Jan Filipiak (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724779#comment-14724779 ] Jan Filipiak commented on HDFS-8836: Thanks for taking this into consideration [~kanaka], one could

[jira] [Commented] (HDFS-8946) Improve choosing datanode storage for block placement

2015-08-31 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724799#comment-14724799 ] Hudson commented on HDFS-8946: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2255 (See

  1   2   >