[jira] [Updated] (HDFS-8898) Create API and command-line argument to get quota without need to get file and directory counts

2015-09-16 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8898: -- Attachment: HDFS-8898.patch Here is the patch just to illustrate the idea. Applications have the option to get {

[jira] [Updated] (HDFS-9004) Add upgrade domain to DatanodeInfo

2015-09-17 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9004: -- Attachment: HDFS-9004-3.patch Thanks [~eddyxu]! Here is the updated patch that addresses your comment. The faile

[jira] [Commented] (HDFS-9004) Add upgrade domain to DatanodeInfo

2015-09-17 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804935#comment-14804935 ] Ming Ma commented on HDFS-9004: --- Hmm, the test failure seems to indicate hadoop-hdfs unit tes

[jira] [Commented] (HDFS-9004) Add upgrade domain to DatanodeInfo

2015-09-22 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903600#comment-14903600 ] Ming Ma commented on HDFS-9004: --- Thanks [~eddyxu], [~shahrs87] and [~ctrezzo]. > Add upgrad

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-09-22 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903774#comment-14903774 ] Ming Ma commented on HDFS-8647: --- Thanks [~brahmareddy] for the useful analysis! My preference

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-09-25 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908908#comment-14908908 ] Ming Ma commented on HDFS-8647: --- Thanks [~brahmareddy]. Maybe we can move {{hasClusterEverBee

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-09-28 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934622#comment-14934622 ] Ming Ma commented on HDFS-8647: --- Thanks [~brahmareddy]! Here are couple comments. * NetworkT

[jira] [Updated] (HDFS-9015) Refactor TestReplicationPolicy to test different block placement policies

2015-09-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9015: -- Attachment: HDFS-9015.patch The patch refactors TestReplicationPolicy* test code in the following ways: * Move c

[jira] [Updated] (HDFS-9015) Refactor TestReplicationPolicy to test different block placement policies

2015-09-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9015: -- Assignee: Ming Ma Status: Patch Available (was: Open) > Refactor TestReplicationPolicy to test different b

[jira] [Commented] (HDFS-7877) Support maintenance state for datanodes

2015-10-01 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940676#comment-14940676 ] Ming Ma commented on HDFS-7877: --- Maybe we should try to support persistence for timeout. We c

[jira] [Commented] (HDFS-9015) Refactor TestReplicationPolicy to test different block placement policies

2015-10-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941576#comment-14941576 ] Ming Ma commented on HDFS-9015: --- Thanks [~eddyxu]! > Refactor TestReplicationPolicy to test

[jira] [Commented] (HDFS-9142) Namenode Http address is not configured correctly for federated cluster in MiniDFSCluster

2015-10-05 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943562#comment-14943562 ] Ming Ma commented on HDFS-9142: --- Thanks [~l201514] for identifying the issue. Having each nam

[jira] [Updated] (HDFS-9006) Provide BlockPlacementPolicy that supports upgrade domain

2015-10-05 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9006: -- Attachment: HDFS-9006.patch Here is the draft patch that implements the upgrade domain policy. > Provide BlockPl

[jira] [Updated] (HDFS-9006) Provide BlockPlacementPolicy that supports upgrade domain

2015-10-05 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9006: -- Assignee: Ming Ma Status: Patch Available (was: Open) > Provide BlockPlacementPolicy that supports upgrade

[jira] [Commented] (HDFS-9142) Namenode Http address is not configured correctly for federated cluster in MiniDFSCluster

2015-10-08 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949145#comment-14949145 ] Ming Ma commented on HDFS-9142: --- [~l201514], there are some debug statements like "System.out

[jira] [Updated] (HDFS-9006) Provide BlockPlacementPolicy that supports upgrade domain

2015-10-08 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9006: -- Attachment: HDFS-9006-2.patch Thanks [~eddyxu] for the great input! Here is the updated patch. bq. We can make c

[jira] [Updated] (HDFS-9006) Provide BlockPlacementPolicy that supports upgrade domain

2015-10-09 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9006: -- Attachment: HDFS-9006-3.patch Failed tests aren't related. Updated the patch to address checkstyle and whitespac

[jira] [Updated] (HDFS-9142) Update MiniDFSCluster to use separate Configuration object for each namenode

2015-10-09 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9142: -- Summary: Update MiniDFSCluster to use separate Configuration object for each namenode (was: Separating Configur

[jira] [Commented] (HDFS-9142) Separating Configuration object for namenode(s) in MiniDFSCluster

2015-10-09 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950917#comment-14950917 ] Ming Ma commented on HDFS-9142: --- +1 on the latest patch. I have committed to trunk. However i

[jira] [Updated] (HDFS-9142) Separating Configuration object for namenode(s) in MiniDFSCluster

2015-10-09 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9142: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks [~l20151

[jira] [Commented] (HDFS-9142) Separating Configuration object for namenode(s) in MiniDFSCluster

2015-10-09 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950972#comment-14950972 ] Ming Ma commented on HDFS-9142: --- Thanks [~liuml07] for the code review. > Separating Configu

[jira] [Commented] (HDFS-9006) Provide BlockPlacementPolicy that supports upgrade domain

2015-10-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954023#comment-14954023 ] Ming Ma commented on HDFS-9006: --- Thanks [~eddyxu] and [~ctrezzo]! > Provide BlockPlacementPo

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954317#comment-14954317 ] Ming Ma commented on HDFS-8647: --- [~brahmareddy], regarding the {{verifyBlockPlacement}} part,

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-13 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956200#comment-14956200 ] Ming Ma commented on HDFS-8647: --- Thanks [~brahmareddy]! Most of the change looks good. It se

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-14 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957248#comment-14957248 ] Ming Ma commented on HDFS-8647: --- Thanks [~vinayrpet]! bq. how about waiting for commit to br

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-15 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960050#comment-14960050 ] Ming Ma commented on HDFS-8647: --- [~brahmareddy], can you please check again if TestBalancer a

[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota without need to get file and directory counts

2015-10-16 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960823#comment-14960823 ] Ming Ma commented on HDFS-8898: --- Thanks [~kihwal]! I will update the patch with new unit test

[jira] [Created] (HDFS-9259) Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario

2015-10-16 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9259: - Summary: Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario Key: HDFS-9259 URL: https://issues.apache.org/jira/browse/HDFS-9259 Project: Hadoop HDFS

[jira] [Updated] (HDFS-9259) Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario

2015-10-16 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9259: -- Description: We recently found that cross-DC hdfs write could be really slow. Further investigation identified t

[jira] [Updated] (HDFS-9259) Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario

2015-10-16 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9259: -- Assignee: Mingliang Liu Thanks [~liuml07]! I have assigned it to you. > Make SO_SNDBUF size configurable at DFSC

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-18 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962697#comment-14962697 ] Ming Ma commented on HDFS-8647: --- Most of the test failures are due to mismatch between BlockM

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964327#comment-14964327 ] Ming Ma commented on HDFS-8647: --- Thanks [~brahmareddy]! It looks great. Sorry, there is one q

[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964506#comment-14964506 ] Ming Ma commented on HDFS-8647: --- Thanks [~brahmareddy]. It seems you also need to change the

[jira] [Updated] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-20 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8647: -- Status: Patch Available (was: Open) > Abstract BlockManager's rack policy into BlockPlacementPolicy > --

[jira] [Updated] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-20 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8647: -- Status: Open (was: Patch Available) > Abstract BlockManager's rack policy into BlockPlacementPolicy > --

[jira] [Updated] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-10-21 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8647: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) +1 on the lates

[jira] [Updated] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy

2015-10-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9007: -- Attachment: HDFS-9007.patch Here is the initial patch. The basic idea is to add a new method called {{isMovable}

[jira] [Updated] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy

2015-10-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9007: -- Assignee: Ming Ma Status: Patch Available (was: Open) > Fix HDFS Balancer to honor upgrade domain policy >

[jira] [Commented] (HDFS-9259) Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario

2015-10-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975177#comment-14975177 ] Ming Ma commented on HDFS-9259: --- Thanks [~liuml07]! The patch looks good. Just some format ni

[jira] [Created] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen

2015-10-26 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9313: - Summary: Possible NullPointerException in BlockManager if no excess replica can be chosen Key: HDFS-9313 URL: https://issues.apache.org/jira/browse/HDFS-9313 Project: Hadoop HDFS

[jira] [Updated] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen

2015-10-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9313: -- Assignee: Ming Ma Status: Patch Available (was: Open) > Possible NullPointerException in BlockManager if n

[jira] [Updated] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen

2015-10-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9313: -- Attachment: HDFS-9313.patch Here is the patch that illustrates the scenario. It is better to guard against this.

[jira] [Created] (HDFS-9314) Improve BlockPlacementPolicyDefault's picking of excess replicas

2015-10-26 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9314: - Summary: Improve BlockPlacementPolicyDefault's picking of excess replicas Key: HDFS-9314 URL: https://issues.apache.org/jira/browse/HDFS-9314 Project: Hadoop HDFS Issue T

[jira] [Created] (HDFS-9315) Update excess storage type list properly when del hint is picked

2015-10-26 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9315: - Summary: Update excess storage type list properly when del hint is picked Key: HDFS-9315 URL: https://issues.apache.org/jira/browse/HDFS-9315 Project: Hadoop HDFS Issue T

[jira] [Updated] (HDFS-9315) Update excess storage type list properly when del hint is picked

2015-10-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9315: -- Attachment: HDFS-9315.patch Here is the patch that explains the scenario and the fix. Update {{excessTypes}} reg

[jira] [Updated] (HDFS-9315) Update excess storage type list properly when del hint is picked

2015-10-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9315: -- Assignee: Ming Ma Status: Patch Available (was: Open) > Update excess storage type list properly when del

[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen

2015-10-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975612#comment-14975612 ] Ming Ma commented on HDFS-9313: --- The question is, even after HDFS-9314 is fixed we might stil

[jira] [Updated] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen

2015-10-27 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9313: -- Attachment: HDFS-9313-2.patch New patch to fix the issue Mingliang pointed out. Thanks Mingliang, Water, Brahma.

[jira] [Updated] (HDFS-9259) Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario

2015-10-27 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9259: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch

[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen

2015-10-27 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976853#comment-14976853 ] Ming Ma commented on HDFS-9313: --- Thanks [~zhz]. If it continues, given no state has changed o

[jira] [Commented] (HDFS-9083) Replication violates block placement policy.

2015-10-27 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14977301#comment-14977301 ] Ming Ma commented on HDFS-9083: --- LGTM. [~shahrs87] any test failures in branch 2.7(if any, it

[jira] [Created] (HDFS-9360) Storage type usage isn't updated properly after file deletion

2015-11-02 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9360: - Summary: Storage type usage isn't updated properly after file deletion Key: HDFS-9360 URL: https://issues.apache.org/jira/browse/HDFS-9360 Project: Hadoop HDFS Issue Type

[jira] [Updated] (HDFS-9360) Storage type usage isn't updated properly after file deletion

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9360: -- Attachment: HDFS-9360.patch The issue is in {{computeQuotaUsage}}, where {{bsps.getPolicy}} returns the default

[jira] [Updated] (HDFS-9360) Storage type usage isn't updated properly after file deletion

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9360: -- Assignee: Ming Ma Status: Patch Available (was: Open) > Storage type usage isn't updated properly after fi

[jira] [Updated] (HDFS-8898) Create API and command-line argument to get quota without need to get file and directory counts

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8898: -- Attachment: HDFS-8898-2.patch Here is the updated patch with new unit tests. Another change in the updated patch

[jira] [Updated] (HDFS-8898) Create API and command-line argument to get quota without need to get file and directory counts

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8898: -- Assignee: Ming Ma Status: Patch Available (was: Open) > Create API and command-line argument to get quota

[jira] [Updated] (HDFS-9360) Storage type usage isn't updated properly after file deletion

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9360: -- Description: For a directory that doesn't have any storage policy defined, its storage quota usage is deducted wh

[jira] [Updated] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9007: -- Attachment: HDFS-9007-2.patch Thanks [~eddyxu]! With function overloading of getDatanodeInfo, the generics funct

[jira] [Updated] (HDFS-9360) Storage type usage isn't updated properly after file deletion

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9360: -- Attachment: HDFS-9360-2.patch Thanks [~xyao]. Here is the updated patch with your suggestion. > Storage type usa

[jira] [Updated] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9313: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch

[jira] [Updated] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9007: -- Attachment: (was: HDFS-9007-2.patch) > Fix HDFS Balancer to honor upgrade domain policy > ---

[jira] [Updated] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy

2015-11-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9007: -- Attachment: HDFS-9007-2.patch > Fix HDFS Balancer to honor upgrade domain policy > --

[jira] [Updated] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy

2015-11-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9007: -- Attachment: HDFS-9007-branch-2.patch Here is the patch for branch-2. > Fix HDFS Balancer to honor upgrade domain

[jira] [Commented] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy

2015-11-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988773#comment-14988773 ] Ming Ma commented on HDFS-9007: --- The latest test is for the branch-2 patch. I ran the failed

[jira] [Updated] (HDFS-8898) Create API and command-line argument to get quota without need to get file and directory counts

2015-11-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8898: -- Attachment: HDFS-8898-3.patch Thanks [~vinayrpet]! Here is the updated patch that addresses your comments, findb

[jira] [Commented] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy

2015-11-04 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990185#comment-14990185 ] Ming Ma commented on HDFS-9007: --- Thanks [~eddyxu]! > Fix HDFS Balancer to honor upgrade doma

[jira] [Commented] (HDFS-9360) Storage type usage isn't updated properly after file deletion

2015-11-05 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991909#comment-14991909 ] Ming Ma commented on HDFS-9360: --- Thanks [~xyao]. > Storage type usage isn't updated properly

[jira] [Commented] (HDFS-8820) Simplify enabling NameNode RPC congestion control and FairCallQueue

2015-11-05 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992079#comment-14992079 ] Ming Ma commented on HDFS-8820: --- Thanks [~arpitagarwal]. Regarding the {{RpcEngine}} backwar

[jira] [Created] (HDFS-9388) Refactor decommission related code to support maintenance state for datanodes

2015-11-05 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9388: - Summary: Refactor decommission related code to support maintenance state for datanodes Key: HDFS-9388 URL: https://issues.apache.org/jira/browse/HDFS-9388 Project: Hadoop HDFS

[jira] [Created] (HDFS-9389) Add maintenance states to AdminStates

2015-11-05 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9389: - Summary: Add maintenance states to AdminStates Key: HDFS-9389 URL: https://issues.apache.org/jira/browse/HDFS-9389 Project: Hadoop HDFS Issue Type: Sub-task Re

[jira] [Created] (HDFS-9390) Block management for maintenance states

2015-11-05 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9390: - Summary: Block management for maintenance states Key: HDFS-9390 URL: https://issues.apache.org/jira/browse/HDFS-9390 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Created] (HDFS-9391) Update webUI/JMX/fsck to display maintenance state info

2015-11-05 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9391: - Summary: Update webUI/JMX/fsck to display maintenance state info Key: HDFS-9391 URL: https://issues.apache.org/jira/browse/HDFS-9391 Project: Hadoop HDFS Issue Type: Sub-t

[jira] [Created] (HDFS-9392) Admins support for maintenance state

2015-11-05 Thread Ming Ma (JIRA)
Ming Ma created HDFS-9392: - Summary: Admins support for maintenance state Key: HDFS-9392 URL: https://issues.apache.org/jira/browse/HDFS-9392 Project: Hadoop HDFS Issue Type: Sub-task Rep

[jira] [Commented] (HDFS-9005) Provide support for upgrade domain script

2015-11-10 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998911#comment-14998911 ] Ming Ma commented on HDFS-9005: --- [~jmeagher] provided some input from admins point of view. T

[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness

2015-11-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003219#comment-15003219 ] Ming Ma commented on HDFS-9239: --- Sorry for the jumping in late for the discussion. While we h

[jira] [Commented] (HDFS-9413) getContentSummary() on standby should throw StandbyException

2015-11-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003365#comment-15003365 ] Ming Ma commented on HDFS-9413: --- Thanks [~brahmareddy]. You bring up an interesting point, wh

[jira] [Commented] (HDFS-9413) getContentSummary() on standby should throw StandbyException

2015-11-13 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005129#comment-15005129 ] Ming Ma commented on HDFS-9413: --- Nit: Maybe use {{HAUtil.setAllowStandbyReads(conf, false);}}

[jira] [Updated] (HDFS-9413) getContentSummary() on standby should throw StandbyException

2015-11-16 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9413: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch

[jira] [Updated] (HDFS-9392) Admins support for maintenance state

2016-08-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9392: -- Attachment: HDFS-9392.patch Here is the draft patch. It also includes datanode state machine transition under so

[jira] [Updated] (HDFS-9392) Admins support for maintenance state

2016-08-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9392: -- Status: Patch Available (was: Open) > Admins support for maintenance state > ---

[jira] [Updated] (HDFS-9392) Admins support for maintenance state

2016-08-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9392: -- Attachment: HDFS-9392-2.patch Updated patch to fix checkstyle, whitespace and unit tests. > Admins support for m

[jira] [Updated] (HDFS-9392) Admins support for maintenance state

2016-08-22 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9392: -- Attachment: HDFS-9392-3.patch Thanks [~eddyxu]! Here is the new patch with the checkstyle fixes. > Admins suppor

[jira] [Updated] (HDFS-9392) Admins support for maintenance state

2016-08-23 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9392: -- Attachment: HDFS-9392-4.patch New patch with additional checkstyle fixes. > Admins support for maintenance state

[jira] [Updated] (HDFS-9392) Admins support for maintenance state

2016-08-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9392: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Status: Resolved (was: Patch

[jira] [Reopened] (HDFS-9392) Admins support for maintenance state

2016-08-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma reopened HDFS-9392: --- > Admins support for maintenance state > > > Key: HDFS-9392 >

[jira] [Resolved] (HDFS-9392) Admins support for maintenance state

2016-08-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma resolved HDFS-9392. --- Resolution: Fixed > Admins support for maintenance state > > >

[jira] [Assigned] (HDFS-9392) Admins support for maintenance state

2016-08-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma reassigned HDFS-9392: - Assignee: Ming Ma > Admins support for maintenance state > > >

[jira] [Updated] (HDFS-9390) Block management for maintenance states

2016-09-23 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9390: -- Assignee: Ming Ma Status: Patch Available (was: Open) > Block management for maintenance states >

[jira] [Updated] (HDFS-9390) Block management for maintenance states

2016-09-23 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9390: -- Attachment: HDFS-9390.patch [~eddyxu] sorry for the delay. Due to the big difference between trunk and 2.6 which

[jira] [Updated] (HDFS-9390) Block management for maintenance states

2016-09-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9390: -- Attachment: HDFS-9390-2.patch Updated patch to address the checkstyle and test code issues. > Block management f

[jira] [Commented] (HDFS-10629) Federation Router

2016-09-28 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531543#comment-15531543 ] Ming Ma commented on HDFS-10629: Sorry for [~elgoiri] and [~jakace] for the late reply. Ye

[jira] [Updated] (HDFS-9390) Block management for maintenance states

2016-09-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9390: -- Attachment: HDFS-9390-3.patch Thanks [~eddyxu] for the review! Here is the new patch. bq. If we change it to the

[jira] [Updated] (HDFS-9390) Block management for maintenance states

2016-09-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9390: -- Attachment: HDFS-9390-4.patch Updated patch to address checkstyle issues. > Block management for maintenance sta

[jira] [Comment Edited] (HDFS-10629) Federation Router

2016-10-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543484#comment-15543484 ] Ming Ma edited comment on HDFS-10629 at 10/3/16 9:29 PM: - Thanks [

[jira] [Commented] (HDFS-10629) Federation Router

2016-10-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543484#comment-15543484 ] Ming Ma commented on HDFS-10629: Thanks [~jakace] for the explanation and new patch. bq.

[jira] [Comment Edited] (HDFS-10629) Federation Router

2016-10-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543484#comment-15543484 ] Ming Ma edited comment on HDFS-10629 at 10/3/16 9:31 PM: - Thanks [

[jira] [Comment Edited] (HDFS-10629) Federation Router

2016-10-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543484#comment-15543484 ] Ming Ma edited comment on HDFS-10629 at 10/3/16 9:33 PM: - Thanks [

[jira] [Commented] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

2016-10-10 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563120#comment-15563120 ] Ming Ma commented on HDFS-10967: Thanks [~zhz]. Indeed that is an issue when the cluster h

[jira] [Comment Edited] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

2016-10-10 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563120#comment-15563120 ] Ming Ma edited comment on HDFS-10967 at 10/10/16 8:56 PM: -- Thanks

[jira] [Commented] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

2016-10-11 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15565729#comment-15565729 ] Ming Ma commented on HDFS-10967: Thanks [~zhz] for the summary! * Good idea to merge Avai

<    1   2   3   4   5   6   7   8   >