Re: [VOTE] HDFS-1312 - Diskbalancer branch merge
This feature is frequently requested by users and would be a useful addition to HDFS. I’ve code reviewed many of the sub-tasks and have tried out Disk Balancer in a test cluster. Suggested a couple of usability improvements. These are tracked by open Jiras but they need not hold up the merge. The documentation looks great. +1 for merging with HDFS-10557 fixed. On 6/15/16, 5:38 PM, "Anu Engineer" wrote: Hi All, I would like to propose a merge vote for HDFS-1312 (Disk balancer) branch to trunk. This branch creates a new tool that allows balancing of data on a datanode. The voting commences now and will run for 7 days till Jun/22/2016 5:00 PM PST. This tool distributes data evenly between the disks of same type on a datanode. This is useful if a disk has been replaced or if some disks are out of space compared to rest of the disks. The current set of commands supported are: 1. Plan - Allows user to create a plan and review it. The plan describes how the data will be moved in the data node. 2. Execute - Allows execution of a plan against a datanode. 3. Query – Queries the status of disk balancer execution. 4. Cancel - cancels a running disk balancer plan. 5. Report – Reports the current state of data distribution on a node. · The original proposal that captures the rationale and possible solution is here. [ https://issues.apache.org/jira/secure/attachment/12755226/disk-balancer-proposal.pdf ] · The updated architecture and test plan document is here. [ https://issues.apache.org/jira/secure/attachment/12810720/Architecture_and_test_update.pdf ] · The merge patch that is a diff against trunk is posted here. [ https://issues.apache.org/jira/secure/attachment/12810943/HDFS-1312.001.patch ] · The user documentation which will be part of apache is posted here. [ https://issues.apache.org/jira/secure/attachment/12805976/HDFS-9547-HDFS-1312.002.patch ] HDFS-1312 has a set of sub-tasks and they are ordered in the same sequence as they were committed to HDFS-1312. Hopefully this will make it easy to code review this branch. There are a set of commands which we would like to do later, including discovering which datanodes in the cluster would benefit by running disk balancer. Appropriate JIRAs for these future work items are filed under HDFS-1312. Disk Balancer is made possible due to the work of many community members including Arpit Agarwal, Vinayakumar B, Mingliang Liu, Tsz Wo Nicholas Sze, Lei (Eddy) Xu and Xiaobing Zhou. I would like to thank them all for the effort and support. Thanks Anu
[jira] [Created] (HDFS-10560) Reuse ObjectMapper instance to improve the performance
Yiqun Lin created HDFS-10560:
Summary: Reuse ObjectMapper instance to improve the performance
Key: HDFS-10560
URL: https://issues.apache.org/jira/browse/HDFS-10560
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer & mover
Affects Versions: HDFS-1312
Reporter: Yiqun Lin
Assignee: Yiqun Lin
In branch HDFS-1312, there are many places use {{ObjectMapper}} instances to do
the json-object transform. But {{ObjectMapper}} instance is relatively heavy,
we should reuse them as possible. And In addition, {{ObjectMapper}} is thread
safe, can see this link:http://wiki.fasterxml.com/JacksonFAQ.
Here are related issues: HDFS-9724, HDFS-9768. We can see detail info in this
issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/ [Jun 21, 2016 10:12:28 AM] (brahma) HDFS-9530. ReservedSpace is not cleared for abandoned Blocks [Jun 21, 2016 11:16:22 AM] (aajisaka) HADOOP-13296. Cleanup javadoc for Path. Contributed by Daniel Templeton. [Jun 21, 2016 6:25:11 PM] (sjlee) MAPREDUCE-6719. The list of -libjars archives should be replaced with a [Jun 21, 2016 6:28:52 PM] (cnauroth) HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS [Jun 21, 2016 9:25:58 PM] (jianhe) MAPREDUCE-6197. Cache MapOutputLocations in ShuffleHandler. Contributed [Jun 22, 2016 2:00:16 AM] (aajisaka) HDFS-7597. DelegationTokenIdentifier should cache the TokenIdentifier to -1 overall The following subsystems voted -1: unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_test_libhdfs_zerocopy_hdfs_static test_test_native_mini_dfs Failed junit tests : hadoop.hdfs.server.namenode.TestNamenodeRetryCache hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.namenode.TestCacheDirectives hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.client.api.impl.TestYarnClient hadoop.yarn.client.cli.TestLogsCLI hadoop.mapred.TestMRCJCFileOutputCommitter cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/diff-compile-javac-root.txt [172K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/diff-javadoc-javadoc-root.txt [2.3M] CTEST: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt [8.0K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [676K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [92K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/71/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [120K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10561) test_native_mini_dfs fails by NoClassDefFoundError
Akira AJISAKA created HDFS-10561:
Summary: test_native_mini_dfs fails by NoClassDefFoundError
Key: HDFS-10561
URL: https://issues.apache.org/jira/browse/HDFS-10561
Project: Hadoop HDFS
Issue Type: Bug
Components: native, test
Reporter: Akira AJISAKA
After HDFS-10256, test_libhdfs_threaded_hdfs_static,
test_libhdfs_zerocopy_hdfs_static, and test_test_native_mini_dfs are failing by
NoClassDefFoundError.
{noformat}
nmdCreate: Builder#build error:
java.lang.NoClassDefFoundError: org/junit/Assert
at
org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:205)
at
org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:150)
at
org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:159)
at
org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2745)
at
org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2735)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:800)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:490)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:449)
Caused by: java.lang.ClassNotFoundException: org.junit.Assert
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
TEST_ERROR: failed on
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_native_mini_dfs.c:35
(errno: 2): got NULL from cl
Test time = 1.05 sec
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
Thanks Allen for reporting. Agreed that HDFS-10256 broke the tests. Filed HDFS-10561. Regards, Akira On 6/19/16 03:54, Allen Wittenauer wrote: On Jun 17, 2016, at 7:04 AM, Apache Jenkins Server wrote: For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/66/ I suspect this change: [Jun 16, 2016 11:17:06 AM] (vinayakumarb) HDFS-10256. Use GenericTestUtils.getTestDir method in tests for broke these tests: Specific tests: Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_test_libhdfs_zerocopy_hdfs_static test_test_native_mini_dfs - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10562) DiskBalancer: update documentation on how to report issues and debug
Anu Engineer created HDFS-10562: --- Summary: DiskBalancer: update documentation on how to report issues and debug Key: HDFS-10562 URL: https://issues.apache.org/jira/browse/HDFS-10562 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer & mover Affects Versions: HDFS-1312 Reporter: Anu Engineer Assignee: Anu Engineer Priority: Minor Fix For: HDFS-1312 Add a section in the diskbalancer documentation on how to report issues and how to debug diskbalancer usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10563) Block reports could be silently dropped by NN
Staffan Friberg created HDFS-10563: -- Summary: Block reports could be silently dropped by NN Key: HDFS-10563 URL: https://issues.apache.org/jira/browse/HDFS-10563 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0-beta1 Reporter: Staffan Friberg Reading through the block reporting code I think I've spotted a case when block reports can silently be dropped and leave thread waiting indefinitely on a FutureTask that never will be executed. The BlockReportProcessingThread.enqueue method doesn't return any status on if the enqueuing of the task was successful and does not handle the case when the queue is full and offer return false. Going back through the call stack to BlockManager.runBlockOp, which indirectly calls enqueue with a FutureTask and then proceeds to do get() om the task. So if the internal queue in the BlockReportingProcessingThread is full, the BR would never be handled and the thread queuing the task would wait indefinitely on the FutureTask that will never be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10564) UNDER MIN REPL'D BLOCKS should be prioritized for replication
Elliott Clark created HDFS-10564: Summary: UNDER MIN REPL'D BLOCKS should be prioritized for replication Key: HDFS-10564 URL: https://issues.apache.org/jira/browse/HDFS-10564 Project: Hadoop HDFS Issue Type: Improvement Reporter: Elliott Clark When datanodes get drained they are probably being drained because the hardware is bad, or suspect. The blocks that have no live nodes should be prioritized. However it appears not to be the case at all. Draining full nodes with lots of blocks but only a handful of under min replicated blocks takes about the full time before fsck reports clean again. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10565) Erasure Coding: Document about the current allowed storage policies for EC Striped mode files
Uma Maheswara Rao G created HDFS-10565: -- Summary: Erasure Coding: Document about the current allowed storage policies for EC Striped mode files Key: HDFS-10565 URL: https://issues.apache.org/jira/browse/HDFS-10565 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Affects Versions: 3.0.0-alpha1 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G HDFS-10473 implemented to allow only ALL_SSD, HOT, COLD policies to take effect while moving/placing blocks for Striped EC files. This is JIRA to track the documentation about the behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10566) Submit plan request should throw exception if Datanode is undergoing an upgrade.
Jitendra Nath Pandey created HDFS-10566: --- Summary: Submit plan request should throw exception if Datanode is undergoing an upgrade. Key: HDFS-10566 URL: https://issues.apache.org/jira/browse/HDFS-10566 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] HDFS-1312 - Diskbalancer branch merge
I think this is a very useful feature. I have file an improvement request, but that doesn’t block the merge. I think the core functionality is complete, with good test coverage and documentation. +1 for the merge. On Jun 22, 2016, at 12:28 AM, Arpit Agarwal wrote: > This feature is frequently requested by users and would be a useful addition > to HDFS. > > I’ve code reviewed many of the sub-tasks and have tried out Disk Balancer in > a test cluster. Suggested a couple of usability improvements. These are > tracked by open Jiras but they need not hold up the merge. The documentation > looks great. > > +1 for merging with HDFS-10557 fixed. > > > On 6/15/16, 5:38 PM, "Anu Engineer" wrote: > > Hi All, > > I would like to propose a merge vote for HDFS-1312 (Disk balancer) branch to > trunk. This branch creates a new tool that allows balancing of data on a > datanode. > > The voting commences now and will run for 7 days till Jun/22/2016 5:00 PM PST. > > This tool distributes data evenly between the disks of same type on a > datanode. > This is useful if a disk has been replaced or if some disks are out of space > compared to rest of the disks. > > The current set of commands supported are: > > 1. Plan - Allows user to create a plan and review it. The plan describes how > the data will be moved in the data node. > > 2. Execute - Allows execution of a plan against a datanode. > > 3. Query – Queries the status of disk balancer execution. > > 4. Cancel - cancels a running disk balancer plan. > > 5. Report – Reports the current state of data distribution on a node. > > > · The original proposal that captures the rationale and possible > solution is here. [ > https://issues.apache.org/jira/secure/attachment/12755226/disk-balancer-proposal.pdf > ] > > · The updated architecture and test plan document is here. [ > https://issues.apache.org/jira/secure/attachment/12810720/Architecture_and_test_update.pdf > ] > > · The merge patch that is a diff against trunk is posted here. [ > https://issues.apache.org/jira/secure/attachment/12810943/HDFS-1312.001.patch > ] > > · The user documentation which will be part of apache is posted here. > [ > https://issues.apache.org/jira/secure/attachment/12805976/HDFS-9547-HDFS-1312.002.patch > ] > > > HDFS-1312 has a set of sub-tasks and they are ordered in the same sequence as > they were committed to HDFS-1312. Hopefully this will make it easy to code > review this branch. > > There are a set of commands which we would like to do later, including > discovering which datanodes in the cluster would benefit by running disk > balancer. > Appropriate JIRAs for these future work items are filed under HDFS-1312. > > Disk Balancer is made possible due to the work of many community members > including Arpit Agarwal, Vinayakumar B, Mingliang Liu, Tsz Wo Nicholas Sze, > Lei (Eddy) Xu and Xiaobing Zhou. I would like to thank them all for the > effort and support. > > Thanks > Anu > > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] HDFS-1312 - Diskbalancer branch merge
Thanks for the great work, Anu. I have ran a test on a local cluster. One suggestion is that the CLI help messages should be more consistent with the rest of rest of hdfs command line tools. +1 for the merge. On Wed, Jun 22, 2016 at 2:30 PM, Jitendra Pandey wrote: > I think this is a very useful feature. > I have file an improvement request, but that doesn’t block the merge. I think > the core functionality is complete, with good test coverage and documentation. > > +1 for the merge. > > On Jun 22, 2016, at 12:28 AM, Arpit Agarwal wrote: > >> This feature is frequently requested by users and would be a useful addition >> to HDFS. >> >> I’ve code reviewed many of the sub-tasks and have tried out Disk Balancer in >> a test cluster. Suggested a couple of usability improvements. These are >> tracked by open Jiras but they need not hold up the merge. The documentation >> looks great. >> >> +1 for merging with HDFS-10557 fixed. >> >> >> On 6/15/16, 5:38 PM, "Anu Engineer" wrote: >> >> Hi All, >> >> I would like to propose a merge vote for HDFS-1312 (Disk balancer) branch to >> trunk. This branch creates a new tool that allows balancing of data on a >> datanode. >> >> The voting commences now and will run for 7 days till Jun/22/2016 5:00 PM >> PST. >> >> This tool distributes data evenly between the disks of same type on a >> datanode. >> This is useful if a disk has been replaced or if some disks are out of space >> compared to rest of the disks. >> >> The current set of commands supported are: >> >> 1. Plan - Allows user to create a plan and review it. The plan describes how >> the data will be moved in the data node. >> >> 2. Execute - Allows execution of a plan against a datanode. >> >> 3. Query – Queries the status of disk balancer execution. >> >> 4. Cancel - cancels a running disk balancer plan. >> >> 5. Report – Reports the current state of data distribution on a node. >> >> >> · The original proposal that captures the rationale and possible >> solution is here. [ >> https://issues.apache.org/jira/secure/attachment/12755226/disk-balancer-proposal.pdf >> ] >> >> · The updated architecture and test plan document is here. [ >> https://issues.apache.org/jira/secure/attachment/12810720/Architecture_and_test_update.pdf >> ] >> >> · The merge patch that is a diff against trunk is posted here. [ >> https://issues.apache.org/jira/secure/attachment/12810943/HDFS-1312.001.patch >> ] >> >> · The user documentation which will be part of apache is posted >> here. [ >> https://issues.apache.org/jira/secure/attachment/12805976/HDFS-9547-HDFS-1312.002.patch >> ] >> >> >> HDFS-1312 has a set of sub-tasks and they are ordered in the same sequence >> as they were committed to HDFS-1312. Hopefully this will make it easy to >> code review this branch. >> >> There are a set of commands which we would like to do later, including >> discovering which datanodes in the cluster would benefit by running disk >> balancer. >> Appropriate JIRAs for these future work items are filed under HDFS-1312. >> >> Disk Balancer is made possible due to the work of many community members >> including Arpit Agarwal, Vinayakumar B, Mingliang Liu, Tsz Wo Nicholas Sze, >> Lei (Eddy) Xu and Xiaobing Zhou. I would like to thank them all for the >> effort and support. >> >> Thanks >> Anu >> >> >> > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > -- Lei (Eddy) Xu Software Engineer, Cloudera - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10567) Improve plan command help message
Lei (Eddy) Xu created HDFS-10567:
Summary: Improve plan command help message
Key: HDFS-10567
URL: https://issues.apache.org/jira/browse/HDFS-10567
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Lei (Eddy) Xu
{code}
--bandwidth Maximum disk bandwidth to be consumed by
diskBalancer. e.g. 10
--maxerror Describes how many errors can be
tolerated while copying between a pair
of disks.
--outFile to write output to, if not
specified defaults will be used.
--plan creates a plan for datanode.
--thresholdPercentagePercentage skew that wetolerate before
diskbalancer starts working e.g. 10
--v Print out the summary of the plan on
console
{code}
We should
* Put the unit into {{--bandwidth}}, or its help message. Is it an integer or
float / double number? Not clear in CLI message.
* Give more details about {{--plan}}. It is not clear what the {{}} is for.
* {{--thresholdPercentage}}, has typo {{wetolerate}} in the error message.
Also it needs to indicated that it is the difference between space utilization
between two disks / volumes. Is it an integer or float / double number?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[RESULT] Re: [VOTE] HDFS-1312 - Diskbalancer branch merge
Hi All, This vote passes with +4 binding votes. Thank you all for taking time out to review, test and contribute patches to disk balancer. Thanks Anu On 6/22/16, 3:41 PM, "Lei Xu" wrote: >Thanks for the great work, Anu. > >I have ran a test on a local cluster. One suggestion is that the CLI >help messages should be more consistent with the rest of rest of hdfs >command line tools. > >+1 for the merge. > >On Wed, Jun 22, 2016 at 2:30 PM, Jitendra Pandey > wrote: >> I think this is a very useful feature. >> I have file an improvement request, but that doesn’t block the merge. I >> think the core functionality is complete, with good test coverage and >> documentation. >> >> +1 for the merge. >> >> On Jun 22, 2016, at 12:28 AM, Arpit Agarwal wrote: >> >>> This feature is frequently requested by users and would be a useful >>> addition to HDFS. >>> >>> I’ve code reviewed many of the sub-tasks and have tried out Disk Balancer >>> in a test cluster. Suggested a couple of usability improvements. These are >>> tracked by open Jiras but they need not hold up the merge. The >>> documentation looks great. >>> >>> +1 for merging with HDFS-10557 fixed. >>> >>> >>> On 6/15/16, 5:38 PM, "Anu Engineer" wrote: >>> >>> Hi All, >>> >>> I would like to propose a merge vote for HDFS-1312 (Disk balancer) branch >>> to trunk. This branch creates a new tool that allows balancing of data on a >>> datanode. >>> >>> The voting commences now and will run for 7 days till Jun/22/2016 5:00 PM >>> PST. >>> >>> This tool distributes data evenly between the disks of same type on a >>> datanode. >>> This is useful if a disk has been replaced or if some disks are out of >>> space compared to rest of the disks. >>> >>> The current set of commands supported are: >>> >>> 1. Plan - Allows user to create a plan and review it. The plan describes >>> how the data will be moved in the data node. >>> >>> 2. Execute - Allows execution of a plan against a datanode. >>> >>> 3. Query – Queries the status of disk balancer execution. >>> >>> 4. Cancel - cancels a running disk balancer plan. >>> >>> 5. Report – Reports the current state of data distribution on a node. >>> >>> >>> · The original proposal that captures the rationale and possible >>> solution is here. [ >>> https://issues.apache.org/jira/secure/attachment/12755226/disk-balancer-proposal.pdf >>> ] >>> >>> · The updated architecture and test plan document is here. [ >>> https://issues.apache.org/jira/secure/attachment/12810720/Architecture_and_test_update.pdf >>> ] >>> >>> · The merge patch that is a diff against trunk is posted here. [ >>> https://issues.apache.org/jira/secure/attachment/12810943/HDFS-1312.001.patch >>> ] >>> >>> · The user documentation which will be part of apache is posted >>> here. [ >>> https://issues.apache.org/jira/secure/attachment/12805976/HDFS-9547-HDFS-1312.002.patch >>> ] >>> >>> >>> HDFS-1312 has a set of sub-tasks and they are ordered in the same sequence >>> as they were committed to HDFS-1312. Hopefully this will make it easy to >>> code review this branch. >>> >>> There are a set of commands which we would like to do later, including >>> discovering which datanodes in the cluster would benefit by running disk >>> balancer. >>> Appropriate JIRAs for these future work items are filed under HDFS-1312. >>> >>> Disk Balancer is made possible due to the work of many community members >>> including Arpit Agarwal, Vinayakumar B, Mingliang Liu, Tsz Wo Nicholas Sze, >>> Lei (Eddy) Xu and Xiaobing Zhou. I would like to thank them all for the >>> effort and support. >>> >>> Thanks >>> Anu >>> >>> >>> >> >> >> - >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > > > >-- >Lei (Eddy) Xu >Software Engineer, Cloudera >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/72/ [Jun 22, 2016 2:38:20 PM] (iwasakims) HADOOP-12588. addendum patch to fix intermittent failure of [Jun 22, 2016 2:45:25 PM] (stevel) HADOOP-13203 S3A: Support fadvise "random" mode for high performance [Jun 22, 2016 6:17:43 PM] (uma.gangumalla) HDFS-10473: Allow only suitable storage policies to be set on striped [Jun 23, 2016 12:20:13 AM] (ozawa) MAPREDUCE-6721. mapreduce.reduce.shuffle.memory.limit.percent=0.0 should [Jun 23, 2016 2:04:54 AM] (Arun Suresh) YARN-5171. Extend DistributedSchedulerProtocol to notify RM of [Jun 23, 2016 4:26:10 AM] (aajisaka) MAPREDUCE-6641. TestTaskAttempt fails in trunk. Contributed by Haibo [Jun 23, 2016 4:48:49 AM] (xgong) YARN-5266. Wrong exit code while trying to get app logs using regex via [Jun 23, 2016 5:06:13 AM] (uma.gangumalla) HDFS-10555: Unable to loadFSEdits due to a failure in readCachePoolInfo. [Jun 23, 2016 5:28:12 AM] (aajisaka) YARN-5278. Remove unused argument in [Jun 23, 2016 6:11:58 AM] (aajisaka) HADOOP-13307. add rsync to Dockerfile so that precommit archive works. -1 overall The following subsystems voted -1: docker Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
