[jira] [Comment Edited] (HADOOP-12857) Rework hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189620#comment-15189620 ] Allen Wittenauer edited comment on HADOOP-12857 at 3/10/16 5:50 PM: -02: * documentation * eliminate HADOOP_TOOLS_PATH since it makes zero sense anymore with this layout and the other capabilities of the shell code in trunk * rework to hopefully work with Windows. :D Should I break this apart to send through Jenkins or ... ? was (Author: aw): -02: * documentation * eliminate HADOOP_TOOLS_PATH since it makes zero sense anymore with this layout and the other capabilities of the shell code in trunk Should I break this apart to send through Jenkins or ... ? > Rework hadoop-tools > --- > > Key: HADOOP-12857 > URL: https://issues.apache.org/jira/browse/HADOOP-12857 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-12857.00.patch, HADOOP-12857.01.patch, > HADOOP-12857.02.patch > > > As hadoop-tools grows bigger and bigger, it's becoming evident that having a > single directory that gets sucked in is starting to become a big burden as > the number of tools grows. Let's rework this to be smarter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-12857) Rework hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189620#comment-15189620 ] Allen Wittenauer edited comment on HADOOP-12857 at 3/10/16 5:50 PM: -02: * documentation * eliminate HADOOP_TOOLS_PATH since it makes zero sense anymore with this layout and the other capabilities of the shell code in trunk Should I break this apart to send through Jenkins or ... ? was (Author: aw): -02: * documentation * eliminate HADOOP_TOOLS_PATH since it makes zero sense anymore with this layout and the other capabilities of the shell code in trunk > Rework hadoop-tools > --- > > Key: HADOOP-12857 > URL: https://issues.apache.org/jira/browse/HADOOP-12857 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-12857.00.patch, HADOOP-12857.01.patch, > HADOOP-12857.02.patch > > > As hadoop-tools grows bigger and bigger, it's becoming evident that having a > single directory that gets sucked in is starting to become a big burden as > the number of tools grows. Let's rework this to be smarter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-12857) Rework hadoop-tools-dist
[ https://issues.apache.org/jira/browse/HADOOP-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177090#comment-15177090 ] Allen Wittenauer edited comment on HADOOP-12857 at 3/3/16 3:25 AM: --- Why does the hdfs haadmin command require hadoop-tools in the classpath? Is this actually a long standing bug/misunderstanding of where toolrunner comes from? was (Author: aw): Why does the hdfs haadmin command require hadoop-tools in the classpath? Is this actually a long standing bug? > Rework hadoop-tools-dist > > > Key: HADOOP-12857 > URL: https://issues.apache.org/jira/browse/HADOOP-12857 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > > As hadoop-tools grows bigger and bigger, it's becoming evident that having a > single directory that gets sucked in is starting to become a big burden as > the number of tools grows. Let's rework this to be smarter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-12857) Rework hadoop-tools-dist
[ https://issues.apache.org/jira/browse/HADOOP-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172799#comment-15172799 ] Allen Wittenauer edited comment on HADOOP-12857 at 2/29/16 10:57 PM: - FWIW, I've got some stupid/simple shell code that takes the output of mvn dependency:list and builds a shell profile script. Some random notes: * It currently looks for *ALL* of the depended upon jars in the tools dir. This is less than efficient for what are hopefully obvious reasons. * HADOOP-10115 pretty much means that the shell profiles will need to be built well after we've processed the hadoop-tools dir in order to know what is/isn't already bundled via hadoop-common. So contemplating two approaches in order to make the latter option work: # Try to trigger mvn dependency:list in the build stage for those modules that need it. Push the output through the build process up until hadoop-dist gets triggered. Take that output and generate the profiles then. # In hadoop-dist, run mvn dependency:list for all (except some blacklisted ones) modules under hadoop-tools (and thus effectively having mvn running mvn), and then generate profiles as in #1. To make matters more complicated, I've been informed over the weekend that Big Top based distributions stupidly merge all of hadoop-tools into hadoop-common's lib dir. So they'll always have the perf hit and other issues that having a flat dir structure causes. was (Author: aw): FWIW, I've got some stupid/simple shell code that takes the output of mvn dependency:list and builds a shell profile script. Some random notes: * It currently looks for *ALL* of the jars in the tools dir. This is less than efficient for what are hopefully obvious reasons. * HADOOP-10115 pretty much means that the shell profiles will need to be built well after we've processed the hadoop-tools dir in order to know what is/isn't already bundled via hadoop-common. So contemplating two approaches in order to make the latter option work: # Try to trigger mvn dependency:list in the build stage for those modules that need it. Push the output through the build process up until hadoop-dist gets triggered. Take that output and generate the profiles then. # In hadoop-dist, run mvn dependency:list for all (except some blacklisted ones) modules under hadoop-tools (and thus effectively having mvn running mvn), and then generate profiles as in #1. To make matters more complicated, I've been informed over the weekend that Big Top based distributions stupidly merge all of hadoop-tools into hadoop-common's lib dir. So they'll always have the perf hit and other issues that having a flat dir structure causes. > Rework hadoop-tools-dist > > > Key: HADOOP-12857 > URL: https://issues.apache.org/jira/browse/HADOOP-12857 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer > > As hadoop-tools grows bigger and bigger, it's becoming evident that having a > single directory that gets sucked in is starting to become a big burden as > the number of tools grows. Let's rework this to be smarter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
