[
https://issues.apache.org/jira/browse/HADOOP-10115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353115#comment-14353115
]
Sean Busbey commented on HADOOP-10115:
--------------------------------------
One possible gap caused by just skipping the jars (rather than symlinking) is
that if folks rely on the directory layout at deployment time to grab needed
jars they might miss out. Presumably they're already grabbing the common share
dir though?
{quote}
Keeping in mind I'm not a Maven expert by any stretch, at what point should
dist-layout-stitching.sh just be something in dev-support/ rather than having a
long string of shell inside the pom.xml file?
{quote}
IMHO, we should do this sooner rather than later. One good reason to do it as a
follow-on is that we could switch to using an maven assembly instead of a shell
script.
{code}
+ # Shellcheck SC2086
+ ROOT=$(cd ../..;pwd)
{code}
Could we use a maven variable for this instead of cd/pwd?
{code}
+ run copy
"$ROOT"/hadoop-common-project/hadoop-common/target/hadoop-common-${project.version}
.
+ run copy
"$ROOT"/hadoop-common-project/hadoop-nfs/target/hadoop-nfs-${project.version} .
+ run copy
"$ROOT"/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-${project.version} .
+ run copy
"$ROOT"/hadoop-hdfs-project/hadoop-hdfs-nfs/target/hadoop-hdfs-nfs-${project.version}
.
+ run copy
"$ROOT"/hadoop-yarn-project/target/hadoop-yarn-project-${project.version} .
+ run copy
"$ROOT"/hadoop-mapreduce-project/target/hadoop-mapreduce-${project.version} .
+ run copy
"$ROOT"/hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-${project.version}
.
{code}
Could you add a comment here that it's important we process the hadoop-common
project first, so that common always has all the dependencies it declares?
Should the yarn get processed before the NFS projects?
> Exclude duplicate jars in hadoop package under different component's lib
> ------------------------------------------------------------------------
>
> Key: HADOOP-10115
> URL: https://issues.apache.org/jira/browse/HADOOP-10115
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.0.0, 2.2.0
> Reporter: Vinayakumar B
> Assignee: Vinayakumar B
> Attachments: HADOOP-10115-004.patch, HADOOP-10115-005.patch,
> HADOOP-10115-006.patch, HADOOP-10115.patch, HADOOP-10115.patch,
> HADOOP-10115.patch
>
>
> In the hadoop package distribution there are more than 90% of the jars are
> duplicated in multiple places.
> For Ex:
> almost all jars in share/hadoop/hdfs/lib are already there in
> share/hadoop/common/lib
> Same case for all other lib in share directory.
> Anyway for all the daemon processes all directories are added to classpath.
> So to reduce the package distribution size and the classpath overhead, remove
> the duplicate jars from the distribution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
