[ https://issues.apache.org/jira/browse/OOZIE-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018108#comment-16018108 ]
Robert Kanter commented on OOZIE-2886: -------------------------------------- Ya, there's a lot of Hadoop jars and many things use many of them - it's hard to get them all. I'm still seeing hadoop jars in the hive and hive2 sharelibs: {noformat} >> [179] 15:10 : oozie-git (master) :: find . -name hadoop*.jar ./client/target/oozie-client-4.4.0-SNAPSHOT-client/oozie-client-4.4.0-SNAPSHOT/lib/hadoop-auth-2.6.0.jar ./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-annotations-2.6.0.jar ./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-auth-2.6.0.jar ./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-api-2.6.0.jar ./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-common-2.6.0.jar ./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar ./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-common-2.6.0.jar ./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-resourcemanager-2.6.0.jar ./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-web-proxy-2.6.0.jar ./server/target/dependency/hadoop-annotations-2.6.0.jar ./server/target/dependency/hadoop-auth-2.6.0.jar ./server/target/dependency/hadoop-common-2.6.0.jar ./server/target/dependency/hadoop-yarn-api-2.6.0.jar ./server/target/dependency/hadoop-yarn-common-2.6.0.jar ./server/target/dependency/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar ./server/target/dependency/hadoop-yarn-server-common-2.6.0.jar ./server/target/dependency/hadoop-yarn-server-resourcemanager-2.6.0.jar ./server/target/dependency/hadoop-yarn-server-web-proxy-2.6.0.jar ./sharelib/distcp/target/partial-sharelib/share/lib/distcp/hadoop-distcp-2.6.0.jar ./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-annotations-2.6.0.jar ./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-api-2.6.0.jar ./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-common-2.6.0.jar ./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar ./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-common-2.6.0.jar ./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-resourcemanager-2.6.0.jar ./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-web-proxy-2.6.0.jar ./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-annotations-2.6.0.jar ./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-api-2.6.0.jar ./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-common-2.6.0.jar ./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar ./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-common-2.6.0.jar ./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-resourcemanager-2.6.0.jar ./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-web-proxy-2.6.0.jar ./sharelib/streaming/target/partial-sharelib/share/lib/mapreduce-streaming/hadoop-streaming-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/distcp/hadoop-distcp-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-annotations-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-api-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-common-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-common-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-resourcemanager-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-web-proxy-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-annotations-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-api-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-common-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-common-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-resourcemanager-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-web-proxy-2.6.0.jar ./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/mapreduce-streaming/hadoop-streaming-2.6.0.jar ./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-annotations-2.6.0.jar ./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-auth-2.6.0.jar ./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-api-2.6.0.jar ./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-common-2.6.0.jar ./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar ./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-server-common-2.6.0.jar ./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-server-resourcemanager-2.6.0.jar ./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-server-web-proxy-2.6.0.jar {noformat} Just taking a quick look at the changes in the hive sharelib pom, it's adding exclusions to the {{hadoop-client}} artifact, which is "provided." Provided dependencies aren't included anyway, only "compile" ones are (default if unspecified is compile). So the exclusions on {{hadoop-client}} aren't doing anything. Looking at the dependency tree, it looks like {{hive-cli}} is bring in {{hive-shims}}, which is bringing in a bunch of the hadoop artifacts. > Ensure consistent versioning of hadoop jars in sharelibs > -------------------------------------------------------- > > Key: OOZIE-2886 > URL: https://issues.apache.org/jira/browse/OOZIE-2886 > Project: Oozie > Issue Type: Bug > Components: build > Affects Versions: 5.0.0 > Reporter: Robert Kanter > Assignee: Artem Ervits > Attachments: OOZIE-2886-0.patch, OOZIE-2886-0.patch, > OOZIE-2886-1.patch, OOZIE-2886-2.patch, OOZIE-2886-3.patch, OOZIE-2886-4.patch > > > The hive and hive2 sharelibs are pulling in Hadoop jars - they shouldn't be. > Also, we should ensure that if you specify {{-Dhadoop.version=foo}} (e.g. > 2.7.0), that all Hadoop jars end up as that version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)