[ https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440964#comment-16440964 ]
Sean Busbey commented on HBASE-20332: ------------------------------------- bq. But I think I have a deeper problem than that. As you mention, surely we are using some of the hbase-server classes and having it at provided in hbase-mapreduce would mean none of it shows up in the shaded jar. I suspect my polluted YARN classpath is providing some of those classes. I'm going to do a quick verification and then if necessary run through my tests again. Yeah, tons of classes loading from my yarn environment (from hbase-client, hbase-common, hbase-server, etc) :( back in a while once I clean it out and run through things again. > shaded mapreduce module shouldn't include hadoop > ------------------------------------------------ > > Key: HBASE-20332 > URL: https://issues.apache.org/jira/browse/HBASE-20332 > Project: HBase > Issue Type: Sub-task > Components: mapreduce, shading > Affects Versions: 2.0.0 > Reporter: Sean Busbey > Assignee: Sean Busbey > Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20332.0.patch > > > AFAICT, we should just entirely skip including hadoop in our shaded mapreduce > module > 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}} > 2) those commands include all the needed Hadoop jars in your classpath by > default (both client side and in the containers) > 3) If you try to use "user classpath first" for your job as a workaround > (e.g. for some library your application needs that hadoop provides) then our > inclusion of *some but not all* hadoop classes then causes everything to fall > over because of mixing rewritten and non-rewritten hadoop classes > 4) if you don't use "user classpath first" then all of our > non-relocated-but-still-shaded hadoop classes are ignored anyways so we're > just wasting space -- This message was sent by Atlassian JIRA (v7.6.3#76005)