[
https://issues.apache.org/jira/browse/PIG-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061583#comment-14061583
]
fang fang chen commented on PIG-4047:
-------------------------------------
Thanks for explain.
For our mapreduce 1.* and hdfs 2.* hadoop cluster, I generated pig tar with no
hadoopversion is specified. I assume the default version would be 20. As I
thought the most common connection between pig and hadoop is at mapreduce side.
Please help correct me if I was wrong.
Till now, one issue was found and was fixed at hadoop side. I am not sure
whether some potential issues existed.
Do you have any concern if we did like this? I mean generated pig package with
no hadoopversion specified and running this package in hybrid hadoop, that is
mapreduce 1.* and hdfs2.*.
Or could you describe some test cases for me to test?
Thanks
> Break up pig withouthadoop and fat jar
> --------------------------------------
>
> Key: PIG-4047
> URL: https://issues.apache.org/jira/browse/PIG-4047
> Project: Pig
> Issue Type: Improvement
> Components: build
> Affects Versions: site
> Reporter: fang fang chen
> Assignee: fang fang chen
> Labels: build
> Fix For: 0.14.0
>
> Attachments: PIG-4047-1.patch, PIG-4047-2.patch, PIG-4047.patch
>
>
> pig-withouthadoop jar is packaging pig core and pig core dependencies. But
> this jar should be removed due to following items:
> 1. the name is confusing. User did not know what the jar is used for at a
> glance.
> 2. it is not absolutely clear for user what the core dependencies are.
> 3. it is hard to maintain dependencies, like dependencies version update.
> Maybe user want to try different version avro without repackaging.
> It is better to not use pig-withouthadoop jar, instead:
> 1. devided without hadoop jar into pig core and pig core dependencies.
> 2. save jars in 1# in lib directory
> 3. in pig script, always add all the jars in lib directory into classpath and
> add pig core jar into classpath.
> I used pig in this way since version 0.8.1 via launching pig grunt. No issue
> found yet.
> Current branch-0.13 is packaging following jars into pig-withouhadoop jar:
> <include name="antlr-runtime-${antlr.version}.jar"/>
> <include name="ST4-${stringtemplate.version}.jar"/>
> <include name="jline-${jline.version}.jar"/>
> <include name="jackson-mapper-asl-${jackson.version}.jar"/>
> <include name="jackson-core-asl-${jackson.version}.jar"/>
> <include name="joda-time-${joda-time.version}.jar"/>
> <include name="guava-${guava.version}.jar"/>
> <include name="automaton-${automaton.version}.jar"/>
> <include name="jansi-${jansi.version}.jar"/>
> <include name="avro-${avro.version}.jar"/>
> <include name="avro-mapred-${avro.version}.jar"/>
> <include name="trevni-core-${avro.version}.jar"/>
> <include name="trevni-avro-${avro.version}.jar"/>
> <include name="snappy-java-${snappy. version}.jar"/>
> We could save upper jars and pig-core jar into lib directory.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
