The idea is only include dependencies of most popular UDF/LoadFunc/StoreFunc in lib. If Pig include dependencies of all existing Pig UDF/LoadFunc/StoreFunc, Pig might end up bundling too many jars. Clearly popularity is not a measurable term and it will change over time, and we make such a decision subjectively. If we get enough interest in a particular UDF/LoadFunc/StoreFunc, we can include dependencies in.
This is independent with fatjar change. Fatjar change makes dependent jars more transparent to users so user will not hit mysterious jar conflict due to the uber pig.jar. Thanks, Daniel On 7/7/15, 4:40 AM, "Lorand Bendig" <[email protected]> wrote: > >Hi Oleksiy, > >Initially the idea was that not to include an additional dependency to >the pig fatjar. Instead, let the >user ship the necessary parquet bundle. >However, with PIG-3737 the dependent jars are now copied to the >$PIG_HOME/lib directory. >I suspect, you are right, the patch in PIG-3737 need to be extended in >order to have parquet-pig-bundle-*.jar >in the /lib directory as well. >On the other hand, it would be also great to bump parquet-bundle version >from 1.2.3 to 1.7.0. > >@Daniel, what do you think? > >Thanks, >Lorand > >On 06/07/15 18:34, Олексій Саянкін wrote: >> Hi team! >> >> I have found strange issue using pig and parquet files. There is no >> parquet-bundle*jar in pig/lib folder so I have to manually add it to >>avoid >> this exception: >> >> pig script failed to validate: >> org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could >>not >> resolve parquet.pig.ParquetLoader using imports: [, java.lang., >> org.apache.pig.builtin., org.apache.pig.impl.builtin.] >> >> I have investigated build.xml files from pig-0.12 to pig-0.15 and found >> that parquet-bundle*jar is only compile time dependency. ANT does not >> copy parquet-bundle*jar >> to lib folder. Similar issue you can see here >> https://issues.apache.org/jira/browse/PIG-3445 (see last comment in the >> thread). >> >> So my question is: Was absence of parquet-bundle*jar file done on >>purpose >> or we have a bug here? >> >> Thanks. >> Oleksiy Sayankin. >> > >
