I think the link you sent got malformatted, but try separating the jars with a comma http://issues.apache.org/jira/browse/HADOOP-4864
On Thu, Jan 14, 2010 at 7:40 AM, Rob Stewart <robstewar...@googlemail.com> wrote: > Hi Dmitriy, > > OK, well it seems that since 0.20.0 the order as specified on the Pig wiki > is no longer relevant: > doop jar -libjars $zipfjar $datagenjar org.apache.pig.test.utils.datagen. > DataGenerator </pig/DataGenerator> -conf $conf_file [options] colspec... > > See this patch over at Hive for 0.20.0: > http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/200907.mbox/< > dfd95197f3ae8c45b0a96c2f4ba3a2556c8358c...@sc-mbxc1.thefacebook.com> > > I have tried a few combinations, but I can't seem to fit in the "-libjars > $zipfjar" in anywhere now. > > Any ideas? > > Thanks for your help. > > Rob > > > > > 2010/1/14 Dmitriy Ryaboy <dvrya...@gmail.com> > >> Rob, >> You need to tell Hadoop which jars you need it to ship to the worker >> nodes. You include datagen.jar, etc, on the classpath, which makes >> them discoverable locally, but you aren't telling Hadoop to ship them. >> You want to list them, comma-separated, in the -libjars parameter. >> >> -D >> >> On Thu, Jan 14, 2010 at 6:49 AM, Rob Stewart >> <robstewar...@googlemail.com> wrote: >> > Hi there. >> > >> > I am well underway with comparing Pig, Hive, JAQL etc... >> > >> > The DataGenerator is proving a valuable tool for me. Thanks for that. >> > >> > I have one query. I am able to use it in local mode, no problem, and some >> > experiments are complete. >> > >> > However, I cannot seem to use it in MapReduce mode on the cluster. This >> is >> > my file "generateData" contents: >> > ------------------ >> > export pigjar=$HOME/installation/pig/pig-0.5.0/pig-0.5.0-core.jar >> > export zipfjar=$HOME/installation/pig/pig-0.5.0/sdsuLibJKD14.jar >> > export datagenjar=$HOME/rs46/installation/DataGenerator/dist/MyPig.jar >> > export conf_file=/usr/lib/hadoop/conf/hadoop-site.xml >> > export HADOOP_CLASSPATH=$pigjar:$zipfjar:$datagenjar >> > /usr/lib/hadoop/bin/hadoop jar $datagenjar >> > org.apache.pig.test.utils.datagen.DataGenerator -conf $conf_file -m 1 >> -rows >> > 10000000 -f words.dat s:8:50:z:0 >> > ------------------ >> > >> > The error I receive when trying to run it with "-m 1" option (in cluster >> > mode): >> > Caused by: java.lang.ClassNotFoundException: sdsu.algorithms.data.Zipf >> > >> > So in local mode, it successfully picks up the jar file sdsuLibJKD14.jar >> , >> > but when running it in cluster mode, this classpath is not found? >> > >> > >> > thanks. >> > >> > Rob Stewart >> > >> >
