Just to be doubly sure can you share the error inside the log file
mentioned in the output ?
On Nov 3, 2016 10:12, "mingda li" <limingda1...@gmail.com> wrote:
> My query is as following:
>
> pig
> -Dpig.additional.jars=/home/hadoop-user/pig-branch-0.lib/
> datafu-pig-incubating-1.3.1.jar
>
>
> To open pig.
>
> Then, input:
>
>
> *REGISTER*
> /home/hadoop-user/pig-branch-0.15/lib/datafu-pig-incubating-1.3.1.jar
>
> data = LOAD 'hdfs://SCAI01.CS.UCLA.EDU:9000/clash/datasets/1.txt' using
> PigStorage() as (val:int);
>
> define MurmurH32 datafu.pig.hash.Hasher('murmur3-32');
>
> dat= FOREACH data GENERATE MurmurH32(val);
>
> On Wed, Nov 2, 2016 at 9:35 PM, mingda li <limingda1...@gmail.com> wrote:
>
> > En, thanks Debabrata, but actually, I register each time ( forget to tell
> > you) before i run the commands.
> > I use *REGISTER* /home/hadoop-user/pig-branch-0.15/lib/datafu-pig-
> > incubating-1.3.1.jar.
> > But cannot help me.
> >
> > Any other reason?
> >
> > Thanks
> >
> > On Wed, Nov 2, 2016 at 8:03 PM, Debabrata Pani <android.p...@gmail.com>
> > wrote:
> >
> >> It says that pig could not find the class Hasher. Start grunt with
> >> -Dpig.additional.jars (before other pig arguments) or do a "register" of
> >> individual jars before typing in your scripts.
> >>
> >> Regards,
> >> Debabrata
> >>
> >> On Nov 3, 2016 07:09, "mingda li" <limingda1...@gmail.com> wrote:
> >>
> >> > Thanks. I have tried to install the datafu and finish quickstart
> >> > successfully http://datafu.incubator.apache.org/docs/quick-start.html
> >> >
> >> > But when i use the murmur hash, it failed. I do not know why.
> >> >
> >> > grunt> data = LOAD 'hdfs://***.UCLA.EDU:9000/clash/datasets/1.txt'
> >> using
> >> > PigStorage() as (val:int);
> >> >
> >> > grunt> data_out = FOREACH data GENERATE val;
> >> >
> >> > grunt> dat= FOREACH data GENERATE MurmurH32(val);
> >> >
> >> > 2016-11-02 18:25:18,424 [main] ERROR org.apache.pig.tools.grunt.Grunt
> -
> >> > ERROR 1070: Could not resolve datafu.pig.hash.Hasher using imports: [,
> >> > java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> >> >
> >> > Details at logfile: /home/hadoop-user/pig-branch-
> >> > 0.15/bin/pig_1478136031217.log
> >> >
> >> >
> >> > The log file is in attachment.
> >> >
> >> >
> >> > Bests,
> >> >
> >> > Mingda
> >> >
> >> >
> >> > On Wed, Nov 2, 2016 at 2:04 PM, Daniel Dai <da...@hortonworks.com>
> >> wrote:
> >> >
> >> >> I see datafu has a patch for the UDF: https://issues.apache.org/jira
> >> >> /browse/DATAFU-47
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 11/2/16, 11:45 AM, "mingda li" <limingda1...@gmail.com> wrote:
> >> >>
> >> >> >Dear all,
> >> >> >
> >> >> >Hi, now I wants to import a UDF function to pig command. Has anyone
> >> ever
> >> >> >done so? I want to import google's guava/murmur3_32 to pig. Could
> >> anyone
> >> >> >give some useful materials or suggestion？
> >> >> >
> >> >> >Bests,
> >> >> >Mingda
> >> >> >
> >> >> >On Wed, Nov 2, 2016 at 2:11 AM, mingda li <limingda1...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Yeah, I see. Thanks for your reply.
> >> >> >>
> >> >> >> Bests,
> >> >> >> Mingda
> >> >> >>
> >> >> >> On Tue, Nov 1, 2016 at 9:20 PM, Daniel Dai <da...@hortonworks.com
> >
> >> >> wrote:
> >> >> >>
> >> >> >>> Yes, you need to dump/store xxx_OrderRes to kick off the job. You
> >> will
> >> >> >>> see two MapReduce jobs corresponding to the first and second
> join.
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>> Daniel
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> On 11/1/16, 10:52 AM, "mingda li" <limingda1...@gmail.com>
> wrote:
> >> >> >>>
> >> >> >>> >Dear Dai,
> >> >> >>> >
> >> >> >>> >Thanks for your reply.
> >> >> >>> >What I want to do is to compare the two different order of join.
> >> The
> >> >> >>> query
> >> >> >>> >is as following:
> >> >> >>> >
> >> >> >>> >*Bad_OrderIn = JOIN inventory BY inv_item_sk, catalog_sales BY
> >> >> >>> cs_item_sk;*
> >> >> >>> >*Bad_OrderRes = JOIN Bad_OrderIn BY (cs_item_sk,
> >> cs_order_number),
> >> >> >>> >catalog_returns BY (cr_item_sk, cr_order_number);*
> >> >> >>> >*Dump or Store Bad_OrderRes;*
> >> >> >>> >
> >> >> >>> >*Good_OrderIn = JOIN catalog_returns BY (cr_item_sk,
> >> >> cr_order_number),
> >> >> >>> >catalog_sales BY (cs_item_sk, cs_order_number);*
> >> >> >>> >*Good_OrderRes = JOIN Good_OrderIn BY cs_item_sk, inventory BY
> >> >> >>> > inv_item_sk;*
> >> >> >>> >*Dump or Store Good_OrderRes;*
> >> >> >>> >
> >> >> >>> >Since Pig execute the query lazily, I think only by Dump or
> Store
> >> the
> >> >> >>> >result, I can know the time of MapReduce Job, is it right? If it
> >> is,
> >> >> >>> then I
> >> >> >>> >need to count the time to Dump or Store the result as the time
> for
> >> >> the
> >> >> >>> >different orders' join.
> >> >> >>> >
> >> >> >>> >Bests,
> >> >> >>> >Mingda
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >On Tue, Nov 1, 2016 at 10:39 AM, Daniel Dai <
> >> da...@hortonworks.com>
> >> >> >>> wrote:
> >> >> >>> >
> >> >> >>> >> Hi, Mingda,
> >> >> >>> >>
> >> >> >>> >> Pig does not do join reordering and will execute the query as
> >> the
> >> >> way
> >> >> >>> it
> >> >> >>> >> is written. Note you can join multiple relations in one join
> >> >> statement.
> >> >> >>> >>
> >> >> >>> >> Do you want execution time for each join in your statement? I
> >> >> assume
> >> >> >>> you
> >> >> >>> >> are using regular join and running with MapReduce, every join
> >> >> statement
> >> >> >>> >> will be a separate MapReduce job and the join runtime is the
> >> >> runtime
> >> >> >>> for
> >> >> >>> >> its MapReduce job.
> >> >> >>> >>
> >> >> >>> >> Thanks,
> >> >> >>> >> Daniel
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> On 10/31/16, 8:21 PM, "mingda li" <limingda1...@gmail.com>
> >> wrote:
> >> >> >>> >>
> >> >> >>> >> >Dear all,
> >> >> >>> >> >
> >> >> >>> >> >I am doing optimization for multiple join. I am not sure if
> Pig
> >> >> can
> >> >> >>> decide
> >> >> >>> >> >the join order in optimization layer. Does anyone know about
> >> >> this? Or
> >> >> >>> Pig
> >> >> >>> >> >just execute the query as the way it is written.
> >> >> >>> >> >
> >> >> >>> >> >And, I want to do the multiple way Join on different keys.
> Can
> >> the
> >> >> >>> >> >following query work?
> >> >> >>> >> >
> >> >> >>> >> >Res =
> >> >> >>> >> >JOIN
> >> >> >>> >> >(JOIN catalog_sales BY cs_item_sk, inventory BY inv_item_sk)
> >> BY
> >> >> >>> >> >(cs_item_sk, cs_order_number), catalog_returns BY
> (cr_item_sk,
> >> >> >>> >> >cr_order_number);
> >> >> >>> >> >
> >> >> >>> >> >BTW, each time, I run the query, it is finished in one
> second.
> >> Is
> >> >> >>> there a
> >> >> >>> >> >way to see the execution time? I have set the
> >> >> pig.udf.profile=true.
> >> >> >>> Where
> >> >> >>> >> >can I find the time?
> >> >> >>> >> >
> >> >> >>> >> >Bests,
> >> >> >>> >> >Mingda
> >> >> >>> >>
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >>
> >> >
> >> >
> >>
> >
> >
>
