Re: Two-level access in Pig 0.8.1

2011-11-01 Thread Andrew Clegg
Sorry, scratch that question, clearly it doesn't, because I'm using it in 0.8.1 and it's not setting two-level access. Oh well. On 1 November 2011 11:03, Andrew Clegg wrote: > Would it be fair to assume that SchemaUtil.newBagSchema(...) etc. will > do the right thing for whi

Re: Two-level access in Pig 0.8.1

2011-11-01 Thread Andrew Clegg
better make your UDF work >> even if twoLevelAccess is dropped. >> >> Daniel >> >> On Mon, Oct 31, 2011 at 2:48 PM, Andrew Clegg >> wrote: >> >>> Thanks chaps. Just to clarify something though -- Daniel's answer >>> suggests twoLevelAccess is

Re: Two-level access in Pig 0.8.1

2011-10-31 Thread Andrew Clegg
ai wrote: >> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it >> any more. >> >> Daniel >> >> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg < >> andrew.clegg+mah...@gmail.com> wrote: >> >>> Hi, >>

Re: Is there a way to set reducer number of pig besides using parallel keyword?

2011-10-12 Thread Andrew Clegg
> page_info, >> >    flatten((bag{tuple(map[])})page_links) as page_links; >> > C = foreach B generate user, >> >    (action == 1 ? page_info#'a' : page_links#'b') as header; >> > D = group C by user parallel 40; >> > E = foreach D generate group, COUNT(C) as cnt; >> > store E into 'L1out'; >> > >> > Best, >> > Hui >> > >> > -- http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg

Re: outputSchema for UDF EvalFunc returning a DataBag

2011-10-04 Thread Andrew Clegg
4 October 2011 00:14, Raghu Angadi wrote: > Utils.getSchemaFromString() seems like exactly what you want ( > from org_apache_pig_impl_util ). > > Raghu. > > [btw. my two previous attempts to send to the list got rejected as spam ] > > On Mon, Oct 3, 2011 at 3:41 PM, And

Re: outputSchema for UDF EvalFunc returning a DataBag

2011-10-03 Thread Andrew Clegg
Angadi wrote: > my understanding is that Pig 0.8 expects the first form and Pig 0.9 requires > the second. > > Raghu. > > On Mon, Oct 3, 2011 at 8:27 AM, Andrew Clegg > wrote: > >> Hi, >> >> When you have a UDF that returns a bag, and you're writin

Re: Does the pig optimizer keep track of relations that are already sorted when doing a JOIN?

2011-08-21 Thread Andrew Clegg
ion? >> >> -- >> >> Founder/CEO Spinn3r.com >> >> Location: *San Francisco, CA* >> Skype: *burtonator* >> >> Skype-in: *(415) 871-0687* >> > -- http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg

Schema changes when storing to a file

2011-07-22 Thread Andrew Clegg
pecting the first element to be the key tuple but instead got the artistid! Thanks again, Andrew. -- http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg

Re: Confused by FOREACH .. GENERATE .. TOP semantics

2011-07-22 Thread Andrew Clegg
wrote: > >> The syntax looks legal. Can you do an explain? >> >> Daniel >> >> On Thu, Jul 21, 2011 at 5:15 AM, Andrew Clegg < >> andrew.clegg+mah...@gmail.com >> > wrote: >> >> > Hi, >> > >> > I have some code t

Confused by FOREACH .. GENERATE .. TOP semantics

2011-07-21 Thread Andrew Clegg
: {key: (artistid: int,country: int,week: chararray),timestamp: long,albumid: int,numtracks: long,reach: int,title_len: long,score: long}} It didn't affect the error, though. Thanks for any suggestions, Andrew. -- http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg