hc, Good stuff. I was thinking along very similar lines with regards to allowing mapping a function over a bag. I suspect a MAP can actually be written as a udf. We'd just have to pass the name of the function to be mapped and call InstantiateFuncFromSpec on it.
We may want a different name for it, as "map" and "reduce" are associated with the Hadoop map and reduce stages when talking about Pig, and at some point Pig may want to allow users to explicitly set up map and reduce jobs -- as opposed to mapping functions to members of bags. -D On Fri, Mar 12, 2010 at 2:00 PM, hc busy <hc.b...@gmail.com> wrote: > Hmm, okay, I read the documentation further and it appears that this has > already been discussed previously > (here<http://wiki.apache.org/pig/PigTypesFunctionalSpec>).There > seem to be a question of what's the right thing to do. It seems clear to me > though. When an operation like '*' is applied, this is clearly an item-wise > operation that is to be applied to each member of the bag. If a function is > aggregate (SUM), then it operates across an entire bag. > > When a COGROUP occurs, just do what SQL does. Which is to say, perform > cross > join if an aggregate has been applied across several bags. And do so > automatically, so we don't have to type out the separate FLATTEN's > > grouped = COGROUP employee BY name, bonuses BY name; > flattened = FOREACH grouped GENERATE group, *FLATTEN(employee), > FLATTEN(bonuses);grouped_again = GROUP flattened BY group; > total_compensation = FOREACH grouped_again GENERATE group, > SUM(employee:salary * bonuses:multiplier);* > > So this should do the same: > > grouped = COGROUP employee BY name, bonuses BY name; > total_compensation = FOREACH grouped GENERATE group, > SUM(employee:salary * bonuses:multiplier); > > > automatically, because that can only have one meaning. > > Alternatively, if it is desired to stay with a low-level language, the > solution to all of this confusion around UDF's that take bag's and UDF's > that operate on members of bags can be resolved if we do two things. > > 1.) Allow UDF's to actually become first class citizens. This way we can > pass UDF's to other UDF's. > 2.) introduce the concept of map() and reduce() operator over bags. > > This two things allows us more freedom and follows the paradigm of > map-reducing more closely. > > grouped = COGROUP employee BY name, bonuses BY name; > total_compensation = FOREACH grouped GENERATE group, > reduce(SUM,map(*,employee::salary,bonuses::multiplier)); > > > Actually, this may deserve a separate keyword. Because map and reduce > operate on single bags where as Pig introduces this concept of co-grouping, > so we should have *comap *and *coreduce* that take functions and operate on > multiple bags that are results of a *cogroup*. > > grouped = COGROUP employee BY name, bonuses BY name; > total_compensation = FOREACH grouped GENERATE group, > REDUCE(SUM,COMAP(*, employee::salary,bonuses::multiplier)); > > > This allows us to write efficiently, on one line, what would other wise be > several aliases and unnecessary FLATTENed cross products. > > A second thing that I see is the recommendation of implementing looping > constructs. I wonder if I may suggest, as a follow up to the above, that we > beef up UDF's as first class citizens and add the ability to create UDF > functions in Pig Latin with the ability to recurse. > > The reason why I think this is a better way to loop than *for(;;)* and * > while(){}* and *do{}while()* statements is that recursive calls are > functional and are more easily optimizable than imperative programming. The > PigJournal <http://wiki.apache.org/pig/PigJournal> has an entry for all of > these constructs and functions under the heading "Extending Pig to Include > Branching, Looping, and Functions", but because map-reduce paradigm is > inherently functional, I would rather think that staying functional would > be > a better way to approach this improvement. So the minimal amount of > additional features needed is to implement functions and branching and we > would have loops as a side-effect of those improvements. > > In order for the optimizations to be available to PigLatin interpreter, the > functions and branching *must* be implemented within the Pig system. If it > is externalized, or implemented as UDL of some other language, then > opportunities for optimization of the execution vanishes. > > > Anyways, a couple of cents on a rainy day. > > > > > On Wed, Mar 10, 2010 at 10:15 AM, hc busy <hc.b...@gmail.com> wrote: > > > An additional thought... we can define udf's like > > > > ADD(bag{(int,int)}), DIVIDE(bag{(int,int)}), MULTIPLY(bag{(int,int)}), > > SQRT(bag{(float)}).. > > > > basically vectorize most of the common arithmetic operations, but then > the > > language has to support it by converting > > > > bag.a + bag.b > > > > to > > > > ADD(bag.(a,b)) > > > > I guess there are some difficulties, for instance: > > > > SQRT(bag.a)+bag.b > > > > How would this work? because sqrt(bag.a) returns a bag, how would we > > convert it to the correct per tuple operation? It's almost like we want > to > > convert an expression > > > > SUM(SQRT(bag.a),bag.b) > > > > into a function F such that > > > > SUM(SQRT(bag.a),bag.b) = F(bag.a,bag.b) > > > > and then the F is computed by iterating through on each tuple of the bag. > > > > FOREACH ... GENERATE ..., F(bag.(a,b)); > > > > > > > > > > > > > > On Wed, Mar 10, 2010 at 9:31 AM, hc busy <hc.b...@gmail.com> wrote: > > > >> > >> So, pig team, what is the right way to accomplish this? > >> > >> > >> On Tue, Mar 9, 2010 at 10:50 PM, Mridul Muralidharan < > >> mrid...@yahoo-inc.com> wrote: > >> > >>> On Tuesday 09 March 2010 04:13 AM, hc busy wrote: > >>> > >>>> okay. Here's the bag that I have: > >>>> > >>>> {group: (a: int,b: chararray,c: chararray,d: int), TABLE: {number1: > >>>> int, > >>>> number2:int}} > >>>> > >>>> > >>>> > >>>> and I want to do this > >>>> > >>>> grunt> CALCULATE= FOREACH TABLE_group GENERATE group, > SUM(TABLE.number1 > >>>> / > >>>> TABLE.number2); > >>>> > >>> > >>> > >>> TABLE.number1 actually gives you the bag of number1 values found in > TABLE > >>> - but I am never really sure of the semantics in these situations since > I am > >>> slightly nervous that it is impl dependent ... my understanding is, > what you > >>> are attempting should not work, but I could be wrong. > >>> > >>> I do know that TABLE.(number1, number2) will consistently project and > >>> pair up the fields : so to 'fix' this, you can write your own > DIVIDE_SUM > >>> which does something like this : > >>> > >>> grunt> CALCULATE= FOREACH TABLE_group GENERATE group, > >>> DIVIDE_SUM(TABLE.(number1 , number2)); > >>> > >>> And DIVIDE_SUM udf impl takes in a bag with tuples containing schema > >>> (numerator, denominator) : and returns : > >>> > >>> result == sum ( foreach tuple ( tuple.numerator / tuple.denominator ) > ); > >>> > >>> > >>> Obviously, this is not as 'elegant' as your initial code and is > >>> definitely more cumbersome ... so clarifying this behavior with someone > from > >>> pig team will definitely be better before you attempt this. > >>> > >>> > >>> Regards, > >>> Mridul > >>> > >>> > >>> > >>>> grunt> DUMP CALCULATE; > >>>> > >>>> 2010-03-08 14:02:41,055 [main] ERROR org.apache.pig.tools.grunt.Grunt > - > >>>> ERROR 1039: Incompatible types in Multiplication Operator left hand > >>>> side:bag > >>>> right hand side:bag > >>>> > >>>> > >>>> > >>>> This seems useful that I may want to calculate an agg. of some > >>>> arithmetic > >>>> operations on member of a bag. Any suggestions? > >>>> > >>>> ... Looking at the documentation it looks like I want to do something > >>>> like > >>>> > >>>> SUM(TABLE.(number1 / number2)) > >>>> > >>>> but that doesn't work either :-( > >>>> > >>> > >>> > >> > > >
