Well for the step you're describing (which I need to do as a preliminary step to accumulating the hours), I just do something in the vein of
NewRel = GROUP OldRel BY timestamp/3600; HourlyRel = FOREACH NewRel GENERATE group as hour, OldRel.something AS something,...; (Noting that timestamp is stored as a long, so I get integer division and the GROUP does what's wanted) Dmitriy was right both about what I was trying to to, and that it's an inherently serial operation. Thanks, Kris On Fri, Dec 17, 2010 at 06:32:38PM -0500, Zach Bailey wrote: > > I believe what you're trying to do is this. You have some sort of data, and > a timestamp: > > > What you want to figure out is how many times each possible value of "data" > appears in a certain time period (say, hourly). > > > Let's say data can have three possible string values: {'a', 'b', 'c'} > > > Your timestamp for convenience sake is a Unix UTC timestamp or ISO formatted > date (I would strongly recommend using one of these since there are already > piggybank functions to slice and dice them). > > > To accumulate all the times that the data 'a' appeared in an hour you would > do something like this: > > > --register piggybank.jar for iso date functions > REGISTER ./piggybank.jar > allData = load ... as (string:chararray, ts:long); > --convert ts to ISO Date, and truncate to the hour > allDataISODates = FOREACH allData GENERATE string, > org.apache.pig.piggybank.evaluation.datetime.truncate.ISOToDay(org.apache.pig.piggybank.evaluation.datetime.convert.UnixToISO(ts)) > as isoHour; > -- group by hour and string > groupedByStringAndHour = GROUP allDataISODates BY (string, isoHour); > -- append counts > stringHourCounts = FOREACH groupedByStringAndHour GENERATE group.string as > string, group.isoHour as isoHour, COUNT(allDataISODates.string) as count; > > > You will now have a relation that looks like: > {'a', '2010-12-13T12:00:00', 2334} > {'b', '2010-12-13T12:00:00', 123} > {'c', '2010-12-13T12:00:00', 3} > {'a', '2010-12-13T13:00:00', 34231} > {'b', '2010-12-13T13:00:00', 34} > {'c', '2010-12-13T13:00:00', 134} > > > Is that the sort of thing you're looking to do? > > -Zach > > > On Friday, December 17, 2010 at 6:22 PM, Dmitriy Ryaboy wrote: > > > What you are suggesting seems to be a fundamentally single-threaded process > > (well, it can be parallelized, but it's not pretty and involves multiple > > passes), so it's not a good fit for the map-reduce paradigm (how would you > > do accumulative totals for 25 billion entries?). Pig tends to avoid > > implementing methods that restrict scaling computations in this way. Your > > idea of streaming through a script would work; you could also write an > > accumulative UDF and use it on the result of doing a GROUP ALL on your > > relation. > > > > -Dmitriy > > > > On Fri, Dec 17, 2010 at 11:31 AM, Kris Coward <k...@melon.org> wrote: > > > > > > > Hello, > > > > > > Is there some sort of mechanism by which I could cause a value to > > > accumulate within a relation? What I'd like to do is something along the > > > lines of having a long called accumulator, and an outer bag called > > > hourlyTotals with a schema of (hour:int, collected:int) > > > > > > accumulator = 0L; -- I know this line doesn't work > > > ORDER hourlyTotals BY collected; > > > cumulativeTotals = FOREACH hourlyTotals { > > > accumulator += collected; > > > GENERATE day, accumulator AS collected; > > > } > > > > > > Could something like this be made to work? Is there something similar > > > that > > > I can do instead? Do I just need to pipe the relation through an > > > external script to get what I want? > > > > > > Thanks, > > > Kris > > > > > > -- > > > Kris Coward http://unripe.melon.org/ > > > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3 > > > > > > > > > > > > > > > > > > > > -- Kris Coward http://unripe.melon.org/ GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3