Store functions can run in either map or reduce depending on your script. If your script has any operator that requires a reduce (most joins, group by, order by, distinct, limit) then the store function will be in a reduce.
Alan. On Jan 11, 2013, at 9:14 AM, Corbin Hoenes wrote: > Hi all, > > I am a little unclear about which speculative execution you must disable. > What phase do storage functions run at map or reduce? > > I've always just done both. > set mapred.reduce.tasks.speculative.execution false > set mapred.map.tasks.speculative.execution false > > Thanks for any hints! > > On Fri, Mar 2, 2012 at 4:18 PM, Bill Graham <[email protected]> wrote: > >> I tried to test turning this off in the setStoreLocation method but without >> that change I wasn't able to get a job to run with SE happening. As a >> result I can't verify that the setting is doing anything. Russell, if you >> can reproduce SE I'd be curious to hear if you could turn it off in >> setStoreLocation. >> >> On Fri, Mar 2, 2012 at 2:40 PM, Russell Jurney <[email protected] >>> wrote: >> >>> I thought it was too late in the workflow to do this, but it would be >>> really cool if you could. I don't think to think about MapReduce much >> when >>> I Pig, except to group my scripts by jobs... so this was a surprise for >> me. >>> Made sense once i thought of it. But it was a surprise. >>> >>> >>> On Fri, Mar 2, 2012 at 1:49 PM, Bill Graham <[email protected]> >> wrote: >>> >>>> I was also curious about this and will try it, but my initial thought >> was >>>> that at that point it might be tool late in the workflow of the job. >> I'll >>>> give it a shot and report back. >>>> >>>> >>>> On Fri, Mar 2, 2012 at 1:45 PM, Dmitriy Ryaboy <[email protected]> >>>> wrote: >>>> >>>>> In a StoreFunc, you could do that when you get passed the jobconf, >>>> right? >>>>> >>>>> On Thu, Mar 1, 2012 at 9:37 PM, Bill Graham <[email protected]> >>>> wrote: >>>>>> I don't think so. We just do it in the pig script before using the >>>> store >>>>>> func: >>>>>> >>>>>> SET mapred.map.tasks.speculative.execution false >>>>>> >>>>>> >>>>>> On Thu, Mar 1, 2012 at 4:19 PM, Russell Jurney < >>>> [email protected] >>>>>> wrote: >>>>>> >>>>>>> Can you turn off speculative execution in a StoreFunc? I believe >> it >>>> is >>>>>>> leading to duplicates in MongoStorage(); >>>>>>> >>>>>>> -- >>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>> datasyndrome.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Note that I'm no longer using my Yahoo! email address. Please email >>>> me >>>>> at >>>>>> [email protected] going forward.* >>>>> >>>> >>>> >>>> >>>> -- >>>> *Note that I'm no longer using my Yahoo! email address. Please email me >> at >>>> [email protected] going forward.* >>>> >>> >>> >>> >>> -- >>> Russell Jurney twitter.com/rjurney [email protected]. >>> com >>> >> >> >> >> -- >> *Note that I'm no longer using my Yahoo! email address. Please email me at >> [email protected] going forward.* >>
