Hi, As Hitesh suggested, please see below. We have an issue with the UNION operator while using the Tez engine with Pig.
Is this a bug? Thanks, ---------- Forwarded message ---------- From: Hitesh Shah <[email protected]> Date: Thu, Nov 5, 2015 at 6:17 PM Subject: Re: Pig job freezes occasionally To: [email protected] Cc: [email protected] This is a question probably better suited for the pig mailing list. thanks — Hitesh On Nov 5, 2015, at 4:41 AM, Gal Vinograd <[email protected]> wrote: > Hey all, > > I'm experiencing a weird behaviour, when I run the script it freeze 2 out of 3 times, and when it doesn't it takes slightly longer then Yarn. I found that if I remove the following part from the end of the script it significantly faster and do not freezes. > > ```pig > a1 = FOREACH metrics GENERATE ...; > a2 = FOREACH metrics GENERATE ...; > ... > a20 = FOREACH metrics GENERATE ...; > u = UNION a1, a2, ..., a20; > STORE u INTO ... Using PigStorage('\t'); > ``` > > I use Tez 0.7.0, Pig 0.15.0, Hadoop 2.6.0. The setting pig.tez.opt.union is disabled, an internal exception is thrown otherwise. I've attached a dump of a frozen job, what could be the problem? > > Thanks :) > <worker_jstack.txt><am_jstack.txt> -- Ron Reiter VP R&D [email protected] +972-545898887 Skype: ron.reiter [image: Crosswise] <http://www.crosswise.com/>
