Re: unique-id for the mapper task with tez execution engine
Ok Thanks.
Can I get this *, ***from
some conf to be absolutely sure that I get unique id ?
Regards,
Amey
On Wed, Dec 23, 2015 at 12:06 PM, Gopal Vijayaraghavan
wrote:
> Hi,
>
> > So what do you suggest to get unique-id for mapper task with tez
> >execution engine?
> >
> > conf.get("mapreduce.task.partition");
> >
> > Is this correct?
>
> Yes, that is correct - but it can only be unique within a Mapper vertex.
>
> Tez plans sort of look like this for complex queries
>
> http://people.apache.org/~gopalv/tpch-plans/q2_minimum_cost_supplier.svg
>
>
> Every MapTezProcessor there will go from 0-n, for that config parameter.
>
> So all those Map 1, Map 2 ... Map 17 will have a task.partition 0 with the
> same in the attempt id, because there's no room
> for a vertex-id in there.
>
> Cheers,
> Gopal
>
>
>
>
>
>
>
>
Re: unique-id for the mapper task with tez execution engine
Hi,
> So what do you suggest to get unique-id for mapper task with tez
>execution engine?
>
> conf.get("mapreduce.task.partition");
>
> Is this correct?
Yes, that is correct - but it can only be unique within a Mapper vertex.
Tez plans sort of look like this for complex queries
http://people.apache.org/~gopalv/tpch-plans/q2_minimum_cost_supplier.svg
Every MapTezProcessor there will go from 0-n, for that config parameter.
So all those Map 1, Map 2 ... Map 17 will have a task.partition 0 with the
same in the attempt id, because there's no room
for a vertex-id in there.
Cheers,
Gopal
Re: unique-id for the mapper task with tez execution engine
Thanks Gopal!
So what do you suggest to get unique-id for mapper task with tez execution
engine?
conf.get("*mapreduce.task.**partition*");
Is this correct?
Regards,
Amey
On Wed, Dec 23, 2015 at 10:58 AM, Gopal Vijayaraghavan
wrote:
> Hi,
>
> (x-posts to bcc:)
>
> On 12/22/15, 9:19 PM, "Amey Barve" wrote:
>
> >conf.get("mapreduce.task.id");
> >
> >Now I want to run same hive queries with tez engine and I want to know
> >what
> >should be my unique-id. Is there any property from configuration or other
> >that can give me unique-id from the mapper task?
> >
> >I am using conf.get("mapreduce.task.partition") property, Is that
> >guranteed
> >to give unique - id for tez?
>
> That's populated in Tez in pretty much the exact format used by MRv2.
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hi
> ve/ql/exec/tez/TezProcessor.java#L112
>
>
> But beware, each mapper & reducer think it's the only one of its kind out
> there, so there are collisions between DAG stages for the unique name.
>
> Cheers,
> Gopal
>
>
>
Re: unique-id for the mapper task with tez execution engine
Hi,
(x-posts to bcc:)
On 12/22/15, 9:19 PM, "Amey Barve" wrote:
>conf.get("mapreduce.task.id");
>
>Now I want to run same hive queries with tez engine and I want to know
>what
>should be my unique-id. Is there any property from configuration or other
>that can give me unique-id from the mapper task?
>
>I am using conf.get("mapreduce.task.partition") property, Is that
>guranteed
>to give unique - id for tez?
That's populated in Tez in pretty much the exact format used by MRv2.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hi
ve/ql/exec/tez/TezProcessor.java#L112
But beware, each mapper & reducer think it's the only one of its kind out
there, so there are collisions between DAG stages for the unique name.
Cheers,
Gopal
