From the explain output, it looks like the small table is of size 57 MB whereas the no conditional task size is set to 10MB. If you bump up the no conditional task size to be larger than 57MB, tez would convert the join to a map join.
Thanks Vikram. From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Reply-To: user <[email protected]<mailto:[email protected]>> Date: Tuesday, May 12, 2015 at 6:21 PM To: user <[email protected]<mailto:[email protected]>> Subject: Re: Re: hive on tez not convert map join to broadcast join I usehive 1.1.0 + tez0.53 set hive.mapjoin.smalltable.filesize = 60000000; set hive.auto.convert.join=true; set hive.auto.convert.join.noconditionaltask=true set hive.auto.convert.join.noconditionaltask.size=10000000 ________________________________ [email protected]<mailto:[email protected]> From: Vikram Dixit<mailto:[email protected]> Date: 2015-05-13 08:47 To: user<mailto:[email protected]> Subject: Re: hive on tez not convert map join to broadcast join Hi, Can you share your configuration values for the following when you run in tez: hive.auto.convert.join hive.auto.convert.join.noconditionaltask hive.auto.convert.join.noconditionaltask.size What are the sizes of the tables? Stored in which format? Size of the tables and size of the tables on disc? Thanks Vikram. From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Reply-To: user <[email protected]<mailto:[email protected]>> Date: Tuesday, May 12, 2015 at 5:42 PM To: user <[email protected]<mailto:[email protected]>> Subject: hive on tez not convert map join to broadcast join In MR query plan is Map Join Operator condition map: Left Outer Join0 to 1 keys: 0 ordr_code (type: string), cart_prod_id (type: bigint) 1 parnt_ordr_code (type: string), comb_prod_id (type: bigint) outputColumnNames: _col1, _col2, _col3, _col5, _col10, _col11, _col15, _col16, But in tez Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) No broadcast edge Reducer 3 <- Map 5 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE) Merge Join Operator condition map: Left Outer Join0 to 1 keys: 0 ordr_code (type: string), cart_prod_id (type: bigint) 1 parnt_ordr_code (type: string), comb_prod_id (type: bigint) ________________________________ [email protected]<mailto:[email protected]>
