set hive.mapjoin.smalltable.filesize is no effect fot hive tez?
[email protected] 发件人： Vikram Dixit 发送时间： 2015-05-13 09:26 收件人： user 主题： Re: hive on tez not convert map join to broadcast join From the explain output, it looks like the small table is of size 57 MB whereas the no conditional task size is set to 10MB. If you bump up the no conditional task size to be larger than 57MB, tez would convert the join to a map join. Thanks Vikram. From: "[email protected]" <[email protected]> Reply-To: user <[email protected]> Date: Tuesday, May 12, 2015 at 6:21 PM To: user <[email protected]> Subject: Re: Re: hive on tez not convert map join to broadcast join I usehive 1.1.0 + tez0.53 set hive.mapjoin.smalltable.filesize = 60000000; set hive.auto.convert.join=true; set hive.auto.convert.join.noconditionaltask=true set hive.auto.convert.join.noconditionaltask.size=10000000 [email protected] From: Vikram Dixit Date: 2015-05-13 08:47 To: user Subject: Re: hive on tez not convert map join to broadcast join Hi, Can you share your configuration values for the following when you run in tez: hive.auto.convert.join hive.auto.convert.join.noconditionaltask hive.auto.convert.join.noconditionaltask.size What are the sizes of the tables? Stored in which format? Size of the tables and size of the tables on disc? Thanks Vikram. From: "[email protected]" <[email protected]> Reply-To: user <[email protected]> Date: Tuesday, May 12, 2015 at 5:42 PM To: user <[email protected]> Subject: hive on tez not convert map join to broadcast join In MR query plan is Map Join Operator condition map: Left Outer Join0 to 1 keys: 0 ordr_code (type: string), cart_prod_id (type: bigint) 1 parnt_ordr_code (type: string), comb_prod_id (type: bigint) outputColumnNames: _col1, _col2, _col3, _col5, _col10, _col11, _col15, _col16, But in tez Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) No broadcast edge Reducer 3 <- Map 5 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE) Merge Join Operator condition map: Left Outer Join0 to 1 keys: 0 ordr_code (type: string), cart_prod_id (type: bigint) 1 parnt_ordr_code (type: string), comb_prod_id (type: bigint) [email protected]
