Hi Mahender Sarangam, 1st : Didn't work the following option in Tez?
set mapreduce.job.reduces=100 or set mapred.reduce.tasks=100 (deprecated) 2nd : Possibility of data skew. It happens when handling null sometimes. Goun 2017-01-21 9:58 GMT+09:00 Mahender Sarangam <[email protected]>: > Hi All, > > We have ORC table which is of 2 GB size. When we perform operation on > top of this ORC table, Tez always deduce 1009 reducer every time. I > searched 1009 is considered as Maximum value of number of Tez task. Is > there a way to reduce the number of reducer. I see file generated > underlying ORC some of them 500 MB or 1 GB etc. Is there way to > distribute file size to same value/same size. > > > My Second scenario, we have join on 5 tables all of them are left join. > Query goes fast till reached 99%. From 99% to 100% it takes too much > time. We are not involving our partition column as part of LEFT JOIN > Statement, Is there better way to resolving issues on 99% hanging > condition. My table is of 20 GB we are left joining with another table ( > 9,00,00,000) records. > > > Mahens > >
