Re: Hive Alter Partition Location Issue
Nope Daniel, That extra '/' isn't causing any trouble. -Harsha On Thu, Apr 30, 2015 at 7:03 AM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > I think you have an extra '/' in the HDFS URI > > Daniel > > On 30 באפר׳ 2015, at 16:46, Harsha N wrote: > > Thanks for your reply, > > analyze table table1 partition (dt=201501) compute statistics;--returns > the same error > *Total jobs = 1* > *Launching Job 1 out of 1* > *Number of reduce tasks is set to 0 since there's no reduce operator* > *java.io.IOException: cannot find dir = > hdfs:///data/dt =201501/1430201400/* > *in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/]* > * at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:347)* > > To Add On > I am working on External tables in Hive 0.13.1-cdh5.3.2 > > -Harsha > > On Thu, Apr 30, 2015 at 12:37 AM, Mich Talebzadeh > wrote: > >> Hi Harsha, >> >> >> >> Have you updated stats on table1 after partition adding? In other words >> it is possible that the optimiser is not aware of that partition yet? >> >> >> >> analyze table table1 partition (dt=201501) compute statistics; >> >> >> >> HTH >> >> >> >> Mich Talebzadeh >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase** >> ASE 15", **ISBN 978-0-9563693-0-7*. >> >> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN >> 978-0-9759693-0-4* >> >> *Publications due shortly:* >> >> *Creating in-memory Data Grid for Trading Systems with Oracle TimesTen >> and Coherence Cache* >> >> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume >> one out shortly >> >> >> >> NOTE: The information in this email is proprietary and confidential. This >> message is for the designated recipient only, if you are not the intended >> recipient, you should destroy it immediately. Any information in this >> message shall not be understood as given or endorsed by Peridale Ltd, its >> subsidiaries or their employees, unless expressly so stated. It is the >> responsibility of the recipient to ensure that this email is virus free, >> therefore neither Peridale Ltd, its subsidiaries nor their employees accept >> any responsibility. >> >> >> >> *From:* Harsha N [mailto:harsha.hadoo...@gmail.com] >> *Sent:* 30 April 2015 07:24 >> *To:* user@hive.apache.org >> *Subject:* Hive Alter Partition Location Issue >> >> >> >> Hi All, >> >> Can experts share your view on Hive behaviour in below scenario. >> >> >> >> I am facing below issue on using alter partition locations in hive. >> >> >> >> *select count(*) from table1 where dt = 201501;* >> >> >> >> *Total jobs = 1* >> >> *Launching Job 1 out of 1* >> >> *Number of reduce tasks determined at compile time: 1* >> >> *In order to change the average load for a reducer (in bytes):* >> >> * set hive.exec.reducers.bytes.per.reducer=* >> >> *In order to limit the maximum number of reducers:* >> >> * set hive.exec.reducers.max=* >> >> *In order to set a constant number of reducers:* >> >> * set mapreduce.job.reduces=* >> >> *java.io.IOException: cannot find dir = >> hdfs:///data/dt =201501/1430201400/* >> >> *in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/]* >> >> >> >> Below are the steps I have followed. >> >> I have altered a partition location in hive using below command. >> >> ALTER TABLE table1 PARTITION (dt=201501) SET LOCATION >> 'hdfs:///data/dt=201501/1430201400/'; >> >> >> >> I have inserted new data into this new location. >> >> >> >> INSERT INTO TABLE table1 >> >> SELECT * FROM table2 where dt=201501 >> >> >> >> *select count(*) from table1 where dt = 201501; doesn't work but * >> >> *select * from table1 where dt = 201501 works good.* >> >> >> >> Please let me know if you need more information. >> >> >> >> Thanks >> >> Harsha >> > >
Re: Hive Alter Partition Location Issue
I think you have an extra '/' in the HDFS URI Daniel > On 30 באפר׳ 2015, at 16:46, Harsha N wrote: > > Thanks for your reply, > > analyze table table1 partition (dt=201501) compute statistics;--returns the > same error > Total jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > java.io.IOException: cannot find dir = hdfs:///data/dt =201501/1430201400/ > in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/] > at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:347) > > To Add On > I am working on External tables in Hive 0.13.1-cdh5.3.2 > > -Harsha > >> On Thu, Apr 30, 2015 at 12:37 AM, Mich Talebzadeh >> wrote: >> Hi Harsha, >> >> >> >> Have you updated stats on table1 after partition adding? In other words it >> is possible that the optimiser is not aware of that partition yet? >> >> >> >> analyze table table1 partition (dt=201501) compute statistics; >> >> >> >> HTH >> >> >> >> Mich Talebzadeh >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", >> ISBN 978-0-9563693-0-7. >> >> co-author "Sybase Transact SQL Guidelines Best Practices", ISBN >> 978-0-9759693-0-4 >> >> Publications due shortly: >> >> Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and >> Coherence Cache >> >> Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume >> one out shortly >> >> >> >> NOTE: The information in this email is proprietary and confidential. This >> message is for the designated recipient only, if you are not the intended >> recipient, you should destroy it immediately. Any information in this >> message shall not be understood as given or endorsed by Peridale Ltd, its >> subsidiaries or their employees, unless expressly so stated. It is the >> responsibility of the recipient to ensure that this email is virus free, >> therefore neither Peridale Ltd, its subsidiaries nor their employees accept >> any responsibility. >> >> >> >> From: Harsha N [mailto:harsha.hadoo...@gmail.com] >> Sent: 30 April 2015 07:24 >> To: user@hive.apache.org >> Subject: Hive Alter Partition Location Issue >> >> >> >> Hi All, >> >> Can experts share your view on Hive behaviour in below scenario. >> >> >> >> I am facing below issue on using alter partition locations in hive. >> >> >> >> select count(*) from table1 where dt = 201501; >> >> >> >> Total jobs = 1 >> >> Launching Job 1 out of 1 >> >> Number of reduce tasks determined at compile time: 1 >> >> In order to change the average load for a reducer (in bytes): >> >> set hive.exec.reducers.bytes.per.reducer= >> >> In order to limit the maximum number of reducers: >> >> set hive.exec.reducers.max= >> >> In order to set a constant number of reducers: >> >> set mapreduce.job.reduces= >> >> java.io.IOException: cannot find dir = hdfs:///data/dt =201501/1430201400/ >> >> in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/] >> >> >> >> Below are the steps I have followed. >> >> I have altered a partition location in hive using below command. >> >> ALTER TABLE table1 PARTITION (dt=201501) SET LOCATION >> 'hdfs:///data/dt=201501/1430201400/'; >> >> >> >> I have inserted new data into this new location. >> >> >> >> INSERT INTO TABLE table1 >> >> SELECT * FROM table2 where dt=201501 >> >> >> >> select count(*) from table1 where dt = 201501; doesn't work but >> >> select * from table1 where dt = 201501 works good. >> >> >> >> Please let me know if you need more information. >> >> >> >> Thanks >> >> Harsha >> >
Re: Hive Alter Partition Location Issue
Thanks for your reply, analyze table table1 partition (dt=201501) compute statistics;--returns the same error *Total jobs = 1* *Launching Job 1 out of 1* *Number of reduce tasks is set to 0 since there's no reduce operator* *java.io.IOException: cannot find dir = hdfs:///data/dt =201501/1430201400/* *in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/]* * at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:347)* To Add On I am working on External tables in Hive 0.13.1-cdh5.3.2 -Harsha On Thu, Apr 30, 2015 at 12:37 AM, Mich Talebzadeh wrote: > Hi Harsha, > > > > Have you updated stats on table1 after partition adding? In other words it > is possible that the optimiser is not aware of that partition yet? > > > > analyze table table1 partition (dt=201501) compute statistics; > > > > HTH > > > > Mich Talebzadeh > > > > http://talebzadehmich.wordpress.com > > > > Author of the books* "A Practitioner’s Guide to Upgrading to Sybase** ASE > 15", **ISBN 978-0-9563693-0-7*. > > co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN > 978-0-9759693-0-4* > > *Publications due shortly:* > > *Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and > Coherence Cache* > > *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume > one out shortly > > > > NOTE: The information in this email is proprietary and confidential. This > message is for the designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Ltd, its > subsidiaries or their employees, unless expressly so stated. It is the > responsibility of the recipient to ensure that this email is virus free, > therefore neither Peridale Ltd, its subsidiaries nor their employees accept > any responsibility. > > > > *From:* Harsha N [mailto:harsha.hadoo...@gmail.com] > *Sent:* 30 April 2015 07:24 > *To:* user@hive.apache.org > *Subject:* Hive Alter Partition Location Issue > > > > Hi All, > > Can experts share your view on Hive behaviour in below scenario. > > > > I am facing below issue on using alter partition locations in hive. > > > > *select count(*) from table1 where dt = 201501;* > > > > *Total jobs = 1* > > *Launching Job 1 out of 1* > > *Number of reduce tasks determined at compile time: 1* > > *In order to change the average load for a reducer (in bytes):* > > * set hive.exec.reducers.bytes.per.reducer=* > > *In order to limit the maximum number of reducers:* > > * set hive.exec.reducers.max=* > > *In order to set a constant number of reducers:* > > * set mapreduce.job.reduces=* > > *java.io.IOException: cannot find dir = > hdfs:///data/dt =201501/1430201400/* > > *in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/]* > > > > Below are the steps I have followed. > > I have altered a partition location in hive using below command. > > ALTER TABLE table1 PARTITION (dt=201501) SET LOCATION > 'hdfs:///data/dt=201501/1430201400/'; > > > > I have inserted new data into this new location. > > > > INSERT INTO TABLE table1 > > SELECT * FROM table2 where dt=201501 > > > > *select count(*) from table1 where dt = 201501; doesn't work but * > > *select * from table1 where dt = 201501 works good.* > > > > Please let me know if you need more information. > > > > Thanks > > Harsha >
RE: Hive Alter Partition Location Issue
Hi Harsha, Have you updated stats on table1 after partition adding? In other words it is possible that the optimiser is not aware of that partition yet? analyze table table1 partition (dt=201501) compute statistics; HTH Mich Talebzadeh http://talebzadehmich.wordpress.com Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4 Publications due shortly: Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Harsha N [mailto:harsha.hadoo...@gmail.com] Sent: 30 April 2015 07:24 To: user@hive.apache.org Subject: Hive Alter Partition Location Issue Hi All, Can experts share your view on Hive behaviour in below scenario. I am facing below issue on using alter partition locations in hive. select count(*) from table1 where dt = 201501; Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapreduce.job.reduces= java.io.IOException: cannot find dir = hdfs:///data/dt =201501/1430201400/ in pathToPartitionInfo: [hdfs:/data/dt=201501/1430201400/] Below are the steps I have followed. I have altered a partition location in hive using below command. ALTER TABLE table1 PARTITION (dt=201501) SET LOCATION 'hdfs:///data/dt=201501/1430201400/'; I have inserted new data into this new location. INSERT INTO TABLE table1 SELECT * FROM table2 where dt=201501 select count(*) from table1 where dt = 201501; doesn't work but select * from table1 where dt = 201501 works good. Please let me know if you need more information. Thanks Harsha
