Re: Issue while quering Hive
Look at the error message Caused by: java.io.IOException: hdfs://localhost:54310/user/ hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile Did you create table with sequencefile ? On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku rinku.g...@fisglobal.comwrote: Hi All, ** ** I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error. ** ** ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException 2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.init(HadoopShimsSecure.java:316) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355) ... 10 more Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805) at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1714)*** * at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1728)*** * at org.apache.hadoop.mapred.SequenceFileRecordReader.init(SequenceFileRecordReader.java:43) at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.init(CombineHiveRecordReader.java:65) ... 15 more ** ** Can anybody help me on this. ** ** Thanks Regards, *Rinku Garg* ** ** ** ** _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. -- Nitin Pawar
RE: Issue while quering Hive
Hi Nitin, Yes, I created the table with sequencefile. Thanks Regards, Rinku Garg From: Nitin Pawar [mailto:nitinpawar...@gmail.com] Sent: 16 September 2013 14:19 To: user@hive.apache.org Subject: Re: Issue while quering Hive Look at the error message Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile Did you create table with sequencefile ? On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Hi All, I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error. ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException 2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.init(HadoopShimsSecure.java:316) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355) ... 10 more Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805) at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1714) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1728) at org.apache.hadoop.mapred.SequenceFileRecordReader.init(SequenceFileRecordReader.java:43) at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.init(CombineHiveRecordReader.java:65) ... 15 more Can anybody help me on this. Thanks Regards, Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. -- Nitin Pawar _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
RE: Issue while quering Hive
Thanks Nitin, That way it worked, But in that case Hadoop will not be able to split my file into chunks/blocks and run multiple maps in parallel. This can cause under-utilization of my cluster's 'mapping' power. Is that rue?? Thanks Regards, Rinku Garg From: Nitin Pawar [mailto:nitinpawar...@gmail.com] Sent: 16 September 2013 15:57 To: user@hive.apache.org Subject: Re: Issue while quering Hive Does your .gz file contains the data in sequencefile ? or its a plain csv? I think looking at the filename its a plain csv file, so I would recommend that you create a normal table with TextInputFormat (the default) and load data in the new table and give it a try. On Mon, Sep 16, 2013 at 3:36 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Hi Nitin, Yes, I created the table with sequencefile. Thanks Regards, Rinku Garg From: Nitin Pawar [mailto:nitinpawar...@gmail.commailto:nitinpawar...@gmail.com] Sent: 16 September 2013 14:19 To: user@hive.apache.orgmailto:user@hive.apache.org Subject: Re: Issue while quering Hive Look at the error message Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile Did you create table with sequencefile ? On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Hi All, I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error. ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException 2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.init(HadoopShimsSecure.java:316) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355) ... 10 more Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805) at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1714) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1728) at org.apache.hadoop.mapred.SequenceFileRecordReader.init(SequenceFileRecordReader.java:43) at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.init(CombineHiveRecordReader.java:65) ... 15 more Can anybody help me on this. Thanks Regards, Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review
Re: Issue while quering Hive
As per my understanding, hadoop 1.x does not provide you any help on processing compressing files in parallel manner. (Atleast this was the case few months back). This bzip2 splitting etc is added in hadoop2.x as per my understanding. On Mon, Sep 16, 2013 at 5:18 PM, Garg, Rinku rinku.g...@fisglobal.comwrote: Thanks Nitin, ** ** That way it worked, But in that case Hadoop will not be able to split my file into chunks/blocks and run multiple maps in parallel. This can cause under-utilization of my cluster's 'mapping' power. Is that rue?? ** ** Thanks Regards, *Rinku Garg* *From:* Nitin Pawar [mailto:nitinpawar...@gmail.com] *Sent:* 16 September 2013 15:57 *To:* user@hive.apache.org *Subject:* Re: Issue while quering Hive ** ** Does your .gz file contains the data in sequencefile ? or its a plain csv? ** ** I think looking at the filename its a plain csv file, so I would recommend that you create a normal table with TextInputFormat (the default) and load data in the new table and give it a try. ** ** ** ** On Mon, Sep 16, 2013 at 3:36 PM, Garg, Rinku rinku.g...@fisglobal.com wrote: Hi Nitin, Yes, I created the table with sequencefile. Thanks Regards, *Rinku Garg* *From:* Nitin Pawar [mailto:nitinpawar...@gmail.com] *Sent:* 16 September 2013 14:19 *To:* user@hive.apache.org *Subject:* Re: Issue while quering Hive Look at the error message Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile Did you create table with sequencefile ? On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku rinku.g...@fisglobal.com wrote: Hi All, I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error. ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException 2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.init(HadoopShimsSecure.java:316) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355) ... 10 more Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805) at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1714)*** * at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1728)*** * at org.apache.hadoop.mapred.SequenceFileRecordReader.init(SequenceFileRecordReader.java:43
Re: Issue while quering Hive
With regards to splitting an compression there are 2 options really as of now If u r using Sequence Files , then Snappy If u r using TXT files then LZO us great (u have to cross a few minor hoops to get LZO to work and I can provide guidance on that) Please don't use GZ (not splittable) / or worse BZ2 (took slow to compress/decompress) - too slow for comfort The only compelling reason u want to use GZIP as I am using in production is that my log files are MULTIPLE LINES…so if I use regular TXT files then splitting can happen between records sanjay From: Nitin Pawar nitinpawar...@gmail.commailto:nitinpawar...@gmail.com Reply-To: user@hive.apache.orgmailto:user@hive.apache.org user@hive.apache.orgmailto:user@hive.apache.org Date: Monday, September 16, 2013 5:07 AM To: user@hive.apache.orgmailto:user@hive.apache.org user@hive.apache.orgmailto:user@hive.apache.org Subject: Re: Issue while quering Hive As per my understanding, hadoop 1.x does not provide you any help on processing compressing files in parallel manner. (Atleast this was the case few months back). This bzip2 splitting etc is added in hadoop2.x as per my understanding. On Mon, Sep 16, 2013 at 5:18 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Thanks Nitin, That way it worked, But in that case Hadoop will not be able to split my file into chunks/blocks and run multiple maps in parallel. This can cause under-utilization of my cluster's 'mapping' power. Is that rue?? Thanks Regards, Rinku Garg From: Nitin Pawar [mailto:nitinpawar...@gmail.commailto:nitinpawar...@gmail.com] Sent: 16 September 2013 15:57 To: user@hive.apache.orgmailto:user@hive.apache.org Subject: Re: Issue while quering Hive Does your .gz file contains the data in sequencefile ? or its a plain csv? I think looking at the filename its a plain csv file, so I would recommend that you create a normal table with TextInputFormat (the default) and load data in the new table and give it a try. On Mon, Sep 16, 2013 at 3:36 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Hi Nitin, Yes, I created the table with sequencefile. Thanks Regards, Rinku Garg From: Nitin Pawar [mailto:nitinpawar...@gmail.commailto:nitinpawar...@gmail.com] Sent: 16 September 2013 14:19 To: user@hive.apache.orgmailto:user@hive.apache.org Subject: Re: Issue while quering Hive Look at the error message Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile Did you create table with sequencefile ? On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Hi All, I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error. ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException 2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.init(HadoopShimsSecure.java:316) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.shims.HadoopShimsSecure
