Re: ClassCastExceptions using Hiveserver2 Thrift interface, Hive 0.11
I can report I see this same error when working with the Perl thrift Hive client ( http://search.cpan.org/~dmor/Thrift-API-HiveClient2-0.011/lib/Thrift/API/HiveClient2.pm), and with the newly released Hive 0.12.0. Is anyone successfully using Hiveserver2, or do I have something odd in my environment? On 14 October 2013 17:44, Andy Sykes andy.sy...@forward3d.com wrote: Hi there, We're using the rbhive gem (https://github.com/forward3d/rbhive) to connect to the Hiveserver2 Thrift service running on Hive 0.11. I've turned SASL auth off with the following in hive-site.xml: property namehive.server2.authentication/name valueNOSASL/value /property When I connect, I get the following stack trace in the Hiveserver2 logs: 13/10/14 17:42:17 ERROR server.TThreadPoolServer: Error occurred during processing of message. java.lang.ClassCastException: org.apache.thrift.transport.TSocket cannot be cast to org.apache.thrift.transport.TSaslServerTransport at org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:35) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) This didn't happen with Hive 0.10 - we could connect with SASL turned off without a problem. Can anyone shed any light on this? -- *Andy Sykes* DevOps Engineer +4420 7806 5904 +4475 9005 2109 19 Mandela Street Floor 2, Centro 3 London NW1 0DU Privacy Email Policy http://forward3d.com/terms.html -- *Andy Sykes* DevOps Engineer +4420 7806 5904 +4475 9005 2109 19 Mandela Street Floor 2, Centro 3 London NW1 0DU Privacy Email Policy http://forward3d.com/terms.html
HS2 ODBC incompatibility
Hi, Source under ODBC seems to be incompatible with HS2's changed RPC thrift interface. Are there any plans on getting an updated version out any time sooner ? Thanks,
Re: hive_service.thrift vs TCLIService.thrift
Hi Haroon, hive_service.thrift is the interface definition language (IDL) file for HiveServer (older one), whereas TCLIService.thrift is the IDL for HiveServer2. HiveServer2 was introduced in Hive-0.11. Check the docs here: https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2 Thanks, --Vaibhav On Mon, Oct 21, 2013 at 11:03 AM, Haroon Muhammad muhammad.har...@live.comwrote: Hi, Can anyone share what's the difference between hive_service.thrift and TCLIService.thrift? Thanks, -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
ArrayIndexOutOfBoundsException while writing MapReduce output as RCFile
Hi All,
I have a scenario where I've to read an RCFile, process it and write the
output as an RCFile using a MapReduce program.
My Hadoop version is *CDH 4.2.1 *
*
*
*Mapper*
Map Input Key,Value = LongWritable, BytesRefArrayWritable
Map Output Key,Value = Text, BytesRefArrayWritable (Record)
***CODE BEGINS***
//Mapper
public static class AbcMapper extends MapperLongWritable,
BytesRefArrayWritable, Text, BytesRefArrayWritable{
public void map(LongWritable key, BytesRefArrayWritable value, Context
context) throws IOException, InterruptedException {
.
// I am passing a text key and BytesRefArrayWritable value (record) as map
output.
context.write(new Text(keys),value);
}
//Reducer
public static class AbcReducer extends
ReducerText,BytesRefArrayWritable,Text,BytesRefArrayWritable {
public void reduce(Text keyz, IterableBytesRefArrayWritable
values, Context context) throws IOException, InterruptedException {
//Based on some logic, I pick one BytesRefArrayWritable record from the
list of BytesRefArrayWritable values obtained in the reduce input.
BytesRefArrayWritable outRecord= new BytesRefArrayWritable(5);
for (BytesRefArrayWritable val : values) {
if (some condition)
outRecord = val;
}
outRecord.size(); //Value here is getting logged as 5.
context.write(new Text(keyz), outRecord);
}
***CODE ENDS***
I've added the following in the main method :
job.setInputFormatClass(RCFileInputFormat.*class*);
job.setOutputFormatClass (RCFileOutputFormat.*class* );
job.setOutputValueClass (BytesRefArrayWritable.*class* );
Before writing to reduce output, the value of outRecord.size() is 5.
But still I'm getting ArrayIndexOutOfBoundsException.
*Stacktrace : *
*
*
*java.lang.ArrayIndexOutOfBoundsException: 0*
at
org.apache.hadoop.hive.ql.io.RCFile$Writer.append(RCFile.java:890)
at poc.RCFileOutputFormat$1.write(RCFileOutputFormat.java:82)
at poc.RCFileOutputFormat$1.write(RCFileOutputFormat.java:1)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
at
org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:99)
at poc.Test$PurgeReducer.reduce(Purge.java:98)
at poc.Test$AReducer.reduce(Purge.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
I tried a lot to find which array was empty causing this exception
*java.lang.ArrayIndexOutOfBoundsException:
0, *but have not found anything yet
Could you please give me any pointers that will help me identify/resolve
the issue ?
Thanks!
In Beeline what is the syntax for ALTER TABLE ?
Hi guys Using Hive0.10.0+198 CDH4 Getting this error for ALTER table command jdbc:hive2://dev-thdp5.corp.nextag.com:100 ALTER TABLE outpdir_seller_hidden ADD IF NOT EXISTS PARTITION (header_date_partition='2013-10-17', header_servername_partition='lu3') LOCATION '/data/output/impressions/outpdir/2013-10-17/008-131018121904385-oozie-oozi-W/outpdir_seller_hidden/sellerhidden/2013-10-17/lu3'; Error: Error while processing statement: FAILED: IllegalArgumentException URI '' in invalid. Must start with file:// or hdfs:// (state=42000,code=4) Thanks Regards sanjay CONFIDENTIALITY NOTICE == This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: HS2 ODBC incompatibility
Yes, the current odbc driver source in hive is not compatible with hive server2. I am not aware of any body working on it. But you can download odbc driver add on for hive server2, for free from the hortonworks website - http://hortonworks.com/download/download-archives/ On Mon, Oct 21, 2013 at 5:06 AM, Haroon Muhammad muhammad.har...@live.com wrote: Hi, Source under ODBC seems to be incompatible with HS2's changed RPC thrift interface. Are there any plans on getting an updated version out any time sooner ? Thanks, -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: HS2 ODBC incompatibility
Cloudera has an ODBC driver that is compatible with both HiveServer1 and HiveServer2. More info here: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Connectors/PDF/Cloudera-ODBC-Driver-for-Apache-Hive-Install-Guide.pdf Thanks. Carl On Mon, Oct 21, 2013 at 12:01 PM, Thejas Nair the...@hortonworks.comwrote: Yes, the current odbc driver source in hive is not compatible with hive server2. I am not aware of any body working on it. But you can download odbc driver add on for hive server2, for free from the hortonworks website - http://hortonworks.com/download/download-archives/ On Mon, Oct 21, 2013 at 5:06 AM, Haroon Muhammad muhammad.har...@live.com wrote: Hi, Source under ODBC seems to be incompatible with HS2's changed RPC thrift interface. Are there any plans on getting an updated version out any time sooner ? Thanks, -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: In Beeline what is the syntax for ALTER TABLE ?
beeline does not have any query syntax of its own, it just sends the query to hive server2, which uses same code as hive-cli to run the query. ie query syntax is same as hive cli On Mon, Oct 21, 2013 at 11:41 AM, Sanjay Subramanian sanjay.subraman...@wizecommerce.com wrote: Hi guys Using Hive 0.10.0+198 CDH4 Getting this error for ALTER table command jdbc:hive2://dev-thdp5.corp.nextag.com:100 ALTER TABLE outpdir_seller_hidden ADD IF NOT EXISTS PARTITION (header_date_partition='2013-10-17', header_servername_partition='lu3') LOCATION '/data/output/impressions/outpdir/2013-10-17/008-131018121904385-oozie-oozi-W/outpdir_seller_hidden/sellerhidden/2013-10-17/lu3'; Error: Error while processing statement: FAILED: IllegalArgumentException URI '' in invalid. Must start with file:// or hdfs:// (state=42000,code=4) Thanks Regards sanjay CONFIDENTIALITY NOTICE == This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: ArrayIndexOutOfBoundsException while writing MapReduce output as RCFile
Seems you did not set the number of
columns (RCFileOutputFormat.setColumnNumber(Configuration conf, int
columnNum)). Can you set it in your main method and see if your MR program
works?
Thanks,
Yin
On Mon, Oct 21, 2013 at 2:38 PM, Krishnan K kkrishna...@gmail.com wrote:
Hi All,
I have a scenario where I've to read an RCFile, process it and write the
output as an RCFile using a MapReduce program.
My Hadoop version is *CDH 4.2.1 *
*
*
*Mapper*
Map Input Key,Value = LongWritable, BytesRefArrayWritable
Map Output Key,Value = Text, BytesRefArrayWritable (Record)
***CODE BEGINS***
//Mapper
public static class AbcMapper extends MapperLongWritable,
BytesRefArrayWritable, Text, BytesRefArrayWritable{
public void map(LongWritable key, BytesRefArrayWritable value, Context
context) throws IOException, InterruptedException {
.
// I am passing a text key and BytesRefArrayWritable value (record) as map
output.
context.write(new Text(keys),value);
}
//Reducer
public static class AbcReducer extends
ReducerText,BytesRefArrayWritable,Text,BytesRefArrayWritable {
public void reduce(Text keyz, IterableBytesRefArrayWritable
values, Context context) throws IOException, InterruptedException {
//Based on some logic, I pick one BytesRefArrayWritable record from the
list of BytesRefArrayWritable values obtained in the reduce input.
BytesRefArrayWritable outRecord= new BytesRefArrayWritable(5);
for (BytesRefArrayWritable val : values) {
if (some condition)
outRecord = val;
}
outRecord.size(); //Value here is getting logged as 5.
context.write(new Text(keyz), outRecord);
}
***CODE ENDS***
I've added the following in the main method :
job.setInputFormatClass(RCFileInputFormat.*class*);
job.setOutputFormatClass (RCFileOutputFormat.*class* );
job.setOutputValueClass (BytesRefArrayWritable.*class* );
Before writing to reduce output, the value of outRecord.size() is 5.
But still I'm getting ArrayIndexOutOfBoundsException.
*Stacktrace : *
*
*
*java.lang.ArrayIndexOutOfBoundsException: 0*
at
org.apache.hadoop.hive.ql.io.RCFile$Writer.append(RCFile.java:890)
at poc.RCFileOutputFormat$1.write(RCFileOutputFormat.java:82)
at poc.RCFileOutputFormat$1.write(RCFileOutputFormat.java:1)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
at
org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:99)
at poc.Test$PurgeReducer.reduce(Purge.java:98)
at poc.Test$AReducer.reduce(Purge.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
I tried a lot to find which array was empty causing this exception
*java.lang.ArrayIndexOutOfBoundsException:
0, *but have not found anything yet
Could you please give me any pointers that will help me identify/resolve
the issue ?
Thanks!
