Cannot create an instance of InputFormat
My Steps:
I define a class public class myInputFormat extends TextInputFormat
implements JobConfigurable to specify input format.
hive add jar /home/biadmin/hiveudf/myFileFormat.jar;
Added /home/biadmin/hiveudf/myFileFormat.jar to class path
Added resource: /home/biadmin/hiveudf/myFileFormat.jar
hive list jars;
/home/biadmin/hiveudf/myFileFormat.jar
hive create table IOtable(str1 string, str2 string, str3 string) stored as
INPUTFORMAT 'com.mytest.fileformat.myInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat' ;
OK
Time taken: 0.081 seconds
hive load data local inpath '/home/biadmin/hivetbl/IOtable_data.txt' into
table IOtable;
Copying data from file:/home/biadmin/hivetbl/IOtable_data.txt
Copying file: file:/home/biadmin/hivetbl/IOtable_data.txt
Loading data to table default.iotable
OK
Time taken: 0.147 seconds
hive select * from IOtable;
OK
Failed with exception java.io.IOException:java.io.IOException: Cannot
create an instance of InputFormat class com.mytest.fileformat.myInputFormat
as specified in mapredWork!
Time taken: 0.059 seconds
*Here is my source code :*
===
package com.mytest.fileformat;
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.mapred.FileSplit;
import org.apache.hadoop.mapred.InputSplit;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.JobConfigurable;
import org.apache.hadoop.mapred.LineRecordReader;
import org.apache.hadoop.mapred.RecordReader;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.InputFormat;
import org.apache.hadoop.mapred.TextInputFormat;
@SuppressWarnings(deprecation)
public class myInputFormat extends TextInputFormat implements
JobConfigurable {
TextInputFormat format;
JobConf job;
public myInputFormat() {
format = new TextInputFormat();
}
@Override
public void configure(JobConf job) {
this.job = job;
format.configure(job);
}
public RecordReaderLongWritable, Text getRecordReader(
InputSplit genericSplit, JobConf job, Reporter reporter)
throws IOException {
reporter.setStatus(genericSplit.toString());
return new myLineRecordReader(job, (FileSplit) genericSplit);
}
public static class myLineRecordReader implements
RecordReaderLongWritable, Text {
LineRecordReader lineReader;
LongWritable lineKey;
Text lineValue;
public myLineRecordReader(JobConf job, FileSplit split) throws
IOException {
lineReader = new LineRecordReader(job, split);
lineKey = lineReader.createKey();
lineValue = lineReader.createValue();
}
public boolean next(LongWritable key, Text value) throws IOException {
while (lineReader.next(lineKey, lineValue)) {
String strReplace = lineValue.toString().toLowerCase().replace(
, \001 );
Text txtReplace = new Text();
txtReplace.set(strReplace);
value.set(txtReplace.getBytes(), 0, txtReplace.getLength());
return true ;
}
// no more data
return false;
} /** end next **/
public LongWritable createKey() {
return lineReader.createKey();
}
public Text createValue() {
return lineReader.createValue();
}
public long getPos() throws IOException{
return lineReader.getPos();
}
public float getProgress() throws IOException{
return lineReader.getProgress();
}
public void close() throws IOException{
lineReader.close();
}
} /** end class myLineRecordReader **/
}
Error while starting HiveServer
Hello, While starting hive server I am getting error as: hadoop@ubuntu:~/hive-0.7.1$ bin/hive --service hiveserver Starting Hive Thrift Server org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:1. at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:99) at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:80) at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:73) at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:384) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) Pls suggest me why this error come and how to remove it? -- Regards, Bhavesh Shah
Re: Error while starting HiveServer
check your hosts file its should have the entry of NN . On Tue, Jan 17, 2012 at 5:36 PM, Bhavesh Shah bhavesh25s...@gmail.comwrote: Hello, While starting hive server I am getting error as: hadoop@ubuntu:~/hive-0.7.1$ bin/hive --service hiveserver Starting Hive Thrift Server org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:1. at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:99) at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:80) at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:73) at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:384) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) Pls suggest me why this error come and how to remove it? -- Regards, Bhavesh Shah
Related to INSERT QUERY in Hive
Hello, I am using Hive-0.7.1. I want to append the data in table. Is hive-0.7.1 support appending feature or just support OVERWRITE feature? When I tried for the appending, the query is not working. What should I do in such case so that I append the data in table? -- Thanks and Regards, Bhavesh Shah
Re: Related to INSERT QUERY in Hive
how about Load data inpath? On Tue, Jan 17, 2012 at 9:00 PM, Bhavesh Shah bhavesh25s...@gmail.comwrote: Hello, I am using Hive-0.7.1. I want to append the data in table. Is hive-0.7.1 support appending feature or just support OVERWRITE feature? When I tried for the appending, the query is not working. What should I do in such case so that I append the data in table? -- Thanks and Regards, Bhavesh Shah -- ...:::Aniket:::... Quetzalco@tl
Re: Related to INSERT QUERY in Hive
Load data inpath will append data into Hive table .. this feature is also supported in Hive-0.7.*. Ex: load data local inpath './examples/files/kv1.txt' into table test1; This commad will append data into test1 table. On Wed, Jan 18, 2012 at 10:55 AM, Aniket Mokashi aniket...@gmail.comwrote: how about Load data inpath? On Tue, Jan 17, 2012 at 9:00 PM, Bhavesh Shah bhavesh25s...@gmail.comwrote: Hello, I am using Hive-0.7.1. I want to append the data in table. Is hive-0.7.1 support appending feature or just support OVERWRITE feature? When I tried for the appending, the query is not working. What should I do in such case so that I append the data in table? -- Thanks and Regards, Bhavesh Shah -- ...:::Aniket:::... Quetzalco@tl
RE: Related to INSERT QUERY in Hive
Hi Aniket/Bhavesh, Actually load data inpath is used for appending file and the path refers to the HDFS path. We can use load data local inpath in case the input file to be loaded is not in HDFS but it is present in the local system. Both of these append the data. In case, you need to overwrite the data, use load data inpath '/foo/write/bar.txt' OVERWRITE into table tblname; OR load data local inpath '/foo/write/bar.txt' OVERWRITE into table tblname; (in case the input file to be loaded is not in HDFS but it is present in the local system) Regards, Krishnan From: Aniket Mokashi [mailto:aniket...@gmail.com] Sent: Wednesday, January 18, 2012 10:56 AM To: user@hive.apache.org Subject: Re: Related to INSERT QUERY in Hive how about Load data inpath? On Tue, Jan 17, 2012 at 9:00 PM, Bhavesh Shah bhavesh25s...@gmail.commailto:bhavesh25s...@gmail.com wrote: Hello, I am using Hive-0.7.1. I want to append the data in table. Is hive-0.7.1 support appending feature or just support OVERWRITE feature? When I tried for the appending, the query is not working. What should I do in such case so that I append the data in table? -- Thanks and Regards, Bhavesh Shah -- ...:::Aniket:::... Quetzalco@tl CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS***
