Hi Huan,
I guess this problem is caused by incompatibility between hadoop 0.17.0 and
0.18.3. Please ignore my previous comments about compiling hive against
0.17.0 and use it on 0.18.3 then.
Can you try this?
*ant clean
ant -Dtarget.dir=/hive -Dhadoop.version='0.18.3' package
*
I took a deeper look at the code. I think the only possibility that you
encounter that "AbstractMethod" problem is that you built hive using "ant
package", and then "ant package -Dhadoop.version=0.18.3" without "ant
clean".
Our build.xml rules get confused because of that.
Please try and let us know how it goes.
Zheng
On Tue, Feb 24, 2009 at 9:38 AM, hc busy <hc.b...@gmail.com> wrote:
> Okay, so our people here looked at this further. In the Hadoop source, the
> exception is thrown every time the constructor is called so it means
> nothing.
>
> We're in a crunch to get this up and running today and would really
> appreciate some help:
>
> *The console hangs with any query that causes a map/reduce ( so select *
> from table works, but that’s it). From what I can tell, any query that tries
> to submit a job to Hadoop hangs, everything else works. So creating tables,
> deleting them, importing and exporting data all work.
>
> A simple query we can try is
>
> *select count(1) from english;
> *
>
> This fails, with example output below.
>
> Other very simple queries that trigger a Map/Reduce do the same.
>
> Thanks!*
>
> hive> select count(1) from english;
> Total MapReduce jobs = 2
> Number of reduce tasks not specified. Defaulting to jobconf value of: 16
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=<number>
> *
> The logs have nothing useful that I can see after that. Instead, here is a
> stack dump of the CLI threads that may be of interest (using jstack).
> Additionally, the client no longer responds to Ctrl-D for shutdown, and
> only stops with the more forceful Ctrl-C.*
>
> "main" prio=10 tid=0x0000000055e70000 nid=0x4605 in Object.wait()
> [0x0000000040208000..0x0000000040209ec0]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaadef20cc8> (a org.apache.hadoop.ipc.Client$Call)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.hadoop.ipc.Client.call(Client.java:710)
> - locked <0x00002aaadef20cc8> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at org.apache.hadoop.mapred.$Proxy1.getNewJobId(Unknown Source)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:725)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:391)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:238)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:306)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>
> Locked ownable synchronizers:
> - None
>
> "IPC Client (47) connection to /10.3.0.66:55554 from an unknown user"
> daemon prio=10 tid=0x00002aaaf9038000 nid=0x464a runnable
> [0x000000004171e000..0x000000004171ea00]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00002aaab416d6a8> (a sun.nio.ch.Util$1)
> - locked <0x00002aaab416d690> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00002aaab416d300> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:260)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:155)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:150)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:123)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:273)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> - locked <0x00002aaadeec6288> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readInt(DataInputStream.java:370)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:500)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:442)
>
> Locked ownable synchronizers:
> - None
>
>
>
>
>
> On Mon, Feb 23, 2009 at 9:00 PM, hc busy <hc.b...@gmail.com> wrote:
>
>>
>> Okay, great! got some help from Zheng, and got it past that point. But
>> now, instead of throwing that exception, the hive CLI hangs with the hadoop
>> cluster idle. We went in and looked at the logs and here's what it says:
>>
>> 2009-02-23 20:55:49,706 DEBUG parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:genSelectPlan(1120)) - genSelectPlan: input =
>> {((tok_function count 1),0: bigint)}
>> 2009-02-23 20:55:49,707 DEBUG parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:genSelectPlan(1196)) - Created Select Plan for
>> clause: insclause-0 row schema: null{(_c0,0: bigint)}
>> 2009-02-23 20:55:49,707 DEBUG serde2.MetadataTypedColumnsetSerDe
>> (MetadataTypedColumnsetSerDe.java:initialize(121)) -
>> org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe: initialized with
>> columnNames: [_c0] and separator code=1 lastColumnTakesRest=false
>> splitLimit=-1
>> 2009-02-23 20:55:49,707 DEBUG parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:genFileSinkPlan(2092)) - Created FileSink Plan for
>> clause: insclause-0dest_path: /tmp/hive-candiru/490583327.10000 row schema:
>> null{(_c0,0: bigint)}
>> 2009-02-23 20:55:49,708 DEBUG parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:genBodyPlan(2784)) - Created Body Plan for Query
>> Block null
>> 2009-02-23 20:55:49,708 DEBUG parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:genPlan(3072)) - Created Plan for Query Block null
>> 2009-02-23 20:55:49,710 INFO parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:analyzeInternal(3266)) - Completed partition pruning
>> 2009-02-23 20:55:49,710 INFO parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:analyzeInternal(3270)) - Completed sample pruning
>> 2009-02-23 20:55:49,720 DEBUG conf.Configuration
>> (Configuration.java:<init>(157)) - java.io.IOException: config()
>> at
>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:157)
>> at
>> org.apache.hadoop.hive.ql.exec.ExecDriver.<clinit>(ExecDriver.java:128)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at java.lang.Class.newInstance0(Class.java:355)
>> at java.lang.Class.newInstance(Class.java:308)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskFactory.get(TaskFactory.java:101)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskFactory.get(TaskFactory.java:116)
>> at
>> org.apache.hadoop.hive.ql.optimizer.GenMRTableScan1.process(GenMRTableScan1.java:54)
>> at
>> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:80)
>> at
>> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:83)
>> at
>> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:51)
>> at
>> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:95)
>> at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:3202)
>> at
>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:3277)
>> at
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:71)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:193)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:306)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>>
>> 2009-02-23 20:55:49,741 DEBUG optimizer.GenMapRedUtils
>> (GenMapRedUtils.java:setTaskPlan(209)) - Adding
>> hdfs://machine:port/user/hive/warehouse/english of tableenglish
>> 2009-02-23 20:55:49,741 DEBUG optimizer.GenMapRedUtils
>> (GenMapRedUtils.java:setTaskPlan(216)) - Information added for path
>> hdfs://machine:port/user/hive/warehouse/english
>> 2009-02-23 20:55:49,742 DEBUG optimizer.GenMapRedUtils
>> (GenMapRedUtils.java:setTaskPlan(221)) - Created Map Work for english
>> 2009-02-23 20:55:49,743 INFO parse.SemanticAnalyzer
>> (SemanticAnalyzer.java:analyzeInternal(3279)) - Completed plan generation
>> 2009-02-23 20:55:49,744 DEBUG conf.Configuration
>> (Configuration.java:<init>(171)) - java.io.IOException: config(config)
>> at
>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:171)
>> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:130)
>> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:140)
>> at
>> org.apache.hadoop.hive.ql.exec.ExecDriver.initialize(ExecDriver.java:88)
>> at org.apache.hadoop.hive.ql.exec.Task.initialize(Task.java:84)
>> at
>> org.apache.hadoop.hive.ql.exec.ExecDriver.initialize(ExecDriver.java:87)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:215)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:306)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>>
>> On Mon, Feb 23, 2009 at 7:57 PM, hc busy <hc.b...@gmail.com> wrote:
>>
>>>
>>> Also, we tried branch-0.2 and trunk against 0.18.2 and 0.18.3; We saw a
>>> few notes on this error before related building against proper hadoop
>>> version but that didn't seem to help here, what else could be causing this?
>>>
>>>
>>>
>>>
>>> On Mon, Feb 23, 2009 at 7:48 PM, hc busy <hc.b...@gmail.com> wrote:
>>>
>>>> *Setting*:
>>>> Hadoop 0.18.3, on several nodes that is able to run mr jobs.
>>>> Hive trunk built with "ant -Dtarget.dir=/hive -Dhadoop.version='0.18.3'
>>>> package", and then deployed by copying build/dist/* to /hive;
>>>> $HADOOP_HOME, $HADOOP, $HIVE_HOME are all configured correctly.
>>>>
>>>> I imported some list of English words into a table called english. It is
>>>> a table with one column of string that I can do 'select * from english;'
>>>> BUT!! the following fails. Can anybody help?
>>>>
>>>> .
>>>> .
>>>> .
>>>> courtesan
>>>> courtesanry
>>>> courtesans
>>>> courtesanship
>>>> courtesied
>>>> courtesies
>>>> courtesy
>>>> courtesy
>>>> Time taken: 4.584 seconds
>>>> *hive*> *select count(1) from english;*
>>>>
>>>> Total MapReduce jobs = 2
>>>> Number of reduce tasks not specified. Defaulting to jobconf value of: 16
>>>> In order to change the average load for a reducer (in bytes):
>>>> set hive.exec.reducers.bytes.per.reducer=<number>
>>>> In order to limit the maximum number of reducers:
>>>> set hive.exec.reducers.max=<number>
>>>> In order to set a constant number of reducers:
>>>> set mapred.reduce.tasks=<number>
>>>> java.lang.AbstractMethodError:
>>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.validateInput(Lorg/apache/hadoop/mapred/JobConf;)V
>>>> at
>>>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:735)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:391)
>>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:238)
>>>> at
>>>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
>>>> at
>>>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
>>>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:306)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>>>> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>>>>
>>>>
>>>
>>
>
--
Yours,
Zheng
