Hi sir, the root cause of your issues seems to be java.io.EOFException, that based on the java doc description means the following:
"Signals that an end of file or end of stream has been reached unexpectedly
during input."
What is the health status of the box with ip 10.6.0.55? Isn't it by any chance
having some issues? Port 8021 is used for TaskTracker, so I would start by
connecting to that box and checking the TT logs.
Jarcec
On Thu, Feb 21, 2013 at 02:44:09PM +0300, Павел Мезенцев wrote:
> Hello!
>
> I use Hive 0.7.1 over Hadoop 0.20.2 (CHD3u3) on 70 nodes cluster.
> I have a trouble with query like this:
>
> *FROM* ( *SELECT* *id*, {expressions} *FROM* table1 *WHERE*
> day='2013-02-16' *AND* ({conditions1})*UNION* *ALL* *SELECT* *id*,
> {expressions} *FROM* table2 *WHERE* day='2013-02-16' *AND*
> (conditions)*UNION* *ALL* *SELECT* *id*, {expressions} *FROM* table3
> *WHERE* day='2013-02-16' *AND* (conditions)*UNION* *ALL* *SELECT*
> *id*, {expressions} *FROM* table4 *WHERE* day='2013-02-16' *AND*
> (conditions)) union_tmp*INSERT* OVERWRITE *table* result_table
> *PARTITION* (day='2013-02-16')*SELECT* *id*, transformations
> (expressions)*GROUP* *BY* *id*;
>
> it had 4865 map tasks and 100 reduce tasks.
> first 4780 map taks completed succefull and last
> 85 tasks hangs.
>
> All this tasks hands with no progress,
>
>
> and no tasks attempts for each:
>
> One hour later after hang situation starting, job failed with exeption:
>
> 2013-02-20 20:02:02,000 Stage-1 map = 0%, reduce = 0%
> 2013-02-20 20:02:40,679 Stage-1 map = 1%, reduce = 0%
> 2013-02-20 20:02:54,022 Stage-1 map = 2%, reduce = 0%
> 2013-02-20 20:03:14,129 Stage-1 map = 3%, reduce = 0%
> ......................................................
> 2013-02-20 21:18:00,361 Stage-1 map = 98%, reduce = 22%
> 2013-02-20 21:18:05,691 Stage-1 map = 98%, reduce = 23%
> java.io.IOException: Call to statlabjt/10.6.0.55:8021 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
> at org.apache.hadoop.ipc.Client.call(Client.java:1110)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> at org.apache.hadoop.mapred.$Proxy8.getJobStatus(Unknown Source)
> at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:1053)
> at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:1065)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.progress(ExecDriver.java:351)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:672)
> at
> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:425)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:815)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:724)
> Ended Job = job_201302152355_4764 with exception
> 'java.io.IOException(Call to statlabjt/10.6.0.55:8021 failed on local
> exception: java.io.EOFException)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
>
> How to find reasons of such situation?
> How to prevent such situations in future?
>
> Best regards
> Mezentsev Pavel
signature.asc
Description: Digital signature
