[
https://issues.apache.org/jira/browse/HIVE-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417965#comment-13417965
]
Zhenxiao Luo commented on HIVE-3275:
------------------------------------
The reason is:
1. On hadoop0.20 or Hadoop0.23 MR1,
JobClient jc = new JobClient(job);
this line is throwing exception in ExecDriver.java.
It calls into JobClient.java:
public JobClient(JobConf conf) throws IOException {
setConf(conf);
init(conf);
}
/**
* Connect to the default {@link JobTracker}.
* @param conf the job configuration.
* @throws IOException
*/
public void init(JobConf conf) throws IOException {
String tracker = conf.get("mapred.job.tracker", "local");
if ("local".equals(tracker)) {
this.jobSubmitClient = new LocalJobRunner(conf);
} else {
this.jobSubmitClient = createRPCProxy(JobTracker.getAddress(conf), conf);
}
}
When createRPCProxy() is called, jobtracker is trying to getAddress(conf) of
the non-existed host(abracadabra), and throws the expected exception:
java.lang.IllegalArgumentException: Does not contain a valid host:port
authority: abracadabra
Here is the log and stack trace when running in hadoop0.20 to proof the above
observation:
2012-07-18 17:53:38,210 ERROR exec.Task (SessionState.java:printError(400)) -
Job Submission failed with exception 'java.lang.IllegalArgumentException(Does
not contain a valid host:port authority: abracadabra)'
java.lang.IllegalArgumentException: Does not contain a valid host:port
authority: abracadabra
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:206)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:158)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:147)
at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:2119)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:497)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:469)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:418)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1324)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1110)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:944)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:671)
at
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_autolocal1(TestNegativeCliDriver.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
2012-07-18 17:53:38,215 ERROR ql.Driver (SessionState.java:printError(400)) -
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MapRedTask
2. When running in hadoop0.23 MR2. MapReduce2 is using Yarn framework,
JobClient jc = new JobClient(job)
this line in ExecDriver.java is calling into MR2's implementation of JobClient:
public JobClient(Configuration conf) throws IOException {
init(new JobConf(conf));
}
/**
* Connect to the default cluster
* @param conf the job configuration.
* @throws IOException
*/
public void init(JobConf conf) throws IOException {
setConf(conf);
cluster = new Cluster(conf);
clientUgi = UserGroupInformation.getCurrentUser();
}
In MR2's implementation of init(JobConf), it does not try to getAddress(conf)
of the non-existed host(abracadabra). It only tries to initialize the cluster.
This is working OK with a proof of the following log:
2012-07-18 17:34:26,539 INFO exec.ExecDriver
(ExecDriver.java:addInputPaths(860)) - Adding input file
pfile:/home/cloudera/Code/hive/build/ql/test/data/warehouse/src
2012-07-18 17:34:26,539 INFO exec.Utilities (Utilities.java:isEmptyPath(1804))
- Content Summary
pfile:/home/cloudera/Code/hive/build/ql/test/data/warehouse/srclength: 5868 num
files: 2 num directories: 1
2012-07-18 17:34:26,877 DEBUG mapreduce.Cluster (Cluster.java:initialize(91)) -
Trying ClientProtocolProvider :
org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-07-18 17:34:26,877 DEBUG mapreduce.Cluster (Cluster.java:initialize(109))
- Cannot pick org.apache.hadoop.mapred.LocalClientProtocolProvider as the
ClientProtocolProvider - returned null protocol
2012-07-18 17:34:26,879 DEBUG mapreduce.Cluster (Cluster.java:initialize(91)) -
Trying ClientProtocolProvider :
org.apache.hadoop.mapred.YarnClientProtocolProvider
2012-07-18 17:34:26,972 DEBUG ipc.YarnRPC (YarnRPC.java:create(57)) - Creating
YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
2012-07-18 17:34:27,000 DEBUG mapred.ResourceMgrDelegate
(ResourceMgrDelegate.java:<init>(89)) - Connecting to ResourceManager at
/0.0.0.0:8032
2012-07-18 17:34:27,002 DEBUG ipc.HadoopYarnProtoRPC
(HadoopYarnProtoRPC.java:getProxy(45)) - Creating a HadoopYarnProtoRpc proxy
for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
2012-07-18 17:34:27,038 DEBUG ipc.Server
(Server.java:registerProtocolEngine(197)) - rpcKind=RPC_PROTOCOL_BUFFER,
rpcRequestWrapperClass=class
org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWritable,
rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@1b34cd7b
2012-07-18 17:34:27,166 DEBUG mapred.ResourceMgrDelegate
(ResourceMgrDelegate.java:<init>(93)) - Connected to ResourceManager at
/0.0.0.0:8032
2012-07-18 17:34:27,172 DEBUG security.UserGroupInformation
(UserGroupInformation.java:logPrivilegedAction(1254)) - PrivilegedAction
as:cloudera (auth:SIMPLE)
from:org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:319)
2012-07-18 17:34:27,186 DEBUG mapreduce.Cluster (Cluster.java:initialize(104))
- Picked org.apache.hadoop.mapred.YarnClientProtocolProvider as the
ClientProtocolProvider
An exception occurs in ExecDriver.java, when it is submitting the job to yarn:
rj = jc.submitJob(job);
Due to the non-existed host(abracadabra), JobClient.submitJob() fails with
exception, with a proof of the following log:
2012-07-18 17:34:27,235 DEBUG mapred.ResourceMgrDelegate
(ResourceMgrDelegate.java:getStagingAreaDir(276)) - getStagingAreaDir:
dir=/tmp/hadoop-yarn/staging/cloudera/.staging
2012-07-18 17:34:27,280 DEBUG ipc.Client (Client.java:<init>(262)) - The ping
interval is 60000 ms.
2012-07-18 17:34:27,287 DEBUG ipc.Client (Client.java:<init>(305)) - Use SIMPLE
authentication for protocol ClientRMProtocolPB
2012-07-18 17:34:27,287 DEBUG ipc.Client (Client.java:setupIOstreams(560)) -
Connecting to /0.0.0.0:8032
2012-07-18 17:34:28,295 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 0 time(s).
2012-07-18 17:34:29,297 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 1 time(s).
2012-07-18 17:34:30,299 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 2 time(s).
2012-07-18 17:34:31,301 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 3 time(s).
2012-07-18 17:34:32,303 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 4 time(s).
2012-07-18 17:34:33,305 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 5 time(s).
2012-07-18 17:34:34,306 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 6 time(s).
2012-07-18 17:34:35,308 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 7 time(s).
2012-07-18 17:34:36,311 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 8 time(s).
2012-07-18 17:34:37,312 INFO ipc.Client
(Client.java:handleConnectionFailure(683)) - Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 9 time(s).
2012-07-18 17:34:37,314 DEBUG ipc.Client (Client.java:close(917)) - closing ipc
connection to 0.0.0.0/0.0.0.0:8032: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:524)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:472)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:566)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:215)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1271)
at org.apache.hadoop.ipc.Client.call(Client.java:1141)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
at $Proxy9.getNewApplication(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:132)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:181)
at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:214)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:338)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1324)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1110)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:944)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:671)
at
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_autolocal1(TestNegativeCliDriver.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at junit.framework.TestCase.runTest(TestCase.java:168)
> Fix autolocal1.q testcase failure when building hive on hadoop0.23 MR2
> ----------------------------------------------------------------------
>
> Key: HIVE-3275
> URL: https://issues.apache.org/jira/browse/HIVE-3275
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
>
> autolocal1.q is failing only on hadoop0.23 MR2, due to cluster initialization
> problem:
> Begin query: autolocal1.q
> diff -a
> /var/lib/jenkins/workspace/zhenxiao-CDH4-Hive-0.9.0/build/ql/test/logs/clientnegative/autolocal1.q.out
>
> /var/lib/jenkins/workspace/zhenxiao-CDH4-Hive-0.9.0/ql/src/test/results/clientnegative/autolocal1.q.out
> 5c5
> < Job Submission failed with exception 'java.io.IOException(Cannot initialize
> Cluster. Please check your configuration for mapreduce.framework.name and the
> correspond server addresses.)'
> —
> > Job Submission failed with exception
> > 'java.lang.IllegalArgumentException(Does not contain a valid host:port
> > authority: abracadabra)'
> Exception: Client execution results failed with error code = 1
> See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get
> more logs.
> Failed query: autolocal1.q
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
