[
https://issues.apache.org/jira/browse/TEZ-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah reassigned TEZ-1097:
--------------------------------
Assignee: Hitesh Shah
> Tez assumes that the scratch directory has to be same as the default
> filesystem
> -------------------------------------------------------------------------------
>
> Key: TEZ-1097
> URL: https://issues.apache.org/jira/browse/TEZ-1097
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Chuan Liu
> Assignee: Hitesh Shah
>
> In HDInsight Azure, we have hive scratch directory set to using HDFS. However
> the default filesystem is WASB.
> Due to this configuration, Tez seems to fail.
> The relevant error logs are as follows.
> {noformat}
> 2014-05-01 04:09:15,007 INFO tez.TezSessionState
> (TezSessionState.java:open(142)) - User of session id
> 41905b13-36e7-4755-af34-265768778443 is chuan
> 2014-05-01 04:09:15,054 INFO tez.DagUtils
> (DagUtils.java:getHiveJarDirectory(760)) - Jar dir is null/directory doesn't
> exist. Choosing HIVE_INSTALL_DIR -
> wasb://[email protected]/user/chuan/.hiveJars
> 2014-05-01 04:09:15,382 INFO tez.DagUtils
> (DagUtils.java:localizeResource(839)) - Localizing resource because it does
> not exist:
> file:/C:/apps/dist/hive-0.13.0.2.1.2.0-1660/lib/hive-exec-0.13.0.2.1.2.0-1660.jar
> to dest:
> wasb://[email protected]/user/chuan/.hiveJars/hive-exec-0.13.0.2.1.2.0-1660-3abc67a6088fef5988020d9608d28d3025115d4201be785434c57c8a90d4cb5f.jar
> 2014-05-01 04:09:15,413 INFO tez.DagUtils
> (DagUtils.java:localizeResource(843)) - Looks like another thread is writing
> the same file will wait.
> 2014-05-01 04:09:15,413 INFO tez.DagUtils
> (DagUtils.java:localizeResource(850)) - Number of wait attempts: 5. Wait
> interval: 5000
> 2014-05-01 04:09:15,444 INFO tez.DagUtils
> (DagUtils.java:createLocalResource(603)) - Resource modification time:
> 1398916679000
> 2014-05-01 04:09:15,444 ERROR exec.Task (TezTask.java:execute(192)) - Failed
> to execute tez graph.
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://headnodehost:9000/hive/scratch/_tez_session_dir/41905b13-36e7-4755-af34-265768778443,
> expected: wasb://[email protected]
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
> at
> org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.checkPath(NativeAzureFileSystem.java:436)
> at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:747)
> at org.apache.tez.client.AMConfiguration.<init>(AMConfiguration.java:81)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:174)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:154)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> 2014-05-01 04:09:15,444 ERROR ql.Driver (SessionState.java:printError(546)) -
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
