I am running DAGs generated by Hive using my custom Tez Client. So I
serialize a DAG, load it back and submit it later. Everything works great
the first time; however, on second runs the I get a RunTime exception
(snippet below)
My guess, it since the same DAG is run again, the output tables (have same
id) and that prevents overwrite.
Where should i introduce randomness in the file name ? Should I change some
name field in FileSinkDescriptor every time I re-run the dag ?
Thanks
Raajay
Vertex failed, vertexName=Reducer 3,
vertexId=vertex_1441949856963_0011_1_04, diagnostics=[Task failed,
taskId=task_1441949856963_0011_1_04_000000, diagnostics=[TaskAttempt 0
failed, info=[Error: Failure while running task:java.lang.RuntimeException:
java.lang.RuntimeException: Hive Runtime Error while closing operators:
Unable to rename output from: hdfs://
10.10.1.2:8020/apps/hive/output_tab/.hive-staging_hive_2015-09-11_00-07-40_043_6365145769624003668-1/_task_tmp.-ext-10000/_tmp.000000_0
to: hdfs://
10.10.1.2:8020/apps/hive/output_tab/.hive-staging_hive_2015-09-11_00-07-40_043_6365145769624003668-1/_tmp.-ext-10000/000000_0
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:345)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171
