Lifeng Wang created HIVE-12229:
----------------------------------
Summary: Custom script in query cannot be executed in yarn-cluster
mode [Spark Branch].
Key: HIVE-12229
URL: https://issues.apache.org/jira/browse/HIVE-12229
Project: Hive
Issue Type: Bug
Components: Spark
Affects Versions: 1.1.0
Reporter: Lifeng Wang
Added one python script in the query and the python script cannot be found
during execution in yarn-cluster mode.
{noformat}
15/10/21 21:10:55 INFO exec.ScriptOperator: Executing [/usr/bin/python,
q2-sessionize.py, 3600]
15/10/21 21:10:55 INFO exec.ScriptOperator: tablename=null
15/10/21 21:10:55 INFO exec.ScriptOperator: partname=null
15/10/21 21:10:55 INFO exec.ScriptOperator: alias=null
15/10/21 21:10:55 INFO spark.SparkRecordHandler: processing 10 rows: used
memory = 324896224
15/10/21 21:10:55 INFO exec.ScriptOperator: ErrorStreamProcessor calling
reporter.progress()
/usr/bin/python: can't open file 'q2-sessionize.py': [Errno 2] No such file or
directory
15/10/21 21:10:55 INFO exec.ScriptOperator: StreamThread OutputProcessor done
15/10/21 21:10:55 INFO exec.ScriptOperator: StreamThread ErrorProcessor done
15/10/21 21:10:55 INFO spark.SparkRecordHandler: processing 100 rows: used
memory = 325619920
15/10/21 21:10:55 ERROR exec.ScriptOperator: Error in writing to script: Stream
closed
15/10/21 21:10:55 INFO exec.ScriptOperator: The script did not consume all
input data. This is considered as an error.
15/10/21 21:10:55 INFO exec.ScriptOperator: set
hive.exec.script.allow.partial.consumption=true; to ignore it.
15/10/21 21:10:55 ERROR spark.SparkReduceRecordHandler: Fatal error:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while processing row
(tag=0)
{"key":{"reducesinkkey0":2,"reducesinkkey1":3316240655},"value":{"_col0":5529}}
org.apache.hadoop.hive.ql.metadata.HiveException: Error while processing row
(tag=0)
{"key":{"reducesinkkey0":2,"reducesinkkey1":3316240655},"value":{"_col0":5529}}
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processKeyValues(SparkReduceRecordHandler.java:340)
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processRow(SparkReduceRecordHandler.java:289)
at
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.processNextRecord(HiveReduceFunctionResultList.java:49)
at
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.processNextRecord(HiveReduceFunctionResultList.java:28)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.insertAll(BypassMergeSortShuffleWriter.java:99)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20001]: An
error occurred while reading or writing to your custom script. It may have
crashed with an error.
at
org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:453)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processKeyValues(SparkReduceRecordHandler.java:331)
... 14 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
