Gopal V created HIVE-6748:
-----------------------------
Summary: FileSinkOperator needs to cleanup held references for
container reuse
Key: HIVE-6748
URL: https://issues.apache.org/jira/browse/HIVE-6748
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 0.13.0
Reporter: Gopal V
Assignee: Gopal V
The current implementation of FileSinkOperator runs into trouble when reusing
the same query pipeline aggressively with container reuse.
This is due to a prevFSP writer which is left referenced after closeOp() and
which is not reset even for initializeOp().
{code}
014-03-25 14:46:31,744 FATAL [main]
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor:
org.apache.hadoop.hive.ql.metadata.HiveException:
java.nio.channels.ClosedChannelException
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:170)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.startGroup(FileSinkOperator.java:833)
at
org.apache.hadoop.hive.ql.exec.Operator.defaultStartGroup(Operator.java:497)
at org.apache.hadoop.hive.ql.exec.Operator.startGroup(Operator.java:520)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:223)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:159)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:306)
at
org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:549)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:538)
Caused by: java.nio.channels.ClosedChannelException
at
org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1526)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.hive.ql.io.orc.WriterImpl$DirectStream.output(WriterImpl.java:316)
at org.apache.hadoop.hive.ql.io.orc.OutStream.flush(OutStream.java:242)
at
org.apache.hadoop.hive.ql.io.orc.WriterImpl.writeMetadata(WriterImpl.java:1923)
at
org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2017)
at
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:98)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:167)
... 13 more
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
