[ https://issues.apache.org/jira/browse/FLINK-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188106#comment-17188106 ]
Jingsong Lee commented on FLINK-18915: -------------------------------------- CC: [~gaoyunhaii] > FIXED_PATH(dummy Hadoop Path) with WriterImpl may cause ORC writer OOM > ---------------------------------------------------------------------- > > Key: FLINK-18915 > URL: https://issues.apache.org/jira/browse/FLINK-18915 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) > Affects Versions: 1.11.0, 1.11.1 > Reporter: wei > Priority: Critical > Fix For: 1.11.2 > > > # OrcBulkWriterFactory > {code:java} > @Override > public BulkWriter<T> create(FSDataOutputStream out) throws IOException { > OrcFile.WriterOptions opts = getWriterOptions(); > opts.physicalWriter(new PhysicalWriterImpl(out, opts)); > return new OrcBulkWriter<>(vectorizer, new WriterImpl(null, FIXED_PATH, > opts)); > }{code} > > # MemoryManagerImpl > {code:java} > // > public void addWriter(Path path, long requestedAllocation, > Callback callback) throws IOException { > checkOwner(); > WriterInfo oldVal = writerList.get(path); > // this should always be null, but we handle the case where the memory > // manager wasn't told that a writer wasn't still in use and the task > // starts writing to the same path. > if (oldVal == null) { > oldVal = new WriterInfo(requestedAllocation, callback); > writerList.put(path, oldVal); > totalAllocation += requestedAllocation; > } else { > // handle a new writer that is writing to the same path > totalAllocation += requestedAllocation - oldVal.allocation; > oldVal.allocation = requestedAllocation; > oldVal.callback = callback; > } > updateScale(true); > } > {code} > SinkTask may have multi BulkWriter create, FIXED_PATH will cause overlay the > last writer callback;Last writer's WriterImpl#checkMemory will never called; > > -- This message was sent by Atlassian Jira (v8.3.4#803005)