zhaolong created HIVE-30192:
-------------------------------
Summary: RCFileRecordReader reuses stale sync-cache offset after
file overwrite, causing silent data loss in insert-overwrite self read/write
Key: HIVE-30192
URL: https://issues.apache.org/jira/browse/HIVE-30192
Project: Hive
Issue Type: Bug
Components: Reader
Affects Versions: 4.2.1, 3.1.3, 3.1.0
Reporter: zhaolong
Problem
When the same container JVM processes a self-read/write INSERT OVERWRITE ...
SELECT * FROM <same RCFile table> where the RCFile exceeds the split size, the
task writes fewer rows than expected. The map log shows EOFException, but the
job does not fail — data is silently writte short.
Trigger conditions (all required)
1. Table stored as RCFile;
2. File larger than split size, producing multiple logical InputSplits;
3. Two consecutive SQLs process the same file path in the same container;
4. The later SQL reads the file after it has been overwritten by the earlier
SQL;
5. No GC happens in that container between the two reads (cache entries are
not cleared);
6. hive.exec.rcfile.use.sync.cache is enabled (default).
Root cause
RCFileRecordReader keeps a JVM-wide static syncCache (Map<String,
RCFileSyncEntry>). Its key is only path + offset — it carries no file-length /
modification-time / version info. Workflow:
- While reading split, after finishing a logical slice the reader records key
= path + readEndOffset, value = lastSeenSyncPos (next read offset).
- Later, the same container opens the same path (already overwritten, new
content/length), and syncCache.get() hits the stale entry because GC has not
cleared it.
- The reader then seek(oldSync) and resumes from a position that points into
the middle of a record of the new file, so the requested read length exceeds
the remaining file size → java.io.EOFException raised in DataInputStream, and
the exception is not propagated — the record reader stops early and fewer
rows are written.
master branch still has this problem in rcfile.java
try {
ret = nextKeyBuffer();
} catch (EOFException eof) {
LOG.warn("Failed to get next key buffer", eof);
}
Suggested fix / workaround (upstream direction)
1. set hive.exec.rcfile.use.sync.cache=false (verified no performance impact).
2. throw eof exception
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
