[
https://issues.apache.org/jira/browse/HIVE-30192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhaolong updated HIVE-30192:
----------------------------
Description:
Problem
When the same container JVM processes a self-read/write INSERT OVERWRITE ...
SELECT * FROM <same RCFile table> where the RCFile exceeds the split size, the
task writes fewer rows than expected. The map log shows EOFException, but the
job does not fail — data is silently writte short.
Trigger conditions (all required)
1. Table stored as RCFile;
2. File larger than split size, producing multiple logical InputSplits;
3. Two consecutive SQLs process the same file path in the same container;
4. The later SQL reads the file after it has been overwritten by the earlier
SQL;
5. No GC happens in that container between the two reads (cache entries are
not cleared);
6. hive.exec.rcfile.use.sync.cache is enabled (default).
Root cause
RCFileRecordReader keeps a JVM-wide static syncCache (Map<String,
RCFileSyncEntry>). Its key is only path + offset — it carries no file-length /
modification-time / version info. Workflow:
- While reading split, after finishing a logical slice the reader records key
= path + readEndOffset, value = lastSeenSyncPos (next read offset).
- Later, the same container opens the same path (already overwritten, new
content/length), and syncCache.get() hits the stale entry because GC has not
cleared it.
- The reader then seek(oldSync) and resumes from a position that points into
the middle of a record of the new file, so the requested read length exceeds
the remaining file size → java.io.EOFException raised in DataInputStream, and
the exception is not propagated — the record reader stops early and fewer
rows are written.
master branch still has this problem in rcfile.java
try
{ ret = nextKeyBuffer(); }
catch (EOFException eof)
{ LOG.warn("Failed to get next key buffer", eof); }
Suggested fix / workaround (upstream direction)
1. set hive.exec.rcfile.use.sync.cache=false (verified no performance impact).
2. throw eof exception
was:
Problem
When the same container JVM processes a self-read/write INSERT OVERWRITE ...
SELECT * FROM <same RCFile table> where the RCFile exceeds the split size, the
task writes fewer rows than expected. The map log shows EOFException, but the
job does not fail — data is silently writte short.
Trigger conditions (all required)
1. Table stored as RCFile;
2. File larger than split size, producing multiple logical InputSplits;
3. Two consecutive SQLs process the same file path in the same container;
4. The later SQL reads the file after it has been overwritten by the earlier
SQL;
5. No GC happens in that container between the two reads (cache entries are
not cleared);
6. hive.exec.rcfile.use.sync.cache is enabled (default).
Root cause
RCFileRecordReader keeps a JVM-wide static syncCache (Map<String,
RCFileSyncEntry>). Its key is only path + offset — it carries no file-length /
modification-time / version info. Workflow:
- While reading split, after finishing a logical slice the reader records key
= path + readEndOffset, value = lastSeenSyncPos (next read offset).
- Later, the same container opens the same path (already overwritten, new
content/length), and syncCache.get() hits the stale entry because GC has not
cleared it.
- The reader then seek(oldSync) and resumes from a position that points into
the middle of a record of the new file, so the requested read length exceeds
the remaining file size → java.io.EOFException raised in DataInputStream, and
the exception is not propagated — the record reader stops early and fewer
rows are written.
master branch still has this problem in rcfile.java
try {
ret = nextKeyBuffer();
} catch (EOFException eof) {
LOG.warn("Failed to get next key buffer", eof);
}
Suggested fix / workaround (upstream direction)
1. set hive.exec.rcfile.use.sync.cache=false (verified no performance impact).
2. throw eof exception
> RCFileRecordReader reuses stale sync-cache offset after file overwrite,
> causing silent data loss in insert-overwrite self read/write
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30192
> URL: https://issues.apache.org/jira/browse/HIVE-30192
> Project: Hive
> Issue Type: Bug
> Components: Reader
> Affects Versions: 3.1.0, 3.1.3, 4.2.1
> Reporter: zhaolong
> Priority: Critical
>
> Problem
> When the same container JVM processes a self-read/write INSERT OVERWRITE
> ... SELECT * FROM <same RCFile table> where the RCFile exceeds the split
> size, the task writes fewer rows than expected. The map log shows
> EOFException, but the job does not fail — data is silently writte short.
>
> Trigger conditions (all required)
> 1. Table stored as RCFile;
> 2. File larger than split size, producing multiple logical InputSplits;
> 3. Two consecutive SQLs process the same file path in the same container;
> 4. The later SQL reads the file after it has been overwritten by the
> earlier SQL;
> 5. No GC happens in that container between the two reads (cache entries are
> not cleared);
> 6. hive.exec.rcfile.use.sync.cache is enabled (default).
>
> Root cause
> RCFileRecordReader keeps a JVM-wide static syncCache (Map<String,
> RCFileSyncEntry>). Its key is only path + offset — it carries no file-length
> / modification-time / version info. Workflow:
> - While reading split, after finishing a logical slice the reader records
> key = path + readEndOffset, value = lastSeenSyncPos (next read offset).
> - Later, the same container opens the same path (already overwritten, new
> content/length), and syncCache.get() hits the stale entry because GC has not
> cleared it.
> - The reader then seek(oldSync) and resumes from a position that points
> into the middle of a record of the new file, so the requested read length
> exceeds the remaining file size → java.io.EOFException raised in
> DataInputStream, and the exception is not propagated — the record reader
> stops early and fewer rows are written.
> master branch still has this problem in rcfile.java
> try
> { ret = nextKeyBuffer(); }
> catch (EOFException eof)
> { LOG.warn("Failed to get next key buffer", eof); }
>
> Suggested fix / workaround (upstream direction)
> 1. set hive.exec.rcfile.use.sync.cache=false (verified no performance impact).
> 2. throw eof exception
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
