ad1happy2go commented on issue #19340:
URL: https://github.com/apache/hudi/issues/19340#issuecomment-5507221924
@wangyinsheng Thanks for the detailed report. The signals you've listed
actually narrow this down a lot: the footer is intact, page CRCs pass, and only
part of the column chunk is unreadable. That tells us the file wasn't
physically damaged or truncated, it was encoded wrong in memory before the
bytes ever hit disk. A dictionary index landing out of range fits that too.
This kind of thing is almost always a concurrency or buffer-reuse race in the
writer, not storage.
A few questions to pin it down:
1. Is spark.speculation turned on? This is the most common cause of
exactly this symptom. A speculative task attempt can close and recycle a pooled
compressor while another attempt is still writing the same file. If it's on,
please set spark.speculation=false and let us know if new corruption stops
showing up.
2. For the instants that produced the bad files (like 20260524205029382),
were there any task or stage retries, or rollbacks? Do you ever see two
attempts writing the same fileId?
3. Can you share your write configs: hoodie.parquet.dictionary.enabled,
hoodie.parquet.compression.codec, hoodie.write.markers.type, and
hoodie.datasource.write.row.writer.enable?
4. For one corrupt file, could you run parquet-cli meta <file> and
parquet-cli dump --disable-data <file> and paste the output? We mainly want to
see which encoding the bad column uses.
5. Is it always string/binary columns, or have you seen it on numeric
columns too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
