akshatshenoi-db opened a new pull request, #57588: URL: https://github.com/apache/spark/pull/57588
### What changes were proposed in this pull request? commons-compress recommends preferring the random-access `ZipFile` over the streaming `ZipArchiveInputStream`: the streaming reader parses local file headers sequentially (so it can return entries absent from the central directory, duplicate names, and incomplete metadata) and cannot handle STORED entries sized only by a trailing data descriptor. `ZipFile` reads the central directory first, giving correct metadata and reliable extraction across all compression methods. This reuses the seekable-channel abstraction already built for 7z (`HadoopSeekableByteChannel`): `openArchiveStream` now returns an `Iterator[(ArchiveEntry, InputStream)]`, letting each container expose its natural shape -- tar/7z advance one forward cursor, while zip serves each entry's own `getInputStream` from the central directory. `ZipFile#canReadEntryData` flags encrypted / unsupported-method entries with a clear `CANNOT_READ_ZIP_ENTRY` error. The tar and 7z paths are unchanged behaviorally. This is part of the archive-reader series (SPARK-57135 CSV, SPARK-57321 CSV-infer, SPARK-57419 JSON, SPARK-57478 text, SPARK-57479 XML, SPARK-57481 Avro, SPARK-57705 zip container). ### Why are the changes needed? The streaming reader rejects valid zips -- notably STORED entries whose size lives only in a trailing data descriptor -- and exposes incomplete/duplicated entry metadata. Reading the central directory fixes both, and the per-entry byte offsets it exposes are a prerequisite for future zip parallelization. ### Does this PR introduce _any_ user-facing change? Yes, gated by `spark.sql.files.archive.reader.enabled` (default false): zip archives that previously failed to read (e.g. a STORED entry sized by a trailing data descriptor) now read successfully. ### How was this patch tested? Updated `SupportsArchiveFormatSuite`: the previously-non-streamable STORED+data-descriptor zip now reads successfully (assertion flipped); added duplicate-name, encrypted-entry (asserting `CANNOT_READ_ZIP_ENTRY` via `checkError`), and corrupt-archive cases. The existing tar and 7z cases are unchanged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
