akshatshenoi-db opened a new pull request, #57388:
URL: https://github.com/apache/spark/pull/57388
### What changes were proposed in this pull request?
This extends the tar/zip archive-reader feature to Parquet, continuing the
series
(SPARK-57135 CSV, SPARK-57321 CSV inference, SPARK-57419 JSON, SPARK-57478
text,
SPARK-57479 XML, SPARK-57481 Avro, SPARK-57705 zip container, and
SPARK-58110 which
extracted the shared `SupportsArchiveFormat` trait). With the feature
enabled, an
archive (`.tar`/`.tar.gz`/`.tgz`/`.zip`) of Parquet files is read like a
directory of
its entries.
Parquet is the first *random-access* archive format. The streaming formats
parse each
entry straight from an `InputStream`, but Parquet needs its trailing footer,
so an
archive entry cannot be streamed. This PR:
- Adds the random-access localization surface to `SupportsArchiveFormat`:
the trait
gains `readLocalizedEntries` (unpack each kept entry to a temp file, one
at a time,
read it, delete it) and the `archiveEntryFilter` hook; the companion gains
`copyEntryToLocalFile` and `localizeEntries` (so an executor-side
inference path can
localize without a trait instance). This is the surface that was deferred
from
SPARK-58110 until its first consumer landed - this is that consumer.
- `ParquetFileFormat` mixes in the trait, returns `isSplitable = false` for
an archive
(the archive is one split), dispatches an archive read through
`readLocalizedEntries`
reusing the existing plain per-file reader, and folds per-entry footer
reads into
`readParquetFootersInParallel` so schema inference expands archives. A
corrupt entry
condemns its whole archive during inference (matching the whole-file
semantics of a
corrupt loose Parquet file), while a corrupt entry during a read skips the
rest of
that archive as the streaming formats do.
- `input_file_name()` / `_metadata.file_path` report the archive path, not a
per-entry
temp path.
The SPARK-37089 use-after-free guard is preserved: the per-task completion
listener
deletes only the temp dir; `FileScanRDD` closes the per-entry reader so the
vectorized
reader's off-heap column vectors are not freed while still in use.
This also fixes a stale doc reference left by SPARK-58110 in
`SupportsArchiveFormatSuite` (the comment pointed at the removed
`ZipArchiveReader`'s
scaladoc).
### Why are the changes needed?
Parquet is a common archived format, and the archive-reader feature already
supports the
text-oriented formats and Avro. Extending it to Parquet lets users ingest
tarred/zipped
Parquet without unpacking first, and establishes the random-access path that
ORC and
other footer-based formats will reuse.
### Does this PR introduce _any_ user-facing change?
Yes, gated by `spark.sql.files.archive.reader.enabled` (default `false`).
When enabled,
reading a path that is a supported archive of Parquet files returns the rows
of its
entries; when disabled (the default), behavior is unchanged. Loose Parquet
reads are
unaffected either way.
### How was this patch tested?
Added `ParquetTarArchiveReadSuite` and `ParquetZipArchiveReadSuite` (each
binding the new
`ParquetArchiveReadBase` to the tar and zip harnesses), covering vectorized
and
row-based reads, schema inference (including `mergeSchema` union and type
widening across
entries), `input_file_name`/`_metadata` parity with the archive path,
per-entry temp-file
cleanup on an abandoned (`LIMIT`) read, and
`ignoreCorruptFiles`/`ignoreMissingFiles`
behavior for both read and inference. Extended the shared
`ArchiveReadSuiteBase` with an
`inferenceSamplesOneFile` hook (Parquet samples one part-file per
SPARK-11500) and an
`input_file_name` parity test.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
