akshatshenoi-db opened a new pull request, #56527: URL: https://github.com/apache/spark/pull/56527
### What changes were proposed in this pull request? SPARK-57135 added reading CSV files packed in tar archives (`.tar`/`.tar.gz`/`.tgz`) and SPARK-57321 added schema inference for them; SPARK-57419 extended both to JSON. All are gated by `spark.sql.files.archive.reader.enabled`. This extends archive reading to the text data source. When the flag is enabled, the V1 text data source reads a tar archive as if it were a directory of its entries: each entry is streamed through `ArchiveReader` (never unpacked to disk) and read exactly like a standalone text file -- one row per line, or a single row holding the whole entry when `wholeText` is set (`TextFileFormat.readArchive`). The whole archive is one non-splittable unit (`isSplitable` returns false for an archive path), and a corrupt/missing archive is skipped as a unit under `ignoreCorruptFiles`/`ignoreMissingFiles`. Text has a fixed `value STRING` schema, so there is no schema inference. Archive scanning is wired into the V1 file source only; the DSv2 reader is left untouched. ### Why are the changes needed? To let text ingestion read tar archives without unpacking them to disk, matching the CSV and JSON behavior already in Spark. ### Does this PR introduce _any_ user-facing change? Yes. With `spark.sql.files.archive.reader.enabled=true` (default false), the text data source can read `.tar`/`.tar.gz`/`.tgz` files. ### How was this patch tested? New `TextTarArchiveReadSuite`: reads of multi-entry archives across all three extensions, parity with a directory read of the same files, `wholeText` and a custom line separator, empty and corrupt archives, the single-partition guarantee, and an archive mixed with loose files in the same directory. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
