[
https://issues.apache.org/jira/browse/SPARK-57135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshat Shenoi updated SPARK-57135:
----------------------------------
Description:
Spark cannot currently read CSV files packaged inside tar archives (.tar,
.tar.gz, .tgz); users must unpack them externally first.
This adds opt-in support (spark.sql.files.archive.reader.enabled, default
false) for reading such archives through the CSV data source by streaming each
entry through the CSV parser, without materializing entries to local disk:
* A streaming ArchiveReader opens the tar once and yields one bounded
InputStream per entry, advancing lazily so memory
stays bounded regardless of archive size. Directories and dot-prefixed
entries are skipped. .tar.gz is decompressed via
Hadoop's codec factory; .tgz is gunzipped explicitly. ArchiveReader is an
abstract base (TarArchiveReader is the only
implementation today), so other archive formats can be added as additive
subclasses.
* CSVFileFormat treats archives as non-splittable (one split per archive) and
streams each entry through UnivocityParser,
handling each entry as a standalone CSV file (headers, multiLine, delimiters,
column pruning).
Scope: CSV reads over tar only. Schema inference from archives, and other file
formats (e.g. JSON, text, XML), are left to follow-ups. Streaming supports
formats parseable sequentially; formats needing random access (Parquet/ORC
footers) cannot stream from a tar and are out of scope.
was:
Spark cannot currently read CSV files packaged inside tar archives (.tar,
.tar.gz, .tgz); users must unpack them externally first.
This adds opt-in support (spark.sql.files.archive.enabled, default false) for
reading such archives through the CSV data source by streaming each entry
through the CSV parser, without materializing entries to local disk:
* A streaming ArchiveReader opens the tar once and yields one bounded
InputStream per entry, advancing lazily so memory stays bounded regardless of
archive size. Directories and dot-prefixed entries are skipped. .tar.gz is
decompressed via Hadoop's codec factory; .tgz is gunzipped explicitly.
* CSVFileFormat treats archives as non-splittable (one split per archive) and
streams each entry through UnivocityParser, handling each entry as a standalone
CSV file (headers, multiLine, delimiters, column pruning).
* Schema inference streams entries through the same CSVInferSchema path as a
multi-file CSV read.
Scope: CSV only. Streaming supports formats parseable sequentially; formats
needing random access (Parquet/ORC footers) cannot stream from a tar and are
out of scope.
(Supersedes the original proposal, which materialized entries to temp files via
a format-agnostic ArchiveFormat.)
> [SQL] Support reading CSV files inside tar archives
> ---------------------------------------------------
>
> Key: SPARK-57135
> URL: https://issues.apache.org/jira/browse/SPARK-57135
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Akshat Shenoi
> Priority: Major
> Labels: pull-request-available
>
> Spark cannot currently read CSV files packaged inside tar archives (.tar,
> .tar.gz, .tgz); users must unpack them externally first.
> This adds opt-in support (spark.sql.files.archive.reader.enabled, default
> false) for reading such archives through the CSV data source by streaming
> each entry through the CSV parser, without materializing entries to local
> disk:
> * A streaming ArchiveReader opens the tar once and yields one bounded
> InputStream per entry, advancing lazily so memory
> stays bounded regardless of archive size. Directories and dot-prefixed
> entries are skipped. .tar.gz is decompressed via
> Hadoop's codec factory; .tgz is gunzipped explicitly. ArchiveReader is an
> abstract base (TarArchiveReader is the only
> implementation today), so other archive formats can be added as additive
> subclasses.
> * CSVFileFormat treats archives as non-splittable (one split per archive)
> and streams each entry through UnivocityParser,
> handling each entry as a standalone CSV file (headers, multiLine,
> delimiters, column pruning).
> Scope: CSV reads over tar only. Schema inference from archives, and other
> file formats (e.g. JSON, text, XML), are left to follow-ups. Streaming
> supports formats parseable sequentially; formats needing random access
> (Parquet/ORC footers) cannot stream from a tar and are out of scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
