lukekim opened a new pull request, #27593:
URL: https://github.com/apache/datafusion/pull/27593
## Which issue does this PR close?
- Closes #27592.
## Rationale for this change
A CSV or JSON file compressed with bzip2, xz or zstd and made of several
concatenated members, such as `cat a.csv.zst b.csv.zst` produces, returns only
the first member's rows when it is read from an object store other than the
local file system (HTTP, S3, GCS, Azure, the in-memory store), with no error.
Read from the local file system, the same file returns every row. #27592 has a
`datafusion-cli` reproduction over HTTP.
The file openers decode a `GetResultPayload::File` with
`FileCompressionType::convert_read` and a `GetResultPayload::Stream` with
`FileCompressionType::convert_stream`. `convert_read` decodes every member for
all four codecs (#5628). `convert_stream` enabled `multiple_members` only on
the gzip decoder (#9175).
## What changes are included in this PR?
- `convert_stream` calls `multiple_members(true)` on the bzip2, xz and zstd
decoders, as it already does for gzip. Its doc comment now says that every
member is decoded, as with `convert_read`.
- Tests (below).
## What is the testing strategy for this PR?
- `datafusion/core/tests/datasource/compression.rs` (`core_integration`),
`concatenated_members_read_in_full`: for each codec, 20,000 rows of CSV,
newline-delimited JSON and JSON array, split partway through a row into two
members, are read with `read_csv` / `read_json` from a local file and from an
`InMemory` object store, and `count(a), sum(a)` must equal `(20000,
199990000)`. gzip and the local-file reads are controls that already pass.
- `datafusion/datasource/src/file_compression_type.rs` unit tests, for
decoder behavior that a query cannot control:
- `concatenated_members_decode_in_full`: three members (5,000 lines, an
empty member, 2,000 lines) per codec, decoded with `convert_read` and with
`convert_stream`, with the stream fed as one chunk, one chunk per member, and
in 1-byte and 13-byte chunks, so member boundaries fall at every position
within a chunk.
- `bytes_after_the_last_member_are_an_error`: a member followed by
`garbage` is an error through both `convert_read` and `convert_stream`.
- The members are compressed with each codec's own encoder crate (`flate2`,
`bzip2`, `liblzma`, `zstd`), not with DataFusion.
- No `.slt` test: sqllogictest reads local files, which are scanned through
`convert_read` and already return every row.
### Verification
Run on macOS (arm64), rustc 1.99.1, at `be69f8cfa` on `main` at `657d2cab4`.
The pushed commit, `f35486a31`, differs from it only in its message (a
`Generated-by:` trailer); both have tree `e7b7d216`.
The new tests, without the fix (this branch with `convert_stream`'s decoders
restored to `main`'s) and with it:
```
$ cargo test -p datafusion-datasource --lib file_compression_type
# without the fix
test file_compression_type::tests::bytes_after_the_last_member_are_an_error
... FAILED
test file_compression_type::tests::concatenated_members_decode_in_full ...
FAILED
"FileCompressionType { variant: BZIP2 } convert_stream: [97, 10]",
"FileCompressionType { variant: ZSTD } convert_stream, one chunk (1):
23890 bytes",
... # of 33890: every chunking, also XZ
test result: FAILED. 2 passed; 2 failed
# with the fix
test result: ok. 4 passed; 0 failed
$ cargo test -p datafusion --test core_integration -- datasource::compression
# without the fix
"Csv.csv.zst from object store: (10926, 59673442)", # also
.bz2, .xz
"NdJson.json.zst from object store: (10463, 54731953)", # also
.bz2, .xz
"JsonArray.json.zst from object store: Arrow error: Json error:
Truncated record whilst reading string",
test result: FAILED. 0 passed; 1 failed
# with the fix
test datasource::compression::concatenated_members_read_in_full ... ok
```
The reproduction in #27592, with `datafusion-cli` built from this branch:
the HTTP table returns `20000 | 199990000`, the same as the local file (55.3.0
returns `10000 | 49995000`). The same holds for `.bz2` and `.xz`.
Data after the last member, read from a local file and from an `InMemory`
store (a temporary test, not in this PR):
| After one member | Store, before | Store, after | Local file, before and
after |
|---|---|---|---|
| `garbage` (bzip2, xz, zstd) | rows returned | error | error |
| half of a second member (bzip2, xz, zstd) | rows returned | error | error |
| xz 4-byte stream padding | rows returned | rows returned | rows returned |
| zstd skippable frame | rows returned | rows returned | rows returned |
Checks at `be69f8cfa`:
- `ci/scripts/rust_fmt.sh`, `ci/scripts/license_header.sh`,
`ci/scripts/typos_check.sh`, `ci/scripts/rust_clippy.sh` (both feature sets)
and `ci/scripts/rust_docs.sh`: all exit 0.
- `cargo test --profile ci --exclude datafusion-examples --exclude
datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins
--features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
--no-fail-fast`: 13,540 passed, 0 failed, 8 ignored.
- Not run locally: the TOML, prettier, link, `cargo audit` and `cargo
machete` steps of `dev/rust_lint.sh`, since no TOML, docs or dependency file
changes. The fork CI run covers the full workflow.
<details>
<summary>Single-member files: decode latency without and with the
fix</summary>
Latency of decoding one single-member file through `convert_stream`,
measured by a temporary test (not in this PR) in a release build. Eight
interleaved runs (without, with, with, without, without, with, with, without)
on an Apple M5 Ultra, with background load of about 7 to 9 on 36 cores. gzip is
the control: its code is the same in both builds. Values are per run, in µs.
| File (compressed → decoded) | Samples per run | P99 without | P99 with |
P99.9 without | P99.9 with |
|---|---|---|---|---|---|
| gzip, 349 B → 673 B (control) | 20,000 | 4.5, 4.5, 5.3, 4.4 | 4.4, 4.4,
4.5, 5.4 | 9.2, 6.3, 20.7, 6.2 | 8.7, 6.2, 7.9, 11.3 |
| bzip2, 302 B → 673 B | 20,000 | 22.2, 22.7, 23.5, 22.5 | 22.0, 22.7, 23.0,
24.5 | 28.3, 28.2, 42.0, 28.5 | 29.5, 28.9, 37.6, 39.5 |
| xz, 360 B → 673 B | 20,000 | 10.0, 10.2, 12.4, 10.0 | 9.9, 9.9, 9.8, 10.0
| 16.7, 14.6, 24.2, 18.6 | 15.5, 15.6, 15.9, 22.9 |
| zstd, 331 B → 673 B | 20,000 | 1.0, 1.1, 1.2, 1.0 | 1.1, 1.0, 1.0, 1.1 |
1.2, 1.2, 1.3, 1.2 | 4.5, 1.2, 1.2, 1.2 |
| gzip, 472 KB → 1.27 MB (control) | 2,000 | 2110, 2178, 2167, 2164 | 2096,
2095, 2146, 2158 | | |
| zstd, 196 KB → 1.27 MB | 2,000 | 1021, 1097, 1061, 1069 | 1023, 1024,
1040, 1023 | | |
The two builds overlap in every row, and they vary from run to run as much
as the unchanged gzip control does. There is no difference above that noise.
With 2,000 samples a run, P99.9 would rest on 2 samples, so it is not reported
for the large files.
</details>
Security review of this diff:
- `/security-review`: passed on `f35486a31176` after 1 review by
`security-audit` focused review of `upstream/main...HEAD` (the PR targets
upstream/main, and /security-review reads only origin/HEAD): no Medium or
higher finding is left open
- Review 1 of `f35486a31176` by `security-audit`: no findings
## Are there any user-facing changes?
- A bzip2, xz or zstd CSV or JSON file made of several members returns every
row when it is read from an object store, as it already does from the local
file system.
- CSV schema inference reads through `convert_stream` for every store, the
local file system included (`datafusion/datasource-csv/src/file_format.rs`), so
it now reads past the first member too.
- Bytes after the last member that do not start another member are now an
error from an object store, for bzip2, xz and zstd, as they already were from
the local file system and for gzip from both. Before this change they were
ignored there, and a truncated second member silently returned only the first
member's rows. Data the formats allow after a member still reads: xz stream
padding and zstd skippable frames.
No public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]