Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
alamb commented on issue #7489:
URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2992673044
`label_issue.py` automatically added labels {'parquet'} from #7555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
alamb closed issue #7489: Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` URL: https://github.com/apache/arrow-rs/issues/7489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
alamb commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2939499927 > > Yes I agree -- here is a PR that proposes defaulting it to 64 (to match page stats). Any thoughts? > > I don't have an opinion on what should be the default 😄 Whatever other tools do I guess 😅 > > I think this issue could be closed now, right? :) The only remaining issue is purely on the `pyarrow` side, but also not super relevant, since files with so huge, untruncated statistics header are overblown massively anyways. Sounds good -- let's call this closed by - https://github.com/apache/arrow-rs/pull/7555 Thanks again for all the help reporting and testing this @jonded94 -- I think everyone will benefit from the result -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
jonded94 commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2939166002 > Yes I agree -- here is a PR that proposes defaulting it to 64 (to match page stats). Any thoughts? I don't have an opinion on what should be the default 😄 Whatever other tools do I guess 😅 I think this issue could be closed now, right? :) The only remaining issue is purely on the `pyarrow` side, but also not super relevant, since files with so huge, untruncated statistics header are overblown massively anyways. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
alamb commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2936283780 > @etseidl @alamb it seems like the issue is fixed now! :) Now that https://github.com/apache/arrow-rs/pull/7555 is merged, I checked out arrow and parquet with the recent git version: @etseidl does great work :) > The file sizes seem like there really should be a statistics_truncate_length default somewhere below or equal to 1024 or so. Yes I agree -- here is a PR that proposes defaulting it to 64 (to match page stats). Any thoughts? - https://github.com/apache/arrow-rs/issues/7490 - https://github.com/apache/arrow-rs/pull/7578 > @alamb a slightly offtopic question: Comparing EnabledStatistics CHUNK and PAGE (with statistics_truncate_length == None) it seems that in PAGE mode compared to CHUNK mode, the file is 3 times the size? Yes, it is currently 3 times the size. As @etseidl mentions, we have a plan to fix this: - https://github.com/apache/arrow-rs/issues/7580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
etseidl commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2936268367 Setting `PAGE` also enables writing of the [page index](https://github.com/apache/parquet-format/blob/master/PageIndex.md), so there's the 3X. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
jonded94 commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2936239923 @alamb a slightly offtopic question: Comparing `EnabledStatistics` `CHUNK` and `PAGE` (with `statistics_truncate_length` == `None`) it seems that in `PAGE` mode compared to `CHUNK` mode, the file is 3 times the size? `101450724 / 34341803 = 2.954...` As this is a 2 row parquet file, we surely have only a single data page and a single row group. I would have expected that it's writing statistics once per row group (from `CHUNK` mode), and once per data page (from `PAGE` mode). Thereby, I would have expected double the file size, not tripple? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
jonded94 commented on issue #7489:
URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2936182415
@etseidl @alamb it seems like the issue is fixed now! :) Now that #7555 is
merged, I checked out `arrow` and `parquet` with the recent git version:
```
arrow = { git = "https://github.com/apache/arrow-rs.git";, rev =
"0ae9f66d10141c8d5054fd77f73168c7a2ea2819", features = ["pyarrow"] }
parquet = { git = "https://github.com/apache/arrow-rs.git";, rev =
"0ae9f66d10141c8d5054fd77f73168c7a2ea2819", features = ["async"] }
```
In the previously linked `pytest` test, I added cases where I truncate
statistics to 7Mi (slightly below the threshold after which `pyarrow` crashes
on reads, at least with `EnabledStatistics` == `PAGE`) and also 8Mi (leading to
`pyarrow` crashes).
Not only are file sizes now actually changing, but the `pyarrow` crashes are
exactly as expected (i.e. only for page level statistics longer than 8MiB per
value)! 😁
Here is an overview over file sizes & `pyarrow` crashes for generated test
files:
| EnabledStatistics | statistics_truncate_length | size [bytes] | fails
with `pyarrow` |
| --- | --- | --- | --- |
| None | None |787344 | No |
| Chunk | 1024 |789415 | No |
| Chunk | 7Mi |15467435 | No |
| Chunk | 8Mi |17564587 | No |
| Chunk | None |34341803 | No |
| Page | 1024 |793552 | No |
| Page | 7Mi |44827620 | No |
| Page | 8Mi |51119076 | Yes |
| Page | None |101450724 | Yes |
The file sizes seem like there really should be a
`statistics_truncate_length` default somewhere below or equal to 1024 or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
alamb commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2925664872 BTW @etseidl has found a bug in arrow-rs (the truncate setting is ignored), and has a fix up. I filed a ticket to track the actual bug here: - https://github.com/apache/arrow-rs/issues/7579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
etseidl commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2914935120 @jonded94 could you try your test with the branch in #7555? I'm hoping that will at least allow the workaround to work. Then we can see about truncating statistics by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
etseidl commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2914412964 I think I found it...you say pyarrow is failing because the page headers are too large. So this implicates the page stats rather than chunk or column index stats. I'm looking at the page writing code and it seems the page stats do not perform truncation (but I may be missing something). https://github.com/apache/arrow-rs/blob/72c72941e2d6caf9349399882f0a175ffdbc3b5d/parquet/src/column/writer/mod.rs#L963-L980 I'll try writing some tests for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
jonded94 commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2906534218 > I am sorry but I don't seem to be able to download the file myself. When I try I get Yeah, this fileupload service I chose only keeps them available for download for a week. Do you know another upload service with longer availability times? > maybe the statistics are still large if there are many many data pages (e.g. you are writing row groups with many columns and rows) The code snippet I shared (which you in principle could also compile and use, at least the Rust part) actually writes a single column, two row parquet file. I think it is as minimal as it gets 😄 I'm unavailable now for at least a week, but I could have a look with this parquet viewer you shared then. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
alamb commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2905428698 One thought I had was that maybe the statistics are still large if there are many many data pages (e.g. you are writing row groups with many columns and rows) so even if the statistics are truncated the size is still exceeded? I think dumping the metadata and looking at what got written is the right next step -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
alamb commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2905423680 One think maybe you could do is use the viewer from @XiangpengHao at https://parquet-viewer.xiangpeng.systems/ To see if the changes to the settings you made resulted in the desired changes in the parquet file I am sorry but I don't seem to be able to download the file myself. When I try I get https://github.com/user-attachments/assets/127d31a0-bc2b-4971-aa36-319eeaf754ef"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
jonded94 commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2903922051 @alamb do you have an idea how I could further debug this problem? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
jonded94 commented on issue #7489:
URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2877156966
@alamb I tried setting `statistics_truncate_length` as well as
`column_index_truncate_length`, but for some reason, this didn't enable
`pyarrow` to read the new file.
I used this Rust code to initialize a Parquet Writer:
```Rust
#[pyclass]
pub struct ParquetFileWriter {
writer: Mutex>>,
}
impl ParquetFileWriter {
#[allow(clippy::too_many_arguments)]
fn try_new(
file: FileWriter,
schema: Schema,
target_rows_per_row_group: NonZeroUsize,
column_compression: Option>,
compression: Option,
statistics_enabled: Option,
statistics_truncate_length: Option,
column_index_truncate_length: Option,
) -> Result {
let props = {
let mut builder = WriterProperties::builder()
.set_max_row_group_size(target_rows_per_row_group.get())
.set_key_value_metadata(Some(convert_arrow_metadata_to_parquet_metadata(
schema.metadata.clone(),
)))
.set_statistics_truncate_length(statistics_truncate_length.map(|value|
value.get()))
.set_column_index_truncate_length(
column_index_truncate_length.map(|value| value.get()),
);
dbg!(statistics_truncate_length);
dbg!(column_index_truncate_length);
if let Some(compression) = compression {
builder = builder.set_compression(compression);
}
if let Some(column_compression) = column_compression {
for (column_path, compression) in
column_compression.into_iter() {
builder = builder.set_column_compression(column_path,
compression);
}
}
if let Some(statistics_enabled) = statistics_enabled {
builder = builder.set_statistics_enabled(statistics_enabled);
}
builder.build()
};
Ok(Self {
writer: Mutex::new(Some(ArrowWriter::try_new_with_options(
file,
SchemaRef::new(schema),
ArrowWriterOptions::new().with_properties(props),
)?)),
})
}
fn write_batch(&self, batch: RecordBatch) -> Result<(),
DiscoParquetError> {
if let Some(file) = self.writer.lock()?.as_mut() {
file.write(&batch)?;
Ok(())
} else {
Err(PyValueError::new_err("File is already closed.").into())
}
}
```
And I used this pytest code to verify which test cases are readable or are
not readable with `pyarrow`:
```Python
@pytest.mark.parametrize(
"statistics_level,statistics_truncate_length,expected_fail",
[
(EnabledStatistics.NONE, 1, False),
(EnabledStatistics.CHUNK, 1024, False),
(EnabledStatistics.CHUNK, 16 * 1024 * 1024, False),
(EnabledStatistics.CHUNK, None, False),
(EnabledStatistics.PAGE, 1024, False),
(EnabledStatistics.PAGE, 16 * 1024 * 1024, True),
(EnabledStatistics.PAGE, None, True),
],
)
def test_page_statistics_pyarrow_compatibility(
statistics_level: EnabledStatistics, statistics_truncate_length: int |
None, expected_fail: bool, tmp_path: Path
) -> None:
# 16MiB, to get statistic headers definitely above 16MiB which pyarrow
doesn't support right now (version 20.0.0)
length = 16 * 1024 * 1024
schema = pyarrow.schema([pyarrow.field("data", pyarrow.binary())])
data = [{"data": b"0" * length} for _ in range(2)]
b = pyarrow.RecordBatch.from_pylist(data, schema=schema)
path = tmp_path / "test.parquet"
with ParquetFileWriter(
path,
schema,
statistics_enabled=statistics_level,
statistics_truncate_length=statistics_truncate_length,
column_index_truncate_length=statistics_truncate_length,
) as writer:
writer.write_batch(b)
reader = pyarrow.parquet.ParquetFile(path)
with pytest.raises(OSError) if expected_fail else
contextlib.nullcontext():
reader.read_row_group(0)
```
### Expectation
- With `EnabledStatistics` == `None`, it should never fail since no
statistics at all are written
- With `EnabledStatistics` == `Chunk`, it should not matter to which length
the statistics are truncated since `pyarrow` appears to be able to read
ColumnChunk/RowGroup level statistics that are arbitrarily large?
- With `EnabledStatistics` == `Page`, it should fail whenever the statistics
are not truncated at all or only to large value (16MiB for exa
Re: [I] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow-rs]
alamb commented on issue #7489: URL: https://github.com/apache/arrow-rs/issues/7489#issuecomment-2873250301 Here is a related ticket in arrow which I think describes the same symptoms: - https://github.com/apache/arrow/issues/46404 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
