Rodrigo-Palma opened a new issue, #1760:
URL: https://github.com/apache/datafusion-python/issues/1760
**Describe the bug**
`DataFrame.write_parquet` accepts `write_options: DataFrameWriteOptions |
None`, documents it ("Options that impact how the DataFrame is written") and
declares it in the `@overload` for the `ParquetWriterOptions` form. But that
branch delegates without forwarding it:
```python
if isinstance(compression, ParquetWriterOptions):
if compression_level is not None:
msg = "compression_level should be None when using
ParquetWriterOptions"
raise ValueError(msg)
self.write_parquet_with_options(path, compression) # write_options
dropped
return
```
`write_parquet_with_options(path, options, write_options=None)` takes the
parameter, so everything in `DataFrameWriteOptions` (`partition_by`,
`single_file_output`, `insert_operation`, `sort_by`) is silently ignored
whenever the compression argument is a `ParquetWriterOptions`. No error, no
warning: the files just land in the wrong layout.
**To Reproduce**
```python
import tempfile, pathlib
from datafusion import SessionContext
from datafusion.dataframe import ParquetWriterOptions, DataFrameWriteOptions
ctx = SessionContext()
df = ctx.from_pydict({"part": ["a", "a", "b"], "v": [1, 2, 3]})
wo = lambda: DataFrameWriteOptions(partition_by="part")
with tempfile.TemporaryDirectory() as d:
out = pathlib.Path(d) / "x"
df.write_parquet(out, ParquetWriterOptions(), write_options=wo())
print(sorted(p.name for p in out.iterdir()))
```
Measured with `datafusion 54.0.0`:
```
write_parquet(ParquetWriterOptions, write_options)
['IDuOjvMa3pdEDotb_0.parquet'] <-- not partitioned
write_parquet_with_options(..., write_options) ['part=a', 'part=b']
write_parquet('zstd', write_options) ['part=a', 'part=b']
```
Same `DataFrameWriteOptions` in all three calls; only the
`ParquetWriterOptions` branch loses the Hive partitioning.
**Expected behavior**
The Hive partitioning, and the rest of `write_options`, should be applied,
exactly as the other two spellings already do: `['part=a', 'part=b']`.
**Additional context**
The branch was added in ef62fa89 (#1169, "Add compression_level support to
ParquetWriterOptions and enhance write_parquet to accept full options object"),
while `write_options` came earlier in #857, so the new delegation path was
written without carrying over the existing parameter. Note that the same `if`
explicitly *refuses* `compression_level` with a `ValueError`, so incompatible
arguments in this branch are rejected on purpose when intended; `write_options`
was not refused, just not passed along.
I have a one-line fix plus a regression test ready and will open a PR
against this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
