NoahKusaba opened a new issue, #22: URL: https://github.com/apache/datafusion-iceberg/issues/22
Reported by @mbutrovich in #19. ### Unpartitioned tables: NULLs are written into required columns The unpartitioned path skips `project_with_partition` ([`table/mod.rs`](https://github.com/apache/datafusion-iceberg/blob/b12e9871dd2b2632cd05786d9f12dd6cde47dc2e/crates/datafusion/src/table/mod.rs#L185-L189)), and `IcebergWriteExec` passes the input's own schema as the sink schema to `execute_input_stream` ([`write.rs`](https://github.com/apache/datafusion-iceberg/blob/b12e9871dd2b2632cd05786d9f12dd6cde47dc2e/crates/datafusion/src/physical_plan/write.rs#L277-L280)). DataFusion only runs [`check_not_null_constraints`](https://github.com/apache/datafusion/blob/d5552342012888b7d1a3ab88d92e3d292fc0cde0/datafusion/physical-plan/src/execution_plan.rs#L1882-L1913) for columns that are non-nullable in the sink schema and nullable in the input. Because both schemas are the input's here, the check never runs. Repro: inserting `SELECT * FROM source` into an unpartitioned table with a required `id: int` column, from a `MemTable` whose nullable `id` holds `[1, NULL]`, succeeds and reads back `0, 1`. The NULL is written as `0`. ### Partitioned tables: nullable sources are rejected at plan time The partitioned path goes the other way. `project_with_partition` rejects any source column that is nullable where the table column is required, even when the source holds no nulls. DataFusion's own sinks accept that case and check at runtime. ### Possible fix Passing the table's Arrow schema (plus the partition column) as the sink schema to `execute_input_stream` would likely fix both: nulls would be caught at runtime on either path, and the plan-time rejection of nullable sources could be relaxed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
