GGraziadei opened a new issue, #17658: URL: https://github.com/apache/iceberg/issues/17658
### Apache Iceberg version main (branch) ### Query engine Spark ### Please describe the bug 🐞 Two independent defects in the Spark Z-order clustering path, both present in `spark/v3.5`, `spark/v4.0` and `spark/v4.1`. They were spotted by @manuzhang while reviewing #16827, which adds a Hilbert strategy that reuses this same code (https://github.com/apache/iceberg/pull/16827#discussion_r3780935134 and https://github.com/apache/iceberg/pull/16827#discussion_r3780990670). They are filed separately from that PR because fixing them changes existing Z-order behaviour, which deserves its own review rather than riding along with a new feature. Once fixed here, both strategies benefit, since the Hilbert runner reuses `SparkZOrderUDF.sortedLexicographically` for the per-column byte conversion. --- #### 1. `NullPointerException` when a clustered column contains nulls `SparkZOrderUDF` registers Java UDFs whose parameters are boxed, but the values are then unboxed with no null check. Spark passes nulls straight through to Java UDFs, so any null in a clustered column aborts the rewrite. The most direct case is `booleanToOrderedBytesUDF()`: ```java (Boolean value) -> { ByteBuffer buffer = inputBuffer(position, ZOrderByteUtils.PRIMITIVE_BUFFER_SIZE); buffer.put(0, (byte) (value ? -127 : 0)); // unboxes a possibly-null Boolean return buffer.array(); } ``` The numeric converters have the same problem by a different route — the lambdas take boxed types while the `ZOrderByteUtils` methods take primitives, so the unboxing happens at the call site: ```java (Byte value) -> ZOrderByteUtils.tinyintToOrderedBytes(value, ...) // byte val (Integer value) -> ZOrderByteUtils.intToOrderedBytes(value, ...) // int val (Double value) -> ZOrderByteUtils.doubleToOrderedBytes(value, ...) // double val ``` `stringToOrderedBytes` and `byteTruncateOrFill` do handle null explicitly, so the behaviour is currently inconsistent across types: nullable string and binary columns work, nullable primitives throw. ##### To reproduce ```sql CREATE TABLE db.tbl (id INT, flag BOOLEAN) USING iceberg; INSERT INTO db.tbl VALUES (1, true), (2, null); CALL system.rewrite_data_files( table => 'db.tbl', strategy => 'sort', sort_order => 'zorder(id, flag)' ); ``` ##### Fix to decide The mechanical part is a null guard in each converter. The part that needs a decision is what a null should sort as — lowest, highest, or a reserved sentinel — and whether that should match the null-ordering semantics Iceberg sort orders already express (`NULLS FIRST` / `NULLS LAST`). Sorting nulls low (all-zero bytes) is the smallest change and matches what `stringToOrderedBytes` already does for a null string. There is currently no test covering null values on the Z-order path, so whatever is chosen should come with one. --- #### 2. Case-insensitive validation keeps the caller's spelling `SparkZOrderFileRewriteRunner.validZOrderColNames` resolves the column case-insensitively when `spark.sql.caseSensitive` is `false`, but then stores the string the caller supplied rather than the resolved field name: ```java Types.NestedField field = caseSensitive ? schema.findField(colName) : schema.caseInsensitiveFindField(colName); Preconditions.checkArgument(field != null, "Cannot find column '%s' ..."); ... validZOrderColNames.add(colName); // caller's spelling, not field.name() ``` Validation therefore passes, but the mis-cased name is what later reaches `df.schema().apply(...)`, which can fail with `FIELD_NOT_FOUND` further down instead of being caught by the check that was supposed to catch it. The fix is to add `field.name()` instead of `colName`, plus a test with a mis-cased column name and `spark.sql.caseSensitive=false`. ### Willingness to contribute - [x] I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
