waterWang opened a new pull request, #17685:
URL: https://github.com/apache/iceberg/pull/17685
### Problem
When using Dynamic Sink with DistributionMode.HASH and records alternating
between two writer schema variants (different field IDs), HashKeyGenerator's
SelectorKey cache key uses only the table schema ID and nulls the record's
schema. This causes the cache to return the same PartitionKeySelector for both
variants, but the selector reads the wrong field position from the row data,
producing ClassCastException.
### Root Cause
Two issues in `HashKeyGenerator.java`:
1. **SelectorKey cache key**: When `tableSchemaId` is provided, the record's
actual `schema` field is nulled out (`this.schema = tableSchemaId == null ?
schema : null`). Records with different dynamic schemas but the same table
schema ID share the same cache key, returning a stale PartitionKeySelector.
2. **Key selector construction**: `getKeySelector()` is called with
`effectiveSchema` (the table schema) instead of `dynamicRecord.schema()`. The
row data is organized per the record's schema, so the key selector must be
built with the record's schema.
### Fix
- Always include the record's schema and spec in the SelectorKey (remove the
nulling condition)
- Use `dynamicRecord.schema()` and `dynamicRecord.spec()` when building the
PartitionKeySelector
### Stack trace
```
ClassCastException: Cannot cast java.lang.String to java.lang.Integer
at StructTransform.get(StructTransform.java:89)
at PartitionKey.get(PartitionKey.java:30)
at PartitionSpec.partitionToPath(PartitionSpec.java:225)
at PartitionKeySelector.getKey(PartitionKeySelector.java:64)
at HashKeyGenerator.generateKey(HashKeyGenerator.java:104)
```
Fixes #17651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
