SEPURI-SAI-KRISHNA commented on issue #19667:
URL: https://github.com/apache/hudi/issues/19667#issuecomment-5374094065
I'm working on this one, but it is blocked behind #19648, and while
measuring it I found the fix I
suggested above is not sufficient. Details below so the issue stays accurate.
**It cannot currently be reproduced on master.** The row-writer half of the
comparison throws
before it ever produces a partition path:
```
java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String
cannot be cast to class java.lang.String
at
org.apache.hudi.keygen.PartitionPathFormatterBase.combine(PartitionPathFormatterBase.java:66)
at
org.apache.hudi.keygen.BuiltinKeyGenerator.combinePartitionPathUnsafe(BuiltinKeyGenerator.java:143)
at
org.apache.spark.sql.hudi.command.SqlKeyGenerator.getPartitionPath(SqlKeyGenerator.scala:122)
```
That is #19647, fixed by #19648. So the divergence is only observable, and a
fix only verifiable,
once that lands.
**What I did measure.** Same table shape, one `TIMESTAMP` partition column,
value
`2026-01-05 18:00:00`:
| table | write path | `_hoodie_partition_path` |
|---|---|---|
| no slash config | Avro | `2026-01-05 18%3A00%3A00` |
| no slash config | row writer | `2026-01-05 18%3A00%3A00` |
| slash config | Avro | `1767636000000000` |
| slash config | row writer | throws, see above |
Without the config the two paths already agree, because
`convertPartitionPathToSqlType` reaches the
`TimestampType` arm and normalizes both. With the config the Avro path
returns raw micros, because
the `slashSeparatedDatePartitioning` arm sits ahead of `TimestampType` and
shadows it.
**Why moving the arm into `case _` is not enough.** Two further details,
both of which follow from
`KeyGenUtils.DEFAULT_PARTITION_PATH_SEPARATOR` being `"/"`:
1. The arm's body, `partitionValue.replace('/', '-')`, cannot do anything.
`partitionValue` is a
fragment of `partitionPath.split("/")`, so it never contains a `/`. The
arm is a no-op that only
suppresses the `TimestampType` branch.
2. The conversion is bypassed entirely for a genuinely slash-separated path.
The guard
`partitionFragments.size != partitionSchema.get.size` returns
`partitionPath` untouched, and a
slash-separated value has more fragments than the table has partition
columns. That is why the
Avro path above still reached the arm at all -- the raw micros value
contains no dash, so it
stayed a single fragment.
So after simply reordering the match, the Avro path would normalize to
`2026-01-05 18%3A00%3A00` while the row-writer path would early-return
`2026/01/05 18:00:00.0`. Still two different directories, just different
ones.
**The two real options**, replacing the suggested fix in the description:
1. **Make `convertPartitionPathToSqlType` slash-aware.** For a single
partition column, rejoin the
fragments before matching them against the schema, run the existing
conversion, then re-apply
the substitution. Both paths then land on `2026/01/05 18%3A00%3A00`. It
only works
unambiguously for a single partition column, so it is coupled to #19666.
2. **Reject a `TIMESTAMP` partition column when slash-separated date
partitioning is enabled.** The
config is documented for `yyyy-MM-dd` date values, and a timestamp
carrying a time of day
produces a directory like `2026/01/05 18%3A00%3A00`, which is consistent
but not obviously
what anyone wants.
I lean towards 2, for the same reason I gave on #19669: the combination is
outside what the feature
documents, and silently accepting it is the actual bug. Happy to implement 1
instead if maintainers
would rather keep it working. Either way I will pick this up once #19648
merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
