voonhous commented on code in PR #19687:
URL: https://github.com/apache/hudi/pull/19687#discussion_r3878133250
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -194,11 +196,16 @@ protected BulkInsertPartitioner<JavaRDD<HoodieRecord<T>>>
getRDDPartitioner(Map<
private <I> BulkInsertPartitioner<I> getPartitioner(Map<String, String>
strategyParams,
HoodieSchema schema,
boolean
isRowPartitioner) {
+ // Trim: the config and inline paths pass the list through verbatim (`id,
ts`), while the
+ // partitioners look up the column names as given.
Option<String[]> orderByColumnsOpt =
Option.ofNullable(strategyParams.get(PLAN_STRATEGY_SORT_COLUMNS.key()))
- .map(listStr -> listStr.split(","));
+ .map(listStr ->
Arrays.stream(listStr.split(",")).map(String::trim).toArray(String[]::new));
return orderByColumnsOpt.map(orderByColumns -> {
+ // The custom-columns partitioners re-validate in their constructors;
this earlier check
+ // additionally covers the spatial-curve (ZORDER/HILBERT) partitioners
below.
+ SortUtils.validateSortableColumns(orderByColumns, schema);
Review Comment:
Rejected outright. Both curve builders need an exact top-level name (DIRECT
logs and returns the frame unordered on a miss; SAMPLE dies in a bare
`NoSuchElementException` from `RangeSample`'s column map), so a new
`SpatialCurveSortPartitionerBase.validateOrderByColumns` fails any sort column
that is not exactly a top-level field of the meta-field schema, naming the
column and the strategy. It runs in the ZORDER/HILBERT arm here (before either
partitioner is built) and in `run_clustering` right after the write client
resolves `hoodie.layout.optimize.strategy`, so `order => 's.level',
order_strategy => 'z-order'` is rejected before a plan is scheduled. Same rule
catches a case-mismatched or misspelled name, which the helper dropped the same
way. Pinned in the new `TestSpatialCurveSortPartitionerBase` and a
`TestClusteringProcedure` leg (zero pending plans). Fixed in d1087568a6e9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
