voonhous commented on code in PR #19592:
URL: https://github.com/apache/hudi/pull/19592#discussion_r3920793663
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,28 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| hoodie.datasource.write.row.writer.enable | true | When enabled, clustering
rewrites file groups through the Spark row writer instead of the RDD path. |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
+ from a datasource write takes the row-writer path unless you turn it off.
+* Clustering also applies its own fallback when the config is **absent** from
the write config entirely — which
+ is what a standalone or async clustering job sees. That fallback has not
been stable across releases: it was
Review Comment:
**minor:** "standalone or async clustering job" is not quite the set that
sees an absent key. In-process async clustering from a Spark datasource
streaming write (`HoodieStreamingSink` passes the datasource write client to
`SparkStreamingAsyncClusteringService`) and `CALL run_clustering`
(`HoodieCLIUtils.createHoodieWriteClient` applies
`parametersWithWriteDefaults`) both carry the key as `true`; only
`HoodieClusteringJob` (spark-submit / hudi-cli) and Hudi Streamer (`StreamSync`
builds from raw props) hit the fallback. Not blocking on 1.x where both values
agree, but could we name those two here instead?
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,28 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| hoodie.datasource.write.row.writer.enable | true | When enabled, clustering
rewrites file groups through the Spark row writer instead of the RDD path. |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
+ from a datasource write takes the row-writer path unless you turn it off.
+* Clustering also applies its own fallback when the config is **absent** from
the write config entirely — which
+ is what a standalone or async clustering job sees. That fallback has not
been stable across releases: it was
+ `false` in 0.14.0, 0.15.0 and 0.15.1, and `true` in 0.14.1 and from 1.0.0
onwards. On this release it is
Review Comment:
**minor:** This release list reads as exhaustive but skips 0.14.2, where the
fallback is `false` again: `release-0.14.2`
`MultipleSparkJobExecutionStrategy.java:118` reads
`getBooleanOrDefault("hoodie.datasource.write.row.writer.enable", false)`
(flipped by a1ac3078495c, #18132). Could we add it here and in the five
versioned copies?
```suggestion
`false` in 0.14.0, 0.14.2, 0.15.0 and 0.15.1, and `true` in 0.14.1 and
from 1.0.0 onwards. On this release it is
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
