rangareddy commented on code in PR #19592:
URL: https://github.com/apache/hudi/pull/19592#discussion_r3948317086
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,28 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| hoodie.datasource.write.row.writer.enable | true | When enabled, clustering
rewrites file groups through the Spark row writer instead of the RDD path. |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
+ from a datasource write takes the row-writer path unless you turn it off.
+* Clustering also applies its own fallback when the config is **absent** from
the write config entirely — which
+ is what a standalone or async clustering job sees. That fallback has not
been stable across releases: it was
+ `false` in 0.14.0, 0.15.0 and 0.15.1, and `true` in 0.14.1 and from 1.0.0
onwards. On this release it is
Review Comment:
Fixed in all six copies, and your reading is confirmed against the tags:
MultipleSparkJobExecutionStrategy reads false at 0.14.0, true at 0.14.1, false
at 0.14.2, false at 0.15.0 and 0.15.1, and true from 1.0.0. 0.14.2 was the one
gap.
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,28 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| hoodie.datasource.write.row.writer.enable | true | When enabled, clustering
rewrites file groups through the Spark row writer instead of the RDD path. |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
+ from a datasource write takes the row-writer path unless you turn it off.
+* Clustering also applies its own fallback when the config is **absent** from
the write config entirely — which
+ is what a standalone or async clustering job sees. That fallback has not
been stable across releases: it was
Review Comment:
Reworded to name the two that actually see an absent key,
`HoodieClusteringJob` (spark-submit or hudi-cli) and Hudi Streamer, and to say
why the other two do not. Confirmed `HoodieCLIUtils:63` applies
`HoodieWriterUtils.parametersWithWriteDefaults` and that nothing in
hudi-utilities does, which is exactly the split you describe.
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,28 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
Review Comment:
Taken, the row now follows the convention of the tables above it with the
key and default in backticks and `Config Param: ENABLE_ROW_WRITER` / `Since
Version: 0.9.0` appended, both read from `DataSourceOptions`.
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,28 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
Review Comment:
Added, in the same edit as the formatting change below: the Description cell
now says the `true` is the config's own default and that the fallback applied
when the config is absent differs by release, so the table no longer
over-promises on its own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
