rangareddy commented on code in PR #19592:
URL: https://github.com/apache/hudi/pull/19592#discussion_r3957888493
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,31 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| `hoodie.datasource.write.row.writer.enable` | `true` | When enabled,
clustering rewrites file groups through the Spark row writer instead of the RDD
path. This is the config's own default; the fallback applied when the config is
absent differs by release, see below.<br /><br />`Config Param:
ENABLE_ROW_WRITER`<br />`Since Version: 0.9.0` |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
Review Comment:
Good catch, added. Confirmed the infer function in `DataSourceOptions`: it
returns `"false"` when the operation is `bulk_insert`,
`!metaFieldsMode.toLegacyPopulateMetaFields`, and `COMBINE_BEFORE_INSERT` is
on. The bullet now names that combination and says it is the first thing to
check when clustering did not use the row writer and nothing in the config says
so.
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,31 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| `hoodie.datasource.write.row.writer.enable` | `true` | When enabled,
clustering rewrites file groups through the Spark row writer instead of the RDD
path. This is the config's own default; the fallback applied when the config is
absent differs by release, see below.<br /><br />`Config Param:
ENABLE_ROW_WRITER`<br />`Since Version: 0.9.0` |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
+ from a datasource write takes the row-writer path unless you turn it off.
+* Clustering also applies its own fallback when the config is **absent** from
the write config entirely. That is
+ what `HoodieClusteringJob` (spark-submit or hudi-cli) and Hudi Streamer see,
since both build their write
+ config from raw properties. In-process async clustering from a Spark
datasource streaming write and
+ `CALL run_clustering` do not: both go through the datasource write defaults,
so they carry the key as `true`.
+ The fallback has not been stable across releases: it was `false` in 0.14.0,
0.14.2, 0.15.0 and 0.15.1, and
+ `true` in 0.14.1 and from 1.0.0 onwards. On this release it is `true`, so
those two paths use the row writer
+ by default as well.
+
+To force the RDD path, set `hoodie.datasource.write.row.writer.enable=false`
in the same properties the
Review Comment:
Added the wider-scope point, with one correction. The config being general
is right, and its own documentation says so, so turning it off for an inline or
async clustering job attached to a datasource write also disables the row
writer for that job's ingestion writes; the standalone `HoodieClusteringJob`
note is in too. But Hudi Streamer is not an example of it:
`StreamSync.isRowWriterEnabled` reads a separate hardcoded
`hoodie.streamer.write.row.writer.enable` defaulting to `false`, with no alias
to the datasource key, so setting the datasource key does not change Streamer
ingestion. Only Streamer's clustering reads this key, and the text now says
that explicitly so the two paragraphs do not read as contradictory.
##########
website/versioned_docs/version-1.0.0/clustering.md:
##########
@@ -156,6 +156,37 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| `hoodie.datasource.write.row.writer.enable` | `true` | When enabled,
clustering rewrites file groups through the Spark row writer instead of the RDD
path. This is the config's own default; the fallback applied when the config is
absent differs by release, see below.<br /><br />`Config Param:
ENABLE_ROW_WRITER`<br />`Since Version: 0.9.0` |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
+ from a datasource write takes the row-writer path unless you turn it off.
+* Clustering also applies its own fallback when the config is **absent** from
the write config entirely. That is
+ what `HoodieClusteringJob` (spark-submit or hudi-cli) and Hudi Streamer see,
since both build their write
+ config from raw properties. In-process async clustering from a Spark
datasource streaming write and
+ `CALL run_clustering` do not: both go through the datasource write defaults,
so they carry the key as `true`.
+ The fallback has not been stable across releases: it was `false` in 0.14.0,
0.14.2, 0.15.0 and 0.15.1, and
+ `true` in 0.14.1 and from 1.0.0 onwards. On this release it is `true`, so
those two paths use the row writer
+ by default as well.
+
+To force the RDD path, set `hoodie.datasource.write.row.writer.enable=false`
in the same properties the
+clustering job reads.
+
+On 1.0.x there is one further condition: even with the config enabled, Hudi
falls back to the RDD path when the
Review Comment:
Added to all three 1.0.x copies: the property defaults to `true`, so the
fallback only applies if it has been explicitly disabled, and it is not
something a table hits by accident. Also dropped an em-dash from that paragraph
while there.
Unrelated but worth flagging since you are in these files: `—props` appears
instead of `--props` in the Hudi Streamer paragraph, in eight clustering.md
copies including three this PR does not touch (0.14.2, 0.15.0, 0.15.1). It is a
copy-paste-breaking typo rather than style. I have left it out to keep this
diff scoped; happy to send it as its own change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
