This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git


The following commit(s) were added to refs/heads/master by this push:
     new f740680aa408 perf(clustering): share group-independent state across 
clustering groups (#20533)
f740680aa408 is described below

commit f740680aa40862bf008f0f9c6ada11cb8d35e37c
Author: Harsha Gudladona <[email protected]>
AuthorDate: Wed Nov 18 01:44:17 2026 +0000

    perf(clustering): share group-independent state across clustering groups 
(#20533)
    
    * test(clustering): show per-group payloads in the clustering write-stage 
task binary
    
    MultipleSparkJobExecutionStrategy builds one RDD per clustering group and
    unions them into one write stage. Spark serializes the whole stage lineage
    into a single task binary that every task deserializes, so anything a
    group's lineage captures is shipped to every task, for every group.
    
    This test runs the real SparkSortAndSizeExecutionStrategy on a 2-group and
    an 8-group subset of one plan, serializes each write-stage RDD the way the
    closure serializer does, and counts distinct instances of payloads that do
    not depend on the group. They must not grow with the number of groups.
    
    On master it fails; per extra group the task binary gains:
      HoodieSchema +1, HoodieWriteConfig +1, TypedProperties +14,
      ReaderContextFactory +1, StructType +3, SQLConfInjectingRDD sqlConfigs +1,
      ~73 KB.
    
    * perf(clustering): share group-independent state across clustering groups
    
    Clustering builds one RDD per group and unions them into a single write
    stage, so Spark ships every group's lineage to every task of the stage.
    Java serialization writes an instance once per stream however many closures
    reference it, but writes every distinct instance in full: whatever was built
    per group was deserialized once per group by every task, and held by every
    running task. Build what does not depend on the group once per run instead:
    
    - table schema with meta fields, resolved once in performClustering;
    - ReaderContextFactory (each one also creates three broadcasts) and the
      reader properties;
    - one write config per run (getGroupWriteConfig), re-derived if the
      strategy's write config is replaced. withBulkInsertParallelism is dropped:
      it was overwritten by withProps, and the clustering write paths pass the
      number of output groups explicitly and never read the setting;
    - the bulk-insert writer schema, through a new HoodieDatasetBulkInsertHelper
      .bulkInsert overload that takes it (getGroupWriterSchema);
    - SQLConfInjectingRDD shares one configs map across RDDs that capture
      equal SQL configs.
    
    TestClusteringTaskBinaryPayloads now passes: on a 2- and 8-group subset of
    one plan, every payload count is flat, and the task binary grows by
    ~10.5 KB per extra group instead of ~73 KB.
    
    * test(clustering): count deserialized SQL config maps instead of reading a 
private field
    
    Both tests read SQLConfInjectingRDD's private sqlConfigs field by 
reflection.
    Count what a task actually deserializes instead: distinct maps equal to the
    captured SQL configs, seen by ObjectInputStream.resolveObject. The read side
    is needed because a Scala map is written through a serialization proxy
    (writeReplace), so replaceObject never sees the map itself.
    TestSQLConfInjectingRDD calls sharedConfigs directly where it checks 
sharing.
    
    * refactor(clustering): hold the state groups share in one per-run holder
    
    The values every clustering group of a run shares (reader schema, reader
    context factory, reader properties, and the write-config and writer-schema
    caches) now live in one ClusteringRunInvariants holder, built by
    performClustering before any group is submitted and dropped when all groups
    have been submitted. This makes the lifetime explicit:
    
    - performClustering builds it after a subclass has settled the write config
      (SparkBinaryCopyClusteringExecutionStrategy replaces it first), so the
      write-config identity check is no longer needed.
    - runClusteringForGroupAsync and runClusteringForGroupAsyncAsRow keep their
      original signatures and read the shared values from the holder.
    - The caches are concurrent maps (groups run on the clustering thread pool)
      and only memoize; the writer-schema cache stays keyed by dataset schema 
and
      table schema. Called outside a run, getGroupWriteConfig and
      getGroupWriterSchema build fresh values.
    
    No behavior change: every group still gets the same shared instances.
    
    * refactor(clustering): fall back to per-group invariants outside a run; 
cover both write paths
    
    runClusteringForGroupAsync and runClusteringForGroupAsyncAsRow are 
protected:
    a custom strategy may override performClustering and call them itself. Such 
a
    caller worked before the per-run holder and failed with 
IllegalStateException
    after it. With no run in progress, each hook now builds the invariants of 
its
    own write path for that one group: the per-group cost this class otherwise
    avoids, but the behaviour such callers had before.
    
    - performClustering's finally comment now covers the failure path: join()
      throws first, so a group may still be running when the holder is cleared;
      it keeps the reader state it captured and builds its own write config and
      writer schema from there.
    - SQLConfInjectingRDD documents why the last config map is held strongly: a
      weak reference could be collected between two groups' captures, and the
      next group would ship a copy of its own. It holds nothing an RDD does not
      already carry to its tasks.
    - TestClusteringTaskBinaryPayloads runs both 
SparkSortAndSizeExecutionStrategy
      and SparkSingleFileSortExecutionStrategy, each with the row writer on and
      off, covering the records path's invariants and both parquetMaxFileSize
      keys of the write-config cache. On the records path no group builds a
      DataFrame, so no SQL config map is shipped at all.
    - A new case submits a group through each hook outside performClustering and
      checks it writes every record of its input files; against the previous
      checkState both cases fail with IllegalStateException.
---
 .../MultipleSparkJobExecutionStrategy.java         | 184 ++++++++---
 ...onsistentBucketClusteringExecutionStrategy.java |  12 +-
 .../SparkSingleFileSortExecutionStrategy.java      |  16 +-
 .../SparkSortAndSizeExecutionStrategy.java         |  17 +-
 .../hudi/HoodieDatasetBulkInsertHelper.scala       |  25 +-
 .../spark/sql/execution/SQLConfInjectingRDD.scala  |  37 ++-
 .../sql/execution/TestSQLConfInjectingRDD.scala    | 159 ++++++++++
 .../strategy/TestClusteringTaskBinaryPayloads.java | 350 +++++++++++++++++++++
 8 files changed, 731 insertions(+), 69 deletions(-)

diff --git 
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java
 
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java
index 773327527598..c2070e6214cc 100644
--- 
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java
+++ 
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java
@@ -18,6 +18,7 @@
 
 package org.apache.hudi.client.clustering.run.strategy;
 
+import org.apache.hudi.HoodieDatasetBulkInsertHelper;
 import org.apache.hudi.HoodieSchemaConversionUtils;
 import org.apache.hudi.SparkAdapterSupport$;
 import org.apache.hudi.avro.model.HoodieClusteringGroup;
@@ -25,6 +26,7 @@ import org.apache.hudi.avro.model.HoodieClusteringPlan;
 import org.apache.hudi.client.SparkTaskContextSupplier;
 import org.apache.hudi.client.WriteStatus;
 import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieStorageConfig;
 import org.apache.hudi.common.config.TypedProperties;
 import org.apache.hudi.common.data.HoodieData;
 import org.apache.hudi.common.engine.HoodieEngineContext;
@@ -47,6 +49,7 @@ import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.SortUtils;
 import org.apache.hudi.common.util.collection.ClosableIterator;
 import org.apache.hudi.common.util.collection.LazyConcatenatingIterator;
+import org.apache.hudi.common.util.collection.Pair;
 import org.apache.hudi.config.HoodieClusteringConfig;
 import org.apache.hudi.config.HoodieWriteConfig;
 import org.apache.hudi.data.CloseableIteratorListener;
@@ -81,6 +84,8 @@ import java.util.Iterator;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.function.Supplier;
@@ -97,10 +102,56 @@ import static 
org.apache.hudi.config.HoodieClusteringConfig.PLAN_STRATEGY_SORT_C
 public abstract class MultipleSparkJobExecutionStrategy<T>
     extends ClusteringExecutionStrategy<T, HoodieData<HoodieRecord<T>>, 
HoodieData<HoodieKey>, HoodieData<WriteStatus>> {
 
+  /*
+   * Clustering builds one RDD per group and unions them into a single write 
stage, so Spark ships the lineage of
+   * every group to every task of that stage. Java serialization writes an 
instance once per stream however many
+   * closures reference it, but writes each distinct instance in full: an 
object built per group is deserialized
+   * once per group by every task, and held by every running task. Whatever 
does not depend on the group (table
+   * schema, reader context factory, reader properties, write config, writer 
schema) is therefore built once per run
+   * and shared by all groups: see ClusteringRunInvariants.
+   */
+  private transient volatile ClusteringRunInvariants runInvariants;
+
   public MultipleSparkJobExecutionStrategy(HoodieTable table, 
HoodieEngineContext engineContext, HoodieWriteConfig writeConfig) {
     super(table, engineContext, writeConfig);
   }
 
+  /**
+   * The write config for a group's bulk insert: this strategy's write config, 
with the max Parquet file size
+   * overridden when one is given. Every group of a run gets the same instance 
(see above); a
+   * {@link HoodieWriteConfig} holds a full copy of the write properties for 
itself and for each of its sub-configs.
+   * The returned config must not be mutated. Called outside {@link 
#performClustering}, it builds a new config.
+   *
+   * <p>Bulk-insert parallelism is deliberately not set: the clustering write 
paths pass the number of output groups
+   * to the partitioner and to the bulk-insert helper explicitly, and nothing 
on them reads
+   * {@link HoodieWriteConfig#BULKINSERT_PARALLELISM_VALUE}.
+   */
+  protected HoodieWriteConfig getGroupWriteConfig(Option<Long> 
parquetMaxFileSize) {
+    ClusteringRunInvariants run = runInvariants;
+    return run != null
+        ? run.groupWriteConfigs.computeIfAbsent(parquetMaxFileSize, 
this::newGroupWriteConfig)
+        : newGroupWriteConfig(parquetMaxFileSize);
+  }
+
+  private HoodieWriteConfig newGroupWriteConfig(Option<Long> 
parquetMaxFileSize) {
+    HoodieWriteConfig config = 
HoodieWriteConfig.newBuilder().withProps(getWriteConfig().getProps()).build();
+    parquetMaxFileSize.ifPresent(bytes -> 
config.setValue(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE, 
String.valueOf(bytes)));
+    return config;
+  }
+
+  /**
+   * The schema a group's bulk-insert writers use, as {@link 
HoodieDatasetBulkInsertHelper#writerSchema} derives it.
+   * Every group with an equal dataset schema and the same table schema gets 
the same instance (see above);
+   * {@link StructType} is immutable. Called outside {@link 
#performClustering}, it derives a new schema.
+   */
+  protected StructType getGroupWriterSchema(StructType datasetSchema, 
HoodieWriteConfig config) {
+    ClusteringRunInvariants run = runInvariants;
+    return run != null
+        ? run.groupWriterSchemas.computeIfAbsent(Pair.of(datasetSchema, 
config.getSchema()),
+            key -> HoodieDatasetBulkInsertHelper.writerSchema(datasetSchema, 
config))
+        : HoodieDatasetBulkInsertHelper.writerSchema(datasetSchema, config);
+  }
+
   @Override
   public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(final 
HoodieClusteringPlan clusteringPlan, final HoodieSchema schema, final String 
instantTime) {
     JavaSparkContext engineContext = 
HoodieSparkEngineContext.getSparkContext(getEngineContext());
@@ -110,23 +161,18 @@ public abstract class MultipleSparkJobExecutionStrategy<T>
         new CustomizedThreadFactory("clustering-job-group", true));
     try {
       boolean canUseRowWriter = 
getWriteConfig().getBooleanOrDefault("hoodie.datasource.write.row.writer.enable",
 true);
+      Map<String, String> strategyParams = 
clusteringPlan.getStrategy().getStrategyParams();
+      // Before any group is submitted, and after a subclass has settled the 
write config
+      // (SparkBinaryCopyClusteringExecutionStrategy replaces it before 
calling this method).
+      runInvariants = canUseRowWriter ? newRowWriterRunInvariants() : 
newRecordsRunInvariants();
       // execute clustering for each group async and collect WriteStatus
       Stream<HoodieData<WriteStatus>> writeStatusesStream = FutureUtils.allOf(
               clusteringPlan.getInputGroups().stream()
-                  .map(inputGroup -> {
-                    if (canUseRowWriter) {
-                      return runClusteringForGroupAsyncAsRow(inputGroup,
-                          clusteringPlan.getStrategy().getStrategyParams(),
-                          shouldPreserveMetadata,
-                          instantTime,
-                          clusteringExecutorService);
-                    }
-                    return runClusteringForGroupAsync(inputGroup,
-                        clusteringPlan.getStrategy().getStrategyParams(),
-                        shouldPreserveMetadata,
-                        instantTime,
-                        clusteringExecutorService);
-                  })
+                  .map(inputGroup -> canUseRowWriter
+                      ? runClusteringForGroupAsyncAsRow(inputGroup, 
strategyParams, shouldPreserveMetadata, instantTime,
+                          clusteringExecutorService)
+                      : runClusteringForGroupAsync(inputGroup, strategyParams, 
shouldPreserveMetadata, instantTime,
+                          clusteringExecutorService))
                   .collect(Collectors.toList()))
           .join()
           .stream();
@@ -137,10 +183,72 @@ public abstract class MultipleSparkJobExecutionStrategy<T>
       writeMetadata.setWriteStatuses(HoodieJavaRDD.of(writeStatusRDD));
       return writeMetadata;
     } finally {
+      // On success every group has been joined, and the groups' RDDs hold 
what they need from the run invariants.
+      // On failure join() throws first, so a group may still be running: it 
reads the reader state it captured when
+      // submitted, and builds its own write config and writer schema from 
here on. The run fails either way.
+      runInvariants = null;
       clusteringExecutorService.shutdown();
     }
   }
 
+  /**
+   * What every group of one {@link #performClustering} run shares (see the 
class comment), fixed for the run: built on
+   * the driver before the groups are submitted, dropped when they have all 
been submitted. The caches only memoize:
+   * each derives its value from the key and the run's write config, so a 
key's value never changes within the run.
+   * Never serialized: groups capture its values, not the holder.
+   */
+  private static final class ClusteringRunInvariants {
+    /** The schema the groups' records have, with meta fields, as handed to 
performClusteringWithRecords*. */
+    private final HoodieSchema readerSchema;
+    private final ReaderContextFactory<?> readerContextFactory;
+    /** Row writer only: {@link #readerSchema} as Spark types. */
+    private final StructType sparkReaderSchema;
+    /** Row writer only: the file group readers' properties. */
+    private final TypedProperties readerProperties;
+    /** Groups run concurrently, on the clustering thread pool. */
+    private final ConcurrentMap<Option<Long>, HoodieWriteConfig> 
groupWriteConfigs = new ConcurrentHashMap<>();
+    /** Keyed by dataset schema and table schema: groups may read different 
schemas. */
+    private final ConcurrentMap<Pair<StructType, String>, StructType> 
groupWriterSchemas = new ConcurrentHashMap<>();
+
+    private ClusteringRunInvariants(HoodieSchema readerSchema, 
ReaderContextFactory<?> readerContextFactory,
+                               StructType sparkReaderSchema, TypedProperties 
readerProperties) {
+      this.readerSchema = readerSchema;
+      this.readerContextFactory = readerContextFactory;
+      this.sparkReaderSchema = sparkReaderSchema;
+      this.readerProperties = readerProperties;
+    }
+
+    @SuppressWarnings("unchecked")
+    private <R> ReaderContextFactory<R> readerContextFactory() {
+      return (ReaderContextFactory<R>) readerContextFactory;
+    }
+  }
+
+  /**
+   * The invariants of the {@link #performClustering} run that is submitting 
groups, or, for a group submitted from
+   * outside a run (a subclass's own performClustering calling the protected 
hooks), invariants built for that group
+   * alone: it pays the per-group cost this class otherwise avoids, but runs 
as it did before the holder existed.
+   */
+  private ClusteringRunInvariants 
currentRunInvariants(Supplier<ClusteringRunInvariants> forOneGroup) {
+    ClusteringRunInvariants run = runInvariants;
+    return run != null ? run : forOneGroup.get();
+  }
+
+  /** The row writer reads with the table schema resolved from the timeline: 
see getTableSchemaWithMetaFields. */
+  private ClusteringRunInvariants newRowWriterRunInvariants() {
+    HoodieSchema tableSchemaWithMetaFields = getTableSchemaWithMetaFields();
+    return new ClusteringRunInvariants(tableSchemaWithMetaFields,
+        
getEngineContext().getReaderContextFactory(getHoodieTable().getMetaClient()),
+        
HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(tableSchemaWithMetaFields),
+        
getReaderProperties(MergeUtils.getMaxMemoryPerCompaction(getEngineContext().getTaskContextSupplier(),
 writeConfig)));
+  }
+
+  /** The records path reads with the write schema; its tasks size their own 
merge memory. */
+  private ClusteringRunInvariants newRecordsRunInvariants() {
+    return new 
ClusteringRunInvariants(HoodieSchemaUtils.addMetadataFields(HoodieSchema.parse(getWriteConfig().getSchema())),
+        
getEngineContext().getReaderContextFactory(getHoodieTable().getMetaClient()), 
null, null);
+  }
+
   /**
    * Execute clustering to write inputRecords into new files based on 
strategyParams.
    * Different from {@link 
MultipleSparkJobExecutionStrategy#performClusteringWithRecordsRDD}, this method 
take {@link Dataset<Row>}
@@ -241,10 +349,11 @@ public abstract class MultipleSparkJobExecutionStrategy<T>
   protected CompletableFuture<HoodieData<WriteStatus>> 
runClusteringForGroupAsync(HoodieClusteringGroup clusteringGroup, Map<String, 
String> strategyParams,
                                                                                
   boolean preserveHoodieMetadata, String instantTime,
                                                                                
   ExecutorService clusteringExecutorService) {
+    ClusteringRunInvariants run = 
currentRunInvariants(this::newRecordsRunInvariants);
+    HoodieSchema readerSchema = run.readerSchema;
     return CompletableFuture.supplyAsync(() -> {
       JavaSparkContext jsc = 
HoodieSparkEngineContext.getSparkContext(getEngineContext());
-      HoodieData<HoodieRecord<T>> inputRecords = readRecordsForGroup(jsc, 
clusteringGroup, instantTime);
-      HoodieSchema readerSchema = 
HoodieSchemaUtils.addMetadataFields(HoodieSchema.parse(getWriteConfig().getSchema()));
+      HoodieData<HoodieRecord<T>> inputRecords = readRecordsForGroup(jsc, 
clusteringGroup, instantTime, run.readerContextFactory());
       // NOTE: Record have to be cloned here to make sure if it holds 
low-level engine-specific
       //       payload pointing into a shared, mutable (underlying) buffer we 
get a clean copy of
       //       it since these records will be shuffled later.
@@ -264,16 +373,12 @@ public abstract class MultipleSparkJobExecutionStrategy<T>
                                                                                
        boolean shouldPreserveHoodieMetadata,
                                                                                
        String instantTime,
                                                                                
        ExecutorService clusteringExecutorService) {
+    ClusteringRunInvariants run = 
currentRunInvariants(this::newRowWriterRunInvariants);
+    HoodieSchema tableSchemaWithMetaFields = run.readerSchema;
     return CompletableFuture.supplyAsync(() -> {
-      JavaSparkContext jsc = 
HoodieSparkEngineContext.getSparkContext(getEngineContext());      // incase of 
MIT, config.getSchema may not contain the full table schema
-      HoodieSchema tableSchemaWithMetaFields = null;
-      try {
-        tableSchemaWithMetaFields = HoodieSchemaUtils.addMetadataFields(new 
TableSchemaResolver(getHoodieTable().getMetaClient()).getTableSchema(false),
-            getWriteConfig().allowOperationMetadataField());
-      } catch (Exception e) {
-        throw new HoodieException("Failed to get table schema during 
clustering", e);
-      }
-      Dataset<Row> inputRecords = readRecordsForGroupAsRow(jsc, 
clusteringGroup, instantTime, tableSchemaWithMetaFields);
+      JavaSparkContext jsc = 
HoodieSparkEngineContext.getSparkContext(getEngineContext());
+      Dataset<Row> inputRecords = readRecordsForGroupAsRow(jsc, 
clusteringGroup, instantTime, tableSchemaWithMetaFields,
+          run.sparkReaderSchema, run.readerContextFactory(), 
run.readerProperties);
 
       List<HoodieFileGroupId> inputFileIds = 
clusteringGroup.getSlices().stream()
           .map(info -> new HoodieFileGroupId(info.getPartitionPath(), 
info.getFileId()))
@@ -283,13 +388,26 @@ public abstract class MultipleSparkJobExecutionStrategy<T>
     }, clusteringExecutorService);
   }
 
+  /**
+   * The table schema with meta fields, resolved from the timeline: in case of 
MIT, config.getSchema may not contain
+   * the full table schema.
+   */
+  private HoodieSchema getTableSchemaWithMetaFields() {
+    try {
+      return HoodieSchemaUtils.addMetadataFields(new 
TableSchemaResolver(getHoodieTable().getMetaClient()).getTableSchema(false),
+          getWriteConfig().allowOperationMetadataField());
+    } catch (Exception e) {
+      throw new HoodieException("Failed to get table schema during 
clustering", e);
+    }
+  }
+
   /**
    * Get RDD of all records for the group. This includes all records from file 
slice (Apply updates from log files, if any).
    */
-  private HoodieData<HoodieRecord<T>> readRecordsForGroup(JavaSparkContext 
jsc, HoodieClusteringGroup clusteringGroup, String instantTime) {
+  private HoodieData<HoodieRecord<T>> readRecordsForGroup(JavaSparkContext 
jsc, HoodieClusteringGroup clusteringGroup, String instantTime,
+                                                          
ReaderContextFactory<T> readerContextFactory) {
     List<ClusteringOperation> clusteringOps = 
clusteringGroup.getSlices().stream().map(ClusteringOperation::create).collect(Collectors.toList());
     int readParallelism = 
Math.min(writeConfig.getClusteringGroupReadParallelism(), clusteringOps.size());
-    ReaderContextFactory<T> readerContextFactory = 
getEngineContext().getReaderContextFactory(getHoodieTable().getMetaClient());
     return HoodieJavaRDD.of(jsc.parallelize(clusteringOps, 
readParallelism).mapPartitions(clusteringOpsPartition -> {
       List<Supplier<ClosableIterator<HoodieRecord<T>>>> suppliers = new 
ArrayList<>();
       long maxMemoryPerCompaction = MergeUtils.getMaxMemoryPerCompaction(new 
SparkTaskContextSupplier(), getWriteConfig());
@@ -308,20 +426,16 @@ public abstract class MultipleSparkJobExecutionStrategy<T>
   private Dataset<Row> readRecordsForGroupAsRow(JavaSparkContext jsc,
                                                 HoodieClusteringGroup 
clusteringGroup,
                                                 String instantTime,
-                                                HoodieSchema 
tableSchemaWithMetaFields) {
+                                                HoodieSchema 
tableSchemaWithMetaFields,
+                                                StructType 
sparkSchemaWithMetaFields,
+                                                
ReaderContextFactory<InternalRow> readerContextFactory,
+                                                TypedProperties 
readerProperties) {
     List<ClusteringOperation> clusteringOps = 
clusteringGroup.getSlices().stream()
         .map(ClusteringOperation::create).collect(Collectors.toList());
     String basePath = getWriteConfig().getBasePath();
-    // construct supporting cast that executors might need
-    long maxMemoryPerCompaction = 
MergeUtils.getMaxMemoryPerCompaction(getEngineContext().getTaskContextSupplier(),
 writeConfig);
-    TypedProperties readerProperties = 
getReaderProperties(maxMemoryPerCompaction);
     final boolean usePosition = 
getWriteConfig().getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
     String internalSchemaStr = getWriteConfig().getInternalSchema();
-
-    // broadcast reader context.
     HoodieTableMetaClient metaClient = getHoodieTable().getMetaClient();
-    ReaderContextFactory<InternalRow> readerContextFactory = 
getEngineContext().getReaderContextFactory(metaClient);
-    StructType sparkSchemaWithMetaFields = 
HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(tableSchemaWithMetaFields);
 
     RDD<InternalRow> internalRowRDD = jsc.parallelize(clusteringOps, 
clusteringOps.size()).flatMap(new FlatMapFunction<ClusteringOperation, 
InternalRow>() {
       @Override
diff --git 
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java
 
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java
index 379f2cb4b858..30cb61555c90 100644
--- 
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java
+++ 
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java
@@ -27,6 +27,7 @@ import org.apache.hudi.common.model.HoodieFileGroupId;
 import org.apache.hudi.common.model.HoodieRecord;
 import org.apache.hudi.common.model.HoodieRecordPayload;
 import org.apache.hudi.common.schema.HoodieSchema;
+import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.collection.Pair;
 import org.apache.hudi.config.HoodieWriteConfig;
 import org.apache.hudi.exception.HoodieClusteringException;
@@ -43,7 +44,6 @@ import org.apache.spark.sql.Row;
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
-import java.util.Properties;
 
 /**
  * Clustering execution strategy specifically for consistent hashing index
@@ -67,9 +67,7 @@ public class 
SparkConsistentBucketClusteringExecutionStrategy<T extends HoodieRe
                                                                    boolean 
shouldPreserveHoodieMetadata,
                                                                    Map<String, 
String> extraMetadata) {
     log.info("Starting clustering for a group, parallelism:{} commit:{}", 
numOutputGroups, instantTime);
-    Properties props = getWriteConfig().getProps();
-
-    HoodieWriteConfig newConfig = 
HoodieWriteConfig.newBuilder().withProps(props).build();
+    HoodieWriteConfig newConfig = getGroupWriteConfig(Option.empty());
 
     Pair<String, List<ConsistentHashingNode>> childNodesPair = 
extractChildNodes(extraMetadata);
     ConsistentBucketIndexBulkInsertPartitionerWithRows partitioner = new 
ConsistentBucketIndexBulkInsertPartitionerWithRows(getHoodieTable(), 
strategyParams, shouldPreserveHoodieMetadata,
@@ -78,7 +76,8 @@ public class 
SparkConsistentBucketClusteringExecutionStrategy<T extends HoodieRe
     Dataset<Row> repartitionedRecords = 
partitioner.repartitionRecords(inputRecords, numOutputGroups);
 
     return HoodieDatasetBulkInsertHelper.bulkInsert(repartitionedRecords, 
instantTime, getHoodieTable(), newConfig,
-        partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata);
+        partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata,
+        getGroupWriterSchema(repartitionedRecords.schema(), newConfig));
   }
 
   @Override
@@ -87,8 +86,7 @@ public class 
SparkConsistentBucketClusteringExecutionStrategy<T extends HoodieRe
                                                                  boolean 
preserveHoodieMetadata, Map<String, String> extraMetadata) {
 
     log.info("Starting clustering for a group, parallelism:{} commit:{}", 
numOutputGroups, instantTime);
-    Properties props = getWriteConfig().getProps();
-    HoodieWriteConfig newConfig = 
HoodieWriteConfig.newBuilder().withProps(props).build();
+    HoodieWriteConfig newConfig = getGroupWriteConfig(Option.empty());
 
     Pair<String, List<ConsistentHashingNode>> childNodesPair = 
extractChildNodes(extraMetadata);
     RDDConsistentBucketBulkInsertPartitioner<T> partitioner = new 
RDDConsistentBucketBulkInsertPartitioner<>(getHoodieTable(), strategyParams, 
preserveHoodieMetadata,
diff --git 
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkSingleFileSortExecutionStrategy.java
 
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkSingleFileSortExecutionStrategy.java
index ea778ce4fc90..a03e2788bf36 100644
--- 
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkSingleFileSortExecutionStrategy.java
+++ 
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkSingleFileSortExecutionStrategy.java
@@ -21,12 +21,12 @@ package org.apache.hudi.client.clustering.run.strategy;
 
 import org.apache.hudi.HoodieDatasetBulkInsertHelper;
 import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.common.config.HoodieStorageConfig;
 import org.apache.hudi.common.data.HoodieData;
 import org.apache.hudi.common.engine.HoodieEngineContext;
 import org.apache.hudi.common.model.HoodieFileGroupId;
 import org.apache.hudi.common.model.HoodieRecord;
 import org.apache.hudi.common.schema.HoodieSchema;
+import org.apache.hudi.common.util.Option;
 import org.apache.hudi.config.HoodieWriteConfig;
 import org.apache.hudi.exception.HoodieClusteringException;
 import org.apache.hudi.io.SingleFileHandleCreateFactory;
@@ -69,18 +69,15 @@ public class SparkSingleFileSortExecutionStrategy<T>
     }
     log.info("Starting clustering for a group, parallelism:{} commit:{}", 
numOutputGroups, instantTime);
 
-    HoodieWriteConfig newConfig = HoodieWriteConfig.newBuilder()
-        .withBulkInsertParallelism(numOutputGroups)
-        .withProps(getWriteConfig().getProps()).build();
-
     // Since clustering will write to single file group using 
HoodieUnboundedCreateHandle, set max file size to a large value.
-    newConfig.setValue(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE, 
String.valueOf(Long.MAX_VALUE));
+    HoodieWriteConfig newConfig = 
getGroupWriteConfig(Option.of(Long.MAX_VALUE));
 
     BulkInsertPartitioner<Dataset<Row>> partitioner = 
getRowPartitioner(strategyParams, schema);
     Dataset<Row> repartitionedRecords = 
partitioner.repartitionRecords(inputRecords, numOutputGroups);
 
     return HoodieDatasetBulkInsertHelper.bulkInsert(repartitionedRecords, 
instantTime, getHoodieTable(), newConfig,
-        partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata);
+        partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata,
+        getGroupWriterSchema(repartitionedRecords.schema(), newConfig));
   }
 
   @Override
@@ -97,11 +94,8 @@ public class SparkSingleFileSortExecutionStrategy<T>
     }
     log.info("Starting clustering for a group, parallelism:{} commit:{}", 
numOutputGroups, instantTime);
 
-    HoodieWriteConfig newConfig = HoodieWriteConfig.newBuilder()
-        .withBulkInsertParallelism(numOutputGroups)
-        .withProps(getWriteConfig().getProps()).build();
     // Since clustering will write to single file group using 
HoodieUnboundedCreateHandle, set max file size to a large value.
-    newConfig.setValue(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE, 
String.valueOf(Long.MAX_VALUE));
+    HoodieWriteConfig newConfig = 
getGroupWriteConfig(Option.of(Long.MAX_VALUE));
 
     return (HoodieData<WriteStatus>) 
SparkBulkInsertHelper.newInstance().bulkInsert(inputRecords, instantTime, 
getHoodieTable(), newConfig,
         false, getRDDPartitioner(strategyParams, schema), true, 
numOutputGroups,
diff --git 
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkSortAndSizeExecutionStrategy.java
 
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkSortAndSizeExecutionStrategy.java
index 05d9b23a8934..97ad508ffdc1 100644
--- 
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkSortAndSizeExecutionStrategy.java
+++ 
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkSortAndSizeExecutionStrategy.java
@@ -20,12 +20,12 @@ package org.apache.hudi.client.clustering.run.strategy;
 
 import org.apache.hudi.HoodieDatasetBulkInsertHelper;
 import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.common.config.HoodieStorageConfig;
 import org.apache.hudi.common.data.HoodieData;
 import org.apache.hudi.common.engine.HoodieEngineContext;
 import org.apache.hudi.common.model.HoodieFileGroupId;
 import org.apache.hudi.common.model.HoodieRecord;
 import org.apache.hudi.common.schema.HoodieSchema;
+import org.apache.hudi.common.util.Option;
 import org.apache.hudi.config.HoodieWriteConfig;
 import org.apache.hudi.io.CreateHandleFactory;
 import org.apache.hudi.table.BulkInsertPartitioner;
@@ -63,17 +63,14 @@ public class SparkSortAndSizeExecutionStrategy<T>
                                                                    boolean 
shouldPreserveHoodieMetadata,
                                                                    Map<String, 
String> extraMetadata) {
     log.info("Starting clustering for a group, parallelism:{} commit:{}", 
numOutputGroups, instantTime);
-    HoodieWriteConfig newConfig = HoodieWriteConfig.newBuilder()
-        .withBulkInsertParallelism(numOutputGroups)
-        .withProps(getWriteConfig().getProps()).build();
-
-    newConfig.setValue(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE, 
String.valueOf(getWriteConfig().getClusteringTargetFileMaxBytes()));
+    HoodieWriteConfig newConfig = 
getGroupWriteConfig(Option.of(getWriteConfig().getClusteringTargetFileMaxBytes()));
 
     BulkInsertPartitioner<Dataset<Row>> partitioner = 
getRowPartitioner(strategyParams, schema);
     Dataset<Row> repartitionedRecords = 
partitioner.repartitionRecords(inputRecords, numOutputGroups);
 
     return HoodieDatasetBulkInsertHelper.bulkInsert(repartitionedRecords, 
instantTime, getHoodieTable(), newConfig,
-        partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata);
+        partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata,
+        getGroupWriterSchema(repartitionedRecords.schema(), newConfig));
   }
 
   @Override
@@ -87,11 +84,7 @@ public class SparkSortAndSizeExecutionStrategy<T>
                                                                  final 
Map<String, String> extraMetadata) {
     log.info("Starting clustering for a group, parallelism:{} commit:{}", 
numOutputGroups, instantTime);
 
-    HoodieWriteConfig newConfig = HoodieWriteConfig.newBuilder()
-        .withBulkInsertParallelism(numOutputGroups)
-        .withProps(getWriteConfig().getProps()).build();
-
-    newConfig.setValue(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE, 
String.valueOf(getWriteConfig().getClusteringTargetFileMaxBytes()));
+    HoodieWriteConfig newConfig = 
getGroupWriteConfig(Option.of(getWriteConfig().getClusteringTargetFileMaxBytes()));
 
     return (HoodieData<WriteStatus>) 
SparkBulkInsertHelper.newInstance().bulkInsert(inputRecords, instantTime, 
getHoodieTable(),
         newConfig, false, getRDDPartitioner(strategyParams, schema), true, 
numOutputGroups, new CreateHandleFactory(shouldPreserveHoodieMetadata));
diff --git 
a/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala
 
b/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala
index 02e3c0934ffe..1695552bd6af 100644
--- 
a/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala
+++ 
b/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala
@@ -160,8 +160,29 @@ object HoodieDatasetBulkInsertHelper
                  table: HoodieTable[_, _, _, _],
                  writeConfig: HoodieWriteConfig,
                  arePartitionRecordsSorted: Boolean,
-                 shouldPreserveHoodieMetadata: Boolean): 
HoodieData[WriteStatus] = {
-    val schema = 
HoodieSchemaConversionUtils.alignFieldsNullability(dataset.schema, 
HoodieSchema.parse(writeConfig.getSchema))
+                 shouldPreserveHoodieMetadata: Boolean): 
HoodieData[WriteStatus] =
+    bulkInsert(dataset, instantTime, table, writeConfig, 
arePartitionRecordsSorted, shouldPreserveHoodieMetadata,
+      writerSchema(dataset.schema, writeConfig))
+
+  /**
+   * The schema the bulk-insert writers use for a dataset: its schema, with 
field nullability aligned to the table
+   * schema in `writeConfig`.
+   */
+  def writerSchema(datasetSchema: StructType, writeConfig: HoodieWriteConfig): 
StructType =
+    HoodieSchemaConversionUtils.alignFieldsNullability(datasetSchema, 
HoodieSchema.parse(writeConfig.getSchema))
+
+  /**
+   * Same as above, with the writer schema (see [[writerSchema]]) supplied by 
the caller. The write closure captures
+   * it, so a caller that unions many bulk inserts into one stage (clustering) 
passes one shared instance: Spark ships
+   * the whole stage lineage to every task, and a schema built per call is 
deserialized once per call by every task.
+   */
+  def bulkInsert(dataset: Dataset[Row],
+                 instantTime: String,
+                 table: HoodieTable[_, _, _, _],
+                 writeConfig: HoodieWriteConfig,
+                 arePartitionRecordsSorted: Boolean,
+                 shouldPreserveHoodieMetadata: Boolean,
+                 schema: StructType): HoodieData[WriteStatus] = {
     HoodieJavaRDD.of(
       injectSQLConf(dataset.queryExecution.toRdd.mapPartitions(iter => {
         val taskContextSupplier: TaskContextSupplier = 
table.getTaskContextSupplier
diff --git 
a/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/SQLConfInjectingRDD.scala
 
b/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/SQLConfInjectingRDD.scala
index 31b2305fcf4a..c75a042099b1 100644
--- 
a/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/SQLConfInjectingRDD.scala
+++ 
b/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/SQLConfInjectingRDD.scala
@@ -35,7 +35,7 @@ import scala.reflect.ClassTag
  * @param conf the `SQLConf` to apply to the execution of the SQL plan
  */
 class SQLConfInjectingRDD[T: ClassTag](var sqlRDD: RDD[T], @transient conf: 
SQLConf) extends RDD[T](sqlRDD) {
-  private val sqlConfigs = conf.getAllConfs
+  private val sqlConfigs = SQLConfInjectingRDD.sharedConfigs(conf)
   private lazy val sqlConfExecutorSide = {
     val newConf = new SQLConf()
     sqlConfigs.foreach { case (k, v) => newConf.setConfString(k, v) }
@@ -58,4 +58,37 @@ class SQLConfInjectingRDD[T: ClassTag](var sqlRDD: RDD[T], 
@transient conf: SQLC
       firstParent[T].iterator(split, context)
     }
   }
-}
\ No newline at end of file
+}
+
+object SQLConfInjectingRDD {
+
+  /**
+   * The last config map captured; driver-side only, guarded by this object. 
See [[sharedConfigs]].
+   *
+   * Held strongly on purpose, for the life of the driver JVM: one map, 
replaced by the next capture whose configs
+   * differ. A weak reference would let the map be collected between two 
groups' captures, and the next group would
+   * then ship a copy of its own. The map holds what every SQLConfInjectingRDD 
already carries and ships to its tasks,
+   * credentials included; keeping it adds no new place they go, only time 
past the session that set them.
+   */
+  private var lastConfigs: Map[String, String] = Map.empty
+
+  /**
+   * The configs of `conf`, as the same map instance for every RDD that 
captures equal configs.
+   *
+   * Clustering wraps every group's RDD in its own [[SQLConfInjectingRDD]] and 
unions all groups into one stage, so
+   * every task deserializes every group's RDD. Java serialization writes an 
instance shared by several objects once
+   * per stream, so a shared map costs one copy per task, where a map built 
per RDD costs one copy per group in every
+   * task. Keyed by content: a changed conf yields a new map, never a stale 
one. Clustering creates the groups' RDDs
+   * concurrently, so the compare-and-set is synchronized: otherwise 
concurrent callers that all miss would each keep
+   * their own map.
+   */
+  private[execution] def sharedConfigs(conf: SQLConf): Map[String, String] = {
+    val current = conf.getAllConfs
+    synchronized {
+      if (lastConfigs != current) {
+        lastConfigs = current
+      }
+      lastConfigs
+    }
+  }
+}
diff --git 
a/hudi-client/hudi-spark-client/src/test/scala/org/apache/spark/sql/execution/TestSQLConfInjectingRDD.scala
 
b/hudi-client/hudi-spark-client/src/test/scala/org/apache/spark/sql/execution/TestSQLConfInjectingRDD.scala
new file mode 100644
index 000000000000..c3af24d67662
--- /dev/null
+++ 
b/hudi-client/hudi-spark-client/src/test/scala/org/apache/spark/sql/execution/TestSQLConfInjectingRDD.scala
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.apache.hudi.HoodieSparkUtils.injectSQLConf
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.{AfterAll, BeforeAll, Test, TestInstance}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertNotSame, 
assertTrue}
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, 
ObjectInputStream, ObjectOutputStream}
+import java.util.{Collections, IdentityHashMap}
+import java.util.concurrent.{Callable, CountDownLatch, Executors, TimeUnit}
+
+/**
+ * Pins down how [[SQLConfInjectingRDD]] ships its captured SQL configs.
+ *
+ * Clustering wraps every group's RDD in its own [[SQLConfInjectingRDD]] and 
unions all groups into one stage, so each
+ * task deserializes every group's RDD. Java serialization writes an instance 
shared by several objects once per
+ * stream, but writes one copy of each distinct instance: a config map built 
per RDD is rebuilt once per group in
+ * every task (measured on a 4,000-group production plan: 16,000 maps of ~120 
entries, ~97 MB per executor with
+ * 4 tasks running). Equal configs must therefore share one map instance.
+ */
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+class TestSQLConfInjectingRDD {
+
+  private val Groups = 12
+  private val ConcurrentRounds = 500
+  private var spark: SparkSession = _
+
+  @BeforeAll
+  def start(): Unit = {
+    spark = 
SparkSession.builder().master("local[2]").appName("sqlconf-injecting-rdd")
+      .config("spark.ui.enabled", "false").getOrCreate()
+  }
+
+  @AfterAll
+  def stop(): Unit = {
+    if (spark != null) {
+      spark.stop()
+    }
+  }
+
+  @Test
+  def aTaskDeserializesOneConfigMapForAllGroups(): Unit = {
+    // What a task binary does: all groups' RDDs written to, and read back 
from, a single stream.
+    val conf = spark.sessionState.conf
+    val maps = configMapsDeserialized(perGroupRdds(conf), conf.getAllConfs)
+    assertEquals(1, maps, s"one config map per task, not one per group (got 
$maps)")
+  }
+
+  @Test
+  def concurrentlyCreatedGroupsShareOneConfigMap(): Unit = {
+    // Clustering creates the groups' RDDs on a thread pool. Each round starts 
from a different captured conf, so all
+    // threads miss at once; the race window is small, hence many rounds.
+    val conf = spark.sessionState.conf
+    val other = conf.clone()
+    other.setConfString(SQLConf.SHUFFLE_PARTITIONS.key, "13")
+    val pool = Executors.newFixedThreadPool(Groups)
+    try {
+      (0 until ConcurrentRounds).foreach { round =>
+        SQLConfInjectingRDD.sharedConfigs(other)
+        val start = new CountDownLatch(1)
+        val futures = (0 until Groups).map { _ =>
+          pool.submit(new Callable[Map[String, String]] {
+            override def call(): Map[String, String] = {
+              start.await()
+              SQLConfInjectingRDD.sharedConfigs(conf)
+            }
+          })
+        }
+        start.countDown()
+        val maps = futures.map(f => System.identityHashCode(f.get(30, 
TimeUnit.SECONDS))).distinct
+        assertEquals(1, maps.size, s"round $round: concurrent groups must 
share one config map (got ${maps.size})")
+      }
+    } finally {
+      pool.shutdownNow()
+    }
+  }
+
+  @Test
+  def changedConfGetsItsOwnMapNeverAStaleOne(): Unit = {
+    val base = spark.sessionState.conf
+    val shared = SQLConfInjectingRDD.sharedConfigs(base)
+
+    val changed = base.clone()
+    changed.setConfString(SQLConf.SHUFFLE_PARTITIONS.key, "7")
+    assertNotSame(shared, SQLConfInjectingRDD.sharedConfigs(changed))
+    // An RDD capturing the changed conf ships the changed configs, not the 
earlier ones.
+    val changedRdd = injectSQLConf(spark.sparkContext.parallelize(Seq(1), 1), 
changed)
+    assertEquals(1, configMapsDeserialized(changedRdd, changed.getAllConfs))
+    assertEquals(0, configMapsDeserialized(changedRdd, base.getAllConfs))
+
+    // Back to the original configs: the captured content must match them 
exactly.
+    assertEquals(base.getAllConfs, SQLConfInjectingRDD.sharedConfigs(base))
+  }
+
+  @Test
+  def tasksRunWithTheCapturedConf(): Unit = {
+    val conf = spark.sessionState.conf.clone()
+    conf.setConfString(SQLConf.SHUFFLE_PARTITIONS.key, "7")
+    // The captured conf is in effect while compute() builds the parent's 
iterator (like Spark's SQLExecutionRDD),
+    // not while that iterator is consumed: read it in the mapPartitions body, 
which runs during compute().
+    val seen = injectSQLConf(
+      spark.sparkContext.parallelize(Seq(1), 1).mapPartitions { _ =>
+        Iterator(SQLConf.get.getConfString(SQLConf.SHUFFLE_PARTITIONS.key))
+      },
+      conf).collect()
+    assertTrue(seen.sameElements(Seq("7")), s"task saw ${seen.mkString(",")}")
+  }
+
+  /** One wrapped RDD per clustering group, each built from the same captured 
conf, as clustering does. */
+  private def perGroupRdds(conf: SQLConf): Seq[RDD[Int]] =
+    (0 until Groups).map(g => 
injectSQLConf(spark.sparkContext.parallelize(Seq(g), 1).map(identity), conf))
+
+  /**
+   * Serializes `value` into one stream, as a task binary is, reads it back, 
and returns how many distinct maps equal
+   * to `configs` the reader built. Counted on the read side: a Scala map is 
written through a serialization proxy
+   * (`writeReplace`), and only `resolveObject` sees the map itself.
+   */
+  private def configMapsDeserialized(value: AnyRef, configs: Map[String, 
String]): Int = {
+    val bytes = new ByteArrayOutputStream()
+    val out = new ObjectOutputStream(bytes)
+    out.writeObject(value)
+    out.close()
+    val maps = Collections.newSetFromMap(new IdentityHashMap[AnyRef, 
java.lang.Boolean]())
+    val in = new ObjectInputStream(new 
ByteArrayInputStream(bytes.toByteArray)) {
+      enableResolveObject(true)
+
+      override protected def resolveObject(obj: AnyRef): AnyRef = {
+        obj match {
+          case map: scala.collection.Map[_, _] if map == configs => 
maps.add(map)
+          case _ =>
+        }
+        obj
+      }
+    }
+    in.readObject()
+    maps.size
+  }
+}
diff --git 
a/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/clustering/run/strategy/TestClusteringTaskBinaryPayloads.java
 
b/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/clustering/run/strategy/TestClusteringTaskBinaryPayloads.java
new file mode 100644
index 000000000000..05545ff5e550
--- /dev/null
+++ 
b/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/clustering/run/strategy/TestClusteringTaskBinaryPayloads.java
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.DataSourceWriteOptions;
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteClientTestUtils;
+import org.apache.hudi.client.WriteStatus;
+import 
org.apache.hudi.client.clustering.plan.strategy.SparkSingleFileSortPlanStrategy;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.schema.HoodieSchema;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.table.marker.MarkerType;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.table.HoodieSparkTable;
+import org.apache.hudi.testutils.HoodieClientTestBase;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.rdd.RDD;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.OutputStream;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static 
org.apache.hudi.common.testutils.HoodieTestUtils.INSTANT_GENERATOR;
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Measures what the write stage of {@link 
MultipleSparkJobExecutionStrategy#performClustering} ships to every task.
+ *
+ * <p>Clustering builds one RDD per group and unions them into a single write 
stage. Spark serializes the stage
+ * lineage into one task binary that every task deserializes, so an object a 
group's lineage captures is shipped
+ * to, and deserialized by, every task of the stage, for every group. Objects 
that do not depend on the group
+ * (table schema, write config, reader properties, reader context factory, SQL 
configs) should therefore be shared
+ * instances: Java serialization writes a shared instance once per stream.
+ *
+ * <p>The test runs each strategy, on the row writer and on the records path, 
on a small and a large subset of one
+ * plan, serializes the resulting lineage the way Spark's closure serializer 
does (one {@link ObjectOutputStream}),
+ * and counts the distinct instances of each such payload. The counts must not 
grow with the number of groups.
+ *
+ * <p>Groups submitted outside a run, through the protected hooks, build what 
they would have shared, and are still
+ * written in full.
+ */
+@Tag("functional")
+public class TestClusteringTaskBinaryPayloads extends HoodieClientTestBase {
+
+  private static final int SMALL_PLAN_GROUPS = 2;
+  private static final int LARGE_PLAN_GROUPS = 8;
+  // dataGen spreads inserts over 3 partitions; with small-file handling off, 
each batch adds one file group
+  // per partition, and the single-file plan strategy emits one clustering 
group per file group.
+  private static final int PARTITIONS = 3;
+  private static final int INSERT_BATCHES = (LARGE_PLAN_GROUPS + PARTITIONS - 
1) / PARTITIONS;
+  private static final int RECORDS_PER_BATCH = 30;
+  private static final String SQL_CONF_MAPS = "SQL config maps";
+
+  /** Payloads that do not depend on the group, so their count must not depend 
on the number of groups. */
+  private static final Map<String, Predicate<Object>> SHARED_PAYLOADS = new 
LinkedHashMap<>();
+
+  static {
+    SHARED_PAYLOADS.put("HoodieSchema", HoodieSchema.class::isInstance);
+    SHARED_PAYLOADS.put("HoodieWriteConfig", 
HoodieWriteConfig.class::isInstance);
+    SHARED_PAYLOADS.put("TypedProperties", TypedProperties.class::isInstance);
+    SHARED_PAYLOADS.put("ReaderContextFactory", 
ReaderContextFactory.class::isInstance);
+    SHARED_PAYLOADS.put("StructType", StructType.class::isInstance);
+  }
+
+  /** The strategies whose groups share a write config of their own: each keys 
the shared config by its file size. */
+  private enum Strategy {
+    SORT_AND_SIZE {
+      @Override
+      MultipleSparkJobExecutionStrategy<?> create(HoodieSparkTable table, 
HoodieEngineContext context, HoodieWriteConfig config) {
+        return new SparkSortAndSizeExecutionStrategy<>(table, context, config);
+      }
+    },
+    SINGLE_FILE_SORT {
+      @Override
+      MultipleSparkJobExecutionStrategy<?> create(HoodieSparkTable table, 
HoodieEngineContext context, HoodieWriteConfig config) {
+        return new SparkSingleFileSortExecutionStrategy<>(table, context, 
config);
+      }
+    };
+
+    abstract MultipleSparkJobExecutionStrategy<?> create(HoodieSparkTable 
table, HoodieEngineContext context,
+                                                         HoodieWriteConfig 
config);
+  }
+
+  /** Each strategy on each write path: the row writer and the records path 
build their run invariants apart. */
+  private static Stream<Arguments> strategiesAndWritePaths() {
+    return Stream.of(Strategy.values())
+        .flatMap(strategy -> Stream.of(true, false).map(rowWriter -> 
Arguments.of(strategy, rowWriter)));
+  }
+
+  @ParameterizedTest(name = "{0}, row writer {1}")
+  @MethodSource("strategiesAndWritePaths")
+  public void taskBinaryPayloadsDoNotGrowWithClusteringGroups(Strategy 
strategy, boolean rowWriter) throws Exception {
+    HoodieWriteConfig config = clusteringConfig(rowWriter);
+    String clusteringInstant = scheduleClustering(config);
+    HoodieClusteringPlan plan = clusteringPlan(clusteringInstant);
+    HoodieSchema schema = new 
TableSchemaResolver(metaClient).getTableSchema(false);
+
+    TaskBinary small = taskBinary(strategy, config, firstGroups(plan, 
SMALL_PLAN_GROUPS), schema, clusteringInstant);
+    TaskBinary large = taskBinary(strategy, config, firstGroups(plan, 
LARGE_PLAN_GROUPS), schema, clusteringInstant);
+    String report = report(small, large);
+    LOG.info("Clustering task binary payloads, {}, row writer {}:\n{}", 
strategy, rowWriter, report);
+
+    int extraGroups = large.groups - small.groups;
+    for (String payload : small.counts.keySet()) {
+      // Spark's SQLExecutionRDD (Dataset.queryExecution.toRdd) captures its 
own copy of the SQL configs, once per
+      // group, as long as each group's RDD goes through a DataFrame; Hudi's 
SQLConfInjectingRDD map must be shared.
+      int expectedGrowth = payload.equals(SQL_CONF_MAPS) && rowWriter ? 
extraGroups : 0;
+      assertEquals(small.counts.get(payload) + expectedGrowth, 
large.counts.get(payload),
+          payload + " is shipped per clustering group; every task deserializes 
all of them.\n" + report);
+    }
+  }
+
+  /**
+   * A custom strategy may override performClustering and submit groups 
through the protected hooks itself. No run's
+   * invariants are held then, so each hook builds its own for the group: the 
group must still be written in full.
+   */
+  @ParameterizedTest(name = "row writer {0}")
+  @ValueSource(booleans = {true, false})
+  public void groupSubmittedOutsideARunIsWrittenInFull(boolean rowWriter) 
throws Exception {
+    HoodieWriteConfig config = clusteringConfig(rowWriter);
+    String clusteringInstant = scheduleClustering(config);
+    HoodieClusteringPlan plan = clusteringPlan(clusteringInstant);
+    HoodieClusteringGroup group = plan.getInputGroups().get(0);
+    long groupRecords = group.getSlices().stream()
+        .mapToLong(slice -> 
context.getSqlContext().read().parquet(slice.getDataFilePath()).count())
+        .sum();
+
+    MultipleSparkJobExecutionStrategy<?> strategy =
+        Strategy.SORT_AND_SIZE.create(HoodieSparkTable.create(config, context, 
metaClient), context, config);
+    Map<String, String> strategyParams = 
plan.getStrategy().getStrategyParams();
+    ExecutorService groupThread = Executors.newSingleThreadExecutor();
+    List<WriteStatus> statuses;
+    try {
+      HoodieData<WriteStatus> written = (rowWriter
+          ? strategy.runClusteringForGroupAsyncAsRow(group, strategyParams, 
true, clusteringInstant, groupThread)
+          : strategy.runClusteringForGroupAsync(group, strategyParams, true, 
clusteringInstant, groupThread)).get();
+      statuses = written.collectAsList();
+    } finally {
+      groupThread.shutdown();
+    }
+
+    assertNoWriteErrors(statuses);
+    assertEquals(groupRecords, 
statuses.stream().mapToLong(WriteStatus::getTotalRecords).sum(),
+        "records written for the group, submitted outside performClustering");
+  }
+
+  /** Small-file handling off and one clustering group per file group, so a 
few batches give many groups. */
+  private HoodieWriteConfig clusteringConfig(boolean rowWriter) {
+    return getConfigBuilder()
+        
.withProps(Collections.singletonMap(DataSourceWriteOptions.ENABLE_ROW_WRITER().key(),
 Boolean.toString(rowWriter)))
+        // Direct markers: what is measured here does not involve the timeline 
server.
+        .withEmbeddedTimelineServerEnabled(false)
+        .withMarkersType(MarkerType.DIRECT.name())
+        
.withCompactionConfig(HoodieCompactionConfig.newBuilder().compactionSmallFileSize(0).build())
+        .withClusteringConfig(HoodieClusteringConfig.newBuilder()
+            
.withClusteringPlanStrategyClass(SparkSingleFileSortPlanStrategy.class.getName())
+            .withClusteringMaxNumGroups(INSERT_BATCHES * PARTITIONS)
+            .build())
+        .build();
+  }
+
+  /** Inserts {@value #INSERT_BATCHES} batches and schedules clustering over 
them; the clustering instant. */
+  private String scheduleClustering(HoodieWriteConfig config) {
+    try (SparkRDDWriteClient client = getHoodieWriteClient(config)) {
+      for (int batch = 0; batch < INSERT_BATCHES; batch++) {
+        String instant = WriteClientTestUtils.createNewInstantTime();
+        WriteClientTestUtils.startCommitWithTime(client, instant);
+        JavaRDD<WriteStatus> statuses = 
client.insert(jsc.parallelize(dataGen.generateInserts(instant, 
RECORDS_PER_BATCH), 1), instant);
+        List<WriteStatus> collected = statuses.collect();
+        assertNoWriteErrors(collected);
+        client.commit(instant, jsc.parallelize(collected));
+      }
+      return (String) client.scheduleClustering(Option.empty()).get();
+    }
+  }
+
+  private HoodieClusteringPlan clusteringPlan(String clusteringInstant) {
+    metaClient = HoodieTableMetaClient.reload(metaClient);
+    HoodieClusteringPlan plan = ClusteringUtils.getClusteringPlan(
+        metaClient, 
INSTANT_GENERATOR.getClusteringCommitRequestedInstant(clusteringInstant)).map(Pair::getRight).get();
+    assertTrue(plan.getInputGroups().size() >= LARGE_PLAN_GROUPS,
+        "need at least " + LARGE_PLAN_GROUPS + " groups, plan has " + 
plan.getInputGroups().size());
+    return plan;
+  }
+
+  private static HoodieClusteringPlan firstGroups(HoodieClusteringPlan plan, 
int groups) {
+    return 
HoodieClusteringPlan.newBuilder(plan).setInputGroups(plan.getInputGroups().subList(0,
 groups)).build();
+  }
+
+  private TaskBinary taskBinary(Strategy strategy, HoodieWriteConfig config, 
HoodieClusteringPlan plan, HoodieSchema schema,
+                                String instant) throws IOException {
+    HoodieSparkTable table = HoodieSparkTable.create(config, context, 
metaClient);
+    RDD<WriteStatus> writeStage = 
HoodieJavaRDD.getJavaRDD(strategy.create(table, context, config)
+        .performClustering(plan, schema, instant).getWriteStatuses()).rdd();
+    // The DAGScheduler resolves partitions and dependencies before 
serializing the task binary, which fills
+    // the lineage fields that get shipped.
+    resolveLineage(writeStage, Collections.newSetFromMap(new 
IdentityHashMap<>()));
+    ByteArrayOutputStream out = new ByteArrayOutputStream();
+    Map<String, Integer> counts;
+    try (PayloadCountingObjectOutputStream oos = new 
PayloadCountingObjectOutputStream(out)) {
+      oos.writeObject(writeStage);
+      oos.flush();
+      counts = oos.counts();
+    }
+    counts.put(SQL_CONF_MAPS, sqlConfMapsDeserialized(out.toByteArray()));
+    return new TaskBinary(plan.getInputGroups().size(), out.size(), counts);
+  }
+
+  /**
+   * How many distinct SQL config maps a task deserializing {@code taskBinary} 
builds. One SQLConfInjectingRDD per group
+   * is inherent; the configs map it carries need not be. Spark's 
SQLExecutionRDD adds one map per group of its own.
+   * Counted on the read side: a Scala map is written through a serialization 
proxy ({@code writeReplace}), and only
+   * {@code resolveObject} sees the map itself.
+   */
+  private int sqlConfMapsDeserialized(byte[] taskBinary) throws IOException {
+    Object sqlConfigs = 
context.getSqlContext().sparkSession().sessionState().conf().getAllConfs();
+    Set<Object> maps = Collections.newSetFromMap(new IdentityHashMap<>());
+    try (ObjectInputStream in = new ObjectInputStream(new 
ByteArrayInputStream(taskBinary)) {
+      {
+        enableResolveObject(true);
+      }
+
+      @Override
+      protected Object resolveObject(Object obj) {
+        if (obj instanceof scala.collection.Map && obj.equals(sqlConfigs)) {
+          maps.add(obj);
+        }
+        return obj;
+      }
+    }) {
+      in.readObject();
+    } catch (ClassNotFoundException e) {
+      throw new IllegalStateException(e);
+    }
+    return maps.size();
+  }
+
+  private static void resolveLineage(RDD<?> rdd, Set<RDD<?>> visited) {
+    if (!visited.add(rdd)) {
+      return;
+    }
+    rdd.partitions();
+    scala.collection.Iterator<?> deps = rdd.dependencies().iterator();
+    while (deps.hasNext()) {
+      resolveLineage(((org.apache.spark.Dependency<?>) deps.next()).rdd(), 
visited);
+    }
+  }
+
+  private static String report(TaskBinary small, TaskBinary large) {
+    int extraGroups = large.groups - small.groups;
+    String rows = small.counts.keySet().stream()
+        .map(payload -> String.format("  %-38s %6d %6d   %+.1f per group", 
payload, small.counts.get(payload),
+            large.counts.get(payload), (large.counts.get(payload) - 
small.counts.get(payload)) / (double) extraGroups))
+        .collect(Collectors.joining("\n"));
+    return String.format("  %-38s %6s %6s%n", "groups", small.groups, 
large.groups)
+        + rows + "\n"
+        + String.format("  %-38s %6d %6d   %+.0f bytes per group", "task 
binary bytes", small.bytes, large.bytes,
+            (large.bytes - small.bytes) / (double) extraGroups);
+  }
+
+  private static final class TaskBinary {
+    final int groups;
+    final long bytes;
+    final Map<String, Integer> counts;
+
+    TaskBinary(int groups, long bytes, Map<String, Integer> counts) {
+      this.groups = groups;
+      this.bytes = bytes;
+      this.counts = counts;
+    }
+  }
+
+  /** Counts each distinct object once, as Java serialization writes it once 
per stream. */
+  private static final class PayloadCountingObjectOutputStream extends 
ObjectOutputStream {
+    private final Map<String, Integer> counts = new LinkedHashMap<>();
+
+    PayloadCountingObjectOutputStream(OutputStream out) throws IOException {
+      super(out);
+      SHARED_PAYLOADS.keySet().forEach(payload -> counts.put(payload, 0));
+      enableReplaceObject(true);
+    }
+
+    @Override
+    protected Object replaceObject(Object obj) {
+      SHARED_PAYLOADS.forEach((payload, matches) -> {
+        if (matches.test(obj)) {
+          counts.merge(payload, 1, Integer::sum);
+        }
+      });
+      return obj;
+    }
+
+    Map<String, Integer> counts() {
+      return new LinkedHashMap<>(counts);
+    }
+  }
+}

Reply via email to