danny0405 commented on code in PR #20533:
URL: https://github.com/apache/hudi/pull/20533#discussion_r4488416352


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -97,10 +102,58 @@
 public abstract class MultipleSparkJobExecutionStrategy<T>
     extends ClusteringExecutionStrategy<T, HoodieData<HoodieRecord<T>>, 
HoodieData<HoodieKey>, HoodieData<WriteStatus>> {
 
+  /*
+   * Clustering builds one RDD per group and unions them into a single write 
stage, so Spark ships the lineage of
+   * every group to every task of that stage. Java serialization writes an 
instance once per stream however many
+   * closures reference it, but writes each distinct instance in full: an 
object built per group is deserialized
+   * once per group by every task, and held by every running task. Whatever 
does not depend on the group (table
+   * schema, reader context factory, reader properties, write config, writer 
schema) is therefore built once per run
+   * and shared by all groups. The state below is driver-side only.
+   */
+  private transient HoodieWriteConfig groupWriteConfigSource;
+  private transient Map<Option<Long>, HoodieWriteConfig> groupWriteConfigs;
+  private transient Map<Pair<StructType, String>, StructType> 
groupWriterSchemas;

Review Comment:
   Could we consolidate the shared reader state and these caches into a 
transient per-run state holder, initialized in `performClustering` before 
submitting the group tasks? This would make the lifetime explicit, remove the 
source-config identity check, and let the existing protected hooks access 
shared state without changing their signatures. Keep the writer-schema cache 
keyed by dataset schema and table schema to handle groups with different 
schemas.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to