nsivabalan commented on code in PR #20533:
URL: https://github.com/apache/hudi/pull/20533#discussion_r4533082744
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/SQLConfInjectingRDD.scala:
##########
@@ -58,4 +58,30 @@ class SQLConfInjectingRDD[T: ClassTag](var sqlRDD: RDD[T],
@transient conf: SQLC
firstParent[T].iterator(split, context)
}
}
-}
\ No newline at end of file
+}
+
+object SQLConfInjectingRDD {
+
+ /** The last config map captured; driver-side only, guarded by this object.
See [[sharedConfigs]]. */
+ private var lastConfigs: Map[String, String] = Map.empty
Review Comment:
Minor: this keeps the last SQL conf map on the driver for the life of the
JVM, past the session that produced it, and session confs can carry
credentials. A `WeakReference`, or a sentence saying the retention is
intentional, would cover it.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -137,10 +184,67 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>>
performClustering(final Hood
writeMetadata.setWriteStatuses(HoodieJavaRDD.of(writeStatusRDD));
return writeMetadata;
} finally {
+ // Every group has been submitted and joined: the groups' RDDs hold what
they need from the run invariants.
Review Comment:
nit: on the failure path `join()` throws, so not every group has necessarily
been joined when this runs. Harmless (a late group falls back to building its
own config), but the comment could say so.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -137,10 +184,67 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>>
performClustering(final Hood
writeMetadata.setWriteStatuses(HoodieJavaRDD.of(writeStatusRDD));
return writeMetadata;
} finally {
+ // Every group has been submitted and joined: the groups' RDDs hold what
they need from the run invariants.
+ runInvariants = null;
clusteringExecutorService.shutdown();
}
}
+ /**
+ * What every group of one {@link #performClustering} run shares (see the
class comment), fixed for the run: built on
+ * the driver before the groups are submitted, dropped when they have all
been submitted. The caches only memoize:
+ * each derives its value from the key and the run's write config, so a
key's value never changes within the run.
+ * Never serialized: groups capture its values, not the holder.
+ */
+ private static final class ClusteringRunInvariants {
+ /** The schema the groups' records have, with meta fields, as handed to
performClusteringWithRecords*. */
+ private final HoodieSchema readerSchema;
+ private final ReaderContextFactory<?> readerContextFactory;
+ /** Row writer only: {@link #readerSchema} as Spark types. */
+ private final StructType sparkReaderSchema;
+ /** Row writer only: the file group readers' properties. */
+ private final TypedProperties readerProperties;
+ /** Groups run concurrently, on the clustering thread pool. */
+ private final ConcurrentMap<Option<Long>, HoodieWriteConfig>
groupWriteConfigs = new ConcurrentHashMap<>();
+ /** Keyed by dataset schema and table schema: groups may read different
schemas. */
+ private final ConcurrentMap<Pair<StructType, String>, StructType>
groupWriterSchemas = new ConcurrentHashMap<>();
+
+ private ClusteringRunInvariants(HoodieSchema readerSchema,
ReaderContextFactory<?> readerContextFactory,
+ StructType sparkReaderSchema, TypedProperties
readerProperties) {
+ this.readerSchema = readerSchema;
+ this.readerContextFactory = readerContextFactory;
+ this.sparkReaderSchema = sparkReaderSchema;
+ this.readerProperties = readerProperties;
+ }
+
+ @SuppressWarnings("unchecked")
+ private <R> ReaderContextFactory<R> readerContextFactory() {
+ return (ReaderContextFactory<R>) readerContextFactory;
+ }
+ }
+
+ /** The invariants of the {@link #performClustering} run that is submitting
groups. */
+ private ClusteringRunInvariants currentRunInvariants() {
+ ClusteringRunInvariants run = runInvariants;
+ ValidationUtils.checkState(run != null, "Clustering groups are submitted
by performClustering");
Review Comment:
A custom strategy that overrides `performClustering` and calls the protected
`runClusteringForGroupAsync` / `runClusteringForGroupAsyncAsRow` hooks itself
worked before this PR and now fails here with `IllegalStateException`. No
in-repo subclass does that, but these hooks are protected.
`getGroupWriteConfig` / `getGroupWriterSchema` already fall back to building
fresh state when the holder is null. Could the hooks do the same, e.g. build
the invariants for that call when `runInvariants` is null? Such callers keep
the old per-group cost but don't break.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/clustering/run/strategy/TestClusteringTaskBinaryPayloads.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteClientTestUtils;
+import org.apache.hudi.client.WriteStatus;
+import
org.apache.hudi.client.clustering.plan.strategy.SparkSingleFileSortPlanStrategy;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.schema.HoodieSchema;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.table.marker.MarkerType;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.table.HoodieSparkTable;
+import org.apache.hudi.testutils.HoodieClientTestBase;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.rdd.RDD;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.OutputStream;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import static
org.apache.hudi.common.testutils.HoodieTestUtils.INSTANT_GENERATOR;
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Measures what the write stage of {@link
MultipleSparkJobExecutionStrategy#performClustering} ships to every task.
+ *
+ * <p>Clustering builds one RDD per group and unions them into a single write
stage. Spark serializes the stage
+ * lineage into one task binary that every task deserializes, so an object a
group's lineage captures is shipped
+ * to, and deserialized by, every task of the stage, for every group. Objects
that do not depend on the group
+ * (table schema, write config, reader properties, reader context factory, SQL
configs) should therefore be shared
+ * instances: Java serialization writes a shared instance once per stream.
+ *
+ * <p>The test runs the real strategy on a small and a large subset of one
plan, serializes the resulting lineage
+ * the way Spark's closure serializer does (one {@link ObjectOutputStream}),
and counts the distinct instances of
+ * each such payload. The counts must not grow with the number of groups.
+ */
+@Tag("functional")
+public class TestClusteringTaskBinaryPayloads extends HoodieClientTestBase {
+
+ private static final int SMALL_PLAN_GROUPS = 2;
+ private static final int LARGE_PLAN_GROUPS = 8;
+ // dataGen spreads inserts over 3 partitions; with small-file handling off,
each batch adds one file group
+ // per partition, and the single-file plan strategy emits one clustering
group per file group.
+ private static final int PARTITIONS = 3;
+ private static final int INSERT_BATCHES = (LARGE_PLAN_GROUPS + PARTITIONS -
1) / PARTITIONS;
+ private static final int RECORDS_PER_BATCH = 30;
+ private static final String SQL_CONF_MAPS = "SQL config maps";
+
+ /** Payloads that do not depend on the group, so their count must not depend
on the number of groups. */
+ private static final Map<String, Predicate<Object>> SHARED_PAYLOADS = new
LinkedHashMap<>();
+
+ static {
+ SHARED_PAYLOADS.put("HoodieSchema", HoodieSchema.class::isInstance);
+ SHARED_PAYLOADS.put("HoodieWriteConfig",
HoodieWriteConfig.class::isInstance);
+ SHARED_PAYLOADS.put("TypedProperties", TypedProperties.class::isInstance);
+ SHARED_PAYLOADS.put("ReaderContextFactory",
ReaderContextFactory.class::isInstance);
+ SHARED_PAYLOADS.put("StructType", StructType.class::isInstance);
+ }
+
+ @Test
+ public void taskBinaryPayloadsDoNotGrowWithClusteringGroups() throws
Exception {
Review Comment:
This covers the row-writer `SparkSortAndSizeExecutionStrategy` path only.
The records path (`hoodie.datasource.write.row.writer.enable=false`) has its
own invariants (`newRecordsRunInvariants`) and shared config; could we
parameterize over the row writer on/off? Running
`SparkSingleFileSortExecutionStrategy` once would also cover the other
`parquetMaxFileSize` key of the config cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]