voonhous commented on code in PR #20545:
URL: https://github.com/apache/hudi/pull/20545#discussion_r4534780625


##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieEngineContext.java:
##########
@@ -82,6 +82,27 @@ public <T> HoodieData<T> parallelize(List<T> data) {
 
   public abstract <T> HoodieData<T> parallelize(List<T> data, int parallelism);
 
+  /**
+   * Unions all the given {@link HoodieData} into a single {@link HoodieData}.
+   *
+   * <p>The default implementation folds the inputs with {@link 
HoodieData#union(HoodieData)}, which
+   * does not nest when {@link #emptyHoodieData()} is eager, since every step 
materializes its result.
+   * Engines whose union nests per call should override this with a single 
flat union: on Spark, a
+   * chain of a few hundred {@code UnionRDD} levels is enough to overflow the 
stack during task
+   * serialization.
+   *
+   * @param dataList the {@link HoodieData} to union
+   * @param <T>      type of object
+   * @return the union of all the inputs, or {@link #emptyHoodieData()} if 
there is none
+   */
+  public <T> HoodieData<T> union(List<HoodieData<T>> dataList) {

Review Comment:
   Closed #17591 as superseded: it edits the pre-#18348 init path and conflicts 
with master. #20545 credits it in the description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to