hansva commented on code in PR #9593:
URL: https://github.com/apache/hop/pull/9593#discussion_r4644253377


##########
plugins/engines/spark/src/main/java/org/apache/hop/spark/pipeline/HopPipelineMetaToSparkConverter.java:
##########
@@ -91,19 +91,109 @@ public class HopPipelineMetaToSparkConverter {
           SparkConst.TABLE_INPUT_PLUGIN_ID);
 
   /**
-   * Plugin ids that must not run as partition-local Hop mini-pipelines. Keep 
in lockstep with
-   * {@link org.apache.hop.spark.engines.SparkPipelineEngine#supports}.
+   * Plugin ids that must not run as partition-local Hop mini-pipelines, with 
the reason shown to
+   * the user: they give wrong rows or none on the native Spark engine. Keep 
in lockstep with {@link
+   * org.apache.hop.spark.engines.SparkPipelineEngine#supports} and the 
"Native Spark=no" mark on
+   * the transform's documentation page.
    */
   public static final Map<String, String> HARD_BANNED_PLUGIN_IDS =
-      Map.of(
-          SparkConst.GROUP_BY_PLUGIN_ID,
-          "Group By is not supported on the native Spark engine. Use Memory 
Group By (native Spark shuffle) instead, or run on the Local engine.",
-          SparkConst.UNIQUE_ROWS_BY_HASH_SET_PLUGIN_ID,
-          "Unique Rows By Hashset is not supported on the native Spark engine. 
Every partition would keep its own hash set, so duplicates spread over 
different partitions would survive. Use Unique Rows (native Spark distinct) or 
Memory Group By instead.",
-          SparkConst.JOIN_ROWS_PLUGIN_ID,
-          "Join Rows is not supported on the native Spark engine. A cartesian 
product needs every row of every input in one place, but every partition would 
only combine the rows it happens to hold, so combinations would go missing. Add 
the same constant field to both inputs and use Merge Join on that field 
instead.",
-          SparkConst.MASK_FIELDS_PLUGIN_ID,
-          "Mask fields is not supported on the native Spark engine. Every 
partition keeps its own mapping and sequence, so the same source value would 
not stay the same token.");
+      Map.ofEntries(
+          Map.entry(
+              SparkConst.GROUP_BY_PLUGIN_ID,
+              "Group By is not supported on the native Spark engine. Use 
Memory Group By (native Spark shuffle) instead, or run on the Local engine."),
+          Map.entry(
+              SparkConst.UNIQUE_ROWS_BY_HASH_SET_PLUGIN_ID,
+              "Unique Rows By Hashset is not supported on the native Spark 
engine. Every partition would keep its own hash set, so duplicates spread over 
different partitions would survive. Use Unique Rows (native Spark distinct) or 
Memory Group By instead."),
+          Map.entry(
+              SparkConst.JOIN_ROWS_PLUGIN_ID,
+              "Join Rows is not supported on the native Spark engine. A 
cartesian product needs every row of every input in one place, but every 
partition would only combine the rows it happens to hold, so combinations would 
go missing. Add the same constant field to both inputs and use Merge Join on 
that field instead."),
+          Map.entry(
+              SparkConst.MASK_FIELDS_PLUGIN_ID,
+              "Mask fields is not supported on the native Spark engine. Every 
partition keeps its own mapping and sequence, so the same source value would 
not stay the same token."),
+          Map.entry(
+              "BlockUntilTransformsFinish",
+              "Block until transforms finish is not supported on the native 
Spark engine. It waits for other transforms of the same Hop pipeline, but a 
Spark partition only runs the transforms of its own stage, so the transforms it 
waits for are never found."),
+          Map.entry(
+              "CSVInput",

Review Comment:
   Reverted in 4a9b6d5585: CSV file input is supported on native Spark again, 
as on main. It is in `SparkVerifiedTransforms` and its page says `Native 
Spark=yes`. The lazy conversion `ClassCastException` is unchanged from main; 
converting binary-string values in the Hop → Spark conversion will come later, 
in a separate PR.



##########
integration-tests/scripts/run-configurations/spark-native-local.json:
##########
@@ -0,0 +1,14 @@
+{
+  "engineRunConfiguration": {
+    "SparkPipelineEngine": {
+      "sparkMaster": "local[4]",
+      "sparkAppName": "hop-it-spark-native",
+      "sparkConfigs": 
"spark.ui.enabled=false\nspark.driver.host=localhost\nspark.sql.shuffle.partitions=4\nspark.default.parallelism=4",

Review Comment:
   Agreed, but not in this PR. Turning off partition coalescing in the IT run 
configuration, rerunning the baseline, and moving whatever changes off the 
verified list will come later, in a separate PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to