bamaer commented on code in PR #9593:
URL: https://github.com/apache/hop/pull/9593#discussion_r4643434956


##########
plugins/engines/spark/src/main/java/org/apache/hop/spark/pipeline/HopPipelineMetaToSparkConverter.java:
##########
@@ -91,19 +91,109 @@ public class HopPipelineMetaToSparkConverter {
           SparkConst.TABLE_INPUT_PLUGIN_ID);
 
   /**
-   * Plugin ids that must not run as partition-local Hop mini-pipelines. Keep 
in lockstep with
-   * {@link org.apache.hop.spark.engines.SparkPipelineEngine#supports}.
+   * Plugin ids that must not run as partition-local Hop mini-pipelines, with 
the reason shown to
+   * the user: they give wrong rows or none on the native Spark engine. Keep 
in lockstep with {@link
+   * org.apache.hop.spark.engines.SparkPipelineEngine#supports} and the 
"Native Spark=no" mark on
+   * the transform's documentation page.
    */
   public static final Map<String, String> HARD_BANNED_PLUGIN_IDS =
-      Map.of(
-          SparkConst.GROUP_BY_PLUGIN_ID,
-          "Group By is not supported on the native Spark engine. Use Memory 
Group By (native Spark shuffle) instead, or run on the Local engine.",
-          SparkConst.UNIQUE_ROWS_BY_HASH_SET_PLUGIN_ID,
-          "Unique Rows By Hashset is not supported on the native Spark engine. 
Every partition would keep its own hash set, so duplicates spread over 
different partitions would survive. Use Unique Rows (native Spark distinct) or 
Memory Group By instead.",
-          SparkConst.JOIN_ROWS_PLUGIN_ID,
-          "Join Rows is not supported on the native Spark engine. A cartesian 
product needs every row of every input in one place, but every partition would 
only combine the rows it happens to hold, so combinations would go missing. Add 
the same constant field to both inputs and use Merge Join on that field 
instead.",
-          SparkConst.MASK_FIELDS_PLUGIN_ID,
-          "Mask fields is not supported on the native Spark engine. Every 
partition keeps its own mapping and sequence, so the same source value would 
not stay the same token.");
+      Map.ofEntries(
+          Map.entry(
+              SparkConst.GROUP_BY_PLUGIN_ID,
+              "Group By is not supported on the native Spark engine. Use 
Memory Group By (native Spark shuffle) instead, or run on the Local engine."),
+          Map.entry(
+              SparkConst.UNIQUE_ROWS_BY_HASH_SET_PLUGIN_ID,
+              "Unique Rows By Hashset is not supported on the native Spark 
engine. Every partition would keep its own hash set, so duplicates spread over 
different partitions would survive. Use Unique Rows (native Spark distinct) or 
Memory Group By instead."),
+          Map.entry(
+              SparkConst.JOIN_ROWS_PLUGIN_ID,
+              "Join Rows is not supported on the native Spark engine. A 
cartesian product needs every row of every input in one place, but every 
partition would only combine the rows it happens to hold, so combinations would 
go missing. Add the same constant field to both inputs and use Merge Join on 
that field instead."),
+          Map.entry(
+              SparkConst.MASK_FIELDS_PLUGIN_ID,
+              "Mask fields is not supported on the native Spark engine. Every 
partition keeps its own mapping and sequence, so the same source value would 
not stay the same token."),
+          Map.entry(
+              "BlockUntilTransformsFinish",
+              "Block until transforms finish is not supported on the native 
Spark engine. It waits for other transforms of the same Hop pipeline, but a 
Spark partition only runs the transforms of its own stage, so the transforms it 
waits for are never found."),
+          Map.entry(
+              "CSVInput",

Review Comment:
   On main, CSV file input with lazy conversion off gives the same rows on 
native Spark as on the local engine (1000-row file → Memory Group By: 1000 
rows, sum of id 500500, 60 distinct states, on local and on `local[4]`). Its 
page says `Native Spark=yes` on main, so this refusal stops pipelines that work 
today.
   
   With lazy conversion on, it fails loudly instead of giving wrong rows: 
`ClassCastException: [B cannot be cast to java.lang.Long` in 
`HopSparkRowConverter.toSparkValue`. The value still has binary-string storage 
when it is handed to Spark.
   
   Suggested fix: keep CSV file input supported, and convert binary-string 
values to normal storage in the Hop → Spark conversion, using the row meta the 
transform actually produced. If file names from a field really give wrong rows, 
refuse only that case.



##########
integration-tests/scripts/run-configurations/spark-native-local.json:
##########
@@ -0,0 +1,14 @@
+{
+  "engineRunConfiguration": {
+    "SparkPipelineEngine": {
+      "sparkMaster": "local[4]",
+      "sparkAppName": "hop-it-spark-native",
+      "sparkConfigs": 
"spark.ui.enabled=false\nspark.driver.host=localhost\nspark.sql.shuffle.partitions=4\nspark.default.parallelism=4",

Review Comment:
   On IT-sized data, Spark's adaptive execution merges the 4 shuffle partitions 
back into one. Generic sources already run as one partition (`repartition(1)`), 
so the baseline never ran a generic transform on more than one partition. Data 
grid → Add sequence → Sort rows (100 rows) → transform → Memory Group By:
   
   | Transform after Sort rows | Local | This run config | Plus 
`spark.sql.adaptive.coalescePartitions.enabled=false` |
   |---|---|---|---|
   | Add sequence | 1..100 | 1..100 | max 25, 25 distinct |
   | Sample rows `1..10` | 10 rows | 10 rows | 40 rows |
   
   With real data, a Spark file input source or a native shuffle gives several 
partitions, and Add sequence produces duplicate values. That's the reason the 
PR gives for refusing Add value fields changing sequence. Both are in 
`SparkVerifiedTransforms` (`Sequence` L152, `SampleRows` L148; 
`ReservoirSampling` L141 most likely as well).
   
   Suggested fix: add that setting here, rerun the baseline, and move whatever 
changes out of the verified list. The comment in `run-tests.sh` (L427, "every 
source spread over 4 partitions") doesn't match what runs either.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to