hansva commented on code in PR #9593:
URL: https://github.com/apache/hop/pull/9593#discussion_r4644253377
##########
plugins/engines/spark/src/main/java/org/apache/hop/spark/pipeline/HopPipelineMetaToSparkConverter.java:
##########
@@ -91,19 +91,109 @@ public class HopPipelineMetaToSparkConverter {
SparkConst.TABLE_INPUT_PLUGIN_ID);
/**
- * Plugin ids that must not run as partition-local Hop mini-pipelines. Keep
in lockstep with
- * {@link org.apache.hop.spark.engines.SparkPipelineEngine#supports}.
+ * Plugin ids that must not run as partition-local Hop mini-pipelines, with
the reason shown to
+ * the user: they give wrong rows or none on the native Spark engine. Keep
in lockstep with {@link
+ * org.apache.hop.spark.engines.SparkPipelineEngine#supports} and the
"Native Spark=no" mark on
+ * the transform's documentation page.
*/
public static final Map<String, String> HARD_BANNED_PLUGIN_IDS =
- Map.of(
- SparkConst.GROUP_BY_PLUGIN_ID,
- "Group By is not supported on the native Spark engine. Use Memory
Group By (native Spark shuffle) instead, or run on the Local engine.",
- SparkConst.UNIQUE_ROWS_BY_HASH_SET_PLUGIN_ID,
- "Unique Rows By Hashset is not supported on the native Spark engine.
Every partition would keep its own hash set, so duplicates spread over
different partitions would survive. Use Unique Rows (native Spark distinct) or
Memory Group By instead.",
- SparkConst.JOIN_ROWS_PLUGIN_ID,
- "Join Rows is not supported on the native Spark engine. A cartesian
product needs every row of every input in one place, but every partition would
only combine the rows it happens to hold, so combinations would go missing. Add
the same constant field to both inputs and use Merge Join on that field
instead.",
- SparkConst.MASK_FIELDS_PLUGIN_ID,
- "Mask fields is not supported on the native Spark engine. Every
partition keeps its own mapping and sequence, so the same source value would
not stay the same token.");
+ Map.ofEntries(
+ Map.entry(
+ SparkConst.GROUP_BY_PLUGIN_ID,
+ "Group By is not supported on the native Spark engine. Use
Memory Group By (native Spark shuffle) instead, or run on the Local engine."),
+ Map.entry(
+ SparkConst.UNIQUE_ROWS_BY_HASH_SET_PLUGIN_ID,
+ "Unique Rows By Hashset is not supported on the native Spark
engine. Every partition would keep its own hash set, so duplicates spread over
different partitions would survive. Use Unique Rows (native Spark distinct) or
Memory Group By instead."),
+ Map.entry(
+ SparkConst.JOIN_ROWS_PLUGIN_ID,
+ "Join Rows is not supported on the native Spark engine. A
cartesian product needs every row of every input in one place, but every
partition would only combine the rows it happens to hold, so combinations would
go missing. Add the same constant field to both inputs and use Merge Join on
that field instead."),
+ Map.entry(
+ SparkConst.MASK_FIELDS_PLUGIN_ID,
+ "Mask fields is not supported on the native Spark engine. Every
partition keeps its own mapping and sequence, so the same source value would
not stay the same token."),
+ Map.entry(
+ "BlockUntilTransformsFinish",
+ "Block until transforms finish is not supported on the native
Spark engine. It waits for other transforms of the same Hop pipeline, but a
Spark partition only runs the transforms of its own stage, so the transforms it
waits for are never found."),
+ Map.entry(
+ "CSVInput",
Review Comment:
Reverted in 4a9b6d5585: CSV file input is supported on native Spark again,
as on main. It is in `SparkVerifiedTransforms` and its page says `Native
Spark=yes`. The lazy conversion `ClassCastException` is unchanged from main;
converting binary-string values in the Hop → Spark conversion will come later,
in a separate PR.
##########
integration-tests/scripts/run-configurations/spark-native-local.json:
##########
@@ -0,0 +1,14 @@
+{
+ "engineRunConfiguration": {
+ "SparkPipelineEngine": {
+ "sparkMaster": "local[4]",
+ "sparkAppName": "hop-it-spark-native",
+ "sparkConfigs":
"spark.ui.enabled=false\nspark.driver.host=localhost\nspark.sql.shuffle.partitions=4\nspark.default.parallelism=4",
Review Comment:
Agreed, but not in this PR. Turning off partition coalescing in the IT run
configuration, rerunning the baseline, and moving whatever changes off the
verified list will come later, in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]