bamaer commented on code in PR #9593:
URL: https://github.com/apache/hop/pull/9593#discussion_r4643434956
##########
plugins/engines/spark/src/main/java/org/apache/hop/spark/pipeline/HopPipelineMetaToSparkConverter.java:
##########
@@ -91,19 +91,109 @@ public class HopPipelineMetaToSparkConverter {
SparkConst.TABLE_INPUT_PLUGIN_ID);
/**
- * Plugin ids that must not run as partition-local Hop mini-pipelines. Keep
in lockstep with
- * {@link org.apache.hop.spark.engines.SparkPipelineEngine#supports}.
+ * Plugin ids that must not run as partition-local Hop mini-pipelines, with
the reason shown to
+ * the user: they give wrong rows or none on the native Spark engine. Keep
in lockstep with {@link
+ * org.apache.hop.spark.engines.SparkPipelineEngine#supports} and the
"Native Spark=no" mark on
+ * the transform's documentation page.
*/
public static final Map<String, String> HARD_BANNED_PLUGIN_IDS =
- Map.of(
- SparkConst.GROUP_BY_PLUGIN_ID,
- "Group By is not supported on the native Spark engine. Use Memory
Group By (native Spark shuffle) instead, or run on the Local engine.",
- SparkConst.UNIQUE_ROWS_BY_HASH_SET_PLUGIN_ID,
- "Unique Rows By Hashset is not supported on the native Spark engine.
Every partition would keep its own hash set, so duplicates spread over
different partitions would survive. Use Unique Rows (native Spark distinct) or
Memory Group By instead.",
- SparkConst.JOIN_ROWS_PLUGIN_ID,
- "Join Rows is not supported on the native Spark engine. A cartesian
product needs every row of every input in one place, but every partition would
only combine the rows it happens to hold, so combinations would go missing. Add
the same constant field to both inputs and use Merge Join on that field
instead.",
- SparkConst.MASK_FIELDS_PLUGIN_ID,
- "Mask fields is not supported on the native Spark engine. Every
partition keeps its own mapping and sequence, so the same source value would
not stay the same token.");
+ Map.ofEntries(
+ Map.entry(
+ SparkConst.GROUP_BY_PLUGIN_ID,
+ "Group By is not supported on the native Spark engine. Use
Memory Group By (native Spark shuffle) instead, or run on the Local engine."),
+ Map.entry(
+ SparkConst.UNIQUE_ROWS_BY_HASH_SET_PLUGIN_ID,
+ "Unique Rows By Hashset is not supported on the native Spark
engine. Every partition would keep its own hash set, so duplicates spread over
different partitions would survive. Use Unique Rows (native Spark distinct) or
Memory Group By instead."),
+ Map.entry(
+ SparkConst.JOIN_ROWS_PLUGIN_ID,
+ "Join Rows is not supported on the native Spark engine. A
cartesian product needs every row of every input in one place, but every
partition would only combine the rows it happens to hold, so combinations would
go missing. Add the same constant field to both inputs and use Merge Join on
that field instead."),
+ Map.entry(
+ SparkConst.MASK_FIELDS_PLUGIN_ID,
+ "Mask fields is not supported on the native Spark engine. Every
partition keeps its own mapping and sequence, so the same source value would
not stay the same token."),
+ Map.entry(
+ "BlockUntilTransformsFinish",
+ "Block until transforms finish is not supported on the native
Spark engine. It waits for other transforms of the same Hop pipeline, but a
Spark partition only runs the transforms of its own stage, so the transforms it
waits for are never found."),
+ Map.entry(
+ "CSVInput",
Review Comment:
On main, CSV file input with lazy conversion off gives the same rows on
native Spark as on the local engine (1000-row file → Memory Group By: 1000
rows, sum of id 500500, 60 distinct states, on local and on `local[4]`). Its
page says `Native Spark=yes` on main, so this refusal stops pipelines that work
today.
With lazy conversion on, it fails loudly instead of giving wrong rows:
`ClassCastException: [B cannot be cast to java.lang.Long` in
`HopSparkRowConverter.toSparkValue`. The value still has binary-string storage
when it is handed to Spark.
Suggested fix: keep CSV file input supported, and convert binary-string
values to normal storage in the Hop → Spark conversion, using the row meta the
transform actually produced. If file names from a field really give wrong rows,
refuse only that case.
##########
integration-tests/scripts/run-configurations/spark-native-local.json:
##########
@@ -0,0 +1,14 @@
+{
+ "engineRunConfiguration": {
+ "SparkPipelineEngine": {
+ "sparkMaster": "local[4]",
+ "sparkAppName": "hop-it-spark-native",
+ "sparkConfigs":
"spark.ui.enabled=false\nspark.driver.host=localhost\nspark.sql.shuffle.partitions=4\nspark.default.parallelism=4",
Review Comment:
On IT-sized data, Spark's adaptive execution merges the 4 shuffle partitions
back into one. Generic sources already run as one partition (`repartition(1)`),
so the baseline never ran a generic transform on more than one partition. Data
grid → Add sequence → Sort rows (100 rows) → transform → Memory Group By:
| Transform after Sort rows | Local | This run config | Plus
`spark.sql.adaptive.coalescePartitions.enabled=false` |
|---|---|---|---|
| Add sequence | 1..100 | 1..100 | max 25, 25 distinct |
| Sample rows `1..10` | 10 rows | 10 rows | 40 rows |
With real data, a Spark file input source or a native shuffle gives several
partitions, and Add sequence produces duplicate values. That's the reason the
PR gives for refusing Add value fields changing sequence. Both are in
`SparkVerifiedTransforms` (`Sequence` L152, `SampleRows` L148;
`ReservoirSampling` L141 most likely as well).
Suggested fix: add that setting here, rerun the baseline, and move whatever
changes out of the verified list. The comment in `run-tests.sh` (L427, "every
source spread over 4 partitions") doesn't match what runs either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]