hansva opened a new issue, #9586: URL: https://github.com/apache/hop/issues/9586
### What needs to happen? The native Spark engine runs a transform either natively (a Spark handler) or as Hop's own transform on every partition. For most transforms nobody has written down which of the two happens, or whether the result is right: - `SparkPipelineEngine.supports()` only knows the transforms with an explicit handler (`SUPPORTED`) and the few in `HopPipelineMetaToSparkConverter.HARD_BANNED_PLUGIN_IDS` (`UNSUPPORTED`). Everything else is `UNKNOWN`: the palette's "Design for: Native Spark" shows it, the run dialog and `hop-run` let it run, and some of those transforms lose rows, write a file per partition or fail at runtime. - Every transform page carries an engine line, for example `:page-engines: Hop Engine=yes, Single Threaded=yes, Native Spark=no, ...`. The `Native Spark` values were set by hand and no longer match what the engine does. ### Goal For every transform, the code and the docs agree on one of: | Value | Meaning | |---|---| | `native` | Runs as a Spark handler | | `yes` | Runs as Hop's own transform on the Spark partitions, with the same result as the local engine | | `no` | Doesn't work on native Spark: refused before the run starts, with the reason | ### Work 1. **Baseline.** Run the integration test projects with the native Spark engine, using a run mode of the IT runner that swaps the project's local run configuration for a native Spark one (`local[4]`, several source partitions). Run them per test. Classify every transform the tests use as `native`, `yes` or `no`, with the failing test and the reason for every `no`. 2. **Mark what doesn't work.** Add every `no` transform to `HARD_BANNED_PLUGIN_IDS` with a reason a user can act on, for example "writes one file per partition: use the Local engine" or "needs a run-wide step before the rows: not supported yet". `supports()` already turns that into `UNSUPPORTED`, so the palette filter, the run dialog and `hop-run` (unless `--allow-unsupported`) all refuse it. 3. **Mark what does.** Transforms verified as `yes` go in an explicit list, so `supports()` answers `SUPPORTED` for them instead of `UNKNOWN`. Transforms not covered by any test stay `UNKNOWN`. 4. **Docs.** Set `Native Spark=` on every transform page to the value from step 1, and list the `no` transforms with their reasons on the native Spark engine page. 5. **Keep them in sync.** A unit test in the Spark plugin compares the `Native Spark` value of every transform page with what `supports()` and the handler list say, so code and docs can't drift apart again. ### Issue Priority Priority: 2 ### Issue Component Component: Spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
