hansva opened a new issue, #9586:
URL: https://github.com/apache/hop/issues/9586

   ### What needs to happen?
   
   The native Spark engine runs a transform either natively (a Spark handler) 
or as Hop's own transform on every partition. For most transforms nobody has 
written down which of the two happens, or whether the result is right:
   
   - `SparkPipelineEngine.supports()` only knows the transforms with an 
explicit handler (`SUPPORTED`) and the few in 
`HopPipelineMetaToSparkConverter.HARD_BANNED_PLUGIN_IDS` (`UNSUPPORTED`). 
Everything else is `UNKNOWN`: the palette's "Design for: Native Spark" shows 
it, the run dialog and `hop-run` let it run, and some of those transforms lose 
rows, write a file per partition or fail at runtime.
   - Every transform page carries an engine line, for example `:page-engines: 
Hop Engine=yes, Single Threaded=yes, Native Spark=no, ...`. The `Native Spark` 
values were set by hand and no longer match what the engine does.
   
   ### Goal
   
   For every transform, the code and the docs agree on one of:
   
   | Value | Meaning |
   |---|---|
   | `native` | Runs as a Spark handler |
   | `yes` | Runs as Hop's own transform on the Spark partitions, with the same 
result as the local engine |
   | `no` | Doesn't work on native Spark: refused before the run starts, with 
the reason |
   
   ### Work
   
   1. **Baseline.** Run the integration test projects with the native Spark 
engine, using a run mode of the IT runner that swaps the project's local run 
configuration for a native Spark one (`local[4]`, several source partitions). 
Run them per test. Classify every transform the tests use as `native`, `yes` or 
`no`, with the failing test and the reason for every `no`.
   2. **Mark what doesn't work.** Add every `no` transform to 
`HARD_BANNED_PLUGIN_IDS` with a reason a user can act on, for example "writes 
one file per partition: use the Local engine" or "needs a run-wide step before 
the rows: not supported yet". `supports()` already turns that into 
`UNSUPPORTED`, so the palette filter, the run dialog and `hop-run` (unless 
`--allow-unsupported`) all refuse it.
   3. **Mark what does.** Transforms verified as `yes` go in an explicit list, 
so `supports()` answers `SUPPORTED` for them instead of `UNKNOWN`. Transforms 
not covered by any test stay `UNKNOWN`.
   4. **Docs.** Set `Native Spark=` on every transform page to the value from 
step 1, and list the `no` transforms with their reasons on the native Spark 
engine page.
   5. **Keep them in sync.** A unit test in the Spark plugin compares the 
`Native Spark` value of every transform page with what `supports()` and the 
handler list say, so code and docs can't drift apart again.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Spark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to