rmannibucau opened a new pull request, #9353:
URL: https://github.com/apache/hop/pull/9353

   ### What is the problem being solved?
   
   Table Input in Apache Hop reads in a single database session. Large reads 
keep one CPU core busy while the storage and network still idle, and on the 
**native Spark** engine the transform currently runs only through the generic 
single-task handler (source transforms are `repartition(1)`, and 
`HopMapPartitionsFn` replays the source per Spark partition), so a requested 
parallel read has always had to be written by hand with copy-number predicates.
   
   ### What is being proposed?
   
   Four shared partition settings on Table Input metadata and its Options tab:
   - `jdbc_partition_column`
   - `jdbc_lower_bound`
   - `jdbc_upper_bound`
   - `jdbc_num_partitions`
   
   `TableInputPartitioning` mirrors the Apache Spark JDBC data-source partition 
arithmetic exactly (`JDBCRelation#columnPartition`), so the two engines read 
the same rows when given the same settings:
   - **Classic engine** — one disjoint range per transform copy, extra copies 
read an empty range (no duplicates).
   - **Native Spark** — a new `SparkJdbcInputHandler` reads through 
`spark.read.jdbc` with Spark partitions; it falls back to the generic handler 
for per-row execution, named parameters, row limits, or incoming parameter 
rows. Stored passwords are decrypted and execution metrics are forwarded to the 
fallback handler.
   
   Incomplete or invalid partition settings now fail fast instead of silently 
falling back. There is no dedicated partition metadata type key; the four 
properties join the existing transform metadata, and the existing 
`<exclude>.yca/**</exclude>` was not part of this change (see note below).
   
   ### What has been done
   
   - Added the four `@HopMetadataProperty` fields and the new test coverage.
   - Added `TableInputPartitioning` and wired classic slicing into `TableInput`.
   - Added `SparkJdbcInputHandler`, the `SparkConst.TABLE_INPUT_PLUGIN_ID` 
registration, explicit-handler support in `HopPipelineMetaToSparkConverter`, 
and the needed `dependencies.xml` paths so the Spark plugin can load the 
transform and JDBC drivers.
   - Added the four fields to the Table Input GUI Options tab (inside a *JDBC 
partitioning* group), locale keys for the non-empty bundles, and user-manual 
documentation.
   - Split Spark tests into guard unit tests and H2 integration tests 
(`SparkJdbcInputHandlerIT`), and asserted the new handler registration in the 
converter test.
   - Classic-engine and Spark partitioned-read integration tests.
   
   ### Verification
   
   - `mvn -pl plugins/transforms/tableinput,plugins/engines/spark test` (unit 
tests) and the H2 integration tests pass on JDK 21.
   - The classic partitioned-pipeline IT and the Spark H2 IT both pass.
   - End-to-end: built the Hop 2.21.0-SNAPSHOT client, ran a partitioned H2 
pipeline with `hop-run.sh -r spark-smoke` on the native Spark engine (4 rows 
read), and confirmed the log shows the native JDBC partitioned read.
   - `apache-rat:check` passes (`.yca/**` was added to the RAT exclusion list; 
this was an environment concern unrelated to the feature but is included in the 
diff).
   
   Note: this branch was cut from the current `main`; the fork's `main` is 
behind, so the PR head is this new branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to