rmannibucau opened a new pull request, #9353: URL: https://github.com/apache/hop/pull/9353
### What is the problem being solved? Table Input in Apache Hop reads in a single database session. Large reads keep one CPU core busy while the storage and network still idle, and on the **native Spark** engine the transform currently runs only through the generic single-task handler (source transforms are `repartition(1)`, and `HopMapPartitionsFn` replays the source per Spark partition), so a requested parallel read has always had to be written by hand with copy-number predicates. ### What is being proposed? Four shared partition settings on Table Input metadata and its Options tab: - `jdbc_partition_column` - `jdbc_lower_bound` - `jdbc_upper_bound` - `jdbc_num_partitions` `TableInputPartitioning` mirrors the Apache Spark JDBC data-source partition arithmetic exactly (`JDBCRelation#columnPartition`), so the two engines read the same rows when given the same settings: - **Classic engine** — one disjoint range per transform copy, extra copies read an empty range (no duplicates). - **Native Spark** — a new `SparkJdbcInputHandler` reads through `spark.read.jdbc` with Spark partitions; it falls back to the generic handler for per-row execution, named parameters, row limits, or incoming parameter rows. Stored passwords are decrypted and execution metrics are forwarded to the fallback handler. Incomplete or invalid partition settings now fail fast instead of silently falling back. There is no dedicated partition metadata type key; the four properties join the existing transform metadata, and the existing `<exclude>.yca/**</exclude>` was not part of this change (see note below). ### What has been done - Added the four `@HopMetadataProperty` fields and the new test coverage. - Added `TableInputPartitioning` and wired classic slicing into `TableInput`. - Added `SparkJdbcInputHandler`, the `SparkConst.TABLE_INPUT_PLUGIN_ID` registration, explicit-handler support in `HopPipelineMetaToSparkConverter`, and the needed `dependencies.xml` paths so the Spark plugin can load the transform and JDBC drivers. - Added the four fields to the Table Input GUI Options tab (inside a *JDBC partitioning* group), locale keys for the non-empty bundles, and user-manual documentation. - Split Spark tests into guard unit tests and H2 integration tests (`SparkJdbcInputHandlerIT`), and asserted the new handler registration in the converter test. - Classic-engine and Spark partitioned-read integration tests. ### Verification - `mvn -pl plugins/transforms/tableinput,plugins/engines/spark test` (unit tests) and the H2 integration tests pass on JDK 21. - The classic partitioned-pipeline IT and the Spark H2 IT both pass. - End-to-end: built the Hop 2.21.0-SNAPSHOT client, ran a partitioned H2 pipeline with `hop-run.sh -r spark-smoke` on the native Spark engine (4 rows read), and confirmed the log shows the native JDBC partitioned read. - `apache-rat:check` passes (`.yca/**` was added to the RAT exclusion list; this was an environment concern unrelated to the feature but is included in the diff). Note: this branch was cut from the current `main`; the fork's `main` is behind, so the PR head is this new branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
