CarlosJuncher03 opened a new issue, #7992: URL: https://github.com/apache/hop/issues/7992
### What would you like to happen? Currently, the Remote Pipeline Engine allows executing an entire pipeline on a Hop Server, while the Pipeline Executor allows calling a secondary pipeline using a specific Run Configuration. However, there is no simple way to use multiple Hop Servers as a pool of workers directly within the data flow. It would be useful to allow a pipeline to send groups of rows to a remote pipeline—executed by one of the available Hop Servers—and receive the processed rows back to continue the original flow. Conceptually, this could be represented by a transform similar to a **Remote Mapping** or **Distributed Pipeline Executor**: 1. The transform receives rows from the main pipeline. 2. Rows are grouped by quantity, size, field, or time interval. 3. Each batch is sent to an available Hop Server. 4. The server executes a secondary pipeline on that batch. 5. The resulting rows return to the transform. 6. The main pipeline continues processing normally. There could also be a metadata object similar to a **Hop Server Pool**, containing multiple Hop Servers or an address representing a dynamic set of workers. This model would allow the use of different types of infrastructure. In a simple environment, the pool could contain manually registered servers. In Kubernetes, it could point to a service with multiple Hop Server replicas, allowing capacity to scale up or down without modifying the pipeline. Important features for this type of processing could include: * Distribution via round-robin, by key, or to the least busy server. * Limits on rows, bytes, or time per batch. * Maximum number of simultaneous batches. * Backpressure control when all workers are busy. * Retry or redirection logic upon server failure. * Definition of behavior regarding row ordering. * Consolidated metrics for remote executions. * Identification of the server that processed each batch. * Error handling integrated with the pipeline's error hops. * Ability to select the server or Run Configuration on a per-transform basis. The proposal does not aim to automatically distribute every transform or create a comprehensive new distributed processing engine. Remote processing would be applied only at points explicitly defined by the user. This would allow the Hop Server to function as a worker layer for specific—and potentially resource-intensive—operations, while preserving the pipeline's visual simplicity: `Local pipeline → distributed remote processing → local continuation` This approach would also create a middle-ground alternative between local execution and full-scale distributed engines like Beam or Spark. Native Spark demonstrates that specialized distributed execution paths can exist without necessarily relying on Beam; this proposal would apply that same conceptual flexibility to native remote processing between pipelines and Hop Servers. The primary benefit would be making the Hop Server more useful as a processing resource within the pipelines themselves, enabling the creation of hybrid, distributed workflows without requiring the entire solution to be migrated to a different execution engine. Note: I realize this might be challenging, but it’s just an idea—I believe there could be simpler ways to achieve horizontal pipeline parallelization in Hop. My point is that, currently, using Beam to run pipelines in parallel is quite complex, and I think a simpler approach could be implemented using the Hop engine itself. ### Issue Priority Priority: 3 ### Issue Component Component: Infrastructure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
