det101 opened a new issue, #18320: URL: https://github.com/apache/dolphinscheduler/issues/18320
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement. ### Description Currently, enterprise users are increasingly adopting a "Batch-Stream Integration" architecture. In this landscape, **Apache StreamPark** has emerged as a leading Stream DevOps platform, specializing in Flink/Spark application management, Git/Maven-based automated compilation, and advanced cloud-native deployment (`Flink on Kubernetes Application/Session`). While DolphinScheduler currently provides excellent support for batch processing and contains a `Dinky` plugin (which focuses on Flink SQL IDE development), it still lacks a native, lifecycle-level task node to orchestrate and control **existing complex stream applications (especially JAR-based custom jobs on K8s)** managed by StreamPark. To bridge the gap between workflow orchestration and stream operations, we propose adding a native **StreamPark Task Plugin** and an **StreamPark DataSource**. ### Use case 1. **Batch-Stream Mixed Orchestration**: Automatically trigger a StreamPark-managed Flink job to restart, resume, or perform data backfilling immediately after a DolphinScheduler offline batch workflow finishes (e.g., ClickHouse/Hive aggregation). 2. **End-to-End Automated Stream Operations (DataOps)**: Fully automate the CI/CD and upgrade pipeline of streaming jobs without any external custom Shell scripts. Users can orchestrate a seamless workflow completely using StreamPark nodes: `StreamPark Node (STOP with Savepoint)` -> `StreamPark Node (BUILD/Recompile from Git & rebuild K8s Image)` -> `StreamPark Node (START/Resume from the newly created Savepoint)`. 3. **Secure Credential Management**: Centrally manage StreamPark's OpenAPI endpoints and Access Tokens in the DolphinScheduler DataSource Center, preventing developers from hardcoding credentials in raw HTTP or Shell scripts. ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
