BenJFan commented on issue #1669: URL: https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1099077819
> surprised and glad to people have similar ideas with me. > > I have been working for integrate flink and seatunnel with web admin and I'm willing to share my works for you: > > * submit seatunnel job. flink cluster support Standalone、YARN、Native Kubernetes resource providers and Session、Application、Per-Job deployment mode, so it is a hard work for supporting all of them. Although flink provides cli、rest、python and scala REPL for job submission and monitor, there are still absence of Java client for submiting jobs and access flink cluster and jobs. I have spent much time providing such a Java client and published it to maven central repository. > * thin seatunnel jar. Seatunnel jar contains all connector plugins and would distribute to all flink cluster nodes with job. As we finded, flink support application deployment mode for reducing job jar dependencies network overload. For this problem, I have concluded two solutions, just same as mentioned before: > > * one is `--classpath` parameters. Actually it is great for solving seatunnel distribution between nodes, people just configures seatunnel environment in advance on each nodes like jdk or something else, which is easy and receive huge payback. > * another is `PipelineOptions#JARS` option. `Clifrontend` would add job jar and libs as `PipelineOptions#JARS` option, then job jar and their dependencies would be upload to JobManager. Through our Java client, we have demonstrated the practicability that a thin seatunnel jar containing none connector plugin and specified connector plugin jars job submission. Thanks guys advise. @kalencaya @tmljob . I had readed flinkx's code, it also use `PipelineOptions#JARS` option. I will learn how make this way work on SeaTunnel. Thanks again. By the way, @kalencaya is the jar will also upload to TaskManager? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
