kalencaya commented on issue #1669:
URL:
https://github.com/apache/incubator-seatunnel/issues/1669#issuecomment-1096744296
surprised and glad to people have similar ideas with me.
I have been working for integrate flink and seatunnel with web admin and I'm
willing to share my works for you:
* submit seatunnel job. flink cluster support Standalone、YARN、Native
Kubernetes resource providers and Session、Application、Per-Job deployment mode,
so it is a hard work for supporting all of them. Although flink provides
cli、rest、python and scala REPL for job submission and monitor, there are still
absence of Java client for submiting jobs and access flink cluster and jobs. I
have spent much time providing such a Java client and published it to maven
central repository.
* thin seatunnel jar. Seatunnel jar contains all connector plugins and would
distribute to all flink cluster nodes with job. As we finded, flink support
application deployment mode for reducing job jar dependencies network overload.
For this problem, I have concluded two solutions, just same as mentioned before:
* one is `--classpath` parameters. Actually it is great for solving
seatunnel distribution between nodes, people just configures seatunnel
environment in advance on each nodes like jdk or something else, which is easy
and receive huge payback.
* another is `PipelineOptions#JARS` option. `Clifrontend` would add job
jar and libs as `PipelineOptions#JARS` option, then job jar and their
dependencies would be upload to JobManager. Through our Java client, we have
demonstrated the practicability that a thin seatunnel jar containing none
connector plugin and specified connector plugin jars job submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
