Hi,
thanks for starting the discussion, Tison!
I'd like to fix this dependency mess rather sooner than later, but we do
have to consider the fact that we are breaking the dependency setup of
users. If they they only had a dependency on flink-streaming-java before
but used classes from flink-clients they would have to explicitly add
this dependency now.
Let's see what others think.
Best,
Aljoscha
On 05.03.20 02:53, tison wrote:
Hi devs,
Here is a proposal to reverse the dependency from flink-streaming-java to
flink-client, for a proper
module dependency graph. Since it changes current structure, it should be
discussed publicly.
The original idea comes from that flink-streaming-java acts as an API only
module just as what
we do in its batch companion flink-java. If a Flink user want to write a
minimum DataStream
program, the only dependency should be flink-streaming java.
However, currently as it is implemented, flink-client and even
flink-runtime are transitively polluted
in when user depends on flink-streaming-java. These dependencies polluted
in as
flink-client:
- previously, ClusterClient, which is removed by FLIP-73 Executors
- accidentally, ProgramInvocationException, we just throw in place as it
is accessible.
- transitively, flink-optimizer, for one utility.
- transitively, flink-java, for several utilities.
flink-runtime:
- mainly for JobGraph generating.
With a previous discussion with @Aljoscha Krettek <aljos...@apache.org> our
goal is briefly making flink-streaming-java
an API only module. As a first step we can break the dependency from
flink-streaming-java to
flink-client[1][2].
With this first step, continuously we factor out common utilities in
flink-java to
flink-core and eventually eliminate dependencies from streaming to batch;
while
orthogonally, we factor out job compilation logic into
flink-streaming-compiler module and
break the dependency to flink-runtime. The final dependency graph will be:
flink-client -> flink-streaming-compiler -> flink-runtime
\->
flink-streaming-java
Looking forward to your feedback. Basically whether or not it is in a right
direction, and if so,
how the community integrates this proposal.
Best,
tison.
[1] https://issues.apache.org/jira/browse/FLINK-15090
[2] https://issues.apache.org/jira/browse/FLINK-16427
