Big +1 also from my side. This will eliminate some work-arounds used so far to bypass the module structure (like code using reflection to extract a JobGraph from a Pipeline).
I agree with Stephan that with proper documentation, release notes and tooling update, it will hopefully not be a big hassle for users to migrate. Also I think it should be done as early in the release as possible, so that we can give it enough exposure and testing. In the past, such deep changes late in the release have led to longer release-testing periods and, eventually, longer release cycles. Cheers, Kostas On Thu, Mar 5, 2020 at 3:35 PM Stephan Ewen <se...@apache.org> wrote: > > +1 to this fix, in general. > > If the main issue is that users have to now add "flink-clients" explicitly, > then I think this is okay, if we spell it out prominently in the release > notes, and make sure quickstarts / etc are updated, and have a good error > message when client/runtime classes are not found. > > On Thu, Mar 5, 2020 at 2:56 PM Aljoscha Krettek <aljos...@apache.org> wrote: > > > Hi, > > > > thanks for starting the discussion, Tison! > > > > I'd like to fix this dependency mess rather sooner than later, but we do > > have to consider the fact that we are breaking the dependency setup of > > users. If they they only had a dependency on flink-streaming-java before > > but used classes from flink-clients they would have to explicitly add > > this dependency now. > > > > Let's see what others think. > > > > Best, > > Aljoscha > > > > On 05.03.20 02:53, tison wrote: > > > Hi devs, > > > > > > Here is a proposal to reverse the dependency from flink-streaming-java to > > > flink-client, for a proper > > > module dependency graph. Since it changes current structure, it should be > > > discussed publicly. > > > > > > The original idea comes from that flink-streaming-java acts as an API > > only > > > module just as what > > > we do in its batch companion flink-java. If a Flink user want to write a > > > minimum DataStream > > > program, the only dependency should be flink-streaming java. > > > > > > However, currently as it is implemented, flink-client and even > > > flink-runtime are transitively polluted > > > in when user depends on flink-streaming-java. These dependencies polluted > > > in as > > > > > > flink-client: > > > - previously, ClusterClient, which is removed by FLIP-73 Executors > > > - accidentally, ProgramInvocationException, we just throw in place as > > it > > > is accessible. > > > - transitively, flink-optimizer, for one utility. > > > - transitively, flink-java, for several utilities. > > > flink-runtime: > > > - mainly for JobGraph generating. > > > > > > With a previous discussion with @Aljoscha Krettek <aljos...@apache.org> > > our > > > goal is briefly making flink-streaming-java > > > an API only module. As a first step we can break the dependency from > > > flink-streaming-java to > > > flink-client[1][2]. > > > > > > With this first step, continuously we factor out common utilities in > > > flink-java to > > > flink-core and eventually eliminate dependencies from streaming to batch; > > > while > > > orthogonally, we factor out job compilation logic into > > > flink-streaming-compiler module and > > > break the dependency to flink-runtime. The final dependency graph will > > be: > > > > > > > > > flink-client -> flink-streaming-compiler -> flink-runtime > > > \-> > > > flink-streaming-java > > > > > > Looking forward to your feedback. Basically whether or not it is in a > > right > > > direction, and if so, > > > how the community integrates this proposal. > > > > > > Best, > > > tison. > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-15090 > > > [2] https://issues.apache.org/jira/browse/FLINK-16427 > > > > >
