First of all I don't have much(if not at all) experience with working with a multi repository project of Flink's size. I would like to mention a few thoughts of mine, though. In general I am slightly against splitting the repository. I fear that what we actually want to do is to introduce double standards for different modules with the repository split.
As I understand there are two issues we want to solve with the split: 1) long build/testing time 2) increasing number of PRs Ad. 1 I agree this is a problem and that we don't necessarily need to run all the tests with every change or build the whole project all the time. However, I think we could achieve that in a single repository and at the same time keep the option to build all modules at once. If I am not mistaken this the approach that Apache Beam community decided to take (see e.g. https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PreCommit_Java.groovy where they define paths to file that if changed trigger the corresponding CI job). Maybe we could make it easier if we restructure the repository? To something like: flink/ |--flink-main/ |--flink-core/ |--flink-runtime/ ... |--flink-connectors/ ... |--flink-filesystems.../ ... |--root.pom In my opinion the Releases section from Chesnay's message shows well that it might not be the best option to split the repository. The option a) looks for me equivalent to what I suggested above but with a split. The option b) looks for me super complicated and I can see no benefit over option a). The option c) would be the most reasonable one if we decided to split the repository, if you ask me. The problem with this approach is the compatibility matrix (which versions of connectors work with which versions of Flink?). Moreover, for me it is an indicator of what I mentioned that we introduce double standards for those modules. I am not saying that I am totally against that, but I think this should be a conscious decision. Ad. 2 I can't see how repository split could help with that rather than moving some of the PRs to a separate list (that probably even less people would look into). Also I think we can achieve something like that already with github filters, no? To sum up my thoughts: 1. I think it is a good idea to split our CI builds to sub-modules (connectors being the first candidate), that would trigger on a changed path basis, but without splitting the repo. 2. My feeling is that the real question is if we want to change our stability guarantees of certain modules to be "just best effort". 3. If we were to vote on this proposal I would vote -0. I am slightly against this change, but wouldn't oppose. Best, Dawid On 08/08/2019 13:23, Chesnay Schepler wrote: > > I would like to also raise an additional issue: currently quite > some bugs (like release blockers [1]) are being discovered by ITCases > of the connectors. It means that at least initially, the main > repository will lose some test coverage. > > True, but I think this is more a symptom of us not properly testing > the contracts that are exposed to connectors. > That we lose lose test coverage is already a big red flag as it > implies that issues were fixed and are now verified by a connector > test, and not by a test in the Flink core. > We could also look into tooling surrounding the CI bot for running the > connectors tests on-demand, although this is very much long-term. > > On 08/08/2019 13:14, Piotr Nowojski wrote: >> Hi, >> >> Thanks for proposing and writing this down Chesney. >> >> Generally speaking +1 from my side for the idea. It will create >> additional pain for cross repository development, like some new >> feature in connectors that need some change in the main repository. >> I’ve worked in such setup before and the teams then regretted having >> such split. But I agree that we should try this to try solve the >> stability/build time issues. >> >> I have no experience in making such kind of splits so I can not help >> here. >> >> I would like to also raise an additional issue: currently quite some >> bugs (like release blockers [1]) are being discovered by ITCases of >> the connectors. It means that at least initially, the main repository >> will lose some test coverage. >> >> Piotrek >> >> [1] https://issues.apache.org/jira/browse/FLINK-13593 >> <https://issues.apache.org/jira/browse/FLINK-13593> >> >>> On 7 Aug 2019, at 13:14, Chesnay Schepler <ches...@apache.org> wrote: >>> >>> Hello everyone, >>> >>> The Flink project sees an ever-increasing amount of dev activity, >>> both in terms of reworked and new features. >>> >>> This is of course an excellent situation to be in, but we are >>> getting to a point where the associate downsides are becoming >>> increasingly troublesome. >>> >>> The ever increasing build times, in addition to unstable tests, >>> significantly slow down the develoment process. >>> Additionally, pull requests for smaller features frequently slip >>> through the crasks as they are being buried under a mountain of >>> other pull requests. >>> >>> As a result I'd like to start a discussion on splitting the Flink >>> repository. >>> >>> In this mail I will outline the core idea, and what problems I >>> currently envision. >>> >>> I'd specifically like to encourage those who were part of similar >>> initiatives in other projects to share the experiences and ideas. >>> >>> >>> General Idea >>> >>> For starters, the idea is to create a new repository for >>> "flink-connectors". >>> For the remainder of this mail, the current Flink repository is >>> referred to as "flink-main". >>> >>> There are also other candidates that we could discuss in the future, >>> like flink-libraries (the next top-priority repo to ease flink-ml >>> development), metric reporters, filesystems and flink-formats. >>> >>> Moving out flink-connectors provides the most benefits, as we >>> straight away save at-least an hour of testing time, and not being >>> included in the binary distribution simplifies a few things. >>> >>> >>> Problems to solve >>> >>> To make this a reality there's a number of questions we have to >>> discuss; some in the short-term, others in the long-term. >>> >>> 1) Git history >>> >>> We have to decide whether we want to rewrite the history of sub >>> repositories to only contain diffs/commits related to this part of >>> Flink, or whether we just fork from some commit in flink-main and >>> add a commit to the connector repo that "transforms" it from >>> flink-main to flink-connectors (i.e., remove everything unrelated to >>> connectors + update module structure etc.). >>> >>> The latter option would have the advantage that our commit book >>> keeping in JIRA would still be correct, but it would create a >>> significant divide between the current and past state of the >>> repository. >>> >>> 2) Maven >>> >>> We should look into whether there's a way to share dependency/plugin >>> configurations and similar, so we don't have to keep them in sync >>> manually across multiple repositories. >>> >>> A new parent Flink pom that all repositories define as their parent >>> could work; this would imply splicing out part of the current room >>> pom.xml. >>> >>> 3) Documentation >>> >>> Splitting the repository realistically also implies splitting the >>> documentation source files (At the beginning we can get by with >>> having it still in flink-main). >>> We could just move the relevant files to the respective repository >>> (while maintaining the directory structure), and merge them when >>> building the docs. >>> >>> We also have to look at how we can handle java-/scaladocs; e.g. >>> whether it is possible to aggregate them across projects. >>> >>> 4) CI (end-to-end tests) >>> >>> The very basic question we have to answer is whether we want E2E >>> tests in the sub repositories. If so, we need to find a way to share >>> e2e-tooling. >>> >>> 5) Releases >>> >>> We have to discuss how our release process will look like. This may >>> also have repercussions on how repositories may depend on each other >>> (SNAPSHOT vs LATEST). Note that this should be discussed for each >>> repo separately. >>> >>> The current options I see are the following: >>> >>> a) Single release >>> >>> Release all repositories at once as a single product. >>> >>> The source release would be a collection of repositories, like >>> flink/ >>> |--flink-main/ >>> |--flink-core/ >>> |--flink-runtime/ >>> ... >>> |--flink-connectors/ >>> ... >>> |--flink-.../ >>> ... >>> >>> This option requires a SNAPSHOT dependency between Flink >>> repositories, but it is pretty much how things work at the >>> moment. >>> >>> b) Synced releases >>> >>> Similar to a), except that each repository gets their own source >>> release that they may released independent of other >>> repositories. >>> For a given release cycle each repo would produce exactly one >>> release. >>> >>> This option requires a SNAPSHOT dependency between Flink >>> repositories. Once any repositories has created an RC or >>> finished it's release, release-branches in other repos can >>> switch to that version. >>> >>> This approach is a tad more flexible than a), but requires more >>> coordination between the repos. >>> >>> c) Separate releases >>> >>> Just like we handle flink-shaded; entirely separate release >>> cycles; some repositories may have more releases in a given time >>> period than others. >>> >>> This option implies a LATEST dependency between Flink >>> repositories. >>> >>> Note that hybrid approaches would also make sense, like doing b) for >>> major versions and c) for bugfix releases. >>> >>> For something like flink-libraries this question may also have >>> repercussions on how/whether they are bundled in the distribution; >>> options a)/b) would maintain the status-quo, c) and hybrid >>> approaches will likely necessitate the exclusion from the >>> distribution. >>> >> >
signature.asc
Description: OpenPGP digital signature
