In practice you can get incremental builds, its not just theory. On Tue, Nov 28, 2017 at 10:12 PM, Romain Manni-Bucau <[email protected]> wrote:
> > > Le 29 nov. 2017 01:29, "Robert Bradshaw" <[email protected]> a écrit : > > I am curious what you mean by the "Python Build" as by nature there's > no (significant) compilation that happens. Python tests, on the other > hands, are completely sequential (and very redundant--each test file > is run at least three times). With the potential of Java builds > becoming fast enough that Python becomes the long (or at least a > significant) factor, I'm staring to get motivated to look at this more > closely :). > > > The python generated and executed code. I didnt look more than that bit it > looks like setup (pip) and tests are not that fast even if several are > skipped (?? A missing dep shouldnt reduce the coverage right?) > > > With respect to correct incremental builds, not only is it possible, > but it can be done at scale (e.g. > http://google-engtools.blogspot.com/2011/08/build-in-cloud- > how-build-system-works.html > https://blog.bazel.build/2017/08/25/introducing-sandboxfs.html > https://nixos.org/nix/) One trick is to disallow tools to even look at > inputs that are not declared. As you've probably guessed, I'm partial > to reproducible builds (and reproducible research in general) myself. > Unfortunately the explorations into bazel seem to indicate that the > third-party dependency system isn't sufficiently mature (vs. the > one-codebase environment it was developed in). > > > It is a theory yes but in practise for a multimodule project you have to > not respect it. It only impacts you with indirect code path like SPI or > reflection. Beam does the first one at least. > > Dont get me wrong, it is not a blocker but it is also not magic and you > will need to add a no daemon and clean build to avoid caching issues it is > easy to have with gradle in dev mode. > > > With regards to the original topic, hopefully the build system can > just take care of testing exactly what needs to be tested based on > changes. Note that this will likely cross language barriers more and > more as the portability work progresses (e.g. we'll have > Python-on-Flink tests that should be run if either the Python SDK or > the Flink runner gets modified). > > > > The git integration of maven we talked earlier does that. I guess/hope > gradle has the same but it is bound to the scm rather than the build tool. > Worse case a git (status based) script should enable to launch that. > > > - Robert > > > > On Tue, Nov 28, 2017 at 12:49 PM, Romain Manni-Bucau > <[email protected]> wrote: > > Lukasz: only for an isolated "system" which is a module - assuming you > still > > want to be able to build a submodule without building and revalidating > the > > whole tree which is important in dev IMO. This means you shouldnt handle > > inputs outside "current" module and therefore miss easily some (typically > > test) execution/coverage - incremental compilation/style check etc is not > > that costly compared to tests. So this is far to be as straight forward > as > > it looks if you want to keep the same guarantees. Once again for > > checkstyle/findbugs/javac/rat etc this is trivial - and even with maven > > actually - but doesnt save a significant amount of time in dev whereas > tests > > do. > > > > > > Le 28 nov. 2017 20:11, "Kenneth Knowles" <[email protected]> a écrit : > > > > I seem to remember a tool called `make` that was pretty good at this. > > > > On Tue, Nov 28, 2017 at 10:47 AM, Lukasz Cwik <[email protected]> wrote: > >> > >> Its been well shown that a build system that uses input/output set > change > >> detection can correctly implement incremental builds. Build systems are > not > >> tied to knowing the internal details of how Java compiles things. > Knowing > >> that there are some inputs, a process, and some outputs is enough to > know > >> when the process needs to be rerun. > >> > >> On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau > >> <[email protected]> wrote: > >>> > >>> Hmm, no. > >>> > >>> Incremental build is never correctly implemented cause there is just no > >>> way to detect some dependencies statically with java code - or any > dynamic > >>> language. > >>> > >>> Side note: same applies for gradle daemon usage BTW. > >>> > >>> After if the list is not maintained it is a bug at the same level than > >>> coding a toString() with "null.toString()". This is not very hard to > handle > >>> the list of modules and worse case a mvnextension can make it coded if > you > >>> feel more comfortable with this kind of solution. > >>> > >>> Le 27 nov. 2017 23:12, "Lukasz Cwik" <[email protected]> a > écrit : > >>>> > >>>> Manually whitelisting/blacklisting sub-modules is error prone since it > >>>> hides issues due to incorrectly maintaining that list is the same > >>>> argument > >>>> as if the build process doesn't correctly invoke an incremental build > >>>> process. > >>>> > >>>> On Mon, Nov 27, 2017 at 1:45 PM, Romain Manni-Bucau > >>>> <[email protected]> > >>>> wrote: > >>>> > >>>> > Well for validation builds- pre PR, incremental support is pointless > >>>> > since > >>>> > it easily hides issues die to caching so a solution saving half of > the > >>>> > build without loosing anuyhing would still be good IMHO. > >>>> > > >>>> > Le 27 nov. 2017 21:12, "Lukasz Cwik" <[email protected]> a > >>>> > écrit : > >>>> > > >>>> > > Incremental builds aren't correctly setup right now so your likely > >>>> > > to see > >>>> > > Python/Go rebuild even if there were no changes. See > >>>> > > https://issues.apache.org/jira/browse/BEAM-3253 > >>>> > > > >>>> > > On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau < > >>>> > > [email protected]> > >>>> > > wrote: > >>>> > > > >>>> > > > that was the goal: validate there was no side effect of the > >>>> > > > changes on > >>>> > > > the whole project. Now the "not java" part of the build will not > >>>> > > > be > >>>> > > > impacted by java changed so this is the part i want to skip > since > >>>> > > > it > >>>> > > > takes a lot of time and I have guarantees it is safe to skip > them. > >>>> > > > > >>>> > > > Romain Manni-Bucau > >>>> > > > @rmannibucau | Blog | Old Blog | Github | LinkedIn > >>>> > > > > >>>> > > > > >>>> > > > 2017-11-27 20:28 GMT+01:00 Lukasz Cwik <[email protected] > >: > >>>> > > > > Romain, that will build the entire project. I think you want > to > >>>> > execute > >>>> > > > > (from the root of the project): > >>>> > > > > ./gradlew :beam-sdks-parent:beam-sdks-python:build > >>>> > > > > > >>>> > > > > On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau < > >>>> > > > [email protected]> > >>>> > > > > wrote: > >>>> > > > > > >>>> > > > >> gradle build --no-daemon > >>>> > > > >> > >>>> > > > >> (with gradle 4.2) > >>>> > > > >> > >>>> > > > >> Romain Manni-Bucau > >>>> > > > >> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >>>> > > > >> > >>>> > > > >> > >>>> > > > >> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles > >>>> > > > >> <[email protected] > >>>> > >: > >>>> > > > >> > What is the gradle command you are using to build just the > >>>> > > > >> > Python > >>>> > > SDK? > >>>> > > > >> > > >>>> > > > >> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau < > >>>> > > > >> [email protected]> > >>>> > > > >> > wrote: > >>>> > > > >> > > >>>> > > > >> >> Hmm, > >>>> > > > >> >> > >>>> > > > >> >> issue is the same with gradle (locally python build takes > >>>> > > > >> >> 15mn > >>>> > > alone > >>>> > > > >> >> which is as much as the java build and it is not > >>>> > > > >> >> parallelized I > >>>> > > > think) > >>>> > > > >> >> > >>>> > > > >> >> pl is not as smooth since it means doing it on each > command > >>>> > whereas > >>>> > > > >> >> the proposal is automatically activated through > settings.xml > >>>> > > > >> >> > >>>> > > > >> >> Romain Manni-Bucau > >>>> > > > >> >> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >>>> > > > >> >> > >>>> > > > >> >> > >>>> > > > >> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles > >>>> > <[email protected] > >>>> > > >: > >>>> > > > >> >> > I think you can already mostly do this with mvn -pl > >>>> > > > >> >> > sdks/XYZ > >>>> > -am > >>>> > > > >> -amd. I > >>>> > > > >> >> > think that we have other work (gradle support) underway > >>>> > > > >> >> > that > >>>> > will > >>>> > > > make > >>>> > > > >> >> this > >>>> > > > >> >> > a non-issue since gradle automatically does even better > >>>> > > > >> >> > than > >>>> > the > >>>> > > > >> profile > >>>> > > > >> >> or > >>>> > > > >> >> > -am -amd. > >>>> > > > >> >> > > >>>> > > > >> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau < > >>>> > > > >> >> [email protected]> > >>>> > > > >> >> > wrote: > >>>> > > > >> >> > > >>>> > > > >> >> >> Hi guys, > >>>> > > > >> >> >> > >>>> > > > >> >> >> java/python/go/xxx support is great but as a developer > >>>> > > > >> >> >> you > >>>> > > rarely > >>>> > > > >> hack > >>>> > > > >> >> >> on them all. > >>>> > > > >> >> >> > >>>> > > > >> >> >> For that reason I opened https://github.com/apache/ > >>>> > > beam/pull/4173 > >>>> > > > . > >>>> > > > >> >> >> > >>>> > > > >> >> >> Goal is to give each developer a way to build the whole > >>>> > project > >>>> > > > and > >>>> > > > >> >> >> all the code he can impact at once but without caring > of > >>>> > > > >> >> >> the > >>>> > > code > >>>> > > > he > >>>> > > > >> >> >> doesn't modify at all - other languages. > >>>> > > > >> >> >> > >>>> > > > >> >> >> Wdyt? > >>>> > > > >> >> >> > >>>> > > > >> >> >> Romain Manni-Bucau > >>>> > > > >> >> >> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >>>> > > > >> >> >> > >>>> > > > >> >> > >>>> > > > >> > >>>> > > > > >>>> > > > >>>> > > >> > >> > > > > > > >
