Re: [build system] speeding up maven build building only changed modules compared to master branch

2019-01-28 Thread Reynold Xin
This might be useful to do. BTW, based on my experience with different build systems in the past few years (extensively SBT/Maven/Bazel, and to a less extent Gradle/Cargo), I think the longer term solution is to move to Bazel. It is so much easier to understand and use, and also much more

Re: [build system] speeding up maven build building only changed modules compared to master branch

2019-01-28 Thread Gabor Somogyi
Do you have some numbers how much is this faster? I'm asking it because previously I've evaluated another plugin and found the following: - Incremental build didn't bring too much even in bigger than spark projects - Incremental test was buggy and sometimes the required tests were not executed

Re: [build system] speeding up maven build building only changed modules compared to master branch

2019-01-26 Thread Sean Owen
Sounds interesting; would it be able to handle R and Python modules built by this project ? The home grown solution here does I think and that is helpful. On Sat, Jan 26, 2019, 6:57 AM vaclavkosar I think it would be good idea to use gitflow-incremental-builder maven > plugin for Spark builds.

[build system] speeding up maven build building only changed modules compared to master branch

2019-01-26 Thread vaclavkosar
I think it would be good idea to use gitflow-incremental-builder maven plugin for Spark builds. It saves resources by building only modules that are impacted by changes compared to git master branch via gitflow-incremental-builder maven plugin. For example if there is only a change introduced