I'm not enthusiastic about making the docker build process mandatory. It's bad enough having to remember to type -DskipShade to avoid a 5-10 minute delay every time I do a build of a different branch, which I have to do every single time I change from one PR to another.
I do not see why the docker build needs to be forced onto everyone. If I had a choice, I'd make every patch which went near hadoop-common to have been retested against the real object stores of hadoop-aws, hadoop-azure, etc. But I dont do that and instead get to find out when some change has accidentally broken those builds (usually artifact updates, bouncy-castle being needed by miniyarn, etc). I'm OK with that. So why can't the popel who want the docker build turn it on. > Do we want to have inline docker build process in maven? Yes to the build, -1 to it being mandatory on a normal build-the-JARs "mvn clean install -DskipTests" run. > If yes, it would be developer’s responsibility to pass -DskipDocker flag to skip docker. Docker is mandatory for default build. see above > If no, what is the release flow for docker images going to look like? the -Pdist profile exists for releasing things right now. On Wed, Mar 13, 2019 at 10:24 PM Eric Yang <ey...@hortonworks.com> wrote: > Hi Hadoop developers, > > In the recent months, there were various discussions on creating docker > build process for Hadoop. There was convergence to make docker build > process inline in the mailing list last month when Ozone team is planning > new repository for Hadoop/ozone docker images. New feature has started to > add docker image build process inline in Hadoop build. > A few lessons learnt from making docker build inline in YARN-7129. The > build environment must have docker to have a successful docker build. > BUILD.txt stated for easy build environment use Docker. There is logic in > place to ensure that absence of docker does not trigger docker build. The > inline process tries to be as non-disruptive as possible to existing > development environment with one exception. If docker’s presence is > detected, but user does not have rights to run docker. This will cause the > build to fail. > > Now, some developers are pushing back on inline docker build process > because existing environment did not make docker build process mandatory. > However, there are benefits to use inline docker build process. The listed > benefits are: > > 1. Source code tag, maven repository artifacts and docker hub artifacts > can all be produced in one build. > 2. Less manual labor to tag different source branches. > 3. Reduce intermediate build caches that may exist in multi-stage builds. > 4. Release engineers and developers do not need to search a maze of build > flags to acquire artifacts. > > The disadvantages are: > > 1. Require developer to have access to docker. > 2. Default build takes longer. > > There is workaround for above disadvantages by using -DskipDocker flag to > avoid docker build completely or -pl !modulename to bypass subprojects. > Hadoop development did not follow Maven best practice because a full > Hadoop build requires a number of profile and configuration parameters. > Some evolutions are working against Maven design and require fork of > separate source trees for different subprojects and pom files. Maven best > practice (https://dzone.com/articles/maven-profile-best-practices) has > explained that do not use profile to trigger different artifact builds > because it will introduce maven artifact naming conflicts on maven > repository using this pattern. Maven offers flags to skip certain > operations, such as -DskipTests -Dmaven.javadoc.skip=true -pl or > -DskipDocker. It seems worthwhile to make some corrections to follow best > practice for Hadoop build. > > Some developers have advocated for separate build process for docker > images. We need consensus on the direction that will work best for Hadoop > development community. Hence, my questions are: > > Do we want to have inline docker build process in maven? > If yes, it would be developer’s responsibility to pass -DskipDocker flag > to skip docker. Docker is mandatory for default build. > If no, what is the release flow for docker images going to look like? > > Thank you for your feedback. > > Regards, > Eric >
