Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320 to look at the new patch set (#4). Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images ...................................................................... IMPALA-10016: Split jars for Impala exec and coord Docker images Maven Changes: Splits out all executor specific jar files into a separate pom file under mvn-deps/executor-deps. The new pom file lists out all executor specific jar files. fe/pom.xml has a dependency on mvn-deps/executor-deps/pom.xml so that all executor specific jars are still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml writes out a build-classpath.txt file that contains all dependencies in the pom.xml file (similar to what is already done in fe/pom.xml). Docker Build Changes: setup_build_context.py was changed to leverage the aformentioned Maven changes. The script still copies all dependencies into the lib/ folder, but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/ folder contains all dependencies necessary to run Impala Executors, but excludes any dependencies that are Coordinator specific. The statestore-lib/ folder excludes all jar files entirely since it does not run an embedded JVM. The docker/CMakeLists.txt was modified to support the new library layout created by setup_build_context.py. Prior to this patch only the build for the Impala base image has access to the dependencies created by setup_build_context.py. This patch changes the build logic so all images have access to the dependencies. This does increase build time because the built context has to be copied and sent to the Docker daemon for each image build. Docker Image Changes: The copy command for the lib/ folder was removed from the impala_base Dockerfile and a corresponding copy command was added to each daemon Docker image. This only each daemon image to only copy in the dependencies it actually requires to run. Other: * Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds * Moved shaded-deps into the mvn-deps folder Overall, this decreases the size of the impalad_executor image by 70 MB, and the statestored image by 700 MB. Further improvements might be possible by decreasing the number of transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover, any new Coordinator specific jar files will not be included in the Executor image. Testing: * Ran core tests Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd --- M CMakeLists.txt M bin/rat_exclude_files.txt M docker/CMakeLists.txt M docker/README.md M docker/catalogd/Dockerfile M docker/impala_base/Dockerfile D docker/impalad_coord_exec/Dockerfile A docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M docker/impalad_executor/Dockerfile M docker/setup_build_context.py M docker/statestored/Dockerfile M fe/CMakeLists.txt M fe/pom.xml A mvn-deps/executor-deps/CMakeLists.txt A mvn-deps/executor-deps/pom.xml R mvn-deps/shaded-deps/.gitignore R mvn-deps/shaded-deps/CMakeLists.txt R mvn-deps/shaded-deps/pom.xml 19 files changed, 419 insertions(+), 306 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/4 -- To view, visit http://gerrit.cloudera.org:8080/16320 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd Gerrit-Change-Number: 16320 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>