[Impala-ASF-CR] Don't build with shared objects in bootstrap build.sh
Hello Impala Public Jenkins, Jim Apple, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12910 to look at the new patch set (#2). Change subject: Don't build with shared objects in bootstrap_build.sh .. Don't build with shared objects in bootstrap_build.sh Building with shared objects is not supported currently because of various problems. All tested configurations of Impala are built with the supporting libraries being linked statically. The patch removes the '-so' flag from bootstrap_build.sh to make the compile-only check work the same way as the regular builds. Change-Id: Id89a19c6fc29b7ac21e5d6174d490ddc8a6f9c0b --- M bin/bootstrap_build.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/12910/2 -- To view, visit http://gerrit.cloudera.org:8080/12910 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id89a19c6fc29b7ac21e5d6174d490ddc8a6f9c0b Gerrit-Change-Number: 12910 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] Don't build with shared objects in bootstrap build.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/12910 ) Change subject: Don't build with shared objects in bootstrap_build.sh .. Patch Set 2: This update is more about testing Gerrit/Jenkins integration after a Gerrit plugin update. -- To view, visit http://gerrit.cloudera.org:8080/12910 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id89a19c6fc29b7ac21e5d6174d490ddc8a6f9c0b Gerrit-Change-Number: 12910 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Sat, 28 Dec 2019 12:24:44 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-9265: Support for toolchain Kudu to provide Java artifacts
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15072 ) Change subject: IMPALA-9265: Support for toolchain Kudu to provide Java artifacts .. Patch Set 2: Took a first pass through it, it looks pretty good. There is one issue that makes me wonder: 1.we explicitly include the C/C++ compiler version in the resulting tarballs' name 2. We now start including Java binaries in the same tarballs. Java binaries can (in theory at least, if not in current practice) be produced by different JDK versions, so should we start including the JDK version (or distro+version) string in the artifact names? Currently we build only with JDK 8, but as JDK 8 is nearing its End of Support Date, this may change one day. We can also defer the decision and establish the convention that no explicit Java version means JDK 8, and everything else is marked; this can be left to our future selves. -- To view, visit http://gerrit.cloudera.org:8080/15072 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba03dfe9c302513b825cbed7146c582e7d97c3af Gerrit-Change-Number: 15072 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 20 Jan 2020 22:11:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9279: Update the Kudu version to include VARCHAR support
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15134 ) Change subject: IMPALA-9279: Update the Kudu version to include VARCHAR support .. Patch Set 2: > Patch Set 2: > The verify job failed because kudu-3ba5ec5d0 (kudu-1.12.0-SNAPSHOT) has a new > run-time dependency: libcurl.so.4 which is not available in the > ubuntu-16.04-configured jenkins worker label. I'm discussing with laszlog the > possibility of adding libcurls.so.4 to the worker labe;. > If we decide to take this new Kudu version as a dependency, then the correct way to handle libcurl.so.4 as a new runtime dependency is to add it to the list of packages we install in bin/bootstrap_system.sh. The worker image referenced above is only minimally preconfigured to allow fast startup times; Impala runtime/development time dependencies should be managed in the bootstrap scripts. Additionally, the dependency on libcurl.so.4 should be evaluated for all OS platforms we claim to have support for: e.g. a brief scan of this article[1] claims that running both libcurl.so.3 and libcurl.so.4 on Ubuntu 18.04 is at least non-trivial to set up. [1]: https://dev.to/jake/using-libcurl3-and-libcurl4-on-ubuntu-1804-bionic-184g, "Using libcurl3 and libcurl4 on Ubuntu 18.04 (Bionic)" -- To view, visit http://gerrit.cloudera.org:8080/15134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iafe56342d43cb63e35c0bbb1b4a99327dda0a44a Gerrit-Change-Number: 15134 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 31 Jan 2020 17:12:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9279: Update the Kudu version to include VARCHAR support
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15134 ) Change subject: IMPALA-9279: Update the Kudu version to include VARCHAR support .. Patch Set 2: [reply thread elided for brevity's sake] I have hijacked an unused (for now) build label, adding the GNU-TLS flavor of libcurl.so in the instance startup script. I started 2 builds there: 1. Attila's change (patch set #2) with core mode tests: this passed at https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/9901/ 2. Core mode tests on master to see if adding libcurl.so.4 leads to any regression. This test passed as well, see: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/9896/ I think the main question here is if we want/need to pick up a pre-release snapshot of Kudu. -- To view, visit http://gerrit.cloudera.org:8080/15134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iafe56342d43cb63e35c0bbb1b4a99327dda0a44a Gerrit-Change-Number: 15134 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 31 Jan 2020 21:25:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9390: Change ant and maven download URLs to valid locations
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15229 Change subject: IMPALA-9390: Change ant and maven download URLs to valid locations .. IMPALA-9390: Change ant and maven download URLs to valid locations Looks like there was a change in the Apache download infrastructure, and the URLs used by the bootstrap scripts to download ant and maven are no longer valid. Manually traversing the download pages yielded the new locations, https://downloads.apache.org/..., which was used to update the bootstrap scripts. Change-Id: I299bdd9cbe999ff4d483c022fa69ec88b37612f6 --- M bin/bootstrap_build.sh M bin/bootstrap_system.sh 2 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/15229/1 -- To view, visit http://gerrit.cloudera.org:8080/15229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I299bdd9cbe999ff4d483c022fa69ec88b37612f6 Gerrit-Change-Number: 15229 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] IMPALA-9402: Relax bootstrap condition for PostgreSQL permission tweaks
Hello David Knupp, Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/15243 to review the following change. Change subject: IMPALA-9402: Relax bootstrap condition for PostgreSQL permission tweaks .. IMPALA-9402: Relax bootstrap condition for PostgreSQL permission tweaks bin/bootstrap_system.sh was looking for a strict combination of OS platform / pg_hba.conf location / default authN method when setting up the default Impala build environment. This combo proved to be too strict for CentOS 7.4 running with PostreSQL 9.2, and failed to set up this environment correctly, which lead to a failure in the dataload phase. The patch fixes the permission tweak for this combination by relaxing the initial condition where the tweak may be applied. Change-Id: I31cbee6c22f9c27b1ff2b9200bb356be14429f39 --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/15243/1 -- To view, visit http://gerrit.cloudera.org:8080/15243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I31cbee6c22f9c27b1ff2b9200bb356be14429f39 Gerrit-Change-Number: 15243 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9390: Change ant and maven download URLs to valid locations
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15229 ) Change subject: IMPALA-9390: Change ant and maven download URLs to valid locations .. Patch Set 1: (1 comment) > Patch Set 1: > > (1 comment) http://gerrit.cloudera.org:8080/#/c/15229/1/bin/bootstrap_build.sh File bin/bootstrap_build.sh: http://gerrit.cloudera.org:8080/#/c/15229/1/bin/bootstrap_build.sh@50 PS1, Line 50: -nv > I think we can remove the "-nv" option to improve debuggability. The output I prefer to keep it: the default progress indicator is too chatty (at least to me), and -nv still prints a one-line summary of what happened even in a successful case. Ultimately it's a matter of personal taste, so I don't feel too strongly about it, but my personal opinion is that some of our run logs are still too verbose in a couple of places. -- To view, visit http://gerrit.cloudera.org:8080/15229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I299bdd9cbe999ff4d483c022fa69ec88b37612f6 Gerrit-Change-Number: 15229 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 20 Feb 2020 15:06:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9390: Change ant and maven download URLs to valid locations
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15229 ) Change subject: IMPALA-9390: Change ant and maven download URLs to valid locations .. Patch Set 2: Verified+1 > Patch Set 2: > > The GVO dies failed with this in clang tidy: > [ERROR] Plugin net.sourceforge.czt.dev:cup-maven-plugin:1.6-cdh or one of its > dependencies could not be resolved: Could not find artifact > net.sourceforge.czt.dev:cup-maven-plugin:jar:1.6-cdh in > cloudera.thirdparty.repo > (https://repository.cloudera.com/content/repositories/third-party) -> [Help 1] Yeah; I retriggered the clang-tidy-ub1604 step and now it passed. Looks like downloading the maven plugin from the Cloudera thirdparty repo got flaky for some reason. Overriding Verified -1 to Verified +1 based on this evidence: https://jenkins.impala.io/job/clang-tidy-ub1604/10400/ -- To view, visit http://gerrit.cloudera.org:8080/15229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I299bdd9cbe999ff4d483c022fa69ec88b37612f6 Gerrit-Change-Number: 15229 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 20 Feb 2020 22:14:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9390: Change ant and maven download URLs to valid locations
Laszlo Gaal has removed a vote on this change. Change subject: IMPALA-9390: Change ant and maven download URLs to valid locations .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/15229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I299bdd9cbe999ff4d483c022fa69ec88b37612f6 Gerrit-Change-Number: 15229 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9390: Change ant and maven download URLs to valid locations
Laszlo Gaal has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15229 ) Change subject: IMPALA-9390: Change ant and maven download URLs to valid locations .. IMPALA-9390: Change ant and maven download URLs to valid locations Looks like there was a change in the Apache download infrastructure, and the URLs used by the bootstrap scripts to download ant and maven are no longer valid. Manually traversing the download pages yielded the new locations, https://downloads.apache.org/..., which was used to update the bootstrap scripts. Change-Id: I299bdd9cbe999ff4d483c022fa69ec88b37612f6 Reviewed-on: http://gerrit.cloudera.org:8080/15229 Reviewed-by: Impala Public Jenkins Tested-by: Laszlo Gaal --- M bin/bootstrap_build.sh M bin/bootstrap_system.sh 2 files changed, 3 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Laszlo Gaal: Verified -- To view, visit http://gerrit.cloudera.org:8080/15229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I299bdd9cbe999ff4d483c022fa69ec88b37612f6 Gerrit-Change-Number: 15229 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR](asf-site) IMPALA-9462: Point to downloads.apache.org for current release
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15443 Change subject: IMPALA-9462: Point to downloads.apache.org for current release .. IMPALA-9462: Point to downloads.apache.org for current release The Apache Infra team deprecated the existing download point https://www.apache.org/dist for the new https://downloads.apache.org. This patch changes the Impala download page to point to the new address for downloading the current version. Tested by loading the updated webpage from the local file system into Chrome and clicking the changed links and verifying that the Impala package and the signatures can be retrieved from their official locations. Change-Id: Iccc15ef785529d477958012a3f5c83715580a26d --- M downloads.html 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/15443/1 -- To view, visit http://gerrit.cloudera.org:8080/15443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newchange Gerrit-Change-Id: Iccc15ef785529d477958012a3f5c83715580a26d Gerrit-Change-Number: 15443 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] WIP: Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15461 Change subject: WIP: Add CentOS 8.1 support to bootstrap_system.sh .. WIP: Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a vrtual machine, or inside a Docker container. Details: - curl is added to the list of required packages, required by IMPALA-9149 - snappy-devel moved to the PowerTools repo, so it needs to get installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. Do not merge this change in the current state! The last change will most likely break other platforms, it will have to be fixed before this change can be merged to master. Change-Id: I67a58ec007219020e1fb562216d7a0d1ff38b0bd --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 99 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/15461/1 -- To view, visit http://gerrit.cloudera.org:8080/15461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I67a58ec007219020e1fb562216d7a0d1ff38b0bd Gerrit-Change-Number: 15461 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[native-toolchain-CR] WIP: Add platform tag for Ubuntu 14.04
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15469 ) Change subject: WIP: Add platform tag for Ubuntu 14.04 .. Patch Set 1: I must be missing something here: I was convinced that the earlier change, https://gerrit.cloudera.org/c/15464/1 removed the Ubuntu-14.04 toolchain build (so artifacts wouldn't be prodiced either). How come that the build is still producing artifacts then? Wouldn't the right fix be to make sure that Ubuntu-14.04 bits are just not built? -- To view, visit http://gerrit.cloudera.org:8080/15469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32dac434ddf12ec3a2367f6a0974b3504bd137c5 Gerrit-Change-Number: 15469 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 17 Mar 2020 17:37:49 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15461 ) Change subject: WIP: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/15461/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15461/1//COMMIT_MSG@27 PS1, Line 27: - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. Add remark for updating the docker-based tests as well http://gerrit.cloudera.org:8080/#/c/15461/1//COMMIT_MSG@28 PS1, Line 28: Add description of chronyd vs ntpd for Centos8 http://gerrit.cloudera.org:8080/#/c/15461/1/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/15461/1/bin/bootstrap_system.sh@262 PS1, Line 262: fix indentation http://gerrit.cloudera.org:8080/#/c/15461/1/bin/bootstrap_system.sh@282 PS1, Line 282: redhat8 inycloud sudo alternatives --install /usr/bin/python python /usr/bin/python2 90 --slave /usr/bin/pip pip /usr/bin/pip2 line too long -- To view, visit http://gerrit.cloudera.org:8080/15461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67a58ec007219020e1fb562216d7a0d1ff38b0bd Gerrit-Change-Number: 15461 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 17 Mar 2020 18:02:19 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add support for centos8
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15465 ) Change subject: Add support for centos8 .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/15465/1/docker/all/postinstall.sh File docker/all/postinstall.sh: http://gerrit.cloudera.org:8080/#/c/15465/1/docker/all/postinstall.sh@31 PS1, Line 31: alternatives --set python /usr/bin/python2 Just a question: does this alias pip2 to pip as well? http://gerrit.cloudera.org:8080/#/c/15465/1/docker/redhat8.df File docker/redhat8.df: http://gerrit.cloudera.org:8080/#/c/15465/1/docker/redhat8.df@48 PS1, Line 48: # We get a newer java-1.8.0-openjdk-devel from centos:7.4. : # The java-1.8.0-openjdk version shipped with centos:7.2 is unable to handle ECDHE : # ciphers. nit: I guess this comment is stale now -- To view, visit http://gerrit.cloudera.org:8080/15465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc15c202f61e251761fd0b1dc9aa0b15c27b3363 Gerrit-Change-Number: 15465 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 17 Mar 2020 20:13:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15461 ) Change subject: WIP: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 1: General remark: this change gets the build bootstrap process to the point of building the Impala virtualenv. I have tested it both in a regular Docker container using docker/test-woth-docker.py (the Centos 8 base container there is a pretty slim one) and with a Centos 8.1 container instance on our internal build cloud. The bootstrap process progresses, then fails the same way at the same point in both environments. -- To view, visit http://gerrit.cloudera.org:8080/15461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67a58ec007219020e1fb562216d7a0d1ff38b0bd Gerrit-Change-Number: 15461 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 17 Mar 2020 20:24:04 + Gerrit-HasComments: No
[native-toolchain-CR] Add support for centos8
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15465 ) Change subject: Add support for centos8 .. Patch Set 2: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/15465/1/docker/all/postinstall.sh File docker/all/postinstall.sh: http://gerrit.cloudera.org:8080/#/c/15465/1/docker/all/postinstall.sh@31 PS1, Line 31: alternatives --set python /usr/bin/python2 > Kinda. Thanks for the explananation Hector, I get it now . http://gerrit.cloudera.org:8080/#/c/15465/1/docker/redhat8.df File docker/redhat8.df: http://gerrit.cloudera.org:8080/#/c/15465/1/docker/redhat8.df@48 PS1, Line 48: RUN postinstall.sh : : COPY ./all > Done Done -- To view, visit http://gerrit.cloudera.org:8080/15465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc15c202f61e251761fd0b1dc9aa0b15c27b3363 Gerrit-Change-Number: 15465 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Mar 2020 20:34:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. Patch Set 1: Code-Review+2 LGTM. Thanks for fixing this; it has bothered me for some time too. -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Sun, 22 Mar 2020 14:34:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 4: Code-Review+1 LGTM. Tim or Zoltan, do you want to take a look? -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Sun, 22 Mar 2020 14:35:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9107 (part 1): Add scripts to produce an m2 archive
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/14562 ) Change subject: IMPALA-9107 (part 1): Add scripts to produce an m2 archive .. Patch Set 15: Code-Review+2 Rebased, no change in the files, carry +2 from Tim -- To view, visit http://gerrit.cloudera.org:8080/14562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I043912f5fbc7cf24ee80b2855354656aa587ca9f Gerrit-Change-Number: 14562 Gerrit-PatchSet: 15 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 22 Mar 2020 14:48:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9406 Add some new packages to download
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15253 ) Change subject: IMPALA-9406 Add some new packages to download .. Patch Set 1: Hey Zhao, I wonder if this is still necessary. You mention that you need these packages for building Kudu, so I assume you need to build Kudu because the CDH_KUDU and CDP_KUDU are not provided on ARM for you ARM efforts. However, since commit 14ae6eae1e3e0ff78040ae2bad92d36d745cf365 on master (commit title: "IMPALA-9279: Update the Kudu version to include VARCHAR support") Kudu can be integrated (again) via the toolchain, which may be a cleaner way for you. Please note that I have no problems adding the packages, I'm just curious about the use case that requires them. Thanks, - Laszlo -- To view, visit http://gerrit.cloudera.org:8080/15253 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b3404b7c067a78689197c774b3b63115ea4339e Gerrit-Change-Number: 15253 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Sun, 22 Mar 2020 14:59:09 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-9462: Point to downloads.apache.org for current release
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15443 ) Change subject: IMPALA-9462: Point to downloads.apache.org for current release .. Patch Set 1: > Patch Set 1: > > I was thinking I would update this when I upload the Impala 3.4 artifacts. > Impala 3.3 will move to having an archive.apache.org URL and Impala 3.4 will > go directly to downloads.apache.org. SGTM -- To view, visit http://gerrit.cloudera.org:8080/15443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Iccc15ef785529d477958012a3f5c83715580a26d Gerrit-Change-Number: 15443 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 12:38:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15537 ) Change subject: IMPALA-9107: Add timestamp to maven logging options. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e Gerrit-Change-Number: 15537 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 24 Mar 2020 21:20:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9401: primitive include-what-you-use script and mappings
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15552 ) Change subject: IMPALA-9401: primitive include-what-you-use script and mappings .. Patch Set 1: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/15552/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15552/1//COMMIT_MSG@12 PS1, Line 12: build nit: may be a typo? http://gerrit.cloudera.org:8080/#/c/15552/1/bin/iwyu/iwyu.sh File bin/iwyu/iwyu.sh: http://gerrit.cloudera.org:8080/#/c/15552/1/bin/iwyu/iwyu.sh@31 PS1, Line 31: echo "" Maybe add an example invocation with environment variables filled out based on a (somewhat) concrete Impala checkout? I'd expect that most people would not run the tool often enough to recall all the arguments readily. -- To view, visit http://gerrit.cloudera.org:8080/15552 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf5f9ba865313afb0c581e6482514ef7f1c65367 Gerrit-Change-Number: 15552 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Thu, 26 Mar 2020 21:52:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9406 Add some new packages to download
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15253 ) Change subject: IMPALA-9406 Add some new packages to download .. Patch Set 1: > Patch Set 1: > > Thank you for watching this, Laszlo. Because now, there are no valid and > released kudu versions which support aarch64, I need to build it from source > code which have been modified for adapting aarch64. Anyway, this commit is > not very critical, I can download and install the packages myself. Maybe in > future when there is a aarch64 version kudu, this commit is useless, we can > just abandon it. I am not opposing these packages; they are fairly trivial. I was more interested in your workflow, as you mentioned that you need to build Kudu yourself, given that there is currently no Kudu distribution ready for aarch64. Given that: 1. Impala usually pulls in Kudu through the native-toolchain project, and since IMPALA-9279 this is again possible for the master branch, and 2. I noticed that there were several commits added to the native-toolchain project targeting aarch64 compatibility, I thought that maybe the best way to build Kudu would be through the native-toolchain project (which project most likely has these packages too). Anyway, it is also OK if you follow a staged approach; these packages are unlikely to cause problems. -- To view, visit http://gerrit.cloudera.org:8080/15253 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b3404b7c067a78689197c774b3b63115ea4339e Gerrit-Change-Number: 15253 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Thu, 26 Mar 2020 22:01:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9401: primitive include-what-you-use script and mappings
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15552 ) Change subject: IMPALA-9401: primitive include-what-you-use script and mappings .. Patch Set 2: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/15552/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15552/1//COMMIT_MSG@12 PS1, Line 12: built > Done Done http://gerrit.cloudera.org:8080/#/c/15552/1/bin/iwyu/iwyu.sh File bin/iwyu/iwyu.sh: http://gerrit.cloudera.org:8080/#/c/15552/1/bin/iwyu/iwyu.sh@31 PS1, Line 31: echo "" > Done Done -- To view, visit http://gerrit.cloudera.org:8080/15552 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf5f9ba865313afb0c581e6482514ef7f1c65367 Gerrit-Change-Number: 15552 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 27 Mar 2020 21:48:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has abandoned this change. ( http://gerrit.cloudera.org:8080/15461 ) Change subject: WIP: Add CentOS 8.1 support to bootstrap_system.sh .. Abandoned Will resubmit a much newer version -- To view, visit http://gerrit.cloudera.org:8080/15461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I67a58ec007219020e1fb562216d7a0d1ff38b0bd Gerrit-Change-Number: 15461 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Use Python from the toolchain for Impala
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15624 Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing mative code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an assert to Impala tests checking that they are being run with Python 2.7.x - Add an assert to the Impala virtualenv checking that HTTPS modules were loaded successfully - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M docker/entrypoint.sh M infra/python/bootstrap_virtualenv.py A infra/python/tmp.py M tests/common/impala_test_suite.py 8 files changed, 102 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/1 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15623 Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a vrtual machine, or inside a Docker container. Details: - curl is added to the list of required packages, required by IMPALA-9149 - snappy-devel moved to the PowerTools repo, so it needs to be installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - For some unknown reason, when the downloaded Maven tarball is extracted in a Docker-based test, the "bin" and "boot" directories are created with owner-only permissions. The 'impdev' users has no access to the maven executable, which then breaks the build. This patch forcibly restores the correct permissions on these directories; this is a no-op when the extraction happens correctly. - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 125 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15623/1 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15623 to look at the new patch set (#2). Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a vrtual machine, or inside a Docker container. Details: - curl is added to the list of required packages, required by IMPALA-9149 - snappy-devel moved to the PowerTools repo, so it needs to be installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - For some unknown reason, when the downloaded Maven tarball is extracted in a Docker-based test, the "bin" and "boot" directories are created with owner-only permissions. The 'impdev' users has no access to the maven executable, which then breaks the build. This patch forcibly restores the correct permissions on these directories; this is a no-op when the extraction happens correctly. - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 83 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15623/2 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15623 to look at the new patch set (#3). Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a vrtual machine, or inside a Docker container. Details: - curl is added to the list of required packages, required by IMPALA-9149 - snappy-devel moved to the PowerTools repo, so it needs to be installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - For some unknown reason, when the downloaded Maven tarball is extracted in a Docker-based test, the "bin" and "boot" directories are created with owner-only permissions. The 'impdev' users has no access to the maven executable, which then breaks the build. This patch forcibly restores the correct permissions on these directories; this is a no-op when the extraction happens correctly. - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 83 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15623/3 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Use Python from the toolchain for Impala
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#3). Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing mative code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an assert to Impala tests checking that they are being run with Python 2.7.x - Add an assert to the Impala virtualenv checking that HTTPS modules were loaded successfully - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M docker/entrypoint.sh M infra/python/bootstrap_virtualenv.py A infra/python/tmp.py M tests/common/impala_test_suite.py 8 files changed, 101 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/3 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15623 ) Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 3: I had to update IMPALA_TOOLCHAIN_BUILD_ID to make the change build (the previous toolchain ID contained only Centos 8 artifacts, but nothing else, which broke GVO on Ubuntu 16.04) -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Wed, 01 Apr 2020 16:01:55 + Gerrit-HasComments: No
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15623 to look at the new patch set (#4). Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a vrtual machine, or inside a Docker container. Details: - curl is added to the list of required packages, required by IMPALA-9149 - snappy-devel moved to the PowerTools repo, so it needs to be installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. The installer is adaptive, it performs only the necessry steps, so it works in various environments. The installer logic is also shared between bin/bootstrap_system.sh and docker/entrypoint.sh - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - For some unknown reason, when the downloaded Maven tarball is extracted in a Docker-based test, the "bin" and "boot" directories are created with owner-only permissions. The 'impdev' users has no access to the maven executable, which then breaks the build. This patch forcibly restores the correct permissions on these directories; this is a no-op when the extraction happens correctly. - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 108 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15623/4 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 4 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Use Python from the toolchain for Impala
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#4). Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing mative code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an assert to Impala tests checking that they are being run with Python 2.7.x - Add an assert to the Impala virtualenv checking that HTTPS modules were loaded successfully - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M docker/entrypoint.sh M infra/python/bootstrap_virtualenv.py A infra/python/tmp.py M tests/common/impala_test_suite.py 7 files changed, 104 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/4 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 4 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Use Python from the toolchain for Impala
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#5). Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing mative code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an assert to Impala tests checking that they are being run with Python 2.7.x - Add an assert to the Impala virtualenv checking that HTTPS modules were loaded successfully - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M docker/entrypoint.sh M infra/python/bootstrap_virtualenv.py A infra/python/tmp.py M tests/common/impala_test_suite.py 7 files changed, 104 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/5 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 5 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Use Python from the toolchain for Impala
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15624 ) Change subject: Use Python from the toolchain for Impala .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py File infra/python/bootstrap_virtualenv.py: http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py@50 PS6, Line 50: # Ensure the ssl modules are loaded correctly. : import ssl : from httplib import HTTPConnection : from urllib2 import HTTPSHandle Tim, are these items still needed/useful here with tmp.py duplicating it? -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 6 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 02 Apr 2020 17:10:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15623 ) Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/15623/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15623/5//COMMIT_MSG@37 PS5, Line 37: self-inflicted: add details on how it was tested -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 5 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 02 Apr 2020 17:14:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Use Python from the toolchain for Impala
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15624 ) Change subject: Use Python from the toolchain for Impala .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG@39 PS6, Line 39: Add details on how it was tested -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 6 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 02 Apr 2020 17:14:40 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add build script to create and publish Docker containers
Hello Hector Acosta, Joe McDonnell, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/15675 to review the following change. Change subject: Add build script to create and publish Docker containers .. Add build script to create and publish Docker containers Adds automation to build and publish the containers used by the Docker-based toolchain builds. These can be used by the toolchain builds to save time by using pre-created containers instead of building the containers on the fly. They can also be used to provide a stable, controlled environment for the toolchain builds. The script can also be invoked from a Jenkins build environment. Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 --- M docker/buildall.py A jenkins/build.sh 2 files changed, 59 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/75/15675/1 -- To view, visit http://gerrit.cloudera.org:8080/15675 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 Gerrit-Change-Number: 15675 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-9097: Don't require minicluster for backend tests
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15641 ) Change subject: IMPALA-9097: Don't require minicluster for backend tests .. Patch Set 1: Code-Review+2 Looks good, thanks for the fix! -- To view, visit http://gerrit.cloudera.org:8080/15641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8f1b1385853fb23df28d24d38761237e6e5c97a7 Gerrit-Change-Number: 15641 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Wed, 08 Apr 2020 13:30:44 + Gerrit-HasComments: No
[Impala-ASF-CR] Use Python from the toolchain for Impala
Laszlo Gaal has posted comments on this change. (
http://gerrit.cloudera.org:8080/15624 )
Change subject: Use Python from the toolchain for Impala
..
Patch Set 7:
(1 comment)
> Patch Set 6:
>
> (1 comment)
Passed a core-mode test run on private infrastructure on CentOS 7.4
http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py
File infra/python/bootstrap_virtualenv.py:
http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py@211
PS6, Line 211: sys.path.append(os.environ.get("IMPALA_HOME") + "/bin")
: from bootstrap_toolchain import (ToolchainPackage)
> I'm not sure there's anything to be gained by not having these at the top o
I opted for the local import when the debug output from the SSL imports above
made me realize that bootstrap_virtualenv.py is invoked every time a Python
script with the impala_python shebang is loaded. I have not measured how much
latency the import would add to it, but it seemed like a good way to avoid
adding latency and increasing coupling at the same time.
That said, the impact is probably not high, so I'm OK with placing the import
at the top of the file, if you think it improves readability.
--
To view, visit http://gerrit.cloudera.org:8080/15624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b
Gerrit-Change-Number: 15624
Gerrit-PatchSet: 7
Gerrit-Owner: Laszlo Gaal
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 08 Apr 2020 15:18:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15623 ) Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG@14 PS6, Line 14: vrtual nit:virtual http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG@18 PS6, Line 18: curl is added to the list of required packages, required by : IMPALA-9149 delete this: curl was added for the no-NTP patch http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG@25 PS6, Line 25: necessry nit: necessary http://gerrit.cloudera.org:8080/#/c/15623/5/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/15623/5/bin/bootstrap_system.sh@266 PS5, Line 266: sudo alternatives --add-slave python /usr/bin/python2 /usr/bin/pip pip /usr/bin/pip2 > I need to look at the other reviews, but there are very few places where we TL;DR: Impala (minus the shell) is still Python2 dependent, and Impala development requires serious commitment as far as system configuration is concerned. The Impala utility scripts (the ones not dependent on impala-python) should move to Python3 compatibility (I'll fiel a Jira if there's none yet). To me that looks like a separate, although important issue; until that's achieved, this option looks like a passable interim solution. @Tim: I agree that this is far from ideal for long-running environments. For better or worse, bootstrap_system.sh makes other changes as well, although probably none of them are as significant as this one if you assume a Python3 default before it is run. On the other hand Impala and Hadoop seems to be coupled to Python2 so much that it is hardly a surprise that Impala (as of now) assumes default Python to be Python2. @David: I've run a quick `git grep` for shebangs: laszlog@laszlog2:~/upstream/Impala$ git grep -l '^#!.* python' | wc -l 32 laszlog@laszlog2:~/upstream/Impala$ git grep -l '^#!.* python' -- shell | wc -l 13 IOW we have 32 scripts with "#!/usr/bin/env python", of which 13 files are located in the "shell" subtree. Most of the rest are utility scripts, like push_to_asf.py or the config generators like fe/src/test/resources/hive-site.xml.py; I assume these can be made Python3-ready (or tolerant) relatively easy. There are a few product scripts too, like common/function-registry/gen_builtins_catalog.py, or common/thrift/generate_error_codes.py, which may be more difficult, but I assume it would be more like a matter of time. Note that this doesn't consider any possible scripts outside the Impala tree (e.g. various private tooling or test frameworks or systems), neither does it consider scripts in other Hadoop components that might get invoked. I also saw that there are multiple existing Jira tickets about making various Python components compatible with Python3; I'll check those and file a new one for script updates if needed. http://gerrit.cloudera.org:8080/#/c/15623/5/bin/bootstrap_system.sh@367 PS5, Line 367: redhat8 indocker sudo -u postgres PGDATA=/var/lib/pgsql/data bash -c \ > Not a blocker, but it would be nice to have a way to avoid the duplication Yes, the support matrix grew large enough (redhat[6]/7/8, Ubuntu 14/16/18[/20 in the near future]) that some refactoring is called for. I'll file a Jira. -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 6 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Apr 2020 16:49:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Hello Zoltan Borok-Nagy, David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15623 to look at the new patch set (#7). Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a vrtual machine, or inside a Docker container. Details: - curl is added to the list of required packages, required by IMPALA-9149 - snappy-devel moved to the PowerTools repo, so it needs to be installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. The installer is adaptive, it performs only the necessry steps, so it works in various environments. The installer logic is also shared between bin/bootstrap_system.sh and docker/entrypoint.sh - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - For some unknown reason, when the downloaded Maven tarball is extracted in a Docker-based test, the "bin" and "boot" directories are created with owner-only permissions. The 'impdev' users has no access to the maven executable, which then breaks the build. This patch forcibly restores the correct permissions on these directories; this is a no-op when the extraction happens correctly. - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. - Add Centos8-specific bootstrap code to the Docker-based tests. Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 115 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15623/7 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 7 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Use Python from the toolchain for Impala
Hello David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#8). Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing native code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an auto-use fixture to conftest.py to check that the tests are being run with Python 2.7.x - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M infra/python/bootstrap_virtualenv.py A infra/python/tmp.py M tests/common/impala_test_suite.py M tests/conftest.py 7 files changed, 106 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/8 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 8 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Use Python from the toolchain for Impala
Hello David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#9). Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing native code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an auto-use fixture to conftest.py to check that the tests are being run with Python 2.7.x - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M infra/python/bootstrap_virtualenv.py M tests/common/impala_test_suite.py M tests/conftest.py 6 files changed, 69 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/9 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 9 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Use Python from the toolchain for Impala
Laszlo Gaal has posted comments on this change. (
http://gerrit.cloudera.org:8080/15624 )
Change subject: Use Python from the toolchain for Impala
..
Patch Set 10:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG@17
PS6, Line 17: native
> Nit: native
Done
http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG@20
PS6, Line 20: Python2
: version from the toolchain
> This is larger question, but would we ever consider having both python3 and
Python3 is already built in the toolchain in addition to Python2 (see
https://github.com/cloudera/native-toolchain/blob/master/buildall.sh#L71),
although I'm not sure how much testing it gets, if any.
http://gerrit.cloudera.org:8080/#/c/15624/6/bin/set-pythonpath.sh
File bin/set-pythonpath.sh:
http://gerrit.cloudera.org:8080/#/c/15624/6/bin/set-pythonpath.sh@34
PS6, Line 34: PYTHONPATH=${PYTHONPATH}:${IMPALA_HOME}/infra/python/env/lib
> Should this be .../infra/python/env/lib ?
Done
http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py
File infra/python/bootstrap_virtualenv.py:
http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py@50
PS6, Line 50: LOG =
logging.getLogger(os.path.splitext(os.path.basename(__file__))[0])
:
: DEPS_DIR = os.path.join(os.path.dirname(__file__), "deps")
: ENV_DIR = os.path.join(os.path.
> I think this was for my debugging only, so we can remove it.
Done
http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py@92
PS6, Line 92: def exec_cmd(args, **kwargs):
: '''Executes a command and
> Can these lines be removed? I think Tim suggested that we didn't need tmp.p
Done
http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py@209
PS6, Line 209: if not os.path.exists(python_cmd):
> This feels like a bit of a hack, but I think we can live with it.
Ack
http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/tmp.py
File infra/python/tmp.py:
PS6:
> I think we can remove this, this was just for debugging issues I was having
Done
http://gerrit.cloudera.org:8080/#/c/15624/6/tests/common/impala_test_suite.py
File tests/common/impala_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/15624/6/tests/common/impala_test_suite.py@156
PS6, Line 156: cls.client = None
> I almost feel like this check might better be done via a fixture in conftes
Done
--
To view, visit http://gerrit.cloudera.org:8080/15624
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b
Gerrit-Change-Number: 15624
Gerrit-PatchSet: 10
Gerrit-Owner: Laszlo Gaal
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 08 Apr 2020 18:50:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Hello Zoltan Borok-Nagy, David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15623 to look at the new patch set (#9). Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a virtual machine, or inside a Docker container. Details: - curl is added to the list of required packages, required by IMPALA-9149 - snappy-devel moved to the PowerTools repo, so it needs to be installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. The installer is adaptive, it performs only the necessary steps, so it works in various environments. The installer logic is also shared between bin/bootstrap_system.sh and docker/entrypoint.sh - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - For some unknown reason, when the downloaded Maven tarball is extracted in a Docker-based test, the "bin" and "boot" directories are created with owner-only permissions. The 'impdev' users has no access to the maven executable, which then breaks the build. This patch forcibly restores the correct permissions on these directories; this is a no-op when the extraction happens correctly. - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. - Add Centos8-specific bootstrap code to the Docker-based tests. Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 115 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15623/9 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 9 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Use Python from the toolchain for Impala
Hello David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#11). Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing native code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an auto-use fixture to conftest.py to check that the tests are being run with Python 2.7.x - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Tests: - Passed core-mode tests on CentOS 7.4 - Passed core-mode tests in Docker-based mode for centos:7 and ubuntu:16.04 Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M infra/python/bootstrap_virtualenv.py M tests/conftest.py 5 files changed, 68 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/11 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 11 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Use Python from the toolchain for Impala
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15624 ) Change subject: Use Python from the toolchain for Impala .. Patch Set 11: (3 comments) http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG@17 PS6, Line 17: native > Done Done http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG@20 PS6, Line 20: Python2 : version from the toolchain > Python3 is already built in the toolchain in addition to Python2 (see https Done http://gerrit.cloudera.org:8080/#/c/15624/6//COMMIT_MSG@39 PS6, Line 39: > Add details on how it was tested Done -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 11 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 18:55:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Hello Zoltan Borok-Nagy, David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15623 to look at the new patch set (#10). Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a virtual machine, or inside a Docker container. Details: - curl is added to the list of required packages, required by IMPALA-9149 - snappy-devel moved to the PowerTools repo, so it needs to be installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. The installer is adaptive, it performs only the necessary steps, so it works in various environments. The installer logic is also shared between bin/bootstrap_system.sh and docker/entrypoint.sh - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - For some unknown reason, when the downloaded Maven tarball is extracted in a Docker-based test, the "bin" and "boot" directories are created with owner-only permissions. The 'impdev' users has no access to the maven executable, which then breaks the build. This patch forcibly restores the correct permissions on these directories; this is a no-op when the extraction happens correctly. - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. - Add Centos8-specific bootstrap code to the Docker-based tests. Tested: - ran the Docker-based tests with --base-image=centos:8 to verify the following build phases are successful: * system prep * build * dataload and that test can start. Passing all tests is was not a requirement for this step, although plausible test results (i.e. not all of the tests fail) were. - ran the Docker-based tests to verify nonregression with --base-image set to the following: centos:7, ubuntu:16.04, ubuntu:18.04. On centos:7 and ubuntu:16.04 the only failure was IMPALA-9097 (BE tests fail without the minicluster running); ubuntu:18.04 showed the same failures as the current upstream code. - passed a core-mode test run on private infrastructure on Centos 7.4 - ran buildall.sh in core mode manually inside a Docker container, simulating a developer workflow (prep-build-dataload-test). There were several observed test failures, but the workflow itself was run to completion with no problems. Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 115 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15623/10 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 10 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Use Python from the toolchain for Impala
Hello David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#12). Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing native code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an auto-use fixture to conftest.py to check that the tests are being run with Python 2.7.x - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Tests: - Passed core-mode tests on CentOS 7.4 - Passed core-mode tests in Docker-based mode for centos:7 and ubuntu:16.04 Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M infra/python/bootstrap_virtualenv.py M tests/conftest.py 5 files changed, 68 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/12 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 12 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15623 ) Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 10: Code-Review+1 (3 comments) Carry +1 from Tim http://gerrit.cloudera.org:8080/#/c/15623/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15623/5//COMMIT_MSG@37 PS5, Line 37: - Add Centos8-specific bootstrap code to the Docker-based tests. > self-inflicted: add details on how it was tested Done http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG@14 PS6, Line 14: virtua > nit:virtual Done http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG@25 PS6, Line 25: necessar > nit: necessary Done -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 10 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Apr 2020 18:59:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Use Python from the toolchain for Impala
Hello David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#13). Change subject: Use Python from the toolchain for Impala .. Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing native code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an auto-use fixture to conftest.py to check that the tests are being run with Python 2.7.x - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Tests: - Passed core-mode tests on CentOS 7.4 - Passed core-mode tests in Docker-based mode for centos:7 and ubuntu:16.04 Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M infra/python/bootstrap_virtualenv.py M tests/conftest.py 5 files changed, 67 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/13 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 13 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Use Python from the toolchain for Impala
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15624 ) Change subject: Use Python from the toolchain for Impala .. Patch Set 13: (1 comment) Carry +1 from Tim http://gerrit.cloudera.org:8080/#/c/15624/11/tests/conftest.py File tests/conftest.py: http://gerrit.cloudera.org:8080/#/c/15624/11/tests/conftest.py@620 PS11, Line 620: > nit: extra blank line Done -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 13 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 19:01:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Use Python from the toolchain for Impala
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15624 ) Change subject: Use Python from the toolchain for Impala .. Patch Set 13: Note: There is one final update coming before this patch can be submitted: IMPALA_TOOLCHAIN_BUILD_ID needs to be bumped to a published version that has final Python 2.7.16 bits, and contains binaries for all Impala 4.x platforms. -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 13 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 19:04:26 + Gerrit-HasComments: No
[Impala-ASF-CR] Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15623 ) Change subject: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 10: Note: There is one final update coming before this patch can be submitted: IMPALA_TOOLCHAIN_BUILD_ID needs to be bumped to a published version that has final Python 2.7.16 bits, and contains binaries for all Impala 4.x platforms. -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 10 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Apr 2020 19:04:20 + Gerrit-HasComments: No
[native-toolchain-CR] Add build script to create and publish Docker containers
Hello Hector Acosta, David Knupp, Tim Armstrong, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15675 to look at the new patch set (#2). Change subject: Add build script to create and publish Docker containers .. Add build script to create and publish Docker containers Adds automation to build and publish the containers used by the Docker-based toolchain builds. These can be used by the toolchain builds to save time by using pre-created containers instead of building the containers on the fly. They can also be used to provide a stable, controlled environment for the toolchain builds. The script can also be invoked from a Jenkins build environment. Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 --- M docker/buildall.py A jenkins/build.sh 2 files changed, 67 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/75/15675/2 -- To view, visit http://gerrit.cloudera.org:8080/15675 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 Gerrit-Change-Number: 15675 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Add build script to create and publish Docker containers
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15675 ) Change subject: Add build script to create and publish Docker containers .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/15675/1/jenkins/build.sh File jenkins/build.sh: http://gerrit.cloudera.org:8080/#/c/15675/1/jenkins/build.sh@21 PS1, Line 21: # the extension '.df' > This defaults to All, right? Done http://gerrit.cloudera.org:8080/#/c/15675/1/jenkins/build.sh@43 PS1, Line 43: > nit: I think this should probably be quoted, otherwise it could be expanded Yes, of course; it was even quoted in line 52. rewrote it to have only a single definition of the pattern. -- To view, visit http://gerrit.cloudera.org:8080/15675 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 Gerrit-Change-Number: 15675 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 22:38:09 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add build script to create and publish Docker containers
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15675 ) Change subject: Add build script to create and publish Docker containers .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/15675/1/jenkins/build.sh File jenkins/build.sh: http://gerrit.cloudera.org:8080/#/c/15675/1/jenkins/build.sh@26 PS1, Line 26: leave the > resulting Done -- To view, visit http://gerrit.cloudera.org:8080/15675 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 Gerrit-Change-Number: 15675 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 22:40:30 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add build script to create and publish Docker containers
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15675 ) Change subject: Add build script to create and publish Docker containers .. Patch Set 3: Verified+1 Verified by running the script through a matched Jenkins job in all the main modes of operation: - building the container images from scratch - forced rebuilding of the images - publishing the images to a test Docker repo in a private registry, and building the toolchain inside those containers -- To view, visit http://gerrit.cloudera.org:8080/15675 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 Gerrit-Change-Number: 15675 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 Apr 2020 08:46:47 + Gerrit-HasComments: No
[native-toolchain-CR] Add build script to create and publish Docker containers
Laszlo Gaal has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15675 ) Change subject: Add build script to create and publish Docker containers .. Add build script to create and publish Docker containers Adds automation to build and publish the containers used by the Docker-based toolchain builds. These can be used by the toolchain builds to save time by using pre-created containers instead of building the containers on the fly. They can also be used to provide a stable, controlled environment for the toolchain builds. The script can also be invoked from a Jenkins build environment. Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 Reviewed-on: http://gerrit.cloudera.org:8080/15675 Reviewed-by: Tim Armstrong Tested-by: Laszlo Gaal --- M docker/buildall.py A jenkins/build.sh 2 files changed, 67 insertions(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved Laszlo Gaal: Verified -- To view, visit http://gerrit.cloudera.org:8080/15675 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 Gerrit-Change-Number: 15675 Gerrit-PatchSet: 4 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9626: Use Python from the toolchain for Impala
Hello David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15624 to look at the new patch set (#14). Change subject: IMPALA-9626: Use Python from the toolchain for Impala .. IMPALA-9626: Use Python from the toolchain for Impala Historically Impala used the Python2 version that was available on the hosting platform, as long as that version was at least v2.6. This caused constant headache as all Python syntax had to be kept compatible with Python 2.6 (for Centos 6). It also caused a recent problem on Centos 8: here the system Python version was compiled with the system's GCC version (v8.3), which was much more recent than the Impala standard compiler version (GCC 4.9.2). When the Impala virtualenv was built, the system Python version supplied C compiler switches for models containing native code that were unknown for the Impala version of GCC, thus breaking virtualenv installation. This patch changes the Impala virtualenv to always use the Python2 version from the toolchain, which is built with the toolchain compiler. This ensures that - Impala always has a known Python 2.7 version for all its scripts, - virtualenv modules based on native code will always be installable, as the Python environment and the modules are built with the same compiler version. Additional changes: - Add an auto-use fixture to conftest.py to check that the tests are being run with Python 2.7.x - Make bootstrap_toolchain.py independent from the Impala virtualenv: remove the dependency on the "sh" library Tests: - Passed core-mode tests on CentOS 7.4 - Passed core-mode tests in Docker-based mode for centos:7 and ubuntu:16.04 Most content in this patch was developed but not published earlier by Tim Armstrong. Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-pythonpath.sh M infra/python/bootstrap_virtualenv.py M tests/conftest.py 5 files changed, 67 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/15624/14 -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 14 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9629: Add CentOS 8.1 support to bootstrap system.sh
Hello Zoltan Borok-Nagy, David Knupp, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15623 to look at the new patch set (#11). Change subject: IMPALA-9629: Add CentOS 8.1 support to bootstrap_system.sh .. IMPALA-9629: Add CentOS 8.1 support to bootstrap_system.sh CentOS 8.1 is a new major version of the CentOS family. It is now stable and popular enough to start supporting it for Impala development. Prepare a raw CentOS 8.1 system to support Impala development and testing. This should work on a standalone computer, on a virtual machine, or inside a Docker container. Details: - snappy-devel moved to the PowerTools repo, so it needs to be installed from there - CentOS 8 has no default Python version. The bootstrap script installs (or configures) Python2 with pip2, then makes them the default via the "alternatives" mechanism. The installer is adaptive, it performs only the necessary steps, so it works in various environments. The installer logic is also shared between bin/bootstrap_system.sh and docker/entrypoint.sh - The toolchain package tag "ec2-centos-8" is added to bootstrap_toolchain.py - For some unknown reason, when the downloaded Maven tarball is extracted in a Docker-based test, the "bin" and "boot" directories are created with owner-only permissions. The 'impdev' users has no access to the maven executable, which then breaks the build. This patch forcibly restores the correct permissions on these directories; this is a no-op when the extraction happens correctly. - TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries. - Centos8-specific bootstrap code was added to the Docker-based tests. Tested: - ran the Docker-based tests with --base-image=centos:8 to verify the following build phases are successful: * system prep * build * dataload and that test can start. Passing all tests is was not a requirement for this step, although plausible test results (i.e. not all of the tests fail) were. - ran the Docker-based tests to verify nonregression with --base-image set to the following: centos:7, ubuntu:16.04, ubuntu:18.04. On centos:7 and ubuntu:16.04 the only failure was IMPALA-9097 (BE tests fail without the minicluster running); ubuntu:18.04 showed the same failures as the current upstream code. - passed a core-mode test run on private infrastructure on Centos 7.4 - ran buildall.sh in core mode manually inside a Docker container, simulating a developer workflow (prep-build-dataload-test). There were several observed test failures, but the workflow itself was run to completion with no problems. Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M docker/entrypoint.sh 4 files changed, 116 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/15623/11 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 11 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9629: Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15623 ) Change subject: IMPALA-9629: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 12: Code-Review+1 (3 comments) IMPALA_TOOLCHAIN_BUILD_ID bumped to an officially published one containing Centos8 binaries. Carry +1 from Tim http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15623/6//COMMIT_MSG@18 PS6, Line 18: snappy-devel moved to the PowerTools repo, so it needs to be installed : from there > delete this: curl was added for the no-NTP patch Done http://gerrit.cloudera.org:8080/#/c/15623/5/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/15623/5/bin/bootstrap_system.sh@266 PS5, Line 266: # 1. Link pip's version to Python's version > TL;DR: Impala (minus the shell) is still Python2 dependent, and Impala deve Filed IMPALA-9627 for the utility scripts http://gerrit.cloudera.org:8080/#/c/15623/5/bin/bootstrap_system.sh@367 PS5, Line 367: "pg_ctl start -w --timeout=120 >> /var/lib/pgsql/pg.log 2>&1" > Yes, the support matrix grew large enough (redhat[6]/7/8, Ubuntu 14/16/18[/ filed IMPALA-9628 -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 12 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 10 Apr 2020 17:12:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9626: Use Python from the toolchain for Impala
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15624 ) Change subject: IMPALA-9626: Use Python from the toolchain for Impala .. Patch Set 15: Code-Review+1 (2 comments) Carry +1 from Tim http://gerrit.cloudera.org:8080/#/c/15624/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15624/11//COMMIT_MSG@7 PS11, Line 7: IMPALA-9626: Use Python from the toolchain for Impala > I think this is significant enough to deserve a JIRA. I filed IMPALA-9626 Thank you. Added the Jira reference to the commit message. http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py File infra/python/bootstrap_virtualenv.py: http://gerrit.cloudera.org:8080/#/c/15624/6/infra/python/bootstrap_virtualenv.py@211 PS6, Line 211: : def install_deps(): > Thanks for the explanation. I think have a marginal preference for placing Done -- To view, visit http://gerrit.cloudera.org:8080/15624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7b40cef89cfb3b467b61b2d54a94e708642882b Gerrit-Change-Number: 15624 Gerrit-PatchSet: 15 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 10 Apr 2020 17:37:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9629: Add CentOS 8.1 support to bootstrap system.sh
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15623 ) Change subject: IMPALA-9629: Add CentOS 8.1 support to bootstrap_system.sh .. Patch Set 12: > Patch Set 12: Code-Review+2 > > (1 comment) The concerns are perfectly valid, and I don't consider this patch to be the ideal solution either. The way I see it it's a manageable first step that enables Impala on Centos8 without regressing on platforms already supported. As such, it opens the way for further improvements, but as with every first step, there are caveats. I also assume (maybe wrongly) that any developer moving to CentOS 8 will be exposed to the Python version dilemma anyway; CentOS (or RedHat) essentially ensured that they can't away from it. They will be aware of the problem, and they would (hopefully) be able to manage it. With that said, I also agree that nailing down the default version of Python is a pretty drastic step; I just hope that follow-up steps will quickly resolve the issues with the Python scripts so that we can get rid of this limitation. As for the scripts and their default environments: I tend to group them into two categories (loosely): - scripts performing some function inside the Impala dev environment, which are reasonably coupled to the Impala virtualenv. - scripts that would be nice to execute without an long setup process, even in a freshly cloned repo. An good example would be https://github.com/apache/impala/blob/master/docker/test-with-docker.py#L21, where we state this explicitly; bootstrap_toolchain.py is another good example. Ideally these scripts could run either with Python2 or Python3, or they should be tied explicitly to one of the major versions. I have already filed IMPALA-9627 (with you on cc:) to improve the utility scripts to be able to let people choose their Python version freely -- maybe that would be a better place to continue the discussion about the details. -- To view, visit http://gerrit.cloudera.org:8080/15623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e Gerrit-Change-Number: 15623 Gerrit-PatchSet: 12 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 Apr 2020 13:03:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9668: Obey SKIP TOOLCHAIN BOOTSTRAP during virtualenv bootstrap
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15759 Change subject: IMPALA-9668: Obey SKIP_TOOLCHAIN_BOOTSTRAP during virtualenv bootstrap .. IMPALA-9668: Obey SKIP_TOOLCHAIN_BOOTSTRAP during virtualenv bootstrap IMPALA-9626 broke the use case where the toolchain binaries are not downloaded from the native-toolchain S3 bucket, because SKIP_TOOLCHAIN_BOOTSTRAP is set to true. Fix this use case by checking SKIP_TOOLCHAIN_BOOTSTRAP in bin/bootstrap_environment.py: - if true: just check if the specified version of the Python binary is present at the expected toolchain location. If it is there, use it, otherwise throw an exception and abort the bootstrap process. - in any other case: proceed to download the Python binary as in bootstrap_toolchain.py. Test: - simulate the custom toolchain setup by downloading the toolchain binaries from the S3 bucket, copying them to a separate directory, symlinking them into Impala/toolchain, then executing buildall.sh with SKIP_BOOTSTRAP_TOOLCHAIN set to "true". Change-Id: Ic51b3c327b3cebc08edff90de931d07e35e0c319 --- M bin/bootstrap_toolchain.py M infra/python/bootstrap_virtualenv.py 2 files changed, 8 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/15759/1 -- To view, visit http://gerrit.cloudera.org:8080/15759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic51b3c327b3cebc08edff90de931d07e35e0c319 Gerrit-Change-Number: 15759 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] IMPALA-9668: Obey SKIP TOOLCHAIN BOOTSTRAP during virtualenv bootstrap
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15759 ) Change subject: IMPALA-9668: Obey SKIP_TOOLCHAIN_BOOTSTRAP during virtualenv bootstrap .. Patch Set 2: Code-Review+2 Rebased, otherwise unchanged, carry +2 from David -- To view, visit http://gerrit.cloudera.org:8080/15759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic51b3c327b3cebc08edff90de931d07e35e0c319 Gerrit-Change-Number: 15759 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 22 Apr 2020 17:30:07 + Gerrit-HasComments: No
[native-toolchain-CR] Clone the Kudu repo using HTTPS by default
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15832 Change subject: Clone the Kudu repo using HTTPS by default .. Clone the Kudu repo using HTTPS by default Since GitHub started enforcing TLSv1.2 for HTTPS connections, the Kudu repo on Github was cloned via SSH. Unfortunately the default SSH access used for this depended on obscure default credentials, which failed to work in some of the cases. WIth Impala 4.x dropping support for older OS platforms, now all the supported OSs are capable of TLSv1.2, so it is safe to reset default access to HTTPS. Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 --- M source/kudu/build.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/32/15832/1 -- To view, visit http://gerrit.cloudera.org:8080/15832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Gerrit-Change-Number: 15832 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[native-toolchain-CR] IMPALA-9686: Add readline to Python
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15831 Change subject: IMPALA-9686: Add readline to Python .. IMPALA-9686: Add readline to Python Add the readline developer packages to the toolchain build containers. These are needed by the Python build, as it figures out which packages to build during compilation time by going through the environment and finding these packages dynamically. Any packages that miss their runtime dependencies are dropped from the build; the only sign is some warning messages in the build log. SLES12 puts ncurses include file in a separate subdirectory under /usr/include. This breaks compiling readline on SLES12 if the extra directoyr is not added to the include path, so fix this in Python's build.sh Runtime asserts were also added to the Python build driver for the issues uncovered so far. Tests: - built the container images - built the toolchain in the new container configs - tested Python from the new build for the presence of readline support Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 --- M docker/debian8.df M docker/redhat7.df M docker/redhat8.df M docker/sles12.df M docker/ubuntu1604.df M docker/ubuntu1804.df M source/python/build.sh 7 files changed, 9 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/31/15831/1 -- To view, visit http://gerrit.cloudera.org:8080/15831 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 Gerrit-Change-Number: 15831 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[native-toolchain-CR] Clone the Kudu repo using HTTPS by default
Laszlo Gaal has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15832 ) Change subject: Clone the Kudu repo using HTTPS by default .. Clone the Kudu repo using HTTPS by default Since GitHub started enforcing TLSv1.2 for HTTPS connections, the Kudu repo on Github was cloned via SSH. Unfortunately the default SSH access used for this depended on obscure default credentials, which failed to work in some of the cases. WIth Impala 4.x dropping support for older OS platforms, now all the supported OSs are capable of TLSv1.2, so it is safe to reset default access to HTTPS. Tested by rebuilding the toolchain in a local Docker container; verified that cloning teh Kudu repo is successful with no user interaction required. Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 --- M source/kudu/build.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/32/15832/2 -- To view, visit http://gerrit.cloudera.org:8080/15832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Gerrit-Change-Number: 15832 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal
[native-toolchain-CR] IMPALA-9686: Add readline to Python
Laszlo Gaal has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15831 ) Change subject: IMPALA-9686: Add readline to Python .. IMPALA-9686: Add readline to Python Add the readline developer packages to the toolchain build containers. These are needed by the Python build, as it figures out which packages to build during compilation time by going through the environment and finding these packages dynamically. Any packages that miss their runtime dependencies are dropped from the build; the only sign is some warning messages in the build log. SLES12 puts ncurses include file in a separate subdirectory under /usr/include. This breaks compiling readline on SLES12 if the extra directoyr is not added to the include path, so fix this in Python's build.sh Runtime asserts were also added to the Python build driver for the issues uncovered so far. Tests: - built the container images - built the toolchain in the new container configs - tested Python from the new build for the presence of readline support Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 --- M docker/debian8.df M docker/redhat7.df M docker/redhat8.df M docker/sles12.df M docker/ubuntu1604.df M docker/ubuntu1804.df M source/python/build.sh 7 files changed, 9 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/31/15831/2 -- To view, visit http://gerrit.cloudera.org:8080/15831 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 Gerrit-Change-Number: 15831 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] Bump test-with-docker test concurrency for large instances
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15834 Change subject: Bump test-with-docker test concurrency for large instances .. Bump test-with-docker test concurrency for large instances The concurrency limits (i.e. how many concurrent Docker containers are running test shards at the same time) were conservative at the high end: the largest memory configuration they considered was under 100 GBs. Bump these limits for the usual m5.12xlarge test worker that has 192 GBs of RAM, of which about 186 GBs are available. Also, swap the order of FE and BE tests, as FE tests have now grown pretty long with the long delay in AuthorizationStmtTest. Test: ran test-with-docker.py with all default parameters. Verified that default concurrency was 6 on an m5.12xlarge and core-mode tests passed in an Ubuntu 16.04 container. Change-Id: I5c03a78ee65d09212d9bfa007e87fd069cdaabb6 --- M docker/test-with-docker.py 1 file changed, 5 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/15834/1 -- To view, visit http://gerrit.cloudera.org:8080/15834 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5c03a78ee65d09212d9bfa007e87fd069cdaabb6 Gerrit-Change-Number: 15834 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[native-toolchain-CR] IMPALA-9686: Add readline to Python
Hello Hector Acosta, David Knupp, Tim Armstrong, Wenzhe Zhou, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15831 to look at the new patch set (#3). Change subject: IMPALA-9686: Add readline to Python .. IMPALA-9686: Add readline to Python Add the readline developer packages to the toolchain build containers. These are needed by the Python build, as it figures out which packages to build during compilation time by going through the environment and finding these packages dynamically. Any packages that miss their runtime dependencies are dropped from the build; the only sign is some warning messages in the build log. SLES12 puts ncurses include file in a separate subdirectory under /usr/include. This breaks compiling readline on SLES12 if the extra directoyr is not added to the include path, so fix this in Python's build.sh Runtime asserts were also added to the Python build driver to check that the following modules were built successfully: - readline - ssl - bz2 Tests: - built the container images - built the toolchain in the new container configs - tested Python from the new build for the presence of readline support Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 --- M docker/debian8.df M docker/redhat7.df M docker/redhat8.df M docker/sles12.df M docker/ubuntu1604.df M docker/ubuntu1804.df M source/python/build.sh 7 files changed, 12 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/31/15831/3 -- To view, visit http://gerrit.cloudera.org:8080/15831 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 Gerrit-Change-Number: 15831 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou
[native-toolchain-CR] IMPALA-9686: Add readline to Python
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15831 ) Change subject: IMPALA-9686: Add readline to Python .. Patch Set 3: (1 comment) Another test (nonpublishing) build succeeded with the Python end-of-build asserts added on Centos 7 and 8, Ubuntu 16.04 and 18.04 and SLES12 sp3. http://gerrit.cloudera.org:8080/#/c/15831/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15831/2//COMMIT_MSG@21 PS2, Line 21: Runtime asserts were also added to the Python build driver to check > Which were these? Thanks for catching this; the asserts got lost when I switched branches. Added them to source/python/build.sh. -- To view, visit http://gerrit.cloudera.org:8080/15831 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 Gerrit-Change-Number: 15831 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 29 Apr 2020 22:15:09 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-9686: Add readline to Python
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15831 ) Change subject: IMPALA-9686: Add readline to Python .. Patch Set 4: Verified+1 Code-Review+2 Rebased, unchanged, carry +2 from Tim Verified by a repeated full build with the new Kudu githash from the rebase. -- To view, visit http://gerrit.cloudera.org:8080/15831 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 Gerrit-Change-Number: 15831 Gerrit-PatchSet: 4 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 30 Apr 2020 15:23:07 + Gerrit-HasComments: No
[native-toolchain-CR] Clone the Kudu repo using HTTPS by default
Hello Thomas Tauber-Marshall, Zoltan Borok-Nagy, Attila Jeges, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15832 to look at the new patch set (#5). Change subject: Clone the Kudu repo using HTTPS by default .. Clone the Kudu repo using HTTPS by default Since GitHub started enforcing TLSv1.2 for HTTPS connections, the Kudu repo on Github was cloned via SSH. Unfortunately the default SSH access used for this depended on obscure default credentials, which failed to work in some of the cases. WIth Impala 4.x dropping support for older OS platforms, now all the supported OSs are capable of TLSv1.2, so it is safe to reset default access to HTTPS. Tested by rebuilding the toolchain in a local Docker container; verified that cloning teh Kudu repo is successful with no user interaction required. Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 --- M source/kudu/build.sh 1 file changed, 2 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/32/15832/5 -- To view, visit http://gerrit.cloudera.org:8080/15832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Gerrit-Change-Number: 15832 Gerrit-PatchSet: 5 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[native-toolchain-CR] Clone the Kudu repo using HTTPS by default
Hello Thomas Tauber-Marshall, Zoltan Borok-Nagy, Attila Jeges, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15832 to look at the new patch set (#6). Change subject: Clone the Kudu repo using HTTPS by default .. Clone the Kudu repo using HTTPS by default Since GitHub started enforcing TLSv1.2 for HTTPS connections, the Kudu repo on Github was cloned via SSH. Unfortunately the default SSH access used for this depended on obscure default credentials, which failed to work in some of the cases. WIth Impala 4.x dropping support for older OS platforms, now all the supported OSs are capable of TLSv1.2, so it is safe to reset default access to HTTPS. Tested by rebuilding the toolchain in a local Docker container; verified that cloning the Kudu repo is successful with no user interaction required. Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 --- M source/kudu/build.sh 1 file changed, 2 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/32/15832/6 -- To view, visit http://gerrit.cloudera.org:8080/15832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Gerrit-Change-Number: 15832 Gerrit-PatchSet: 6 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[native-toolchain-CR] Clone the Kudu repo using HTTPS by default
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15832 ) Change subject: Clone the Kudu repo using HTTPS by default .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/15832/2/source/kudu/build.sh File source/kudu/build.sh: http://gerrit.cloudera.org:8080/#/c/15832/2/source/kudu/build.sh@72 PS2, Line 72: > Please update comment Done -- To view, visit http://gerrit.cloudera.org:8080/15832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Gerrit-Change-Number: 15832 Gerrit-PatchSet: 5 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 30 Apr 2020 15:43:58 + Gerrit-HasComments: Yes
[native-toolchain-CR] Clone the Kudu repo using HTTPS by default
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15832 ) Change subject: Clone the Kudu repo using HTTPS by default .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/15832/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15832/5//COMMIT_MSG@18 PS5, Line 18: the nit: typo -- To view, visit http://gerrit.cloudera.org:8080/15832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Gerrit-Change-Number: 15832 Gerrit-PatchSet: 6 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 30 Apr 2020 15:44:36 + Gerrit-HasComments: Yes
[native-toolchain-CR] Clone the Kudu repo using HTTPS by default
Laszlo Gaal has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15832 ) Change subject: Clone the Kudu repo using HTTPS by default .. Clone the Kudu repo using HTTPS by default Since GitHub started enforcing TLSv1.2 for HTTPS connections, the Kudu repo on Github was cloned via SSH. Unfortunately the default SSH access used for this depended on obscure default credentials, which failed to work in some of the cases. WIth Impala 4.x dropping support for older OS platforms, now all the supported OSs are capable of TLSv1.2, so it is safe to reset default access to HTTPS. Tested by rebuilding the toolchain in a local Docker container; verified that cloning the Kudu repo is successful with no user interaction required. Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Reviewed-on: http://gerrit.cloudera.org:8080/15832 Reviewed-by: Laszlo Gaal Tested-by: Laszlo Gaal --- M source/kudu/build.sh 1 file changed, 2 insertions(+), 3 deletions(-) Approvals: Laszlo Gaal: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Gerrit-Change-Number: 15832 Gerrit-PatchSet: 7 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[native-toolchain-CR] Clone the Kudu repo using HTTPS by default
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15832 ) Change subject: Clone the Kudu repo using HTTPS by default .. Patch Set 6: Verified+1 Code-Review+2 (1 comment) Carry +2 from Tim Verified by a full build for all platforms with the rebased Kudu git hash. http://gerrit.cloudera.org:8080/#/c/15832/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15832/5//COMMIT_MSG@18 PS5, Line 18: the > nit: typo Done -- To view, visit http://gerrit.cloudera.org:8080/15832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2486b34e7ae62eecb49ab0a602308b9fe898c8d0 Gerrit-Change-Number: 15832 Gerrit-PatchSet: 6 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 30 Apr 2020 15:45:29 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-9686: Add readline to Python
Laszlo Gaal has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15831 ) Change subject: IMPALA-9686: Add readline to Python .. IMPALA-9686: Add readline to Python Add the readline developer packages to the toolchain build containers. These are needed by the Python build, as it figures out which packages to build during compilation time by going through the environment and finding these packages dynamically. Any packages that miss their runtime dependencies are dropped from the build; the only sign is some warning messages in the build log. SLES12 puts ncurses include file in a separate subdirectory under /usr/include. This breaks compiling readline on SLES12 if the extra directoyr is not added to the include path, so fix this in Python's build.sh Runtime asserts were also added to the Python build driver to check that the following modules were built successfully: - readline - ssl - bz2 Tests: - built the container images - built the toolchain in the new container configs - tested Python from the new build for the presence of readline support Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 Reviewed-on: http://gerrit.cloudera.org:8080/15831 Reviewed-by: Laszlo Gaal Tested-by: Laszlo Gaal --- M docker/debian8.df M docker/redhat7.df M docker/redhat8.df M docker/sles12.df M docker/ubuntu1604.df M docker/ubuntu1804.df M source/python/build.sh 7 files changed, 12 insertions(+), 3 deletions(-) Approvals: Laszlo Gaal: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15831 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5bbe6fa16305a4462278306947535fad468cb3d8 Gerrit-Change-Number: 15831 Gerrit-PatchSet: 5 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou
[native-toolchain-CR] Fix Debian 8 Dockerfile to handle expired repos
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15843 Change subject: Fix Debian 8 Dockerfile to handle expired repos .. Fix Debian 8 Dockerfile to handle expired repos Debian 8 (jessie) is nearing end-of-life (set for 2020-06-30), and repo keys have started expiring. Update keys and source lists to keep the Dockerfile working. Patch is based onthe following articles: - https://unix.stackexchange.com/a/508728/2419 - https://linux-audit.com/how-to-solve-an-expired-key-keyexpired-with-apt/ Change-Id: Ic1fe3a43cc1ef753d63d8fd76a029cf9ef657219 --- M docker/debian/8/99no-check-valid-until M docker/debian/8/sources.list M docker/debian8.df 3 files changed, 13 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/43/15843/1 -- To view, visit http://gerrit.cloudera.org:8080/15843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic1fe3a43cc1ef753d63d8fd76a029cf9ef657219 Gerrit-Change-Number: 15843 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] Bump Kudu to 389d4f1e1, fix toolchain Python
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15845 Change subject: Bump Kudu to 389d4f1e1, fix toolchain Python .. Bump Kudu to 389d4f1e1, fix toolchain Python Pick up the rebuilt toolchain Python, now with readline support. Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/15845/1 -- To view, visit http://gerrit.cloudera.org:8080/15845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 Gerrit-Change-Number: 15845 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] Bump Kudu to 389d4f1e1, fix toolchain Python
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15845 ) Change subject: Bump Kudu to 389d4f1e1, fix toolchain Python .. Patch Set 2: Rebased, unchanged, carry +2 from Thomas -- To view, visit http://gerrit.cloudera.org:8080/15845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 Gerrit-Change-Number: 15845 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 05 May 2020 11:39:44 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump Kudu to 389d4f1e1, fix toolchain Python
Hello Thomas Tauber-Marshall, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15845 to look at the new patch set (#3). Change subject: Bump Kudu to 389d4f1e1, fix toolchain Python .. Bump Kudu to 389d4f1e1, fix toolchain Python Pick up the rebuilt toolchain Python, now with readline support. The new toolchain version contains a newer Kudu version, so bump the Kudu versions to the one too. Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 --- M bin/impala-config.sh 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/15845/3 -- To view, visit http://gerrit.cloudera.org:8080/15845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 Gerrit-Change-Number: 15845 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] Bump Kudu to 389d4f1e1, fix toolchain Python
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15845 ) Change subject: Bump Kudu to 389d4f1e1, fix toolchain Python .. Patch Set 4: @Thomas, @Wenzhe: the latest patch takes Kudu from 1.12 to 1.13, because that's the version present in the toolchain build I'm picking up. Could you please comment if this is planned and/or desirable? Thank you! -- To view, visit http://gerrit.cloudera.org:8080/15845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 Gerrit-Change-Number: 15845 Gerrit-PatchSet: 4 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 13 May 2020 15:03:40 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump test-with-docker test concurrency for large instances
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15834 ) Change subject: Bump test-with-docker test concurrency for large instances .. Patch Set 3: Rebased, unchanged, carry +2 from Joe -- To view, visit http://gerrit.cloudera.org:8080/15834 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c03a78ee65d09212d9bfa007e87fd069cdaabb6 Gerrit-Change-Number: 15834 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Wed, 13 May 2020 19:47:53 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump test-with-docker test concurrency for large instances
Laszlo Gaal has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15834 ) Change subject: Bump test-with-docker test concurrency for large instances .. Bump test-with-docker test concurrency for large instances The concurrency limits (i.e. how many concurrent Docker containers are running test shards at the same time) were conservative at the high end: the largest memory configuration they considered was under 100 GBs. Bump these limits for the usual m5.12xlarge test worker that has 192 GBs of RAM, of which about 186 GBs are available. Also, swap the order of FE and BE tests, as FE tests have now grown pretty long with the long delay in AuthorizationStmtTest. Test: ran test-with-docker.py with all default parameters. Verified that default concurrency was 6 on an m5.12xlarge and core-mode tests passed in an Ubuntu 16.04 container. Change-Id: I5c03a78ee65d09212d9bfa007e87fd069cdaabb6 Reviewed-on: http://gerrit.cloudera.org:8080/15834 Tested-by: Impala Public Jenkins Reviewed-by: Laszlo Gaal --- M docker/test-with-docker.py 1 file changed, 5 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Verified Laszlo Gaal: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15834 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5c03a78ee65d09212d9bfa007e87fd069cdaabb6 Gerrit-Change-Number: 15834 Gerrit-PatchSet: 4 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Bump test-with-docker test concurrency for large instances
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15834 ) Change subject: Bump test-with-docker test concurrency for large instances .. Patch Set 3: Code-Review+2 Carry +2 from Joe, this time with vote added too :) -- To view, visit http://gerrit.cloudera.org:8080/15834 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c03a78ee65d09212d9bfa007e87fd069cdaabb6 Gerrit-Change-Number: 15834 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Thu, 14 May 2020 08:20:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9192: Move Avro-Java and Parquet dependencies to the CDP version
Hello Joe McDonnell, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/15933 to review the following change. Change subject: IMPALA-9192: Move Avro-Java and Parquet dependencies to the CDP version .. IMPALA-9192: Move Avro-Java and Parquet dependencies to the CDP version IMPALA-9731 adopted the CDP version of many Hadoop dependencies. This patch moves the Avro and Parquet Java components to their CDP versions so that they are aligned with the other Hadoop components. Test: Ran tests successfully in core mode. Change-Id: I49c7c5832b5ba53a00b098642f6c64616eb944bd --- M bin/impala-config.sh M fe/pom.xml M impala-parent/pom.xml M testdata/pom.xml 4 files changed, 31 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/15933/1 -- To view, visit http://gerrit.cloudera.org:8080/15933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I49c7c5832b5ba53a00b098642f6c64616eb944bd Gerrit-Change-Number: 15933 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 3: (2 comments) Went through the logic; found a preexisting failure case not even described in the original ticket. http://gerrit.cloudera.org:8080/#/c/15898/3/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/15898/3/docker/entrypoint.sh@319 PS3, Line 319: copy_cluster_logs I'd recommend moving this line before L317: according to its comments, that line attempts to remove the very symlinks that copy_cluster_logs" is trying to save for later (although the exact flag and predicate combination seems to be wrong there). http://gerrit.cloudera.org:8080/#/c/15898/3/docker/entrypoint.sh@319 PS3, Line 319: copy_cluster_logs There is also another problem here (not covered in the original ticket): For each instantiation of the container (whether for build or test purposes) the driver script test_with_docker.py mounts a different log directory at the containers "/log" mount point (see L580-581: https://github.com/apache/impala/blob/master/docker/test-with-docker.py#L580, L655, L815 https://github.com/apache/impala/blob/master/docker/test-with-docker.py#L815). This means anything created there from within the container will not be present in the log tree of a different container run, unless there is a mechanism in test-with-docker.py that replicates or recreates the artifact. Add to this that the cluster log symlinks are created by testdata/cluster/admin in the function create_cluster(), which is run only in the build container, but not in any of the test containers. This has the end result that none of the test containers will actually contain the symlinks from /logs/cluster to the node logs, because the test containers run only testdata/cluster/admin start_cluster(), which does not ensure that the symlinks are present. I see two ways out of this. a) either fix testdata/cluster/admin so that start_cluster() (re)creates these symlink if they are gone b) or fix entrypoint.sh to the same effect. Once this is fixed, then the call to copy_cluster_logs should also be invoked from boot_container() (or possibly from test_suite() ) to ensure that the symlinks are there to capture the log files, so that clpy_cluster_logs can later pick them up and archive them. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Mon, 18 May 2020 17:40:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump Kudu to 389d4f1e1, fix toolchain Python
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15845 ) Change subject: Bump Kudu to 389d4f1e1, fix toolchain Python .. Patch Set 6: Code-Review+2 Rebased across recent changes, the patch proper is unchanged. Carry the +2 from Thomas. -- To view, visit http://gerrit.cloudera.org:8080/15845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 Gerrit-Change-Number: 15845 Gerrit-PatchSet: 6 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 29 May 2020 15:11:53 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump Kudu to 389d4f1e1, fix toolchain Python
Laszlo Gaal has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15845 ) Change subject: Bump Kudu to 389d4f1e1, fix toolchain Python .. Bump Kudu to 389d4f1e1, fix toolchain Python Pick up the rebuilt toolchain Python, now with readline support. The new toolchain version contains a newer Kudu version, so bump the Kudu versions to the one too. Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 Reviewed-on: http://gerrit.cloudera.org:8080/15845 Tested-by: Impala Public Jenkins Reviewed-by: Laszlo Gaal --- M bin/impala-config.sh 1 file changed, 3 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Laszlo Gaal: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idcc2d95e2ca4d9819a6cc0b4eb2865d60e21f153 Gerrit-Change-Number: 15845 Gerrit-PatchSet: 7 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9192: Move Avro-Java and Parquet dependencies to the CDP version
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15933 to look at the new patch set (#2). Change subject: IMPALA-9192: Move Avro-Java and Parquet dependencies to the CDP version .. IMPALA-9192: Move Avro-Java and Parquet dependencies to the CDP version IMPALA-9731 adopted the CDP version of many Hadoop dependencies. This patch moves the Avro and Parquet Java components to their CDP versions so that they are aligned with the other Hadoop components. Test: Ran tests successfully in core mode. Change-Id: I49c7c5832b5ba53a00b098642f6c64616eb944bd --- M bin/impala-config.sh M fe/pom.xml M impala-parent/pom.xml M testdata/pom.xml 4 files changed, 25 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/15933/2 -- To view, visit http://gerrit.cloudera.org:8080/15933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I49c7c5832b5ba53a00b098642f6c64616eb944bd Gerrit-Change-Number: 15933 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-9192: Move Avro-Java and Parquet dependencies to the CDP version
Laszlo Gaal has posted comments on this change. (
http://gerrit.cloudera.org:8080/15933 )
Change subject: IMPALA-9192: Move Avro-Java and Parquet dependencies to the CDP
version
..
Patch Set 2: Code-Review+2
(1 comment)
Carry +2 from Joe after removing the unneeded _URL shell variables. Again a
core mode build was successful.
http://gerrit.cloudera.org:8080/#/c/15933/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/15933/1/bin/impala-config.sh@223
PS1, Line 223: export CDP_HIVE_SOURCE_URL=${CDP_HIVE_SOURCE_U
> Nit: The *_URL env variables are used by bootstrap_toolchain.py. If we aren
Done
--
To view, visit http://gerrit.cloudera.org:8080/15933
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I49c7c5832b5ba53a00b098642f6c64616eb944bd
Gerrit-Change-Number: 15933
Gerrit-PatchSet: 2
Gerrit-Owner: Laszlo Gaal
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Comment-Date: Tue, 09 Jun 2020 22:22:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9845: Point Maven and Ant downloads to stable locations
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16062 Change subject: IMPALA-9845: Point Maven and Ant downloads to stable locations .. IMPALA-9845: Point Maven and Ant downloads to stable locations Ant released a new version in May 2020, which made the URL in bootstrap_system.sh obsolete. At the same time Apache created new rules for the download locations, moving older releases to archive.apache.org. This patch changes the download URLs for Maven and Ant to point to the stable locations at archive.apache.org. These locations don't change when a new version of a project is released, so downloads pulling a specific version will not be affected by a new release. At the same time new releases are stored at the archive site as well, so this location works for all versions. Change-Id: I1875f260b931ef096fc91a4723f91310225c55c9 --- M bin/bootstrap_system.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/16062/1 -- To view, visit http://gerrit.cloudera.org:8080/16062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1875f260b931ef096fc91a4723f91310225c55c9 Gerrit-Change-Number: 16062 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
