As for IMPALA_LZO and the Jenkins build script: cdh5-trunk is an ancient branch of Impala-lzo, it supported the 2.x branch of Impala. The last commit on impala-lzo/cdh5-trunk is from 10/18/2018, which is roughly the same timeframe when the community agreed to stop active maintenance on the Impala 2.x branch; e.g. see Philip's mail at https://lists.apache.org/thread/lghnszw8y2lyqr12zr477jhyy6m1jl7o
My proposal (and offer) for the Jenkins build script is to change the default branch it clones to asf-3.4 per Quanlong's investigation above. I'll change it later today unless someone objects. Thank you, - LaszloG On Fri, Mar 25, 2022 at 4:35 AM Quanlong Huang <huangquanl...@gmail.com> wrote: > Thank Zoltan and Laszlo for checking this! > > For the git-ignore issue mentioned by Zoltan, I think we should remove > the line of "thirdparty" in .gitignore. All files in be/src/thirdparty > are tracked in the git index. In my local repo, if I add --no-index > option to git-check-ignore, I can see thirdparty is ignored as what we > saw in the release tarball: > > $ git check-ignore -nv --no-index be/**/* > ... > .gitignore:4:thirdparty be/src/thirdparty > ... > > In the release tarball, be/src/thirdparty is not in the git index > since there are no history. So they are ignored finally. I track the > history of why we have "thirdparty" in .gitignore. It's there in the > first commit of that file. But the purpose is not ignoring > be/src/thirdparty: > > commit edc3a5518439ed59e6c46b04d0393018aabfa8c5 > Author: carl <c...@cloudera.com> > Date: Fri Jul 29 15:49:39 2011 -0700 > IMP-8. Update build scripts, etc., to reflect thirdparty/hadoop > and thirdparty/hive > ... > create mode 100644 .gitignore > ... > > At that time we don't have be/src/thirdparty yet (It's created in > IMPALA-3223). I filed a JIRA to track this issue: > https://issues.apache.org/jira/browse/IMPALA-11201 > > -------------------------------------------------------------------- > For the LZO build issue, I planned to fix it by this commit: > > https://github.com/apache/impala/commit/eb1ed66fa435a722fa8c6a7c58ff53edc10c572e > It changes bootstrap_system.sh to clone the asf-3.4 branch of > Impala-lzo. However, the all-build-options-ub1604 job still fails. > Then I realized that it uses the cdh5-trunk branch by default. The > build script has codes like this: > > IMPALA_LZO=${IMPALA_LZO:-} > if [[ -n "${IMPALA_LZO}" ]]; then > git init ~/Impala-lzo > pushd ~/Impala-lzo > GitWithRetry fetch https://github.com/cloudera/impala-lzo.git > IMPALA_LZO_BRANCH="${IMPALA_LZO_BRANCH:-cdh5-trunk}" > GitWithRetry fetch https://github.com/cloudera/impala-lzo.git > "${IMPALA_LZO_BRANCH}" > GitWithRetry checkout FETCH_HEAD > GitWithRetry rev-parse FETCH_HEAD > popd > ... > > By default IMPALA_LZO is not set. Thus the cdh5-trunk branch is used. > > I think we don't need to be blocked by our Jenkins jobs, because users > are building Impala in their own envs. Do you think it's ok to go > ahead with RC3? https://github.com/apache/impala/tree/3.4.1-rc3 > > Thanks, > Quanlong > > > On Fri, Mar 25, 2022 at 12:21 AM Laszlo Gaal <laszlo.g...@cloudera.com> > wrote: > > > > Thanks for jumping on it, Quanlong. > > > > > It's a pity that the logs don't print the commit hash. Maybe it's due > > > to some config differences. > > > > If you mean that the Jenkins run summary page doesn't display the git > > hash(es) of the project(s) they build: > > this happens because the current job configurations don't rely on > Jenkins's > > built-in Git > > infrastructure to handle the operations. All (or at least all commonly > > used) jobs handle these on their own, > > in shell script fragments replicated throughout the various jobs. > > > > I'll make a pass through the jobs to let Jenkins handle the Git > operations, > > but it will take me at least a few days to get to this. > > > > On another note, this problem also highlights the fact that the > > release-test job > > needs to know which impala-lzo branch to use. The job does not currently > > take this > > parameter, so it should be added, otherwise it will never be able to > verify > > this release correctly. > > > > I'll let the community know when the fixes/enhancements land. > > In the meantime the suggested workaround for the release-test-ub1604 job > is > > to check the results, and if the only failure is from the > > "all-build-options-ub1604" job then > > re-run just that job against the release candidate, passing in "asf-3.4" > > for the IMPALA_LZO_BRANCH parameter. > > > > Thank you for your patience, > > > > - LaszloG > > > > > > > > On Thu, Mar 24, 2022 at 10:18 AM Quanlong Huang <huangquanl...@gmail.com > > > > wrote: > > > > > Sorry that there is a build issue on Impala-lzo. Thanks Laszlo Gaal > > > and Riza Suminto for reporting this to me offline! > > > > > > Impala-3.4 should depend on the asf-3.4 branch of Impala-lzo, instead > > > of the master branch. The build issue is revealed by the > > > all-build-options job: > > > https://jenkins.impala.io/job/all-build-options-ub1604/8069/ > > > > > > I'm confused why my builds passed: > > > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/16037/ > > > https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/5440/ > > > > > > It's a pity that the logs don't print the commit hash. Maybe it's due > > > to some config differences. > > > > > > Anyway, I'll post another RC soon. Thanks again for your votes and > > > sorry for the mistake! > > > > > > Quanlong > > > > > > On Thu, Mar 24, 2022 at 6:35 AM Joe McDonnell < > joemcdonn...@cloudera.com> > > > wrote: > > > > > > > > +1 (binding) > > > > > > > > I verified the tarball contents, the signatures, the sha512 sum, > built it > > > > from source, and verified the tests. > > > > > > > > On Wed, Mar 23, 2022 at 1:19 AM skyyws <sky...@163.com> wrote: > > > > > > > > > +1, thanks for Quanlong! > > > > > > > > > > > > > > > > > > > > > > > > > On 03/23/2022 15:28，Quanlong Huang<stigahu...@apache.org> wrote： > > > > > Hi all, > > > > > > > > > > This is a vote for Impala 3.4.1. > > > > > > > > > > The artifacts for testing can be downloaded from: > > > > > https://dist.apache.org/repos/dist/dev/impala/3.4.1/RC2/ > > > > > Git tag: 3.4.1-rc2 > > > https://github.com/apache/impala/releases/tag/3.4.1-rc2 > > > > > Tree hash: 7358fb59d9c09558d732fe680311a833eb2966af > > > > > > > > > > Please vote +1 or -1. -1 votes should be accompanied by an > explanation > > > > > of the reason. Only PMC members have binding votes, but other > > > > > community members are encouraged to cast non-binding votes. This > vote > > > > > will pass if there are 3 binding +1 votes and more binding +1 votes > > > > > than -1 votes. > > > > > > > > > > This wiki page describes how to check the release before you vote: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#HowtoRelease-HowtoVoteonaReleaseCandidate > > > > > > > > > > I tested this by running exhaustive jobs for both with/without > > > LocalCatalog > > > > > mode, i.e. ubuntu-16.04-from-scratch and > ubuntu-16.04-dockerised-tests. > > > > > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/16037/ > > > > > https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/5440/ > > > > > > > > > > Note that this is RC2 since RC1 was blocked for having > IMPALA-9242. We > > > > > will add back IMPALA-9242 in 3.4.2 after we fix IMPALA-10326. > (Thank > > > > > Csaba for pointing out this!) > > > > > > > > > > Thanks, > > > > > Quanlong > > > > > > > > >
