Yes, there are two discussion threads before that are relative to this. One for stopping the cherrypick-2.x-and-test jenkins job: https://lists.apache.org/thread.html/2b4b62d4c07661b27a5203618cb0425a429f6460f2eb505acbcd26c6@%3Cdev.impala.apache.org%3E
The other for removing support for hadoop 2 in master branch: https://lists.apache.org/thread.html/49f9b68ed3d6d2c0fdee16a877b259922545e4824e1233479227a657@%3Cdev.impala.apache.org%3E I'm +1 with the second thread that we only support Hadoop 2 in branch-2.x and support Hadoop 3 in the master branch to be more focused. I'm also agree with Paul's concern. It's such a dilemma that if we skip some commits, things will be harder and harder as we moving forward; if we cherry-pick, review, and test the commits one by one, branch-2.x will never catch up the master branch, which is an obstacle if someone (like me) wants to backport his/her new patch to branch-2.x but waits too long and finally fogets details of the patch. I roughly investigated how other systems deal with multiple branches. The efforts to backport a patch could be the same for the original patch. It's not a easy go, so the Hive community declares that "The decision to port a feature from master to branch-1 is at the discretion of the contributor and committer. However no features that break backwards compatibility will be accepted on branch-1." I think it's a chance to understand more parts of Impala by learning and backporting the patches, since they have execellent commit messages and were strictly reviewed. So I volunteer for the job to move forward the branch-2.x. Hopes patch authors could give some pointers when I'm blocked! I'll try approach (b) first and switch to (a) when (b) becomes impossible after too many commits are skipped. I'll confirm with the author if I think a patch should be skipped. For the first patch, "0b540b025 IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition", the cherry-pick conflicts is due to the revert of IMPALA-6479 in 2.x. I'm testing branch-2.x with IMPALA-6479 being picked back. Does anyone know why we revert it? (I also comment in the JIRA). On Mon, Jan 28, 2019 at 12:43 PM Philip Zeyliger <[email protected]> wrote: > As for Quanlong's question, I think the answer is however the folks who > want to do the work prefer to do it. As you noticed in the CDH changelists, > Cloudera's distribution has opted for something more like approach (a), > choosing to backport individual features. For a while, we were doing > automation for cherry-picking things automatically, and it got tedious > enough that we decided to turn it off. > > On Sun, Jan 27, 2019 at 7:37 PM Paul Rogers <[email protected]> wrote: > > > Hi Quanlong, > > > > Thanks for the suggestion. I wonder if there is a third strategy: > > > > c) Isolate the Hadoop 2.x/3.x differences into clearly-defined driver > > layer so that basically all of 3.x can be applied to the 2.x branch. Said > > another way, a single source base can work against either Hadoop 2.x or > > 3.x, with the build (C++) or runtime (Java) choosing the proper “driver” > > classes. > > > > We had such a layer for a while, where Impala master could be built against > either Hadoop3 or Hadoop2. We decided to clean it up in commit > e4ae605b083ab536c68552e37ca3c46f6bff4c76. > > commit e4ae605b083ab536c68552e37ca3c46f6bff4c76 > Author: Fredy Wijaya <[email protected]> > Date: Thu Jul 12 17:01:13 2018 -0700 > > IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 > > This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that > uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code > from > IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code > changes in this patch, there is no code change for the shims. The shims > for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default > implementation. > > Testing: > - Ran core and exhaustive tests > > Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 > Reviewed-on: http://gerrit.cloudera.org:8080/10940 > Reviewed-by: Impala Public Jenkins <[email protected] > > > Tested-by: Impala Public Jenkins <[email protected]> >
