Thank Zoltan for listing these commits! IMPALA-10413 can be resolved. I marked the fix version as 4.0 since we are going to pick it. Also did the same for IMPALA-10704.
Besides that, I think we should also pick the doc fix about removing Sentry related stuffs: - 1a84a14 IMPALA-9770: [DOCS] Remove Sentry references in documentation IMPALA-7501 was recently resolved. I'd be good if we pick it as well, because it fixes the scalability issue of local-catalog mode on large tables (avoiding OOM and reducing memory footprints). - bb30621 IMPALA-7501: Slim down partition metadata in LocalCatalog mode Thanks, Quanlong On Fri, Jun 4, 2021 at 5:47 PM Zoltán Borók-Nagy <borokna...@apache.org> wrote: > I didn't have any specifics in mind when I wrote there are good commits on > master since May 1. > My principle was only that it would be nice to release what's latest, since > some time has passed since 4.0 was branched. > > Now that I looked at the commit list, maybe we could consider adding the > followings at least: > > - IMPALA-10683: Skip test 'test_double_precision' for non-HDFS test env > - IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64 > - IMPALA-10413: fix impalad crashes when canceling the retrying query > - IMPALA-5121: Fix AVG() on timestamp col with > use_local_tz_for_unix_timestamp_conversions > - IMPALA-10704: Fix retried query id not being unregistered when retry > fails > > IMPALA-9690 <https://issues.apache.org/jira/browse/IMPALA-9690> and > IMPALA-5121 <https://issues.apache.org/jira/browse/IMPALA-5121> are even > targeted for 4.0 in Jira. > IMPALA-10413 <https://issues.apache.org/jira/browse/IMPALA-10413> is > targeted for 4.0, but the Jira is not resolved yet. A patch has been > already merged, so I asked at the Jira ticket whether we can resolve it. > IMPALA-10704 <https://issues.apache.org/jira/browse/IMPALA-10704> is > targeted for 4.1, but fixes an issue introduced by IMPALA-10413. > > Thanks, > Zoltan > > > On Fri, Jun 4, 2021 at 5:06 AM Jim Apple <jbap...@apache.org> wrote: > > > Once we know the answer to that last question, I'd be interested in > reading > > more about your vision for what should go in 4.0, Quanlong -- > > i.e."everything committed on June 2 except patches X and Y". > > > > On Thu, Jun 3, 2021 at 2:10 AM Quanlong Huang <huangquanl...@gmail.com> > > wrote: > > > > > Sorry to be late here! I'm -1 for merging HEAD into 4.0. Because it > will > > > bump the CDP_BUILD_NUMBER again which increases the difficulty for 3.x > > > users (who may still be using Hive 2.x or 3.x) to adopt Impala-4.0. > There > > > are two new commits in HEAD that bump the CDP_BUILD_NUMBER. > > > > > > We may have already depended on some Hive 4.0 APIs. But those features > > may > > > not be needed by 3.x users. They can revert some changes to make > > Impala-4.0 > > > compatible with older versions of Hive. > > > The lower GBN we have, the easier it is for them to upgrade. > > > > > > Does those good commits of HEAD depend on bumping the GBN? If not, we > can > > > just pick them to the 4.0 branch. > > > > > > Thanks, > > > Quanlong > > > > > > On Wed, May 26, 2021 at 8:52 PM Zoltán Borók-Nagy < > borokna...@apache.org > > > > > > wrote: > > > > > > > There's been a handful of good commits since May 1, so +1 for merging > > > HEAD > > > > into 4.0. > > > > > > > > Cheers, > > > > Zoltan > > > > > > > > On Wed, May 26, 2021 at 2:11 PM Jim Apple <jbap...@apache.org> > wrote: > > > > > > > > > We're only waiting on one patch for the docs, but since I cut the > 4.0 > > > > > branch on May 1, a number of things have landed. Maybe it would be > > > best > > > > to > > > > > merge HEAD into 4.0.0 once the Sentry doc change lands, thereby > > > including > > > > > everything we've talked about cherrypicking -- including 9690. > > > > > > > > > > Thoughts? > > > > > > > > > > On Wed, May 26, 2021 at 12:40 AM Quanlong Huang < > > > huangquanl...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I see IMPALA-9690 is done but still missing in branch-4.0.0 and I > > > think > > > > > we > > > > > > should pick it > > > > > > > > > > > > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for > > > > x86_64 > > > > > > > > > > > > I'd be good to also carry on these fixes for query retry: > > > > > > > > > > > > - 42684b4 IMPALA-10413: fix impalad crashes when canceling the > > > > > retrying > > > > > > query > > > > > > - d111443 IMPALA-10704: Fix retried query id not being > > > unregistered > > > > > when > > > > > > retry fails > > > > > > > > > > > > If you are ok, I can push them to branch-4.0.0. > > > > > > > > > > > > Thanks, > > > > > > Quanlong > > > > > > > > > > > > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jbap...@apache.org> > > > wrote: > > > > > > > > > > > > > Sounds good! > > > > > > > > > > > > > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell < > > > > > joemcdonn...@cloudera.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Jim, > > > > > > > > > > > > > > > > I talked with the developer for the Sentry docs bug this past > > > week, > > > > > and > > > > > > > we > > > > > > > > went through the list of changes needed for that. The > developer > > > is > > > > > > hoping > > > > > > > > to work on it next week. > > > > > > > > > > > > > > > > If there is any further delay, I'm open to options of working > > > > around > > > > > > it. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Joe > > > > > > > > > > > > > > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple < > jbap...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > > > This Sentry one I'm concerned about. The ticket has been > open > > > > for a > > > > > > > year. > > > > > > > > > We could do a 4.0 release without the docs, or we could > maybe > > > > put a > > > > > > > note > > > > > > > > on > > > > > > > > > the docs that Sentry support is no longer available. > > > > > > > > > > > > > > > > > > Any other ideas on how to tackle this while the ticket > waits > > > for > > > > a > > > > > > > patch? > > > > > > > > > > > > > > > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell < > > > > > > > joemcdonn...@cloudera.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I agree that it is a good idea for us to get going on the > > 4.0 > > > > > > > release. > > > > > > > > > > > > > > > > > > > > I'm working on a change to fix IMPALA-9690. I will post > an > > > > > updated > > > > > > > > review > > > > > > > > > > once a test run completes. If everything is looking > good, I > > > > would > > > > > > > like > > > > > > > > to > > > > > > > > > > get this in the 4.0 release. > > > > > > > > > > > > > > > > > > > > There is also one documentation fix that I think belongs > in > > > the > > > > > > > Impala > > > > > > > > > 4.0 > > > > > > > > > > release: > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 - > Remove > > > > > Sentry > > > > > > > > > > references > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple < > > apa...@jbapple.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Sure thing! Let us know if you find owners for anything > > > that > > > > > > > someone > > > > > > > > > > wants > > > > > > > > > > > to champion as a 4.0 blocker. > > > > > > > > > > > > > > > > > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang < > > > > > > > > huangquanl...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thank Jim! I planned to go through all the JIRAs > target > > > at > > > > > 4.0: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22 > > > > > > > > > > > > > > > > > > > > > > > > But due to some other urgent works and the recent > > > holiday, > > > > my > > > > > > > > > progress > > > > > > > > > > is > > > > > > > > > > > > slow. Anyway, I think it's time to branch out. We've > > been > > > > > > waiting > > > > > > > > too > > > > > > > > > > > long. > > > > > > > > > > > > Thanks for creating the branch. > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Quanlong > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple < > > > > apa...@jbapple.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > I've created a 4.0 branch (with all changes through > > the > > > > end > > > > > > of > > > > > > > > > April) > > > > > > > > > > > and > > > > > > > > > > > > > started testing a release candidate. Everyone who > > wants > > > > to > > > > > > > vote, > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release > > > > > > > > > > is a > > > > > > > > > > > > > guide to both releasing and voting (at the bottom). > > > I'll > > > > > let > > > > > > > you > > > > > > > > > all > > > > > > > > > > > know > > > > > > > > > > > > > when I have something to vote on. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > Jim > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer < > > > > > > > > > > > > csringho...@cloudera.com > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > About IMPALA-9690 (AVX support): > > > > > > > > > > > > > > My preferred solution would be to deprecate > support > > > for > > > > > x64 > > > > > > > > > without > > > > > > > > > > > > AVX2 > > > > > > > > > > > > > in > > > > > > > > > > > > > > 4.0, but not start removing the related logic > yet. > > > > > > > > > > > > > > - We could even add a DCHECK + flag to crash by > > > default > > > > > if > > > > > > no > > > > > > > > > AVX2 > > > > > > > > > > is > > > > > > > > > > > > > > detected, and a message that points them to > Impala > > > > > mailing > > > > > > > > > > > > lists/related > > > > > > > > > > > > > > Jira. > > > > > > > > > > > > > > - After 4.0 is released people would be free to > > > > implement > > > > > > > > things > > > > > > > > > > that > > > > > > > > > > > > > > assume AVX2 support on x64, for example compiling > > > > > > Impala/the > > > > > > > > > whole > > > > > > > > > > > > > > toolchain with -mavx2. I think that currently > > people > > > > are > > > > > > held > > > > > > > > > back > > > > > > > > > > > from > > > > > > > > > > > > > > doing this - why put effort in > doing+benchmarking a > > > > > change > > > > > > if > > > > > > > > you > > > > > > > > > > > can't > > > > > > > > > > > > > > push it to master. > > > > > > > > > > > > > > - If a user appears with a valid use case that > > needs > > > > x64 > > > > > > > > without > > > > > > > > > > AVX2 > > > > > > > > > > > > > then > > > > > > > > > > > > > > we can revise our plans, but we shouldn't ignore > > low > > > > > > hanging > > > > > > > > > fruits > > > > > > > > > > > for > > > > > > > > > > > > > the > > > > > > > > > > > > > > sake of imaginary users :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Csaba > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian < > > > > > > > > > > > > > sh...@superdupershant.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time > to > > > > clean > > > > > up > > > > > > > the > > > > > > > > > old > > > > > > > > > > > > > > > conditional cpu logic it would be great, but if > > the > > > > ARM > > > > > > > folks > > > > > > > > > > still > > > > > > > > > > > > > > > need some of that logic it's not critical > enough > > to > > > > > > block, > > > > > > > > > just a > > > > > > > > > > > > very > > > > > > > > > > > > > > > nice to have? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Shant > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong < > > > > > > > > > > > > > tim.g.armstr...@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that it would be good to get this > done > > > even > > > > > if > > > > > > > it's > > > > > > > > > not > > > > > > > > > > > > > > perfect, > > > > > > > > > > > > > > > > there's so much good stuff. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell < > > > > > > > > > > > > > joemcdonn...@cloudera.com> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A > > > large > > > > > > amount > > > > > > > > of > > > > > > > > > > good > > > > > > > > > > > > > work > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > gone in and it belongs in a release. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We need to decide which breaking changes > are > > > > truly > > > > > > > > blockers > > > > > > > > > > for > > > > > > > > > > > > an > > > > > > > > > > > > > > > Impala 4 > > > > > > > > > > > > > > > > > release. My feeling is that we can't hold > the > > > > > release > > > > > > > for > > > > > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > > breaking changes unless someone signs up to > > do > > > > > those > > > > > > > > > changes. > > > > > > > > > > > > > > Breaking > > > > > > > > > > > > > > > > > compatibility is useful sometimes, but it > > can't > > > > be > > > > > an > > > > > > > > > > > indefinite > > > > > > > > > > > > > hold > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > releases. I think a release discussion > thread > > > is > > > > a > > > > > > good > > > > > > > > way > > > > > > > > > > to > > > > > > > > > > > > > > > stimulate > > > > > > > > > > > > > > > > > this discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one > > other > > > > > change > > > > > > > > > > mentioned > > > > > > > > > > > > in > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > original email is switching to use the new > > > > > on-demand > > > > > > > > > metadata > > > > > > > > > > > by > > > > > > > > > > > > > > > default. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong > > Huang < > > > > > > > > > > > > > > huangquanl...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sure. If there are no objections, I'll > > raise > > > a > > > > > > > > discussion > > > > > > > > > > > > thread > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > 4.0 release. There are still some > > unresolved > > > > > > breaking > > > > > > > > > > > changes, > > > > > > > > > > > > > e.g. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the > default > > > file > > > > > > > format > > > > > > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU > > > > > > > requirements > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > Quanlong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim > Apple > > < > > > > > > > > > > > jbap...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for bringing this up again, > > > Quanlong! > > > > I > > > > > > > would > > > > > > > > > love > > > > > > > > > > > to > > > > > > > > > > > > > see > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > 4.0 > > > > > > > > > > > > > > > > > > > release soon so that 4.1 releases and > > what > > > > not > > > > > > can > > > > > > > be > > > > > > > > > > > > prepared. > > > > > > > > > > > > > > > Already > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > changlist from the 3.x line is going to > > be > > > > > > enormous > > > > > > > > and > > > > > > > > > > > could > > > > > > > > > > > > > be > > > > > > > > > > > > > > > hard > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > users to digest. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM > Quanlong > > > > Huang > > > > > < > > > > > > > > > > > > > > > > > huangquanl...@gmail.com> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Reviving this thread. I think it's > time > > > to > > > > > > create > > > > > > > > the > > > > > > > > > > > 4.0.0 > > > > > > > > > > > > > > > branch > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > prepare for the release now. Any > > breaking > > > > > > change > > > > > > > is > > > > > > > > > > > landing > > > > > > > > > > > > > and > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > > > wait? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Quanlong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim > > > > > Armstrong < > > > > > > > > > > > > > > > > > tarmstr...@cloudera.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another thing that we're looking > into > > > is > > > > > > > changing > > > > > > > > > the > > > > > > > > > > > > > encoded > > > > > > > > > > > > > > > > > runtime > > > > > > > > > > > > > > > > > > > > > profile representation to be more > > > > > efficient - > > > > > > > see > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. > > > > > > > > > > So > > > > > > > > > > > > that > > > > > > > > > > > > > > > might > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > default we will try to change in > > Impala > > > > > 4.0. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM > Tim > > > > > > Armstrong > > > > > > > < > > > > > > > > > > > > > > > > > > > tarmstr...@cloudera.com> > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > An addendum to this - we're also > > > > > > considering > > > > > > > > > > whether > > > > > > > > > > > to > > > > > > > > > > > > > > > increase > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > minimum CPU version so that we > can > > > > > simplify > > > > > > > > > > > development > > > > > > > > > > > > > and > > > > > > > > > > > > > > > focus > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > optimising for the most common > > CPUs - > > > > see > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Running on commodity hardware is > > > > > important > > > > > > > and > > > > > > > > we > > > > > > > > > > > want > > > > > > > > > > > > > > > people to > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > > > > > to develop on whatever machine > they > > > > have > > > > > > > access > > > > > > > > > to. > > > > > > > > > > > At > > > > > > > > > > > > > most > > > > > > > > > > > > > > > we're > > > > > > > > > > > > > > > > > > > > likely > > > > > > > > > > > > > > > > > > > > > > going to require AVX2 support, > > which > > > > has > > > > > > been > > > > > > > > > > > supported > > > > > > > > > > > > > by > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > vast > > > > > > > > > > > > > > > > > > > > > > majority of CPUs for the better > > part > > > > of a > > > > > > > > decade. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM > Joe > > > > > > > McDonnell < > > > > > > > > > > > > > > > > > > > > joemcdonn...@cloudera.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Impala is starting development > on > > > > Impala > > > > > > > 4.0. > > > > > > > > > > There > > > > > > > > > > > > are > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > exciting > > > > > > > > > > > > > > > > > > > > > >> projects underway including full > > > > support > > > > > > for > > > > > > > > > Hive > > > > > > > > > > > ACID > > > > > > > > > > > > > > > tables, > > > > > > > > > > > > > > > > > > > > improved > > > > > > > > > > > > > > > > > > > > > >> multithreading support, and > fault > > > > > > tolerance > > > > > > > > > > > > > improvements, > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > Impala > > > > > > > > > > > > > > > > > > > > 4.0 > > > > > > > > > > > > > > > > > > > > > >> also serves as a time to alter > > > > behavior > > > > > > and > > > > > > > > > > platform > > > > > > > > > > > > > > > support. > > > > > > > > > > > > > > > > > Here > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > >> current list of notable breaking > > > > changes > > > > > > > that > > > > > > > > > are > > > > > > > > > > > > > planned > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > Impala > > > > > > > > > > > > > > > > > > > > 4.0: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Remove support for older > operating > > > > > > systems: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Maintaining support for older > > > > operating > > > > > > > > systems > > > > > > > > > > > > > requires a > > > > > > > > > > > > > > > > > > > > considerable > > > > > > > > > > > > > > > > > > > > > >> amount of effort, especially > with > > > > > changing > > > > > > > > > > security > > > > > > > > > > > > > > > > > requirements. > > > > > > > > > > > > > > > > > > > Each > > > > > > > > > > > > > > > > > > > > > >> extra operating system consumes > > > > valuable > > > > > > > > effort > > > > > > > > > > and > > > > > > > > > > > > can > > > > > > > > > > > > > > > conflict > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > >> support for newer operating > > systems, > > > > so > > > > > we > > > > > > > > have > > > > > > > > > > > > decided > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > remove > > > > > > > > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > > > > > > > >> for some older operating > systems. > > > > > > > > Specifically, > > > > > > > > > we > > > > > > > > > > > > plan > > > > > > > > > > > > > to > > > > > > > > > > > > > > > drop > > > > > > > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > > > > > > > for: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> - > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Centos 6 > > > > > > > > > > > > > > > > > > > > > >> - > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Ubuntu 14 > > > > > > > > > > > > > > > > > > > > > >> - > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Debian 8 > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Each of these is approaching or > > past > > > > its > > > > > > end > > > > > > > > of > > > > > > > > > > > life. > > > > > > > > > > > > > > Since > > > > > > > > > > > > > > > > > > Centos6 > > > > > > > > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > > > > > > >> the only operating system still > > > > needing > > > > > > > Python > > > > > > > > > 2.6 > > > > > > > > > > > and > > > > > > > > > > > > > > > Python > > > > > > > > > > > > > > > > > 2.6 > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > long > > > > > > > > > > > > > > > > > > > > > >> past its end of life, Python 2.6 > > > will > > > > no > > > > > > > > longer > > > > > > > > > be > > > > > > > > > > > > > > > supported. In > > > > > > > > > > > > > > > > > > > > > addition, > > > > > > > > > > > > > > > > > > > > > >> Centos 7 support will be focused > > on > > > > > newer > > > > > > > > > versions > > > > > > > > > > > > such > > > > > > > > > > > > > as > > > > > > > > > > > > > > > 7.5 > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > above. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Remove support for Sentry: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Over the past year and a half, > > > > Impala's > > > > > > > Ranger > > > > > > > > > > > > > > authorization > > > > > > > > > > > > > > > > > > > > > >> functionality has achieved > parity > > > and > > > > > > > > surpassed > > > > > > > > > > the > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > Sentry > > > > > > > > > > > > > > > > > > > > > >> authorization functionality. > > > Impala's > > > > > > Sentry > > > > > > > > > > support > > > > > > > > > > > > > > > requires > > > > > > > > > > > > > > > > > > > ongoing > > > > > > > > > > > > > > > > > > > > > >> effort to test and maintain, > > > including > > > > > > > > > maintenance > > > > > > > > > > > on > > > > > > > > > > > > > > Sentry > > > > > > > > > > > > > > > > > > itself > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > >> address security updates. Given > > the > > > > > > > diminished > > > > > > > > > > > > activity > > > > > > > > > > > > > in > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > Sentry > > > > > > > > > > > > > > > > > > > > > >> community, there is no timeline > > for > > > > > > > addressing > > > > > > > > > > these > > > > > > > > > > > > > > > security > > > > > > > > > > > > > > > > > > > updates. > > > > > > > > > > > > > > > > > > > > > >> Unless there is a maintainer for > > > > Sentry > > > > > > > > support, > > > > > > > > > > > > Impala > > > > > > > > > > > > > > > plans to > > > > > > > > > > > > > > > > > > > focus > > > > > > > > > > > > > > > > > > > > > its > > > > > > > > > > > > > > > > > > > > > >> efforts on Ranger for its > > > > authorization > > > > > > > system > > > > > > > > > and > > > > > > > > > > > > drop > > > > > > > > > > > > > > > Sentry. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Impala-lzo provides code to > allow > > > > Impala > > > > > > to > > > > > > > > read > > > > > > > > > > the > > > > > > > > > > > > LZO > > > > > > > > > > > > > > > > > > compressed > > > > > > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, > which > > > is > > > > > why > > > > > > > this > > > > > > > > > > > support > > > > > > > > > > > > > is > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > included > > > > > > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code > > > > interacts > > > > > > with > > > > > > > > > > > internal > > > > > > > > > > > > > > Impala > > > > > > > > > > > > > > > > > code > > > > > > > > > > > > > > > > > > > at a > > > > > > > > > > > > > > > > > > > > > >> level that is error prone and > > > > intricate. > > > > > > > Given > > > > > > > > > the > > > > > > > > > > > low > > > > > > > > > > > > > > > adoption > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > LZO > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > >> the other compression options > > > > available, > > > > > > > > Impala > > > > > > > > > > > plans > > > > > > > > > > > > to > > > > > > > > > > > > > > > remove > > > > > > > > > > > > > > > > > > > > > Impala-lzo > > > > > > > > > > > > > > > > > > > > > >> support along with the low level > > > > > interface > > > > > > > it > > > > > > > > > > used. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Deprecations: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> In addition, we also plan to > > > deprecate > > > > > > > several > > > > > > > > > > > > existing > > > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > >> they can be removed in a future > > > > release. > > > > > > > Here > > > > > > > > > is a > > > > > > > > > > > > > summary > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > most > > > > > > > > > > > > > > > > > > > > > >> notable ones: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> - > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Impala will default to using > on > > > > > demand > > > > > > > > > metadata > > > > > > > > > > > > > > > management > > > > > > > > > > > > > > > > > > (i.e. > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > >> local catalog implementation) > > as > > > > > > > described > > > > > > > > in > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > documentation: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html > > > > > > > > > > > > > > > > > > > > > >> The old metadata system is > now > > > > > > deprecated > > > > > > > > and > > > > > > > > > > may > > > > > > > > > > > > be > > > > > > > > > > > > > > > removed > > > > > > > > > > > > > > > > > > in a > > > > > > > > > > > > > > > > > > > > > >> future release. > > > > > > > > > > > > > > > > > > > > > >> - > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the > > Beeswax > > > > > > client > > > > > > > > > > protocol > > > > > > > > > > > > > (i.e. > > > > > > > > > > > > > > > > > clients > > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > >> connect via beeswax_port) in > > > favor > > > > of > > > > > > the > > > > > > > > > > > > HiveServer2 > > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > > > > > protocol. > > > > > > > > > > > > > > > > > > > > > >> - > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the old > > > > > > > > > decimal_v2=false > > > > > > > > > > > > > behavior > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > Impala > > > > > > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query > > option > > > > has > > > > > > > > > defaulted > > > > > > > > > > to > > > > > > > > > > > > > true > > > > > > > > > > > > > > > since > > > > > > > > > > > > > > > > > > > Impala > > > > > > > > > > > > > > > > > > > > > 3. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Impala is always open to new > > > > developers, > > > > > > and > > > > > > > > we > > > > > > > > > > > > welcome > > > > > > > > > > > > > > > feedback > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > > > > > > > >> plans. Further discussion of > other > > > > > changes > > > > > > > and > > > > > > > > > > > > > > deprecations > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > ongoing > > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > >> dev@impala.apache.org. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> Joe McDonnell > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
