I agree that it is a good idea for us to get going on the 4.0 release. I'm working on a change to fix IMPALA-9690. I will post an updated review once a test run completes. If everything is looking good, I would like to get this in the 4.0 release.
There is also one documentation fix that I think belongs in the Impala 4.0 release: https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry references Thanks, Joe On Thu, May 6, 2021 at 7:55 PM Jim Apple <apa...@jbapple.com> wrote: > Sure thing! Let us know if you find owners for anything that someone wants > to champion as a 4.0 blocker. > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <huangquanl...@gmail.com> > wrote: > > > Thank Jim! I planned to go through all the JIRAs target at 4.0: > > > > > https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22 > > > > But due to some other urgent works and the recent holiday, my progress is > > slow. Anyway, I think it's time to branch out. We've been waiting too > long. > > Thanks for creating the branch. > > > > Regards, > > Quanlong > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <apa...@jbapple.com> wrote: > > > > > I've created a 4.0 branch (with all changes through the end of April) > and > > > started testing a release candidate. Everyone who wants to vote, > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release is a > > > guide to both releasing and voting (at the bottom). I'll let you all > know > > > when I have something to vote on. > > > > > > Thanks! > > > Jim > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer < > > csringho...@cloudera.com > > > > > > > wrote: > > > > > > > About IMPALA-9690 (AVX support): > > > > My preferred solution would be to deprecate support for x64 without > > AVX2 > > > in > > > > 4.0, but not start removing the related logic yet. > > > > - We could even add a DCHECK + flag to crash by default if no AVX2 is > > > > detected, and a message that points them to Impala mailing > > lists/related > > > > Jira. > > > > - After 4.0 is released people would be free to implement things that > > > > assume AVX2 support on x64, for example compiling Impala/the whole > > > > toolchain with -mavx2. I think that currently people are held back > from > > > > doing this - why put effort in doing+benchmarking a change if you > can't > > > > push it to master. > > > > - If a user appears with a valid use case that needs x64 without AVX2 > > > then > > > > we can revise our plans, but we shouldn't ignore low hanging fruits > for > > > the > > > > sake of imaginary users :) > > > > > > > > -Csaba > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian < > > > sh...@superdupershant.com > > > > > > > > > wrote: > > > > > > > > > +1 > > > > > > > > > > Regarding IMPALA-9690 if someone has the time to clean up the old > > > > > conditional cpu logic it would be great, but if the ARM folks still > > > > > need some of that logic it's not critical enough to block, just a > > very > > > > > nice to have? > > > > > > > > > > -Shant > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong < > > > tim.g.armstr...@gmail.com> > > > > > wrote: > > > > > > > > > > > > I agree that it would be good to get this done even if it's not > > > > perfect, > > > > > > there's so much good stuff. > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell < > > > joemcdonn...@cloudera.com> > > > > > > wrote: > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A large amount of good > > > work > > > > > has > > > > > > > gone in and it belongs in a release. > > > > > > > > > > > > > > We need to decide which breaking changes are truly blockers for > > an > > > > > Impala 4 > > > > > > > release. My feeling is that we can't hold the release for > > > > compatibility > > > > > > > breaking changes unless someone signs up to do those changes. > > > > Breaking > > > > > > > compatibility is useful sometimes, but it can't be an > indefinite > > > hold > > > > > on > > > > > > > releases. I think a release discussion thread is a good way to > > > > > stimulate > > > > > > > this discussion. > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one other change mentioned > > in > > > > the > > > > > > > original email is switching to use the new on-demand metadata > by > > > > > default. > > > > > > > > > > > > > > Thanks, > > > > > > > Joe > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang < > > > > huangquanl...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Sure. If there are no objections, I'll raise a discussion > > thread > > > > for > > > > > the > > > > > > > > 4.0 release. There are still some unresolved breaking > changes, > > > e.g. > > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the default file format > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements > > > > > > > > > > > > > > > > Thanks > > > > > > > > Quanlong > > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple < > jbap...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks for bringing this up again, Quanlong! I would love > to > > > see > > > > a > > > > > 4.0 > > > > > > > > > release soon so that 4.1 releases and what not can be > > prepared. > > > > > Already > > > > > > > > the > > > > > > > > > changlist from the 3.x line is going to be enormous and > could > > > be > > > > > hard > > > > > > > for > > > > > > > > > users to digest. > > > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang < > > > > > > > huangquanl...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Reviving this thread. I think it's time to create the > 4.0.0 > > > > > branch > > > > > > > and > > > > > > > > > > prepare for the release now. Any breaking change is > landing > > > and > > > > > we > > > > > > > > should > > > > > > > > > > wait? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Quanlong > > > > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong < > > > > > > > tarmstr...@cloudera.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Another thing that we're looking into is changing the > > > encoded > > > > > > > runtime > > > > > > > > > > > profile representation to be more efficient - see > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So > > that > > > > > might > > > > > > > be > > > > > > > > a > > > > > > > > > > > default we will try to change in Impala 4.0. > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong < > > > > > > > > > tarmstr...@cloudera.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > An addendum to this - we're also considering whether > to > > > > > increase > > > > > > > > the > > > > > > > > > > > > minimum CPU version so that we can simplify > development > > > and > > > > > focus > > > > > > > > > more > > > > > > > > > > on > > > > > > > > > > > > optimising for the most common CPUs - see > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690 > > > > > > > > > > > > > > > > > > > > > > > > Running on commodity hardware is important and we > want > > > > > people to > > > > > > > be > > > > > > > > > > able > > > > > > > > > > > > to develop on whatever machine they have access to. > At > > > most > > > > > we're > > > > > > > > > > likely > > > > > > > > > > > > going to require AVX2 support, which has been > supported > > > by > > > > > the > > > > > > > vast > > > > > > > > > > > > majority of CPUs for the better part of a decade. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell < > > > > > > > > > > joemcdonn...@cloudera.com > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> Impala is starting development on Impala 4.0. There > > are > > > > many > > > > > > > > > exciting > > > > > > > > > > > >> projects underway including full support for Hive > ACID > > > > > tables, > > > > > > > > > > improved > > > > > > > > > > > >> multithreading support, and fault tolerance > > > improvements, > > > > > but > > > > > > > > Impala > > > > > > > > > > 4.0 > > > > > > > > > > > >> also serves as a time to alter behavior and platform > > > > > support. > > > > > > > Here > > > > > > > > > is > > > > > > > > > > > the > > > > > > > > > > > >> current list of notable breaking changes that are > > > planned > > > > in > > > > > > > > Impala > > > > > > > > > > 4.0: > > > > > > > > > > > >> > > > > > > > > > > > >> Remove support for older operating systems: > > > > > > > > > > > >> > > > > > > > > > > > >> Maintaining support for older operating systems > > > requires a > > > > > > > > > > considerable > > > > > > > > > > > >> amount of effort, especially with changing security > > > > > > > requirements. > > > > > > > > > Each > > > > > > > > > > > >> extra operating system consumes valuable effort and > > can > > > > > conflict > > > > > > > > > with > > > > > > > > > > > >> support for newer operating systems, so we have > > decided > > > to > > > > > > > remove > > > > > > > > > > > support > > > > > > > > > > > >> for some older operating systems. Specifically, we > > plan > > > to > > > > > drop > > > > > > > > > > support > > > > > > > > > > > for: > > > > > > > > > > > >> > > > > > > > > > > > >> - > > > > > > > > > > > >> > > > > > > > > > > > >> Centos 6 > > > > > > > > > > > >> - > > > > > > > > > > > >> > > > > > > > > > > > >> Ubuntu 14 > > > > > > > > > > > >> - > > > > > > > > > > > >> > > > > > > > > > > > >> Debian 8 > > > > > > > > > > > >> > > > > > > > > > > > >> Each of these is approaching or past its end of > life. > > > > Since > > > > > > > > Centos6 > > > > > > > > > > was > > > > > > > > > > > >> the only operating system still needing Python 2.6 > and > > > > > Python > > > > > > > 2.6 > > > > > > > > is > > > > > > > > > > > long > > > > > > > > > > > >> past its end of life, Python 2.6 will no longer be > > > > > supported. In > > > > > > > > > > > addition, > > > > > > > > > > > >> Centos 7 support will be focused on newer versions > > such > > > as > > > > > 7.5 > > > > > > > and > > > > > > > > > > > above. > > > > > > > > > > > >> > > > > > > > > > > > >> Remove support for Sentry: > > > > > > > > > > > >> > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger > > > > authorization > > > > > > > > > > > >> functionality has achieved parity and surpassed the > > > > existing > > > > > > > > Sentry > > > > > > > > > > > >> authorization functionality. Impala's Sentry support > > > > > requires > > > > > > > > > ongoing > > > > > > > > > > > >> effort to test and maintain, including maintenance > on > > > > Sentry > > > > > > > > itself > > > > > > > > > to > > > > > > > > > > > >> address security updates. Given the diminished > > activity > > > in > > > > > the > > > > > > > > > Sentry > > > > > > > > > > > >> community, there is no timeline for addressing these > > > > > security > > > > > > > > > updates. > > > > > > > > > > > >> Unless there is a maintainer for Sentry support, > > Impala > > > > > plans to > > > > > > > > > focus > > > > > > > > > > > its > > > > > > > > > > > >> efforts on Ranger for its authorization system and > > drop > > > > > Sentry. > > > > > > > > > > > >> > > > > > > > > > > > >> Remove support for Impala-lzo: > > > > > > > > > > > >> > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to read the > > LZO > > > > > > > > compressed > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this > support > > > is > > > > > not > > > > > > > > > included > > > > > > > > > > > >> directly. The Impala-lzo code interacts with > internal > > > > Impala > > > > > > > code > > > > > > > > > at a > > > > > > > > > > > >> level that is error prone and intricate. Given the > low > > > > > adoption > > > > > > > of > > > > > > > > > LZO > > > > > > > > > > > and > > > > > > > > > > > >> the other compression options available, Impala > plans > > to > > > > > remove > > > > > > > > > > > Impala-lzo > > > > > > > > > > > >> support along with the low level interface it used. > > > > > > > > > > > >> > > > > > > > > > > > >> Deprecations: > > > > > > > > > > > >> > > > > > > > > > > > >> In addition, we also plan to deprecate several > > existing > > > > > features > > > > > > > > so > > > > > > > > > > that > > > > > > > > > > > >> they can be removed in a future release. Here is a > > > summary > > > > > of > > > > > > > the > > > > > > > > > most > > > > > > > > > > > >> notable ones: > > > > > > > > > > > >> > > > > > > > > > > > >> - > > > > > > > > > > > >> > > > > > > > > > > > >> Impala will default to using on demand metadata > > > > > management > > > > > > > > (i.e. > > > > > > > > > > the > > > > > > > > > > > >> local catalog implementation) as described in > this > > > > > > > > documentation: > > > > > > > > > > > >> > > > > > > > > > > > > > > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html > > > > > > > > > > > >> The old metadata system is now deprecated and may > > be > > > > > removed > > > > > > > > in a > > > > > > > > > > > >> future release. > > > > > > > > > > > >> - > > > > > > > > > > > >> > > > > > > > > > > > >> Impala is deprecating the Beeswax client protocol > > > (i.e. > > > > > > > clients > > > > > > > > > > that > > > > > > > > > > > >> connect via beeswax_port) in favor of the > > HiveServer2 > > > > > client > > > > > > > > > > > protocol. > > > > > > > > > > > >> - > > > > > > > > > > > >> > > > > > > > > > > > >> Impala is deprecating the old decimal_v2=false > > > behavior > > > > > from > > > > > > > > > Impala > > > > > > > > > > > >> 2.x. The decimal_v2 query option has defaulted to > > > true > > > > > since > > > > > > > > > Impala > > > > > > > > > > > 3. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Impala is always open to new developers, and we > > welcome > > > > > feedback > > > > > > > > on > > > > > > > > > > > these > > > > > > > > > > > >> plans. Further discussion of other changes and > > > > deprecations > > > > > is > > > > > > > > > ongoing > > > > > > > > > > > at > > > > > > > > > > > >> dev@impala.apache.org. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks, > > > > > > > > > > > >> > > > > > > > > > > > >> Joe McDonnell > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
