Hi Jim, I talked with the developer for the Sentry docs bug this past week, and we went through the list of changes needed for that. The developer is hoping to work on it next week.
If there is any further delay, I'm open to options of working around it. Thanks, Joe On Sat, May 15, 2021 at 12:43 PM Jim Apple <jbap...@apache.org> wrote: > This Sentry one I'm concerned about. The ticket has been open for a year. > We could do a 4.0 release without the docs, or we could maybe put a note on > the docs that Sentry support is no longer available. > > Any other ideas on how to tackle this while the ticket waits for a patch? > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <joemcdonn...@cloudera.com> > wrote: > > > I agree that it is a good idea for us to get going on the 4.0 release. > > > > I'm working on a change to fix IMPALA-9690. I will post an updated review > > once a test run completes. If everything is looking good, I would like to > > get this in the 4.0 release. > > > > There is also one documentation fix that I think belongs in the Impala > 4.0 > > release: > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry > > references > > > > Thanks, > > Joe > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <apa...@jbapple.com> wrote: > > > > > Sure thing! Let us know if you find owners for anything that someone > > wants > > > to champion as a 4.0 blocker. > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <huangquanl...@gmail.com > > > > > wrote: > > > > > > > Thank Jim! I planned to go through all the JIRAs target at 4.0: > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22 > > > > > > > > But due to some other urgent works and the recent holiday, my > progress > > is > > > > slow. Anyway, I think it's time to branch out. We've been waiting too > > > long. > > > > Thanks for creating the branch. > > > > > > > > Regards, > > > > Quanlong > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <apa...@jbapple.com> wrote: > > > > > > > > > I've created a 4.0 branch (with all changes through the end of > April) > > > and > > > > > started testing a release candidate. Everyone who wants to vote, > > > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release > > is a > > > > > guide to both releasing and voting (at the bottom). I'll let you > all > > > know > > > > > when I have something to vote on. > > > > > > > > > > Thanks! > > > > > Jim > > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer < > > > > csringho...@cloudera.com > > > > > > > > > > > wrote: > > > > > > > > > > > About IMPALA-9690 (AVX support): > > > > > > My preferred solution would be to deprecate support for x64 > without > > > > AVX2 > > > > > in > > > > > > 4.0, but not start removing the related logic yet. > > > > > > - We could even add a DCHECK + flag to crash by default if no > AVX2 > > is > > > > > > detected, and a message that points them to Impala mailing > > > > lists/related > > > > > > Jira. > > > > > > - After 4.0 is released people would be free to implement things > > that > > > > > > assume AVX2 support on x64, for example compiling Impala/the > whole > > > > > > toolchain with -mavx2. I think that currently people are held > back > > > from > > > > > > doing this - why put effort in doing+benchmarking a change if you > > > can't > > > > > > push it to master. > > > > > > - If a user appears with a valid use case that needs x64 without > > AVX2 > > > > > then > > > > > > we can revise our plans, but we shouldn't ignore low hanging > fruits > > > for > > > > > the > > > > > > sake of imaginary users :) > > > > > > > > > > > > -Csaba > > > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian < > > > > > sh...@superdupershant.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time to clean up the > old > > > > > > > conditional cpu logic it would be great, but if the ARM folks > > still > > > > > > > need some of that logic it's not critical enough to block, > just a > > > > very > > > > > > > nice to have? > > > > > > > > > > > > > > -Shant > > > > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong < > > > > > tim.g.armstr...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > I agree that it would be good to get this done even if it's > not > > > > > > perfect, > > > > > > > > there's so much good stuff. > > > > > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell < > > > > > joemcdonn...@cloudera.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A large amount of > > good > > > > > work > > > > > > > has > > > > > > > > > gone in and it belongs in a release. > > > > > > > > > > > > > > > > > > We need to decide which breaking changes are truly blockers > > for > > > > an > > > > > > > Impala 4 > > > > > > > > > release. My feeling is that we can't hold the release for > > > > > > compatibility > > > > > > > > > breaking changes unless someone signs up to do those > changes. > > > > > > Breaking > > > > > > > > > compatibility is useful sometimes, but it can't be an > > > indefinite > > > > > hold > > > > > > > on > > > > > > > > > releases. I think a release discussion thread is a good way > > to > > > > > > > stimulate > > > > > > > > > this discussion. > > > > > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one other change > > mentioned > > > > in > > > > > > the > > > > > > > > > original email is switching to use the new on-demand > metadata > > > by > > > > > > > default. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang < > > > > > > huangquanl...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Sure. If there are no objections, I'll raise a discussion > > > > thread > > > > > > for > > > > > > > the > > > > > > > > > > 4.0 release. There are still some unresolved breaking > > > changes, > > > > > e.g. > > > > > > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the default file format > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > Quanlong > > > > > > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple < > > > jbap...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for bringing this up again, Quanlong! I would > love > > > to > > > > > see > > > > > > a > > > > > > > 4.0 > > > > > > > > > > > release soon so that 4.1 releases and what not can be > > > > prepared. > > > > > > > Already > > > > > > > > > > the > > > > > > > > > > > changlist from the 3.x line is going to be enormous and > > > could > > > > > be > > > > > > > hard > > > > > > > > > for > > > > > > > > > > > users to digest. > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang < > > > > > > > > > huangquanl...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Reviving this thread. I think it's time to create the > > > 4.0.0 > > > > > > > branch > > > > > > > > > and > > > > > > > > > > > > prepare for the release now. Any breaking change is > > > landing > > > > > and > > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > wait? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Quanlong > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong < > > > > > > > > > tarmstr...@cloudera.com > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Another thing that we're looking into is changing > the > > > > > encoded > > > > > > > > > runtime > > > > > > > > > > > > > profile representation to be more efficient - see > > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. > > So > > > > that > > > > > > > might > > > > > > > > > be > > > > > > > > > > a > > > > > > > > > > > > > default we will try to change in Impala 4.0. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong < > > > > > > > > > > > tarmstr...@cloudera.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > An addendum to this - we're also considering > > whether > > > to > > > > > > > increase > > > > > > > > > > the > > > > > > > > > > > > > > minimum CPU version so that we can simplify > > > development > > > > > and > > > > > > > focus > > > > > > > > > > > more > > > > > > > > > > > > on > > > > > > > > > > > > > > optimising for the most common CPUs - see > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Running on commodity hardware is important and we > > > want > > > > > > > people to > > > > > > > > > be > > > > > > > > > > > > able > > > > > > > > > > > > > > to develop on whatever machine they have access > to. > > > At > > > > > most > > > > > > > we're > > > > > > > > > > > > likely > > > > > > > > > > > > > > going to require AVX2 support, which has been > > > supported > > > > > by > > > > > > > the > > > > > > > > > vast > > > > > > > > > > > > > > majority of CPUs for the better part of a decade. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell < > > > > > > > > > > > > joemcdonn...@cloudera.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Impala is starting development on Impala 4.0. > > There > > > > are > > > > > > many > > > > > > > > > > > exciting > > > > > > > > > > > > > >> projects underway including full support for > Hive > > > ACID > > > > > > > tables, > > > > > > > > > > > > improved > > > > > > > > > > > > > >> multithreading support, and fault tolerance > > > > > improvements, > > > > > > > but > > > > > > > > > > Impala > > > > > > > > > > > > 4.0 > > > > > > > > > > > > > >> also serves as a time to alter behavior and > > platform > > > > > > > support. > > > > > > > > > Here > > > > > > > > > > > is > > > > > > > > > > > > > the > > > > > > > > > > > > > >> current list of notable breaking changes that > are > > > > > planned > > > > > > in > > > > > > > > > > Impala > > > > > > > > > > > > 4.0: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Remove support for older operating systems: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Maintaining support for older operating systems > > > > > requires a > > > > > > > > > > > > considerable > > > > > > > > > > > > > >> amount of effort, especially with changing > > security > > > > > > > > > requirements. > > > > > > > > > > > Each > > > > > > > > > > > > > >> extra operating system consumes valuable effort > > and > > > > can > > > > > > > conflict > > > > > > > > > > > with > > > > > > > > > > > > > >> support for newer operating systems, so we have > > > > decided > > > > > to > > > > > > > > > remove > > > > > > > > > > > > > support > > > > > > > > > > > > > >> for some older operating systems. Specifically, > we > > > > plan > > > > > to > > > > > > > drop > > > > > > > > > > > > support > > > > > > > > > > > > > for: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> - > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Centos 6 > > > > > > > > > > > > > >> - > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Ubuntu 14 > > > > > > > > > > > > > >> - > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Debian 8 > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Each of these is approaching or past its end of > > > life. > > > > > > Since > > > > > > > > > > Centos6 > > > > > > > > > > > > was > > > > > > > > > > > > > >> the only operating system still needing Python > 2.6 > > > and > > > > > > > Python > > > > > > > > > 2.6 > > > > > > > > > > is > > > > > > > > > > > > > long > > > > > > > > > > > > > >> past its end of life, Python 2.6 will no longer > be > > > > > > > supported. In > > > > > > > > > > > > > addition, > > > > > > > > > > > > > >> Centos 7 support will be focused on newer > versions > > > > such > > > > > as > > > > > > > 7.5 > > > > > > > > > and > > > > > > > > > > > > > above. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Remove support for Sentry: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger > > > > > > authorization > > > > > > > > > > > > > >> functionality has achieved parity and surpassed > > the > > > > > > existing > > > > > > > > > > Sentry > > > > > > > > > > > > > >> authorization functionality. Impala's Sentry > > support > > > > > > > requires > > > > > > > > > > > ongoing > > > > > > > > > > > > > >> effort to test and maintain, including > maintenance > > > on > > > > > > Sentry > > > > > > > > > > itself > > > > > > > > > > > to > > > > > > > > > > > > > >> address security updates. Given the diminished > > > > activity > > > > > in > > > > > > > the > > > > > > > > > > > Sentry > > > > > > > > > > > > > >> community, there is no timeline for addressing > > these > > > > > > > security > > > > > > > > > > > updates. > > > > > > > > > > > > > >> Unless there is a maintainer for Sentry support, > > > > Impala > > > > > > > plans to > > > > > > > > > > > focus > > > > > > > > > > > > > its > > > > > > > > > > > > > >> efforts on Ranger for its authorization system > and > > > > drop > > > > > > > Sentry. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Remove support for Impala-lzo: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to read > > the > > > > LZO > > > > > > > > > > compressed > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this > > > support > > > > > is > > > > > > > not > > > > > > > > > > > included > > > > > > > > > > > > > >> directly. The Impala-lzo code interacts with > > > internal > > > > > > Impala > > > > > > > > > code > > > > > > > > > > > at a > > > > > > > > > > > > > >> level that is error prone and intricate. Given > the > > > low > > > > > > > adoption > > > > > > > > > of > > > > > > > > > > > LZO > > > > > > > > > > > > > and > > > > > > > > > > > > > >> the other compression options available, Impala > > > plans > > > > to > > > > > > > remove > > > > > > > > > > > > > Impala-lzo > > > > > > > > > > > > > >> support along with the low level interface it > > used. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Deprecations: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> In addition, we also plan to deprecate several > > > > existing > > > > > > > features > > > > > > > > > > so > > > > > > > > > > > > that > > > > > > > > > > > > > >> they can be removed in a future release. Here > is a > > > > > summary > > > > > > > of > > > > > > > > > the > > > > > > > > > > > most > > > > > > > > > > > > > >> notable ones: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> - > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Impala will default to using on demand > metadata > > > > > > > management > > > > > > > > > > (i.e. > > > > > > > > > > > > the > > > > > > > > > > > > > >> local catalog implementation) as described in > > > this > > > > > > > > > > documentation: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html > > > > > > > > > > > > > >> The old metadata system is now deprecated and > > may > > > > be > > > > > > > removed > > > > > > > > > > in a > > > > > > > > > > > > > >> future release. > > > > > > > > > > > > > >> - > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Impala is deprecating the Beeswax client > > protocol > > > > > (i.e. > > > > > > > > > clients > > > > > > > > > > > > that > > > > > > > > > > > > > >> connect via beeswax_port) in favor of the > > > > HiveServer2 > > > > > > > client > > > > > > > > > > > > > protocol. > > > > > > > > > > > > > >> - > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Impala is deprecating the old > decimal_v2=false > > > > > behavior > > > > > > > from > > > > > > > > > > > Impala > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has > defaulted > > to > > > > > true > > > > > > > since > > > > > > > > > > > Impala > > > > > > > > > > > > > 3. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Impala is always open to new developers, and we > > > > welcome > > > > > > > feedback > > > > > > > > > > on > > > > > > > > > > > > > these > > > > > > > > > > > > > >> plans. Further discussion of other changes and > > > > > > deprecations > > > > > > > is > > > > > > > > > > > ongoing > > > > > > > > > > > > > at > > > > > > > > > > > > > >> dev@impala.apache.org. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Joe McDonnell > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
