Hi, vc
I will help to do some tests against the release branch.
BTW, I also found a bug about TimestampBasedKeyGenerator and pushed a pr on it
(https://github.com/apache/hudi/pull/1920), If this release won’t go without
TimestampKeyGenerator issue fixed, this pr may be included.
Best,
wangxianghu
发件人: 刘金辉
发送时间: 2020年8月19日 14:06
收件人: dev
主题: 回复： [DISCUSS] Release 0.6.0 timelines
hello,vc
This week I will devote all of it to it. I do post some problems at the moment,
and I will report them to issues later.
thanks liujinhui
------------------ 原始邮件 ------------------
发件人:
"dev"
<[email protected]>;
发送时间: 2020年8月19日(星期三) 中午12:51
收件人: "dev"<[email protected]>;
主题: Re: [DISCUSS] Release 0.6.0 timelines
We still have 1 blocker issue from TimestampKeyGenerator issue with joda
DateTimeFormatter. Sudha (RM) and Pratyaksh are going to look into this.
In the meantime, here's the progress, plans around testing so far. If folks
in the community can help test the release branch in the next couple of
days, it would be of great help!
- Testing plan here: Stability tested by balaji running spark
streaming,
Correctness (Nishith testing on release branch, Udit to help with
EMR setup)
- Testing plan continued: Spark datasource MOR, .. Marker based
rollback, upgrade-downgrade (vinoth is on this)
- Testing plan : bootstrap (balaji tested for hive), udit has it
working
for Presto in EMR.
- Testing plan: bulk_insert v2 (vinoth tested for correctness, file
sizes. ), performance (tested on a cluster, microbenchmarks)
There will also be some blogs/docs to explain new stuff.
- Blogs to explain new stuff: sort modes (vinoth), simple index
(vinoth), bootstrap (balaji), multi delta-streamer (pratyaksh),
Spark
Streaming/DeltaStreamer continuous mode blog (balaji)
- Docs: All missing docs will be added by Sudha.
On Tue, Aug 18, 2020 at 1:58 AM Bhavani Sudha <[email protected]>
wrote:
> Quick update on the RC.
>
> Found a build issue when building scala 2.12 and sent a PR for that -
> https://github.com/apache/hudi/pull/1976 . Working on resolving this in
> the
> release branch and updating RC. Will update soon.
>
> Thanks,
> Sudha
>
> On Fri, Aug 14, 2020 at 5:56 PM Vinoth Chandar <[email protected]>
wrote:
>
> > Thanks Sudha! This is means master is now open for regular PRs. Thanks
> for
> > your patience, everyone.
> >
> > On Fri, Aug 14, 2020 at 3:51 PM Bhavani Sudha
<[email protected]>
> > wrote:
> >
> > > Hello all,
> > >
> > > We have cut the release branch -
> > > https://github.com/apache/hudi/tree/release-0.6.0 . Since it is
> already
> > > Friday, we will be sending the release candidate early next week
(after
> > > some testing).
> > >
> > > Happy Friday!
> > >
> > > Thanks,
> > > Sudha
> > >
> > > On Wed, Aug 12, 2020 at 3:56 PM [email protected]
<[email protected]
> >
> > > wrote:
> > >
> > > >
> > > > Hi Folks,
> > > > We are continuing to work on CI stabilization and will cut
the
> release
> > > > once we stabilize the builds hopefully tonight/tomorrow.
> > > > Thanks,Balaji.V
> > > > On Tuesday, August 11, 2020,
09:15:05 PM PDT, Vinoth Chandar <
> > > > [email protected]> wrote:
> > > >
> > > > Hello all,
> > > >
> > > > Update on this. We have landed most of the blockers for the
0.6.0
> > release
> > > > and I am currently working on the last major blocker,
HUDI-1013.
> > > > We are working through some unexpected CI flakiness. We
hope to
> > stabilize
> > > > master, cut the RC, and then open up master for regular PR
merges.
> > > > ETA for this is tomorrow night PST (Aug 12, PST).
> > > >
> > > > We will keep this thread posted!
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > > > On Tue, Aug 4, 2020 at 9:47 PM Vinoth Chandar
<[email protected]>
> > wrote:
> > > >
> > > > > Small correction:
> > > > >
> > > > > >> Vinoth working on code review, tests for PR
1876,
> > > > > This is landed!
> > > > >
> > > > >
> > > > > On Tue, Aug 4, 2020 at 9:44 PM Bhavani Sudha <
> > [email protected]>
> > > > > wrote:
> > > > >
> > > > >> Hello all,
> > > > >>
> > > > >> We are targeting the end of this week to cut RC.
Here is an update
> > of
> > > > >> where
> > > > >> we are at release blockers.
> > > > >>
> > > > >> 0.6.0 Release blocker status (board
> > > > >> <
> > > > >>
> > > >
> > >
> >
>
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69
> > > > >> >)
> > > > >> ,
> > > > >>
> > > > >> - Spark Datasource/MOR
> https://github.com/apache/hudi/pull/1848
> > > > needs
> > > > >> to
> > > > >> be tested by gary/balaji (About
to land)
> > > > >> - Hive Sync restructuring
(Review done, about to land)
> > > > >> - Bootstrap
> > > > >> - Vinoth working on
code review, tests for PR 1876,
> > > > >> - then udit will
rework PR 1702 (In Code review)
> > > > >> - then we will
review, land PR 1870, 1869
> > > > >> - Bulk insert V2 PR 1834, lower
risk, independent PR, well
> tested
> > > > >> already
> > > > >> - Dependent PR 1149
to be landed,
> > > > >> - and modes to be
respected in V2 impl as well (At risk)
> > > > >> - Upgrade Downgrade Hooks, PR
1858 : (In Code review)
> > > > >> - HUDI-1054- Marker list perf
improvement, Udit has a PR out
> > > > >> - HUDI-115 : Overwrite with...
ordering issue, Sudha has a PR
> > > nearing
> > > > >> landing
> > > > >> - HUDI-1098 : Marker file issue
with non-existent files. (In
> Code
> > > > >> review)
> > > > >> - Spark Streaming + Async
Compaction , test complete, code
> review
> > > > >> comments and land PR 1752 (About
to land)
> > > > >> - Spark DataSource/Hive MOR
Incremental Query HUDI-920 (At
> risk)
> > > > >> - Flink/Multi Engine refactor,
will need a large rebase and
> > rework,
> > > > >> review, land (At risk for 0.6.0)
> > > > >> - BloomIndex V2 - Global index
implementation. (At risk)
> > > > >> - HUDI-845 : Parallel writing
i.e allow multiple writers
> (Pushed
> > > out
> > > > of
> > > > >> 0.6.0)
> > > > >> - HUDI-860 : Small File Handling
without memory caching (Pushed
> > out
> > > > of
> > > > >> 0.6.0)
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >> Sudha
> > > > >>
> > > > >> On Mon, Aug 3, 2020 at 3:41 PM Vinoth Chandar
<[email protected]>
> > > > wrote:
> > > > >>
> > > > >> > +1 (we need to formalize this well)
> > > > >> > But having just blockers land first, would
help not just with
> > > > rebasing,
> > > > >> but
> > > > >> > also wind down towards cutting an RC by end
of week.
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Aug 3, 2020 at 2:53 PM Bhavani Sudha <
> > > [email protected]
> > > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hello all,
> > > > >> > >
> > > > >> > > As we are all hustling towards getting
the blockers in, I
> wanted
> > > to
> > > > >> > propose
> > > > >> > > a code/merge freeze until we cut a
release for 0.6.0 and
> > restrict
> > > > it
> > > > >> to
> > > > >> > > only merging blockers identified for
this release. It would
> > reduce
> > > > >> > rebasing
> > > > >> > > time for blockers in progress. If we
feel some issue is a
> > serious
> > > > >> blocker
> > > > >> > > we can discuss it here and bump it's
priority.
> > > > >> > >
> > > > >> > > Please share your thoughts or concerns.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Sudha
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Aug 3, 2020 at 8:19 AM Vinoth
Chandar <
> > [email protected]>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Given enough time has passed, Sudha
can be our RM for 0.6.0.
> > > > >> > > >
> > > > >> > > > On the release blocker progress, we
landed few blockers over
> > the
> > > > >> > weekend,
> > > > >> > > > with some almost ready for landing
> > > > >> > > >
> > > > >> > > > Will send out a status update again
tomorrow night PST!
> > > > >> > > >
> > > > >> > > > On Mon, Aug 3, 2020 at 8:17 AM
Vinoth Chandar <
> > > [email protected]>
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > > Hi anton.
> > > > >> > > > >
> > > > >> > > > > We were hoping to cut a
release by last weekend. New
> target
> > is
> > > > >> this
> > > > >> > > > > weekend!
> > > > >> > > > > (tbh we were thrown off a bit
due to COVID in Q2, given a
> > lot
> > > of
> > > > >> > > > > PMC/Committers had additional
kid care duties. Now we are
> > back
> > > > to
> > > > >> > > normal
> > > > >> > > > > cadence)
> > > > >> > > > >
> > > > >> > > > > Going forward, I plan to start
a discussion around
> planning,
> > > > >> > > prioritizing
> > > > >> > > > > and other release processes
after 0.6.0. Would be great to
> > > have
> > > > >> the
> > > > >> > > > > community weigh in even more
in these things upfront.
> > > > >> > > > >
> > > > >> > > > > Thanks
> > > > >> > > > > Vinoth
> > > > >> > > > >
> > > > >> > > > > On Fri, Jul 31, 2020 at 6:49
PM Anton Zuyeu <
> > > > >> [email protected]>
> > > > >> > > > wrote:
> > > > >> > > > >
> > > > >> > > > >> Hi All,
> > > > >> > > > >>
> > > > >> > > > >> I apologize for possibly
dumb question but when was 0.6.0
> > > > >> planned to
> > > > >> > > be
> > > > >> > > > >> released? Can't find any
dates on Hudi related pages.
> > > > >> > > > >>
> > > > >> > > > >> On Thu, Jul 30, 2020 at
10:36 AM Vinoth Chandar <
> > > > >> [email protected]>
> > > > >> > > > >> wrote:
> > > > >> > > > >>
> > > > >> > > > >> > Is anyone able to
help with the at risk items? :)
> > > > >> > > > >> >
> > > > >> > > > >> > On Thu, Jul 30, 2020
at 7:07 AM leesf <
> > [email protected]
> > > >
> > > > >> > wrote:
> > > > >> > > > >> >
> > > > >> > > > >> > > @Vinoth Chandar
<[email protected]> Thanks for the
> > > > reminder,
> > > > >> > > marked
> > > > >> > > > >> to
> > > > >> > > > >> > > blocker, and
next week would be ok to me.
> > > > >> > > > >> > >
> > > > >> > > > >> > > Vinoth Chandar
<[email protected]> 于2020年7月30日周四
> > > > 上午11:35写道：
> > > > >> > > > >> > >
> > > > >> > > > >> > > > @leesf
<[email protected]> can we please mark
> the
> > > > >> relevant
> > > > >> > > > >> > ticket(s)
> > > > >> > > > >> > > > with
blocker priority, so it's easier to track?
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > Looks like
we are nearing a choice for RM.
> > > > >> > > > >> > > > Any more
thoughts on timelines? Looks like everyone
> > so
> > > > far
> > > > >> is
> > > > >> > > > >> leaning
> > > > >> > > > >> > > > towards
completeness of the release over doing it
> > > sooner?
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > On Wed, Jul
29, 2020 at 6:36 PM vino yang <
> > > > >> > > [email protected]>
> > > > >> > > > >> > wrote:
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > > +1 on
Sudha being RM for the release. And looking
> > > > >> forward to
> > > > >> > > > >> 0.6.0.
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > Best,
> > > > >> > > > >> > > > > Vino
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > leesf
<[email protected]> 于2020年7月30日周四
> > 上午9:15写道：
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > >
+1 on Sudha on being RM, and PR#1810
> > > > >> > > > >> > > > > >
https://github.com/apache/hudi/pull/1810
> > (abstract
> > > > >> hive
> > > > >> > > sync
> > > > >> > > > >> > module)
> > > > >> > > > >> > > > > would
> > > > >> > > > >> > > > > >
also goes to this release.
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > >
Sivabalan <[email protected]> 于2020年7月30日周四
> > > > 上午2:18写道：
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > >
> +1 on Sudha being RM for the release. Makes
> > sense
> > > > to
> > > > >> > push
> > > > >> > > > the
> > > > >> > > > >> > > release
> > > > >> > > > >> > > > > by
> > > > >> > > > >> > > > > > a
> > > > >> > > > >> > > > > >
> week.
> > > > >> > > > >> > > > > >
>
> > > > >> > > > >> > > > > >
> On Wed, Jul 29, 2020 at 1:35 AM
> > > [email protected]
> > > > <
> > > > >> > > > >> > > >
[email protected]
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > >
> wrote:
> > > > >> > > > >> > > > > >
>
> > > > >> > > > >> > > > > >
> > +1 on Sudha on being RM for this release.
> > Also
> > > > >> agree
> > > > >> > on
> > > > >> > > > >> > pushing
> > > > >> > > > >> > > > the
> > > > >> > > > >> > > > > >
> > release date by a week.
> > > > >> > > > >> > > > > >
> > Balaji.V
> > > > >> > > > >> > > > > >
> > On Tuesday, July 28, 2020, 10:08:41 PM
> > PDT,
> > > > >> > Bhavani
> > > > >> > > > >> Sudha <
> > > > >> > > > >> > > > > >
> > [email protected]> wrote:
> > > > >> > > > >> > > > > >
> >
> > > > >> > > > >> > > > > >
> > Thanks Vinoth for the update. I can
> > volunteer
> > > to
> > > > >> RM
> > > > >> > > this
> > > > >> > > > >> > > release.
> > > > >> > > > >> > > > > >
> >
> > > > >> > > > >> > > > > >
> > Understand 0.6.0 release is delayed than
> what
> > > we
> > > > >> > > > originally
> > > > >> > > > >> > > > >
discussed.
> > > > >> > > > >> > > > > > Q2
> > > > >> > > > >> > > > > >
> > has been really hard with COVID and
> > everything
> > > > >> going
> > > > >> > on.
> > > > >> > > > >> Given
> > > > >> > > > >> > > that
> > > > >> > > > >> > > > > we
> > > > >> > > > >> > > > > >
> are
> > > > >> > > > >> > > > > >
> > at this point, I feel by delaying the RC
> by a
> > > > week
> > > > >> or
> > > > >> > so
> > > > >> > > > >> more
> > > > >> > > > >> > if
> > > > >> > > > >> > > we
> > > > >> > > > >> > > > > can
> > > > >> > > > >> > > > > >
> get
> > > > >> > > > >> > > > > >
> > some of the 'At risk' items in, I would
> vote
> > > for
> > > > >> that.
> > > > >> > > > That
> > > > >> > > > >> is
> > > > >> > > > >> > > just
> > > > >> > > > >> > > > > my
> > > > >> > > > >> > > > > >
> > personal opinion. I ll let others chime in.
> > > > >> > > > >> > > > > >
> >
> > > > >> > > > >> > > > > >
> > Thanks,
> > > > >> > > > >> > > > > >
> > Sudha
> > > > >> > > > >> > > > > >
> >
> > > > >> > > > >> > > > > >
> > On Tue, Jul 28, 2020 at 9:48 PM Vinoth
> > Chandar
> > > <
> > > > >> > > > >> > >
[email protected]>
> > > > >> > > > >> > > > > >
> wrote:
> > > > >> > > > >> > > > > >
> >
> > > > >> > > > >> > > > > >
> > > Hello all,
> > > > >> > > > >> > > > > >
> > >
> > > > >> > > > >> > > > > >
> > > Just wanted to kickstart a thread to firm
> > up
> > > > the
> > > > >> RC
> > > > >> > > cut
> > > > >> > > > >> date
> > > > >> > > > >> > > for
> > > > >> > > > >> > > > > >
0.6.0
> > > > >> > > > >> > > > > >
> > and
> > > > >> > > > >> > > > > >
> > > pick a RM. (any volunteers?, if not I
> self
> > > > >> nominate
> > > > >> > > > >> myself)
> > > > >> > > > >> > > > > >
> > >
> > > > >> > > > >> > > > > >
> > > Here's an update on where we are at with
> > the
> > > > >> > remaining
> > > > >> > > > >> > release
> > > > >> > > > >> > > > > >
> blockers.
> > > > >> > > > >> > > > > >
> > I
> > > > >> > > > >> > > > > >
> > > have marked items as "At risk" assuming
> we
> > > cut
> > > > RC
> > > > >> > > > sometime
> > > > >> > > > >> > next
> > > > >> > > > >> > > > > week.
> > > > >> > > > >> > > > > >
> > > Please chime in with your thoughts.
> > Ideally,
> > > we
> > > > >> > don't
> > > > >> > > > take
> > > > >> > > > >> > any
> > > > >> > > > >> > > > more
> > > > >> > > > >> > > > > >
> > > blockers. If we also want to knock off
> the
> > at
> > > > >> risk
> > > > >> > > > items,
> > > > >> > > > >> > then
> > > > >> > > > >> > > we
> > > > >> > > > >> > > > > >
would
> > > > >> > > > >> > > > > >
> > > at-least push dates by another week (my
> > > guess).
> > > > >> > > > >> > > > > >
> > >
> > > > >> > > > >> > > > > >
> > > 0.6.0 Release blocker status (board
> > > > >> > > > >> > > > > >
> > > <
> > > > >> > > > >> > > > > >
> > >
> > > > >> > > > >> > > > > >
> >
> > > > >> > > > >> > > > > >
>
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69
> > > > >> > > > >> > > > > >
> > > >)
> > > > >> > > > >> > > > > >
> > > ,
> > > > >> > > > >> > > > > >
> > >
> > > > >> > > > >> > > > > >
> > > - Spark Datasource/MOR
> > > > >> > > > >> > > >
https://github.com/apache/hudi/pull/1848
> > > > >> > > > >> > > > > >
> needs
> > > > >> > > > >> > > > > >
> > > to
> > > > >> > > > >> > > > > >
> > > be tested by gary/balaji
> > > > >> > > > >> > > > > >
> > > - Bootstrap
> > > > >> > > > >> > > > > >
> > > - Vinoth working on code review,
> tests
> > > for
> > > > >> PR
> > > > >> > > 1876,
> > > > >> > > > >> > > > > >
> > > - then udit will rework PR 1702
> > > > >> > > > >> > > > > >
> > > - then we will review, land PR
1870,
> > > 1869
> > > > >> > > > >> > > > > >
> > > - Also need to fix HUDI-999,
> HUDI-1021
> > > > >> > > > >> > > > > >
> > > - Bulk insert V2 PR 1834, lower risk,
> > > > >> independent
> > > > >> > > PR,
> > > > >> > > > >> well
> > > > >> > > > >> > > > > tested
> > > > >> > > > >> > > > > >
> > > already
> > > > >> > > > >> > > > > >
> > > - Dependent PR 1149 to be landed,
> > > > >> > > > >> > > > > >
> > > - and modes to be respected in V2
> impl
> > > as
> > > > >> well
> > > > >> > > (At
> > > > >> > > > >> risk)
> > > > >> > > > >> > > > > >
> > > - Upgrade Downgrade Hooks, PR 1858 :
> > Siva
> > > > has
> > > > >> a
> > > > >> > PR
> > > > >> > > > out,
> > > > >> > > > >> > code
> > > > >> > > > >> > > > > >
> > completing
> > > > >> > > > >> > > > > >
> > > this week
> > > > >> > > > >> > > > > >
> > > - HUDI-1054- Marker list perf
> > improvement,
> > > > >> Udit
> > > > >> > > has a
> > > > >> > > > >> PR
> > > > >> > > > >> > out
> > > > >> > > > >> > > > > >
> > > - HUDI-115 : Overwrite with...
> ordering
> > > > issue,
> > > > >> > > Sudha
> > > > >> > > > >> has a
> > > > >> > > > >> > > PR
> > > > >> > > > >> > > > > >
> nearing
> > > > >> > > > >> > > > > >
> > > landing
> > > > >> > > > >> > > > > >
> > > - HUDI-1098 : Marker file issue with
> > > > >> non-existent
> > > > >> > > > >> files.
> > > > >> > > > >> > > Siva
> > > > >> > > > >> > > > to
> > > > >> > > > >> > > > > >
> begin
> > > > >> > > > >> > > > > >
> > > impl
> > > > >> > > > >> > > > > >
> > > - Spark Streaming + Async Compaction ,
> > > test
> > > > >> > > complete,
> > > > >> > > > >> code
> > > > >> > > > >> > > > > review
> > > > >> > > > >> > > > > >
> > > comments and land PR 1752
> > > > >> > > > >> > > > > >
> > > - Spark DataSource/Hive MOR
> Incremental
> > > > Query
> > > > >> > > > HUDI-920
> > > > >> > > > >> (At
> > > > >> > > > >> > > > risk)
> > > > >> > > > >> > > > > >
> > > - Flink/Multi Engine refactor, will
> > need a
> > > > >> large
> > > > >> > > > rebase
> > > > >> > > > >> > and
> > > > >> > > > >> > > > > >
rework,
> > > > >> > > > >> > > > > >
> > > review, land (At risk for 0.6.0, high
> > > scope,
> > > > >> may
> > > > >> > > not
> > > > >> > > > >> have
> > > > >> > > > >> > > > enough
> > > > >> > > > >> > > > > >
> time)
> > > > >> > > > >> > > > > >
> > > - BloomIndex V2 - Global index
> > > > implementation.
> > > > >> > (At
> > > > >> > > > >> risk)
> > > > >> > > > >> > > > > >
> > > - HUDI-845 : Parallel writing i.e
> allow
> > > > >> multiple
> > > > >> > > > >> writers
> > > > >> > > > >> > (At
> > > > >> > > > >> > > > > risk)
> > > > >> > > > >> > > > > >
> > > - HUDI-860 : Small File Handling
> without
> > > > >> memory
> > > > >> > > > caching
> > > > >> > > > >> > (At
> > > > >> > > > >> > > > > risk)
> > > > >> > > > >> > > > > >
> > >
> > > > >> > > > >> > > > > >
> > >
> > > > >> > > > >> > > > > >
> > > Thanks
> > > > >> > > > >> > > > > >
> > > Vinoth
> > > > >> > > > >> > > > > >
> > >
> > > > >> > > > >> > > > > >
> >
> > > > >> > > > >> > > > > >
>
> > > > >> > > > >> > > > > >
>
> > > > >> > > > >> > > > > >
>
> > > > >> > > > >> > > > > >
> --
> > > > >> > > > >> > > > > >
> Regards,
> > > > >> > > > >> > > > > >
> -Sivabalan
> > > > >> > > > >> > > > > >
>
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > >
> >
>
