Just out of curiosity - what's the blocker - you have an issue? I had originally done the code to make that work.
On Wed, Aug 19, 2020 at 12:51 AM Vinoth Chandar <[email protected]> wrote: > We still have 1 blocker issue from TimestampKeyGenerator issue with joda > DateTimeFormatter. Sudha (RM) and Pratyaksh are going to look into this. > > In the meantime, here's the progress, plans around testing so far. If folks > in the community can help test the release branch in the next couple of > days, it would be of great help! > > > - Testing plan here: Stability tested by balaji running spark streaming, > Correctness (Nishith testing on release branch, Udit to help with EMR > setup) > - Testing plan continued: Spark datasource MOR, .. Marker based > rollback, upgrade-downgrade (vinoth is on this) > - Testing plan : bootstrap (balaji tested for hive), udit has it working > for Presto in EMR. > - Testing plan: bulk_insert v2 (vinoth tested for correctness, file > sizes. ), performance (tested on a cluster, microbenchmarks) > > > There will also be some blogs/docs to explain new stuff. > > - Blogs to explain new stuff: sort modes (vinoth), simple index > (vinoth), bootstrap (balaji), multi delta-streamer (pratyaksh), Spark > Streaming/DeltaStreamer continuous mode blog (balaji) > - Docs: All missing docs will be added by Sudha. > > > > > On Tue, Aug 18, 2020 at 1:58 AM Bhavani Sudha <[email protected]> > wrote: > > > Quick update on the RC. > > > > Found a build issue when building scala 2.12 and sent a PR for that - > > https://github.com/apache/hudi/pull/1976 . Working on resolving this in > > the > > release branch and updating RC. Will update soon. > > > > Thanks, > > Sudha > > > > On Fri, Aug 14, 2020 at 5:56 PM Vinoth Chandar <[email protected]> > wrote: > > > > > Thanks Sudha! This is means master is now open for regular PRs. Thanks > > for > > > your patience, everyone. > > > > > > On Fri, Aug 14, 2020 at 3:51 PM Bhavani Sudha <[email protected] > > > > > wrote: > > > > > > > Hello all, > > > > > > > > We have cut the release branch - > > > > https://github.com/apache/hudi/tree/release-0.6.0 . Since it is > > already > > > > Friday, we will be sending the release candidate early next week > (after > > > > some testing). > > > > > > > > Happy Friday! > > > > > > > > Thanks, > > > > Sudha > > > > > > > > On Wed, Aug 12, 2020 at 3:56 PM [email protected] < > [email protected] > > > > > > > wrote: > > > > > > > > > > > > > > Hi Folks, > > > > > We are continuing to work on CI stabilization and will cut the > > release > > > > > once we stabilize the builds hopefully tonight/tomorrow. > > > > > Thanks,Balaji.V > > > > > On Tuesday, August 11, 2020, 09:15:05 PM PDT, Vinoth Chandar < > > > > > [email protected]> wrote: > > > > > > > > > > Hello all, > > > > > > > > > > Update on this. We have landed most of the blockers for the 0.6.0 > > > release > > > > > and I am currently working on the last major blocker, HUDI-1013. > > > > > We are working through some unexpected CI flakiness. We hope to > > > stabilize > > > > > master, cut the RC, and then open up master for regular PR merges. > > > > > ETA for this is tomorrow night PST (Aug 12, PST). > > > > > > > > > > We will keep this thread posted! > > > > > > > > > > Thanks > > > > > Vinoth > > > > > > > > > > On Tue, Aug 4, 2020 at 9:47 PM Vinoth Chandar <[email protected]> > > > wrote: > > > > > > > > > > > Small correction: > > > > > > > > > > > > >> Vinoth working on code review, tests for PR 1876, > > > > > > This is landed! > > > > > > > > > > > > > > > > > > On Tue, Aug 4, 2020 at 9:44 PM Bhavani Sudha < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > >> Hello all, > > > > > >> > > > > > >> We are targeting the end of this week to cut RC. Here is an > update > > > of > > > > > >> where > > > > > >> we are at release blockers. > > > > > >> > > > > > >> 0.6.0 Release blocker status (board > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69 > > > > > >> >) > > > > > >> , > > > > > >> > > > > > >> - Spark Datasource/MOR > > https://github.com/apache/hudi/pull/1848 > > > > > needs > > > > > >> to > > > > > >> be tested by gary/balaji (About to land) > > > > > >> - Hive Sync restructuring (Review done, about to land) > > > > > >> - Bootstrap > > > > > >> - Vinoth working on code review, tests for PR 1876, > > > > > >> - then udit will rework PR 1702 (In Code review) > > > > > >> - then we will review, land PR 1870, 1869 > > > > > >> - Bulk insert V2 PR 1834, lower risk, independent PR, well > > tested > > > > > >> already > > > > > >> - Dependent PR 1149 to be landed, > > > > > >> - and modes to be respected in V2 impl as well (At risk) > > > > > >> - Upgrade Downgrade Hooks, PR 1858 : (In Code review) > > > > > >> - HUDI-1054- Marker list perf improvement, Udit has a PR out > > > > > >> - HUDI-115 : Overwrite with... ordering issue, Sudha has a PR > > > > nearing > > > > > >> landing > > > > > >> - HUDI-1098 : Marker file issue with non-existent files. (In > > Code > > > > > >> review) > > > > > >> - Spark Streaming + Async Compaction , test complete, code > > review > > > > > >> comments and land PR 1752 (About to land) > > > > > >> - Spark DataSource/Hive MOR Incremental Query HUDI-920 (At > > risk) > > > > > >> - Flink/Multi Engine refactor, will need a large rebase and > > > rework, > > > > > >> review, land (At risk for 0.6.0) > > > > > >> - BloomIndex V2 - Global index implementation. (At risk) > > > > > >> - HUDI-845 : Parallel writing i.e allow multiple writers > > (Pushed > > > > out > > > > > of > > > > > >> 0.6.0) > > > > > >> - HUDI-860 : Small File Handling without memory caching > (Pushed > > > out > > > > > of > > > > > >> 0.6.0) > > > > > >> > > > > > >> > > > > > >> Thanks, > > > > > >> Sudha > > > > > >> > > > > > >> On Mon, Aug 3, 2020 at 3:41 PM Vinoth Chandar < > [email protected]> > > > > > wrote: > > > > > >> > > > > > >> > +1 (we need to formalize this well) > > > > > >> > But having just blockers land first, would help not just with > > > > > rebasing, > > > > > >> but > > > > > >> > also wind down towards cutting an RC by end of week. > > > > > >> > > > > > > >> > > > > > > >> > On Mon, Aug 3, 2020 at 2:53 PM Bhavani Sudha < > > > > [email protected] > > > > > > > > > > > >> > wrote: > > > > > >> > > > > > > >> > > Hello all, > > > > > >> > > > > > > > >> > > As we are all hustling towards getting the blockers in, I > > wanted > > > > to > > > > > >> > propose > > > > > >> > > a code/merge freeze until we cut a release for 0.6.0 and > > > restrict > > > > > it > > > > > >> to > > > > > >> > > only merging blockers identified for this release. It would > > > reduce > > > > > >> > rebasing > > > > > >> > > time for blockers in progress. If we feel some issue is a > > > serious > > > > > >> blocker > > > > > >> > > we can discuss it here and bump it's priority. > > > > > >> > > > > > > > >> > > Please share your thoughts or concerns. > > > > > >> > > > > > > > >> > > Thanks, > > > > > >> > > Sudha > > > > > >> > > > > > > > >> > > > > > > > >> > > On Mon, Aug 3, 2020 at 8:19 AM Vinoth Chandar < > > > [email protected]> > > > > > >> wrote: > > > > > >> > > > > > > > >> > > > Given enough time has passed, Sudha can be our RM for > 0.6.0. > > > > > >> > > > > > > > > >> > > > On the release blocker progress, we landed few blockers > over > > > the > > > > > >> > weekend, > > > > > >> > > > with some almost ready for landing > > > > > >> > > > > > > > > >> > > > Will send out a status update again tomorrow night PST! > > > > > >> > > > > > > > > >> > > > On Mon, Aug 3, 2020 at 8:17 AM Vinoth Chandar < > > > > [email protected]> > > > > > >> > wrote: > > > > > >> > > > > > > > > >> > > > > Hi anton. > > > > > >> > > > > > > > > > >> > > > > We were hoping to cut a release by last weekend. New > > target > > > is > > > > > >> this > > > > > >> > > > > weekend! > > > > > >> > > > > (tbh we were thrown off a bit due to COVID in Q2, given > a > > > lot > > > > of > > > > > >> > > > > PMC/Committers had additional kid care duties. Now we > are > > > back > > > > > to > > > > > >> > > normal > > > > > >> > > > > cadence) > > > > > >> > > > > > > > > > >> > > > > Going forward, I plan to start a discussion around > > planning, > > > > > >> > > prioritizing > > > > > >> > > > > and other release processes after 0.6.0. Would be great > to > > > > have > > > > > >> the > > > > > >> > > > > community weigh in even more in these things upfront. > > > > > >> > > > > > > > > > >> > > > > Thanks > > > > > >> > > > > Vinoth > > > > > >> > > > > > > > > > >> > > > > On Fri, Jul 31, 2020 at 6:49 PM Anton Zuyeu < > > > > > >> [email protected]> > > > > > >> > > > wrote: > > > > > >> > > > > > > > > > >> > > > >> Hi All, > > > > > >> > > > >> > > > > > >> > > > >> I apologize for possibly dumb question but when was > 0.6.0 > > > > > >> planned to > > > > > >> > > be > > > > > >> > > > >> released? Can't find any dates on Hudi related pages. > > > > > >> > > > >> > > > > > >> > > > >> On Thu, Jul 30, 2020 at 10:36 AM Vinoth Chandar < > > > > > >> [email protected]> > > > > > >> > > > >> wrote: > > > > > >> > > > >> > > > > > >> > > > >> > Is anyone able to help with the at risk items? :) > > > > > >> > > > >> > > > > > > >> > > > >> > On Thu, Jul 30, 2020 at 7:07 AM leesf < > > > [email protected] > > > > > > > > > > >> > wrote: > > > > > >> > > > >> > > > > > > >> > > > >> > > @Vinoth Chandar <[email protected]> Thanks for the > > > > > reminder, > > > > > >> > > marked > > > > > >> > > > >> to > > > > > >> > > > >> > > blocker, and next week would be ok to me. > > > > > >> > > > >> > > > > > > > >> > > > >> > > Vinoth Chandar <[email protected]> 于2020年7月30日周四 > > > > > 上午11:35写道： > > > > > >> > > > >> > > > > > > > >> > > > >> > > > @leesf <[email protected]> can we please mark > > the > > > > > >> relevant > > > > > >> > > > >> > ticket(s) > > > > > >> > > > >> > > > with blocker priority, so it's easier to track? > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > Looks like we are nearing a choice for RM. > > > > > >> > > > >> > > > Any more thoughts on timelines? Looks like > everyone > > > so > > > > > far > > > > > >> is > > > > > >> > > > >> leaning > > > > > >> > > > >> > > > towards completeness of the release over doing it > > > > sooner? > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > On Wed, Jul 29, 2020 at 6:36 PM vino yang < > > > > > >> > > [email protected]> > > > > > >> > > > >> > wrote: > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > +1 on Sudha being RM for the release. And > looking > > > > > >> forward to > > > > > >> > > > >> 0.6.0. > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > Best, > > > > > >> > > > >> > > > > Vino > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > leesf <[email protected]> 于2020年7月30日周四 > > > 上午9:15写道： > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > +1 on Sudha on being RM, and PR#1810 > > > > > >> > > > >> > > > > > https://github.com/apache/hudi/pull/1810 > > > (abstract > > > > > >> hive > > > > > >> > > sync > > > > > >> > > > >> > module) > > > > > >> > > > >> > > > > would > > > > > >> > > > >> > > > > > also goes to this release. > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > Sivabalan <[email protected]> 于2020年7月30日周四 > > > > > 上午2:18写道： > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > +1 on Sudha being RM for the release. Makes > > > sense > > > > > to > > > > > >> > push > > > > > >> > > > the > > > > > >> > > > >> > > release > > > > > >> > > > >> > > > > by > > > > > >> > > > >> > > > > > a > > > > > >> > > > >> > > > > > > week. > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > On Wed, Jul 29, 2020 at 1:35 AM > > > > [email protected] > > > > > < > > > > > >> > > > >> > > > [email protected] > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > wrote: > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > > +1 on Sudha on being RM for this > release. > > > Also > > > > > >> agree > > > > > >> > on > > > > > >> > > > >> > pushing > > > > > >> > > > >> > > > the > > > > > >> > > > >> > > > > > > > release date by a week. > > > > > >> > > > >> > > > > > > > Balaji.V > > > > > >> > > > >> > > > > > > > On Tuesday, July 28, 2020, 10:08:41 PM > > > PDT, > > > > > >> > Bhavani > > > > > >> > > > >> Sudha < > > > > > >> > > > >> > > > > > > > [email protected]> wrote: > > > > > >> > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > Thanks Vinoth for the update. I can > > > volunteer > > > > to > > > > > >> RM > > > > > >> > > this > > > > > >> > > > >> > > release. > > > > > >> > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > Understand 0.6.0 release is delayed than > > what > > > > we > > > > > >> > > > originally > > > > > >> > > > >> > > > > discussed. > > > > > >> > > > >> > > > > > Q2 > > > > > >> > > > >> > > > > > > > has been really hard with COVID and > > > everything > > > > > >> going > > > > > >> > on. > > > > > >> > > > >> Given > > > > > >> > > > >> > > that > > > > > >> > > > >> > > > > we > > > > > >> > > > >> > > > > > > are > > > > > >> > > > >> > > > > > > > at this point, I feel by delaying the RC > > by a > > > > > week > > > > > >> or > > > > > >> > so > > > > > >> > > > >> more > > > > > >> > > > >> > if > > > > > >> > > > >> > > we > > > > > >> > > > >> > > > > can > > > > > >> > > > >> > > > > > > get > > > > > >> > > > >> > > > > > > > some of the 'At risk' items in, I would > > vote > > > > for > > > > > >> that. > > > > > >> > > > That > > > > > >> > > > >> is > > > > > >> > > > >> > > just > > > > > >> > > > >> > > > > my > > > > > >> > > > >> > > > > > > > personal opinion. I ll let others chime > in. > > > > > >> > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > Thanks, > > > > > >> > > > >> > > > > > > > Sudha > > > > > >> > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > On Tue, Jul 28, 2020 at 9:48 PM Vinoth > > > Chandar > > > > < > > > > > >> > > > >> > > [email protected]> > > > > > >> > > > >> > > > > > > wrote: > > > > > >> > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > Hello all, > > > > > >> > > > >> > > > > > > > > > > > > > >> > > > >> > > > > > > > > Just wanted to kickstart a thread to > firm > > > up > > > > > the > > > > > >> RC > > > > > >> > > cut > > > > > >> > > > >> date > > > > > >> > > > >> > > for > > > > > >> > > > >> > > > > > 0.6.0 > > > > > >> > > > >> > > > > > > > and > > > > > >> > > > >> > > > > > > > > pick a RM. (any volunteers?, if not I > > self > > > > > >> nominate > > > > > >> > > > >> myself) > > > > > >> > > > >> > > > > > > > > > > > > > >> > > > >> > > > > > > > > Here's an update on where we are at > with > > > the > > > > > >> > remaining > > > > > >> > > > >> > release > > > > > >> > > > >> > > > > > > blockers. > > > > > >> > > > >> > > > > > > > I > > > > > >> > > > >> > > > > > > > > have marked items as "At risk" assuming > > we > > > > cut > > > > > RC > > > > > >> > > > sometime > > > > > >> > > > >> > next > > > > > >> > > > >> > > > > week. > > > > > >> > > > >> > > > > > > > > Please chime in with your thoughts. > > > Ideally, > > > > we > > > > > >> > don't > > > > > >> > > > take > > > > > >> > > > >> > any > > > > > >> > > > >> > > > more > > > > > >> > > > >> > > > > > > > > blockers. If we also want to knock off > > the > > > at > > > > > >> risk > > > > > >> > > > items, > > > > > >> > > > >> > then > > > > > >> > > > >> > > we > > > > > >> > > > >> > > > > > would > > > > > >> > > > >> > > > > > > > > at-least push dates by another week (my > > > > guess). > > > > > >> > > > >> > > > > > > > > > > > > > >> > > > >> > > > > > > > > 0.6.0 Release blocker status (board > > > > > >> > > > >> > > > > > > > > < > > > > > >> > > > >> > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69 > > > > > >> > > > >> > > > > > > > > >) > > > > > >> > > > >> > > > > > > > > , > > > > > >> > > > >> > > > > > > > > > > > > > >> > > > >> > > > > > > > > - Spark Datasource/MOR > > > > > >> > > > >> > > > https://github.com/apache/hudi/pull/1848 > > > > > >> > > > >> > > > > > > needs > > > > > >> > > > >> > > > > > > > > to > > > > > >> > > > >> > > > > > > > > be tested by gary/balaji > > > > > >> > > > >> > > > > > > > > - Bootstrap > > > > > >> > > > >> > > > > > > > > - Vinoth working on code review, > > tests > > > > for > > > > > >> PR > > > > > >> > > 1876, > > > > > >> > > > >> > > > > > > > > - then udit will rework PR 1702 > > > > > >> > > > >> > > > > > > > > - then we will review, land PR > 1870, > > > > 1869 > > > > > >> > > > >> > > > > > > > > - Also need to fix HUDI-999, > > HUDI-1021 > > > > > >> > > > >> > > > > > > > > - Bulk insert V2 PR 1834, lower > risk, > > > > > >> independent > > > > > >> > > PR, > > > > > >> > > > >> well > > > > > >> > > > >> > > > > tested > > > > > >> > > > >> > > > > > > > > already > > > > > >> > > > >> > > > > > > > > - Dependent PR 1149 to be landed, > > > > > >> > > > >> > > > > > > > > - and modes to be respected in V2 > > impl > > > > as > > > > > >> well > > > > > >> > > (At > > > > > >> > > > >> risk) > > > > > >> > > > >> > > > > > > > > - Upgrade Downgrade Hooks, PR 1858 : > > > Siva > > > > > has > > > > > >> a > > > > > >> > PR > > > > > >> > > > out, > > > > > >> > > > >> > code > > > > > >> > > > >> > > > > > > > completing > > > > > >> > > > >> > > > > > > > > this week > > > > > >> > > > >> > > > > > > > > - HUDI-1054- Marker list perf > > > improvement, > > > > > >> Udit > > > > > >> > > has a > > > > > >> > > > >> PR > > > > > >> > > > >> > out > > > > > >> > > > >> > > > > > > > > - HUDI-115 : Overwrite with... > > ordering > > > > > issue, > > > > > >> > > Sudha > > > > > >> > > > >> has a > > > > > >> > > > >> > > PR > > > > > >> > > > >> > > > > > > nearing > > > > > >> > > > >> > > > > > > > > landing > > > > > >> > > > >> > > > > > > > > - HUDI-1098 : Marker file issue with > > > > > >> non-existent > > > > > >> > > > >> files. > > > > > >> > > > >> > > Siva > > > > > >> > > > >> > > > to > > > > > >> > > > >> > > > > > > begin > > > > > >> > > > >> > > > > > > > > impl > > > > > >> > > > >> > > > > > > > > - Spark Streaming + Async > Compaction , > > > > test > > > > > >> > > complete, > > > > > >> > > > >> code > > > > > >> > > > >> > > > > review > > > > > >> > > > >> > > > > > > > > comments and land PR 1752 > > > > > >> > > > >> > > > > > > > > - Spark DataSource/Hive MOR > > Incremental > > > > > Query > > > > > >> > > > HUDI-920 > > > > > >> > > > >> (At > > > > > >> > > > >> > > > risk) > > > > > >> > > > >> > > > > > > > > - Flink/Multi Engine refactor, will > > > need a > > > > > >> large > > > > > >> > > > rebase > > > > > >> > > > >> > and > > > > > >> > > > >> > > > > > rework, > > > > > >> > > > >> > > > > > > > > review, land (At risk for 0.6.0, > high > > > > scope, > > > > > >> may > > > > > >> > > not > > > > > >> > > > >> have > > > > > >> > > > >> > > > enough > > > > > >> > > > >> > > > > > > time) > > > > > >> > > > >> > > > > > > > > - BloomIndex V2 - Global index > > > > > implementation. > > > > > >> > (At > > > > > >> > > > >> risk) > > > > > >> > > > >> > > > > > > > > - HUDI-845 : Parallel writing i.e > > allow > > > > > >> multiple > > > > > >> > > > >> writers > > > > > >> > > > >> > (At > > > > > >> > > > >> > > > > risk) > > > > > >> > > > >> > > > > > > > > - HUDI-860 : Small File Handling > > without > > > > > >> memory > > > > > >> > > > caching > > > > > >> > > > >> > (At > > > > > >> > > > >> > > > > risk) > > > > > >> > > > >> > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > >> > > > >> > > > > > > > > Thanks > > > > > >> > > > >> > > > > > > > > Vinoth > > > > > >> > > > >> > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > -- > > > > > >> > > > >> > > > > > > Regards, > > > > > >> > > > >> > > > > > > -Sivabalan > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > -- *Allen Underwood* Principal Software Engineer Broadcom | Symantec Enterprise Division *Mobile*: 404.808.5926
smime.p7s
Description: S/MIME Cryptographic Signature
