Hi Allen, Yes, it's a serialization (runtime) issue. I am working on fixing it.
On Wed, Aug 19, 2020 at 7:04 PM Sivabalan <[email protected]> wrote: > That would be of great help Allen. Much appreciated. > > On Wed, Aug 19, 2020 at 9:30 AM Allen Underwood > <[email protected]> wrote: > > > Thanks Sivabalan, > > > > That's definitely the issue I had to resolve when introducing Joda time. > > I'll have a look back at my code to see how I got around it. If I > remember > > correctly, in the original code, the DateTime object was being saved as > an > > instance variable on the object and I had to do things differently in the > > Joda time approach - but it turned out not to be a big deal because the > > instance variable wasn't needed. > > > > I'll see if I can get a PR in to help fix this. > > > > The worst part is no unit tests catch this - it's a runtime issue I > assume > > due to how Spark serializes state. > > > > Thanks, > > > > Allen > > > > On Wed, Aug 19, 2020 at 9:23 AM Sivabalan <[email protected]> wrote: > > > >> I am not sure if all findings so far have been documented here. but this > >> is > >> > >> > >> the ticket AFSIK: https://issues.apache.org/jira/browse/HUDI-1177 > >> > >> > >> > >> > >> > >> On Wed, Aug 19, 2020 at 9:15 AM Allen Underwood > >> > >> > >> <[email protected]> wrote: > >> > >> > >> > >> > >> > >> > Just out of curiosity - what's the blocker - you have an issue? I had > >> > >> > >> > originally done the code to make that work. > >> > >> > >> > > >> > >> > >> > On Wed, Aug 19, 2020 at 12:51 AM Vinoth Chandar <[email protected]> > >> wrote: > >> > >> > >> > > >> > >> > >> >> We still have 1 blocker issue from TimestampKeyGenerator issue with > >> joda > >> > >> > >> >> DateTimeFormatter. Sudha (RM) and Pratyaksh are going to look into > >> this. > >> > >> > >> >> > >> > >> > >> >> In the meantime, here's the progress, plans around testing so far. If > >> > >> > >> >> folks > >> > >> > >> >> in the community can help test the release branch in the next couple > of > >> > >> > >> >> days, it would be of great help! > >> > >> > >> >> > >> > >> > >> >> > >> > >> > >> >> - Testing plan here: Stability tested by balaji running spark > >> > >> > >> >> streaming, > >> > >> > >> >> Correctness (Nishith testing on release branch, Udit to help with > >> EMR > >> > >> > >> >> setup) > >> > >> > >> >> - Testing plan continued: Spark datasource MOR, .. Marker based > >> > >> > >> >> rollback, upgrade-downgrade (vinoth is on this) > >> > >> > >> >> - Testing plan : bootstrap (balaji tested for hive), udit has it > >> > >> > >> >> working > >> > >> > >> >> for Presto in EMR. > >> > >> > >> >> - Testing plan: bulk_insert v2 (vinoth tested for correctness, > file > >> > >> > >> >> sizes. ), performance (tested on a cluster, microbenchmarks) > >> > >> > >> >> > >> > >> > >> >> > >> > >> > >> >> There will also be some blogs/docs to explain new stuff. > >> > >> > >> >> > >> > >> > >> >> - Blogs to explain new stuff: sort modes (vinoth), simple index > >> > >> > >> >> (vinoth), bootstrap (balaji), multi delta-streamer (pratyaksh), > >> Spark > >> > >> > >> >> Streaming/DeltaStreamer continuous mode blog (balaji) > >> > >> > >> >> - Docs: All missing docs will be added by Sudha. > >> > >> > >> >> > >> > >> > >> >> > >> > >> > >> >> > >> > >> > >> >> > >> > >> > >> >> On Tue, Aug 18, 2020 at 1:58 AM Bhavani Sudha < > [email protected] > >> > > >> > >> > >> >> wrote: > >> > >> > >> >> > >> > >> > >> >> > Quick update on the RC. > >> > >> > >> >> > > >> > >> > >> >> > Found a build issue when building scala 2.12 and sent a PR for > that - > >> > >> > >> >> > https://github.com/apache/hudi/pull/1976 . Working on resolving > >> this in > >> > >> > >> >> > the > >> > >> > >> >> > release branch and updating RC. Will update soon. > >> > >> > >> >> > > >> > >> > >> >> > Thanks, > >> > >> > >> >> > Sudha > >> > >> > >> >> > > >> > >> > >> >> > On Fri, Aug 14, 2020 at 5:56 PM Vinoth Chandar <[email protected]> > >> > >> > >> >> wrote: > >> > >> > >> >> > > >> > >> > >> >> > > Thanks Sudha! This is means master is now open for regular PRs. > >> Thanks > >> > >> > >> >> > for > >> > >> > >> >> > > your patience, everyone. > >> > >> > >> >> > > > >> > >> > >> >> > > On Fri, Aug 14, 2020 at 3:51 PM Bhavani Sudha < > >> > >> > >> >> [email protected]> > >> > >> > >> >> > > wrote: > >> > >> > >> >> > > > >> > >> > >> >> > > > Hello all, > >> > >> > >> >> > > > > >> > >> > >> >> > > > We have cut the release branch - > >> > >> > >> >> > > > https://github.com/apache/hudi/tree/release-0.6.0 . Since it > is > >> > >> > >> >> > already > >> > >> > >> >> > > > Friday, we will be sending the release candidate early next > week > >> > >> > >> >> (after > >> > >> > >> >> > > > some testing). > >> > >> > >> >> > > > > >> > >> > >> >> > > > Happy Friday! > >> > >> > >> >> > > > > >> > >> > >> >> > > > Thanks, > >> > >> > >> >> > > > Sudha > >> > >> > >> >> > > > > >> > >> > >> >> > > > On Wed, Aug 12, 2020 at 3:56 PM [email protected] < > >> > >> > >> >> [email protected] > >> > >> > >> >> > > > >> > >> > >> >> > > > wrote: > >> > >> > >> >> > > > > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > Hi Folks, > >> > >> > >> >> > > > > We are continuing to work on CI stabilization and will cut > the > >> > >> > >> >> > release > >> > >> > >> >> > > > > once we stabilize the builds hopefully tonight/tomorrow. > >> > >> > >> >> > > > > Thanks,Balaji.V > >> > >> > >> >> > > > > On Tuesday, August 11, 2020, 09:15:05 PM PDT, Vinoth > >> Chandar < > >> > >> > >> >> > > > > [email protected]> wrote: > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > Hello all, > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > Update on this. We have landed most of the blockers for the > >> 0.6.0 > >> > >> > >> >> > > release > >> > >> > >> >> > > > > and I am currently working on the last major blocker, > >> HUDI-1013. > >> > >> > >> >> > > > > We are working through some unexpected CI flakiness. We hope > to > >> > >> > >> >> > > stabilize > >> > >> > >> >> > > > > master, cut the RC, and then open up master for regular PR > >> merges. > >> > >> > >> >> > > > > ETA for this is tomorrow night PST (Aug 12, PST). > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > We will keep this thread posted! > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > Thanks > >> > >> > >> >> > > > > Vinoth > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > On Tue, Aug 4, 2020 at 9:47 PM Vinoth Chandar < > >> [email protected]> > >> > >> > >> >> > > wrote: > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > > Small correction: > >> > >> > >> >> > > > > > > >> > >> > >> >> > > > > > >> Vinoth working on code review, tests for PR 1876, > >> > >> > >> >> > > > > > This is landed! > >> > >> > >> >> > > > > > > >> > >> > >> >> > > > > > > >> > >> > >> >> > > > > > On Tue, Aug 4, 2020 at 9:44 PM Bhavani Sudha < > >> > >> > >> >> > > [email protected]> > >> > >> > >> >> > > > > > wrote: > >> > >> > >> >> > > > > > > >> > >> > >> >> > > > > >> Hello all, > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > >> We are targeting the end of this week to cut RC. Here is > an > >> > >> > >> >> update > >> > >> > >> >> > > of > >> > >> > >> >> > > > > >> where > >> > >> > >> >> > > > > >> we are at release blockers. > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > >> 0.6.0 Release blocker status (board > >> > >> > >> >> > > > > >> < > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > >> > >> > >> >> > > > >> > >> > >> >> > > >> > >> > >> >> > >> > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69 > >> > >> > >> >> > > > > >> >) > >> > >> > >> >> > > > > >> , > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > >> - Spark Datasource/MOR > >> > >> > >> >> > https://github.com/apache/hudi/pull/1848 > >> > >> > >> >> > > > > needs > >> > >> > >> >> > > > > >> to > >> > >> > >> >> > > > > >> be tested by gary/balaji (About to land) > >> > >> > >> >> > > > > >> - Hive Sync restructuring (Review done, about to land) > >> > >> > >> >> > > > > >> - Bootstrap > >> > >> > >> >> > > > > >> - Vinoth working on code review, tests for PR 1876, > >> > >> > >> >> > > > > >> - then udit will rework PR 1702 (In Code review) > >> > >> > >> >> > > > > >> - then we will review, land PR 1870, 1869 > >> > >> > >> >> > > > > >> - Bulk insert V2 PR 1834, lower risk, independent PR, > >> well > >> > >> > >> >> > tested > >> > >> > >> >> > > > > >> already > >> > >> > >> >> > > > > >> - Dependent PR 1149 to be landed, > >> > >> > >> >> > > > > >> - and modes to be respected in V2 impl as well (At > >> risk) > >> > >> > >> >> > > > > >> - Upgrade Downgrade Hooks, PR 1858 : (In Code review) > >> > >> > >> >> > > > > >> - HUDI-1054- Marker list perf improvement, Udit has a > PR > >> out > >> > >> > >> >> > > > > >> - HUDI-115 : Overwrite with... ordering issue, Sudha > has > >> a > >> > >> > >> >> PR > >> > >> > >> >> > > > nearing > >> > >> > >> >> > > > > >> landing > >> > >> > >> >> > > > > >> - HUDI-1098 : Marker file issue with non-existent > files. > >> (In > >> > >> > >> >> > Code > >> > >> > >> >> > > > > >> review) > >> > >> > >> >> > > > > >> - Spark Streaming + Async Compaction , test complete, > >> code > >> > >> > >> >> > review > >> > >> > >> >> > > > > >> comments and land PR 1752 (About to land) > >> > >> > >> >> > > > > >> - Spark DataSource/Hive MOR Incremental Query HUDI-920 > >> (At > >> > >> > >> >> > risk) > >> > >> > >> >> > > > > >> - Flink/Multi Engine refactor, will need a large rebase > >> and > >> > >> > >> >> > > rework, > >> > >> > >> >> > > > > >> review, land (At risk for 0.6.0) > >> > >> > >> >> > > > > >> - BloomIndex V2 - Global index implementation. (At > risk) > >> > >> > >> >> > > > > >> - HUDI-845 : Parallel writing i.e allow multiple > writers > >> > >> > >> >> > (Pushed > >> > >> > >> >> > > > out > >> > >> > >> >> > > > > of > >> > >> > >> >> > > > > >> 0.6.0) > >> > >> > >> >> > > > > >> - HUDI-860 : Small File Handling without memory caching > >> > >> > >> >> (Pushed > >> > >> > >> >> > > out > >> > >> > >> >> > > > > of > >> > >> > >> >> > > > > >> 0.6.0) > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > >> Thanks, > >> > >> > >> >> > > > > >> Sudha > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > >> On Mon, Aug 3, 2020 at 3:41 PM Vinoth Chandar < > >> > >> > >> >> [email protected]> > >> > >> > >> >> > > > > wrote: > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > >> > +1 (we need to formalize this well) > >> > >> > >> >> > > > > >> > But having just blockers land first, would help not just > >> with > >> > >> > >> >> > > > > rebasing, > >> > >> > >> >> > > > > >> but > >> > >> > >> >> > > > > >> > also wind down towards cutting an RC by end of week. > >> > >> > >> >> > > > > >> > > >> > >> > >> >> > > > > >> > > >> > >> > >> >> > > > > >> > On Mon, Aug 3, 2020 at 2:53 PM Bhavani Sudha < > >> > >> > >> >> > > > [email protected] > >> > >> > >> >> > > > > > > >> > >> > >> >> > > > > >> > wrote: > >> > >> > >> >> > > > > >> > > >> > >> > >> >> > > > > >> > > Hello all, > >> > >> > >> >> > > > > >> > > > >> > >> > >> >> > > > > >> > > As we are all hustling towards getting the blockers > in, > >> I > >> > >> > >> >> > wanted > >> > >> > >> >> > > > to > >> > >> > >> >> > > > > >> > propose > >> > >> > >> >> > > > > >> > > a code/merge freeze until we cut a release for 0.6.0 > >> and > >> > >> > >> >> > > restrict > >> > >> > >> >> > > > > it > >> > >> > >> >> > > > > >> to > >> > >> > >> >> > > > > >> > > only merging blockers identified for this release. It > >> would > >> > >> > >> >> > > reduce > >> > >> > >> >> > > > > >> > rebasing > >> > >> > >> >> > > > > >> > > time for blockers in progress. If we feel some issue > is > >> a > >> > >> > >> >> > > serious > >> > >> > >> >> > > > > >> blocker > >> > >> > >> >> > > > > >> > > we can discuss it here and bump it's priority. > >> > >> > >> >> > > > > >> > > > >> > >> > >> >> > > > > >> > > Please share your thoughts or concerns. > >> > >> > >> >> > > > > >> > > > >> > >> > >> >> > > > > >> > > Thanks, > >> > >> > >> >> > > > > >> > > Sudha > >> > >> > >> >> > > > > >> > > > >> > >> > >> >> > > > > >> > > > >> > >> > >> >> > > > > >> > > On Mon, Aug 3, 2020 at 8:19 AM Vinoth Chandar < > >> > >> > >> >> > > [email protected]> > >> > >> > >> >> > > > > >> wrote: > >> > >> > >> >> > > > > >> > > > >> > >> > >> >> > > > > >> > > > Given enough time has passed, Sudha can be our RM > for > >> > >> > >> >> 0.6.0. > >> > >> > >> >> > > > > >> > > > > >> > >> > >> >> > > > > >> > > > On the release blocker progress, we landed few > >> blockers > >> > >> > >> >> over > >> > >> > >> >> > > the > >> > >> > >> >> > > > > >> > weekend, > >> > >> > >> >> > > > > >> > > > with some almost ready for landing > >> > >> > >> >> > > > > >> > > > > >> > >> > >> >> > > > > >> > > > Will send out a status update again tomorrow night > >> PST! > >> > >> > >> >> > > > > >> > > > > >> > >> > >> >> > > > > >> > > > On Mon, Aug 3, 2020 at 8:17 AM Vinoth Chandar < > >> > >> > >> >> > > > [email protected]> > >> > >> > >> >> > > > > >> > wrote: > >> > >> > >> >> > > > > >> > > > > >> > >> > >> >> > > > > >> > > > > Hi anton. > >> > >> > >> >> > > > > >> > > > > > >> > >> > >> >> > > > > >> > > > > We were hoping to cut a release by last weekend. > New > >> > >> > >> >> > target > >> > >> > >> >> > > is > >> > >> > >> >> > > > > >> this > >> > >> > >> >> > > > > >> > > > > weekend! > >> > >> > >> >> > > > > >> > > > > (tbh we were thrown off a bit due to COVID in Q2, > >> > >> > >> >> given a > >> > >> > >> >> > > lot > >> > >> > >> >> > > > of > >> > >> > >> >> > > > > >> > > > > PMC/Committers had additional kid care duties. Now > >> we > >> > >> > >> >> are > >> > >> > >> >> > > back > >> > >> > >> >> > > > > to > >> > >> > >> >> > > > > >> > > normal > >> > >> > >> >> > > > > >> > > > > cadence) > >> > >> > >> >> > > > > >> > > > > > >> > >> > >> >> > > > > >> > > > > Going forward, I plan to start a discussion around > >> > >> > >> >> > planning, > >> > >> > >> >> > > > > >> > > prioritizing > >> > >> > >> >> > > > > >> > > > > and other release processes after 0.6.0. Would be > >> > >> > >> >> great to > >> > >> > >> >> > > > have > >> > >> > >> >> > > > > >> the > >> > >> > >> >> > > > > >> > > > > community weigh in even more in these things > >> upfront. > >> > >> > >> >> > > > > >> > > > > > >> > >> > >> >> > > > > >> > > > > Thanks > >> > >> > >> >> > > > > >> > > > > Vinoth > >> > >> > >> >> > > > > >> > > > > > >> > >> > >> >> > > > > >> > > > > On Fri, Jul 31, 2020 at 6:49 PM Anton Zuyeu < > >> > >> > >> >> > > > > >> [email protected]> > >> > >> > >> >> > > > > >> > > > wrote: > >> > >> > >> >> > > > > >> > > > > > >> > >> > >> >> > > > > >> > > > >> Hi All, > >> > >> > >> >> > > > > >> > > > >> > >> > >> > >> >> > > > > >> > > > >> I apologize for possibly dumb question but when > was > >> > >> > >> >> 0.6.0 > >> > >> > >> >> > > > > >> planned to > >> > >> > >> >> > > > > >> > > be > >> > >> > >> >> > > > > >> > > > >> released? Can't find any dates on Hudi related > >> pages. > >> > >> > >> >> > > > > >> > > > >> > >> > >> > >> >> > > > > >> > > > >> On Thu, Jul 30, 2020 at 10:36 AM Vinoth Chandar < > >> > >> > >> >> > > > > >> [email protected]> > >> > >> > >> >> > > > > >> > > > >> wrote: > >> > >> > >> >> > > > > >> > > > >> > >> > >> > >> >> > > > > >> > > > >> > Is anyone able to help with the at risk items? > :) > >> > >> > >> >> > > > > >> > > > >> > > >> > >> > >> >> > > > > >> > > > >> > On Thu, Jul 30, 2020 at 7:07 AM leesf < > >> > >> > >> >> > > [email protected] > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > >> > wrote: > >> > >> > >> >> > > > > >> > > > >> > > >> > >> > >> >> > > > > >> > > > >> > > @Vinoth Chandar <[email protected]> Thanks > for > >> > >> > >> >> the > >> > >> > >> >> > > > > reminder, > >> > >> > >> >> > > > > >> > > marked > >> > >> > >> >> > > > > >> > > > >> to > >> > >> > >> >> > > > > >> > > > >> > > blocker, and next week would be ok to me. > >> > >> > >> >> > > > > >> > > > >> > > > >> > >> > >> >> > > > > >> > > > >> > > Vinoth Chandar <[email protected]> > >> 于2020年7月30日周四 > >> > >> > >> >> > > > > 上午11:35写道： > >> > >> > >> >> > > > > >> > > > >> > > > >> > >> > >> >> > > > > >> > > > >> > > > @leesf <[email protected]> can we > please > >> > >> > >> >> mark > >> > >> > >> >> > the > >> > >> > >> >> > > > > >> relevant > >> > >> > >> >> > > > > >> > > > >> > ticket(s) > >> > >> > >> >> > > > > >> > > > >> > > > with blocker priority, so it's easier to > >> track? > >> > >> > >> >> > > > > >> > > > >> > > > > >> > >> > >> >> > > > > >> > > > >> > > > Looks like we are nearing a choice for RM. > >> > >> > >> >> > > > > >> > > > >> > > > Any more thoughts on timelines? Looks like > >> > >> > >> >> everyone > >> > >> > >> >> > > so > >> > >> > >> >> > > > > far > >> > >> > >> >> > > > > >> is > >> > >> > >> >> > > > > >> > > > >> leaning > >> > >> > >> >> > > > > >> > > > >> > > > towards completeness of the release over > >> doing > >> > >> > >> >> it > >> > >> > >> >> > > > sooner? > >> > >> > >> >> > > > > >> > > > >> > > > > >> > >> > >> >> > > > > >> > > > >> > > > On Wed, Jul 29, 2020 at 6:36 PM vino yang < > >> > >> > >> >> > > > > >> > > [email protected]> > >> > >> > >> >> > > > > >> > > > >> > wrote: > >> > >> > >> >> > > > > >> > > > >> > > > > >> > >> > >> >> > > > > >> > > > >> > > > > +1 on Sudha being RM for the release. And > >> > >> > >> >> looking > >> > >> > >> >> > > > > >> forward to > >> > >> > >> >> > > > > >> > > > >> 0.6.0. > >> > >> > >> >> > > > > >> > > > >> > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > Best, > >> > >> > >> >> > > > > >> > > > >> > > > > Vino > >> > >> > >> >> > > > > >> > > > >> > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > leesf <[email protected]> > 于2020年7月30日周四 > >> > >> > >> >> > > 上午9:15写道： > >> > >> > >> >> > > > > >> > > > >> > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > +1 on Sudha on being RM, and PR#1810 > >> > >> > >> >> > > > > >> > > > >> > > > > > > https://github.com/apache/hudi/pull/1810 > >> > >> > >> >> > > (abstract > >> > >> > >> >> > > > > >> hive > >> > >> > >> >> > > > > >> > > sync > >> > >> > >> >> > > > > >> > > > >> > module) > >> > >> > >> >> > > > > >> > > > >> > > > > would > >> > >> > >> >> > > > > >> > > > >> > > > > > also goes to this release. > >> > >> > >> >> > > > > >> > > > >> > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > Sivabalan <[email protected]> > >> > >> > >> >> 于2020年7月30日周四 > >> > >> > >> >> > > > > 上午2:18写道： > >> > >> > >> >> > > > > >> > > > >> > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > +1 on Sudha being RM for the release. > >> > >> > >> >> Makes > >> > >> > >> >> > > sense > >> > >> > >> >> > > > > to > >> > >> > >> >> > > > > >> > push > >> > >> > >> >> > > > > >> > > > the > >> > >> > >> >> > > > > >> > > > >> > > release > >> > >> > >> >> > > > > >> > > > >> > > > > by > >> > >> > >> >> > > > > >> > > > >> > > > > > a > >> > >> > >> >> > > > > >> > > > >> > > > > > > week. > >> > >> > >> >> > > > > >> > > > >> > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > On Wed, Jul 29, 2020 at 1:35 AM > >> > >> > >> >> > > > [email protected] > >> > >> > >> >> > > > > < > >> > >> > >> >> > > > > >> > > > >> > > > [email protected] > >> > >> > >> >> > > > > >> > > > >> > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > wrote: > >> > >> > >> >> > > > > >> > > > >> > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > +1 on Sudha on being RM for this > >> > >> > >> >> release. > >> > >> > >> >> > > Also > >> > >> > >> >> > > > > >> agree > >> > >> > >> >> > > > > >> > on > >> > >> > >> >> > > > > >> > > > >> > pushing > >> > >> > >> >> > > > > >> > > > >> > > > the > >> > >> > >> >> > > > > >> > > > >> > > > > > > > release date by a week. > >> > >> > >> >> > > > > >> > > > >> > > > > > > > Balaji.V > >> > >> > >> >> > > > > >> > > > >> > > > > > > > On Tuesday, July 28, 2020, > >> 10:08:41 > >> > >> > >> >> PM > >> > >> > >> >> > > PDT, > >> > >> > >> >> > > > > >> > Bhavani > >> > >> > >> >> > > > > >> > > > >> Sudha < > >> > >> > >> >> > > > > >> > > > >> > > > > > > > [email protected]> wrote: > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > Thanks Vinoth for the update. I > can > >> > >> > >> >> > > volunteer > >> > >> > >> >> > > > to > >> > >> > >> >> > > > > >> RM > >> > >> > >> >> > > > > >> > > this > >> > >> > >> >> > > > > >> > > > >> > > release. > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > Understand 0.6.0 release is delayed > >> than > >> > >> > >> >> > what > >> > >> > >> >> > > > we > >> > >> > >> >> > > > > >> > > > originally > >> > >> > >> >> > > > > >> > > > >> > > > > discussed. > >> > >> > >> >> > > > > >> > > > >> > > > > > Q2 > >> > >> > >> >> > > > > >> > > > >> > > > > > > > has been really hard with COVID and > >> > >> > >> >> > > everything > >> > >> > >> >> > > > > >> going > >> > >> > >> >> > > > > >> > on. > >> > >> > >> >> > > > > >> > > > >> Given > >> > >> > >> >> > > > > >> > > > >> > > that > >> > >> > >> >> > > > > >> > > > >> > > > > we > >> > >> > >> >> > > > > >> > > > >> > > > > > > are > >> > >> > >> >> > > > > >> > > > >> > > > > > > > at this point, I feel by delaying > >> the RC > >> > >> > >> >> > by a > >> > >> > >> >> > > > > week > >> > >> > >> >> > > > > >> or > >> > >> > >> >> > > > > >> > so > >> > >> > >> >> > > > > >> > > > >> more > >> > >> > >> >> > > > > >> > > > >> > if > >> > >> > >> >> > > > > >> > > > >> > > we > >> > >> > >> >> > > > > >> > > > >> > > > > can > >> > >> > >> >> > > > > >> > > > >> > > > > > > get > >> > >> > >> >> > > > > >> > > > >> > > > > > > > some of the 'At risk' items in, I > >> would > >> > >> > >> >> > vote > >> > >> > >> >> > > > for > >> > >> > >> >> > > > > >> that. > >> > >> > >> >> > > > > >> > > > That > >> > >> > >> >> > > > > >> > > > >> is > >> > >> > >> >> > > > > >> > > > >> > > just > >> > >> > >> >> > > > > >> > > > >> > > > > my > >> > >> > >> >> > > > > >> > > > >> > > > > > > > personal opinion. I ll let others > >> chime > >> > >> > >> >> in. > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > Thanks, > >> > >> > >> >> > > > > >> > > > >> > > > > > > > Sudha > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > On Tue, Jul 28, 2020 at 9:48 PM > >> Vinoth > >> > >> > >> >> > > Chandar > >> > >> > >> >> > > > < > >> > >> > >> >> > > > > >> > > > >> > > [email protected]> > >> > >> > >> >> > > > > >> > > > >> > > > > > > wrote: > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > Hello all, > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > Just wanted to kickstart a thread > >> to > >> > >> > >> >> firm > >> > >> > >> >> > > up > >> > >> > >> >> > > > > the > >> > >> > >> >> > > > > >> RC > >> > >> > >> >> > > > > >> > > cut > >> > >> > >> >> > > > > >> > > > >> date > >> > >> > >> >> > > > > >> > > > >> > > for > >> > >> > >> >> > > > > >> > > > >> > > > > > 0.6.0 > >> > >> > >> >> > > > > >> > > > >> > > > > > > > and > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > pick a RM. (any volunteers?, if > >> not I > >> > >> > >> >> > self > >> > >> > >> >> > > > > >> nominate > >> > >> > >> >> > > > > >> > > > >> myself) > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > Here's an update on where we are > at > >> > >> > >> >> with > >> > >> > >> >> > > the > >> > >> > >> >> > > > > >> > remaining > >> > >> > >> >> > > > > >> > > > >> > release > >> > >> > >> >> > > > > >> > > > >> > > > > > > blockers. > >> > >> > >> >> > > > > >> > > > >> > > > > > > > I > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > have marked items as "At risk" > >> > >> > >> >> assuming > >> > >> > >> >> > we > >> > >> > >> >> > > > cut > >> > >> > >> >> > > > > RC > >> > >> > >> >> > > > > >> > > > sometime > >> > >> > >> >> > > > > >> > > > >> > next > >> > >> > >> >> > > > > >> > > > >> > > > > week. > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > Please chime in with your > thoughts. > >> > >> > >> >> > > Ideally, > >> > >> > >> >> > > > we > >> > >> > >> >> > > > > >> > don't > >> > >> > >> >> > > > > >> > > > take > >> > >> > >> >> > > > > >> > > > >> > any > >> > >> > >> >> > > > > >> > > > >> > > > more > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > blockers. If we also want to > knock > >> off > >> > >> > >> >> > the > >> > >> > >> >> > > at > >> > >> > >> >> > > > > >> risk > >> > >> > >> >> > > > > >> > > > items, > >> > >> > >> >> > > > > >> > > > >> > then > >> > >> > >> >> > > > > >> > > > >> > > we > >> > >> > >> >> > > > > >> > > > >> > > > > > would > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > at-least push dates by another > week > >> > >> > >> >> (my > >> > >> > >> >> > > > guess). > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > 0.6.0 Release blocker status > (board > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > < > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > >> > >> > >> >> > > > > >> > > > >> > > > >> > >> > >> >> > > > > >> > > > >> > > >> > >> > >> >> > > > > >> > > > >> > >> > >> > >> >> > > > > >> > > > > >> > >> > >> >> > > > > >> > > > >> > >> > >> >> > > > > >> > > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > > >> > >> > >> >> > > > > >> > >> > >> >> > > > >> > >> > >> >> > > >> > >> > >> >> > >> > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69 > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > >) > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > , > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Spark Datasource/MOR > >> > >> > >> >> > > > > >> > > > >> > > > https://github.com/apache/hudi/pull/1848 > >> > >> > >> >> > > > > >> > > > >> > > > > > > needs > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > to > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > be tested by gary/balaji > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Bootstrap > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Vinoth working on code > >> review, > >> > >> > >> >> > tests > >> > >> > >> >> > > > for > >> > >> > >> >> > > > > >> PR > >> > >> > >> >> > > > > >> > > 1876, > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - then udit will rework PR > >> 1702 > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - then we will review, land > PR > >> > >> > >> >> 1870, > >> > >> > >> >> > > > 1869 > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Also need to fix HUDI-999, > >> > >> > >> >> > HUDI-1021 > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Bulk insert V2 PR 1834, > lower > >> > >> > >> >> risk, > >> > >> > >> >> > > > > >> independent > >> > >> > >> >> > > > > >> > > PR, > >> > >> > >> >> > > > > >> > > > >> well > >> > >> > >> >> > > > > >> > > > >> > > > > tested > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > already > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Dependent PR 1149 to be > >> landed, > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - and modes to be respected > >> in V2 > >> > >> > >> >> > impl > >> > >> > >> >> > > > as > >> > >> > >> >> > > > > >> well > >> > >> > >> >> > > > > >> > > (At > >> > >> > >> >> > > > > >> > > > >> risk) > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Upgrade Downgrade Hooks, PR > >> 1858 > >> > >> > >> >> : > >> > >> > >> >> > > Siva > >> > >> > >> >> > > > > has > >> > >> > >> >> > > > > >> a > >> > >> > >> >> > > > > >> > PR > >> > >> > >> >> > > > > >> > > > out, > >> > >> > >> >> > > > > >> > > > >> > code > >> > >> > >> >> > > > > >> > > > >> > > > > > > > completing > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > this week > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - HUDI-1054- Marker list perf > >> > >> > >> >> > > improvement, > >> > >> > >> >> > > > > >> Udit > >> > >> > >> >> > > > > >> > > has a > >> > >> > >> >> > > > > >> > > > >> PR > >> > >> > >> >> > > > > >> > > > >> > out > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - HUDI-115 : Overwrite with... > >> > >> > >> >> > ordering > >> > >> > >> >> > > > > issue, > >> > >> > >> >> > > > > >> > > Sudha > >> > >> > >> >> > > > > >> > > > >> has a > >> > >> > >> >> > > > > >> > > > >> > > PR > >> > >> > >> >> > > > > >> > > > >> > > > > > > nearing > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > landing > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - HUDI-1098 : Marker file > issue > >> > >> > >> >> with > >> > >> > >> >> > > > > >> non-existent > >> > >> > >> >> > > > > >> > > > >> files. > >> > >> > >> >> > > > > >> > > > >> > > Siva > >> > >> > >> >> > > > > >> > > > >> > > > to > >> > >> > >> >> > > > > >> > > > >> > > > > > > begin > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > impl > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Spark Streaming + Async > >> > >> > >> >> Compaction , > >> > >> > >> >> > > > test > >> > >> > >> >> > > > > >> > > complete, > >> > >> > >> >> > > > > >> > > > >> code > >> > >> > >> >> > > > > >> > > > >> > > > > review > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > comments and land PR 1752 > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Spark DataSource/Hive MOR > >> > >> > >> >> > Incremental > >> > >> > >> >> > > > > Query > >> > >> > >> >> > > > > >> > > > HUDI-920 > >> > >> > >> >> > > > > >> > > > >> (At > >> > >> > >> >> > > > > >> > > > >> > > > risk) > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - Flink/Multi Engine refactor, > >> will > >> > >> > >> >> > > need a > >> > >> > >> >> > > > > >> large > >> > >> > >> >> > > > > >> > > > rebase > >> > >> > >> >> > > > > >> > > > >> > and > >> > >> > >> >> > > > > >> > > > >> > > > > > rework, > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > review, land (At risk for > 0.6.0, > >> > >> > >> >> high > >> > >> > >> >> > > > scope, > >> > >> > >> >> > > > > >> may > >> > >> > >> >> > > > > >> > > not > >> > >> > >> >> > > > > >> > > > >> have > >> > >> > >> >> > > > > >> > > > >> > > > enough > >> > >> > >> >> > > > > >> > > > >> > > > > > > time) > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - BloomIndex V2 - Global index > >> > >> > >> >> > > > > implementation. > >> > >> > >> >> > > > > >> > (At > >> > >> > >> >> > > > > >> > > > >> risk) > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - HUDI-845 : Parallel writing > >> i.e > >> > >> > >> >> > allow > >> > >> > >> >> > > > > >> multiple > >> > >> > >> >> > > > > >> > > > >> writers > >> > >> > >> >> > > > > >> > > > >> > (At > >> > >> > >> >> > > > > >> > > > >> > > > > risk) > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > - HUDI-860 : Small File > Handling > >> > >> > >> >> > without > >> > >> > >> >> > > > > >> memory > >> > >> > >> >> > > > > >> > > > caching > >> > >> > >> >> > > > > >> > > > >> > (At > >> > >> > >> >> > > > > >> > > > >> > > > > risk) > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > Thanks > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > Vinoth > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > -- > >> > >> > >> >> > > > > >> > > > >> > > > > > > Regards, > >> > >> > >> >> > > > > >> > > > >> > > > > > > -Sivabalan > >> > >> > >> >> > > > > >> > > > >> > > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > > >> > >> > >> >> > > > > >> > > > >> > > > > >> > >> > >> >> > > > > >> > > > >> > > > >> > >> > >> >> > > > > >> > > > >> > > >> > >> > >> >> > > > > >> > > > >> > >> > >> > >> >> > > > > >> > > > > > >> > >> > >> >> > > > > >> > > > > >> > >> > >> >> > > > > >> > > > >> > >> > >> >> > > > > >> > > >> > >> > >> >> > > > > >> > >> > >> > >> >> > > > > > > >> > >> > >> >> > > > > >> > >> > >> >> > > > >> > >> > >> >> > > >> > >> > >> >> > >> > >> > >> > > >> > >> > >> > > >> > >> > >> > -- > >> > > > >> > >> > *Allen Underwood* > >> > >> > >> > Principal Software Engineer > >> > >> > >> > Broadcom | Symantec Enterprise Division > >> > >> > >> > *Mobile*: 404.808.5926 > >> > >> > >> > > >> > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> > >> Regards, > >> > >> > >> -Sivabalan > >> > >> > >> > > > > -- > > *Allen Underwood* > > Principal Software Engineer > > Broadcom | Symantec Enterprise Division > > *Mobile*: 404.808.5926 > > > > > > -- > Regards, > -Sivabalan >
