nsivabalan commented on code in PR #11514: URL: https://github.com/apache/hudi/pull/11514#discussion_r1671383928
########## rfc/rfc-78/rfc-78.md: ########## @@ -0,0 +1,301 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +# RFC-76: [Bridge release for 1.x] + +## Proposers + +- @nsivabalan +- @vbalaji + +## Approvers + - @yihua + - @codope + +## Status + +JIRA: https://issues.apache.org/jira/browse/HUDI-7882 + +> Please keep the status updated in `rfc/README.md`. + +## Abstract + +[Hudi 1.x](https://github.com/apache/hudi/blob/ae1ee05ab8c2bd732e57bee11c8748926b05ec4b/rfc/rfc-69/rfc-69.md) is a powerful +re-imagination of the transactional database layer in Hudi to power continued innovation across the community in the coming +years. It introduces lot of differentiating features for Apache Hudi. Feel free to checkout the +[release page](https://hudi.apache.org/releases/release-1.0.0-beta1) for more info. We had beta1 and beta2 releases which was meant for +interested developers/users to give a spin on some of the advanced features. But as we are working towards 1.0 GA, we are proposing +a bridge release (0.16.0) for smoother migration for existing hudi users. + +## Objectives +Goal is to have a smooth migration experience for the users from 0.x to 1.0. We plan to have a 0.16.0 bridge release asking everyone to first migrate to 0.16.0 before they can upgrade to 1.x. + +A typical organization might have a medallion architecture deployed to run 1000s of Hudi pipelines i.e. bronze, silver and gold layer. +For this layout of pipelines, here is how a typical migration might look like(w/o a bridge release) + +a. Existing pipelines are in 0.15.x. (bronze, silver, gold) +b. Migrate gold pipelines to 1.x. +- We need to strictly migrate only gold to 1x. Bcoz, a 0.15.0 reader may not be able to read 1.x hudi tables. So, if we migrate any of silver pipelines to 1.x before migrating entire gold layer, we might end up in a situation, +where a 0.15.0 reader (gold) might end up reading 1.x table (silver). This might lead to failures. So, we have to follow certain order in which we migrate pipelines. +c. Once all of gold is migrated to 1.x, we can move all of silver to 1.x. +d. Once all of gold and silver pipelines are migrated to 1.x, finally we can move all of bronze to 1.x. + +In the end, we would have migrated all of existing hudi pipelines from 0.15.0 to 1.x. +But as you could see, we need some coordination with which we need to migrate. And in a very large organization, sometimes we may not have good control over downstream consumers. +Hence, coordinating entire migration workflow and orchestrating the same might be challenging. + +Hence to ease the migration workflow for 1.x, we are introducing 0.16.0 as a bridge release. + +Here are the objectives with this bridge release: + +- 1.x reader should be able to read 0.14.x to 0.16.x tables w/o any loss in functionality and no data inconsistencies. +- 0.16.x should have read capability for 1.x tables w/ some limitations. For features ported over from 0.x, no loss in functionality should be guaranteed. +But for new features that was introduced in 1.x, we may not be able to support all of them. Will be calling out which new features may not work with 0.16.x reader. +- In this case, we explicitly request users to not turn on these features untill all readers are completely migrated to 1.x so as to not break any readers as applicable. + +Connecting back to our example above, lets see how the migration might look like for an existing user. + +a. Existing pipelines are in 0.15.x. (bronze, silver, gold) +b. Migrate pipelines to 0.16.0 (in any order. we do not have any constraints around which pipeline should be migrated first). +c. Ensure all pipelines are in 0.16.0 (both readers and writers) +d. Start migrating pipelines in a rolling fashion to 1.x. At this juncture, we could have few pipelines in 1.x and few pipelines in 0.16.0. but since 0.16.x +can read 1.x tables, we should be ok here. Just that do not enable new features like Non blocking concurrency control yet. +e. Migrate all of 0.16.0 to 1.x version. +f. Once all readers and writers are in 1.x, we are good to enable any new features (like NBCC) with 1.x tables. + +As you could see, company/org wide coordination to migrate gold before migrating silver or bronze is relaxed with the bridge release. Only requirement to keep a tab on, +is to ensure to migrate all pipelines completely to 0.16.x before starting to migrate to 1.x. + +So, here are the objectives of this RFC with the bridge release. +- 1.x reader should be able to read 0.14.x to 0.16.x tables w/o any loss in functionality and no data inconsistencies. +- 0.16.x should have read capability for 1.x tables w/ some limitations. For features ported over from 0.x, no loss in functionality should be guaranteed. + But for new features that was introduced in 1.x, we may not be able to support all of them. Will be calling out which new features may not work with 0.16.x reader. +- Document steps for rolling upgrade from 0.16.x to 1.x , with minimal downtime. +- Downgrade from 1.x to 0.16.x documented with call outs on any functionality loss. + +### Considerations when choosing Migration strategy +- While migration is happening, we want to allow readers to continue reading data. This means, we cannot employ a stop-the-world strategy when we are migrating. +All the actions that we are performing as part of table upgrade should not have any side-effects of breaking snapshot isolation for readers. +- Also, users should have migrated to 0.16.x before upgrading to 1.x. We do not want to add read support for very old versions of hudi in 1.x(for eg 0.7.0). +- So, in an effort to bring everyone to latest hudi versions, 1.x reader will have full read capabilities for 0.16.x, but for older hudi versions, 1.x reader may not have full reader support. +The reocmmended guideline is to upgrade all readers and writers to 0.16.x. and then slowly start upgrading to 1.x(readers followed by writers). + +Before we dive in further, lets understand the format changes: + +## Format changes +### Table properties +- Payload class ➝ payload type. +- hoodie.record.merge.mode +- New metadata partitions could be added (optionally enabled) + +### MDT changes +- New MDT partitions are available in 1.x. MDT schema upgraded. +- RLI schema is upgraded to hold row positions. + +### Timeline: +- [storage changes] Completed write commits have completed times in the file name. +- [storage changes] Completed and inflight write commits are in avro format which were json in 0.x. +- We are switching the action type for pending clustering from “replace commit” to “cluster”. +- [storage changes] Timeline ➝ LST timeline. There is no archived timeline in 1.x. +- [In-memory changes] HoodieInstant changes due to presence of completion time for completed HoodieInstants. + +### Filegroup/FileSlice changes: +- Log file names contain delta commit time instead of base instant time. +- Log appends are disabled in 1.x. In other words, each log block is already appended to a new log file. +- File Slice determination logic for log files changed. In 0.x, we have base instant time in log files and its straight forward. +In 1.x, we find completion time for a log file and find the base instant time (parsed from base files) for a given HoodieFileGroup, +- which has the highest value lesser than the completion time of the log file) of interest. +- Log file ordering within a file slice. (in 0.x, we use base instant time ➝ log file versions ➝ write token) to order diff log files. in 1.x, we will be using completion time to order). +- Rollbacks in 0.x appends a new rollback block (new log file). While in 1.x, rollback will remove the partially failed log files. + +### Log format changes: +- We have added new header types in 1.x. (IS_PARTIAL) + +## Changes to be ported over to 0.16.x to support reading 1.x tables + +### What will be supported +- For features introduced in 0.x, and tables written in 1.x, 0.16.0 reader should be able to provide consistent reads w/o any breakage. +- +### What will not be supported +- A 0.16 writer cannot write to a table that has been upgraded-to/created using 1.x without downgrading to 0.16. Might be obvious, but calling it out nevertheless. +- For new features introduced in 1.x, we might call out that 0.16.x reader may not support reads. + +| 1.x Features written by 1.x writer | 1.x reader | 0.16.x Reader | +|----------------|-------------------------------------------|-------| +| Deletion vector | supported | Falls back to key based merges giving up on perf optimization +| Partial merges/updates | supported | Fails with clear error message stating that partial merging | +| Functional indexes | supported | Not supported. Perf optimization may not kick in| +| Secondary indexes | supported | Not supported. Perf optimization may not kick in| +| NBCC OR Completion time based log file ordering in a file slice | supported | Not supported. Will be using log file version and write token based ordering.| + + +### Timeline +- Timeline read of 1.x need to be supported. Review Comment: Our idea here is to not port LSM archived timeline reader to 0.16.x. We need to fix 1.x timeline to align uncleaned timeline as active and rest of it as archived or a minimum of 2 days of timeline. For eg, Case 1: Someone configures cleaner to 4 days. 1.x active timeline will be 4 days and rest will go into LSM Case2: Someone configures cleaner to 1 day: 1.x active timeline will be 2 days and rest will go into LSM. If we maintain this semantics, with 0.16.x reader, we don't need to port LSM reader. just reading active timeline would suffice for all of snapshot, time travel and incremental queries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
