[
https://issues.apache.org/jira/browse/HUDI-8076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-8076:
---------------------------------
Description:
(!) Work in Progress.
h3. Basic Idea:
Introduce support for Hudi writer code to produce storage format for the last
2-3 table versions. This enables older readers to continue reading the table
even when writers are upgraded, as long as writer produces storage compatible
with the latest table version the reader can read. The readers can then be
rolling upgraded easily, at different cadence without need for any tight
co-ordination. Additionally, the reader should have ability to "dynamically"
deduce table version based on table properties, such that when the writer is
switched to the latest table version, subsequent reads will just adapt and read
it as the latest table version.
Operators still need to ensure all readers have the latest binary that supports
a given table version, before switching the writer to that version. Special
consideration to table services, as reader/writer processes, that should be
able manage the tables as well. Queries should gracefully fail during table
version switches and start eventually succeeding when writer completes
switching.
h3. High level approach:
was:
(!) Work in Progress.
h3. Basic Idea:
Introduce support for Hudi writer code to produce storage format for the last
2-3 table versions. This enables older readers to continue reading the table
even when writers are upgraded, as long as writer produces storage compatible
with the latest table version the reader can read. The readers can then be
rolling upgraded easily, at different cadence without need for any tight
co-ordination. Additionally, the reader should have ability to "dynamically"
deduce table version based on table properties, such that when the writer is
switched to
h3.
High level approach:
> RFC for backwards compatible writer mode in Hudi 1.0
> ----------------------------------------------------
>
> Key: HUDI-8076
> URL: https://issues.apache.org/jira/browse/HUDI-8076
> Project: Apache Hudi
> Issue Type: New Feature
> Reporter: Ethan Guo
> Assignee: Vinoth Chandar
> Priority: Major
> Fix For: 1.0.0
>
>
> (!) Work in Progress.
> h3. Basic Idea:
> Introduce support for Hudi writer code to produce storage format for the last
> 2-3 table versions. This enables older readers to continue reading the table
> even when writers are upgraded, as long as writer produces storage compatible
> with the latest table version the reader can read. The readers can then be
> rolling upgraded easily, at different cadence without need for any tight
> co-ordination. Additionally, the reader should have ability to "dynamically"
> deduce table version based on table properties, such that when the writer is
> switched to the latest table version, subsequent reads will just adapt and
> read it as the latest table version.
> Operators still need to ensure all readers have the latest binary that
> supports a given table version, before switching the writer to that version.
> Special consideration to table services, as reader/writer processes, that
> should be able manage the tables as well. Queries should gracefully fail
> during table version switches and start eventually succeeding when writer
> completes switching.
> h3. High level approach:
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
