[
https://issues.apache.org/jira/browse/HUDI-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17439599#comment-17439599
]
Vinoth Chandar commented on HUDI-2475:
--------------------------------------
I agree with your assessment above that for multi-writer deployments or
async/separate deployments of cleaning/compaction/clustering.
Writing this up per deployment type for clarity
{quote}{color:#FF0000}Still WIP {color}
{quote}
h2. Deltastreamer continuous mode & Spark Streaming with in-writer process
async table services, with no other writers
No action needed, rolling upgrades work just fine.
(I looked at the code to ensure the writer's write client gets created first,
which will perform upgrade (delete existing metadata table, then rebuild it),
before any async scheduling/execution happens)
h2. Multiple writers (or) Single writers with table services running
out-of-writer process asynchronously
*1) Stop all writers and table services*
- If metadata enabled,
- if metadata enabled later.
*2) Bring up writers serially, first writer will upgrade to 0.10*
*3) Deploy 0.10 to table services and redeploy*
*4) Do not enable metadata on query side until 1-3 are successfully completed*
As usual, it's recommended that readers are upgraded to 0.10 bundles prior to
1-3
- Need to check what errors are thrown if not across engines.
> Upgrade downgrade infra for enabling metadata
> ---------------------------------------------
>
> Key: HUDI-2475
> URL: https://issues.apache.org/jira/browse/HUDI-2475
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: sivabalan narayanan
> Assignee: Vinoth Chandar
> Priority: Blocker
> Fix For: 0.10.0
>
>
> Upgrade downgrade infra for enabling metadata.
>
> If user is having a writer process and clustering/compaction running async.
>
> - New synchronous metadata design, has a constraint that once metadata table
> is bootstrapped, all commits will happen synchronously. In other words, there
> is no catch up business wrt datatable.
> So, it may not be feasible to do rolling upgrade (i.e. upgrade writer first
> while async compaction is running) and then upgrade async compaction.
> Bootstrap has to be done by stopping all processes and then we can restart
> all other processes one by one (by using the upgraded hudi library) w/
> metadata enabled.
> This is the only viable option I can think of.
> 1. Stop all processes. Upgrade to hudi to a version w/ synchronous metadata.
> bring up one writer process w/ metadata config enabled. this will bootstrap
> the metadata table. and from there on, any new commits by the writer will do
> synchronous updates to metadata.
> Note: users can choose to upgrade via hudi-cli if need be. but easier would
> be to just start the writer. Expect some delay for first commit since
> bootstrap will be happening.
> 2. Once first commit in previous writer process completes successfully, we
> can restart all other processes. Upgrade the async table service (to hudi
> version w/ metadata enabled) and restart it. *Ensure metadata table is
> enabled across all processes.* Even if missed on one, could result in data
> loss.
>
> By this, once metadata table is bootstrapped, any new commits from all
> processes will be synced to metadata.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
