[
https://issues.apache.org/jira/browse/HDDS-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814615#comment-16814615
]
Anu Engineer commented on HDDS-1266:
------------------------------------
Hi [~linyiqun],
Thank you for very careful reading. You are right on both counts. I will add
some more comments.
{quote}When do the mapping for HDFS blockId to Ozone blockId, there is one
precondition that HDFS blockId should be unique. The HDFS block id is unique in
a single block pool (namepsace). But with multiple block pool case, I suppose
we cannot promise this.
{quote}
You are right, if we have 2 different block pools we cannot guarantee block
uniqueness. One way to deal with that is a create a namespace like feature –
that is BlockPool1.blockid and BlockPool2.BlockID. Another way to deal with
this issue is to do the upgrades in serial fashion. That is upgrade
HDFS.BlockPool 1 to Ozone, then Upgrade HDFS.BlockPool2. Discussion about both
are missing in the current proposal, since the doc was already 20 pages long.
We will write up smaller more focused design docs to compliment this document,
so that we can dive deeper into issues like multiple block pools.
{quote}From the design doc, we cannot sync the metadata change between HDFS and
Ozone during upgrade time. That is to say that for one system metadata change
cannot reflect in another system. Based on this, seems we can't well supported
concurrently running HDFS and Ozone to provider the service, except one case
that we upgrade some specific HDFS folders that will only accessed by Ozone.
{quote}
This came up multiple times during the initial chats. [~ccondit] had suggested
that one way to solve that problem is to listen to the changes via Journal
Node. That way, we know what changes are happening on HDFS side and when we
verify the Ozone against HDFS after upgrade, we know what we should expect.
There is still a problem of Datanode trying to make a Hard link to an deleted
HDFS block, since during the process of the HDFS the blocks did exist. The
current plan is for the data node to report that information back, and we will
try to verify the information against HDFS at that point of time. The default
catch all, as you have seen from the design doc is the fall back into DistCp,
if the in-place upgrade detects any errors.
> [Ozone upgrade] Support Upgrading HDFS clusters to use Ozone
> ------------------------------------------------------------
>
> Key: HDDS-1266
> URL: https://issues.apache.org/jira/browse/HDDS-1266
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Reporter: Anu Engineer
> Assignee: Anu Engineer
> Priority: Major
> Attachments: InPlaceUpgradesForOzone.pdf
>
>
> This is the master JIRA to support upgrading existing HDFS clusters to have
> Ozone running concurrently. One of the requirements is that we support
> upgrading from HDFS to Ozone, without a full data copy. This requirement is
> called "In Place upgrade", the end result of such an upgrade would be to have
> the HDFS data appear in Ozone as if Ozone has taken a snap-shot of the HDFS
> data. Once upgrade is complete, Ozone and HDFS will act as independent
> systems. I will post a design document soon.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
