[
https://issues.apache.org/jira/browse/HDDS-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDDS-4315:
-
Description:
In a non-Ratis OM, the transaction index used to generate ObjectID is reset on
OM restart. This can lead to duplicate ObjectIDs when the OM is restarted.
ObjectIDs should be unique.
HDDS-2939 and NFS are some of the features which depend on ObjectIds being
unique.
To ensure that objectIDs are unique across restarts in non-ratis OM cluster,
the transaction index should be updated in DB on every flush to DB. This can be
done in a similar fashion to what is being done for ratis enabled cluster
today. TransactionInfo table is updated with transaction index as part of every
batch write operation to DB.
Also, and epoch number is introduced to ensure that objectIDs do not clash with
older clusters in which this fix does not exist. From the 64 bits of ObjectID
(long variable), 2 bits are reserved for epoch and 8 bits for recursive
directory creation, if required. The most significant 2 bits of objectIDs is
set to epoch. For clusters before HDDS-4315 there is no epoch as such. But it
can be safely assumed that the most significant 2 bits of the objectID will be
00 (as it unlikely to reach trxn index > 2^62 in an existing cluster). From
HDDS-4315 onwards, the Epoch for non-ratis OM clusters will be binary 01 (=
decimal 1) and for ratis enabled OM cluster will be binary 10 (= decimal 2).
was:
In a non-Ratis OM, the transaction index used to generate ObjectID is reset on
OM restart. This can lead to duplicate ObjectIDs when the OM is restarted.
ObjectIDs should be unique.
For HDDS-2939 and NFS are some of the features which depend on ObjectIds being
unique.
This Jira aims to introduce an epoch number in OM which is incremented on OM
restarts. The epoch is persisted on disk. This epoch will be used to set the
first 16 bits of the objectID to ensure that objectIDs are unique even after OM
restart.
The highest epoch number is reserved for transactions coming through ratis.
This will take care of the scenario where OM ratis is enabled on an existing
cluster.
> Ensure ObjectIDs are unique across restarts
> ---
>
> Key: HDDS-4315
> URL: https://issues.apache.org/jira/browse/HDDS-4315
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
>
> In a non-Ratis OM, the transaction index used to generate ObjectID is reset
> on OM restart. This can lead to duplicate ObjectIDs when the OM is restarted.
> ObjectIDs should be unique.
> HDDS-2939 and NFS are some of the features which depend on ObjectIds being
> unique.
> To ensure that objectIDs are unique across restarts in non-ratis OM cluster,
> the transaction index should be updated in DB on every flush to DB. This can
> be done in a similar fashion to what is being done for ratis enabled cluster
> today. TransactionInfo table is updated with transaction index as part of
> every batch write operation to DB.
> Also, and epoch number is introduced to ensure that objectIDs do not clash
> with older clusters in which this fix does not exist. From the 64 bits of
> ObjectID (long variable), 2 bits are reserved for epoch and 8 bits for
> recursive directory creation, if required. The most significant 2 bits of
> objectIDs is set to epoch. For clusters before HDDS-4315 there is no epoch as
> such. But it can be safely assumed that the most significant 2 bits of the
> objectID will be 00 (as it unlikely to reach trxn index > 2^62 in an existing
> cluster). From HDDS-4315 onwards, the Epoch for non-ratis OM clusters will be
> binary 01 (= decimal 1) and for ratis enabled OM cluster will be binary 10 (=
> decimal 2).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
