[
https://issues.apache.org/jira/browse/HDDS-14337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saketa Chalamchala updated HDDS-14337:
--------------------------------------
Description:
SCM fails to restart after it is first deployed and started near the end of the
year, say on Dec 31, with the following error due to a Precondition fail that
check if
{code:java}
localId > UniqueId.next() {code}
- `localId`: Uses `LocalDate.of(LocalDate.now().getYear() + 1, 1, 1) <<
Short.SIZE` (shift by 16 bits)
- `UniqueId.next()`: Uses `System.currentTimeMillis() << Short.SIZE` (shift by
16 bits)
Near a year boundary, these two time bases diverge. For example, on Dec 31
evening in US timezones, `localId` may return `2026 Jan 1` while
\{{currentTimeMillis}} is already in the next UTC year `2026 Jan 1 + 7h`. In
that case the seeded `localId` (Jan 1 of next local year) can be less than
`UniqueId.next()`, and the precondition fails.
{code:java}
SCM start failed with exception
java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:129)
at
org.apache.hadoop.hdds.scm.ha.SequenceIdGenerator.upgradeToSequenceId(SequenceIdGenerator.java:363)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:698)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:413)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:616)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:628)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:171)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:145)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:74)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:48)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:103)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:94)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.main(StorageContainerManagerStarter.java:63)
{code}
was:
SCM fails to restart after it is first deployed and started near the end of the
year, say on Dec 31, with the following error due to a Precondition fail that
check if {code:java} localId > UniqueId.next() {code}
- `localId`: Uses millisSinceEpoch << Short.SIZE (shift by 16 bits)
- `UniqueId.next()`: Uses currentTimeMillis << 10 (shift by 10 bits)
Since UniqueId.next() shifts by fewer bits and includes the current time, while
localId shifts by more bits but uses a future date (Jan 1st of next year), the
values can be very close or even inverted, especially near year boundaries.
{code:java}
SCM start failed with exception
java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:129)
at
org.apache.hadoop.hdds.scm.ha.SequenceIdGenerator.upgradeToSequenceId(SequenceIdGenerator.java:363)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:698)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:413)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:616)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:628)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:171)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:145)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:74)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:48)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:103)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:94)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.main(StorageContainerManagerStarter.java:63)
{code}
> SCM does not restart if it was first deployed near the end of year
> ------------------------------------------------------------------
>
> Key: HDDS-14337
> URL: https://issues.apache.org/jira/browse/HDDS-14337
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Saketa Chalamchala
> Assignee: Saketa Chalamchala
> Priority: Major
>
> SCM fails to restart after it is first deployed and started near the end of
> the year, say on Dec 31, with the following error due to a Precondition fail
> that check if
> {code:java}
> localId > UniqueId.next() {code}
> - `localId`: Uses `LocalDate.of(LocalDate.now().getYear() + 1, 1, 1) <<
> Short.SIZE` (shift by 16 bits)
> - `UniqueId.next()`: Uses `System.currentTimeMillis() << Short.SIZE` (shift
> by 16 bits)
> Near a year boundary, these two time bases diverge. For example, on Dec 31
> evening in US timezones, `localId` may return `2026 Jan 1` while
> \{{currentTimeMillis}} is already in the next UTC year `2026 Jan 1 + 7h`. In
> that case the seeded `localId` (Jan 1 of next local year) can be less than
> `UniqueId.next()`, and the precondition fails.
> {code:java}
> SCM start failed with exception
> java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:129)
> at
> org.apache.hadoop.hdds.scm.ha.SequenceIdGenerator.upgradeToSequenceId(SequenceIdGenerator.java:363)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:698)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:413)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:616)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:628)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:171)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:145)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:74)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:48)
> at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
> at picocli.CommandLine.access$1300(CommandLine.java:145)
> at
> picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
> at
> picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
> at picocli.CommandLine.execute(CommandLine.java:2078)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:103)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:94)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.main(StorageContainerManagerStarter.java:63)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
