[
https://issues.apache.org/jira/browse/HDDS-17038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksei Ieshin updated HDDS-17038:
----------------------------------
Description:
After an OM installs a Ratis snapshot (for example, a lagging follower catching
up), all {{OzoneManagerDoubleBufferMetrics}}
({{{}ozone_manager_double_buffer_metrics_*{}}} on {{{}/prom{}}},
{{OzoneManagerDoubleBufferMetrics}} on {{{}/jmx{}}}) stop being published. They
stay missing until the OM JVM restarts. The double buffer itself keeps
flushing; only the metrics source is gone.
Cause:
# {{{}OzoneManagerStateMachine.pause(){}}}, used by snapshot install, calls
{{{}ozoneManagerDoubleBuffer.stop(){}}}. {{OzoneManagerDoubleBuffer.stop()}}
calls {{{}metrics.unRegister(){}}}, which runs
{{{}DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME){}}}.
# {{OzoneManagerStateMachine.unpause()}} then creates a new double buffer via
{{{}buildDoubleBufferForRatis(){}}}. Its field initializer calls
{{{}OzoneManagerDoubleBufferMetrics.create(){}}}.
# {{create()}} is a lazy singleton:
!http://localhost:63342/markdownPreview/651969378//Users/aieshin/git/ozone!
{code:java}
public static synchronized OzoneManagerDoubleBufferMetrics create() {
if (instance != null) {
return instance; // still the old object, whose source was
unregistered in step 1
} ... {code}
{{`unRegister()`}} never resets {{{}instance{}}}, so the new double buffer
updates an object that no longer belongs to the metrics system, and nothing is
exported.
Observed in production on 2.0.0. The same code is in 2.2.1 and master. On a
3-node OM HA cluster, two followers lost these metrics after snapshot installs
(one during a RATIS-2240 snapshot-install deadlock, one during a period of
heavy load). Both got them back only after their pods restarted. The leader,
which never installs a snapshot, was unaffected. This silently blinds
dashboards and alerts built on double-buffer flush time.
*Proposed fix:* make {{unRegister()}} reset the singleton, so the next
{{create()}} registers a fresh source:
{code:java}
public static synchronized void unRegister() { // or keep it an instance
method, but clear the static
DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME);
instance = null;
} {code}
Alternatively, {{create()}} could re-register when {{instance != null}} but the
source is missing from the metrics system.
*Test:* in {{{}TestOzoneManagerStateMachine{}}}, or via
{{{}MiniOzoneHAClusterImpl{}}}, call {{pause()}} and then {{{}unpause(){}}}.
Assert that
{{DefaultMetricsSystem.instance().getSource("OzoneManagerDoubleBufferMetrics")}}
is non-null, and that {{getTotalNumOfFlushOperations()}} increases after a
further write. Possibly related: HDDS-16908 (intermittent failure in
{{TestOzoneManagerDoubleBufferWithDummyResponse}} metrics assertions), since
tests share the static singleton.
was:
After an OM installs a Ratis snapshot (for example, a lagging follower catching
up), all {{OzoneManagerDoubleBufferMetrics}}
({{{}ozone_manager_double_buffer_metrics_*{}}} on {{{}/prom{}}},
{{OzoneManagerDoubleBufferMetrics}} on {{{}/jmx{}}}) stop being published. They
stay missing until the OM JVM restarts. The double buffer itself keeps
flushing; only the metrics source is gone.
Cause:
# {{{}OzoneManagerStateMachine.pause(){}}}, used by snapshot install, calls
{{{}ozoneManagerDoubleBuffer.stop(){}}}. {{OzoneManagerDoubleBuffer.stop()}}
calls {{{}metrics.unRegister(){}}}, which runs
{{{}DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME){}}}.
# {{OzoneManagerStateMachine.unpause()}} then creates a new double buffer via
{{{}buildDoubleBufferForRatis(){}}}. Its field initializer calls
{{{}OzoneManagerDoubleBufferMetrics.create(){}}}.
# {{create()}} is a lazy singleton:
!http://localhost:63342/markdownPreview/651969378//Users/aieshin/git/ozone!
{{public static synchronized OzoneManagerDoubleBufferMetrics create() \{
if (instance != null) {
return instance; // still the old object, whose source was
unregistered in step 1
} ...}}
{{unRegister()}} never resets {{{}instance{}}}, so the new double buffer
updates an object that no longer belongs to the metrics system, and nothing is
exported.
Observed in production on 2.0.0. The same code is in 2.2.1 and master. On a
3-node OM HA cluster, two followers lost these metrics after snapshot installs
(one during a RATIS-2240 snapshot-install deadlock, one during a period of
heavy load). Both got them back only after their pods restarted. The leader,
which never installs a snapshot, was unaffected. This silently blinds
dashboards and alerts built on double-buffer flush time.
*Proposed fix:* make {{unRegister()}} reset the singleton, so the next
{{create()}} registers a fresh source:
!http://localhost:63342/markdownPreview/651969378//Users/aieshin/git/ozone!
{{public static synchronized void unRegister() \{ // or keep it an instance
method, but clear the static
DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME);
instance = null;
}}}
Alternatively, {{create()}} could re-register when {{instance != null}} but the
source is missing from the metrics system.
*Test:* in {{{}TestOzoneManagerStateMachine{}}}, or via
{{{}MiniOzoneHAClusterImpl{}}}, call {{pause()}} and then {{{}unpause(){}}}.
Assert that
{{DefaultMetricsSystem.instance().getSource("OzoneManagerDoubleBufferMetrics")}}
is non-null, and that {{getTotalNumOfFlushOperations()}} increases after a
further write. Possibly related: HDDS-16908 (intermittent failure in
{{TestOzoneManagerDoubleBufferWithDummyResponse}} metrics assertions), since
tests share the static singleton.
> OzoneManagerDoubleBuffer metrics disappear permanently after a follower
> installs a Ratis snapshot
> -------------------------------------------------------------------------------------------------
>
> Key: HDDS-17038
> URL: https://issues.apache.org/jira/browse/HDDS-17038
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM HA
> Affects Versions: 2.0.0, 2.2.0, 2.2.1
> Reporter: Aleksei Ieshin
> Priority: Minor
>
> After an OM installs a Ratis snapshot (for example, a lagging follower
> catching up), all {{OzoneManagerDoubleBufferMetrics}}
> ({{{}ozone_manager_double_buffer_metrics_*{}}} on {{{}/prom{}}},
> {{OzoneManagerDoubleBufferMetrics}} on {{{}/jmx{}}}) stop being published.
> They stay missing until the OM JVM restarts. The double buffer itself keeps
> flushing; only the metrics source is gone.
> Cause:
> # {{{}OzoneManagerStateMachine.pause(){}}}, used by snapshot install, calls
> {{{}ozoneManagerDoubleBuffer.stop(){}}}. {{OzoneManagerDoubleBuffer.stop()}}
> calls {{{}metrics.unRegister(){}}}, which runs
> {{{}DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME){}}}.
> # {{OzoneManagerStateMachine.unpause()}} then creates a new double buffer
> via {{{}buildDoubleBufferForRatis(){}}}. Its field initializer calls
> {{{}OzoneManagerDoubleBufferMetrics.create(){}}}.
> # {{create()}} is a lazy singleton:
>
> !http://localhost:63342/markdownPreview/651969378//Users/aieshin/git/ozone!
> {code:java}
> public static synchronized OzoneManagerDoubleBufferMetrics create() {
> if (instance != null) {
> return instance; // still the old object, whose source was
> unregistered in step 1
> } ... {code}
> {{`unRegister()`}} never resets {{{}instance{}}}, so the new double buffer
> updates an object that no longer belongs to the metrics system, and nothing
> is exported.
> Observed in production on 2.0.0. The same code is in 2.2.1 and master. On a
> 3-node OM HA cluster, two followers lost these metrics after snapshot
> installs (one during a RATIS-2240 snapshot-install deadlock, one during a
> period of heavy load). Both got them back only after their pods restarted.
> The leader, which never installs a snapshot, was unaffected. This silently
> blinds dashboards and alerts built on double-buffer flush time.
> *Proposed fix:* make {{unRegister()}} reset the singleton, so the next
> {{create()}} registers a fresh source:
>
> {code:java}
> public static synchronized void unRegister() { // or keep it an instance
> method, but clear the static
> DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME);
> instance = null;
> } {code}
> Alternatively, {{create()}} could re-register when {{instance != null}} but
> the source is missing from the metrics system.
> *Test:* in {{{}TestOzoneManagerStateMachine{}}}, or via
> {{{}MiniOzoneHAClusterImpl{}}}, call {{pause()}} and then {{{}unpause(){}}}.
> Assert that
> {{DefaultMetricsSystem.instance().getSource("OzoneManagerDoubleBufferMetrics")}}
> is non-null, and that {{getTotalNumOfFlushOperations()}} increases after a
> further write. Possibly related: HDDS-16908 (intermittent failure in
> {{TestOzoneManagerDoubleBufferWithDummyResponse}} metrics assertions), since
> tests share the static singleton.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
