[
https://issues.apache.org/jira/browse/HDDS-17348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17348:
----------------------------------
Labels: pull-request-available (was: )
> Intermittent failure in kubernetes check after restarting datanodes due to
> stale pipeline
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-17348
> URL: https://issues.apache.org/jira/browse/HDDS-17348
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: kubernetes, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> Ozone Client Key Generator | FAIL |
> 255 != 0
> Running command 'ozone freon ockg -t 1 -n1 -p ockgpost 2>&1'.
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Unable to
> allocate a container to the block of size: 268435456, replicationConfig:
> RATIS/THREE. Waiting for one of pipelines to be OPEN failed. The input
> pipeline IDs 13a84c89-fe5b-413a-85af-37b3bfec4a11 cannot be found
> {code}
> The following {{Ozone Client Key Validator}} with prefix {{ockgpost}} then
> fails with {{Key:ockgpost/0 not found}}.
> h3. Root cause
> {{kubernetes/examples/ozone/test.sh}} deletes the datanode pods, then waits
> with {{wait_for_startup}} and {{wait_for_pipeline}} (added in HDDS-14635).
> SCM is not restarted, so both checks can pass right away:
> {{wait_for_pipeline}} only requires any OPEN RATIS/THREE pipeline, and the
> pre restart pipeline is still OPEN in SCM.
> The restarted pods get new IPs. When each datanode registers again,
> {{NodeAddressUpdateHandler}} closes every pipeline that still has the old
> address of that datanode. In this run datanode-0 and datanode-1 registered 3
> seconds apart, so the old pipeline was closed, a new one was allocated with
> the old address of datanode-1 and datanode-2, and then closed and removed
> too. The post restart freon write landed in that window and failed.
> h3. Fix
> Make {{assert_pipeline_exists}} count only OPEN RATIS/THREE pipelines whose
> datanodes all have their current pod IP. Such a pipeline can only become OPEN
> after all its datanodes registered again, so it is not closed later.
> - https://github.com/apache/ozone/actions/runs/39834857275/job/120027666568
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
