Siyao Meng created HDDS-17348:
---------------------------------
Summary: Intermittent failure in kubernetes check after restarting
datanodes due to stale pipeline
Key: HDDS-17348
URL: https://issues.apache.org/jira/browse/HDDS-17348
Project: Apache Ozone
Issue Type: Sub-task
Components: kubernetes, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
Ozone Client Key Generator | FAIL |
255 != 0
Running command 'ozone freon ockg -t 1 -n1 -p ockgpost 2>&1'.
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Unable to
allocate a container to the block of size: 268435456, replicationConfig:
RATIS/THREE. Waiting for one of pipelines to be OPEN failed. The input pipeline
IDs 13a84c89-fe5b-413a-85af-37b3bfec4a11 cannot be found
{code}
The following {{Ozone Client Key Validator}} with prefix {{ockgpost}} then
fails with {{Key:ockgpost/0 not found}}.
h3. Root cause
{{kubernetes/examples/ozone/test.sh}} deletes the datanode pods, then waits
with {{wait_for_startup}} and {{wait_for_pipeline}} (added in HDDS-14635). SCM
is not restarted, so both checks can pass right away: {{wait_for_pipeline}}
only requires any OPEN RATIS/THREE pipeline, and the pre restart pipeline is
still OPEN in SCM.
The restarted pods get new IPs. When each datanode registers again,
{{NodeAddressUpdateHandler}} closes every pipeline that still has the old
address of that datanode. In this run datanode-0 and datanode-1 registered 3
seconds apart, so the old pipeline was closed, a new one was allocated with the
old address of datanode-1 and datanode-2, and then closed and removed too. The
post restart freon write landed in that window and failed.
h3. Fix
Make {{assert_pipeline_exists}} count only OPEN RATIS/THREE pipelines whose
datanodes all have their current pod IP. Such a pipeline can only become OPEN
after all its datanodes registered again, so it is not closed later.
- https://github.com/apache/ozone/actions/runs/39834857275/job/120027666568
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
