errose28 commented on PR #11861:
URL: https://github.com/apache/ozone/pull/11861#issuecomment-6673339171
I was discussing the write path changes in this PR with @sodonnel and he
suggested trying to push the unregistered node/pipeline filter down into the
`PipelineManager` instead of the container providers. By implementing a new
method in `PipelineManager` that only returns pipelines with all registered
nodes we get:
- No changes to writable container providers:
- They just need to switch to calling the filtering methods
in`PipelineManager`
- Pipelines with unregistered nodes count towards the pipeline limit
- This is required to prevent going over the limit, since the scrubber
does not delete excess pipelines
- Minimum pipelines required to exit safemode must be enough to sustain
the cluster until the scrubber kicks in a few minutes later. This is expected.
- Background pipeline creator will not replace the unusable pipelines
containing unregistered nodes until the scrubber starts.
- If we lose any of the valid pipelines after safemode exit, the
background pipeline creator will have already started running after safemode
precheck completes, so they will be replaced
- Datanodes have the interval between safemode exit and scrubber start to
finish registering, otherwise their pipelines will be closed by the scrubber
I've implemented this in #11956 and it looks easier for the write path.
Maybe we should split this task so this PR only handles the read path and
#11956 only handles the write path. There will be a third PR for an EC pipeline
safemode rule required as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]