[ https://issues.apache.org/jira/browse/HDDS-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170623#comment-17170623 ]
Zheng Huang-Mu commented on HDDS-3724: -------------------------------------- It's seems to be fixed by [~xyao] in HDDS-4027? > BackgroundPipelineCreator should not throw Pipeline creation failed exception > due to insufficient nodes > ------------------------------------------------------------------------------------------------------- > > Key: HDDS-3724 > URL: https://issues.apache.org/jira/browse/HDDS-3724 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Reporter: Hanisha Koneru > Priority: Major > > BackgroundPipelineCreator runs a scheduler to keep creating pipelines if > possible. But in a healthy cluster where all the datanodes are part of > pipelines and there are no more nodes available for creating new pipelines, > the BackgroundPipelineCreator should not log an error message. In this > scenario, it is not an error that a pipeline couldn't be created. > SCM logs were flooded with "Failed to create pipeline" error messages because > of this: > {code:java} > 2020-06-04 11:03:24,229 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed > due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of > factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1 > 2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] WARN > org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline > creation failed due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed > due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:05:40,705 [ReplicationMonitor] INFO > org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor > Thread took 0 milliseconds for processing 7 containers. > 2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of > factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1 > 2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] WARN > org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline > creation failed due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed > due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of > factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1 > 2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] WARN > org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline > creation failed due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed > due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:10:40,700 [ReplicationMonitor] INFO > org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor > Thread took 0 milliseconds for processing 7 containers. > 2020-06-04 11:11:24,234 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of > factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1 > 2020-06-04 11:11:24,235 [RatisPipelineUtilsThread] WARN > org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline > creation failed due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:11:24,235 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed > due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:13:24,240 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of > factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1 > 2020-06-04 11:13:24,240 [RatisPipelineUtilsThread] WARN > org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline > creation failed due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:13:24,240 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed > due to no sufficient healthy datanodes. Required 3. Found 1. > 2020-06-04 11:15:24,242 [RatisPipelineUtilsThread] ERROR > org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create > pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of > factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org