[jira] [Updated] (HDFS-16058) If the balancer cannot update the datanode's storage type in time, the balancer will throw NullPointerException

[lei w (Jira)]

     [ 
https://issues.apache.org/jira/browse/HDFS-16058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lei w updated HDFS-16058:
-------------------------
    Attachment: HDFS-16058.patch

> If the balancer cannot update the datanode's storage type in time, the 
> balancer will throw NullPointerException
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-16058
>                 URL: https://issues.apache.org/jira/browse/HDFS-16058
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer & mover
>            Reporter: lei w
>            Priority: Critical
>         Attachments: HDFS-16058.patch
>
>
> The main logic of the balancer is to initialize the cluster information 
> (number of datanodes, datanode's storage type and network topology 
> etc)、calculate the source DataNode and target DataNode that need to move the 
> block and then obtain the movable block through the getblocks(). Finally , 
> move blocks. If a DataNode adds another type of storage and writes an EC 
> block after the cluster information is initialized, and then returns the 
> newly written EC block to balancer by getblocks(), it will cause balancer to 
> throw NullPointerException .The main reason is that the balancer does not 
> record the newly added storage type of the DataNode, which causes the current 
> DataNode information not to be recorded when generate an object of type 
> DBlockStriped . Finally, null is returned when the block of the current 
> DataNode is obtained through the object of type DBlockStriped.
> {code:java}
> 2021-04-23 19:38:21,233 WARN 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: Dispatcher thread failed
> java.lang.NullPointerException
>         at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.chooseProxySource(Dispatcher.java:325)
>         at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.markMovedIfGoodBlock(Dispatcher.java:291)
>         at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.chooseBlockAndProxy(Dispatcher.java:271)
>         at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2500(Dispatcher.java:235)
>         at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.chooseNextMove(Dispatcher.java:886)
>         at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.dispatchBlocks(Dispatcher.java:943)
>         at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.access$3200(Dispatcher.java:751)
>         at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$2.run(Dispatcher.java:1221)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org