[
https://issues.apache.org/jira/browse/HDFS-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041564#comment-17041564
]
zhuqi commented on HDFS-15171:
------------------------------
Hi [~sodonnell]
Thanks for your patient reply.
First, the every 10 minutes thread in CachingGetSpaceUsed, now with a random
jitter time to random the refresh operation, and if we can persist the value to
the cache file when the value refresh, this is the most real time cache.
Second, when the value refresh, we can compare it with last one, if they are
same, we can jump the persist operation to reduce the disk operation.
In order to reduce the disk operation, we can add a fixed time interval which
can be configurated, when the real time fresh time exceed the fixed time
interval , then to persist the value to disk.
Then we can remove the shutdown hook persist operation and don't need to
caculate what dfs.datanode.cached-dfsused.check.interval.ms is suitable
anymore.
And also can reslove my problem, which caused by the datanode shutdown
ungracefully.
What do you think about my advice?
> Add a thread to call saveDfsUsed periodically, to prevent datanode too long
> restart time.
> -------------------------------------------------------------------------------------------
>
> Key: HDFS-15171
> URL: https://issues.apache.org/jira/browse/HDFS-15171
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 3.2.0
> Reporter: zhuqi
> Assignee: zhuqi
> Priority: Major
>
> There are 30 storage dirs per datanode in our production cluster , it will
> take too many time to restart, because sometimes the datanode didn't shutdown
> gracefully. Now only the datanode graceful shut down hook and the
> blockpoolslice shutdown will cause the saveDfsUsed function, that cause the
> restart of datanode can't reuse the dfsuse cache sometimes. I think if we can
> add a thread to periodically call the saveDfsUsed function.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
