[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988315#comment-14988315
]
Daryn Sharp edited comment on HDFS-9260 at 11/3/15 10:36 PM:
-------------------------------------------------------------
I've read the doc now. Sorry I commented before doing so. The results are
interesting until the the final details about a 4x reduction in block updates.
Here are some basic specs to consider:
* 10-80k adds/min
* job submissions increasing replication factor to 10
* at least 1 node/day decommissioning or going dead with 100k-400k blocks
* every few weeks entire racks (40 nodes) are decommissioned for refresh or
reallocation
* balancer is constantly churning to populate recommissioned dead nodes
That's a lot of IBRs which is why a 4x degradation is quite concerning. The
block report processing times seem a bit high in the tests. I'll attach an
image of the BR processing times for some of our busiest clusters. They span
the gamut from 100M-300M blocks with roughly the same number of files. We got
a huge improvement from my BR encoding change + per-storage reports.
BTW, I had/have a working patch that replaced the triplets with sparse yet
densely packed 2-dimensional primitive arrays. Everything is linked via
indices to a greatly reduce the dirty cards to scan. Need to dig up the jira
when my head is above water.
was (Author: daryn):
I've read the doc now. Sorry I commented before doing so. The results are
interesting until the the final details about a 4x reduction in block updates.
Here are some basic specs to consider:
* 10-80k adds/min
* job submissions increasing replication factor to 10
* at least 1 node/day decommissioning or going dead with 100k-400k blocks
* every few weeks entire racks (40 nodes) are decommissioned for refresh or
reallocation
* balancer is constantly churning to populate recommissioned dead nodes
That's a lot of IBRs which is why a 4x degradation is quite concerning. The
block report processing times seem a bit high in the tests. :) I'll attach an
image of the BR processing times for some of our busiest clusters. They span
the gamut from 100M-300M blocks with roughly the same number of files. We got
a huge improvement from my BR encoding change + per-storage reports.
BTW, I had/have a working patch that replaced the triplets with sparse yet
densely packed 2-dimensional primitive arrays. Everything is linked via
indices to a greatly reduce the dirty cards to scan. Need to dig up the jira
when my head is above water.
> Improve performance and GC friendliness of startup and FBRs
> -----------------------------------------------------------
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
> Affects Versions: 2.7.1
> Reporter: Staffan Friberg
> Assignee: Staffan Friberg
> Attachments: FBR processing.png, HDFS Block and Replica Management
> 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch,
> HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch,
> HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change and also some help
> investigating/understanding a few outstanding issues if we are interested in
> moving forward with this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
