[
https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Radia updated HADOOP-2110:
---------------------------------
Attachment: brAsLongsPatch2.txt
The attached patch uses an array of longs instead of an array of Class Block
for the Block report (BR)
A block is 2 longs (block id and block length).
Java's overhead for an object is 16 bytes.
For an array of objects each array entry is a separately allocated object.
Comparison of the block report processing times.
All times are in millisec and are measured from the data node sending the BR
and
receiving an acknowledgement from the NN that the BR was processed.
Approx 50 improvement in the BR processing time
BRs using Class Block || BRs using Longs
Num Time Time || Time Time
Blocks Total per 1K blocks || Total per 1K blocks
---------------------------------------------||-----------------------
40000 651ms 16ms || 230ms 5.75ms
100000 1258ms 12ms || 573ms 5.73ms
300000 3521ms 12ms || 1800ms 6.00ms
500000 5930ms 12ms || 2916ms 5.83ms
> Block Report Optimization: Replace class instance
> -------------------------------------------------
>
> Key: HADOOP-2110
> URL: https://issues.apache.org/jira/browse/HADOOP-2110
> Project: Hadoop
> Issue Type: Sub-task
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Robert Chansler
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: brAsLongsPatch2.txt
>
>
> Block report processing can be more efficient if instances of the block ID
> class are replaced with a sequence of longs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
