[
https://issues.apache.org/jira/browse/HDDS-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153798#comment-17153798
]
Stephen O'Donnell commented on HDDS-1930:
-----------------------------------------
On a 8 datanode cluster (not ideal for Ozone as it should be a multiple of 3
DNs), I ran teragen and then terasort:
{code}
ozone sh volume create o3://ozone1/teragen
ozone sh bucket create o3://ozone1/teragen/bucket
yarn jar
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar
teragen -Dmapreduce.job.maps=8 1000000000 o3fs://bucket.teragen.ozone1/test3
yarn jar
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar
terasort -Dmapreduce.job.maps=30 o3fs://bucket.teragen.ozone1/test3
o3fs://bucket.teragen.ozone1/sort-test3
{code}
The job counts at the end said:
{code}
Killed map tasks=244
Launched map tasks=620
Launched reduce tasks=1
Other local map tasks=244
Data-local map tasks=321
Rack-local map tasks=55
{code}
So it executed 620 -244 = 376 containers successfully, where 321 were
"Data-Local" and the remaining 55 were "Rack-Local". I would like to do a few
more tests to confirm, but from that, it does seem like the locality is making
it through to YARN and it is picking local nodes for the data or at least the
same rack.
> Test Topology Aware Job scheduling with Ozone Topology
> ------------------------------------------------------
>
> Key: HDDS-1930
> URL: https://issues.apache.org/jira/browse/HDDS-1930
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Reporter: Xiaoyu Yao
> Assignee: Stephen O'Donnell
> Priority: Major
> Fix For: 0.6.0
>
>
> My initial results with Terasort does not seem to report the counter
> properly. Most of the requests are handled by rack local but no node local.
> This ticket is opened to add more system testing to validate the feature.
> Total Allocated Containers: 3778
> Each table cell represents the number of NodeLocal/RackLocal/OffSwitch
> containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests.
> Node Local Request Rack Local Request Off Switch Request
> Num Node Local Containers (satisfied by) 0
> Num Rack Local Containers (satisfied by) 0 3648
> Num Off Switch Containers (satisfied by) 0 96 34
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
