[
https://issues.apache.org/jira/browse/HDDS-1785?focusedWorklogId=275849&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-275849
]
ASF GitHub Bot logged work on HDDS-1785:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Jul/19 11:59
Start Date: 12/Jul/19 11:59
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1085: HDDS-1785.
OOM error in Freon due to the concurrency handling
URL: https://github.com/apache/hadoop/pull/1085
## What changes were proposed in this pull request?
Change concurrency in Freon `RandomKeyGenerator`:
* create a worker for each thread
* let each worker create volumes, buckets and keys, without limiting
"inner" objects to specific "outer" ones (eg. create key for any bucket)
Workers coordinate the items they create using "global" counters.
https://issues.apache.org/jira/browse/HDDS-1785
## How was this patch tested?
Tested with various number of volumes/buckets/threads.
```
$ ozone freon rk --numOfVolumes 1 --numOfBuckets 100 --numOfKeys 5
--numOfThreads 1 --replicationType=RATIS --factor=THREE
...
Number of Volumes created: 1
Number of Buckets created: 100
Number of Keys added: 500
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,100
Average Time spent in bucket creation: 00:00:00,304
Average Time spent in key creation: 00:00:01,556
Average Time spent in key write: 00:00:53,509
Total bytes written: 5120000
Total Execution time: 00:01:01,537
```
```
$ ozone freon rk --numOfVolumes 1 --numOfBuckets 100 --numOfKeys 5
--numOfThreads 50 --replicationType=RATIS --factor=THREE
...
Number of Volumes created: 1
Number of Buckets created: 100
Number of Keys added: 500
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,003
Average Time spent in bucket creation: 00:00:00,229
Average Time spent in key creation: 00:00:00,273
Average Time spent in key write: 00:00:10,375
Total bytes written: 5120000
Total Execution time: 00:00:16,872
```
```
$ ozone freon rk --numOfVolumes 10 --numOfBuckets 10 --numOfKeys 500
--numOfThreads 50 --replicationType=RATIS --factor=THREE
...
Number of Volumes created: 10
Number of Buckets created: 100
Number of Keys added: 50000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,052
Average Time spent in bucket creation: 00:00:00,240
Average Time spent in key creation: 00:00:30,742
Average Time spent in key write: 00:10:04,146
Total bytes written: 512000000
Total Execution time: 00:10:42,463
```
```
$ ozone freon rk --numOfVolumes 100 --numOfBuckets 100 --numOfKeys 2
--numOfThreads 50 --replicationType=RATIS --factor=THREE
...
Number of Volumes created: 100
Number of Buckets created: 10000
Number of Keys added: 20000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,266
Average Time spent in bucket creation: 00:00:06,388
Average Time spent in key creation: 00:00:09,324
Average Time spent in key write: 00:03:44,925
Total bytes written: 204800000
Total Execution time: 00:04:11,735
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 275849)
Time Spent: 10m
Remaining Estimate: 0h
> OOM error in Freon due to the concurrency handling
> --------------------------------------------------
>
> Key: HDDS-1785
> URL: https://issues.apache.org/jira/browse/HDDS-1785
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Elek, Marton
> Assignee: Doroszlai, Attila
> Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HDDS-1532 modified the concurrent framework usage of Freon
> (RandomKeyGenerator).
> The new approach uses separated tasks (Runnable) to create the
> volumes/buckets/keys.
> Unfortunately it doesn't work very well in some cases.
> # When Freon starts it creates an executor with fixed number of threads (10)
> # The first loop submits numOfVolumes (10) VolumeProcessor tasks to the
> executor
> # The 10 threads starts to execute the 10 VolumeProcessor tasks
> # Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor
> tasks. All together 10000 tasks are submitted to the executor.
> # The 10 threads starts to execute the first 10 BucketProcessor tasks, they
> starts to create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
> # At this point of the time no keys are generated, but the next 10
> BucketProcessor tasks are started to execute..
> # To execute the first key creation we should process all the
> BucketProcessor tasks which means that all the Key creation tasks (10 * 1000
> * 500 000) are created and added to the executor
> # Which requires a huge amount of time and memory
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
