[
https://issues.apache.org/jira/browse/HDDS-2301?focusedWorklogId=336309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336309
]
ASF GitHub Bot logged work on HDDS-2301:
----------------------------------------
Author: ASF GitHub Bot
Created on: 30/Oct/19 17:16
Start Date: 30/Oct/19 17:16
Worklog Time Spent: 10m
Work Description: supratimdeka commented on pull request #107: HDDS-2301.
Write path: Reduce read contention in rocksDB.
URL: https://github.com/apache/hadoop-ozone/pull/107
https://issues.apache.org/jira/browse/HDDS-2301
This change introduces 'mkdir -p' behaviour in these 2 OM Requests:
1. create directory and
2. create (file)
The idea is to avoid iteration of the key table when creating new files with
a path. Without the patch, during file create - checking the existence of each
parent directory requires a slow DB iterator. With the patch, this check
becomes a point lookup which can be accelerated by the bloom filters in RocksDB.
This patch is a work-in-progress and the following functionality is broken:
1. interoperability with S3
2. ACLs (setting correct ACLs for the parent directories created in the path)
3. rename
Soliciting early feedback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 336309)
Remaining Estimate: 0h
Time Spent: 10m
> Write path: Reduce read contention in rocksDB
> ---------------------------------------------
>
> Key: HDDS-2301
> URL: https://issues.apache.org/jira/browse/HDDS-2301
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Rajesh Balamohan
> Assignee: Supratim Deka
> Priority: Major
> Labels: performance, pull-request-available
> Attachments: om_write_profile.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Benchmark:
> ------------
> Simple benchmark which creates 100 and 1000s of keys (empty directory) in
> OM. This is done in a tight loop and multiple threads from client side to add
> enough load on CPU. Note that intention is to understand the bottlenecks in
> OM (intentionally avoiding interactions with SCM & DN).
> Observation:
> -------------
> During write path, Ozone checks {{OMFileRequest.verifyFilesInPath}}. This
> internally calls {{omMetadataManager.getKeyTable().get(dbKeyName)}} for every
> write operation. This turns out to be expensive and chokes the write path.
> [https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java#L155]
> [https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java#L63]
> In most of the cases, directory creation would be fresh entry. In such cases,
> it would be good to try with {{RocksDB::keyMayExist.}}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
