[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840936#comment-16840936 ]
xudongcao commented on HDDS-1530: --------------------------------- [~anu] [~arpitagarwal] I revert the UUID issue in HDDS-1530.002.patch, Please review it if you have time. > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > ------------------------------------------------------------------------------------------------------ > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test > Reporter: xudongcao > Assignee: xudongcao > Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org