[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530.002.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. Freon lacks option such as "--validateWrites", thus users cannot manually specify that verification is required after writing. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Add a "--validateWrites" option to Freon command line, users can provide this option to indicate that a validation is required after write. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. Freon lacks option such as "--validateWrites", thus users cannot manually specify that verification is required after writing. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Add a "--validateWrites" option to Freon command line, users can provide this option to indicate that a validation is required after write. 4. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially when writting with a small buffer repeatedly. > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840936#comment-16840936 ] xudongcao commented on HDDS-1530: - [~anu] [~arpitagarwal] I revert the UUID issue in HDDS-1530.002.patch, Please review it if you have time. > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530.001.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: (was: HDDS-1530.001.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1532) Freon: Improve the concurrency testing framework.
[ https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1532: Description: Currently, Freon's concurrency framework is just on volume-level, but in actual testing, users are likely to provide a smaller volume number（typically 1）, and a larger bucket number and key number, in which case the existing concurrency framework can not make good use of the thread pool. We need to improve the concurrency framework. make the volume creation task, bucket creation task, and key creation task all can be equally submitted to the thread pool as a general task. was: Currently, Freon's concurrency framework is just on volume-level, but in actual testing, users are likely to provide a smaller volume number, and a larger bucket number and key number, in which case the existing concurrency framework can not make good use of the thread pool. We need to improve the concurrency framework. make the volume creation task, bucket creation task, and key creation task all can be equally submitted to the thread pool as a general task. > Freon: Improve the concurrency testing framework. > - > > Key: HDDS-1532 > URL: https://issues.apache.org/jira/browse/HDDS-1532 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > Currently, Freon's concurrency framework is just on volume-level, but in > actual testing, users are likely to provide a smaller volume number（typically > 1）, and a larger bucket number and key number, in which case the existing > concurrency framework can not make good use of the thread pool. > We need to improve the concurrency framework. make the volume creation task, > bucket creation task, and key creation task all can be equally submitted to > the thread pool as a general task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1532) Freon: Improve the concurrency testing framework.
[ https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1532: Description: Currently, Freon's concurrency framework is just on volume-level, but in actual testing, users are likely to provide a smaller volume number（typically 1）, and a larger bucket number and key number, in which case the existing concurrency framework can not make good use of the thread pool. We need to improve the concurrency policy, make the volume creation task, bucket creation task, and key creation task all can be equally submitted to the thread pool as a general task. was: Currently, Freon's concurrency framework is just on volume-level, but in actual testing, users are likely to provide a smaller volume number（typically 1）, and a larger bucket number and key number, in which case the existing concurrency framework can not make good use of the thread pool. We need to improve the concurrency framework. make the volume creation task, bucket creation task, and key creation task all can be equally submitted to the thread pool as a general task. > Freon: Improve the concurrency testing framework. > - > > Key: HDDS-1532 > URL: https://issues.apache.org/jira/browse/HDDS-1532 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > Currently, Freon's concurrency framework is just on volume-level, but in > actual testing, users are likely to provide a smaller volume number（typically > 1）, and a larger bucket number and key number, in which case the existing > concurrency framework can not make good use of the thread pool. > We need to improve the concurrency policy, make the volume creation task, > bucket creation task, and key creation task all can be equally submitted to > the thread pool as a general task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1532) Freon: Improve the concurrency testing framework.
[ https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1532: Summary: Freon: Improve the concurrency testing framework. (was: Freon: Improve the concurrency framework.) > Freon: Improve the concurrency testing framework. > - > > Key: HDDS-1532 > URL: https://issues.apache.org/jira/browse/HDDS-1532 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > Currently, Freon's concurrency framework is just on volume-level, but in > actual testing, users are likely to provide a smaller volume number, and a > larger bucket number and key number, in which case the existing concurrency > framework can not make good use of the thread pool. > We need to improve the concurrency framework. make the volume creation task, > bucket creation task, and key creation task all can be equally submitted to > the thread pool as a general task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1532) Freon: Improve the concurrency framework.
[ https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao reassigned HDDS-1532: --- Assignee: xudongcao > Freon: Improve the concurrency framework. > - > > Key: HDDS-1532 > URL: https://issues.apache.org/jira/browse/HDDS-1532 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > Currently, Freon's concurrency framework is just on volume-level, but in > actual testing, users are likely to provide a smaller volume number, and a > larger bucket number and key number, in which case the existing concurrency > framework can not make good use of the thread pool. > We need to improve the concurrency framework. make the volume creation task, > bucket creation task, and key creation task all can be equally submitted to > the thread pool as a general task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1532) Freon: Improve the concurrency framework.
[ https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1532: Summary: Freon: Improve the concurrency framework. (was: Improve the concurrency framework) > Freon: Improve the concurrency framework. > - > > Key: HDDS-1532 > URL: https://issues.apache.org/jira/browse/HDDS-1532 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Priority: Major > > Currently, Freon's concurrency framework is just on volume-level, but in > actual testing, users are likely to provide a smaller volume number, and a > larger bucket number and key number, in which case the existing concurrency > framework can not make good use of the thread pool. > We need to improve the concurrency framework. make the volume creation task, > bucket creation task, and key creation task all can be equally submitted to > the thread pool as a general task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1532) Improve the concurrency framework
[ https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1532: Component/s: test > Improve the concurrency framework > - > > Key: HDDS-1532 > URL: https://issues.apache.org/jira/browse/HDDS-1532 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Priority: Major > > Currently, Freon's concurrency framework is just on volume-level, but in > actual testing, users are likely to provide a smaller volume number, and a > larger bucket number and key number, in which case the existing concurrency > framework can not make good use of the thread pool. > We need to improve the concurrency framework. make the volume creation task, > bucket creation task, and key creation task all can be equally submitted to > the thread pool as a general task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1532) Improve the concurrency framework
[ https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1532: Environment: (was: Improve the concurrency framework. Now the volume creation task, bucket creation task, and key creation task all can be submitted to the thread pool concurrently as a general task. ) > Improve the concurrency framework > - > > Key: HDDS-1532 > URL: https://issues.apache.org/jira/browse/HDDS-1532 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: xudongcao >Priority: Major > > Currently, Freon's concurrency framework is just on volume-level, but in > actual testing, users are likely to provide a smaller volume number, and a > larger bucket number and key number, in which case the existing concurrency > framework can not make good use of the thread pool. > We need to improve the concurrency framework. make the volume creation task, > bucket creation task, and key creation task all can be equally submitted to > the thread pool as a general task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1532) Improve the concurrency framework
[ https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1532: Description: Currently, Freon's concurrency framework is just on volume-level, but in actual testing, users are likely to provide a smaller volume number, and a larger bucket number and key number, in which case the existing concurrency framework can not make good use of the thread pool. We need to improve the concurrency framework. make the volume creation task, bucket creation task, and key creation task all can be equally submitted to the thread pool as a general task. was:Currently, Freon's concurrency framework is just on volume-level, but in actual testing, users are likely to provide a smaller volume number, and a larger bucket number and key number, in which case the existing concurrency framework can not make good use of the thread pool. > Improve the concurrency framework > - > > Key: HDDS-1532 > URL: https://issues.apache.org/jira/browse/HDDS-1532 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Environment: Improve the concurrency framework. Now the volume > creation task, bucket creation task, and key creation task all can be > submitted to the thread pool concurrently as a general task. >Reporter: xudongcao >Priority: Major > > Currently, Freon's concurrency framework is just on volume-level, but in > actual testing, users are likely to provide a smaller volume number, and a > larger bucket number and key number, in which case the existing concurrency > framework can not make good use of the thread pool. > We need to improve the concurrency framework. make the volume creation task, > bucket creation task, and key creation task all can be equally submitted to > the thread pool as a general task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1532) Improve the concurrency framework
xudongcao created HDDS-1532: --- Summary: Improve the concurrency framework Key: HDDS-1532 URL: https://issues.apache.org/jira/browse/HDDS-1532 Project: Hadoop Distributed Data Store Issue Type: Bug Environment: Improve the concurrency framework. Now the volume creation task, bucket creation task, and key creation task all can be submitted to the thread pool concurrently as a general task. Reporter: xudongcao Currently, Freon's concurrency framework is just on volume-level, but in actual testing, users are likely to provide a smaller volume number, and a larger bucket number and key number, in which case the existing concurrency framework can not make good use of the thread pool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530.001.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: (was: HDDS-1530.001.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530.001.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: (was: HDDS-1530v0.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: (was: HDFS-14484v0.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530v0.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530v0.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530v0.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Attachment: HDFS-14484v0.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDFS-14484v0.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", so users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. Freon lacks option such as "--validateWrites", thus users cannot manually specify that verification is required after writing. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Add a "--validateWrites" option to Freon command line, users can provide this option to indicate that a validation is required after write. 4. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially when writting with a small buffer repeatedly. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. Freon lacks option such as "--validateWrites", so users cannot manually specify that verification is required after writing. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Add a "--validateWrites" option to Freon command line, users can provide this option to indicate that a validation is required after write. 4. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially when writting with a small buffer repeatedly. > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDFS-14484v0.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Attachment: (was: HDFS-14484v0.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDFS-14484v0.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", so users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Attachment: HDFS-14484v0.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDFS-14484v0.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", so users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Attachment: (was: HDFS-14484.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", so users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839946#comment-16839946 ] xudongcao commented on HDFS-14484: -- In fact, my initial change was to keep this, but I found this would make the code more complicated and didn't make much sense, so I removed it. But if you think this is necessary, I will add it back. > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDFS-14484.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", so users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Attachment: HDFS-14484.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDFS-14484.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", so users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Summary: Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options. (was: Ozone: Freon: Support big files larger than 2GB and introduce "--bufferSize" and "--validateWrites" options.) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", so users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and introduce "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. Freon lacks option such as "--validateWrites", so users cannot manually specify that verification is required after writing. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Add a "--validateWrites" option to Freon command line, users can provide this option to indicate that a validation is required after write. 4. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially when writting with a small buffer repeatedly. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially when writting with a small buffer repeatedly. > Ozone: Freon: Support big files larger than 2GB and introduce "--bufferSize" > and "--validateWrites" options. > > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", so users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and introduce "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Summary: Ozone: Freon: Support big files larger than 2GB and introduce "--bufferSize" and "--validateWrites" options. (was: Ozone: Freon: Support big files larger than 2GB and introduce a "--bufferSize" parameter.) > Ozone: Freon: Support big files larger than 2GB and introduce "--bufferSize" > and "--validateWrites" options. > > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and introduce a "--bufferSize" parameter.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially when writting with a small buffer repeatedly. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially writting with a small buffer repeatedly. > Ozone: Freon: Support big files larger than 2GB and introduce a > "--bufferSize" parameter. > - > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and introduce a "--bufferSize" parameter.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially writting with a small buffer repeatedly. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Remove the process of appending an uuid to each key, which is of little significance and complicates the code. > Ozone: Freon: Support big files larger than 2GB and introduce a > "--bufferSize" parameter. > - > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and introduce a "--bufferSize" parameter.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Remove the process of appending an uuid to each key, which is of little significance and complicates the code. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *And we need:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Remove the process of appending an uuid to each key, which is of little significance and complicates the code. > Ozone: Freon: Support big files larger than 2GB and introduce a > "--bufferSize" parameter. > - > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and introduce a "--bufferSize" parameter.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *And we need:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Remove the process of appending an uuid to each key, which is of little significance and complicates the code. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *And we need:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. > Ozone: Freon: Support big files larger than 2GB and introduce a > "--bufferSize" parameter. > - > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > *And we need:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and introduce a "--bufferSize" parameter.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. *And we need:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. The existing concurrency framework only supports volume-level concurrency, but in actual testing, a user is likely to specify a small number of volumes, and a larger number of buckets and keys . At this point, it can't take full advantage of thread pool to speed up testing. *And we need:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Improve the concurrency framework. Now the volume creation task, bucket creation task, and key creation task all can be submitted to the thread pool concurrently as a general task. > Ozone: Freon: Support big files larger than 2GB and introduce a > "--bufferSize" parameter. > - > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > *And we need:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and introduce a "--bufferSize" parameter.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Summary: Ozone: Freon: Support big files larger than 2GB and introduce a "--bufferSize" parameter. (was: Ozone: Freon: Support big files larger than 2GB and improve concurrent framework.) > Ozone: Freon: Support big files larger than 2GB and introduce a > "--bufferSize" parameter. > - > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. The existing concurrency framework only supports volume-level > concurrency, but in actual testing, a user is likely to specify a small > number of volumes, and a larger number of buckets and keys . At this point, > it can't take full advantage of thread pool to speed up testing. > *And we need:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Improve the concurrency framework. Now the volume creation task, bucket > creation task, and key creation task all can be submitted to the thread pool > concurrently as a general task. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and improve concurrent framework.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Summary: Ozone: Freon: Support big files larger than 2GB and improve concurrent framework. (was: Ozone: Freon: Support big files larger than 2GB and improve concurrent operations.) > Ozone: Freon: Support big files larger than 2GB and improve concurrent > framework. > - > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. The existing concurrency framework only supports volume-level > concurrency, but in actual testing, a user is likely to specify a small > number of volumes, and a larger number of buckets and keys . At this point, > it can't take full advantage of thread pool to speed up testing. > *And we need:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Improve the concurrency framework. Now the volume creation task, bucket > creation task, and key creation task all can be submitted to the thread pool > concurrently as a general task. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and improve concurrent operations.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. The existing concurrency framework only supports volume-level concurrency, but in actual testing, a user is likely to specify a small number of volumes, and a larger number of buckets and keys . At this point, it can't take full advantage of thread pool to speed up testing. *And we need:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Improve the concurrency framework. Now the volume creation task, bucket creation task, and key creation task all can be submitted to the thread pool concurrently as a general task. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use a int type for keysize. 2. Freon allocates the entire key buffer at once (this is exactly the reson why it cannot support files larger than 2GB, because the maximum size of a java array byte[] is 2G), if the test pressure is big enough, freon will report OOM exception frequently. *And we need:* 1. Make sure freon can support big files larger than 2GB. 2. Use an internal small buffer repeatedly than allocating the entire key-size buffer at once, the typical size of the small buffer is 4K and it can be configured. > Ozone: Freon: Support big files larger than 2GB and improve concurrent > operations. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. The existing concurrency framework only supports volume-level > concurrency, but in actual testing, a user is likely to specify a small > number of volumes, and a larger number of buckets and keys . At this point, > it can't take full advantage of thread pool to speed up testing. > *And we need:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Improve the concurrency framework. Now the volume creation task, bucket > creation task, and key creation task all can be submitted to the thread pool > concurrently as a general task. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and improve concurrent operations.
[ https://issues.apache.org/jira/browse/HDFS-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDFS-14484: - Summary: Ozone: Freon: Support big files larger than 2GB and improve concurrent operations. (was: Ozone: Freon: Support big files larger than 2GB and fix OOM.) > Ozone: Freon: Support big files larger than 2GB and improve concurrent > operations. > -- > > Key: HDFS-14484 > URL: https://issues.apache.org/jira/browse/HDFS-14484 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use a int > type for keysize. > 2. Freon allocates the entire key buffer at once (this is exactly the reson > why it cannot support files larger than 2GB, because the maximum size of a > java array byte[] is 2G), if the test pressure is big enough, freon will > report OOM exception frequently. > *And we need:* > 1. Make sure freon can support big files larger than 2GB. > 2. Use an internal small buffer repeatedly than allocating the entire > key-size buffer at once, the typical size of the small buffer is 4K and it > can be configured. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14484) Ozone: Freon: Support big files larger than 2GB and fix OOM.
xudongcao created HDFS-14484: Summary: Ozone: Freon: Support big files larger than 2GB and fix OOM. Key: HDFS-14484 URL: https://issues.apache.org/jira/browse/HDFS-14484 Project: Hadoop HDFS Issue Type: Bug Components: benchmarks Reporter: xudongcao Assignee: xudongcao *Current problems:* 1. Freon does not support big files larger than 2GB because it use a int type for keysize. 2. Freon allocates the entire key buffer at once (this is exactly the reson why it cannot support files larger than 2GB, because the maximum size of a java array byte[] is 2G), if the test pressure is big enough, freon will report OOM exception frequently. *And we need:* 1. Make sure freon can support big files larger than 2GB. 2. Use an internal small buffer repeatedly than allocating the entire key-size buffer at once, the typical size of the small buffer is 4K and it can be configured. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
