[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-1530: - Resolution: Fixed Fix Version/s: 0.4.1 Target Version/s: 0.4.1 (was: 0.5.0) Status: Resolved (was: Patch Available) Thanks [~xudongcao] for the contribution and all for the reviews. I've commit the patch to trunk. > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Status: Patch Available (was: In Progress) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Status: In Progress (was: Patch Available) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Status: Patch Available (was: Open) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1530: - Labels: pull-request-available (was: ) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Labels: pull-request-available > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Attachment: (was: HDDS-1530.002.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Status: Open (was: Patch Available) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Attachment: (was: HDDS-1530.001.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Attachment: (was: HDDS-1530.001.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Attachment: HDDS-1530.001.patch HDDS-1530.002.patch Status: Patch Available (was: Open) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Attachment: (was: HDDS-1530.002.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Status: Open (was: Patch Available) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Status: Patch Available (was: Open) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HDDS-1530: - Status: Open (was: Patch Available) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1530: --- Status: Patch Available (was: Open) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1530: --- Status: Open (was: Patch Available) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1530: Target Version/s: 0.5.0 > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530.002.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Description: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. Freon lacks option such as "--validateWrites", thus users cannot manually specify that verification is required after writing. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Add a "--validateWrites" option to Freon command line, users can provide this option to indicate that a validation is required after write. was: *Current problems:* 1. Freon does not support big files larger than 2GB because it use an int type "keySize" parameter and also "keyValue" buffer size. 2. Freon allocates a entire buffer for each key at once, so if the key size is large and the concurrency is high, freon will report OOM exception frequently. 3. Freon lacks option such as "--validateWrites", thus users cannot manually specify that verification is required after writing. *Some solutions:* 1. Use a long type "keySize" parameter, make sure freon can support big files larger than 2GB. 2. Use a small buffer repeatedly than allocating the entire key-size buffer at once, the default buffer size is 4K and can be configured by "–bufferSize" parameter. 3. Add a "--validateWrites" option to Freon command line, users can provide this option to indicate that a validation is required after write. 4. Remove the process of appending an uuid to each key, which is of little significance and complicates the code, especially when writting with a small buffer repeatedly. > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch, HDDS-1530.002.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530.001.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: (was: HDDS-1530.001.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-1530: - Status: Patch Available (was: Open) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530.001.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: (was: HDDS-1530.001.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530.001.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: (was: HDDS-1530v0.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530.001.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: (was: HDFS-14484v0.patch) > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530v0.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[ https://issues.apache.org/jira/browse/HDDS-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xudongcao updated HDDS-1530: Attachment: HDDS-1530v0.patch > Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and > "--validateWrites" options. > -- > > Key: HDDS-1530 > URL: https://issues.apache.org/jira/browse/HDDS-1530 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: xudongcao >Assignee: xudongcao >Priority: Major > Attachments: HDDS-1530v0.patch > > > *Current problems:* > 1. Freon does not support big files larger than 2GB because it use an int > type "keySize" parameter and also "keyValue" buffer size. > 2. Freon allocates a entire buffer for each key at once, so if the key size > is large and the concurrency is high, freon will report OOM exception > frequently. > 3. Freon lacks option such as "--validateWrites", thus users cannot manually > specify that verification is required after writing. > *Some solutions:* > 1. Use a long type "keySize" parameter, make sure freon can support big > files larger than 2GB. > 2. Use a small buffer repeatedly than allocating the entire key-size buffer > at once, the default buffer size is 4K and can be configured by "–bufferSize" > parameter. > 3. Add a "--validateWrites" option to Freon command line, users can provide > this option to indicate that a validation is required after write. > 4. Remove the process of appending an uuid to each key, which is of little > significance and complicates the code, especially when writting with a small > buffer repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
