[jira] [Updated] (MAPREDUCE-6780) Add support for HDFS directory with erasure code policy to TeraGen and TeraSort
[ https://issues.apache.org/jira/browse/MAPREDUCE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6780: - Attachment: MAPREDUCE-6780-v2.patch Refine some wording > Add support for HDFS directory with erasure code policy to TeraGen and > TeraSort > --- > > Key: MAPREDUCE-6780 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6780 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6780-v1.patch, MAPREDUCE-6780-v2.patch > > > So far, HDFS file with erasure code policy doesn't support hflush and hsync > operation. This task is going to find a way to support writing data to > erasure code policy files in TeraGen and TeraSort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6780) Add support for HDFS directory with erasure code policy to TeraGen and TeraSort
[ https://issues.apache.org/jira/browse/MAPREDUCE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6780: - Status: Patch Available (was: Open) > Add support for HDFS directory with erasure code policy to TeraGen and > TeraSort > --- > > Key: MAPREDUCE-6780 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6780 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6780-v1.patch > > > So far, HDFS file with erasure code policy doesn't support hflush and hsync > operation. This task is going to find a way to support writing data to > erasure code policy files in TeraGen and TeraSort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6780) Add support for HDFS directory with erasure code policy to TeraGen and TeraSort
[ https://issues.apache.org/jira/browse/MAPREDUCE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6780: - Description: So far, HDFS file with erasure code policy doesn't support hflush and hsync operation. This task is going to find a way to support writing data to erasure code policy files in TeraGen and TeraSort. (was: So far, HDFS files with erasure code policy doesn't support hflush and hsync. To support writing files erasure code policy, this is going to add an new option to choose whether hflush and hsync should be used or not when write data in TeraGen and TeraSort. ) > Add support for HDFS directory with erasure code policy to TeraGen and > TeraSort > --- > > Key: MAPREDUCE-6780 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6780 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6780-v1.patch > > > So far, HDFS file with erasure code policy doesn't support hflush and hsync > operation. This task is going to find a way to support writing data to > erasure code policy files in TeraGen and TeraSort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6780) Add support for HDFS directory with erasure code policy to TeraGen and TeraSort
[ https://issues.apache.org/jira/browse/MAPREDUCE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6780: - Attachment: MAPREDUCE-6780-v1.patch Initial patch > Add support for HDFS directory with erasure code policy to TeraGen and > TeraSort > --- > > Key: MAPREDUCE-6780 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6780 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6780-v1.patch > > > So far, HDFS files with erasure code policy doesn't support hflush and hsync. > To support writing files erasure code policy, this is going to add an new > option to choose whether hflush and hsync should be used or not when write > data in TeraGen and TeraSort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6780) Add support for HDFS directory with erasure code policy to TeraGen and TeraSort
SammiChen created MAPREDUCE-6780: Summary: Add support for HDFS directory with erasure code policy to TeraGen and TeraSort Key: MAPREDUCE-6780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6780 Project: Hadoop Map/Reduce Issue Type: Task Reporter: SammiChen Assignee: SammiChen So far, HDFS files with erasure code policy doesn't support hflush and hsync. To support writing files erasure code policy, this is going to add an new option to choose whether hflush and hsync should be used or not when write data in TeraGen and TeraSort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500490#comment-15500490 ] SammiChen commented on MAPREDUCE-6774: -- Thanks Kai for review the patch and help commit to trunk! > Add support for HDFS erasure code policy to TestDFSIO > - > > Key: MAPREDUCE-6774 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: SammiChen >Assignee: SammiChen > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-6774-v1.patch, MAPREDUCE-6774-v2.patch, > MAPREDUCE-6774-v3.patch, MAPREDUCE-6774-v4.patch, MAPREDUCE-6774-v5.patch, > MAPREDUCE-6774-v6.patch > > > HDFS erasure code policy allows user to store directory and file to > predefined erasure code policies. Currently only 3x replication is supported > in TestDFSIO implementation. This is going to add an new option to enable > tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6774: - Attachment: MAPREDUCE-6774-v6.patch Fix 2 style issues > Add support for HDFS erasure code policy to TestDFSIO > - > > Key: MAPREDUCE-6774 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6774-v1.patch, MAPREDUCE-6774-v2.patch, > MAPREDUCE-6774-v3.patch, MAPREDUCE-6774-v4.patch, MAPREDUCE-6774-v5.patch, > MAPREDUCE-6774-v6.patch > > > HDFS erasure code policy allows user to store directory and file to > predefined erasure code policies. Currently only 3x replication is supported > in TestDFSIO implementation. This is going to add an new option to enable > tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6774: - Attachment: MAPREDUCE-6774-v5.patch Improved patch based on Kai's suggestion > Add support for HDFS erasure code policy to TestDFSIO > - > > Key: MAPREDUCE-6774 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6774-v1.patch, MAPREDUCE-6774-v2.patch, > MAPREDUCE-6774-v3.patch, MAPREDUCE-6774-v4.patch, MAPREDUCE-6774-v5.patch > > > HDFS erasure code policy allows user to store directory and file to > predefined erasure code policies. Currently only 3x replication is supported > in TestDFSIO implementation. This is going to add an new option to enable > tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6774: - Attachment: MAPREDUCE-6774-v4.patch > Add support for HDFS erasure code policy to TestDFSIO > - > > Key: MAPREDUCE-6774 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6774-v1.patch, MAPREDUCE-6774-v2.patch, > MAPREDUCE-6774-v3.patch, MAPREDUCE-6774-v4.patch > > > HDFS erasure code policy allows user to store directory and file to > predefined erasure code policies. Currently only 3x replication is supported > in TestDFSIO implementation. This is going to add an new option to enable > tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[
https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486876#comment-15486876
]
SammiChen commented on MAPREDUCE-6774:
--
Thanks Kai, for take time review the patch and give very good suggestion!
The name 'checkStoragePolicy' sounds better than 'isStoragePolicyValid'. So
does the name 'checkErasureCodePolicy'. I uploaded Patch v4 which contains the
improvement. One the other hand, I would suggest keep the return value
{{boolean}} of each check function. Because the {{main}} function depends on
the return value, not the exception, to handle the invalid parameter case.
> Add support for HDFS erasure code policy to TestDFSIO
> -
>
> Key: MAPREDUCE-6774
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: MAPREDUCE-6774-v1.patch, MAPREDUCE-6774-v2.patch,
> MAPREDUCE-6774-v3.patch, MAPREDUCE-6774-v4.patch
>
>
> HDFS erasure code policy allows user to store directory and file to
> predefined erasure code policies. Currently only 3x replication is supported
> in TestDFSIO implementation. This is going to add an new option to enable
> tests of files with erasure code policy enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6774: - Attachment: MAPREDUCE-6774-v3.patch fix style issues > Add support for HDFS erasure code policy to TestDFSIO > - > > Key: MAPREDUCE-6774 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6774-v1.patch, MAPREDUCE-6774-v2.patch, > MAPREDUCE-6774-v3.patch > > > HDFS erasure code policy allows user to store directory and file to > predefined erasure code policies. Currently only 3x replication is supported > in TestDFSIO implementation. This is going to add an new option to enable > tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6774: - Attachment: MAPREDUCE-6774-v2.patch 1. improve the patch, only io_data directory is mattered when apply EC policy 2. fix checkstyle reported issue > Add support for HDFS erasure code policy to TestDFSIO > - > > Key: MAPREDUCE-6774 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: SammiChen >Assignee: SammiChen > Attachments: MAPREDUCE-6774-v1.patch, MAPREDUCE-6774-v2.patch > > > HDFS erasure code policy allows user to store directory and file to > predefined erasure code policies. Currently only 3x replication is supported > in TestDFSIO implementation. This is going to add an new option to enable > tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6774: - Status: Patch Available (was: Open) > Add support for HDFS erasure code policy to TestDFSIO > - > > Key: MAPREDUCE-6774 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: SammiChen > Attachments: MAPREDUCE-6774-v1.patch > > > HDFS erasure code policy allows user to store directory and file to > predefined erasure code policies. Currently only 3x replication is supported > in TestDFSIO implementation. This is going to add an new option to enable > tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated MAPREDUCE-6774: - Attachment: MAPREDUCE-6774-v1.patch > Add support for HDFS erasure code policy to TestDFSIO > - > > Key: MAPREDUCE-6774 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: SammiChen > Attachments: MAPREDUCE-6774-v1.patch > > > HDFS erasure code policy allows user to store directory and file to > predefined erasure code policies. Currently only 3x replication is supported > in TestDFSIO implementation. This is going to add an new option to enable > tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6774) Add support for HDFS erasure code policy to TestDFSIO
SammiChen created MAPREDUCE-6774: Summary: Add support for HDFS erasure code policy to TestDFSIO Key: MAPREDUCE-6774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6774 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: SammiChen HDFS erasure code policy allows user to store directory and file to predefined erasure code policies. Currently only 3x replication is supported in TestDFSIO implementation. This is going to add an new option to enable tests of files with erasure code policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO
[
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434646#comment-15434646
]
SammiChen commented on MAPREDUCE-6578:
--
Thanks Kai's suggestion. If the user input {{storagePolicy}} parameter is
incorrect or invalid, it's better provide user the knowledge that what's are
current supported storage policy list.
Update the patch based on above thoughts.
> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch,
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different
> storage medias according to predefined storage policies. Only 'Default'
> policy is supported in current TestDFSIO implementation. This is going to add
> an new option to enable tests of other storage polices.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
