[jira] [Resolved] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng resolved HADOOP-19085.
Hadoop Flags: Reviewed
Target Version/s: 3.3.6
Resolution: Fixed
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-19085.001.patch, HDFS Compatibility Benchmark
> Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827774#comment-17827774
]
Kai Zheng commented on HADOOP-19085:
For the valuable suggestions mentioned in the discussion, I suggest we follow
up in separate issues, [~han.liu].
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-19085.001.patch, HDFS Compatibility Benchmark
> Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827773#comment-17827773
]
Kai Zheng commented on HADOOP-19085:
The patch was just committed. Thanks [~han.liu] for the contribution, and
[~ste...@apache.org] for the guidance!
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-19085.001.patch, HDFS Compatibility Benchmark
> Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825136#comment-17825136
]
Kai Zheng commented on HADOOP-19085:
Thanks [~han.liu] for the update. The latest codes LGTM and +1. Would anyone
take a look?
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824611#comment-17824611
]
Kai Zheng commented on HADOOP-19085:
[~han.liu] Good work! Just a minor, would you grep the PR codes ""hdfs
compatibility" and refine some bit? Thanks!
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823076#comment-17823076
]
Kai Zheng commented on HADOOP-19085:
Hi [~han.liu] The work looks great overall. Some comments follow and would you
help check? Thanks!
1. About the document HdfsCompatBenchIssue.md, maybe we could reshape it as the
user doc like we have for other hadoop-tools modules.
2. For most classes in the package org/apache/hadoop/fs/compat, we could have a
sub package like 'common' for them, so we could leave the main package much
cleaner.
3. The tool (HdfsCompatibility.java) could be 'HdfsCompatTool'?
4. HdfsCompatFileSystemImpl could be 'HdfsCompatBasics' and moved to stay along
with the function cases together. Actually we could put all the HCFS API cases
together directly under the 'cases' package.
5. We could rename 'modification.t' to 'write.t', and have another file like
'directory.t' for dir related operations.
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819869#comment-17819869
]
Kai Zheng commented on HADOOP-19085:
Just did a quick glance at the code. [~han.liu] Let's do in favor of the
package "org.apache.hadoop.fs.compat" instead of "org.apache.hadoop.compat" to
wrap the new codes since the compat tool focuses on the fs aspect. Hadoop means
lots of things. Thanks.
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819866#comment-17819866
]
Kai Zheng commented on HADOOP-19085:
Got it. Thanks Steve for the refresh, :).
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Assignee: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819454#comment-17819454
]
Kai Zheng commented on HADOOP-19085:
Hi [~han.liu], since you're working on this, would you take it?
[~ste...@apache.org] could you help assign, thanks! I failed because I can't
see his name.
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng moved HDFS-17316 to HADOOP-19085:
---
Key: HADOOP-19085 (was: HDFS-17316)
Project: Hadoop Common (was: Hadoop HDFS)
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350032#comment-16350032 ] Kai Zheng commented on HADOOP-15027: Was the commit already corrected or should we reopen this? Could you help update or clarify some bit for me, [~Sammi]? Thanks. > AliyunOSS: Support multi-thread pre-read to improve sequential read from > Hadoop to Aliyun OSS performance > - > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch, > HADOOP-15027.009.patch, HADOOP-15027.010.patch, HADOOP-15027.011.patch, > HADOOP-15027.012.patch, HADOOP-15027.013.patch, HADOOP-15027.014.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317929#comment-16317929
]
Kai Zheng commented on HADOOP-15104:
Cherry picked to branch-3.0 as well. Thanks folks.
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-15104:
---
Fix Version/s: 3.0.1
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317891#comment-16317891
]
Kai Zheng commented on HADOOP-15104:
Got it, Andrew. Thanks for your nice elaboration :)
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 3.1.0, 2.10.0, 2.9.1
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16315979#comment-16315979
]
Kai Zheng commented on HADOOP-15104:
Thanks [~wujinhu] for the reminding and double check.
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 3.1.0, 2.10.0, 2.9.1
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16315977#comment-16315977
]
Kai Zheng commented on HADOOP-15104:
Sorry for the late notice and response on this, [~rchiang]. So you're sure it's
branch-3.0 instead of branch-3.0.0 we're currently using for the 3.0.x series,
right? If it's not frozen yet and still in time, I'll correct the commit soon.
[~andrew.wang] could you help clarify some bit? What's branch-3.0.0 for and
should we clean it up? Thank you!
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 3.1.0, 2.10.0, 2.9.1
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290282#comment-16290282
]
Kai Zheng commented on HADOOP-15039:
Just done. The commit was corrected.
Sorry for the inconvenience.
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch, HADOOP-15039.004.patch, HADOOP-15039.005.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290276#comment-16290276
]
Kai Zheng commented on HADOOP-15039:
Oops, the commit was incorrect. Thanks [~uncleGen] for the catch. I'll revert
the commit and re-commit the patch.
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch, HADOOP-15039.004.patch, HADOOP-15039.005.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-15104:
---
Fix Version/s: 3.1.0
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-15104:
---
Fix Version/s: (was: 2.9.1)
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng resolved HADOOP-15104.
Resolution: Fixed
Committed to trunk and branch-3.0.0. Thanks Jinhu for the work.
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 2.9.1
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15104) AliyunOSS: change the default value of max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-15104:
---
Summary: AliyunOSS: change the default value of max error retry (was:
AliyunOSS: change default max error retry)
> AliyunOSS: change the default value of max error retry
> --
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 2.9.1
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15104) AliyunOSS: change default max error retry
[
https://issues.apache.org/jira/browse/HADOOP-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283678#comment-16283678
]
Kai Zheng commented on HADOOP-15104:
A minor change that makes sense. +1.
> AliyunOSS: change default max error retry
> -
>
> Key: HADOOP-15104
> URL: https://issues.apache.org/jira/browse/HADOOP-15104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: wujinhu
>Assignee: wujinhu
> Fix For: 3.0.0, 2.9.1
>
> Attachments: HADOOP-15104.001.patch
>
>
> Currently, default number of times we should retry errors is 20, however,
> oss sdk retry delay is
> {code:java}
> long delay = (long)Math.pow(2, retries) * 0.3
> {code}
> when one error occurs. So, if we retry 20 times, sleep time will be about
> 3.64 days and it is unacceptable. So we should change the default behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[ https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281518#comment-16281518 ] Kai Zheng commented on HADOOP-15080: Thanks [~Sammi] for the quick tweak on this! The change LGTM and +1, also having a check on the new sdk and it looks clean. > Cat-X dependency on org.json via derived json-lib > - > > Key: HADOOP-15080 > URL: https://issues.apache.org/jira/browse/HADOOP-15080 > Project: Hadoop Common > Issue Type: Bug > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas >Priority: Blocker > Attachments: HADOOP-15080-branch-3.0.0.001.patch, > HADOOP-15080-branch-3.0.0.002.patch > > > The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library > (from which json-lib may be derived) is released under a > [category-x|https://www.apache.org/legal/resolved.html#json] license. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-15039:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks Genmao for the contribution!
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch, HADOOP-15039.004.patch, HADOOP-15039.005.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279626#comment-16279626
]
Kai Zheng commented on HADOOP-15039:
The latest patch LGTM and +1. Thanks Genmao for the update!
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch, HADOOP-15039.004.patch, HADOOP-15039.005.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276251#comment-16276251
]
Kai Zheng commented on HADOOP-15039:
Thanks [~uncleGen] for the work. It LGTM and the minor comment is, would you
clean up some bit about SemaphoredDelegatingExecutor? The comments in the class
header would be generic since it's moved to the common module a a utility.
+1 pending on the minor.
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-15039:
---
Summary: Move SemaphoredDelegatingExecutor to hadoop-common (was: move
SemaphoredDelegatingExecutor to hadoop-common)
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
[
https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276234#comment-16276234
]
Kai Zheng edited comment on HADOOP-15038 at 12/4/17 2:56 AM:
-
Thanks for the discussion, folks.
The idea to extract MetadataStore into a Hadoop common module sounds good to
me, but when do it, we should be careful not to introduce non-trivial
dependencies.
[~ste...@apache.org] it looks to me the cloudup package would be a good fit for
the new module. Could you introduce the new module when do the work of
HADOOP-14766? If nice, how to name the module (under hadoop-common-project)?
IMO hadoop-cloud-common might be better, sure hadoop-cloud-core is also good to
me.
[~fabbri] do you have any concerns if the main work of MetadataStore is
extracted and put into hadoop-common-project/hadoop-cloud-common?
Not sure if [~chris.douglas] would have more thoughts on this.
Thanks!
was (Author: drankye):
Thanks for the discussion, folks.
The idea to extract MetadataStore into a Hadoop common module sounds good to
me, but when do it, we should be careful not to introduce non-trivial
dependencies.
[~ste...@apache.org] it looks to me the cloudup package would be a good fit for
the new module. Could you introduce the new module when do the work of
HADOOP-14766? If nice, how to name the module (under hadoop-common-project)?
IMO hadoop-cloud-common might be better, sure hadoop-cloud-core is also good to
me.
[~ajfabbri] do you have any concerns if the main work of MetadataStore is
extracted and put into hadoop-common-project/hadoop-cloud-common?
Not sure if [~chris.douglas] would have more thoughts on this.
Thanks!
> Abstract MetadataStore in S3Guard into a common module.
> ---
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
> into a common module.
> Based on this work, other filesystem or object store can implement their own
> metastore for optimization (known issues like consistency problem and
> metadata operation performance). [~ste...@apache.org] and other guys have
> done many base and great works in {{S3Guard}}. It is very helpful to start
> work. I did some perf test in HADOOP-14098, and started related work for
> Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like
> metadata cache inconsistent with S3 and so on. It also will be a problem for
> other object store. However, we can do these works in parallel.
> [~ste...@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
[
https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276234#comment-16276234
]
Kai Zheng commented on HADOOP-15038:
Thanks for the discussion, folks.
The idea to extract MetadataStore into a Hadoop common module sounds good to
me, but when do it, we should be careful not to introduce non-trivial
dependencies.
[~ste...@apache.org] it looks to me the cloudup package would be a good fit for
the new module. Could you introduce the new module when do the work of
HADOOP-14766? If nice, how to name the module (under hadoop-common-project)?
IMO hadoop-cloud-common might be better, sure hadoop-cloud-core is also good to
me.
[~ajfabbri] do you have any concerns if the main work of MetadataStore is
extracted and put into hadoop-common-project/hadoop-cloud-common?
Not sure if [~chris.douglas] would have more thoughts on this.
Thanks!
> Abstract MetadataStore in S3Guard into a common module.
> ---
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
> into a common module.
> Based on this work, other filesystem or object store can implement their own
> metastore for optimization (known issues like consistency problem and
> metadata operation performance). [~ste...@apache.org] and other guys have
> done many base and great works in {{S3Guard}}. It is very helpful to start
> work. I did some perf test in HADOOP-14098, and started related work for
> Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like
> metadata cache inconsistent with S3 and so on. It also will be a problem for
> other object store. However, we can do these works in parallel.
> [~ste...@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.8+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264873#comment-16264873 ] Kai Zheng commented on HADOOP-14964: Thanks [~Sammi] for the update. It sounds good, let's forget 2.7 and get 2.9 done first. We'll still need to discuss about 2.8 in the dev 2.8.3 release thread. The patch and test results for 2.9 LGTM and +1. Sammi, could you help proceed: 1. Commit it to branch 2.9; 2. Update all consolidated issue entries to reflect the fix version. Thank you! > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.8+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch, > HADOOP-14964-branch-2.9.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.8+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-14964: --- Summary: AliyunOSS: backport Aliyun OSS module to branch-2 and 2.8+ branches (was: AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches) > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.8+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch, > HADOOP-14964-branch-2.9.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260388#comment-16260388 ] Kai Zheng commented on HADOOP-14964: bq. Could it also apply to the hadoop-oss module? Yep, that's for the same thing, the one meant in this effort. bq. Sure, but what changed between these versions? I haven't got the chance to look at the codes closely, but the changes are all introduced by the revision upgrade of httpclient. If hadoopk-aliyun module was launched with different httpclient from the expected version by OSS SDK, it won't work in our test. I thought the latest OSS SDK should be using some advanced features in newer httpclient version. I thought the shade approach should be good because in the long run OSS SDK has the freedom to use its own version of httpclient library, not worrying about conflict with outside or elsewhere. > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260352#comment-16260352 ] Kai Zheng commented on HADOOP-14964: bq. but I thought that was the problem shading is designed to solve. If aliyun-oss SDK shades its dep httpclient by relocating and renaming the packages, I agree it should work and can avoid the conflict we met with the different httpclient versions used in Hadoop elsewhere. Thanks for the good suggestion! bq. What are the differences we need to resolve for the 2.9.1 hadoop-oss.jar to be usable in a 2.7 cluster? For 2.7, we have to use a lower version of aliyun-oss SDK to match with the right version of httpclient library used in 2.7. But as you suggested above, this can be avoided if Aliyun OSS team would shape the SDK incorporating the httpclient library by shading. Let me sync up with them to make the Hadoop side easier. Thanks Chris. > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260317#comment-16260317 ] Kai Zheng commented on HADOOP-14964: I'm not sure about the shaded approach. For example, hadoop-oss.jar -> httpclient-x.jar (by shade), hadoop-common.jar -> httpclient-y.jar, and assume we have a client app that uses both hadoop-oss.jar and hadoop-common.jar, is it possible to work? Maybe you meant it by relocating classes when shading? > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260293#comment-16260293 ] Kai Zheng commented on HADOOP-14964: bq. The hope is that the 2.9.1 jar could work in 2.7 and 2.8 clusters without modification. Unfortunately it's not. We have to provide different back port patches relying on different OSS SDKs for the matched httpclient library with Hadoop. Thanks Chris for taking care of this. > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260174#comment-16260174 ] Kai Zheng commented on HADOOP-14964: Thanks Chris for the comments! bq. In the future, please raise active backports in the release threads. Yes, it's a miss we didn't raise this in a release thread instead of here. I admit we didn't plan this well beforehand then found our users want this feature in 2.7+ releases instead of waiting for the 3.0 release. The module was worked out actually in the last year and we didn't make it in a release since then. A real bad and I'm sorry for that :(. bq. It would have been simpler if this issue had been visible before 2.9.0 was released. Now that 2.9.0 is out, this should target 2.10... Oops, I probably got what you meant. So 2.9.0 would be the right chance for a major feature to be in, and after that, 2.9.1+ will just be patch releases for bug fixes, right? bq. As discussed on the dev list, we have not released other FS plugins with patch releases. While initially awkward, it prevents situations where we need to generate a release in a moribund branch for an active component. Ok, I saw the problem. I don't want to cause mess as you said if it's the community release convention. I also received some similar concern from Steve. I'll try to align with my side and see how to proceed. As you suggested, looks like 2.9.1 would be good to target and for 2.7/2.8 releases we provide separate patches or jars hosted somewhere. > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15024) AliyunOSS: support user agent configuration and include that & Hadoop version information to oss server
[ https://issues.apache.org/jira/browse/HADOOP-15024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260103#comment-16260103 ] Kai Zheng commented on HADOOP-15024: Thanks Steve for the commit. My post +1. > AliyunOSS: support user agent configuration and include that & Hadoop version > information to oss server > --- > > Key: HADOOP-15024 > URL: https://issues.apache.org/jira/browse/HADOOP-15024 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0 >Reporter: SammiChen >Assignee: SammiChen > Attachments: HADOOP-15024.000.patch, HADOOP-15024.001.patch, > HADOOP-15024.002.patch > > > Provide oss client side Hadoop version to oss server, to help build access > statistic metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253257#comment-16253257 ] Kai Zheng commented on HADOOP-14964: Hi Steve, bq. FWIW, I think the object stores are less risky here, but packaging changes are. I agree. So for the above raised httpclient issue for 2.7 branch, what we're trying to come up is to do a new OSS SDK release aligning the httpclient version with Hadoop, in order to avoid changing into hadoop common side. Do you think this works for you? Thanks! > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12725) RPC encryption benchmark and optimization prototypes
[ https://issues.apache.org/jira/browse/HADOOP-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252900#comment-16252900 ] Kai Zheng commented on HADOOP-12725: Hello [~shv], Thanks for your interesting on this. We're aligned with HADOOP-10768 and looks like the work is going pretty well over there. Would you look at that one? Thanks! > RPC encryption benchmark and optimization prototypes > > > Key: HADOOP-12725 > URL: https://issues.apache.org/jira/browse/HADOOP-12725 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > > This would implement a benchmark tool to measure and compare the performance > of Hadoop IPC/RPC call when security is enabled and different SASL > QOP(Quality of Protection) is enforced. Given the data collected by this > benchmark, it would then be able to know if any performance concern when > considering to enforce privacy, integration, or authenticy protection level, > and do optimization accordingly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[ https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-14993: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~uncleGen] for the contribution! > AliyunOSS: Override listFiles and listLocatedStatus > > > Key: HADOOP-14993 > URL: https://issues.apache.org/jira/browse/HADOOP-14993 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.1.0 > > Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch, > HADOOP-14993.003.patch > > > Do a bulk listing off all entries under a path in one single operation, there > is no need to recursively walk the directory tree. > Updates: > - override listFiles and listLocatedStatus by using bulk listing > - some minor updates in hadoop-aliyun index.md -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[ https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251174#comment-16251174 ] Kai Zheng commented on HADOOP-14993: The latest patch LGTM and +1. Will commit it shortly. > AliyunOSS: Override listFiles and listLocatedStatus > > > Key: HADOOP-14993 > URL: https://issues.apache.org/jira/browse/HADOOP-14993 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch, > HADOOP-14993.003.patch > > > Do a bulk listing off all entries under a path in one single operation, there > is no need to recursively walk the directory tree. > Updates: > - override listFiles and listLocatedStatus by using bulk listing > - some minor updates in hadoop-aliyun index.md -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250931#comment-16250931 ] Kai Zheng commented on HADOOP-14964: Thanks Steve for the comments. bq. We can worry about 2.9, but I don't expect any patches going in to 2.7-2.8, which are essentially maintenance only right now. Looking at the latest discussions, I roughly agree. It sounds reasonable to freeze 2.7 branch, but for 2.8, I'm not very sure. It is more reasonable to support 2.8 branch longer time like it did for 2.7 branch. Will discuss in the common-dev thread. > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-14964: --- Summary: AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches (was: AliyunOSS: backport Aliyun OSS module to branch-2) > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249031#comment-16249031 ] Kai Zheng commented on HADOOP-14964: Having committed the back port patch to branch-2 branch, will continue the back port for 2.7/2.8/2.9 branches using this. > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HADOOP-14964: -- Assignee: SammiChen (was: Genmao Yu) > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247127#comment-16247127 ] Kai Zheng commented on HADOOP-14964: Thanks Sammi! It LGTM and +1. If no further comments until tomorrow, I will commit the consolidated patch into branch-2 writing the commit message with above commits history info. > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-14964: --- Summary: AliyunOSS: backport Aliyun OSS module to branch-2 (was: AliyunOSS: backport HADOOP-12756 to branch-2) > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport HADOOP-12756 to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247055#comment-16247055 ] Kai Zheng commented on HADOOP-14964: Hi Sammi, Thanks for your post! Another question, could you list the JIRAs that this consolidated patch contains? Thanks! I thought it's more than HADOOP-12756. I will update the title. > AliyunOSS: backport HADOOP-12756 to branch-2 > > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport HADOOP-12756 to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245304#comment-16245304 ] Kai Zheng commented on HADOOP-14964: Thanks Sammi for the patch! 1. Did you have some run on your local environment? If so, would you please post the local test results? 2. For 2.7 branch, I wonder if we could use previous OSS sdk version that matches some appropriate httpclient library. Would you do some check? Thanks! > AliyunOSS: backport HADOOP-12756 to branch-2 > > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[
https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240091#comment-16240091
]
Kai Zheng commented on HADOOP-14993:
Thanks for the work, [~uncleGen]. Some comments follow.
1. Is it possible to refactor {{listFiles}} and {{listLocatedStatus}} since
they look very similar sharing some codes/logic.
2. {{firstListing}} looks useless in {{createLocatedFileStatusIterator}}.
3. How about: isDirectory(OSSObjectSummary object)?
{code}
+ public static boolean objectRepresentsDirectory(final String name,
+ final long size) {
+return !name.isEmpty()
+ && name.charAt(name.length() - 1) == '/'
+ && size == 0L;
+ }
{code}
4. How about: {{FileStatusAcceptor}} => {{OssPathFilter}} that extends
{{PathFilter}}. The comments could be simplified like: "OSS specific path
filter"
{code}
+/**
+ * Interface to implement by the logic deciding whether to accept a summary
+ * entry or path as a valid file or directory.
+ */
+public interface FileStatusAcceptor {
{code}
> AliyunOSS: Override listFiles and listLocatedStatus
>
>
> Key: HADOOP-14993
> URL: https://issues.apache.org/jira/browse/HADOOP-14993
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch
>
>
> Do a bulk listing off all entries under a path in one single operation, there
> is no need to recursively walk the directory tree.
> Updates:
> - override listFiles and listLocatedStatus by using bulk listing
> - some minor updates in hadoop-aliyun index.md
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport HADOOP-12756 to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235405#comment-16235405 ] Kai Zheng commented on HADOOP-14964: Sorry for the late response, Genmao. Steve, could you help with my following questions? Thanks. bq. It's too late for the 2.9 branch... Did you mean the backport to 2.9 branch is too late? If so, I'd wonder why, because I don't aware any release on-going based on the branch and the branch is frozen right. Or did you mean it's too late for branch-2, since as you mentioned, we might not have 2.10? Thanks for your clarifying or confirm. I thought OSS support is rather standing alone and I roughly believe we could backport it to branch-2, branch-2.7, branch-2.8 and branch-2.9, technically. I'm not familiar with release logics, though. Would you cast some insights? Thanks again. > AliyunOSS: backport HADOOP-12756 to branch-2 > > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: Genmao Yu >Priority: Major > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14098) AliyunOSS: improve the performance of object metadata operation
[ https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210803#comment-16210803 ] Kai Zheng commented on HADOOP-14098: Thanks Genmao for working on this! bq. In consideration of localization, I may use some other database instead of DymanoDB. This looks reasonable to me. Have you looked into the work and evaluated how much effort to do so? I'm not sure if it's already pluggable. If not yet, we could make it pluggable first and then support other database. For your note, AFAIK, the HADOOP-13345 work mainly works for solving the consistency issues incurred by S3. For OSS, looks like you mainly wants to optimize the performance. > AliyunOSS: improve the performance of object metadata operation > --- > > Key: HADOOP-14098 > URL: https://issues.apache.org/jira/browse/HADOOP-14098 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > > Open this JIRA to research and address the potential request performance > issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14616) Erasurecode XOR testcase failures
[ https://issues.apache.org/jira/browse/HADOOP-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172886#comment-16172886 ] Kai Zheng commented on HADOOP-14616: That sounds solid. So let's wait for the Jenkins. > Erasurecode XOR testcase failures > -- > > Key: HADOOP-14616 > URL: https://issues.apache.org/jira/browse/HADOOP-14616 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: Huafeng Wang > Attachments: HADOOP-14616.001.patch > > > TestXORCoder, TestXORRawCoderInteroperable2, TestNativeXORRawCoder are the > testcases failing. All with the same error. This happens after the commit of > HADOOP-14479 > java.lang.InternalError: Invalid inputs > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.decodeImpl(Native > Method) > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.performDecodeImpl(NativeXORRawDecoder.java:44) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:58) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:74) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:105) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:163) > at > org.apache.hadoop.io.erasurecode.coder.ErasureDecodingStep.performCoding(ErasureDecodingStep.java:54) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performCodingStep(TestErasureCoderBase.java:126) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:95) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestXORCoder.testCodingNoDirectBuffer_erasing_p0(TestXORCoder.java:51) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14616) Erasurecode XOR testcase failures
[ https://issues.apache.org/jira/browse/HADOOP-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172863#comment-16172863 ] Kai Zheng commented on HADOOP-14616: The patch fixed a code error. LGTM and +1. > Erasurecode XOR testcase failures > -- > > Key: HADOOP-14616 > URL: https://issues.apache.org/jira/browse/HADOOP-14616 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: Huafeng Wang > Attachments: HADOOP-14616.001.patch > > > TestXORCoder, TestXORRawCoderInteroperable2, TestNativeXORRawCoder are the > testcases failing. All with the same error. This happens after the commit of > HADOOP-14479 > java.lang.InternalError: Invalid inputs > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.decodeImpl(Native > Method) > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.performDecodeImpl(NativeXORRawDecoder.java:44) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:58) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:74) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:105) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:163) > at > org.apache.hadoop.io.erasurecode.coder.ErasureDecodingStep.performCoding(ErasureDecodingStep.java:54) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performCodingStep(TestErasureCoderBase.java:126) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:95) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestXORCoder.testCodingNoDirectBuffer_erasing_p0(TestXORCoder.java:51) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14616) Erasurecode XOR testcase failures
[ https://issues.apache.org/jira/browse/HADOOP-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172858#comment-16172858 ] Kai Zheng commented on HADOOP-14616: Hi Huafeng, As the native EC testing hasn't been enabled in the Jenkins system yet, so let's not expect much help from it for this. Would you help run these tests locally and then post the results here? Thank you! > Erasurecode XOR testcase failures > -- > > Key: HADOOP-14616 > URL: https://issues.apache.org/jira/browse/HADOOP-14616 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: Huafeng Wang > Attachments: HADOOP-14616.001.patch > > > TestXORCoder, TestXORRawCoderInteroperable2, TestNativeXORRawCoder are the > testcases failing. All with the same error. This happens after the commit of > HADOOP-14479 > java.lang.InternalError: Invalid inputs > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.decodeImpl(Native > Method) > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.performDecodeImpl(NativeXORRawDecoder.java:44) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:58) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:74) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:105) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:163) > at > org.apache.hadoop.io.erasurecode.coder.ErasureDecodingStep.performCoding(ErasureDecodingStep.java:54) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performCodingStep(TestErasureCoderBase.java:126) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:95) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestXORCoder.testCodingNoDirectBuffer_erasing_p0(TestXORCoder.java:51) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14616) Erasurecode XOR testcase failures
[ https://issues.apache.org/jira/browse/HADOOP-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172857#comment-16172857 ] Kai Zheng commented on HADOOP-14616: Assigned to [~HuafengWang] as he posted the patch and works on it. > Erasurecode XOR testcase failures > -- > > Key: HADOOP-14616 > URL: https://issues.apache.org/jira/browse/HADOOP-14616 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: Huafeng Wang > Attachments: HADOOP-14616.001.patch > > > TestXORCoder, TestXORRawCoderInteroperable2, TestNativeXORRawCoder are the > testcases failing. All with the same error. This happens after the commit of > HADOOP-14479 > java.lang.InternalError: Invalid inputs > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.decodeImpl(Native > Method) > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.performDecodeImpl(NativeXORRawDecoder.java:44) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:58) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:74) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:105) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:163) > at > org.apache.hadoop.io.erasurecode.coder.ErasureDecodingStep.performCoding(ErasureDecodingStep.java:54) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performCodingStep(TestErasureCoderBase.java:126) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:95) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestXORCoder.testCodingNoDirectBuffer_erasing_p0(TestXORCoder.java:51) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14616) Erasurecode XOR testcase failures
[ https://issues.apache.org/jira/browse/HADOOP-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HADOOP-14616: -- Assignee: Huafeng Wang > Erasurecode XOR testcase failures > -- > > Key: HADOOP-14616 > URL: https://issues.apache.org/jira/browse/HADOOP-14616 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: Huafeng Wang > Attachments: HADOOP-14616.001.patch > > > TestXORCoder, TestXORRawCoderInteroperable2, TestNativeXORRawCoder are the > testcases failing. All with the same error. This happens after the commit of > HADOOP-14479 > java.lang.InternalError: Invalid inputs > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.decodeImpl(Native > Method) > at > org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.performDecodeImpl(NativeXORRawDecoder.java:44) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:58) > at > org.apache.hadoop.io.erasurecode.rawcoder.AbstractNativeRawDecoder.doDecode(AbstractNativeRawDecoder.java:74) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:105) > at > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:163) > at > org.apache.hadoop.io.erasurecode.coder.ErasureDecodingStep.performCoding(ErasureDecodingStep.java:54) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performCodingStep(TestErasureCoderBase.java:126) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:95) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestXORCoder.testCodingNoDirectBuffer_erasing_p0(TestXORCoder.java:51) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14869) Upgrade apache kerby verion to v1.0.1
[ https://issues.apache.org/jira/browse/HADOOP-14869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-14869: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-beta1 Status: Resolved (was: Patch Available) Committed to branch 3.0 and trunk. Thanks [~zhouwei] for the contribution! > Upgrade apache kerby verion to v1.0.1 > - > > Key: HADOOP-14869 > URL: https://issues.apache.org/jira/browse/HADOOP-14869 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei Zhou >Assignee: Wei Zhou > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14869.01.patch > > > Kerby v1.0.1 released, it contains some important bug fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14869) Upgrade apache kerby verion to v1.0.1
[ https://issues.apache.org/jira/browse/HADOOP-14869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16167745#comment-16167745 ] Kai Zheng commented on HADOOP-14869: Thanks [~zhouwei] for this. Yes we should use the latest Kerby version for the important fixes. +1. > Upgrade apache kerby verion to v1.0.1 > - > > Key: HADOOP-14869 > URL: https://issues.apache.org/jira/browse/HADOOP-14869 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei Zhou >Assignee: Wei Zhou > Attachments: HADOOP-14869.01.patch > > > Kerby v1.0.1 released, it contains some important bug fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14866) Backport implementation of parallel block copy in Distcp to hadoop 2.8
[ https://issues.apache.org/jira/browse/HADOOP-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164208#comment-16164208 ] Kai Zheng commented on HADOOP-14866: The backport of the nice enhancement and optimization for DistCp to 2.8.x sounds good to me. [~djp] and [~yzhangal], would you share your thoughts? Thanks! > Backport implementation of parallel block copy in Distcp to hadoop 2.8 > -- > > Key: HADOOP-14866 > URL: https://issues.apache.org/jira/browse/HADOOP-14866 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Reporter: Huafeng Wang >Assignee: Huafeng Wang > Fix For: 2.8.2 > > Attachments: HADOOP-14866.001.branch2.8.2.patch > > > The implementation of parallel block copy in Distcp targets to version 2.9. > It would be great to have this feature in version 2.8. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156312#comment-16156312
]
Kai Zheng commented on HADOOP-12862:
Sorry for the late, pretty busy with EC related issues.
The issue was well documented in the description and I totally agree. The patch
looks good overall, with some comments:
1. {{setSslConf}} could be => {{loadSslConf}}.
2. Writing some plain password in configuration file should be discouraged, if
allowed; could we go for the password from reading the file first?
{code}
+ hadoop.security.group.mapping.ldap.ssl.keystore.password
+
+
+Password for LDAP SSL keystore. If this value is empty, Hadoop will attempt
+to read the password from the file in
+hadoop.security.group.mapping.ldap.ssl.keystore.password.file.
+
+
{code}
3. Lots of _empty_ default values defined here. If we don't really appreciate
or favor the *empty* value at all, I guess we could avoid the overhead of
defining them at all.
{code}
public static final String LDAP_KEYSTORE_PASSWORD_DEFAULT = "";
..
public static final String LDAP_KEYSTORE_PASSWORD_FILE_DEFAULT = "";
..
+ /*
+ * Password for the keystore
+ */
+ public static final String LDAP_TRUSTSTORE_PASSWORD_KEY =
+ LDAP_CONFIG_PREFIX +".ssl.truststore.password";
+ public static final String LDAP_TRUSTSTORE_PASSWORD_DEFAULT = "";
+
+ public static final String LDAP_TRUSTSTORE_PASSWORD_FILE_KEY =
+ LDAP_TRUSTSTORE_PASSWORD_KEY + ".file";
+ public static final String LDAP_TRUSTSTORE_PASSWORD_FILE_DEFAULT = "";
+
{code}
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16152022#comment-16152022 ] Kai Zheng commented on HADOOP-14217: [~yufeldman] just adding you as a Hadoop contributor, assigned this to you. Now you should be able to play with this issue now. Enjoy! > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/oss >Reporter: Genmao Yu >Assignee: Yuliya Feldman > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HADOOP-14217: -- Assignee: Yuliya Feldman > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/oss >Reporter: Genmao Yu >Assignee: Yuliya Feldman > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14649) Update aliyun-sdk-oss version to 2.8.1
[ https://issues.apache.org/jira/browse/HADOOP-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137780#comment-16137780 ] Kai Zheng commented on HADOOP-14649: In annual leave and vacation, email response will be delayed. > Update aliyun-sdk-oss version to 2.8.1 > -- > > Key: HADOOP-14649 > URL: https://issues.apache.org/jira/browse/HADOOP-14649 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang >Assignee: Genmao Yu > Attachments: HADOOP-14649.000.patch > > > Update the dependency > com.aliyun.oss:aliyun-sdk-oss:2.4.1 > to the latest (2.8.1). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136955#comment-16136955
]
Kai Zheng commented on HADOOP-12862:
Thanks for this, [~jojochuang]. Will look at this later if not urgent, am in a
vacation.
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-12862:
---
Comment: was deleted
(was: In annual leave and vacation, email response will be delayed. For SSM and
Hadoop 3.0 related please contact with Wei Zhou; for benchmark with NSG
related, please contact with Shunyang; for HAS related, Jiajia.
)
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14194) Aliyun OSS should not use empty endpoint as default
[
https://issues.apache.org/jira/browse/HADOOP-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136934#comment-16136934
]
Kai Zheng commented on HADOOP-14194:
Yes, my missed +1 on this.
> Aliyun OSS should not use empty endpoint as default
> ---
>
> Key: HADOOP-14194
> URL: https://issues.apache.org/jira/browse/HADOOP-14194
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Reporter: Mingliang Liu
>Assignee: Genmao Yu
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14194.000.patch, HADOOP-14194.001.patch
>
>
> In {{AliyunOSSFileSystemStore::initialize()}}, it retrieves the endPoint and
> using empty string as a default value.
> {code}
> String endPoint = conf.getTrimmed(ENDPOINT_KEY, "");
> {code}
> The plain value without validation is passed to OSSClient. If the endPoint is
> not provided (empty string) or the endPoint is not valid, users will get
> exception from Aliyun OSS sdk with raw exception message like:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected
> authority at index 8: https://
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:359)
> at com.aliyun.oss.OSSClient.setEndpoint(OSSClient.java:313)
> at com.aliyun.oss.OSSClient.(OSSClient.java:297)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:134)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:272)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSTestUtils.createTestFileSystem(AliyunOSSTestUtils.java:63)
> at
> org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.setUp(TestAliyunOSSFileSystemContract.java:47)
> at junit.framework.TestCase.runBare(TestCase.java:139)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> Caused by: java.net.URISyntaxException: Expected authority at index 8:
> https://
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.failExpecting(URI.java:2854)
> at java.net.URI$Parser.parseHierarchical(URI.java:3102)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:357)
> {code}
> Let's check endPoint is not null or empty, catch the IllegalArgumentException
> and log it, wrapping the exception with clearer message stating the
> misconfiguration in endpoint or credentials.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136263#comment-16136263
]
Kai Zheng commented on HADOOP-12862:
In annual leave and vacation, email response will be delayed. For SSM and
Hadoop 3.0 related please contact with Wei Zhou; for benchmark with NSG
related, please contact with Shunyang; for HAS related, Jiajia.
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14194) Aliyun OSS should not use empty endpoint as default
[
https://issues.apache.org/jira/browse/HADOOP-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-14194:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks [~uncleGen] for the contribution!
> Aliyun OSS should not use empty endpoint as default
> ---
>
> Key: HADOOP-14194
> URL: https://issues.apache.org/jira/browse/HADOOP-14194
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Reporter: Mingliang Liu
>Assignee: Genmao Yu
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14194.000.patch, HADOOP-14194.001.patch
>
>
> In {{AliyunOSSFileSystemStore::initialize()}}, it retrieves the endPoint and
> using empty string as a default value.
> {code}
> String endPoint = conf.getTrimmed(ENDPOINT_KEY, "");
> {code}
> The plain value without validation is passed to OSSClient. If the endPoint is
> not provided (empty string) or the endPoint is not valid, users will get
> exception from Aliyun OSS sdk with raw exception message like:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected
> authority at index 8: https://
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:359)
> at com.aliyun.oss.OSSClient.setEndpoint(OSSClient.java:313)
> at com.aliyun.oss.OSSClient.(OSSClient.java:297)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:134)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:272)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSTestUtils.createTestFileSystem(AliyunOSSTestUtils.java:63)
> at
> org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.setUp(TestAliyunOSSFileSystemContract.java:47)
> at junit.framework.TestCase.runBare(TestCase.java:139)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> Caused by: java.net.URISyntaxException: Expected authority at index 8:
> https://
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.failExpecting(URI.java:2854)
> at java.net.URI$Parser.parseHierarchical(URI.java:3102)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:357)
> {code}
> Let's check endPoint is not null or empty, catch the IllegalArgumentException
> and log it, wrapping the exception with clearer message stating the
> misconfiguration in endpoint or credentials.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14194) Aliyun OSS should not use empty endpoint as default
[
https://issues.apache.org/jira/browse/HADOOP-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134646#comment-16134646
]
Kai Zheng commented on HADOOP-14194:
The change looks good. Looks like the two minor check styles should be
addressed. Genmao could you update? Thanks!
> Aliyun OSS should not use empty endpoint as default
> ---
>
> Key: HADOOP-14194
> URL: https://issues.apache.org/jira/browse/HADOOP-14194
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Reporter: Mingliang Liu
>Assignee: Genmao Yu
> Attachments: HADOOP-14194.000.patch, HADOOP-14194.001.patch
>
>
> In {{AliyunOSSFileSystemStore::initialize()}}, it retrieves the endPoint and
> using empty string as a default value.
> {code}
> String endPoint = conf.getTrimmed(ENDPOINT_KEY, "");
> {code}
> The plain value without validation is passed to OSSClient. If the endPoint is
> not provided (empty string) or the endPoint is not valid, users will get
> exception from Aliyun OSS sdk with raw exception message like:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected
> authority at index 8: https://
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:359)
> at com.aliyun.oss.OSSClient.setEndpoint(OSSClient.java:313)
> at com.aliyun.oss.OSSClient.(OSSClient.java:297)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:134)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:272)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSTestUtils.createTestFileSystem(AliyunOSSTestUtils.java:63)
> at
> org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.setUp(TestAliyunOSSFileSystemContract.java:47)
> at junit.framework.TestCase.runBare(TestCase.java:139)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> Caused by: java.net.URISyntaxException: Expected authority at index 8:
> https://
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.failExpecting(URI.java:2854)
> at java.net.URI$Parser.parseHierarchical(URI.java:3102)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:357)
> {code}
> Let's check endPoint is not null or empty, catch the IllegalArgumentException
> and log it, wrapping the exception with clearer message stating the
> misconfiguration in endpoint or credentials.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14479) Erasurecode testcase failures with native enabled
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133905#comment-16133905 ] Kai Zheng commented on HADOOP-14479: Ah, HDFS-11066 looks good enough. Thanks Andrew! > Erasurecode testcase failures with native enabled > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-14479.001.patch > > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14479) Erasurecode testcase failures with native enabled
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131950#comment-16131950 ] Kai Zheng commented on HADOOP-14479: Thanks [~andrew.wang] for the reminder. Yes it's scheduled and we'll do it. If it can't be made in BETA1, we'll give some thorough runs against latest ISA-L manually. > Erasurecode testcase failures with native enabled > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-14479.001.patch > > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14649) Update aliyun-sdk-oss version to 2.8.0
[ https://issues.apache.org/jira/browse/HADOOP-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125549#comment-16125549 ] Kai Zheng commented on HADOOP-14649: Hi [~rchiang], Thanks for the ping. I had an offline sync with some folk and hope this can be get touched soon. Wondering how urgent this is to your side. > Update aliyun-sdk-oss version to 2.8.0 > -- > > Key: HADOOP-14649 > URL: https://issues.apache.org/jira/browse/HADOOP-14649 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang > > Update the dependency > com.aliyun.oss:aliyun-sdk-oss:2.4.1 > to the latest (2.8.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14743) CompositeGroupsMapping should not swallow exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119321#comment-16119321
]
Kai Zheng commented on HADOOP-14743:
+1 for the nice change. Thanks!
> CompositeGroupsMapping should not swallow exceptions
>
>
> Key: HADOOP-14743
> URL: https://issues.apache.org/jira/browse/HADOOP-14743
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14743.001.patch, HADOOP-14743.002.patch
>
>
> {code:title=CompositeGroupsMapping#getGroups}
>for (GroupMappingServiceProvider provider : providersList) {
> try {
> groups = provider.getGroups(user);
> } catch (Exception e) {
> //LOG.warn("Exception trying to get groups for user " + user, e);
>
> }
> if (groups != null && ! groups.isEmpty()) {
> groupSet.addAll(groups);
> if (!combined) break;
> }
> }
> {code}
> If anything fails inside the underlying groups mapping service provider,
> there's no way to tell what went wrong.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14649) Update aliyun-sdk-oss version to 2.8.0
[ https://issues.apache.org/jira/browse/HADOOP-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090877#comment-16090877 ] Kai Zheng commented on HADOOP-14649: Ping [~uncleGen]. Sounds good to get this done. > Update aliyun-sdk-oss version to 2.8.0 > -- > > Key: HADOOP-14649 > URL: https://issues.apache.org/jira/browse/HADOOP-14649 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang > > Update the dependency > com.aliyun.oss:aliyun-sdk-oss:2.4.1 > to the latest (2.8.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14649) Update aliyun-sdk-oss version to 2.8.0
[ https://issues.apache.org/jira/browse/HADOOP-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HADOOP-14649: -- Assignee: (was: Kai Zheng) > Update aliyun-sdk-oss version to 2.8.0 > -- > > Key: HADOOP-14649 > URL: https://issues.apache.org/jira/browse/HADOOP-14649 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang > > Update the dependency > com.aliyun.oss:aliyun-sdk-oss:2.4.1 > to the latest (2.8.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HADOOP-12756: -- Assignee: mingfei.shi (was: Kai Zheng) > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: HADOOP-12756 >Reporter: shimingfei >Assignee: mingfei.shi > Fix For: HADOOP-12756, 3.0.0-alpha2 > > Attachments: Aliyun-OSS-integration.pdf, > Aliyun-OSS-integration-v2.pdf, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, > HADOOP-12756.007.patch, HADOOP-12756.008.patch, HADOOP-12756.009.patch, > HADOOP-12756.010.patch, HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14593) Improve TestCodecRawCoderMapping & TestRSErasureCoder to support native RS coder
[ https://issues.apache.org/jira/browse/HADOOP-14593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-14593: --- Resolution: Duplicate Status: Resolved (was: Patch Available) This was done in HADOOP-14479. > Improve TestCodecRawCoderMapping & TestRSErasureCoder to support native RS > coder > > > Key: HADOOP-14593 > URL: https://issues.apache.org/jira/browse/HADOOP-14593 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-14593.001.patch, HADOOP-14593.002.patch > > > Currently this unit test will fail due to not support the native RS coder. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14479) Erasurecode testcase failures with native enabled
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-14479: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha4 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~Sammi] for the contribution! > Erasurecode testcase failures with native enabled > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-14479.001.patch > > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14479) Erasurecode testcase failures with native enabled
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067656#comment-16067656 ] Kai Zheng commented on HADOOP-14479: Thanks [~Sammi] a lot for digging into these failure cases and providing the patch! This clarified these failures have nothing to do with ISA-L version update. The patch LGTM and +1. Will commit it shortly. > Erasurecode testcase failures with native enabled > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-14479.001.patch > > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14479) Erasurecode testcase failures with ISA-L
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055020#comment-16055020 ] Kai Zheng commented on HADOOP-14479: Hmm, I thought the test gap was majorly caused by the precommit missing ISA-L building. Quite some time ago it was done in HADOOP-12626 but removed in HADOOP-13342. I thought we should bring it back and re-enable the ISA-L building & tests in precommit. Andrew, do you think so? If sounds good, we can fire a new issue to do it. Thanks! > Erasurecode testcase failures with ISA-L > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14479) Erasurecode testcase failures with ISA-L
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055010#comment-16055010 ] Kai Zheng commented on HADOOP-14479: Hi [~andrew.wang], bq. We're supposed to be running the ISA-L tests as part of precommit, so I'm not sure what happened here. Seems like there's a test gap for ISA-L vs. the Java coder (time to revisit HDFS-11066?) I agree, it would be good to fill the test gap. Let's revisit HDFS-11066. Thanks! > Erasurecode testcase failures with ISA-L > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14479) Erasurecode testcase failures with ISA-L
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053119#comment-16053119 ] Kai Zheng commented on HADOOP-14479: Per offline [~Sammi] will take this. Sammi, hope you're doing great. Is this caused by different environment like Ubuntu or by the upgrade of Intel ISA-L? Did you get some chance to verify this first? It would be great if you could update on this when any progress. Thanks! > Erasurecode testcase failures with ISA-L > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14479) Erasurecode testcase failures with ISA-L
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HADOOP-14479: -- Assignee: SammiChen > Erasurecode testcase failures with ISA-L > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14146) KerberosAuthenticationHandler should authenticate with SPN in AP-REQ
[ https://issues.apache.org/jira/browse/HADOOP-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049887#comment-16049887 ] Kai Zheng commented on HADOOP-14146: Thanks [~daryn] for the nice update! I still have some concern over the large portion of ASN.1 & DER decoding codes because it'll be a maintain burden for the project. However that utility is quite separate and we can refine the part later. With the added good tests, and based on my trust of you in the domain, I'm good to provide my +1. You might wait a couple of days and then commit it if no other comments. Thanks! > KerberosAuthenticationHandler should authenticate with SPN in AP-REQ > > > Key: HADOOP-14146 > URL: https://issues.apache.org/jira/browse/HADOOP-14146 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.5.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-14146.1.patch, HADOOP-14146.2.patch, > HADOOP-14146.3.patch, HADOOP-14146.patch > > > Many attempts (HADOOP-10158, HADOOP-11628, HADOOP-13565) have tried to add > multiple SPN host and/or realm support to spnego authentication. The basic > problem is the server tries to guess and/or brute force what SPN the client > used. The server should just decode the SPN from the AP-REQ. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14146) KerberosAuthenticationHandler should authenticate with SPN in AP-REQ
[
https://issues.apache.org/jira/browse/HADOOP-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040025#comment-16040025
]
Kai Zheng commented on HADOOP-14146:
In addition to above comments, some more:
1. Ref. below, NT_GSS_KRB5_PRINCIPAL could be NT_GSS_KRB5_PRINCIPAL_OID.
{code}
+ public static final Oid GSS_SPNEGO_MECH_OID =
+ getNumericOidInstance("1.3.6.1.5.5.2");
+ public static final Oid GSS_KRB5_MECH_OID =
+ getNumericOidInstance("1.2.840.113554.1.2.2");
+ public static final Oid NT_GSS_KRB5_PRINCIPAL =
+ getNumericOidInstance("1.2.840.113554.1.2.2.1");
{code}
2. Ref. below, the message would be more specific like "Invalid server
principal {} decoded from client request".
{code}
+final String serverPrincipal =
+KerberosUtil.getTokenServerName(clientToken);
+if (!serverPrincipal.startsWith("HTTP/")) {
+ throw new IllegalArgumentException(
+ "Invalid server principal: " + serverPrincipal);
+}
{code}
3. You get rid of the login check for each HTTP server principal listed in the
keytab, instead, you put them into the server subject directly or manually. Is
it possible a server principal expired or invalid at the time?
{code}
- for (String spnegoPrincipal : spnegoPrincipals) {
-LOG.info("Login using keytab {}, for principal {}",
-keytab, spnegoPrincipal);
-final KerberosConfiguration kerberosConfiguration =
-new KerberosConfiguration(keytab, spnegoPrincipal);
-final LoginContext loginContext =
-new LoginContext("", serverSubject, null, kerberosConfiguration);
-try {
- loginContext.login();
-} catch (LoginException le) {
- LOG.warn("Failed to login as [{}]", spnegoPrincipal, le);
- throw new AuthenticationException(le);
-}
-loginContexts.add(loginContext);
{code}
4. Besides you might want to call {{KerberosUtil.hasKerberosKeyTab}} with the
placed keytab instance in the subject, wonder how the instance would be used in
the subsequent SPNEGO authenticating to the client token. Could you help
explain some bit for me or as comment for the code? Thanks!
{code}
+ KeyTab keytabInstance = KeyTab.getInstance(keytabFile);
+ serverSubject.getPrivateCredentials().add(keytabInstance);
{code}
5. Is is a good chance to move the follow block to somewhere like
{{KerberosUtil}}?
{code}
/* Return the OS login module class name */
private static String getOSLoginModuleName() {
if (IBM_JAVA) {
if (windows) {
return is64Bit ? "com.ibm.security.auth.module.Win64LoginModule"
: "com.ibm.security.auth.module.NTLoginModule";
} else if (aix) {
return is64Bit ? "com.ibm.security.auth.module.AIX64LoginModule"
: "com.ibm.security.auth.module.AIXLoginModule";
} else {
return "com.ibm.security.auth.module.LinuxLoginModule";
}
} else {
return windows ? "com.sun.security.auth.module.NTLoginModule"
: "com.sun.security.auth.module.UnixLoginModule";
}
}
{code}
> KerberosAuthenticationHandler should authenticate with SPN in AP-REQ
>
>
> Key: HADOOP-14146
> URL: https://issues.apache.org/jira/browse/HADOOP-14146
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HADOOP-14146.1.patch, HADOOP-14146.2.patch,
> HADOOP-14146.patch
>
>
> Many attempts (HADOOP-10158, HADOOP-11628, HADOOP-13565) have tried to add
> multiple SPN host and/or realm support to spnego authentication. The basic
> problem is the server tries to guess and/or brute force what SPN the client
> used. The server should just decode the SPN from the AP-REQ.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14426) Upgrade Kerby version from 1.0.0-RC2 to 1.0.0
[ https://issues.apache.org/jira/browse/HADOOP-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014898#comment-16014898 ] Kai Zheng commented on HADOOP-14426: Thanks [~jiajia] and [~jojochuang]! Is there any existing issue/workaround related to Kerby and MiniKDC we can clear by this chance as well? The change here itself is OK and I'm +1. > Upgrade Kerby version from 1.0.0-RC2 to 1.0.0 > - > > Key: HADOOP-14426 > URL: https://issues.apache.org/jira/browse/HADOOP-14426 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Jiajia Li >Assignee: Jiajia Li > Attachments: HADOOP-14426-001.patch > > > Apache Kerby 1.0.0 with some bug fixes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14146) KerberosAuthenticationHandler should authenticate with SPN in AP-REQ
[ https://issues.apache.org/jira/browse/HADOOP-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014891#comment-16014891 ] Kai Zheng commented on HADOOP-14146: I'm in travelling and looking for the time for a full detailed review. Should happen in a few days. Meanwhile, could you address [~shahrs87]'s above comment? Thanks! > KerberosAuthenticationHandler should authenticate with SPN in AP-REQ > > > Key: HADOOP-14146 > URL: https://issues.apache.org/jira/browse/HADOOP-14146 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.5.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-14146.1.patch, HADOOP-14146.patch > > > Many attempts (HADOOP-10158, HADOOP-11628, HADOOP-13565) have tried to add > multiple SPN host and/or realm support to spnego authentication. The basic > problem is the server tries to guess and/or brute force what SPN the client > used. The server should just decode the SPN from the AP-REQ. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14397) Pull up the builder pattern to FileSystem and add AbstractContractCreateTest for it
[
https://issues.apache.org/jira/browse/HADOOP-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002442#comment-16002442
]
Kai Zheng commented on HADOOP-14397:
[~ste...@apache.org],
Should the following from your comments elsewhere be handled here? Thanks!
{quote}
2. There are only tests for HDFS and local, neither of them perfect. Proposed:
move to AbstractContractCreateTest, test for all filesystems, fix tests and FS
where appropriate.
3. Add more tests to generate the failure conditions implied by the updated
filesystem spec. Eg. create over a an existing file, create over a directory,
create with negative buffer size, negative block size, empty dest path, etc,
etc.
This will clarify when precondition checks are made, as well as whether. For
example: should newFSDataOutputStreamBuilder() validate the path immediately?
{quote}
> Pull up the builder pattern to FileSystem and add AbstractContractCreateTest
> for it
> ---
>
> Key: HADOOP-14397
> URL: https://issues.apache.org/jira/browse/HADOOP-14397
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, fs, hdfs-client
>Affects Versions: 2.9.0
>Reporter: Lei (Eddy) Xu
>
> After reach the stability of the Builder APIs, we should promote the API from
> {{DistributedFileSystem}} to {{FileSystem}}, and add necessary contract tests
> to cover the API for all file systems.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978280#comment-15978280
]
Kai Zheng commented on HADOOP-13200:
As now codecs are dynamically identified and loaded along with its raw coder
implementations, for users to use them when define erasure coding policies, a
CLI cmd would be needed to list these codecs. The cmd would simply list the
keys of the map maintained by {{CodecRegistry}}. I suggest we do this
separately.
> Seeking a better approach allowing to customize and configure erasure coders
>
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Tim Yao
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975922#comment-15975922
]
Kai Zheng commented on HADOOP-13200:
Thanks [~timmyyao] for working thru this and updating the patches. LGTM and my
+1.
I'm happy we can get rid of static things like follows,
{code}
- // Default coders for each codec names.
- public static final Map DEFAULT_CODERS_MAP = ImmutableMap.of(
- "rs", IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT,
- "rs-legacy", IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODERS_DEFAULT,
- "xor",IO_ERASURECODE_CODEC_XOR_RAWCODERS_DEFAULT
- );
{code}
by registries like the following, in In
hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderFactory:
{code}
+org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory
+org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory
+org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory
+org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactoryLegacy
+org.apache.hadoop.io.erasurecode.rawcoder.XORRawErasureCoderFactory
{code}
[~andrew.wang], [~jojochuang] maybe you also want to take a look? If no other
comments in a couple of days, I will commit it.
> Seeking a better approach allowing to customize and configure erasure coders
>
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Tim Yao
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12911) Upgrade Hadoop MiniKDC with Kerby
[ https://issues.apache.org/jira/browse/HADOOP-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975877#comment-15975877 ] Kai Zheng commented on HADOOP-12911: Right, the both issues should have been closed when this was in, as they're simple fork to trigger tests in HDFS and YARN. Will close them as duplicates. > Upgrade Hadoop MiniKDC with Kerby > - > > Key: HADOOP-12911 > URL: https://issues.apache.org/jira/browse/HADOOP-12911 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Reporter: Jiajia Li >Assignee: Jiajia Li > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12911-v1.patch, HADOOP-12911-v2.patch, > HADOOP-12911-v3.patch, HADOOP-12911-v4.patch, HADOOP-12911-v5.patch, > HADOOP-12911-v6.patch, HADOOP-12911-v7.patch, HaDOOP-12911-v8.patch > > > As discussed in the mailing list, we’d like to introduce Apache Kerby into > Hadoop. Initially it’s good to start with upgrading Hadoop MiniKDC with Kerby > offerings. Apache Kerby (https://github.com/apache/directory-kerby), as an > Apache Directory sub project, is a Java Kerberos binding. It provides a > SimpleKDC server that borrowed ideas from MiniKDC and implemented all the > facilities existing in MiniKDC. Currently MiniKDC depends on the old Kerberos > implementation in Directory Server project, but the implementation is stopped > being maintained. Directory community has a plan to replace the > implementation using Kerby. MiniKDC can use Kerby SimpleKDC directly to avoid > depending on the full of Directory project. Kerby also provides nice identity > backends such as the lightweight memory based one and the very simple json > one for easy development and test environments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12911) Upgrade Hadoop MiniKDC with Kerby
[
https://issues.apache.org/jira/browse/HADOOP-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975851#comment-15975851
]
Kai Zheng commented on HADOOP-12911:
Hi Andrew,
Did you mean these in hadoop-auth module? These were added back by HADOOP-12082
for the purpose of testing LdapAuthenticationHandler, please see [the
comment|https://issues.apache.org/jira/browse/HADOOP-12082?focusedCommentId=15567566&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567566]
{noformat}
org.apache.directory.server
apacheds-core
${apacheds.version}
test
org.apache.directory.server
apacheds-protocol-ldap
${apacheds.version}
test
org.apache.directory.server
apacheds-ldif-partition
${apacheds.version}
test
org.apache.directory.api
api-ldap-codec-core
${ldap-api.version}
test
org.apache.directory.api
api-ldap-model
${ldap-api.version}
test
org.apache.directory.server
apacheds-server-integ
${apacheds.version}
test
...
{noformat}
> Upgrade Hadoop MiniKDC with Kerby
> -
>
> Key: HADOOP-12911
> URL: https://issues.apache.org/jira/browse/HADOOP-12911
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Jiajia Li
>Assignee: Jiajia Li
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12911-v1.patch, HADOOP-12911-v2.patch,
> HADOOP-12911-v3.patch, HADOOP-12911-v4.patch, HADOOP-12911-v5.patch,
> HADOOP-12911-v6.patch, HADOOP-12911-v7.patch, HaDOOP-12911-v8.patch
>
>
> As discussed in the mailing list, we’d like to introduce Apache Kerby into
> Hadoop. Initially it’s good to start with upgrading Hadoop MiniKDC with Kerby
> offerings. Apache Kerby (https://github.com/apache/directory-kerby), as an
> Apache Directory sub project, is a Java Kerberos binding. It provides a
> SimpleKDC server that borrowed ideas from MiniKDC and implemented all the
> facilities existing in MiniKDC. Currently MiniKDC depends on the old Kerberos
> implementation in Directory Server project, but the implementation is stopped
> being maintained. Directory community has a plan to replace the
> implementation using Kerby. MiniKDC can use Kerby SimpleKDC directly to avoid
> depending on the full of Directory project. Kerby also provides nice identity
> backends such as the lightweight memory based one and the very simple json
> one for easy development and test environments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967028#comment-15967028 ] Kai Zheng commented on HADOOP-13200: Hi Tim, Thanks for your quick work on this! The patch looks not in good form. You need also a rebase. > Seeking a better approach allowing to customize and configure erasure coders > > > Key: HADOOP-13200 > URL: https://issues.apache.org/jira/browse/HADOOP-13200 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Tim Yao >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-13200.01.patch > > > This is a follow-on task for HADOOP-13010 as discussed over there. There may > be some better approach allowing to customize and configure erasure coders > than the current having raw coder factory, as [~cmccabe] suggested. Will copy > the relevant comments here to continue the discussion. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14146) KerberosAuthenticationHandler should authenticate with SPN in AP-REQ
[ https://issues.apache.org/jira/browse/HADOOP-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967025#comment-15967025 ] Kai Zheng commented on HADOOP-14146: Daryn sorry for the late response. I'm OK with the current approach to proceed, though I'd take some time later to explore more and improve it when sounds good. > KerberosAuthenticationHandler should authenticate with SPN in AP-REQ > > > Key: HADOOP-14146 > URL: https://issues.apache.org/jira/browse/HADOOP-14146 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.5.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-14146.1.patch, HADOOP-14146.patch > > > Many attempts (HADOOP-10158, HADOOP-11628, HADOOP-13565) have tried to add > multiple SPN host and/or realm support to spnego authentication. The basic > problem is the server tries to guess and/or brute force what SPN the client > used. The server should just decode the SPN from the AP-REQ. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13665) Erasure Coding codec should support fallback coder
[
https://issues.apache.org/jira/browse/HADOOP-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963918#comment-15963918
]
Kai Zheng commented on HADOOP-13665:
Thanks [~lewuathe] for the update! The latest patch LGTM. Guess [~jojochuang]
will take another look and commit it. Thanks.
> Erasure Coding codec should support fallback coder
> --
>
> Key: HADOOP-13665
> URL: https://issues.apache.org/jira/browse/HADOOP-13665
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io
>Reporter: Wei-Chiu Chuang
>Assignee: Kai Sasaki
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13665.01.patch, HADOOP-13665.02.patch,
> HADOOP-13665.03.patch, HADOOP-13665.04.patch, HADOOP-13665.05.patch,
> HADOOP-13665.06.patch, HADOOP-13665.07.patch, HADOOP-13665.08.patch,
> HADOOP-13665.09.patch, HADOOP-13665.10.patch, HADOOP-13665.11.patch,
> HADOOP-13665.12.patch
>
>
> The current EC codec supports a single coder only (by default pure Java
> implementation). If the native coder is specified but is unavailable, it
> should fallback to pure Java implementation.
> One possible solution is to follow the convention of existing Hadoop native
> codec, such as transport encryption (see {{CryptoCodec.java}}). It supports
> fallback by specifying two or multiple coders as the value of property, and
> loads coders in order.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13665) Erasure Coding codec should support fallback coder
[
https://issues.apache.org/jira/browse/HADOOP-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958477#comment-15958477
]
Kai Zheng commented on HADOOP-13665:
Thanks [~lewuathe] for the update and [~jojochuang] for the comments! It looks
close now. I like the idea of {{createRawEncoderWithFallback}} a lot.
bq. would it make sense to update IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT to
be orgNativeRSRawErasureCoderFactory,org.RSRawErasureCoderFactory?
Yes, we should configure like this, instead of the way as below:
{code}
+ public static final String IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT =
+ "rs,rs-legacy";
{code}
[~lewuathe], for your understanding, *rs* and *rs-legacy* are different codecs
so we'd better not mix them.
> Erasure Coding codec should support fallback coder
> --
>
> Key: HADOOP-13665
> URL: https://issues.apache.org/jira/browse/HADOOP-13665
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io
>Reporter: Wei-Chiu Chuang
>Assignee: Kai Sasaki
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13665.01.patch, HADOOP-13665.02.patch,
> HADOOP-13665.03.patch, HADOOP-13665.04.patch, HADOOP-13665.05.patch,
> HADOOP-13665.06.patch, HADOOP-13665.07.patch, HADOOP-13665.08.patch,
> HADOOP-13665.09.patch
>
>
> The current EC codec supports a single coder only (by default pure Java
> implementation). If the native coder is specified but is unavailable, it
> should fallback to pure Java implementation.
> One possible solution is to follow the convention of existing Hadoop native
> codec, such as transport encryption (see {{CryptoCodec.java}}). It supports
> fallback by specifying two or multiple coders as the value of property, and
> loads coders in order.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950473#comment-15950473 ] Kai Zheng commented on HADOOP-13200: Thanks Andrew for the confirm and assigned to [~timmyyao] for the patch per offline discussion. > Seeking a better approach allowing to customize and configure erasure coders > > > Key: HADOOP-13200 > URL: https://issues.apache.org/jira/browse/HADOOP-13200 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Tim Yao >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > > This is a follow-on task for HADOOP-13010 as discussed over there. There may > be some better approach allowing to customize and configure erasure coders > than the current having raw coder factory, as [~cmccabe] suggested. Will copy > the relevant comments here to continue the discussion. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HADOOP-13200: -- Assignee: Tim Yao (was: Kai Zheng) > Seeking a better approach allowing to customize and configure erasure coders > > > Key: HADOOP-13200 > URL: https://issues.apache.org/jira/browse/HADOOP-13200 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Tim Yao >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > > This is a follow-on task for HADOOP-13010 as discussed over there. There may > be some better approach allowing to customize and configure erasure coders > than the current having raw coder factory, as [~cmccabe] suggested. Will copy > the relevant comments here to continue the discussion. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
