[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255983#comment-15255983 ] Ling Zhou commented on HADOOP-12756: Hi Liu Yi, Ali-oss client does rely on a higher version of httpclient, it is required by Aliyun OSS SDK. Currently it uses 4.4 version, and hadoop uses 4.2.5 version which does not work for OSS SDK. And about the name "oss", we will change it according to your suggestion. Hi Steve Loughran, We have read the filesystem specification docs, this implementation is similar to S3A. So operations like rename and delete are still not atomic. Aliyun OSS is much like general object storage system except it is strong consistent. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: shimingfei >Assignee: shimingfei > Attachments: 0001-OSS-filesystem-integration-with-Hadoop.patch, HCFS > User manual.md, OSS integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255941#comment-15255941 ] Yi Liu commented on HADOOP-12756: - Also the name "oss" is abbreviation of Object Store Service, it's too generic, I think we need to change the name to ali-oss or some other names which other people can understand what it is at first glance. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: shimingfei >Assignee: shimingfei > Attachments: 0001-OSS-filesystem-integration-with-Hadoop.patch, HCFS > User manual.md, OSS integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255929#comment-15255929 ] Yi Liu commented on HADOOP-12756: - Thanks for mingfei and Lei for the work. Hi [~ste...@apache.org] and [~cnauroth], regarding testability, they have talked with me offline, Aliyun created an account for the test and they retained the account for Hadoop, they wanted to pass the username/password through "-D" in mvn command. So the basic functionalities could be verified by unit tests. Does this make sense to you? Mingfei and Lei: About the ali-oss client, does it rely on a different version of httpclient? Could we use the version which hadoop is using? I will post my detailed comments later. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: shimingfei >Assignee: shimingfei > Attachments: 0001-OSS-filesystem-integration-with-Hadoop.patch, HCFS > User manual.md, OSS integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12579) Deprecate and remove WriteableRPCEngine
[
https://issues.apache.org/jira/browse/HADOOP-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-12579:
---
Attachment: HADOOP-12579-v7.patch
Fixed some failures.
> Deprecate and remove WriteableRPCEngine
> ---
>
> Key: HADOOP-12579
> URL: https://issues.apache.org/jira/browse/HADOOP-12579
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Haohui Mai
> Attachments: HADOOP-12579-v1.patch, HADOOP-12579-v3.patch,
> HADOOP-12579-v4.patch, HADOOP-12579-v5.patch, HADOOP-12579-v6.patch,
> HADOOP-12579-v7.patch
>
>
> The {{WriteableRPCEninge}} depends on Java's serialization mechanisms for RPC
> requests. Without proper checks, it has be shown that it can lead to security
> vulnerabilities such as remote code execution (e.g., COLLECTIONS-580,
> HADOOP-12577).
> The current implementation has migrated from {{WriteableRPCEngine}} to
> {{ProtobufRPCEngine}} now. This jira proposes to deprecate
> {{WriteableRPCEngine}} in branch-2 and to remove it in trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-13057) Async IPC server support
[
https://issues.apache.org/jira/browse/HADOOP-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255911#comment-15255911
]
He Tianyi commented on HADOOP-13057:
BTW, would be really appreciate to see further updates on HADOOP-11552.
> Async IPC server support
>
>
> Key: HADOOP-13057
> URL: https://issues.apache.org/jira/browse/HADOOP-13057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>
> On some application, server may run out of handlers when performing many
> blocking I/O operations during processing of each call (e.g. calling another
> service, etc.). A viable solution is increasing number of handlers.
> But this faces the problem that large amount of threads will consume much
> memory (stack, etc.), and performance issues either.
> After HADOOP-12909, work on asynchronization has been done on caller-side.
> This is a similar proposal on server-side.
> Suggesting the ability to handle requests asynchronously.
> For example, in such server, calls may return a Future object instead of
> immediate value. Then sends response to client in {{onSuccess}} or
> {{onFailed}} callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-13057) Async IPC server support
[
https://issues.apache.org/jira/browse/HADOOP-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255907#comment-15255907
]
He Tianyi commented on HADOOP-13057:
Thanks, [~sseth].
Marked as duplicate.
> Async IPC server support
>
>
> Key: HADOOP-13057
> URL: https://issues.apache.org/jira/browse/HADOOP-13057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>
> On some application, server may run out of handlers when performing many
> blocking I/O operations during processing of each call (e.g. calling another
> service, etc.). A viable solution is increasing number of handlers.
> But this faces the problem that large amount of threads will consume much
> memory (stack, etc.), and performance issues either.
> After HADOOP-12909, work on asynchronization has been done on caller-side.
> This is a similar proposal on server-side.
> Suggesting the ability to handle requests asynchronously.
> For example, in such server, calls may return a Future object instead of
> immediate value. Then sends response to client in {{onSuccess}} or
> {{onFailed}} callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-13057) Async IPC server support
[
https://issues.apache.org/jira/browse/HADOOP-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi resolved HADOOP-13057.
Resolution: Duplicate
> Async IPC server support
>
>
> Key: HADOOP-13057
> URL: https://issues.apache.org/jira/browse/HADOOP-13057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>
> On some application, server may run out of handlers when performing many
> blocking I/O operations during processing of each call (e.g. calling another
> service, etc.). A viable solution is increasing number of handlers.
> But this faces the problem that large amount of threads will consume much
> memory (stack, etc.), and performance issues either.
> After HADOOP-12909, work on asynchronization has been done on caller-side.
> This is a similar proposal on server-side.
> Suggesting the ability to handle requests asynchronously.
> For example, in such server, calls may return a Future object instead of
> immediate value. Then sends response to client in {{onSuccess}} or
> {{onFailed}} callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-13057) Async IPC server support
[
https://issues.apache.org/jira/browse/HADOOP-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255896#comment-15255896
]
Siddharth Seth commented on HADOOP-13057:
-
HADOOP-11552 targets the same.
> Async IPC server support
>
>
> Key: HADOOP-13057
> URL: https://issues.apache.org/jira/browse/HADOOP-13057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>
> On some application, server may run out of handlers when performing many
> blocking I/O operations during processing of each call (e.g. calling another
> service, etc.). A viable solution is increasing number of handlers.
> But this faces the problem that large amount of threads will consume much
> memory (stack, etc.), and performance issues either.
> After HADOOP-12909, work on asynchronization has been done on caller-side.
> This is a similar proposal on server-side.
> Suggesting the ability to handle requests asynchronously.
> For example, in such server, calls may return a Future object instead of
> immediate value. Then sends response to client in {{onSuccess}} or
> {{onFailed}} callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-13021) Hadoop swift driver unit test should use unique directory for each run
[
https://issues.apache.org/jira/browse/HADOOP-13021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He updated HADOOP-13021:
-
Attachment: HADOOP-13021.001.patch
> Hadoop swift driver unit test should use unique directory for each run
> --
>
> Key: HADOOP-13021
> URL: https://issues.apache.org/jira/browse/HADOOP-13021
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
>Affects Versions: 2.7.2
>Reporter: Chen He
>Assignee: Chen He
> Labels: unit-test
> Attachments: HADOOP-13021.001.patch
>
>
> Since all "unit test" in swift package are actually functionality test, it
> requires server's information in the core-site.xml file. However, multiple
> unit test runs on difference machines using the same core-site.xml file will
> result in some unit tests failure. For example:
> In TestSwiftFileSystemBasicOps.java
> public void testMkDir() throws Throwable {
> Path path = new Path("/test/MkDir");
> fs.mkdirs(path);
> //success then -so try a recursive operation
> fs.delete(path, true);
> }
> It is possible that machine A and B are running "mvn clean install" using
> same core-site.xml file. However, machine A run testMkDir() first and delete
> the dir, but machine B just tried to run fs.delete(path,true). It will report
> failure. This is just an example. There are many similar cases in the unit
> test sets. I would propose we use a unique dir for each unit test run instead
> of using "Path path = new Path("/test/MkDir")" for all concurrent runs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-13057) Async IPC server support
[
https://issues.apache.org/jira/browse/HADOOP-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255868#comment-15255868
]
He Tianyi commented on HADOOP-13057:
Sure, added links and description.
> Async IPC server support
>
>
> Key: HADOOP-13057
> URL: https://issues.apache.org/jira/browse/HADOOP-13057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>
> On some application, server may run out of handlers when performing many
> blocking I/O operations during processing of each call (e.g. calling another
> service, etc.). A viable solution is increasing number of handlers.
> But this faces the problem that large amount of threads will consume much
> memory (stack, etc.), and performance issues either.
> After HADOOP-12909, work on asynchronization has been done on caller-side.
> This is a similar proposal on server-side.
> Suggesting the ability to handle requests asynchronously.
> For example, in such server, calls may return a Future object instead of
> immediate value. Then sends response to client in {{onSuccess}} or
> {{onFailed}} callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-13057) Async IPC server support
[
https://issues.apache.org/jira/browse/HADOOP-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated HADOOP-13057:
---
Description:
On some application, server may run out of handlers when performing many
blocking I/O operations during processing of each call (e.g. calling another
service, etc.). A viable solution is increasing number of handlers.
But this faces the problem that large amount of threads will consume much
memory (stack, etc.), and performance issues either.
After HADOOP-12909, work on asynchronization has been done on caller-side. This
is a similar proposal on server-side.
Suggesting the ability to handle requests asynchronously.
For example, in such server, calls may return a Future object instead of
immediate value. Then sends response to client in {{onSuccess}} or {{onFailed}}
callbacks.
was:
On some application, server may run out of handlers when performing many
blocking I/O operations during processing of each call (e.g. calling another
service, etc.). A viable solution is increasing number of handlers.
But this faces the problem that large amount of threads will consume much
memory (stack, etc.), and performance issues either.
Suggesting the ability to handle requests asynchronously.
For example, in such server, calls may return a Future object instead of
immediate value. Then sends response to client in {{onSuccess}} or {{onFailed}}
callbacks.
> Async IPC server support
>
>
> Key: HADOOP-13057
> URL: https://issues.apache.org/jira/browse/HADOOP-13057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>
> On some application, server may run out of handlers when performing many
> blocking I/O operations during processing of each call (e.g. calling another
> service, etc.). A viable solution is increasing number of handlers.
> But this faces the problem that large amount of threads will consume much
> memory (stack, etc.), and performance issues either.
> After HADOOP-12909, work on asynchronization has been done on caller-side.
> This is a similar proposal on server-side.
> Suggesting the ability to handle requests asynchronously.
> For example, in such server, calls may return a Future object instead of
> immediate value. Then sends response to client in {{onSuccess}} or
> {{onFailed}} callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-13057) Async IPC server support
[
https://issues.apache.org/jira/browse/HADOOP-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255864#comment-15255864
]
Mingliang Liu commented on HADOOP-13057:
Can you refer to [HDFS-9924] and [ HADOOP-12909] and describe the difference
from here? Thanks.
> Async IPC server support
>
>
> Key: HADOOP-13057
> URL: https://issues.apache.org/jira/browse/HADOOP-13057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>
> On some application, server may run out of handlers when performing many
> blocking I/O operations during processing of each call (e.g. calling another
> service, etc.). A viable solution is increasing number of handlers.
> But this faces the problem that large amount of threads will consume much
> memory (stack, etc.), and performance issues either.
> Suggesting the ability to handle requests asynchronously.
> For example, in such server, calls may return a Future object instead of
> immediate value. Then sends response to client in {{onSuccess}} or
> {{onFailed}} callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12579) Deprecate and remove WriteableRPCEngine
[
https://issues.apache.org/jira/browse/HADOOP-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255848#comment-15255848
]
Hadoop QA commented on HADOOP-12579:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 32s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 5s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 10s
{color} | {color:red} root: patch generated 3 new + 579 unchanged - 28 fixed =
582 total (was 607) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 19s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 34s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 31s
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 33s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{colo
[jira] [Updated] (HADOOP-13057) Async IPC server support
[
https://issues.apache.org/jira/browse/HADOOP-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated HADOOP-13057:
---
Description:
On some application, server may run out of handlers when performing many
blocking I/O operations during processing of each call (e.g. calling another
service, etc.). A viable solution is increasing number of handlers.
But this faces the problem that large amount of threads will consume much
memory (stack, etc.), and performance issues either.
Suggesting the ability to handle requests asynchronously.
For example, in such server, calls may return a Future object instead of
immediate value. Then sends response to client in {{onSuccess}} or {{onFailed}}
callbacks.
was:
On some application, server may run out of handlers when performing many
blocking I/O operations during processing of each call (e.g. calling another
service, etc.). A viable solution is increasing number of handlers.
But this faces the problem that large amount of threads will consume much
memory (stack, etc.), and performance issues either.
Suggesting the ability to handle requests asynchronously.
For example, in such server, calls may return a Future object instead of
immediate value. Then sends response to client in {onSuccess} or {onFailed}
callbacks.
> Async IPC server support
>
>
> Key: HADOOP-13057
> URL: https://issues.apache.org/jira/browse/HADOOP-13057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>
> On some application, server may run out of handlers when performing many
> blocking I/O operations during processing of each call (e.g. calling another
> service, etc.). A viable solution is increasing number of handlers.
> But this faces the problem that large amount of threads will consume much
> memory (stack, etc.), and performance issues either.
> Suggesting the ability to handle requests asynchronously.
> For example, in such server, calls may return a Future object instead of
> immediate value. Then sends response to client in {{onSuccess}} or
> {{onFailed}} callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-13057) Async IPC server support
He Tianyi created HADOOP-13057:
--
Summary: Async IPC server support
Key: HADOOP-13057
URL: https://issues.apache.org/jira/browse/HADOOP-13057
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 2.6.0
Reporter: He Tianyi
On some application, server may run out of handlers when performing many
blocking I/O operations during processing of each call (e.g. calling another
service, etc.). A viable solution is increasing number of handlers.
But this faces the problem that large amount of threads will consume much
memory (stack, etc.), and performance issues either.
Suggesting the ability to handle requests asynchronously.
For example, in such server, calls may return a Future object instead of
immediate value. Then sends response to client in {onSuccess} or {onFailed}
callbacks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-13047) S3a Forward seek in stream length to be configurable
[ https://issues.apache.org/jira/browse/HADOOP-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HADOOP-13047: -- Attachment: HADOOP-13047.WIP.2.patch Agreed. Created "fs.s3a.readahead.buffer.size" which can be used for configuring the buffer size (initially thought of reusing io.file.buffer.size in S3AFileSystem, but it would be better to track s3a readahead separately). Created the patch which includes HADOOP-13028. Main changes are in seekInputStream. Patch should be a lot simplified once HADOOP-13028 is checked in. > S3a Forward seek in stream length to be configurable > > > Key: HADOOP-13047 > URL: https://issues.apache.org/jira/browse/HADOOP-13047 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran > Attachments: HADOOP-13047.WIP.2.patch, HADOOP-13047.WIP.patch > > > Even with lazy seek, tests can show that sometimes a short-distance forward > seek is triggering a close + reopen, because the threshold for the seek is > simply available bytes in the inner stream. > A configurable threshold would allow data to be read and discarded before > that seek. This should be beneficial over long-haul networks as the time to > set up the TCP channel is high, and TCP-slow-start means that the ramp up of > bandwidth is slow. In such deployments, it will better to read forward than > re-open, though the exact "best" number will vary with client and endpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9613) [JDK8] Update jersey version to latest 1.x release
[ https://issues.apache.org/jira/browse/HADOOP-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255763#comment-15255763 ] Tsuyoshi Ozawa commented on HADOOP-9613: [~ste...@apache.org] OK, I will raise this issue to jersey community. Thank you for the comment. > [JDK8] Update jersey version to latest 1.x release > -- > > Key: HADOOP-9613 > URL: https://issues.apache.org/jira/browse/HADOOP-9613 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.4.0, 3.0.0 >Reporter: Timothy St. Clair >Assignee: Tsuyoshi Ozawa > Labels: maven > Attachments: HADOOP-2.2.0-9613.patch, > HADOOP-9613.004.incompatible.patch, HADOOP-9613.005.incompatible.patch, > HADOOP-9613.006.incompatible.patch, HADOOP-9613.007.incompatible.patch, > HADOOP-9613.008.incompatible.patch, HADOOP-9613.009.incompatible.patch, > HADOOP-9613.010.incompatible.patch, HADOOP-9613.011.incompatible.patch, > HADOOP-9613.012.incompatible.patch, HADOOP-9613.013.incompatible.patch, > HADOOP-9613.1.patch, HADOOP-9613.2.patch, HADOOP-9613.3.patch, > HADOOP-9613.patch > > > Update pom.xml dependencies exposed when running a mvn-rpmbuild against > system dependencies on Fedora 18. > The existing version is 1.8 which is quite old. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12579) Deprecate and remove WriteableRPCEngine
[
https://issues.apache.org/jira/browse/HADOOP-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-12579:
---
Attachment: HADOOP-12579-v6.patch
Updated the patch, fixing a missed place.
> Deprecate and remove WriteableRPCEngine
> ---
>
> Key: HADOOP-12579
> URL: https://issues.apache.org/jira/browse/HADOOP-12579
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Haohui Mai
> Attachments: HADOOP-12579-v1.patch, HADOOP-12579-v3.patch,
> HADOOP-12579-v4.patch, HADOOP-12579-v5.patch, HADOOP-12579-v6.patch
>
>
> The {{WriteableRPCEninge}} depends on Java's serialization mechanisms for RPC
> requests. Without proper checks, it has be shown that it can lead to security
> vulnerabilities such as remote code execution (e.g., COLLECTIONS-580,
> HADOOP-12577).
> The current implementation has migrated from {{WriteableRPCEngine}} to
> {{ProtobufRPCEngine}} now. This jira proposes to deprecate
> {{WriteableRPCEngine}} in branch-2 and to remove it in trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[
https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255674#comment-15255674
]
Hadoop QA commented on HADOOP-12892:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 4s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 37s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 33s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
9s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hadoop-project-dist in the patch passed with JDK
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hadoop-assemblies in the patch passed with JDK
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 9s
{color} | {color:green} hadoop-assemblies in the patch passed with JDK
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_77. {color} |
[jira] [Commented] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255661#comment-15255661 ] Hadoop QA commented on HADOOP-12892: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/9165/console in case of problems. > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.patch, > HADOOP-12892.02.patch, HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12892) fix/rewrite create-release
[ https://issues.apache.org/jira/browse/HADOOP-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12892: -- Attachment: HADOOP-12892.03.patch -03: * clean up some output * fix hdfs.h > fix/rewrite create-release > -- > > Key: HADOOP-12892 > URL: https://issues.apache.org/jira/browse/HADOOP-12892 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-12892.00.patch, HADOOP-12892.01.patch, > HADOOP-12892.02.patch, HADOOP-12892.03.patch > > > create-release needs some major surgery. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
