[jira] [Created] (HADOOP-12921) hadoop-minicluster should depend on hadoop-auth

2016-03-13 Thread Haohui Mai (JIRA) 
Haohui Mai created HADOOP-12921:
---

 Summary: hadoop-minicluster should depend on hadoop-auth
 Key: HADOOP-12921
 URL: https://issues.apache.org/jira/browse/HADOOP-12921
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Haohui Mai


Downstream projects fail to spawn a {{MiniDFSCluster}} as AuthenticationFilter 
is not in the dependency of the {{hadoop-minicluster}} package:

{noformat}
java.lang.NoSuchMethodError: 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.constructSecretProvider(Ljavax/servlet/ServletContext;Ljava/util/Properties;Z)Lorg/apache/hadoop/security/authentication/util/SignerSecretProvider;
at 
org.apache.hadoop.http.HttpServer2.constructSecretProvider(HttpServer2.java:447)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:340)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:114)
at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:290)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:126)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:752)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:638)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:811)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:795)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
  ...
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library

2016-03-13 Thread Kai Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HADOOP-11540:
---
Attachment: HADOOP-11540-v5.patch

Fixed part of the check styles.

> Raw Reed-Solomon coder using Intel ISA-L library
> 
>
> Key: HADOOP-11540
> URL: https://issues.apache.org/jira/browse/HADOOP-11540
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Kai Zheng
> Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, 
> HADOOP-11540-v2.patch, HADOOP-11540-v4.patch, HADOOP-11540-v5.patch, 
> HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel 
> ISAL-v1.pdf
>
>
> This is to provide RS codec implementation using Intel ISA-L library for 
> encoding and decoding.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library

2016-03-13 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192669#comment-15192669
 ] 

Hadoop QA commented on HADOOP-11540:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 34s 
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 29s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s 
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 30s 
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 30s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 30s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s 
{color} | {color:red} hadoop-common-project/hadoop-common: patch generated 16 
new + 1 unchanged - 0 fixed = 17 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 22s {color} 
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 55s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
21s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 5s {color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Timed out junit tests | 
org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA

[jira] [Updated] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library

2016-03-13 Thread Kai Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HADOOP-11540:
---
Attachment: HADOOP-11540-v4.patch

Rebased and refined the patch after HADOOP-11996 was in.

> Raw Reed-Solomon coder using Intel ISA-L library
> 
>
> Key: HADOOP-11540
> URL: https://issues.apache.org/jira/browse/HADOOP-11540
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Kai Zheng
> Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, 
> HADOOP-11540-v2.patch, HADOOP-11540-v4.patch, 
> HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel 
> ISAL-v1.pdf
>
>
> This is to provide RS codec implementation using Intel ISA-L library for 
> encoding and decoding.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12724) Let BufferedFSInputStream implement CanUnbuffer

2016-03-13 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HADOOP-12724:

Description: 
When trying to determine reason for test failure over in HBASE-9393, I saw the 
following exception:
{code}
testSeekTo[4](org.apache.hadoop.hbase.io.hfile.TestSeekTo)  Time elapsed: 0.033 
sec  <<< ERROR!
java.lang.UnsupportedOperationException: this stream does not support 
unbuffering.
at 
org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:229)
at 
org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:227)
at 
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:518)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:562)
at 
org.apache.hadoop.hbase.io.hfile.TestSeekTo.testSeekToInternals(TestSeekTo.java:307)
at 
org.apache.hadoop.hbase.io.hfile.TestSeekTo.testSeekTo(TestSeekTo.java:298)
{code}
Here is the cause:
{code}
java.lang.ClassCastException: org.apache.hadoop.fs.BufferedFSInputStream cannot 
be cast to org.apache.hadoop.fs.CanUnbuffer
{code}
See the comments starting with 
https://issues.apache.org/jira/browse/HBASE-9393?focusedCommentId=15105939=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15105939
 for background on the HBase patch.

This issue is to make BufferedFSInputStream implement CanUnbuffer.

This would benefit hbase unit tests.

Thanks to [~cmccabe] for discussion.

  was:
When trying to determine reason for test failure over in HBASE-9393, I saw the 
following exception:
{code}
testSeekTo[4](org.apache.hadoop.hbase.io.hfile.TestSeekTo)  Time elapsed: 0.033 
sec  <<< ERROR!
java.lang.UnsupportedOperationException: this stream does not support 
unbuffering.
at 
org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:229)
at 
org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:227)
at 
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:518)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:562)
at 
org.apache.hadoop.hbase.io.hfile.TestSeekTo.testSeekToInternals(TestSeekTo.java:307)
at 
org.apache.hadoop.hbase.io.hfile.TestSeekTo.testSeekTo(TestSeekTo.java:298)
{code}
Here is the cause:
{code}
java.lang.ClassCastException: org.apache.hadoop.fs.BufferedFSInputStream cannot 
be cast to org.apache.hadoop.fs.CanUnbuffer
{code}
See the comments starting with 
https://issues.apache.org/jira/browse/HBASE-9393?focusedCommentId=15105939=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15105939
 for background on the HBase patch.

This issue is to make BufferedFSInputStream implement CanUnbuffer.

This would benefit hbase unit tests.


Thanks to [~cmccabe] for discussion.


> Let BufferedFSInputStream implement CanUnbuffer
> ---
>
> Key: HADOOP-12724
> URL: https://issues.apache.org/jira/browse/HADOOP-12724
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Ted Yu
>
> When trying to determine reason for test failure over in HBASE-9393, I saw 
> the following exception:
> {code}
> testSeekTo[4](org.apache.hadoop.hbase.io.hfile.TestSeekTo)  Time elapsed: 
> 0.033 sec  <<< ERROR!
> java.lang.UnsupportedOperationException: this stream does not support 
> unbuffering.
>   at 
> org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:229)
>   at 
> org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:227)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:518)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:562)
>   at 
> org.apache.hadoop.hbase.io.hfile.TestSeekTo.testSeekToInternals(TestSeekTo.java:307)
>   at 
> org.apache.hadoop.hbase.io.hfile.TestSeekTo.testSeekTo(TestSeekTo.java:298)
> {code}
> Here is the cause:
> {code}
> java.lang.ClassCastException: org.apache.hadoop.fs.BufferedFSInputStream 
> cannot be cast to org.apache.hadoop.fs.CanUnbuffer
> {code}
> See the comments starting with 
> https://issues.apache.org/jira/browse/HBASE-9393?focusedCommentId=15105939=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15105939
>  for background on the HBase patch.
> This issue is to make BufferedFSInputStream implement CanUnbuffer.
> This would benefit hbase unit tests.
> Thanks to [~cmccabe] for discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12672) RPC timeout should not override IPC ping interval

2016-03-13 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192353#comment-15192353
 ] 

Steve Loughran commented on HADOOP-12672:
-

No worries about the breakage, though I think I'd have been happier if 2.6 
wasn't picking up so much of these RPC changes, not until they were stable. 

Can you add this patch to HDFS and YARN JIRAs so we can verify their 
miniclusters are happy this time round?

> RPC timeout should not override IPC ping interval
> -
>
> Key: HADOOP-12672
> URL: https://issues.apache.org/jira/browse/HADOOP-12672
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc
>Affects Versions: 2.8.0, 2.7.3, 2.6.4
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Attachments: HADOOP-12672.001.patch, HADOOP-12672.002.patch, 
> HADOOP-12672.003.patch, HADOOP-12672.004.patch, HADOOP-12672.005.patch, 
> HADOOP-12672.006.patch
>
>
> Currently if the value of ipc.client.rpc-timeout.ms is greater than 0, the 
> timeout overrides the ipc.ping.interval and client will throw exception 
> instead of sending ping when the interval is passed. RPC timeout should work 
> without effectively disabling IPC ping.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls

2016-03-13 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192352#comment-15192352
 ] 

Steve Loughran commented on HADOOP-12910:
-

In HADOOP-9965 I've been wondering if we could have a method 
"hasFeature(String):" -> yes, no, don-know; we'd have standard strings to probe 
"append", "flush",  The same probe could do "serializable", 
"lineralizable", ...

To be fair, most code shouldn't be checking this stuff as there should be a 
common set of assumptions: Filesystems look like HDFS, asyc operations have no 
guarantees on ordering. But some things (distCp , output committers) may 
benefit if they probe for it, and the FS implementations are honest in their 
replies

> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a 
> better name).  All the APIs in FutureFileSystem are the same as FileSystem 
> except that the return type is wrapped by Future, e.g.
> {code}
>   //FileSystem
>   public boolean rename(Path src, Path dst) throws IOException;
>   //FutureFileSystem
>   public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls

2016-03-13 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192350#comment-15192350
 ] 

Steve Loughran commented on HADOOP-12910:
-

Exception-wise, I'd give the sync method and the future the right to raise 
IOEs; the future may work best as wrapped

+1 for declaring everything as {{throws IOE}}.

bq. f we want to support Java 8 language features, we should also support it 
for FileSystem.

I was thinking in terms of having the async calls hook up to lambda expressions

> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a 
> better name).  All the APIs in FutureFileSystem are the same as FileSystem 
> except that the return type is wrapped by Future, e.g.
> {code}
>   //FileSystem
>   public boolean rename(Path src, Path dst) throws IOException;
>   //FutureFileSystem
>   public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)