date:20160425

[jira] [Updated] (HADOOP-12957) Limit the number of outstanding async calls

2016-04-25 Thread Xiaobing Zhou (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HADOOP-12957:
---
Attachment: HADOOP-12957.001.patch

> Limit the number of outstanding async calls
> ---
>
> Key: HADOOP-12957
> URL: https://issues.apache.org/jira/browse/HADOOP-12957
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12957-HADOOP-12909.000.patch, 
> HADOOP-12957-combo.000.patch, HADOOP-12957.001.patch
>
>
> In async RPC, if the callers don't read replies fast enough, the buffer 
> storing replies could be used up. This is to propose limiting the number of 
> outstanding async calls to eliminate the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12345) Credential length in CredentialsSys.java incorrect

2016-04-25 Thread Akira AJISAKA (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated HADOOP-12345:
---
Assignee: (was: S M Al Mamun)

> Credential length in CredentialsSys.java incorrect
> --
>
> Key: HADOOP-12345
> URL: https://issues.apache.org/jira/browse/HADOOP-12345
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: nfs
>Affects Versions: 2.7.0
>Reporter: Pradeep Nayak Udupi Kadbet
>Priority: Blocker
>
> Hi -
> There is a bug in the way hadoop-nfs sets the credential length in 
> "Credentials" field of the NFS RPC packet when using AUTH_SYS
> In CredentialsSys.java, when we are writing the creds in to XDR object, we 
> set the length as follows:
>  // mStamp + mHostName.length + mHostName + mUID + mGID + mAuxGIDs.count
> 96 mCredentialsLength = 20 + mHostName.getBytes().length;
> (20 corresponds to 4 bytes for mStamp, 4 bytes for mUID, 4 bytes for mGID, 4 
> bytes for length field of hostname, 4 bytes for number of aux 4 gids) and 
> this is okay.
> However when we add the length of the hostname to this, we are not adding the 
> extra padded bytes for the hostname (If the length is not a multiple of 4) 
> and thus when the NFS server reads the packet, it returns GARBAGE_ARGS 
> because it doesn't read the uid field when it is expected to read. I can 
> reproduce this issue constantly on machines where the hostname length is not 
> a multiple of 4.
> A possible fix is to do something this:
> int pad = mHostName.getBytes().length % 4;
>  // mStamp + mHostName.length + mHostName + mUID + mGID + mAuxGIDs.count
> mCredentialsLength = 20 + mHostName.getBytes().length + pad;
> I would be happy to submit the patch but I need some help to commit into 
> mainline. I haven't committed into Hadoop yet.
> Cheers!
> Pradeep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12957) Limit the number of outstanding async calls

2016-04-25 Thread Xiaobing Zhou (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257599#comment-15257599
 ] 

Xiaobing Zhou commented on HADOOP-12957:


I posted patch v001 for review. The idea is:
When the limit is reached, AsyncCallLimitExceededException will be thrown, 
applications need to handle the exception by calling Future#get to retrieve 
results of established async calls in order not to block follow-on async calls.

> Limit the number of outstanding async calls
> ---
>
> Key: HADOOP-12957
> URL: https://issues.apache.org/jira/browse/HADOOP-12957
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12957-HADOOP-12909.000.patch, 
> HADOOP-12957-combo.000.patch, HADOOP-12957.001.patch
>
>
> In async RPC, if the callers don't read replies fast enough, the buffer 
> storing replies could be used up. This is to propose limiting the number of 
> outstanding async calls to eliminate the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12957) Limit the number of outstanding async calls

2016-04-25 Thread Xiaobing Zhou (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HADOOP-12957:
---
Attachment: (was: HADOOP-12957.001.patch)

> Limit the number of outstanding async calls
> ---
>
> Key: HADOOP-12957
> URL: https://issues.apache.org/jira/browse/HADOOP-12957
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12957-HADOOP-12909.000.patch, 
> HADOOP-12957-combo.000.patch, HADOOP-12957.001.patch
>
>
> In async RPC, if the callers don't read replies fast enough, the buffer 
> storing replies could be used up. This is to propose limiting the number of 
> outstanding async calls to eliminate the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12957) Limit the number of outstanding async calls

2016-04-25 Thread Xiaobing Zhou (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HADOOP-12957:
---
Attachment: HADOOP-12957.001.patch

> Limit the number of outstanding async calls
> ---
>
> Key: HADOOP-12957
> URL: https://issues.apache.org/jira/browse/HADOOP-12957
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12957-HADOOP-12909.000.patch, 
> HADOOP-12957-combo.000.patch, HADOOP-12957.001.patch
>
>
> In async RPC, if the callers don't read replies fast enough, the buffer 
> storing replies could be used up. This is to propose limiting the number of 
> outstanding async calls to eliminate the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13010) Refactor raw erasure coders

2016-04-25 Thread Rui Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257562#comment-15257562
 ] 

Rui Li commented on HADOOP-13010:
-

Just created HADOOP-13061 for the refactor of erasure coders. I'll start 
working on it after this one is done.

> Refactor raw erasure coders
> ---
>
> Key: HADOOP-13010
> URL: https://issues.apache.org/jira/browse/HADOOP-13010
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Fix For: 3.0.0
>
> Attachments: HADOOP-13010-v1.patch, HADOOP-13010-v2.patch, 
> HADOOP-13010-v3.patch, HADOOP-13010-v4.patch, HADOOP-13010-v5.patch
>
>
> This will refactor raw erasure coders according to some comments received so 
> far.
> * As discussed in HADOOP-11540 and suggested by [~cmccabe], better not to 
> rely class inheritance to reuse the codes, instead they can be moved to some 
> utility.
> * Suggested by [~jingzhao] somewhere quite some time ago, better to have a 
> state holder to keep some checking results for later reuse during an 
> encode/decode call.
> This would not get rid of some inheritance levels as doing so isn't clear yet 
> for the moment and also incurs big impact. I do wish the end result by this 
> refactoring will make all the levels more clear and easier to follow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-13061) Refactor erasure coders

2016-04-25 Thread Rui Li (JIRA)

Rui Li created HADOOP-13061:
---

 Summary: Refactor erasure coders
 Key: HADOOP-13061
 URL: https://issues.apache.org/jira/browse/HADOOP-13061
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Rui Li
Assignee: Rui Li






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12991) Conflicting default ports in DelegateToFileSystem

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257499#comment-15257499
 ] 

Hadoop QA commented on HADOOP-12991:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s 
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 44s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s 
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 34s 
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 46s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 44s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_92. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 59s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
24s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 11s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12796669/HADOOP-12991.01.patch 
|
| JIRA Issue | HADOOP-12991 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 5856c75191eb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8eadd71 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |

[jira] [Updated] (HADOOP-11603) Metric Snapshot log can be changed #MetricsSystemImpl.java since all the services will be initialized

2016-04-25 Thread Brahma Reddy Battula (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HADOOP-11603:
--
Description: 
Namenode,DataNode,journalnode,ResourceManager, Nodemanager and historyservice 
etc... will initialize DefaultMetricsSystem hence following log will be logged 
for all the classes.. Since there is snapshot feature in  HDFS ,it's better to 
change the following...
{code}
LOG.info("Scheduled snapshot period at "+ period +" second(s).");
{code}

  was:
Namenode,DataNode,journalnode,ResourceManager, Nodemanager and historyservice 
etc... will initialize DefaultMetricsSystem hence following log will be logged 
for all the classes which is not correct.. since snapshot feature is related to 
HDFS and this can be logged when snapshot is enabled.

{code}
LOG.info("Scheduled snapshot period at "+ period +" second(s).");
{code}


> Metric Snapshot log can be changed #MetricsSystemImpl.java since all the 
> services will be initialized
> -
>
> Key: HADOOP-11603
> URL: https://issues.apache.org/jira/browse/HADOOP-11603
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Minor
>  Labels: BB2015-05-TBR
> Attachments: HADOOP-11603.patch, defaultmetricIntialize-classes.JPG
>
>
> Namenode,DataNode,journalnode,ResourceManager, Nodemanager and historyservice 
> etc... will initialize DefaultMetricsSystem hence following log will be 
> logged for all the classes.. Since there is snapshot feature in  HDFS ,it's 
> better to change the following...
> {code}
> LOG.info("Scheduled snapshot period at "+ period +" second(s).");
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11603) Metric Snapshot log can be changed #MetricsSystemImpl.java since all the services will be initialized

2016-04-25 Thread Brahma Reddy Battula (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HADOOP-11603:
--
Summary: Metric Snapshot log can be changed #MetricsSystemImpl.java since 
all the services will be initialized  (was: Snapshot log can be removed 
#MetricsSystemImpl.java since all the services will intialzed)

> Metric Snapshot log can be changed #MetricsSystemImpl.java since all the 
> services will be initialized
> -
>
> Key: HADOOP-11603
> URL: https://issues.apache.org/jira/browse/HADOOP-11603
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Minor
>  Labels: BB2015-05-TBR
> Attachments: HADOOP-11603.patch, defaultmetricIntialize-classes.JPG
>
>
> Namenode,DataNode,journalnode,ResourceManager, Nodemanager and historyservice 
> etc... will initialize DefaultMetricsSystem hence following log will be 
> logged for all the classes which is not correct.. since snapshot feature is 
> related to HDFS and this can be logged when snapshot is enabled.
> {code}
> LOG.info("Scheduled snapshot period at "+ period +" second(s).");
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12991) Conflicting default ports in DelegateToFileSystem

2016-04-25 Thread Kai Sasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Sasaki updated HADOOP-12991:

Status: Patch Available  (was: Open)

> Conflicting default ports in DelegateToFileSystem
> -
>
> Key: HADOOP-12991
> URL: https://issues.apache.org/jira/browse/HADOOP-12991
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.7.2
>Reporter: Kevin Hogeland
>Assignee: Kai Sasaki
> Attachments: HADOOP-12991.01.patch
>
>
> HADOOP-12304 introduced logic to ensure that the {{DelegateToFileSystem}} 
> constructor sets the default port to -1:
> {code:title=DelegateToFileSystem.java}
>   protected DelegateToFileSystem(URI theUri, FileSystem theFsImpl,
>   Configuration conf, String supportedScheme, boolean authorityRequired)
>   throws IOException, URISyntaxException {
> super(theUri, supportedScheme, authorityRequired, 
> getDefaultPortIfDefined(theFsImpl));
> fsImpl = theFsImpl;
> fsImpl.initialize(theUri, conf);
> fsImpl.statistics = getStatistics();
>   }
>   private static int getDefaultPortIfDefined(FileSystem theFsImpl) {
> int defaultPort = theFsImpl.getDefaultPort();
> return defaultPort != 0 ? defaultPort : -1;
>   }
> {code}
> However, {{DelegateToFileSystem#getUriDefaultPort}} returns 0:
> {code:title=DelegateToFileSystem.java}
>   public int getUriDefaultPort() {
> return 0;
>   }
> {code}
> This breaks {{AbstractFileSystem#checkPath}}:
> {code:title=AbstractFileSystem.java}
> int thisPort = this.getUri().getPort(); // If using DelegateToFileSystem, 
> this is -1
> int thatPort = uri.getPort(); // This is -1 by default in java.net.URI
> if (thatPort == -1) {
>   thatPort = this.getUriDefaultPort();  // Sets thatPort to 0
> }
> if (thisPort != thatPort) {
>   throw new InvalidPathException("Wrong FS: " + path + ", expected: "
>   + this.getUri());
> }
> {code}
> Which breaks any subclasses of {{DelegateToFileSystem}} that don't specify a 
> port (S3n, Wasb(s)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11418) Property "io.compression.codec.lzo.class" does not work with other value besides default

2016-04-25 Thread Yuanbo Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257435#comment-15257435
 ] 

Yuanbo Liu commented on HADOOP-11418:
-

Thanks for your great work, Akira and Masatake

> Property "io.compression.codec.lzo.class" does not work with other value 
> besides default
> 
>
> Key: HADOOP-11418
> URL: https://issues.apache.org/jira/browse/HADOOP-11418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io
>Affects Versions: 2.6.0
>Reporter: fang fang chen
>Assignee: Yuanbo Liu
>  Labels: BB2015-05-RFC
> Fix For: 2.8.0
>
> Attachments: HADOOP-11418-004.patch, HADOOP-11418-1.patch, 
> HADOOP-11418-2.patch, HADOOP-11418-3.patch, HADOOP-11418.005.patch, 
> HADOOP-11418.006.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for 
> other codec besides default. Hadoop will always treat it as defaultClazz. I 
> think it is a bug. Please let me know if this is a work as design thing. 
> Thanks
>  77   private static final String defaultClazz =
>  78   "org.apache.hadoop.io.compress.LzoCodec";
>  82   public synchronized boolean isSupported() {
>  83 if (!checked) {
>  84   checked = true;
>  85   String extClazz =
>  86   (conf.get(CONF_LZO_CLASS) == null ? System
>  87   .getProperty(CONF_LZO_CLASS) : null);
>  88   String clazz = (extClazz != null) ? extClazz : defaultClazz;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12579) Deprecate and remove WriteableRPCEngine

2016-04-25 Thread Haohui Mai (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257367#comment-15257367
 ] 

Haohui Mai commented on HADOOP-12579:
-

{code}
   RPC_WRITABLE ((short) 2),// Use WritableRpcEngine,
{code}

Is it possible to get rid of this? +1 once addressed.


> Deprecate and remove WriteableRPCEngine
> ---
>
> Key: HADOOP-12579
> URL: https://issues.apache.org/jira/browse/HADOOP-12579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Haohui Mai
> Attachments: HADOOP-12579-v1.patch, HADOOP-12579-v3.patch, 
> HADOOP-12579-v4.patch, HADOOP-12579-v5.patch, HADOOP-12579-v6.patch, 
> HADOOP-12579-v7.patch
>
>
> The {{WriteableRPCEninge}} depends on Java's serialization mechanisms for RPC 
> requests. Without proper checks, it has be shown that it can lead to security 
> vulnerabilities such as remote code execution (e.g., COLLECTIONS-580, 
> HADOOP-12577).
> The current implementation has migrated from {{WriteableRPCEngine}} to 
> {{ProtobufRPCEngine}} now. This jira proposes to deprecate 
> {{WriteableRPCEngine}} in branch-2 and to remove it in trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12935) API documentation for dynamic subcommands

2016-04-25 Thread Allen Wittenauer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-12935:
--
   Resolution: Fixed
 Assignee: Allen Wittenauer
Fix Version/s: HADOOP-12930
   Status: Resolved  (was: Patch Available)

> API documentation for dynamic subcommands
> -
>
> Key: HADOOP-12935
> URL: https://issues.apache.org/jira/browse/HADOOP-12935
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: scripts
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Fix For: HADOOP-12930
>
> Attachments: HADOOP-12935-HADOOP-12930.01.patch, HADOOP-12935.00.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12935) API documentation for dynamic subcommands

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257349#comment-15257349
 ] 

Hadoop QA commented on HADOOP-12935:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
28s {color} | {color:green} HADOOP-12930 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s 
{color} | {color:green} HADOOP-12930 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
20s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 12s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800680/HADOOP-12935-HADOOP-12930.01.patch
 |
| JIRA Issue | HADOOP-12935 |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux d0b88e1e6d67 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HADOOP-12930 / 8eadd71 |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9178/console |
| Powered by | Apache Yetus 0.3.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> API documentation for dynamic subcommands
> -
>
> Key: HADOOP-12935
> URL: https://issues.apache.org/jira/browse/HADOOP-12935
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: scripts
>Reporter: Allen Wittenauer
> Attachments: HADOOP-12935-HADOOP-12930.01.patch, HADOOP-12935.00.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12935) API documentation for dynamic subcommands

2016-04-25 Thread Allen Wittenauer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-12935:
--
Attachment: HADOOP-12935-HADOOP-12930.01.patch

-01:
* fix errors

> API documentation for dynamic subcommands
> -
>
> Key: HADOOP-12935
> URL: https://issues.apache.org/jira/browse/HADOOP-12935
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: scripts
>Reporter: Allen Wittenauer
> Attachments: HADOOP-12935-HADOOP-12930.01.patch, HADOOP-12935.00.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13060) While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.

2016-04-25 Thread Jenny Dong (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257319#comment-15257319
 ] 

Jenny Dong commented on HADOOP-13060:
-

I was using aws-java-sdk-1.10.69. I added aws-java-sdk-s3:1.10.6 but we would 
still need to include aws-java-sdk.jar. Which version should we be using for 
that? 

> While trying to perform a Distcp command, we see the error Exception in 
> thread "main" java.lang.NoSuchMethodError: 
> com.amazonaws.services.s3.transfer.TransferManager.(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> 
>
> Key: HADOOP-13060
> URL: https://issues.apache.org/jira/browse/HADOOP-13060
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Jenny Dong
>
> While trying to do a distcp from our native HDFS cluster to S3, we get the 
> following error/stacktrace : 
> We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar 
> version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors 
> seen in HADOOP-12420 + other authentication errors).
> Exception in thread "main" java.lang.NoSuchMethodError: 
> com.amazonaws.services.s3.transfer.TransferManager.(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>   at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
>   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
>   at 
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
>   at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
>   at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
>   at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
>   at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
>   at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
> I dug into both classes com.amazonaws.services.s3.transfer.TransferManager & 
> org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem 
> created a ThreadPoolExecutor (which implements AbstractExecutorService which 
> implements ExecutorService). I also checked on the classpath to make sure the 
> version of the jars being picked up is what I expected. 
> Help would be much appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13039) Add documentation for configuration property ipc.maximum.data.length for controlling maximum RPC message size.

2016-04-25 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257238#comment-15257238
 ] 

Mingliang Liu commented on HADOOP-13039:


{quote}{color:red}The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch.{color}{quote}

This is for adding missing documentation for config property. No code change 
was involved.

> Add documentation for configuration property ipc.maximum.data.length for 
> controlling maximum RPC message size.
> --
>
> Key: HADOOP-13039
> URL: https://issues.apache.org/jira/browse/HADOOP-13039
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13039.000.patch, HADOOP-13039.001.patch, 
> HADOOP-13039.001.patch, HADOOP-13039.001.patch
>
>
> The RPC server enforces a maximum length on incoming messages.  Messages 
> larger than the maximum are rejected immediately as potentially malicious.  
> The maximum length can be tuned by setting configuration property 
> {{ipc.maximum.data.length}}, but this is not documented in core-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13039) Add documentation for configuration property ipc.maximum.data.length for controlling maximum RPC message size.

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257221#comment-15257221
 ] 

Hadoop QA commented on HADOOP-13039:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 39s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 3s 
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 47s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s 
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s 
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 49s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 24s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_92. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 35s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 37s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800640/HADOOP-13039.001.patch
 |
| JIRA Issue | HADOOP-13039 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  |
| uname | Linux 23b65dc78f14 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8eadd71 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_92 
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| JDK v1.7.0_95  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9177/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console

[jira] [Commented] (HADOOP-13060) While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.

2016-04-25 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257154#comment-15257154
 ] 

Chris Nauroth commented on HADOOP-13060:


bq. We are using aws-java-sdk.jar version 1.10.69...

HADOOP-12269 changed the dependency from aws-java-sdk:1.7.4 to 
aws-java-sdk-s3:1.10.6.  Can you try checking again that you were using 
"aws-java-sdk-s3" and not "aws-java-sdk"?  We are successfully testing with 
aws-java-sdk-s3:1.10.6.  This is potentially confusing because the names are so 
similar.

> While trying to perform a Distcp command, we see the error Exception in 
> thread "main" java.lang.NoSuchMethodError: 
> com.amazonaws.services.s3.transfer.TransferManager.(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> 
>
> Key: HADOOP-13060
> URL: https://issues.apache.org/jira/browse/HADOOP-13060
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Jenny Dong
>
> While trying to do a distcp from our native HDFS cluster to S3, we get the 
> following error/stacktrace : 
> We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar 
> version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors 
> seen in HADOOP-12420 + other authentication errors).
> Exception in thread "main" java.lang.NoSuchMethodError: 
> com.amazonaws.services.s3.transfer.TransferManager.(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>   at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
>   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
>   at 
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
>   at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
>   at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
>   at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
>   at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
>   at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
> I dug into both classes com.amazonaws.services.s3.transfer.TransferManager & 
> org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem 
> created a ThreadPoolExecutor (which implements AbstractExecutorService which 
> implements ExecutorService). I also checked on the classpath to make sure the 
> version of the jars being picked up is what I expected. 
> Help would be much appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11820) aw jira testing, ignore

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257117#comment-15257117
 ] 

Hadoop QA commented on HADOOP-11820:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 3s {color} 
| {color:red} Docker failed to build yetus/hadoop:fbe3e86. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12798315/0001-parallelize-long-running-yarn-unit-tests.patch
 |
| JIRA Issue | HADOOP-11820 |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9176/console |
| Powered by | Apache Yetus 0.3.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> aw jira testing, ignore
> ---
>
> Key: HADOOP-11820
> URL: https://issues.apache.org/jira/browse/HADOOP-11820
> Project: Hadoop Common
>  Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
> Attachments: 0001-parallelize-long-running-yarn-unit-tests.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13028) add low level counter metrics for S3A; use in read performance tests

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257096#comment-15257096
 ] 

Hadoop QA commented on HADOOP-13028:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
1s {color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 6s 
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 49s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s 
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 2s 
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 54s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 54s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s 
{color} | {color:red} root: patch generated 44 new + 53 unchanged - 2 fixed = 
97 total (was 55) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 44s 
{color} | {color:red} hadoop-tools/hadoop-aws generated 12 new + 0 unchanged - 
0 fixed = 12 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s 
{color} | {color:red} hadoop-aws in the patch failed with JDK v1.8.0_92. 
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 50s 
{color} | {color:red} hadoop-tools_hadoop-aws-jdk1.7.0_95 with JDK v1.7.0_95 
generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 56s {color} 
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s 
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_92. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 10s {color} 
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_95.

[jira] [Commented] (HADOOP-12949) Add metrics and HTrace to the s3a connector

2016-04-25 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257083#comment-15257083
 ] 

Colin Patrick McCabe commented on HADOOP-12949:
---

Since HADOOP-13028 is focusing on metrics for s3a, let's focus this JIRA on 
just HTrace integration.  It's a good idea to read up on HDFS-10175 as well, 
since we've been discussing what interface(s) we'd like the FS metrics to have 
in the future there.

> Add metrics and HTrace to the s3a connector
> ---
>
> Key: HADOOP-12949
> URL: https://issues.apache.org/jira/browse/HADOOP-12949
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Reporter: Madhawa Gunasekara
>Assignee: Madhawa Gunasekara
>
> Hi All, 
> s3, GCS, WASB, and other cloud blob stores are becoming increasingly 
> important in Hadoop. But we don't have distributed tracing for these yet. It 
> would be interesting to add distributed tracing here. It would enable 
> collecting really interesting data like probability distributions of PUT and 
> GET requests to s3 and their impact on MR jobs, etc.
> I would like to implement this feature, Please shed some light on this 
> Thanks,
> Madhawa



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12949) Add HTrace to the s3a connector

2016-04-25 Thread Colin Patrick McCabe (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-12949:
--
Summary: Add HTrace to the s3a connector  (was: Add metrics and HTrace to 
the s3a connector)

> Add HTrace to the s3a connector
> ---
>
> Key: HADOOP-12949
> URL: https://issues.apache.org/jira/browse/HADOOP-12949
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Reporter: Madhawa Gunasekara
>Assignee: Madhawa Gunasekara
>
> Hi All, 
> s3, GCS, WASB, and other cloud blob stores are becoming increasingly 
> important in Hadoop. But we don't have distributed tracing for these yet. It 
> would be interesting to add distributed tracing here. It would enable 
> collecting really interesting data like probability distributions of PUT and 
> GET requests to s3 and their impact on MR jobs, etc.
> I would like to implement this feature, Please shed some light on this 
> Thanks,
> Madhawa



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-13060) While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.

2016-04-25 Thread Jenny Dong (JIRA)

Jenny Dong created HADOOP-13060:
---

 Summary: While trying to perform a Distcp command, we see the 
error Exception in thread "main" java.lang.NoSuchMethodError: 
com.amazonaws.services.s3.transfer.TransferManager.(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
 Key: HADOOP-13060
 URL: https://issues.apache.org/jira/browse/HADOOP-13060
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 2.7.1
Reporter: Jenny Dong


While trying to do a distcp from our native HDFS cluster to S3, we get the 
following error/stacktrace : 

We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar 
version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors 
seen in HADOOP-12420 + other authentication errors).

Exception in thread "main" java.lang.NoSuchMethodError: 
com.amazonaws.services.s3.transfer.TransferManager.(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)

I dug into both classes com.amazonaws.services.s3.transfer.TransferManager & 
org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem 
created a ThreadPoolExecutor (which implements AbstractExecutorService which 
implements ExecutorService). I also checked on the classpath to make sure the 
version of the jars being picked up is what I expected. 

Help would be much appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13039) Add documentation for configuration property ipc.maximum.data.length for controlling maximum RPC message size.

2016-04-25 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13039:
---
Attachment: HADOOP-13039.001.patch

{quote}
The following NEW packages will be installed:
  bats
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.5 kB of archives.
After this operation, 70.7 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  bats
[91mE: There are problems and -y was used without --force-yes
[0mThe command '/bin/sh -c apt-get install -y bats' returned a non-zero code: 
100

Total Elapsed time:   0m  3s

ERROR: Docker failed to build image.
{quote}

Seems the precommit build is failing. Will file a bug for Yetus if it happens 
again.

> Add documentation for configuration property ipc.maximum.data.length for 
> controlling maximum RPC message size.
> --
>
> Key: HADOOP-13039
> URL: https://issues.apache.org/jira/browse/HADOOP-13039
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13039.000.patch, HADOOP-13039.001.patch, 
> HADOOP-13039.001.patch, HADOOP-13039.001.patch
>
>
> The RPC server enforces a maximum length on incoming messages.  Messages 
> larger than the maximum are rejected immediately as potentially malicious.  
> The maximum length can be tuned by setting configuration property 
> {{ipc.maximum.data.length}}, but this is not documented in core-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files

2016-04-25 Thread Ravi Prakash (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257047#comment-15257047
 ] 

Ravi Prakash commented on HADOOP-13018:
---

Hi Steve! I felt {{verify}} shouldn't be Method specific. I'm calling 
{{Credentials.readTokenStorageFile}} and checking it doesn't throw any 
exceptions. Perhaps in the future, we'd want some other Method? Did you have 
something else in mind?

> Make Kdiag check whether hadoop.token.files points to existent and valid files
> --
>
> Key: HADOOP-13018
> URL: https://issues.apache.org/jira/browse/HADOOP-13018
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HADOOP-13018.01.patch, HADOOP-13018.02.patch
>
>
> Steve proposed that KDiag should fail fast to help debug the case that 
> hadoop.token.files points to a file not found. This JIRA is to affect that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files

2016-04-25 Thread Ravi Prakash (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-13018:
--
Target Version/s: 3.0.0, 2.9.0  (was: 3.0.0)

> Make Kdiag check whether hadoop.token.files points to existent and valid files
> --
>
> Key: HADOOP-13018
> URL: https://issues.apache.org/jira/browse/HADOOP-13018
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HADOOP-13018.01.patch, HADOOP-13018.02.patch
>
>
> Steve proposed that KDiag should fail fast to help debug the case that 
> hadoop.token.files points to a file not found. This JIRA is to affect that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12782) Faster LDAP group name resolution with ActiveDirectory

2016-04-25 Thread Wei-Chiu Chuang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-12782:
-
Attachment: HADOOP-12782.004.patch

rev04: fixed checkstyle and javac warnings.

> Faster LDAP group name resolution with ActiveDirectory
> --
>
> Key: HADOOP-12782
> URL: https://issues.apache.org/jira/browse/HADOOP-12782
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12782.001.patch, HADOOP-12782.002.patch, 
> HADOOP-12782.003.patch, HADOOP-12782.004.patch
>
>
> The typical LDAP group name resolution works well under typical scenarios. 
> However, we have seen cases where a user is mapped to many groups (in an 
> extreme case, a user is mapped to more than 100 groups). The way it's being 
> implemented now makes this case super slow resolving groups from 
> ActiveDirectory.
> The current LDAP group resolution implementation sends two queries to a 
> ActiveDirectory server. The first query returns a user object, which contains 
> DN (distinguished name). The second query looks for groups where the user DN 
> is a member. If a user is mapped to many groups, the second query returns all 
> group objects associated with the user, and is thus very slow.
> After studying a user object in ActiveDirectory, I found a user object 
> actually contains a "memberOf" field, which is the DN of all group objects 
> where the user belongs to. Assuming that an organization has no recursive 
> group relation (that is, a user A is a member of group G1, and group G1 is a 
> member of group G2), we can use this properties to avoid the second query, 
> which can potentially run very slow.
> I propose that we add a configuration to only enable this feature for users 
> who want to reduce group resolution time and who does not have recursive 
> groups, so that existing behavior will not be broken.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12949) Add metrics and HTrace to the s3a connector

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256909#comment-15256909
 ] 

Steve Loughran commented on HADOOP-12949:
-

In HADOOP-13028 I've put in some low level counters, primarily to see what's 
going on. That covers a lot of what would be useful, though it happily ignores 
anything directly tracking duration, and HTrace integration

> Add metrics and HTrace to the s3a connector
> ---
>
> Key: HADOOP-12949
> URL: https://issues.apache.org/jira/browse/HADOOP-12949
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Reporter: Madhawa Gunasekara
>Assignee: Madhawa Gunasekara
>
> Hi All, 
> s3, GCS, WASB, and other cloud blob stores are becoming increasingly 
> important in Hadoop. But we don't have distributed tracing for these yet. It 
> would be interesting to add distributed tracing here. It would enable 
> collecting really interesting data like probability distributions of PUT and 
> GET requests to s3 and their impact on MR jobs, etc.
> I would like to implement this feature, Please shed some light on this 
> Thanks,
> Madhawa



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add low level counter metrics for S3A; use in read performance tests

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Status: Patch Available  (was: Open)

> add low level counter metrics for S3A; use in read performance tests
> 
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, 
> HADOOP-13028-004.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13028) add low level counter metrics for S3A; use in read performance tests

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256884#comment-15256884
 ] 

Steve Loughran commented on HADOOP-13028:
-

Patch -004

includes HADOOP-13047; forward read range is configurable. Default is 64K; 
we'll need tests to work out what is good in different deployments (in-EC2; 
remote). For my tests, 640K looks right.

There's a lot of tests for the seek behaviour; seeks with no read to verify 
lazy seek, then some seek+read sequences to see how things slow down on 
different readahead values.

BTW, the readahead can be set on an open stream via 
{{CanSetReadahead.setReadahead(Long)}}; this could enable some code to 
dynamically tune things if it really knew what it was doing. I'm using it in 
the tests to simplify their setup.

> add low level counter metrics for S3A; use in read performance tests
> 
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, 
> HADOOP-13028-004.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add low level counter metrics for S3A; use in read performance tests

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Attachment: HADOOP-13028-004.patch

> add low level counter metrics for S3A; use in read performance tests
> 
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, 
> HADOOP-13028-004.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add low level counter metrics for S3A; use in read performance tests

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Summary: add low level counter metrics for S3A; use in read performance 
tests  (was: add counter and timer metrics for S3A HTTP & low-level operations)

> add low level counter metrics for S3A; use in read performance tests
> 
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add low level counter metrics for S3A; use in read performance tests

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Attachment: 
org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt

Performance run for Patch 004. Notice the inferred detail
{code}
An open() call has the equivalent duration of reading 686,641 bytes
{code}

That effectively defines the forward range I should have on my local system if 
I want to use the US landsat datasets as a source of data...it's faster to skip 
half a megabyte of data than it is to break the link to S3 and set it up again

> add low level counter metrics for S3A; use in read performance tests
> 
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HADOOP-12949) Add metrics and HTrace to the s3a connector

2016-04-25 Thread Madhawa Gunasekara (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-12949 started by Madhawa Gunasekara.
---
> Add metrics and HTrace to the s3a connector
> ---
>
> Key: HADOOP-12949
> URL: https://issues.apache.org/jira/browse/HADOOP-12949
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Reporter: Madhawa Gunasekara
>Assignee: Madhawa Gunasekara
>
> Hi All, 
> s3, GCS, WASB, and other cloud blob stores are becoming increasingly 
> important in Hadoop. But we don't have distributed tracing for these yet. It 
> would be interesting to add distributed tracing here. It would enable 
> collecting really interesting data like probability distributions of PUT and 
> GET requests to s3 and their impact on MR jobs, etc.
> I would like to implement this feature, Please shed some light on this 
> Thanks,
> Madhawa



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HADOOP-12949) Add metrics and HTrace to the s3a connector

2016-04-25 Thread Madhawa Gunasekara (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhawa Gunasekara reassigned HADOOP-12949:
---

Assignee: Madhawa Gunasekara

> Add metrics and HTrace to the s3a connector
> ---
>
> Key: HADOOP-12949
> URL: https://issues.apache.org/jira/browse/HADOOP-12949
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Reporter: Madhawa Gunasekara
>Assignee: Madhawa Gunasekara
>
> Hi All, 
> s3, GCS, WASB, and other cloud blob stores are becoming increasingly 
> important in Hadoop. But we don't have distributed tracing for these yet. It 
> would be interesting to add distributed tracing here. It would enable 
> collecting really interesting data like probability distributions of PUT and 
> GET requests to s3 and their impact on MR jobs, etc.
> I would like to implement this feature, Please shed some light on this 
> Thanks,
> Madhawa



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12101) Add automatic search of default Configuration variables to TestConfigurationFieldsBase

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256865#comment-15256865
 ] 

Hadoop QA commented on HADOOP-12101:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 3s {color} 
| {color:red} Docker failed to build yetus/hadoop:fbe3e86. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800606/HADOOP-12101.011.patch
 |
| JIRA Issue | HADOOP-12101 |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9173/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> Add automatic search of default Configuration variables to 
> TestConfigurationFieldsBase
> --
>
> Key: HADOOP-12101
> URL: https://issues.apache.org/jira/browse/HADOOP-12101
> Project: Hadoop Common
>  Issue Type: Test
>  Components: test
>Affects Versions: 2.7.0
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>  Labels: supportability
> Attachments: HADOOP-12101.001.patch, HADOOP-12101.002.patch, 
> HADOOP-12101.003.patch, HADOOP-12101.004.patch, HADOOP-12101.005.patch, 
> HADOOP-12101.006.patch, HADOOP-12101.007.patch, HADOOP-12101.008.patch, 
> HADOOP-12101.009.patch, HADOOP-12101.010.patch, HADOOP-12101.011.patch
>
>
> Add functionality given a Configuration variable FOO, to at least check the 
> xml file value against DEFAULT_FOO.
> Without waivers and a mapping for exceptions, this can probably never be a 
> test method that generates actual errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add counter and timer metrics for S3A HTTP & low-level operations

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Status: Open  (was: Patch Available)

> add counter and timer metrics for S3A HTTP & low-level operations
> -
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12101) Add automatic search of default Configuration variables to TestConfigurationFieldsBase

2016-04-25 Thread Ray Chiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang updated HADOOP-12101:

Attachment: HADOOP-12101.011.patch

- Same patch.  Re-launch Jenkins.

> Add automatic search of default Configuration variables to 
> TestConfigurationFieldsBase
> --
>
> Key: HADOOP-12101
> URL: https://issues.apache.org/jira/browse/HADOOP-12101
> Project: Hadoop Common
>  Issue Type: Test
>  Components: test
>Affects Versions: 2.7.0
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>  Labels: supportability
> Attachments: HADOOP-12101.001.patch, HADOOP-12101.002.patch, 
> HADOOP-12101.003.patch, HADOOP-12101.004.patch, HADOOP-12101.005.patch, 
> HADOOP-12101.006.patch, HADOOP-12101.007.patch, HADOOP-12101.008.patch, 
> HADOOP-12101.009.patch, HADOOP-12101.010.patch, HADOOP-12101.011.patch
>
>
> Add functionality given a Configuration variable FOO, to at least check the 
> xml file value against DEFAULT_FOO.
> Without waivers and a mapping for exceptions, this can probably never be a 
> test method that generates actual errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13039) Add documentation for configuration property ipc.maximum.data.length for controlling maximum RPC message size.

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256798#comment-15256798
 ] 

Hadoop QA commented on HADOOP-13039:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 3s {color} 
| {color:red} Docker failed to build yetus/hadoop:fbe3e86. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800588/HADOOP-13039.001.patch
 |
| JIRA Issue | HADOOP-13039 |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9171/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> Add documentation for configuration property ipc.maximum.data.length for 
> controlling maximum RPC message size.
> --
>
> Key: HADOOP-13039
> URL: https://issues.apache.org/jira/browse/HADOOP-13039
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13039.000.patch, HADOOP-13039.001.patch, 
> HADOOP-13039.001.patch
>
>
> The RPC server enforces a maximum length on incoming messages.  Messages 
> larger than the maximum are rejected immediately as potentially malicious.  
> The maximum length can be tuned by setting configuration property 
> {{ipc.maximum.data.length}}, but this is not documented in core-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12101) Add automatic search of default Configuration variables to TestConfigurationFieldsBase

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256796#comment-15256796
 ] 

Hadoop QA commented on HADOOP-12101:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 11m 2s 
{color} | {color:red} Docker failed to build yetus/hadoop:fbe3e86. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800592/HADOOP-12101.010.patch
 |
| JIRA Issue | HADOOP-12101 |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9170/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> Add automatic search of default Configuration variables to 
> TestConfigurationFieldsBase
> --
>
> Key: HADOOP-12101
> URL: https://issues.apache.org/jira/browse/HADOOP-12101
> Project: Hadoop Common
>  Issue Type: Test
>  Components: test
>Affects Versions: 2.7.0
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>  Labels: supportability
> Attachments: HADOOP-12101.001.patch, HADOOP-12101.002.patch, 
> HADOOP-12101.003.patch, HADOOP-12101.004.patch, HADOOP-12101.005.patch, 
> HADOOP-12101.006.patch, HADOOP-12101.007.patch, HADOOP-12101.008.patch, 
> HADOOP-12101.009.patch, HADOOP-12101.010.patch
>
>
> Add functionality given a Configuration variable FOO, to at least check the 
> xml file value against DEFAULT_FOO.
> Without waivers and a mapping for exceptions, this can probably never be a 
> test method that generates actual errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256792#comment-15256792
 ] 

Steve Loughran commented on HADOOP-12751:
-

turns out that precommits need to be explicitly triggered if the patch hasn't 
been updated (Yetus is getting cleverer)... I've just kicked one off

> While using kerberos Hadoop incorrectly assumes names with '@' to be 
> non-simple
> ---
>
> Key: HADOOP-12751
> URL: https://issues.apache.org/jira/browse/HADOOP-12751
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.2
> Environment: kerberos
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
>  Labels: kerberos
> Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0001-Remove-check-for-user-name-characters-and.patch, 
> 0002-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0003-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0004-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0005-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0006-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0007-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0007-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0008-HADOOP-12751-leave-user-validation-to-os.patch
>
>
> In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) 
> and Active Directory (ad.local) users can be made available on the OS level 
> by something like sssd. The trusted users will be of the form 'user@ad.local' 
> while other users are will not contain the domain. Executing 'id -Gn 
> user@ad.local' will successfully return the groups the user belongs to if 
> configured correctly. 
> However, it is assumed by Hadoop that users of the format with '@' cannot be 
> correct. This code is in KerberosName.java and seems to be a validator if the 
> 'auth_to_local' rules are applied correctly.
> In my opinion this should be removed or changed to a different kind of check 
> or maybe logged as a warning while still proceeding, as the current behavior 
> limits integration possibilities with other standard tools.
> Workaround are difficult to apply (by having a rewrite by system tools to for 
> example user_ad_local) due to down stream consequences.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12101) Add automatic search of default Configuration variables to TestConfigurationFieldsBase

2016-04-25 Thread Ray Chiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang updated HADOOP-12101:

Attachment: HADOOP-12101.010.patch

- Add support for TestCommonConfigurationFields
- Current fails TestYarnConfigurationFields due to YARN-4992


> Add automatic search of default Configuration variables to 
> TestConfigurationFieldsBase
> --
>
> Key: HADOOP-12101
> URL: https://issues.apache.org/jira/browse/HADOOP-12101
> Project: Hadoop Common
>  Issue Type: Test
>  Components: test
>Affects Versions: 2.7.0
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>  Labels: supportability
> Attachments: HADOOP-12101.001.patch, HADOOP-12101.002.patch, 
> HADOOP-12101.003.patch, HADOOP-12101.004.patch, HADOOP-12101.005.patch, 
> HADOOP-12101.006.patch, HADOOP-12101.007.patch, HADOOP-12101.008.patch, 
> HADOOP-12101.009.patch, HADOOP-12101.010.patch
>
>
> Add functionality given a Configuration variable FOO, to at least check the 
> xml file value against DEFAULT_FOO.
> Without waivers and a mapping for exceptions, this can probably never be a 
> test method that generates actual errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13039) Add documentation for configuration property ipc.maximum.data.length for controlling maximum RPC message size.

2016-04-25 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13039:
---
Attachment: HADOOP-13039.001.patch

Don't know why Jenkins was not triggered. Will upload the same patch again.

> Add documentation for configuration property ipc.maximum.data.length for 
> controlling maximum RPC message size.
> --
>
> Key: HADOOP-13039
> URL: https://issues.apache.org/jira/browse/HADOOP-13039
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13039.000.patch, HADOOP-13039.001.patch, 
> HADOOP-13039.001.patch
>
>
> The RPC server enforces a maximum length on incoming messages.  Messages 
> larger than the maximum are rejected immediately as potentially malicious.  
> The maximum length can be tuned by setting configuration property 
> {{ipc.maximum.data.length}}, but this is not documented in core-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12738) Create unit test to automatically compare Common related classes and core-default.xml

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256687#comment-15256687
 ] 

Hadoop QA commented on HADOOP-12738:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 28s 
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 12s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 54s 
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 1s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 48s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_77. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 52s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
23s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 42s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800560/HADOOP-12738.003.patch
 |
| JIRA Issue | HADOOP-12738 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 59d5afbc2d49 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 10f0f78 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |

[jira] [Commented] (HADOOP-5470) RunJar.unJar() should write the last modified time found in the jar entry to the uncompressed file

2016-04-25 Thread Arpit Agarwal (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256649#comment-15256649
 ] 

Arpit Agarwal commented on HADOOP-5470:
---

Hi [~boky01], I recommend logging a summary message with the number of failed 
files rather than one warning per file. v4 patch lgtm otherwise.

> RunJar.unJar() should write the last modified time found in the jar entry to 
> the uncompressed file
> --
>
> Key: HADOOP-5470
> URL: https://issues.apache.org/jira/browse/HADOOP-5470
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: util
>Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
>Reporter: Colin Evans
>Assignee: Andras Bokor
>Priority: Minor
>  Labels: newbie
> Attachments: HADOOP-5470.01.patch, HADOOP-5470.02.patch, 
> HADOOP-5470.03.patch, HADOOP-5470.04.patch
>
>
> For tools like jruby and jython, last modified times determine if a script 
> gets recompiled.  Losing the correct last modified time causes some 
> unfortunate recompilation race conditions when a job is running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13021) Hadoop swift driver unit test should use unique directory for each run

2016-04-25 Thread Chen He (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256638#comment-15256638
 ] 

Chen He commented on HADOOP-13021:
--

Thank you for the quick reply, [~steve_l]. I agree, we need to clean before and 
after tests. I will update the patch.

> Hadoop swift driver unit test should use unique directory for each run
> --
>
> Key: HADOOP-13021
> URL: https://issues.apache.org/jira/browse/HADOOP-13021
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.7.2
>Reporter: Chen He
>Assignee: Chen He
>  Labels: unit-test
> Attachments: HADOOP-13021.001.patch
>
>
> Since all "unit test" in swift package are actually functionality test, it 
> requires server's information in the core-site.xml file. However, multiple 
> unit test runs on difference machines using the same core-site.xml file will 
> result in some unit tests failure. For example:
> In TestSwiftFileSystemBasicOps.java
> public void testMkDir() throws Throwable {
> Path path = new Path("/test/MkDir");
> fs.mkdirs(path);
> //success then -so try a recursive operation
> fs.delete(path, true);
>   }
> It is possible that machine A and B are running "mvn clean install" using 
> same core-site.xml file. However, machine A run testMkDir() first and delete 
> the dir, but machine B just tried to run fs.delete(path,true). It will report 
> failure. This is just an example. There are many similar cases in the unit 
> test sets. I would propose we use a unique dir for each unit test run instead 
> of using "Path path = new Path("/test/MkDir")" for all concurrent runs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12345) Credential length in CredentialsSys.java incorrect

2016-04-25 Thread Ray Chiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256611#comment-15256611
 ] 

Ray Chiang commented on HADOOP-12345:
-

Given that no patch has been submitted for 8 months, I'd recommend changing 
Target Version to 2.9.0 or getting something up for review ASAP.

> Credential length in CredentialsSys.java incorrect
> --
>
> Key: HADOOP-12345
> URL: https://issues.apache.org/jira/browse/HADOOP-12345
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: nfs
>Affects Versions: 2.7.0
>Reporter: Pradeep Nayak Udupi Kadbet
>Assignee: S M Al Mamun
>Priority: Blocker
>
> Hi -
> There is a bug in the way hadoop-nfs sets the credential length in 
> "Credentials" field of the NFS RPC packet when using AUTH_SYS
> In CredentialsSys.java, when we are writing the creds in to XDR object, we 
> set the length as follows:
>  // mStamp + mHostName.length + mHostName + mUID + mGID + mAuxGIDs.count
> 96 mCredentialsLength = 20 + mHostName.getBytes().length;
> (20 corresponds to 4 bytes for mStamp, 4 bytes for mUID, 4 bytes for mGID, 4 
> bytes for length field of hostname, 4 bytes for number of aux 4 gids) and 
> this is okay.
> However when we add the length of the hostname to this, we are not adding the 
> extra padded bytes for the hostname (If the length is not a multiple of 4) 
> and thus when the NFS server reads the packet, it returns GARBAGE_ARGS 
> because it doesn't read the uid field when it is expected to read. I can 
> reproduce this issue constantly on machines where the hostname length is not 
> a multiple of 4.
> A possible fix is to do something this:
> int pad = mHostName.getBytes().length % 4;
>  // mStamp + mHostName.length + mHostName + mUID + mGID + mAuxGIDs.count
> mCredentialsLength = 20 + mHostName.getBytes().length + pad;
> I would be happy to submit the patch but I need some help to commit into 
> mainline. I haven't committed into Hadoop yet.
> Cheers!
> Pradeep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12738) Create unit test to automatically compare Common related classes and core-default.xml

2016-04-25 Thread Ray Chiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang updated HADOOP-12738:

Attachment: HADOOP-12738.003.patch

- Removed HADOOP-12101 patch bits

> Create unit test to automatically compare Common related classes and 
> core-default.xml
> -
>
> Key: HADOOP-12738
> URL: https://issues.apache.org/jira/browse/HADOOP-12738
> Project: Hadoop Common
>  Issue Type: Test
>Affects Versions: 2.7.1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Minor
>  Labels: supportability
> Attachments: HADOOP-12738.001.patch, HADOOP-12738.002.patch, 
> HADOOP-12738.003.patch
>
>
> Create a unit test that will automatically compare the fields in the various 
> Common related classes and core-default.xml. It should throw an error if a 
> property is missing in either the class or the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13022) S3 MD5 check fails on Server Side Encryption with AWS and default key is used

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256551#comment-15256551
 ] 

Steve Loughran commented on HADOOP-13022:
-

This depends on HADOOP-13050. However, the restrictions there (http component 
versions) also apply

> S3 MD5 check fails on Server Side Encryption with AWS and default key is used
> -
>
> Key: HADOOP-13022
> URL: https://issues.apache.org/jira/browse/HADOOP-13022
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Leonardo Contreras
>
> When server side encryption with "aws:kms" value and no custom key is used in 
> S3A Filesystem, the AWSClient fails when verifing Md5:
> {noformat}
> Exception in thread "main" com.amazonaws.AmazonClientException: Unable to 
> verify integrity of data upload.  Client calculated content hash (contentMD5: 
> 1B2M2Y8AsgTpgAmY7PhCfg== in base 64) didn't match hash (etag: 
> c29fcc646e17c348bce9cca8f9d205f5 in hex) calculated by Amazon S3.  You may 
> need to delete the data stored in Amazon S3. (metadata.contentMD5: null, 
> md5DigestStream: 
> com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@65d9e72a, 
> bucketName: abuse-messages-nonprod, key: 
> venus/raw_events/checkpoint/825eb6aa-543d-46b1-801f-42de9dbc1610/)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1492)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:1295)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:1272)
>   at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:969)
>   at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1888)
>   at 
> org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2077)
>   at 
> org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2074)
>   at scala.Option.map(Option.scala:145)
>   at 
> org.apache.spark.SparkContext.setCheckpointDir(SparkContext.scala:2074)
>   at 
> org.apache.spark.streaming.StreamingContext.checkpoint(StreamingContext.scala:237)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple

2016-04-25 Thread Bolke de Bruin (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256532#comment-15256532
 ] 

Bolke de Bruin commented on HADOOP-12751:
-

How are we doing here?

> While using kerberos Hadoop incorrectly assumes names with '@' to be 
> non-simple
> ---
>
> Key: HADOOP-12751
> URL: https://issues.apache.org/jira/browse/HADOOP-12751
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.2
> Environment: kerberos
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
>  Labels: kerberos
> Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0001-Remove-check-for-user-name-characters-and.patch, 
> 0002-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0003-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0004-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0005-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0006-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0007-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0007-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0008-HADOOP-12751-leave-user-validation-to-os.patch
>
>
> In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) 
> and Active Directory (ad.local) users can be made available on the OS level 
> by something like sssd. The trusted users will be of the form 'user@ad.local' 
> while other users are will not contain the domain. Executing 'id -Gn 
> user@ad.local' will successfully return the groups the user belongs to if 
> configured correctly. 
> However, it is assumed by Hadoop that users of the format with '@' cannot be 
> correct. This code is in KerberosName.java and seems to be a validator if the 
> 'auth_to_local' rules are applied correctly.
> In my opinion this should be removed or changed to a different kind of check 
> or maybe logged as a warning while still proceeding, as the current behavior 
> limits integration possibilities with other standard tools.
> Workaround are difficult to apply (by having a rewrite by system tools to for 
> example user_ad_local) due to down stream consequences.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13047) S3a Forward seek in stream length to be configurable

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256525#comment-15256525
 ] 

Steve Loughran commented on HADOOP-13047:
-

HADOOP-13028 has an attachment which compares read bandwidths to open 
time...from the numbers locally to the US, I'm seeing a forward skip of < 800K 
would be faster than closing and reopening the connection —and that's ignoring 
the effects of TCP slow start on followon reads

{code}
2016-04-25 17:01:09,584 INFO  scale.S3AScaleTestBase 
(S3AScaleTestBase.java:end(181)) - Duration of Open stream: 164,965,000 nS
2016-04-25 17:01:13,613 INFO  scale.S3AScaleTestBase 
(S3AScaleTestBase.java:end(181)) - Duration of Time to read 20405780 bytes: 
4,029,203,000 nS
2016-04-25 17:01:13,614 INFO  scale.TestS3AInputStreamPerformance 
(TestS3AInputStreamPerformance.java:bandwidth(135)) - Bandwidth = 5.064471  MB/S
2016-04-25 17:01:13,614 INFO  scale.TestS3AInputStreamPerformance 
(TestS3AInputStreamPerformance.java:testTimeToOpenAndReadWholeFileByByte(97)) - 
An open() call has the equivalent duration of reading 837,385 bytes

{code}

> S3a Forward seek in stream length to be configurable
> 
>
> Key: HADOOP-13047
> URL: https://issues.apache.org/jira/browse/HADOOP-13047
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
> Attachments: HADOOP-13047.WIP.2.patch, HADOOP-13047.WIP.patch
>
>
> Even with lazy seek, tests can show that sometimes a short-distance forward 
> seek is triggering a close + reopen, because the threshold for the seek is 
> simply available bytes in the inner stream.
> A configurable threshold would allow data to be read and discarded before 
> that seek. This should be beneficial over long-haul networks as the time to 
> set up the TCP channel is high, and TCP-slow-start means that the ramp up of 
> bandwidth is slow. In such deployments, it will better to read forward than 
> re-open, though the exact "best" number will vary with client and endpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13047) S3a Forward seek in stream length to be configurable

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256495#comment-15256495
 ] 

Steve Loughran commented on HADOOP-13047:
-

# I like the simplicity of this being a buffer.
# There's no need to actually have a separate read+discard loop to go past the 
available bytes; .skip() will do that, simply blocking until the data is ready. 
(Though a sanity check at the end is probably wise)
# I'd like that buffer size to be settable via setReadahead(). Why? It would 
allow applications which *really* knew what they were doing to actually set 
readahead for their algorithms. Though I don't see any evidence of that method 
being used anywhere in the Hadoop codebase —does anyone have any data on how 
often code uses it?
# Patch -002 of HADOOP-13028 exposes the statistics of a stream via a visible 
for testing method...this would all you to write a test which verified that a 
forward seek within the buffer range worked. This is where that 
{{setReadahead()}} call would really be useful, as you could change the buffer 
size and verify that forward seeks simply increment the counter of bytes 
skipped on seek; the number of stream close/open events would be unchanged. 

> S3a Forward seek in stream length to be configurable
> 
>
> Key: HADOOP-13047
> URL: https://issues.apache.org/jira/browse/HADOOP-13047
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
> Attachments: HADOOP-13047.WIP.2.patch, HADOOP-13047.WIP.patch
>
>
> Even with lazy seek, tests can show that sometimes a short-distance forward 
> seek is triggering a close + reopen, because the threshold for the seek is 
> simply available bytes in the inner stream.
> A configurable threshold would allow data to be read and discarded before 
> that seek. This should be beneficial over long-haul networks as the time to 
> set up the TCP channel is high, and TCP-slow-start means that the ramp up of 
> bandwidth is slow. In such deployments, it will better to read forward than 
> re-open, though the exact "best" number will vary with client and endpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12738) Create unit test to automatically compare Common related classes and core-default.xml

2016-04-25 Thread Ray Chiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256482#comment-15256482
 ] 

Ray Chiang commented on HADOOP-12738:
-

Oh, that's my bad--I usually put it in my local tree for purposes of running 
the test, but usually exclude it from the patch.  I will resubmit the clean 
patch.

> Create unit test to automatically compare Common related classes and 
> core-default.xml
> -
>
> Key: HADOOP-12738
> URL: https://issues.apache.org/jira/browse/HADOOP-12738
> Project: Hadoop Common
>  Issue Type: Test
>Affects Versions: 2.7.1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Minor
>  Labels: supportability
> Attachments: HADOOP-12738.001.patch, HADOOP-12738.002.patch
>
>
> Create a unit test that will automatically compare the fields in the various 
> Common related classes and core-default.xml. It should throw an error if a 
> property is missing in either the class or the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add counter and timer metrics for S3A HTTP & low-level operations

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Attachment: HADOOP-13028-002.patch

> add counter and timer metrics for S3A HTTP & low-level operations
> -
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add counter and timer metrics for S3A HTTP & low-level operations

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Status: Patch Available  (was: In Progress)

> add counter and timer metrics for S3A HTTP & low-level operations
> -
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add counter and timer metrics for S3A HTTP & low-level operations

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Attachment: 
org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt

The file {{TestS3AInputStreamPerformance}} shows the result of a test run. 
There's a fairly big variance in times of operations, which could be a 
combination of buffering and network delays; someone should test this in EC2 
itself to see what shows up. For anyone trying to optimise things, those 
counters of times skipped, times incomplete, etc, are the ones which provide 
more deterministic results

> add counter and timer metrics for S3A HTTP & low-level operations
> -
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, 
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add counter and timer metrics for S3A HTTP & low-level operations

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Attachment: (was: HADOOP-13028-002.patch)

> add counter and timer metrics for S3A HTTP & low-level operations
> -
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add counter and timer metrics for S3A HTTP & low-level operations

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Priority: Major  (was: Minor)

> add counter and timer metrics for S3A HTTP & low-level operations
> -
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13028) add counter and timer metrics for S3A HTTP & low-level operations

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13028:

Attachment: HADOOP-13028-002.patch

HADOOP-13028 patch 002:
 -include fix for HADOOP-13058: "S3A FS fails during init against a read-only 
FS if multipart purge is enabled"
-include fix for HADOOP-13059 "S3a over-reacts to potentially transient network 
problems in its init() logic"
 -add more counters for # of read operations, readfully, readincomplete (i.e. 
when you got less back than you asked for)
 -the statistics counters for the new operations include some of the arguments; 
these are currently ignored, but would permit more detailed collection 
(histograms )
 -There's a test, {{TestS3AInputStreamPerformance}} which runs operations 
against a (configurable) large file; default is a public 20MB AWS landsat 
CSV.gz file. This file path can be changed to something else for testing 
against other infrastructures
 -FSDataInputStream extends its toString() call to include that of the wrapped 
stream; this allows those streams which add details to get pulled in.


> add counter and timer metrics for S3A HTTP & low-level operations
> -
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch
>
>
> against S3 (and other object stores), opening connections can be expensive, 
> closing connections may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of 
> open/close/failure+reconnect operations, timers of how long things take. This 
> can be used downstream to measure efficiency of the code (how often 
> connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13059) S3a over-reacts to potentially transient network problems in its init() logic

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256420#comment-15256420
 ] 

Steve Loughran commented on HADOOP-13059:
-

{code}
2016-04-25 15:42:03,710 INFO  http.AmazonHttpClient 
(AmazonHttpClient.java:executeHelper(496)) - Unable to execute HTTP request: 
tests3neu.s3.amazonaws.com
java.net.UnknownHostException: tests3neu.s3.amazonaws.com
at java.net.InetAddress.getAllByName0(InetAddress.java:1250)
at java.net.InetAddress.getAllByName(InetAddress.java:1162)
at java.net.InetAddress.getAllByName(InetAddress.java:1098)
at 
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:44)
at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:259)
at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:159)
at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at 
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728)
at 
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at 
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107)
at 
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:240)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2786)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2823)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2805)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:381)
at 
org.apache.spark.cloud.CloudSuite.createFilesystem(CloudSuite.scala:141)
at org.apache.spark.cloud.s3.S3aIOSuite.init(S3aIOSuite.scala:55)
at org.apache.spark.cloud.s3.S3aIOSuite.(S3aIOSuite.scala:43)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:379)
at 
org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:69)
at 
org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:38)
at 
org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:37)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
{code}

> S3a over-reacts to potentially transient network problems in its init() logic
> -
>
> Key: HADOOP-13059
> URL: https://issues.apache.org/jira/browse/HADOOP-13059
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> If there's a reason for s3a not being able to connect to AWS, then the 
> constructor fails, even if this is a potentially transient event.
> This happens because the code to check for a bucket existing will relay the 
> exceptions.
> The constructor should catch IOEs against the remote FS, downgrade to warn 
> and let the code continue; it may fail later, but it may also recover.



--
This message was

[jira] [Created] (HADOOP-13059) S3a over-reacts to potentially transient network problems in its init() logic

2016-04-25 Thread Steve Loughran (JIRA)

Steve Loughran created HADOOP-13059:
---

 Summary: S3a over-reacts to potentially transient network problems 
in its init() logic
 Key: HADOOP-13059
 URL: https://issues.apache.org/jira/browse/HADOOP-13059
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran


If there's a reason for s3a not being able to connect to AWS, then the 
constructor fails, even if this is a potentially transient event.

This happens because the code to check for a bucket existing will relay the 
exceptions.

The constructor should catch IOEs against the remote FS, downgrade to warn and 
let the code continue; it may fail later, but it may also recover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-13058) S3A FS fails during init against a read-only FS if multipart purge is enabled

2016-04-25 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13058:

Summary: S3A FS fails during init against a read-only FS if multipart purge 
is enabled  (was: S3A FS failes during init against a read-only FS if multipart 
purge is enabled)

> S3A FS fails during init against a read-only FS if multipart purge is enabled
> -
>
> Key: HADOOP-13058
> URL: https://issues.apache.org/jira/browse/HADOOP-13058
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> If you try to open a read-only filesystem, and the multipart upload option is 
> set to purge existing uploads, then the FS fails to load with an access 
> denied exception.
> it should catch the exception, downgrade to a debug and await until a file 
> write operation for access exceptions to reject on access rights.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11418) Property "io.compression.codec.lzo.class" does not work with other value besides default

2016-04-25 Thread Akira AJISAKA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256378#comment-15256378
 ] 

Akira AJISAKA commented on HADOOP-11418:


Thanks Masatake!

> Property "io.compression.codec.lzo.class" does not work with other value 
> besides default
> 
>
> Key: HADOOP-11418
> URL: https://issues.apache.org/jira/browse/HADOOP-11418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io
>Affects Versions: 2.6.0
>Reporter: fang fang chen
>Assignee: Yuanbo Liu
>  Labels: BB2015-05-RFC
> Fix For: 2.8.0
>
> Attachments: HADOOP-11418-004.patch, HADOOP-11418-1.patch, 
> HADOOP-11418-2.patch, HADOOP-11418-3.patch, HADOOP-11418.005.patch, 
> HADOOP-11418.006.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for 
> other codec besides default. Hadoop will always treat it as defaultClazz. I 
> think it is a bug. Please let me know if this is a work as design thing. 
> Thanks
>  77   private static final String defaultClazz =
>  78   "org.apache.hadoop.io.compress.LzoCodec";
>  82   public synchronized boolean isSupported() {
>  83 if (!checked) {
>  84   checked = true;
>  85   String extClazz =
>  86   (conf.get(CONF_LZO_CLASS) == null ? System
>  87   .getProperty(CONF_LZO_CLASS) : null);
>  88   String clazz = (extClazz != null) ? extClazz : defaultClazz;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11418) Property "io.compression.codec.lzo.class" does not work with other value besides default

2016-04-25 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HADOOP-11418:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

Committed. Thanks, [~yuanbo], [~fang fang chen] and [~ajisakaa].

> Property "io.compression.codec.lzo.class" does not work with other value 
> besides default
> 
>
> Key: HADOOP-11418
> URL: https://issues.apache.org/jira/browse/HADOOP-11418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io
>Affects Versions: 2.6.0
>Reporter: fang fang chen
>Assignee: Yuanbo Liu
>  Labels: BB2015-05-RFC
> Fix For: 2.8.0
>
> Attachments: HADOOP-11418-004.patch, HADOOP-11418-1.patch, 
> HADOOP-11418-2.patch, HADOOP-11418-3.patch, HADOOP-11418.005.patch, 
> HADOOP-11418.006.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for 
> other codec besides default. Hadoop will always treat it as defaultClazz. I 
> think it is a bug. Please let me know if this is a work as design thing. 
> Thanks
>  77   private static final String defaultClazz =
>  78   "org.apache.hadoop.io.compress.LzoCodec";
>  82   public synchronized boolean isSupported() {
>  83 if (!checked) {
>  84   checked = true;
>  85   String extClazz =
>  86   (conf.get(CONF_LZO_CLASS) == null ? System
>  87   .getProperty(CONF_LZO_CLASS) : null);
>  88   String clazz = (extClazz != null) ? extClazz : defaultClazz;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11418) Property "io.compression.codec.lzo.class" does not work with other value besides default

2016-04-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256371#comment-15256371
 ] 

Hudson commented on HADOOP-11418:
-

FAILURE: Integrated in Hadoop-trunk-Commit #9663 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/9663/])
HADOOP-11418. Property "io.compression.codec.lzo.class" does not work 
(iwasakims: rev bec5b4cd8c1fb27a5b707d601b508224a9c107ed)
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Compression.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestCompression.java


> Property "io.compression.codec.lzo.class" does not work with other value 
> besides default
> 
>
> Key: HADOOP-11418
> URL: https://issues.apache.org/jira/browse/HADOOP-11418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io
>Affects Versions: 2.6.0
>Reporter: fang fang chen
>Assignee: Yuanbo Liu
>  Labels: BB2015-05-RFC
> Attachments: HADOOP-11418-004.patch, HADOOP-11418-1.patch, 
> HADOOP-11418-2.patch, HADOOP-11418-3.patch, HADOOP-11418.005.patch, 
> HADOOP-11418.006.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for 
> other codec besides default. Hadoop will always treat it as defaultClazz. I 
> think it is a bug. Please let me know if this is a work as design thing. 
> Thanks
>  77   private static final String defaultClazz =
>  78   "org.apache.hadoop.io.compress.LzoCodec";
>  82   public synchronized boolean isSupported() {
>  83 if (!checked) {
>  84   checked = true;
>  85   String extClazz =
>  86   (conf.get(CONF_LZO_CLASS) == null ? System
>  87   .getProperty(CONF_LZO_CLASS) : null);
>  88   String clazz = (extClazz != null) ? extClazz : defaultClazz;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11418) Property "io.compression.codec.lzo.class" does not work with other value besides default

2016-04-25 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256350#comment-15256350
 ] 

Masatake Iwasaki commented on HADOOP-11418:
---

+1 from me. Committing this.

> Property "io.compression.codec.lzo.class" does not work with other value 
> besides default
> 
>
> Key: HADOOP-11418
> URL: https://issues.apache.org/jira/browse/HADOOP-11418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io
>Affects Versions: 2.6.0
>Reporter: fang fang chen
>Assignee: Yuanbo Liu
>  Labels: BB2015-05-RFC
> Attachments: HADOOP-11418-004.patch, HADOOP-11418-1.patch, 
> HADOOP-11418-2.patch, HADOOP-11418-3.patch, HADOOP-11418.005.patch, 
> HADOOP-11418.006.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for 
> other codec besides default. Hadoop will always treat it as defaultClazz. I 
> think it is a bug. Please let me know if this is a work as design thing. 
> Thanks
>  77   private static final String defaultClazz =
>  78   "org.apache.hadoop.io.compress.LzoCodec";
>  82   public synchronized boolean isSupported() {
>  83 if (!checked) {
>  84   checked = true;
>  85   String extClazz =
>  86   (conf.get(CONF_LZO_CLASS) == null ? System
>  87   .getProperty(CONF_LZO_CLASS) : null);
>  88   String clazz = (extClazz != null) ? extClazz : defaultClazz;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13058) S3A FS failes during init against a read-only FS if multipart purge is enabled

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256332#comment-15256332
 ] 

Steve Loughran commented on HADOOP-13058:
-

{code}
, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.281 sec <<< FAILURE! - in 
org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance
testTimeToOpenAndReadWholeFileByByte(org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance)
  Time elapsed: 6.973 sec  <<< ERROR!
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: 
Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 
8ECDC7355F5EFCCC)
at 
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at 
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at 
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at 
com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2664)
at 
com.amazonaws.services.s3.transfer.TransferManager.abortMultipartUploads(TransferManager.java:1222)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.initMultipartUploads(S3AFileSystem.java:349)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:244)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2786)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2823)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2811)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:428)
at 
org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance.openFS(TestS3AInputStreamPerformance.java:52)
{code}

> S3A FS failes during init against a read-only FS if multipart purge is enabled
> --
>
> Key: HADOOP-13058
> URL: https://issues.apache.org/jira/browse/HADOOP-13058
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> If you try to open a read-only filesystem, and the multipart upload option is 
> set to purge existing uploads, then the FS fails to load with an access 
> denied exception.
> it should catch the exception, downgrade to a debug and await until a file 
> write operation for access exceptions to reject on access rights.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-13058) S3A FS failes during init against a read-only FS if multipart purge is enabled

2016-04-25 Thread Steve Loughran (JIRA)

Steve Loughran created HADOOP-13058:
---

 Summary: S3A FS failes during init against a read-only FS if 
multipart purge is enabled
 Key: HADOOP-13058
 URL: https://issues.apache.org/jira/browse/HADOOP-13058
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran


If you try to open a read-only filesystem, and the multipart upload option is 
set to purge existing uploads, then the FS fails to load with an access denied 
exception.

it should catch the exception, downgrade to a debug and await until a file 
write operation for access exceptions to reject on access rights.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12579) Deprecate and remove WriteableRPCEngine

2016-04-25 Thread Kai Zheng (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256241#comment-15256241
 ] 

Kai Zheng commented on HADOOP-12579:


The failed tests are not relevant. The remained check styles are not suitable 
to be fixed.
The patch is ready for review.

> Deprecate and remove WriteableRPCEngine
> ---
>
> Key: HADOOP-12579
> URL: https://issues.apache.org/jira/browse/HADOOP-12579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Haohui Mai
> Attachments: HADOOP-12579-v1.patch, HADOOP-12579-v3.patch, 
> HADOOP-12579-v4.patch, HADOOP-12579-v5.patch, HADOOP-12579-v6.patch, 
> HADOOP-12579-v7.patch
>
>
> The {{WriteableRPCEninge}} depends on Java's serialization mechanisms for RPC 
> requests. Without proper checks, it has be shown that it can lead to security 
> vulnerabilities such as remote code execution (e.g., COLLECTIONS-580, 
> HADOOP-12577).
> The current implementation has migrated from {{WriteableRPCEngine}} to 
> {{ProtobufRPCEngine}} now. This jira proposes to deprecate 
> {{WriteableRPCEngine}} in branch-2 and to remove it in trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12666) Support Microsoft Azure Data Lake - as a file system in Hadoop

2016-04-25 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256185#comment-15256185
 ] 

Vishwajeet Dusane commented on HADOOP-12666:


[~cnauroth], [~chris.douglas], [~twu], [~fabbri] and [~mackrorysd]  - Could you 
please let me know if you have any more comments that is not addressed so far ? 
Thanks a lot again for the review.



> Support Microsoft Azure Data Lake - as a file system in Hadoop
> --
>
> Key: HADOOP-12666
> URL: https://issues.apache.org/jira/browse/HADOOP-12666
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: Create_Read_Hadoop_Adl_Store_Semantics.pdf, 
> HADOOP-12666-002.patch, HADOOP-12666-003.patch, HADOOP-12666-004.patch, 
> HADOOP-12666-005.patch, HADOOP-12666-006.patch, HADOOP-12666-007.patch, 
> HADOOP-12666-008.patch, HADOOP-12666-009.patch, HADOOP-12666-010.patch, 
> HADOOP-12666-1.patch
>
>   Original Estimate: 336h
>  Time Spent: 336h
>  Remaining Estimate: 0h
>
> h2. Description
> This JIRA describes a new file system implementation for accessing Microsoft 
> Azure Data Lake Store (ADL) from within Hadoop. This would enable existing 
> Hadoop applications such has MR, HIVE, Hbase etc..,  to use ADL store as 
> input or output.
>  
> ADL is ultra-high capacity, Optimized for massive throughput with rich 
> management and security features. More details available at 
> https://azure.microsoft.com/en-us/services/data-lake-store/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256146#comment-15256146
 ] 

Steve Loughran commented on HADOOP-12756:
-

# I agree: name change, especially as OSS is also the acronym "Open Source 
Software". Make it hadoo-aliyun (or some other obvious name). Keeping it bigger 
than just OSS allows for more features for the platform to go in later
# There's an open JIRA on incrementing http components; HADOOP-12767; I'm 
expecting this to go in for Hadoop 2.9. which is what this patch can target 
(hence: work directly with hadoop-trunk for your dev & patches, not branch-2.8)
# All version dependencies must be declared in {{hadoop-project/pom.xml}}; it's 
how we make sure versions are consistent.
# regarding passing down usernames, this must be done via the 
{{test/resources/auth-keys.xml}} file. Look at the aws or openstack modules to 
see how the tests are automatically skipped if undefined. See also [how to keep 
your credentials 
private|http://steveloughran.blogspot.co.uk/2016/04/testing-against-s3-and-object-stores.html].
 Using the hadoop XML files lets you also test credential provider integration, 
which we'll also expect
# Have a look at the s3a work, especially those items in phase i: 
stabilisation, HADOOP-11571, Make sure that the patch avoids those same 
problems (e.g how to close vs abort streams, swallowing FileNotFoundExceptions 
during the final delete phase). S3a phase II, HADOOP-11694 contains some other 
bugs, but is otherwise performance work. It's probably best to wait one 
iteration before doing the performance version, get things stable first.
# nice to see all the tests!










> Incorporate Aliyun OSS file system implementation
> -
>
> Key: HADOOP-12756
> URL: https://issues.apache.org/jira/browse/HADOOP-12756
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Reporter: shimingfei
>Assignee: shimingfei
> Attachments: 0001-OSS-filesystem-integration-with-Hadoop.patch, HCFS 
> User manual.md, OSS integration.pdf, OSS integration.pdf
>
>
> Aliyun OSS is widely used among China’s cloud users, but currently it is not 
> easy to access data laid on OSS storage from user’s Hadoop/Spark application, 
> because of no original support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, 
> Spark/Hadoop applications can read/write data from OSS without any code 
> change. Narrowing the gap between user’s APP and data storage, like what have 
> been done for S3 in Hadoop 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13021) Hadoop swift driver unit test should use unique directory for each run

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256136#comment-15256136
 ] 

Steve Loughran commented on HADOOP-13021:
-

I'd recommend that the tests clean up before as well as after then, then do a 
bit of spin to verify the path is gone. That way they can recover.

I have to do something simiilar with yarn functional tests: clean up any 
previous app instances in case they were left around

> Hadoop swift driver unit test should use unique directory for each run
> --
>
> Key: HADOOP-13021
> URL: https://issues.apache.org/jira/browse/HADOOP-13021
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.7.2
>Reporter: Chen He
>Assignee: Chen He
>  Labels: unit-test
> Attachments: HADOOP-13021.001.patch
>
>
> Since all "unit test" in swift package are actually functionality test, it 
> requires server's information in the core-site.xml file. However, multiple 
> unit test runs on difference machines using the same core-site.xml file will 
> result in some unit tests failure. For example:
> In TestSwiftFileSystemBasicOps.java
> public void testMkDir() throws Throwable {
> Path path = new Path("/test/MkDir");
> fs.mkdirs(path);
> //success then -so try a recursive operation
> fs.delete(path, true);
>   }
> It is possible that machine A and B are running "mvn clean install" using 
> same core-site.xml file. However, machine A run testMkDir() first and delete 
> the dir, but machine B just tried to run fs.delete(path,true). It will report 
> failure. This is just an example. There are many similar cases in the unit 
> test sets. I would propose we use a unique dir for each unit test run instead 
> of using "Path path = new Path("/test/MkDir")" for all concurrent runs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9613) [JDK8] Update jersey version to latest 1.x release

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256134#comment-15256134
 ] 

Steve Loughran commented on HADOOP-9613:


Did you meant to say common-dev@hadoop, hdfs-def, yarn-dev, mr-dev? That's what 
I was thinking of. YARN does a lot of web stuff.

The main thing everyone needs to know is: this upgrade is inevitable, we've 
held back from fear, especially of HDFS access, but that's now Netty-based, so 
not an issue

> [JDK8] Update jersey version to latest 1.x release
> --
>
> Key: HADOOP-9613
> URL: https://issues.apache.org/jira/browse/HADOOP-9613
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Timothy St. Clair
>Assignee: Tsuyoshi Ozawa
>  Labels: maven
> Attachments: HADOOP-2.2.0-9613.patch, 
> HADOOP-9613.004.incompatible.patch, HADOOP-9613.005.incompatible.patch, 
> HADOOP-9613.006.incompatible.patch, HADOOP-9613.007.incompatible.patch, 
> HADOOP-9613.008.incompatible.patch, HADOOP-9613.009.incompatible.patch, 
> HADOOP-9613.010.incompatible.patch, HADOOP-9613.011.incompatible.patch, 
> HADOOP-9613.012.incompatible.patch, HADOOP-9613.013.incompatible.patch, 
> HADOOP-9613.1.patch, HADOOP-9613.2.patch, HADOOP-9613.3.patch, 
> HADOOP-9613.patch
>
>
> Update pom.xml dependencies exposed when running a mvn-rpmbuild against 
> system dependencies on Fedora 18.  
> The existing version is 1.8 which is quite old. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256133#comment-15256133
 ] 

Steve Loughran commented on HADOOP-13018:
-

Looks great! 

One question: why the reflection? There's nothing that can be done with class 
casting?

> Make Kdiag check whether hadoop.token.files points to existent and valid files
> --
>
> Key: HADOOP-13018
> URL: https://issues.apache.org/jira/browse/HADOOP-13018
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HADOOP-13018.01.patch, HADOOP-13018.02.patch
>
>
> Steve proposed that KDiag should fail fast to help debug the case that 
> hadoop.token.files points to a file not found. This JIRA is to affect that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-13056) Print expected values when rejecting a server's determined principal

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256131#comment-15256131
 ] 

Steve Loughran commented on HADOOP-13056:
-

Test failures *appear* unrelated

> Print expected values when rejecting a server's determined principal
> 
>
> Key: HADOOP-13056
> URL: https://issues.apache.org/jira/browse/HADOOP-13056
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.5.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: HADOOP-13056.000.patch
>
>
> When an address-constructed service principal by a client does not match a 
> provided pattern or the configured principal property, the error is very 
> uninformative on what the specific cause is. Currently the only error printed 
> is, in both cases:
> {code}
>  java.lang.IllegalArgumentException: Server has invalid Kerberos principal: 
> hdfs/host.internal@REALM
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12844) Recover when S3A fails on IOException in read()

2016-04-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256121#comment-15256121
 ] 

Steve Loughran commented on HADOOP-12844:
-

I've pulled this in to HADOOP-13028

The mechanism is actually simpler than this patch, because those socket 
exceptions already handled are IOEs.

All that's needed is
# expand that clause to catch to all IOEs
# move it after the {{EOFException}} as that's the exception we don't want to 
retry.

I'm also now counting how often this event happens on a stream.

> Recover when S3A fails on IOException in read()
> ---
>
> Key: HADOOP-12844
> URL: https://issues.apache.org/jira/browse/HADOOP-12844
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.7.1, 2.7.2
>Reporter: Pieter Reuse
>Assignee: Pieter Reuse
> Attachments: HADOOP-12844.001.patch
>
>
> This simple patch catches IOExceptions in S3AInputStream.read(byte[] buf, int 
> off, int len) and reopens the connection on the same location as it was 
> before the exception.
> This is similar to the functionality introduced in S3N in 
> [HADOOP-6254|https://issues.apache.org/jira/browse/HADOOP-6254], for exactly 
> the same reason.
> Patch developed in cooperation with [~emres].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12579) Deprecate and remove WriteableRPCEngine

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256112#comment-15256112
 ] 

Hadoop QA commented on HADOOP-12579:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 12 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 49s 
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 42s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 9s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 7s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s 
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 5s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 16s 
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m 16s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 16s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 24s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m 24s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 24s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 16s 
{color} | {color:red} root: patch generated 3 new + 579 unchanged - 28 fixed = 
582 total (was 607) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 21s 
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 17s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 33s {color} 
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 3s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 34s 
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed with JDK 
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 2s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. 
{color} |
| {color:red}-1{color} |

[jira] [Commented] (HADOOP-13046) Fix hadoop-dist to adapt to HDFS client library separation

2016-04-25 Thread Teruyoshi Zenmyo (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256048#comment-15256048
 ] 

Teruyoshi Zenmyo commented on HADOOP-13046:
---

The patch only modifies build settings.
I have confirmed hadoop-dist layout is changed as below by this patch.

{code}
 sdiff -s before after
  > include/hdfs.h
  > 
lib/native/libhdfs.a
  > 
lib/native/libhdfs.so
  > 
lib/native/libhdfs.so.0.0.0
  > 
share/hadoop/hdfs/hadoop-hdfs-client-3.0.0-SNAPSHOT-tests.jar
  > 
share/hadoop/hdfs/hadoop-hdfs-client-3.0.0-SNAPSHOT.jar
  > 
share/hadoop/hdfs/hadoop-hdfs-native-client-3.0.0-SNAPSHOT-te
  > 
share/hadoop/hdfs/hadoop-hdfs-native-client-3.0.0-SNAPSHOT.ja
share/hadoop/hdfs/lib/hadoop-hdfs-client-3.0.0-SNAPSHOT.jar   <
  > 
share/hadoop/hdfs/sources/hadoop-hdfs-client-3.0.0-SNAPSHOT-s
  > 
share/hadoop/hdfs/sources/hadoop-hdfs-client-3.0.0-SNAPSHOT-t
{code}

> Fix hadoop-dist to adapt to HDFS client library separation
> --
>
> Key: HADOOP-13046
> URL: https://issues.apache.org/jira/browse/HADOOP-13046
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Teruyoshi Zenmyo
>Assignee: Teruyoshi Zenmyo
> Attachments: HADOOP-13046.patch
>
>
> Some build-related files should be updated to adapt to HDFS client library 
> separation. There exist below issues.
> - hdfs.h is not included.
> - hadoop.component is not set in pom.xml of hdfs client libraries.
> - hdfs-native-client is not include



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11418) Property "io.compression.codec.lzo.class" does not work with other value besides default

2016-04-25 Thread Akira AJISAKA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256044#comment-15256044
 ] 

Akira AJISAKA commented on HADOOP-11418:


Thank you for your comment. The latest patch needs another committer's +1 
because I wrote it. Hi [~iwasakims], would you please review the latest patch?

> Property "io.compression.codec.lzo.class" does not work with other value 
> besides default
> 
>
> Key: HADOOP-11418
> URL: https://issues.apache.org/jira/browse/HADOOP-11418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io
>Affects Versions: 2.6.0
>Reporter: fang fang chen
>Assignee: Yuanbo Liu
>  Labels: BB2015-05-RFC
> Attachments: HADOOP-11418-004.patch, HADOOP-11418-1.patch, 
> HADOOP-11418-2.patch, HADOOP-11418-3.patch, HADOOP-11418.005.patch, 
> HADOOP-11418.006.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for 
> other codec besides default. Hadoop will always treat it as defaultClazz. I 
> think it is a bug. Please let me know if this is a work as design thing. 
> Thanks
>  77   private static final String defaultClazz =
>  78   "org.apache.hadoop.io.compress.LzoCodec";
>  82   public synchronized boolean isSupported() {
>  83 if (!checked) {
>  84   checked = true;
>  85   String extClazz =
>  86   (conf.get(CONF_LZO_CLASS) == null ? System
>  87   .getProperty(CONF_LZO_CLASS) : null);
>  88   String clazz = (extClazz != null) ? extClazz : defaultClazz;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11418) Property "io.compression.codec.lzo.class" does not work with other value besides default

2016-04-25 Thread Yuanbo Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256025#comment-15256025
 ] 

Yuanbo Liu commented on HADOOP-11418:
-

[~ajisakaa] Thanks for your work. This issue has been assigned to me. I've 
reviewed the patch and it looks good. Could you please tell me what the next 
step is so that we can fix and close this issue?

> Property "io.compression.codec.lzo.class" does not work with other value 
> besides default
> 
>
> Key: HADOOP-11418
> URL: https://issues.apache.org/jira/browse/HADOOP-11418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io
>Affects Versions: 2.6.0
>Reporter: fang fang chen
>Assignee: Yuanbo Liu
>  Labels: BB2015-05-RFC
> Attachments: HADOOP-11418-004.patch, HADOOP-11418-1.patch, 
> HADOOP-11418-2.patch, HADOOP-11418-3.patch, HADOOP-11418.005.patch, 
> HADOOP-11418.006.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for 
> other codec besides default. Hadoop will always treat it as defaultClazz. I 
> think it is a bug. Please let me know if this is a work as design thing. 
> Thanks
>  77   private static final String defaultClazz =
>  78   "org.apache.hadoop.io.compress.LzoCodec";
>  82   public synchronized boolean isSupported() {
>  83 if (!checked) {
>  84   checked = true;
>  85   String extClazz =
>  86   (conf.get(CONF_LZO_CLASS) == null ? System
>  87   .getProperty(CONF_LZO_CLASS) : null);
>  88   String clazz = (extClazz != null) ? extClazz : defaultClazz;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation

2016-04-25 Thread Ling Zhou (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255983#comment-15255983
 ] 

Ling Zhou commented on HADOOP-12756:


Hi Liu Yi,
Ali-oss client does rely on a higher version of httpclient, it is required by 
Aliyun OSS SDK. Currently it uses 4.4 version, and hadoop uses 4.2.5 version 
which does not work for OSS SDK. And about the name "oss", we will change it 
according to your suggestion.

Hi Steve Loughran,
We have read the filesystem specification docs, this implementation is similar 
to S3A. So operations like rename and delete are still not atomic. Aliyun OSS 
is much like general object storage system except it is strong consistent.



> Incorporate Aliyun OSS file system implementation
> -
>
> Key: HADOOP-12756
> URL: https://issues.apache.org/jira/browse/HADOOP-12756
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Reporter: shimingfei
>Assignee: shimingfei
> Attachments: 0001-OSS-filesystem-integration-with-Hadoop.patch, HCFS 
> User manual.md, OSS integration.pdf, OSS integration.pdf
>
>
> Aliyun OSS is widely used among China’s cloud users, but currently it is not 
> easy to access data laid on OSS storage from user’s Hadoop/Spark application, 
> because of no original support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, 
> Spark/Hadoop applications can read/write data from OSS without any code 
> change. Narrowing the gap between user’s APP and data storage, like what have 
> been done for S3 in Hadoop 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation

2016-04-25 Thread Yi Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255941#comment-15255941
 ] 

Yi Liu commented on HADOOP-12756:
-

Also the name "oss" is abbreviation of Object Store Service, it's too generic, 
I think we need to change the name to ali-oss or some other names which other 
people can understand what it is at first glance. 

> Incorporate Aliyun OSS file system implementation
> -
>
> Key: HADOOP-12756
> URL: https://issues.apache.org/jira/browse/HADOOP-12756
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Reporter: shimingfei
>Assignee: shimingfei
> Attachments: 0001-OSS-filesystem-integration-with-Hadoop.patch, HCFS 
> User manual.md, OSS integration.pdf, OSS integration.pdf
>
>
> Aliyun OSS is widely used among China’s cloud users, but currently it is not 
> easy to access data laid on OSS storage from user’s Hadoop/Spark application, 
> because of no original support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, 
> Spark/Hadoop applications can read/write data from OSS without any code 
> change. Narrowing the gap between user’s APP and data storage, like what have 
> been done for S3 in Hadoop 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

84 matches

Mail list logo