date:20150401


[ 
https://issues.apache.org/jira/browse/HADOOP-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390151#comment-14390151
 ] 

Kai Zheng commented on HADOOP-11643:


Hi Xinwei,

Good catch ! Would you check it with latest codes in the branch and if 
necessary fire an issue for the fix ? Thanks !

 Define EC schema API for ErasureCodec
 -

 Key: HADOOP-11643
 URL: https://issues.apache.org/jira/browse/HADOOP-11643
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
 Fix For: HDFS-7285

 Attachments: HADOOP-11643-v3.patch, HADOOP-11643_v1.patch, 
 HADOOP-11643_v2.patch


 As part of {{ErasureCodec}} API to be defined in HDFS-7699, {{ECSchema}} API 
 will be first defined here for better sync among related issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11643) Define EC schema API for ErasureCodec


[ 
https://issues.apache.org/jira/browse/HADOOP-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390167#comment-14390167
 ] 

Xinwei Qin  commented on HADOOP-11643:
--

OK, I will fire an issue to fix it.

 Define EC schema API for ErasureCodec
 -

 Key: HADOOP-11643
 URL: https://issues.apache.org/jira/browse/HADOOP-11643
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
 Fix For: HDFS-7285

 Attachments: HADOOP-11643-v3.patch, HADOOP-11643_v1.patch, 
 HADOOP-11643_v2.patch


 As part of {{ErasureCodec}} API to be defined in HDFS-7699, {{ECSchema}} API 
 will be first defined here for better sync among related issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11782) Correct two thrown messages in ECSchema class


 [ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xinwei Qin  updated HADOOP-11782:
-
Status: Patch Available  (was: In Progress)

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11643) Define EC schema API for ErasureCodec


[ 
https://issues.apache.org/jira/browse/HADOOP-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390137#comment-14390137
 ] 

Xinwei Qin  commented on HADOOP-11643:
--

Hi [~drankye]
{quote}
catch (NumberFormatException e) {
  throw new IllegalArgumentException(Option value  +
  options.get(CHUNK_SIZE_KEY) +  for  + CHUNK_SIZE_KEY +
   is found. It should be an integer);
{quote}
In the  patch v3, when check {{NUM_DATA_UNITS_KEY}} and 
{{NUM_PARITY_UNITS_KEY}}  value format, the thrown messages are wrong, they 
should not be {{CHUNK_SIZE_KEY}}, but should be {{NUM_DATA_UNITS_KEY}} and 
{{NUM_PARITY_UNITS_KEY}}.

Please review to verify.

 Define EC schema API for ErasureCodec
 -

 Key: HADOOP-11643
 URL: https://issues.apache.org/jira/browse/HADOOP-11643
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
 Fix For: HDFS-7285

 Attachments: HADOOP-11643-v3.patch, HADOOP-11643_v1.patch, 
 HADOOP-11643_v2.patch


 As part of {{ErasureCodec}} API to be defined in HDFS-7699, {{ECSchema}} API 
 will be first defined here for better sync among related issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11782) Correct two thrown messages in ECSchema class

Xinwei Qin  created HADOOP-11782:


 Summary: Correct two thrown messages in ECSchema class
 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285


when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11782) Correct two thrown messages in ECSchema class


[ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390233#comment-14390233
 ] 

Hadoop QA commented on HADOOP-11782:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12708641/HADOOP-11782.000.patch
  against trunk revision 2daa478.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6037//console

This message is automatically generated.

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HADOOP-11782) Correct two thrown messages in ECSchema class


 [ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-11782 started by Xinwei Qin .

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11782) Correct two thrown messages in ECSchema class


 [ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xinwei Qin  updated HADOOP-11782:
-
Status: Open  (was: Patch Available)

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HADOOP-11782) Correct two thrown messages in ECSchema class


 [ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-11782 started by Xinwei Qin .

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11665) Provide and unify cross platform byteorder support in native code

2015-04-01 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-11665:

Component/s: native

 Provide and unify cross platform byteorder support in native code
 -

 Key: HADOOP-11665
 URL: https://issues.apache.org/jira/browse/HADOOP-11665
 Project: Hadoop Common
  Issue Type: Bug
  Components: native, util
Affects Versions: 2.4.1, 2.6.0
 Environment: PowerPC Big Endian  other Big Endian platforms
Reporter: Binglin Chang
Assignee: Binglin Chang
 Attachments: HADOOP-11665.001.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11775) Fix Javadoc typos in hadoop-openstack module


[ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390040#comment-14390040
 ] 

Hadoop QA commented on HADOOP-11775:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12708607/HADOOP-11775.001.patch
  against trunk revision 2daa478.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-openstack.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6036//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6036//console

This message is automatically generated.

 Fix Javadoc typos in hadoop-openstack module
 

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial
 Attachments: HADOOP-11775.000.patch, HADOOP-11775.001.patch


 Some typos are listed below but not limited to:
 SwiftNativeFileSystemObject.java
   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
 SwiftNativeFileSystem.java
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/
 SwiftRestClient.java
   /**
* Converts Swift path to URI to make request.
* This is public for unit testing
*
* @param path path to object
* @param endpointURI damain url e.g. http://domain.com
* @return valid URI for object
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11782) Correct two thrown messages in ECSchema class


 [ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xinwei Qin  updated HADOOP-11782:
-
Attachment: HADOOP-11782.000.patch

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11665) Provide and unify cross platform byteorder support in native code

2015-04-01 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-11665:

Target Version/s: 2.8.0  (was: 2.7.0)

 Provide and unify cross platform byteorder support in native code
 -

 Key: HADOOP-11665
 URL: https://issues.apache.org/jira/browse/HADOOP-11665
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.4.1, 2.6.0
 Environment: PowerPC Big Endian  other Big Endian platforms
Reporter: Binglin Chang
Assignee: Binglin Chang
 Attachments: HADOOP-11665.001.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11782) Correct two thrown messages in ECSchema class


[ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390525#comment-14390525
 ] 

Kai Zheng commented on HADOOP-11782:


Committed to the branch. Thanks Xinwei for the contribution !

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11782) Correct two thrown messages in ECSchema class


[ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390500#comment-14390500
 ] 

Kai Zheng commented on HADOOP-11782:


The change looks good. +1.

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


 [ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay Bhat updated HADOOP-8320:
---
Attachment: HADOOP-8320.005.patch

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch, HADOOP-8320.002.patch, 
 HADOOP-8320.003.patch, HADOOP-8320.004.patch, HADOOP-8320.005.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11782) Correct two thrown messages in ECSchema class


 [ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng resolved HADOOP-11782.

  Resolution: Fixed
Hadoop Flags: Reviewed

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Bug
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11783) Build failed with JVM IBM JAVA on TestSecureLogins

2015-04-01 Thread pascal oliva (JIRA)

pascal oliva created HADOOP-11783:
-

 Summary: Build failed with JVM IBM JAVA on TestSecureLogins
 Key: HADOOP-11783
 URL: https://issues.apache.org/jira/browse/HADOOP-11783
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
 Environment: $ mvn -version
Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 
2015-03-13T16:10:27-04:00)
Maven home: /opt/apache-maven-3.3.1
Java version: 1.7.0, vendor: IBM Corporation
Java home: /usr/lib/jvm/ibm-java-x86_64-71/jre
Default locale: en_US, platform encoding: UTF-8
OS name: linux, version: 3.10.0-229.el7.x86_64, arch: amd64, family: 
unix

Reporter: pascal oliva


Hadoop build failed with JVM IBM due to the use of com.sun.security.* classes : 
[ERROR] 
/home/ibmadmin/TRUNK/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureLogins.java:[23,36]
 package com.sun.security.auth.module does not exist
[ERROR] 
/home/ibmadmin/TRUNK/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureLogins.java:[138,11]
 cannot find symbol
[ERROR] symbol:   class Krb5LoginModule
[ERROR] location: class org.apache.hadoop.registry.secure.TestSecureLogins



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


[ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390752#comment-14390752
 ] 

Hadoop QA commented on HADOOP-8320:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12708691/HADOOP-8320.005.patch
  against trunk revision 2e79f1c.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common:

  org.apache.hadoop.fs.TestFcLocalFsUtil
  org.apache.hadoop.fs.TestLocalDirAllocator
  org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison
  org.apache.hadoop.fs.shell.TestCopy
  org.apache.hadoop.util.TestDiskChecker
  
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
  
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename
  org.apache.hadoop.fs.TestFileUtil
  org.apache.hadoop.fs.contract.localfs.TestLocalFSContractOpen
  org.apache.hadoop.security.ssl.TestSSLFactory
  org.apache.hadoop.fs.TestFsShellCopy
  
org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem
  org.apache.hadoop.fs.TestLocalFileSystem
  
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractDelete
  org.apache.hadoop.fs.shell.find.TestIname
  org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem
  org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem
  org.apache.hadoop.io.file.tfile.TestTFileStreams
  
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
  org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs
  org.apache.hadoop.util.TestGenericOptionsParser
  org.apache.hadoop.io.TestSetFile
  org.apache.hadoop.fs.TestS3_LocalFileContextURI
  org.apache.hadoop.crypto.key.TestKeyShell
  
org.apache.hadoop.fs.viewfs.TestViewFileSystemDelegationTokenSupport
  org.apache.hadoop.fs.viewfs.TestViewFileSystemDelegation
  
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractMkdir
  org.apache.hadoop.io.compress.TestCodec
  org.apache.hadoop.fs.TestGetFileBlockLocations
  org.apache.hadoop.fs.viewfs.TestViewFsLocalFs
  org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsStreams
  org.apache.hadoop.fs.TestChecksumFileSystem
  org.apache.hadoop.fs.TestTruncatedInputBug
  org.apache.hadoop.security.alias.TestCredShell
  org.apache.hadoop.security.alias.TestCredentialProviderFactory
  
org.apache.hadoop.fs.contract.localfs.TestLocalFSContractCreate
  org.apache.hadoop.io.TestMapFile
  org.apache.hadoop.fs.TestPath
  org.apache.hadoop.fs.viewfs.TestChRootedFs
  org.apache.hadoop.security.TestLdapGroupsMapping
  org.apache.hadoop.fs.TestFilterFileSystem
  org.apache.hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem
  org.apache.hadoop.fs.shell.TestPathData
  org.apache.hadoop.fs.TestFileSystemInitialization
  org.apache.hadoop.fs.shell.find.TestPrint0
  org.apache.hadoop.io.TestArrayFile
  org.apache.hadoop.fs.TestLocalFsFCStatistics
  org.apache.hadoop.fs.TestLocalFileSystemPermission
  org.apache.hadoop.fs.TestListFiles
  org.apache.hadoop.fs.shell.find.TestFind
  org.apache.hadoop.io.file.tfile.TestTFileByteArrays
  org.apache.hadoop.io.file.tfile.TestTFileSplit

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390769#comment-14390769
 ] 

Allen Wittenauer commented on HADOOP-11731:
---

So is this ready to go in?

Which do we prefer, with all subtasks listed or without?

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11783) Build failed with JVM IBM JAVA on TestSecureLogins

2015-04-01 Thread pascal oliva (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

pascal oliva updated HADOOP-11783:
--
Status: Patch Available  (was: Open)

HADOOP-11783-1.patch
tested with IBM JAVA and openJDK
 mvn test -X -Dtest=org.apache.hadoop.registry.secure.TestSecureLogins

 Build failed with JVM IBM JAVA on TestSecureLogins
 --

 Key: HADOOP-11783
 URL: https://issues.apache.org/jira/browse/HADOOP-11783
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
 Environment: $ mvn -version
 Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 
 2015-03-13T16:10:27-04:00)
 Maven home: /opt/apache-maven-3.3.1
 Java version: 1.7.0, vendor: IBM Corporation
 Java home: /usr/lib/jvm/ibm-java-x86_64-71/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux, version: 3.10.0-229.el7.x86_64, arch: amd64, family: 
 unix
Reporter: pascal oliva
 Attachments: HADOOP-11783-1.patch


 Hadoop build failed with JVM IBM due to the use of com.sun.security.* classes 
 : 
 [ERROR] 
 /home/ibmadmin/TRUNK/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureLogins.java:[23,36]
  package com.sun.security.auth.module does not exist
 [ERROR] 
 /home/ibmadmin/TRUNK/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureLogins.java:[138,11]
  cannot find symbol
 [ERROR] symbol:   class Krb5LoginModule
 [ERROR] location: class org.apache.hadoop.registry.secure.TestSecureLogins



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11783) Build failed with JVM IBM JAVA on TestSecureLogins

2015-04-01 Thread pascal oliva (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

pascal oliva updated HADOOP-11783:
--
Attachment: HADOOP-11783-1.patch

 Build failed with JVM IBM JAVA on TestSecureLogins
 --

 Key: HADOOP-11783
 URL: https://issues.apache.org/jira/browse/HADOOP-11783
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
 Environment: $ mvn -version
 Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 
 2015-03-13T16:10:27-04:00)
 Maven home: /opt/apache-maven-3.3.1
 Java version: 1.7.0, vendor: IBM Corporation
 Java home: /usr/lib/jvm/ibm-java-x86_64-71/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux, version: 3.10.0-229.el7.x86_64, arch: amd64, family: 
 unix
Reporter: pascal oliva
 Attachments: HADOOP-11783-1.patch


 Hadoop build failed with JVM IBM due to the use of com.sun.security.* classes 
 : 
 [ERROR] 
 /home/ibmadmin/TRUNK/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureLogins.java:[23,36]
  package com.sun.security.auth.module does not exist
 [ERROR] 
 /home/ibmadmin/TRUNK/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureLogins.java:[138,11]
  cannot find symbol
 [ERROR] symbol:   class Krb5LoginModule
 [ERROR] location: class org.apache.hadoop.registry.secure.TestSecureLogins



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11783) Build failed with JVM IBM JAVA on TestSecureLogins


[ 
https://issues.apache.org/jira/browse/HADOOP-11783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390835#comment-14390835
 ] 

Hadoop QA commented on HADOOP-11783:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12708708/HADOOP-11783-1.patch
  against trunk revision 2e79f1c.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-minikdc 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6039//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6039//console

This message is automatically generated.

 Build failed with JVM IBM JAVA on TestSecureLogins
 --

 Key: HADOOP-11783
 URL: https://issues.apache.org/jira/browse/HADOOP-11783
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
 Environment: $ mvn -version
 Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 
 2015-03-13T16:10:27-04:00)
 Maven home: /opt/apache-maven-3.3.1
 Java version: 1.7.0, vendor: IBM Corporation
 Java home: /usr/lib/jvm/ibm-java-x86_64-71/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: linux, version: 3.10.0-229.el7.x86_64, arch: amd64, family: 
 unix
Reporter: pascal oliva
 Attachments: HADOOP-11783-1.patch


 Hadoop build failed with JVM IBM due to the use of com.sun.security.* classes 
 : 
 [ERROR] 
 /home/ibmadmin/TRUNK/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureLogins.java:[23,36]
  package com.sun.security.auth.module does not exist
 [ERROR] 
 /home/ibmadmin/TRUNK/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureLogins.java:[138,11]
  cannot find symbol
 [ERROR] symbol:   class Krb5LoginModule
 [ERROR] location: class org.apache.hadoop.registry.secure.TestSecureLogins



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11784) failed to locate Winutils for win 32 platform

2015-04-01 Thread Gaurav Tiwari (JIRA)

Gaurav Tiwari created HADOOP-11784:
--

 Summary: failed to locate Winutils for win 32 platform
 Key: HADOOP-11784
 URL: https://issues.apache.org/jira/browse/HADOOP-11784
 Project: Hadoop Common
  Issue Type: Bug
  Components: bin
Affects Versions: 2.6.0
Reporter: Gaurav Tiwari


During the execution of Map reduce example first I got the error telling 
ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path

So I downloaded a version and update the bin folder again executing the same 
command I am getting error like 

ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable 
C:\hadoop-2.6.0\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
at org.apache.hadoop.util.Shell.clinit(Shell.java:363)
at 
org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:438)
at 
org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:484)
at 
org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:170)
at 
org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:153)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
at org.apache.hadoop.examples.Grep.main(Grep.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at 
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
15/04/01 19:12:51 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where ap
licable
15/04/01 19:12:52 INFO Configuration.deprecation: session.id is deprecated. 
Instead, use dfs.metrics.session-id
15/04/01 19:12:52 INFO jvm.JvmMetrics: Initializing JVM Metrics with 
processName=JobTracker, sessionId=
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1011)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:656)
at 
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:444)
at 
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:293)
at 
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
at org.apache.hadoop.examples.Grep.run(Grep.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.Grep.main(Grep.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at

[jira] [Commented] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


[ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391776#comment-14391776
 ] 

Vijay Bhat commented on HADOOP-8320:


[~atm] all test cases now finally pass for the refactor. There are no new 
additional test cases since existing ones capture the functionality of the code 
being refactored. Would appreciate a review when you get a chance.

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch, HADOOP-8320.002.patch, 
 HADOOP-8320.003.patch, HADOOP-8320.004.patch, HADOOP-8320.005.patch, 
 HADOOP-8320.007.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes

2015-04-01 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391771#comment-14391771
 ] 

Andrew Wang commented on HADOOP-11731:
--

I think we should list the subtasks, since they're JIRAs too.

I'll add that I looked at an earlier version of Allen's script and it looked 
basically good. If Colin's happy too, let's just get it in and fix the rest 
later.

Allen, do you mind also updating the release Jenkins job 
(https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/) so it also 
runs this script?

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11785) Reduce number of listStatus operation in distcp buildListing()


[ 
https://issues.apache.org/jira/browse/HADOOP-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391791#comment-14391791
 ] 

Hadoop QA commented on HADOOP-11785:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12708802/distcp-liststatus2.patch
  against trunk revision c94d594.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-distcp.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6045//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6045//console

This message is automatically generated.

 Reduce number of listStatus operation in distcp buildListing()
 --

 Key: HADOOP-11785
 URL: https://issues.apache.org/jira/browse/HADOOP-11785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Priority: Minor
 Attachments: distcp-liststatus.patch, distcp-liststatus2.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Distcp was taking long time in copyListing.buildListing() for large source 
 trees (I was using source of 1.5M files in a tree of about 50K directories). 
 For input at s3 buildListing was taking more than one hour. I've noticed a 
 performance bug in the current code which does listStatus twice for each 
 directory which doubles number of RPCs in some cases (if most directories do 
 not contain 1000 files).
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


[ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391709#comment-14391709
 ] 

Hadoop QA commented on HADOOP-8320:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12708789/HADOOP-8320.007.patch
  against trunk revision c94d594.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6044//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6044//console

This message is automatically generated.

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch, HADOOP-8320.002.patch, 
 HADOOP-8320.003.patch, HADOOP-8320.004.patch, HADOOP-8320.005.patch, 
 HADOOP-8320.007.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11623) Native compilation fails on Solaris due to use of syscall function.

2015-04-01 Thread Malcolm Kavalsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392165#comment-14392165
 ] 

Malcolm Kavalsky commented on HADOOP-11623:
---

Yes




 Native compilation fails on Solaris due to use of syscall function.
 ---

 Key: HADOOP-11623
 URL: https://issues.apache.org/jira/browse/HADOOP-11623
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.6.0
 Environment: Solaris 11.2
Reporter: Malcolm Kavalsky
Assignee: Malcolm Kavalsky
   Original Estimate: 168h
  Remaining Estimate: 168h

 Solaris does not provide the syscall function. Currently, hadoop has very 
 limited use of this function ( only 2 files ). These need to be ifdeffed 
 with the correct alternatives for Solaris.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9805) Refactor RawLocalFileSystem#rename for improved testability.


[ 
https://issues.apache.org/jira/browse/HADOOP-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391882#comment-14391882
 ] 

Hadoop QA commented on HADOOP-9805:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12708851/HADOOP-9805.003.patch
  against trunk revision 75cb1d4.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6046//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6046//console

This message is automatically generated.

 Refactor RawLocalFileSystem#rename for improved testability.
 

 Key: HADOOP-9805
 URL: https://issues.apache.org/jira/browse/HADOOP-9805
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Jean-Pierre Matsumoto
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-9805.001.patch, HADOOP-9805.002.patch, 
 HADOOP-9805.003.patch


 {{RawLocalFileSystem#rename}} contains fallback logic to provide POSIX rename 
 behavior on platforms where {{java.io.File#renameTo}} fails.  The method 
 returns early if {{java.io.File#renameTo}} succeeds, so test runs may not 
 cover the fallback logic depending on the platform.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HADOOP-11742) mkdir by file system shell fails on an empty bucket

2015-04-01 Thread Takenori Sato (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takenori Sato reopened HADOOP-11742:


I confirmed mkdir fails on an empty bucket for AWS as follows:

1. make sure the bucket is empty, but get an exception

{code}
# hadoop-2.7.0-SNAPSHOT/bin/hdfs dfs -Dfs.s3a.access.key=ACCESS_KEY 
-Dfs.s3a.secret.key=SECRET_KEY -ls s3a://s3atest/
15/04/02 01:49:09 DEBUG http.wire:  HEAD / HTTP/1.1[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Host: s3atest.s3.amazonaws.com[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Authorization: AWS XXX=[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Date: Thu, 02 Apr 2015 01:49:08 
GMT[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  User-Agent: aws-sdk-java/1.7.4 
Linux/3.10.0-123.8.1.el7.centos.plus.x86_64 
Java_HotSpot(TM)_64-Bit_Server_VM/24.75-b04/1.7.0_75[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Content-Type: 
application/x-www-form-urlencoded; charset=utf-8[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Connection: Keep-Alive[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  [\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  HTTP/1.1 200 OK[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  x-amz-id-2: XXX[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  x-amz-request-id: XXX[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Date: Thu, 02 Apr 2015 01:49:10 
GMT[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Content-Type: application/xml[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Transfer-Encoding: chunked[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Server: AmazonS3[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  [\r][\n]
15/04/02 01:49:09 DEBUG s3a.S3AFileSystem: Getting path status for 
s3a://s3atest/ ()
15/04/02 01:49:09 DEBUG http.wire:  GET /?delimiter=%2Fmax-keys=1prefix= 
HTTP/1.1[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Host: s3atest.s3.amazonaws.com[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Authorization: AWS XXX=[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Date: Thu, 02 Apr 2015 01:49:09 
GMT[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  User-Agent: aws-sdk-java/1.7.4 
Linux/3.10.0-123.8.1.el7.centos.plus.x86_64 
Java_HotSpot(TM)_64-Bit_Server_VM/24.75-b04/1.7.0_75[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Content-Type: 
application/x-www-form-urlencoded; charset=utf-8[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Connection: Keep-Alive[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  [\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  HTTP/1.1 200 OK[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  x-amz-id-2: XXX[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  x-amz-request-id: XXX[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Date: Thu, 02 Apr 2015 01:49:10 
GMT[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Content-Type: application/xml[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Transfer-Encoding: chunked[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  Server: AmazonS3[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  [\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  fe[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  ?xml version=1.0 
encoding=UTF-8?[\n]
15/04/02 01:49:09 DEBUG http.wire:  ListBucketResult 
xmlns=http://s3.amazonaws.com/doc/2006-03-01/;Names3atest/NamePrefix/PrefixMarker/MarkerMaxKeys1/MaxKeysDelimiter//DelimiterIsTruncatedfalse/IsTruncated/ListBucketResult
15/04/02 01:49:09 DEBUG http.wire:  [\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  0[\r][\n]
15/04/02 01:49:09 DEBUG http.wire:  [\r][\n]
15/04/02 01:49:09 DEBUG s3a.S3AFileSystem: Not Found: s3a://s3atest/
ls: `s3a://s3atest/': No such file or directory
{code}

2. create a directory, but get an exception

{code}
# hadoop-2.7.0-SNAPSHOT/bin/hdfs dfs -Dfs.s3a.access.key=ACCESS_KEY 
-Dfs.s3a.secret.key=SECRET_KEY -mkdir s3a://s3atest/root
15/04/02 01:49:41 DEBUG http.wire:  HEAD / HTTP/1.1[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Host: s3atest.s3.amazonaws.com[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Authorization: AWS XXX=[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Date: Thu, 02 Apr 2015 01:49:41 
GMT[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  User-Agent: aws-sdk-java/1.7.4 
Linux/3.10.0-123.8.1.el7.centos.plus.x86_64 
Java_HotSpot(TM)_64-Bit_Server_VM/24.75-b04/1.7.0_75[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Content-Type: 
application/x-www-form-urlencoded; charset=utf-8[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Connection: Keep-Alive[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  [\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  HTTP/1.1 200 OK[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  x-amz-id-2: XXX[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  x-amz-request-id: XXX[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Date: Thu, 02 Apr 2015 01:49:42 
GMT[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Content-Type: application/xml[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Transfer-Encoding: chunked[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  Server: AmazonS3[\r][\n]
15/04/02 01:49:41 DEBUG http.wire:  [\r][\n]
15/04/02 01:49:41 DEBUG s3a.S3AFileSystem: Getting path

[jira] [Updated] (HADOOP-11757) NFS gateway should shutdown when it can't start UDP or TCP server

2015-04-01 Thread Brandon Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li updated HADOOP-11757:

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

 NFS gateway should shutdown when it can't start UDP or TCP server
 -

 Key: HADOOP-11757
 URL: https://issues.apache.org/jira/browse/HADOOP-11757
 Project: Hadoop Common
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.2.0
Reporter: Brandon Li
Assignee: Brandon Li
 Attachments: HDFS-7989.001.patch, HDFS-7989.002.patch


 Unlike the Portmap, Nfs3 class does shutdown when the service can't start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11757) NFS gateway should shutdown when it can't start UDP or TCP server

2015-04-01 Thread Brandon Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391826#comment-14391826
 ] 

Brandon Li commented on HADOOP-11757:
-

I've committed the patch. Thank you, Jing, for the review.

 NFS gateway should shutdown when it can't start UDP or TCP server
 -

 Key: HADOOP-11757
 URL: https://issues.apache.org/jira/browse/HADOOP-11757
 Project: Hadoop Common
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.2.0
Reporter: Brandon Li
Assignee: Brandon Li
 Fix For: 2.7.0

 Attachments: HDFS-7989.001.patch, HDFS-7989.002.patch


 Unlike the Portmap, Nfs3 class does shutdown when the service can't start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11731) Rework the changelog and releasenotes


 [ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11731:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

I've committed this to trunk.

Thanks everyone. 

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Fix For: 3.0.0

 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch, HADOOP-11731-07.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes

2015-04-01 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391953#comment-14391953
 ] 

Hudson commented on HADOOP-11731:
-

SUCCESS: Integrated in Hadoop-trunk-Commit #7490 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7490/])
HADOOP-11731. Rework the changelog and releasenotes (aw) (aw: rev 
f383fd9b6caf4557613250c5c218b1a1b65a212b)
* hadoop-project/src/site/site.xml
* dev-support/releasedocmaker.py
* hadoop-common-project/hadoop-common/pom.xml
* hadoop-common-project/hadoop-common/CHANGES.txt
* BUILDING.txt
* dev-support/relnotes.py


 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Fix For: 3.0.0

 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch, HADOOP-11731-07.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11757) NFS gateway should shutdown when it can't start UDP or TCP server

2015-04-01 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391956#comment-14391956
 ] 

Hudson commented on HADOOP-11757:
-

SUCCESS: Integrated in Hadoop-trunk-Commit #7490 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7490/])
HADOOP-11757. NFS gateway should shutdown when it can't start UDP or TCP 
server. Contributed by Brandon Li (brandonli: rev 
60ce825a71850fe0622d551159e8d66f32448bb5)
* 
hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleTcpServer.java
* 
hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java
* 
hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleUdpServer.java


 NFS gateway should shutdown when it can't start UDP or TCP server
 -

 Key: HADOOP-11757
 URL: https://issues.apache.org/jira/browse/HADOOP-11757
 Project: Hadoop Common
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.2.0
Reporter: Brandon Li
Assignee: Brandon Li
 Fix For: 2.7.0

 Attachments: HDFS-7989.001.patch, HDFS-7989.002.patch


 Unlike the Portmap, Nfs3 class does shutdown when the service can't start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11731) Rework the changelog and releasenotes


 [ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11731:
--
Attachment: HADOOP-11731-07.patch

-07:
* [~chris.douglas] recommended changing the 'Description' heading to 'Summary' 
to reflect what that field actually is in JIRA.

[~cmccabe], et. al. if you have no other comments, I'll commit this (w/subtasks 
turned on)  tomorrow morning given the +1.  I'll follow it up with a message to 
*-dev and open other JIRAs to finish up moving to automated changes.

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch, HADOOP-11731-07.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11731) Rework the changelog and releasenotes


 [ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11731:
--
Status: Open  (was: Patch Available)

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch, HADOOP-11731-07.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11731) Rework the changelog and releasenotes


 [ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11731:
--
Status: Patch Available  (was: Open)

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch, HADOOP-11731-07.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9805) Refactor RawLocalFileSystem#rename for improved testability.


[ 
https://issues.apache.org/jira/browse/HADOOP-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391823#comment-14391823
 ] 

Colin Patrick McCabe commented on HADOOP-9805:
--

+1 from me.  Will hold off a bit in case others have comments.

 Refactor RawLocalFileSystem#rename for improved testability.
 

 Key: HADOOP-9805
 URL: https://issues.apache.org/jira/browse/HADOOP-9805
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Jean-Pierre Matsumoto
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-9805.001.patch, HADOOP-9805.002.patch, 
 HADOOP-9805.003.patch


 {{RawLocalFileSystem#rename}} contains fallback logic to provide POSIX rename 
 behavior on platforms where {{java.io.File#renameTo}} fails.  The method 
 returns early if {{java.io.File#renameTo}} succeeds, so test runs may not 
 cover the fallback logic depending on the platform.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11781) rewrite smart-apply-patch.sh

2015-04-01 Thread Raymie Stata (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raymie Stata updated HADOOP-11781:
--
Attachment: HADOOP-11781-01.patch

-01: cleaned up a race-condition with file names (c.f. HADOOP-11746).  Added 
feature to take a URL and JIRA issue-identifier instead of a patch filename, so 
one can more quickly apply patches (we want to quickly check the backlog of 
patches hundreds of patches to find ones that no longer apply -- this feature 
will help). 

 rewrite smart-apply-patch.sh
 

 Key: HADOOP-11781
 URL: https://issues.apache.org/jira/browse/HADOOP-11781
 Project: Hadoop Common
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11781-01.patch


 smart-apply-patch.sh has a few race conditions and is just generally crufty.  
 It should be rewritten.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11784) failed to locate Winutils for win 32 platform

2015-04-01 Thread Yong Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391920#comment-14391920
 ] 

Yong Zhang commented on HADOOP-11784:
-

Did you try to set HADOOP_HOME=C:\hadoop-2.6.0?

 failed to locate Winutils for win 32 platform
 -

 Key: HADOOP-11784
 URL: https://issues.apache.org/jira/browse/HADOOP-11784
 Project: Hadoop Common
  Issue Type: Bug
  Components: bin
Affects Versions: 2.6.0
Reporter: Gaurav Tiwari

 During the execution of Map reduce example first I got the error telling 
 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary 
 path
 So I downloaded a version and update the bin folder again executing the same 
 command I am getting error like 
 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary 
 path
 java.io.IOException: Could not locate executable 
 C:\hadoop-2.6.0\bin\winutils.exe in the Hadoop binaries.
 at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
 at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
 at org.apache.hadoop.util.Shell.clinit(Shell.java:363)
 at 
 org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:438)
 at 
 org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:484)
 at 
 org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:170)
 at 
 org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:153)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
 at org.apache.hadoop.examples.Grep.main(Grep.java:101)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:601)
 at 
 org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
 at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
 at 
 org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:601)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
 15/04/01 19:12:51 WARN util.NativeCodeLoader: Unable to load native-hadoop 
 library for your platform... using builtin-java classes where ap
 licable
 15/04/01 19:12:52 INFO Configuration.deprecation: session.id is deprecated. 
 Instead, use dfs.metrics.session-id
 15/04/01 19:12:52 INFO jvm.JvmMetrics: Initializing JVM Metrics with 
 processName=JobTracker, sessionId=
 java.lang.NullPointerException
 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1011)
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
 at org.apache.hadoop.util.Shell.run(Shell.java:455)
 at 
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
 at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)
 at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
 at 
 org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:656)
 at 
 org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:444)
 at 
 org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:293)
 at 
 org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
 at org.apache.hadoop.examples.Grep.run(Grep.java:77)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at org.apache.hadoop.examples.Grep.main(Grep.java:101)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes

[
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391763#comment-14391763
]

Colin Patrick McCabe commented on HADOOP-11731:
---

I think we should support subtasks for now. If we later decide they aren't
important we can make it optional in the script.

I actually am completely neutral on whether we should have release notes for
multiple releases in one file (or changes for CHANGES.txt). I think if we want
to change the format for upcoming releases, we should discuss that on the
mailing list first. However, that issue shouldn't hold up this patch. It's
easy enough to create a file with release notes for multiple releases by
cat'ing together files, or passing multiple revisions on the relnotes.py
command line.

This is certainly a big improvement on what came before, so +1.

Rework the changelog and releasenotes
-

Key: HADOOP-11731
URL: https://issues.apache.org/jira/browse/HADOOP-11731
Project: Hadoop Common
Issue Type: New Feature
Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch,
HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch,
HADOOP-11731-06.patch

The current way we generate these build artifacts is awful. Plus they are
ugly and, in the case of release notes, very hard to pick out what is
important.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code

[
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vijay Bhat updated HADOOP-8320:
---
Release Note:
Moved the checkPath logic from the AbstractFileSystem and FileSystem classes to
a checkFileSystemPath method in the Path class. The AbstracFileSystem and
FileSystem checkPath method will call the common code in checkFileSystemPath
(adapter pattern). This is because the two classes aren't derived from any
common ancestor.

There are no new test cases because existing test cases test the functionality
that is being refactored. Adding new test cases would be redundant.

was:Moved the checkPath logic from the AbstractFileSystem and FileSystem
classes to a checkFileSystemPath method in the Path class. The
AbstracFileSystem and FileSystem checkPath method will call the common code in
checkFileSystemPath (adapter pattern). This is because the two classes aren't
derived from any common ancestor.

FileSystem#checkPath and AbstractFileSystem#checkPath should share code
---

Key: HADOOP-8320
URL: https://issues.apache.org/jira/browse/HADOOP-8320
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
Attachments: HADOOP-8320.001.patch, HADOOP-8320.002.patch,
HADOOP-8320.003.patch, HADOOP-8320.004.patch, HADOOP-8320.005.patch,
HADOOP-8320.007.patch

Per the discussion on HADOOP-8310, these two methods can be refactored to
share some code.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11757) NFS gateway should shutdown when it can't start UDP or TCP server

2015-04-01 Thread Brandon Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li updated HADOOP-11757:

Fix Version/s: 2.7.0

 NFS gateway should shutdown when it can't start UDP or TCP server
 -

 Key: HADOOP-11757
 URL: https://issues.apache.org/jira/browse/HADOOP-11757
 Project: Hadoop Common
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.2.0
Reporter: Brandon Li
Assignee: Brandon Li
 Fix For: 2.7.0

 Attachments: HDFS-7989.001.patch, HDFS-7989.002.patch


 Unlike the Portmap, Nfs3 class does shutdown when the service can't start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391945#comment-14391945
 ] 

Hadoop QA commented on HADOOP-11731:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12708858/HADOOP-11731-07.patch
  against trunk revision 4d14816.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common:

  org.apache.hadoop.ipc.TestRPCWaitForProxy

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6047//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6047//console

This message is automatically generated.

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Fix For: 3.0.0

 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch, HADOOP-11731-07.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391795#comment-14391795
 ] 

Allen Wittenauer commented on HADOOP-11731:
---

Ha, sorry, we crossed each other. 

I'll see what it takes to fix the Jenkins job.  I want to poke around Jenkins 
for HADOOP-11746 anyway. 

Thanks everyone.

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch, HADOOP-11731-07.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-9805) Refactor RawLocalFileSystem#rename for improved testability.

2015-04-01 Thread Jean-Pierre Matsumoto (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Pierre Matsumoto updated HADOOP-9805:
--
Attachment: HADOOP-9805.003.patch

Patch 003 attached. I have taken account Colin's and Chris's feedbacks.

 Refactor RawLocalFileSystem#rename for improved testability.
 

 Key: HADOOP-9805
 URL: https://issues.apache.org/jira/browse/HADOOP-9805
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Jean-Pierre Matsumoto
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-9805.001.patch, HADOOP-9805.002.patch, 
 HADOOP-9805.003.patch


 {{RawLocalFileSystem#rename}} contains fallback logic to provide POSIX rename 
 behavior on platforms where {{java.io.File#renameTo}} fails.  The method 
 returns early if {{java.io.File#renameTo}} succeeds, so test runs may not 
 cover the fallback logic depending on the platform.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10907) Single Node Setup still thinks it is hadoop 1.x

2015-04-01 Thread Jean-Pierre Matsumoto (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-10907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391732#comment-14391732
 ] 

Jean-Pierre Matsumoto commented on HADOOP-10907:


When I read Hadoop 2.6 documentation Single Node Cluster, I think this bug can 
be closed.

 Single Node Setup still thinks it is hadoop 1.x
 ---

 Key: HADOOP-10907
 URL: https://issues.apache.org/jira/browse/HADOOP-10907
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Allen Wittenauer
  Labels: newbie

 # JDK 1.6 is deprecated
 # the *-all.sh scripts are now in sbin.
 # location of hadoop-*-examples.jar is no longer there
 # hadoop jar should be replaced with yarn jar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11776) jdiff is broken in Hadoop 2

2015-04-01 Thread Li Lu (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Li Lu updated HADOOP-11776:
---
Attachment: HADOOP-11776-040115.patch

Worked on this issue on top of [~ozawa]'s HADOOP-11377 patch. That patch
addressed the Null.java problem for me. In this patch I'm doing the following
things:
# Applied a quick fix for the jdiff compatibility param settings to eliminate
the package error. I'd appreciate if there's background information about
this setting, so that we can decide the correct long term fix.
# Run the maven script against branch-2.6.0 to generate two sample API files,
one for hadoop-common-project/hadoop-common and one for
hadoo-hdfs-project/hadoop-hdfs
# Fix the SAXParser not found exception when running jdiff.

After this fix, hadoop hdfs can have API diff report after {{mvn package -Pdocs
-DskipTests}} in its target/site/jdiff/xml folder. Hadoop common still has some
problem with jdiff: jdiff is complainging the following:

[javadoc] Error: duplicate comment id:
org.apache.hadoop.metrics2.MetricsSystem.register_changed(java.lang.String,
java.lang.String, T)

Since this is the only error we got in all components (we're also exploring
YARN-3426 for yarn-api, yarn-client, yarn-common, and yarn-server-common),
after briefly checking the original code of jdiff I highly suspect the error is
triggered by a bug of jdiff. However, I'm not sure if jdiff is still
maintained, so for long term fix we may want to find some alternative tools.

So far the script would generate a diff report for hdfs. I'm extending the YARN
part in YARN-3426.

The patch in HADOOP-11377 appears to work on my local machine.

jdiff is broken in Hadoop 2
---

Key: HADOOP-11776
URL: https://issues.apache.org/jira/browse/HADOOP-11776
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Li Lu
Assignee: Li Lu
Priority: Blocker
Attachments: HADOOP-11776-040115.patch

Seems like we haven't touch the API files from jdiff under dev-support for a
while. For now we're missing the jdiff API files for hadoop 2. We're also
missing YARN when generating the jdiff API files.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11776) jdiff is broken in Hadoop 2

2015-04-01 Thread Li Lu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Lu updated HADOOP-11776:
---
Status: Patch Available  (was: Open)

 jdiff is broken in Hadoop 2
 ---

 Key: HADOOP-11776
 URL: https://issues.apache.org/jira/browse/HADOOP-11776
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Li Lu
Assignee: Li Lu
Priority: Blocker
 Attachments: HADOOP-11776-040115.patch


 Seems like we haven't touch the API files from jdiff under dev-support for a 
 while. For now we're missing the jdiff API files for hadoop 2. We're also 
 missing YARN when generating the jdiff API files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11627) Remove io.native.lib.available from trunk

2015-04-01 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391867#comment-14391867
 ] 

Brahma Reddy Battula commented on HADOOP-11627:
---

Testcase failures are unrelated this patch...

 Remove io.native.lib.available from trunk
 -

 Key: HADOOP-11627
 URL: https://issues.apache.org/jira/browse/HADOOP-11627
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Akira AJISAKA
Assignee: Brahma Reddy Battula
 Attachments: HADOOP-11627-002.patch, HADOOP-11627-003.patch, 
 HADOOP-11627-004.patch, HADOOP-11627-005.patch, HADOOP-11627-006.patch, 
 HADOOP-11627.patch


 According to the discussion in HADOOP-8642, we should remove 
 {{io.native.lib.available}} from trunk, and always use native libraries if 
 they exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11776) jdiff is broken in Hadoop 2


[ 
https://issues.apache.org/jira/browse/HADOOP-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392198#comment-14392198
 ] 

Hadoop QA commented on HADOOP-11776:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12708869/HADOOP-11776-040115.patch
  against trunk revision f383fd9.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

  
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
  org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
  org.apache.hadoop.hdfs.server.namenode.TestStartup

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6048//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6048//console

This message is automatically generated.

 jdiff is broken in Hadoop 2
 ---

 Key: HADOOP-11776
 URL: https://issues.apache.org/jira/browse/HADOOP-11776
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Li Lu
Assignee: Li Lu
Priority: Blocker
 Attachments: HADOOP-11776-040115.patch


 Seems like we haven't touch the API files from jdiff under dev-support for a 
 while. For now we're missing the jdiff API files for hadoop 2. We're also 
 missing YARN when generating the jdiff API files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11786) Fix Javadoc typos in org.apache.hadoop.fs.FileSystem

2015-04-01 Thread Chen He (JIRA)

Chen He created HADOOP-11786:


 Summary: Fix Javadoc typos in org.apache.hadoop.fs.FileSystem
 Key: HADOOP-11786
 URL: https://issues.apache.org/jira/browse/HADOOP-11786
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial


/**
 * Resets all statistics to 0.
 *
 * In order to reset, we add up all the thread-local statistics data, and
 * set rootData to the negative of that.
 *
 * This may seem like a counterintuitive way to reset the statsitics.  Why
 * can't we just zero out all the thread-local data?  Well, thread-local
 * data can only be modified by the thread that owns it.  If we tried to
 * modify the thread-local data from this thread, our modification might get
 * interleaved with a read-modify-write operation done by the thread that
 * owns the data.  That would result in our update getting lost.
 *
 * The approach used here avoids this problem because it only ever reads
 * (not writes) the thread-local data.  Both reads and writes to rootData
 * are done under the lock, so we're free to modify rootData from any thread
 * that holds the lock.
 */

etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Moved] (HADOOP-11787) OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac


 [ 
https://issues.apache.org/jira/browse/HADOOP-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth moved HDFS-7938 to HADOOP-11787:
--

 Target Version/s: 2.7.0  (was: 2.7.0)
Affects Version/s: (was: 2.7.0)
   2.7.0
  Key: HADOOP-11787  (was: HDFS-7938)
  Project: Hadoop Common  (was: Hadoop HDFS)

 OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit 
 Mac
 

 Key: HADOOP-11787
 URL: https://issues.apache.org/jira/browse/HADOOP-11787
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Kiran Kumar M R
Priority: Critical
 Attachments: HDFS-7938-001.patch


 In OpensslSecureRandom.c, pthread_threadid_np is being used with an unsigned 
 long, but the type signature requires a uint64_t.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11782) Correct two thrown messages in ECSchema class

2015-04-01 Thread Zhe Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HADOOP-11782:
---
Issue Type: Sub-task  (was: Bug)
Parent: HADOOP-11264

 Correct two thrown messages in ECSchema class
 -

 Key: HADOOP-11782
 URL: https://issues.apache.org/jira/browse/HADOOP-11782
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Affects Versions: HDFS-7285
Reporter: Xinwei Qin 
Assignee: Xinwei Qin 
 Fix For: HDFS-7285

 Attachments: HADOOP-11782.000.patch


 when check {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}} value format, 
 the thrown messages are wrong, they should not be {{CHUNK_SIZE_KEY}}, but 
 should be {{NUM_DATA_UNITS_KEY}} and {{NUM_PARITY_UNITS_KEY}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11785) Reduce number of listStatus operation in distcp buildListing()


[ 
https://issues.apache.org/jira/browse/HADOOP-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391095#comment-14391095
 ] 

Hadoop QA commented on HADOOP-11785:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12708737/distcp-liststatus.patch
  against trunk revision 4922394.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-distcp.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6041//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6041//console

This message is automatically generated.

 Reduce number of listStatus operation in distcp buildListing()
 --

 Key: HADOOP-11785
 URL: https://issues.apache.org/jira/browse/HADOOP-11785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Priority: Minor
 Attachments: distcp-liststatus.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Distcp was taking long time in copyListing.buildListing() for large source 
 trees (I was using source of 1.5M files in a tree of about 50K directories). 
 For input at s3 buildListing was taking more than one hour. I've noticed a 
 performance bug in the current code which does listStatus twice for each 
 directory which doubles number of RPCs in some cases (if most directories do 
 not contain 1000 files).
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11627) Remove io.native.lib.available from trunk

2015-04-01 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390971#comment-14390971
 ] 

Brahma Reddy Battula commented on HADOOP-11627:
---

Thanks a lot for taking look into this issue..Updated patch..Kindly review the 
same...

 Remove io.native.lib.available from trunk
 -

 Key: HADOOP-11627
 URL: https://issues.apache.org/jira/browse/HADOOP-11627
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Akira AJISAKA
Assignee: Brahma Reddy Battula
 Attachments: HADOOP-11627-002.patch, HADOOP-11627-003.patch, 
 HADOOP-11627-004.patch, HADOOP-11627-005.patch, HADOOP-11627-006.patch, 
 HADOOP-11627.patch


 According to the discussion in HADOOP-8642, we should remove 
 {{io.native.lib.available}} from trunk, and always use native libraries if 
 they exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11785) Reduce number of listStatus operation in distcp buildListing()

Zoran Dimitrijevic created HADOOP-11785:
---

 Summary: Reduce number of listStatus operation in distcp 
buildListing()
 Key: HADOOP-11785
 URL: https://issues.apache.org/jira/browse/HADOOP-11785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Priority: Minor
 Fix For: 3.0.0
 Attachments: distcp-liststatus.patch

Distcp was taking long time in copyListing.buildListing() for large source 
trees (I was using source of 1.5M files in a tree of about 50K directories). 
For input at s3 buildListing was taking more than one hour. I've noticed a 
performance bug in the current code which does listStatus twice for each 
directory which doubles number of RPCs in some cases (if most directories do 
not contain 1000 files).

 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11785) Reduce number of listStatus operation in distcp buildListing()


 [ 
https://issues.apache.org/jira/browse/HADOOP-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoran Dimitrijevic updated HADOOP-11785:

Attachment: distcp-liststatus.patch

 Reduce number of listStatus operation in distcp buildListing()
 --

 Key: HADOOP-11785
 URL: https://issues.apache.org/jira/browse/HADOOP-11785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Priority: Minor
 Fix For: 3.0.0

 Attachments: distcp-liststatus.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Distcp was taking long time in copyListing.buildListing() for large source 
 trees (I was using source of 1.5M files in a tree of about 50K directories). 
 For input at s3 buildListing was taking more than one hour. I've noticed a 
 performance bug in the current code which does listStatus twice for each 
 directory which doubles number of RPCs in some cases (if most directories do 
 not contain 1000 files).
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11785) Reduce number of listStatus operation in distcp buildListing()


[ 
https://issues.apache.org/jira/browse/HADOOP-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391205#comment-14391205
 ] 

Zoran Dimitrijevic commented on HADOOP-11785:
-

I did not change any tests since this is a performance-bug fix and all existing 
tests pass.

Should I mark this as a bug fix instead of improvement?



 Reduce number of listStatus operation in distcp buildListing()
 --

 Key: HADOOP-11785
 URL: https://issues.apache.org/jira/browse/HADOOP-11785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Priority: Minor
 Attachments: distcp-liststatus.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Distcp was taking long time in copyListing.buildListing() for large source 
 trees (I was using source of 1.5M files in a tree of about 50K directories). 
 For input at s3 buildListing was taking more than one hour. I've noticed a 
 performance bug in the current code which does listStatus twice for each 
 directory which doubles number of RPCs in some cases (if most directories do 
 not contain 1000 files).
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11787) OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac


 [ 
https://issues.apache.org/jira/browse/HADOOP-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HADOOP-11787:
---
   Resolution: Fixed
Fix Version/s: 2.7.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

+1 for the patch.  I verified this on Mac.  I committed this to trunk, branch-2 
and branch-2.7.  Kiran, thank you for the patch.  Colin, thank you for 
reporting the problem.

 OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit 
 Mac
 

 Key: HADOOP-11787
 URL: https://issues.apache.org/jira/browse/HADOOP-11787
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Kiran Kumar M R
Priority: Critical
 Fix For: 2.7.0

 Attachments: HDFS-7938-001.patch


 In OpensslSecureRandom.c, pthread_threadid_np is being used with an unsigned 
 long, but the type signature requires a uint64_t.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11787) OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac


 [ 
https://issues.apache.org/jira/browse/HADOOP-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HADOOP-11787:
---
Component/s: native

 OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit 
 Mac
 

 Key: HADOOP-11787
 URL: https://issues.apache.org/jira/browse/HADOOP-11787
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Kiran Kumar M R
Priority: Critical
 Fix For: 2.7.0

 Attachments: HDFS-7938-001.patch


 In OpensslSecureRandom.c, pthread_threadid_np is being used with an unsigned 
 long, but the type signature requires a uint64_t.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Status: Patch Available  (was: Open)

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Status: Open  (was: Patch Available)

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh


[ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391300#comment-14391300
 ] 

Allen Wittenauer commented on HADOOP-11746:
---

I pretty much don't trust ASF's reviewboard, given it has millions of spam 
accounts on it.  I don't think it's secure at all.

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh

2015-04-01 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391288#comment-14391288
 ] 

Sean Busbey commented on HADOOP-11746:
--

would you mind throwing this on reviewboard? 

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-05.patch

-05:
* more major code cleanup: renamed quite a few functions, removed more cruft, 
etc
* shelldoc capability docs
* added a different table for test failures
* slightly changed the overall +/-1 output
* fixed some issues with relative paths
* reworked some branch detection stuff
* reworked the failed test output list to remove the org.apache part

Need someone to review this now

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11627) Remove io.native.lib.available from trunk

2015-04-01 Thread Brahma Reddy Battula (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HADOOP-11627:
--
Attachment: HADOOP-11627-006.patch

 Remove io.native.lib.available from trunk
 -

 Key: HADOOP-11627
 URL: https://issues.apache.org/jira/browse/HADOOP-11627
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Akira AJISAKA
Assignee: Brahma Reddy Battula
 Attachments: HADOOP-11627-002.patch, HADOOP-11627-003.patch, 
 HADOOP-11627-004.patch, HADOOP-11627-005.patch, HADOOP-11627-006.patch, 
 HADOOP-11627.patch


 According to the discussion in HADOOP-8642, we should remove 
 {{io.native.lib.available}} from trunk, and always use native libraries if 
 they exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11785) Reduce number of listStatus operation in distcp buildListing()


 [ 
https://issues.apache.org/jira/browse/HADOOP-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoran Dimitrijevic updated HADOOP-11785:

Attachment: distcp-liststatus2.patch

removed white space diffs.
reused FileSystem sourceFS in traverse as suggested.

 Reduce number of listStatus operation in distcp buildListing()
 --

 Key: HADOOP-11785
 URL: https://issues.apache.org/jira/browse/HADOOP-11785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Priority: Minor
 Attachments: distcp-liststatus.patch, distcp-liststatus2.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Distcp was taking long time in copyListing.buildListing() for large source 
 trees (I was using source of 1.5M files in a tree of about 50K directories). 
 For input at s3 buildListing was taking more than one hour. I've noticed a 
 performance bug in the current code which does listStatus twice for each 
 directory which doubles number of RPCs in some cases (if most directories do 
 not contain 1000 files).
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9805) Refactor RawLocalFileSystem#rename for improved testability.


[ 
https://issues.apache.org/jira/browse/HADOOP-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391495#comment-14391495
 ] 

Colin Patrick McCabe commented on HADOOP-9805:
--

Looks good.

How about changing renamePOSIX to be named handleEmptyDstDirectoryOnWindows?

renamePOSIX sounds like something that should be called on POSIX platforms, 
not on Windows.  Also, it doesn't suggest that its use case is handling empty 
destination directories.

 Refactor RawLocalFileSystem#rename for improved testability.
 

 Key: HADOOP-9805
 URL: https://issues.apache.org/jira/browse/HADOOP-9805
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Jean-Pierre Matsumoto
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-9805.001.patch, HADOOP-9805.002.patch


 {{RawLocalFileSystem#rename}} contains fallback logic to provide POSIX rename 
 behavior on platforms where {{java.io.File#renameTo}} fails.  The method 
 returns early if {{java.io.File#renameTo}} succeeds, so test runs may not 
 cover the fallback logic depending on the platform.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11785) Reduce number of listStatus operation in distcp buildListing()


 [ 
https://issues.apache.org/jira/browse/HADOOP-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11785:
--
Fix Version/s: (was: 3.0.0)
   Status: Patch Available  (was: Open)

Removing fix version, since that should be set at commit time

 Reduce number of listStatus operation in distcp buildListing()
 --

 Key: HADOOP-11785
 URL: https://issues.apache.org/jira/browse/HADOOP-11785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Priority: Minor
 Attachments: distcp-liststatus.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Distcp was taking long time in copyListing.buildListing() for large source 
 trees (I was using source of 1.5M files in a tree of about 50K directories). 
 For input at s3 buildListing was taking more than one hour. I've noticed a 
 performance bug in the current code which does listStatus twice for each 
 directory which doubles number of RPCs in some cases (if most directories do 
 not contain 1000 files).
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11787) OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac

2015-04-01 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391343#comment-14391343
 ] 

Hudson commented on HADOOP-11787:
-

FAILURE: Integrated in Hadoop-trunk-Commit #7487 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7487/])
HADOOP-11787. OpensslSecureRandom.c pthread_threadid_np usage signature is 
wrong on 32-bit Mac. Contributed by Kiran Kumar M R. (cnauroth: rev 
a3a96a07faf0c6f6aa3ed33608271c2b1657e437)
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c


 OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit 
 Mac
 

 Key: HADOOP-11787
 URL: https://issues.apache.org/jira/browse/HADOOP-11787
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Kiran Kumar M R
Priority: Critical
 Fix For: 2.7.0

 Attachments: HDFS-7938-001.patch


 In OpensslSecureRandom.c, pthread_threadid_np is being used with an unsigned 
 long, but the type signature requires a uint64_t.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh

2015-04-01 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391348#comment-14391348
 ] 

Sean Busbey commented on HADOOP-11746:
--

I don't follow. Presuming it is insecure, what could we lose by having the 
patch there? Is there another review tool that's more accessible than locally 
applying an putting comments here in Jira?

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


 [ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay Bhat updated HADOOP-8320:
---
Attachment: HADOOP-8320.007.patch

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch, HADOOP-8320.002.patch, 
 HADOOP-8320.003.patch, HADOOP-8320.004.patch, HADOOP-8320.005.patch, 
 HADOOP-8320.007.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11785) Reduce number of listStatus operation in distcp buildListing()


[ 
https://issues.apache.org/jira/browse/HADOOP-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391432#comment-14391432
 ] 

Colin Patrick McCabe commented on HADOOP-11785:
---

Thanks, [~3opan].  This looks good in general.

bq. Should I mark this as a bug fix instead of improvement?

I don't see this as a bug because the functionality is correct.  It seems to be 
an improvement.

{code}
-   * Collect the list of 
+   * Collect the list of
-   * the the source root is a directory, then the source root entry is not 
+   * the the source root is a directory, then the source root entry is not
-if (fileStatus.getPath().equals(sourcePathRoot)  
+if (fileStatus.getPath().equals(sourcePathRoot) 
{code}

Can you remove these whitespace changes from the patch?  It's distracting and 
it makes it look like things have changed, when in fact they have not.  I think 
there are a few other whitespace changes as well.

{{traverseDirectory}}: Maybe we can optimize this even more.  Can we pass in 
the sourceFS to this function, rather than calling {{Path#getFileSystem}}?  
{{Path#getFileSystem}} requires some synchronization which might add overheads.

It looks good aside from that.  thanks

 Reduce number of listStatus operation in distcp buildListing()
 --

 Key: HADOOP-11785
 URL: https://issues.apache.org/jira/browse/HADOOP-11785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Priority: Minor
 Attachments: distcp-liststatus.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Distcp was taking long time in copyListing.buildListing() for large source 
 trees (I was using source of 1.5M files in a tree of about 50K directories). 
 For input at s3 buildListing was taking more than one hour. I've noticed a 
 performance bug in the current code which does listStatus twice for each 
 directory which doubles number of RPCs in some cases (if most directories do 
 not contain 1000 files).
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh


[ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391453#comment-14391453
 ] 

Hadoop QA commented on HADOOP-11746:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12708777/HADOOP-11746-05.patch
  against trunk revision ed72daa.

{color:red}-1 @author{color}.  The patch appears to contain 13 @author tags 
which the Hadoop community has agreed to not allow in code contributions.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6043//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6043//console

This message is automatically generated.

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11787) OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac

2015-04-01 Thread Kiran Kumar M R (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391337#comment-14391337
 ] 

Kiran Kumar M R commented on HADOOP-11787:
--

Chris, thanks for verifying on Mac and comitting

 OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit 
 Mac
 

 Key: HADOOP-11787
 URL: https://issues.apache.org/jira/browse/HADOOP-11787
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Kiran Kumar M R
Priority: Critical
 Fix For: 2.7.0

 Attachments: HDFS-7938-001.patch


 In OpensslSecureRandom.c, pthread_threadid_np is being used with an unsigned 
 long, but the type signature requires a uint64_t.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11623) Native compilation fails on Solaris due to use of syscall function.

2015-04-01 Thread Kiran Kumar M R (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391374#comment-14391374
 ] 

Kiran Kumar M R commented on HADOOP-11623:
--

syscall was used in
{{hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c}}
{{hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c}}

OpensslSecureRandom.c has been fixed in  HADOOP-11638. It has Solaris specific 
code.
NativeIO.c still has syscall.

Are you referring to these two files?

 Native compilation fails on Solaris due to use of syscall function.
 ---

 Key: HADOOP-11623
 URL: https://issues.apache.org/jira/browse/HADOOP-11623
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.6.0
 Environment: Solaris 11.2
Reporter: Malcolm Kavalsky
Assignee: Malcolm Kavalsky
   Original Estimate: 168h
  Remaining Estimate: 168h

 Solaris does not provide the syscall function. Currently, hadoop has very 
 limited use of this function ( only 2 files ). These need to be ifdeffed 
 with the correct alternatives for Solaris.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9805) Refactor RawLocalFileSystem#rename for improved testability.


[ 
https://issues.apache.org/jira/browse/HADOOP-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391612#comment-14391612
 ] 

Chris Nauroth commented on HADOOP-9805:
---

It's unfortunate that the new method has to run {{pathToFile}} on the source 
and destination again.  There is quite a bit of work behind that method call, 
and the result is already available in the scope of {{rename}}.  
I suggest that the new method accept the arguments as both {{Path}} and 
{{File}} to avoid the redundant call.  Combining that with Colin's rename 
suggestion, the signature would be:

{code}
public final boolean handleEmptyDstDirectoryOnWindows(Path src, File srcFile, 
Path dst, File dstFile) throws IOException
{code}

Admittedly, that's a strange method signature, but this is only meant to be 
visible for testing, not as a public API.  To that end, let's also mark the 
method {{final}}.

bq. Of course this bug is Windows-specific and we could easily fix it in a 
follow-on JIRA (probably via JNI code).

I'm not sure this is true.  Last time I checked, it looked like the Windows API 
did not offer any way to match the POSIX semantics exactly.  I'd be happy to be 
proven wrong though, because the current approach is admittedly not atomic.  :-)

 Refactor RawLocalFileSystem#rename for improved testability.
 

 Key: HADOOP-9805
 URL: https://issues.apache.org/jira/browse/HADOOP-9805
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Jean-Pierre Matsumoto
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-9805.001.patch, HADOOP-9805.002.patch


 {{RawLocalFileSystem#rename}} contains fallback logic to provide POSIX rename 
 behavior on platforms where {{java.io.File#renameTo}} fails.  The method 
 returns early if {{java.io.File#renameTo}} succeeds, so test runs may not 
 cover the fallback logic depending on the platform.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh


[ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391635#comment-14391635
 ] 

Allen Wittenauer commented on HADOOP-11746:
---

I actively try to avoid services that are so obviously compromised and not 
maintained since who knows what other problems may exist... iframe injections, 
etc.   Plus, Hadoop uses JIRA as the platform for this.  I understand that 
other projects use different tools, but we don't.

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11627) Remove io.native.lib.available from trunk


[ 
https://issues.apache.org/jira/browse/HADOOP-11627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391663#comment-14391663
 ] 

Hadoop QA commented on HADOOP-11627:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12708734/HADOOP-11627-006.patch
  against trunk revision 2e79f1c.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

  org.apache.hadoop.mapred.pipes.TestPipeApplication
  org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
  org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
  org.apache.hadoop.mapred.TestMRTimelineEventHandling
  org.apache.hadoop.mapred.TestClusterMRNotification

  The following test timeouts occurred in 
hadoop-common-project/hadoop-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapreduce.TestMRJobClient
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapreduce.TestLargeSort
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapreduce.v2.TestUberAM

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6040//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6040//console

This message is automatically generated.

 Remove io.native.lib.available from trunk
 -

 Key: HADOOP-11627
 URL: https://issues.apache.org/jira/browse/HADOOP-11627
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Akira AJISAKA
Assignee: Brahma Reddy Battula
 Attachments: HADOOP-11627-002.patch, HADOOP-11627-003.patch, 
 HADOOP-11627-004.patch, HADOOP-11627-005.patch, HADOOP-11627-006.patch, 
 HADOOP-11627.patch


 According to the discussion in HADOOP-8642, we should remove 
 {{io.native.lib.available}} from trunk, and always use native libraries if 
 they exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh


[ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391696#comment-14391696
 ] 

Colin Patrick McCabe commented on HADOOP-11746:
---

[~busbey], have you tried out 
https://chrome.google.com/webstore/detail/git-patch-viewer/hkoggakcdopbgnaeeidcmopfekipkleg
 ?  I find it really helpful for viewing diffs on JIRA.

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, 
 HADOOP-11746-05.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HADOOP-11746) rewrite test-patch.sh