[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[ https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686432#comment-13686432 ] Hadoop QA commented on HADOOP-9637: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588295/HADOOP-9637-trunk.2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2663//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2663//console This message is automatically generated. Adding Native Fstat for Windows as needed by YARN - Key: HADOOP-9637 URL: https://issues.apache.org/jira/browse/HADOOP-9637 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.patch In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, {{SecureIOUtils.openForRead()}} was called to enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this. The JIRA try to provide a Windows implementation of {{NativeIO.Posix.getFstat()}}. TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message Owner ... for path ... did not match expected owner The check on file owner happens at {{AggregatedLogFormat.write()}} method. The method calls {{SecureIOUtils.openForRead()}} to read the log files before writing out to the OutputStream. {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the file owner and group. We don't have {{NativeIO.Posix.getFstat()}} implementation on Windows; thus, the failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9515) Add general interface for NFS and Mount
[ https://issues.apache.org/jira/browse/HADOOP-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686579#comment-13686579 ] Hudson commented on HADOOP-9515: Integrated in Hadoop-Yarn-trunk #244 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/244/]) CHANGES.txt change for HADOOP-9515 (Revision 1493929) HADOOP-9515. Add general interface for NFS and Mount. Contributed by Brandon Li (Revision 1493924) Result = SUCCESS brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493929 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493924 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountEntry.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountInterface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsFileType.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsTime.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3 * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/FileHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3FileAttributes.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Interface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Status.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/ACCESS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/COMMIT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/CREATE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSINFO3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSSTAT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/GETATTR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/LOOKUP3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/MKDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/PATHCONF3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READ3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIRPLUS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READLINK3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/REMOVE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RENAME3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RMDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RequestWithHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/SETATTR3Request.java *
[jira] [Commented] (HADOOP-9515) Add general interface for NFS and Mount
[ https://issues.apache.org/jira/browse/HADOOP-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686675#comment-13686675 ] Hudson commented on HADOOP-9515: Integrated in Hadoop-Hdfs-trunk #1434 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1434/]) CHANGES.txt change for HADOOP-9515 (Revision 1493929) HADOOP-9515. Add general interface for NFS and Mount. Contributed by Brandon Li (Revision 1493924) Result = FAILURE brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493929 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493924 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountEntry.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountInterface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsFileType.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsTime.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3 * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/FileHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3FileAttributes.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Interface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Status.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/ACCESS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/COMMIT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/CREATE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSINFO3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSSTAT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/GETATTR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/LOOKUP3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/MKDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/PATHCONF3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READ3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIRPLUS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READLINK3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/REMOVE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RENAME3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RMDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RequestWithHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/SETATTR3Request.java *
[jira] [Commented] (HADOOP-9515) Add general interface for NFS and Mount
[ https://issues.apache.org/jira/browse/HADOOP-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686746#comment-13686746 ] Hudson commented on HADOOP-9515: Integrated in Hadoop-Mapreduce-trunk #1461 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1461/]) CHANGES.txt change for HADOOP-9515 (Revision 1493929) HADOOP-9515. Add general interface for NFS and Mount. Contributed by Brandon Li (Revision 1493924) Result = SUCCESS brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493929 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493924 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountEntry.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountInterface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsFileType.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsTime.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3 * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/FileHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3FileAttributes.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Interface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Status.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/ACCESS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/COMMIT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/CREATE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSINFO3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSSTAT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/GETATTR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/LOOKUP3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/MKDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/PATHCONF3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READ3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIRPLUS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READLINK3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/REMOVE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RENAME3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RMDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RequestWithHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/SETATTR3Request.java *
[jira] [Created] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
Steve Loughran created HADOOP-9651: -- Summary: Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists Key: HADOOP-9651 URL: https://issues.apache.org/jira/browse/HADOOP-9651 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Steve Loughran Priority: Minor While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if you try to create a file that exists and you have set {{overwrite=false}}, {{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it impossible to distinguish a create operation failing from a fixable problem (the file is there) and something more fundamental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[ https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686763#comment-13686763 ] Steve Loughran commented on HADOOP-9651: Here I propose # saying MUST throw {{FileAlreadyExistsException}} # fixing all the hadoop filesystems to do this where they do not today. # making sure the FS tests check for this. HDFS already has the correct behaviour, as do the blobstore(s). Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists - Key: HADOOP-9651 URL: https://issues.apache.org/jira/browse/HADOOP-9651 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Steve Loughran Priority: Minor While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if you try to create a file that exists and you have set {{overwrite=false}}, {{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it impossible to distinguish a create operation failing from a fixable problem (the file is there) and something more fundamental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[ https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686765#comment-13686765 ] Steve Loughran commented on HADOOP-9651: stack {code} ava.io.IOException: File already exists: file:/Users/stevel/tmp/buffer.json at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:276) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:269) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.init(ChecksumFileSystem.java:382) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:430) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:411) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:888) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:869) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:768) {code} Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists - Key: HADOOP-9651 URL: https://issues.apache.org/jira/browse/HADOOP-9651 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Steve Loughran Priority: Minor While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if you try to create a file that exists and you have set {{overwrite=false}}, {{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it impossible to distinguish a create operation failing from a fixable problem (the file is there) and something more fundamental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[ https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686983#comment-13686983 ] Chris Nauroth commented on HADOOP-9637: --- Thanks for addressing my questions. It all makes sense. I have just one more comment about the new patch. {code} if (Shell.WINDOWS) { int i = owner.indexOf('\\'); if (i != -1) this.owner = owner.substring(i + 1); } {code} This code is identical for username and group, so let's refactor it behind a helper function. We also can add comments stating that the reason for this is to remove domain. Adding Native Fstat for Windows as needed by YARN - Key: HADOOP-9637 URL: https://issues.apache.org/jira/browse/HADOOP-9637 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.patch In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, {{SecureIOUtils.openForRead()}} was called to enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this. The JIRA try to provide a Windows implementation of {{NativeIO.Posix.getFstat()}}. TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message Owner ... for path ... did not match expected owner The check on file owner happens at {{AggregatedLogFormat.write()}} method. The method calls {{SecureIOUtils.openForRead()}} to read the log files before writing out to the OutputStream. {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the file owner and group. We don't have {{NativeIO.Posix.getFstat()}} implementation on Windows; thus, the failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687022#comment-13687022 ] Jason Lowe commented on HADOOP-9582: bq. I did do this in one my previous patch,however Steve(on 22/May/13 06:01) recommended that raising AssertionFailedError would be better for diagnostics. Ah, yes, that makes sense for better diagnostics. I'm +1 on the latest patch and will commit later today. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582-4.txt, HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[ https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré updated HADOOP-9651: - Attachment: HADOOP-9651.patch Patch for the RawLocalFileSystem (including unit tests) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists - Key: HADOOP-9651 URL: https://issues.apache.org/jira/browse/HADOOP-9651 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Steve Loughran Priority: Minor Attachments: HADOOP-9651.patch While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if you try to create a file that exists and you have set {{overwrite=false}}, {{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it impossible to distinguish a create operation failing from a fixable problem (the file is there) and something more fundamental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[ https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré updated HADOOP-9651: - Status: Patch Available (was: Open) This patch updates the RawLocalFileSystem to throw FileAlreadyExistsException. Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists - Key: HADOOP-9651 URL: https://issues.apache.org/jira/browse/HADOOP-9651 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Steve Loughran Priority: Minor Attachments: HADOOP-9651.patch While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if you try to create a file that exists and you have set {{overwrite=false}}, {{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it impossible to distinguish a create operation failing from a fixable problem (the file is there) and something more fundamental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[ https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687119#comment-13687119 ] Jean-Baptiste Onofré commented on HADOOP-9651: -- I'm reviewing the other filesystems to see if some updates are required to correctly throw FileAlreadyExistsException. Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists - Key: HADOOP-9651 URL: https://issues.apache.org/jira/browse/HADOOP-9651 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Steve Loughran Priority: Minor Attachments: HADOOP-9651.patch While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if you try to create a file that exists and you have set {{overwrite=false}}, {{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it impossible to distinguish a create operation failing from a fixable problem (the file is there) and something more fundamental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8040) Add symlink support to FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687131#comment-13687131 ] Colin Patrick McCabe commented on HADOOP-8040: -- {code} protected static void checkNotSchemeWithRelative(final Path path) protected static void checkNotRelative(final Path path) public static String getPathWithoutSchemeAndAuthority(Path path) {code} Why are these static methods rather than instance methods? They seem to just operate on a single path, and don't do anything smart when you pass null, so I can't see why you'd want them to be static. I also don't see why you would want {{protected}} here, rather than package-private (i.e. no qualifier.). Also, a function whose name begins with getPath... should return a path. Let the caller call toString if he wants to. {code} System.out.println(RLFS wd: + workingDir); {code} Debug printout left in. {code} private FileStatus getFileLinkStatusInternal(final Path f) throws IOException { String target = readLink(f); try { FileStatus fs = getFileStatus(f); // If f refers to a regular file or directory if (target.isEmpty()) { return fs; } // Otherwise f refers to a symlink return new FileStatus(fs.getLen(), false, fs.getReplication(), fs.getBlockSize(), fs.getModificationTime(), fs.getAccessTime(), fs.getPermission(), fs.getOwner(), fs.getGroup(), new Path(target), f); {code} Symlinks do not necessarily have the same owner as the file they point to. Here's any example: {code} cmccabe@keter:~/hadoop2 touch a cmccabe@keter:~/hadoop2 ln -s a b cmccabe@keter:~/hadoop2 sudo chown bob b root's password: cmccabe@keter:~/hadoop2 ls -l a b -rw-r--r-- 1 bob users 0 Jun 18 12:14 a lrwxrwxrwx 1 cmccabe users 1 Jun 18 12:14 b - a {code} Symlinks have a file mode (aka permission bits) which are independent of the files they point to. On certain UNIXes (but not Linux), symlinks can have permissions other than 0777. I'm not sure if you can get access to lstat from JDK6, or if you'll need JNI for this. {code} final public FileStatus makeQualified(URI defaultUri, Path path) { {code} This is basically a copy constructor that also modifies {{HdfsFileStatus#path}}. Are you sure it wouldn't be easier to just have a method to change the path of the object? As new fields are added to HdfsFileStatus, it's going to be very, very easy for people to forget to change this function to copy them. In fact, you seem to have left a field out here yourself-- fileId. This is especially an issue when subclasses start coming into play. All the problems that plague clone() plague this method too. I also feel like having a mutator like setParentPath(URI defaultURI, Path parent) would be more efficient, as well as more maintainable. FSLinkResolver: it seems like you only need one of these per function. Since they are stateless, there isn't any point in creating garbage that you'll have to collect later. Just make a static FSLinkResolverContentSummary and use that in getContentSummary. etc. etc. Also, should MAX_PATH_LINKS be configurable? Add symlink support to FileSystem - Key: HADOOP-8040 URL: https://issues.apache.org/jira/browse/HADOOP-8040 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha Reporter: Eli Collins Assignee: Andrew Wang Attachments: hadoop-8040-1.patch, hadoop-8040-2.patch, hadoop-8040-3.patch, hadoop-8040-4.patch, hadoop-8040-5.patch HADOOP-6421 added symbolic links to FileContext. Resolving symlinks is done on the client-side, and therefore requires client support. An HDFS symlink (created by FileContext) when accessed by FileSystem will result in an unhandled UnresolvedLinkException. Because not all users will migrate from FileSystem to FileContext in lock step, and we want users of FileSystem to be able to access all paths created by FileContext, we need to support symlink resolution in FileSystem as well, to facilitate migration to FileContext. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HADOOP-9421: -- Priority: Blocker (was: Major) Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9648) Fix build native library on mac osx
[ https://issues.apache.org/jira/browse/HADOOP-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687136#comment-13687136 ] Arpit Agarwal commented on HADOOP-9648: --- Thanks for taking this up Kirill. I verified that the native library builds on OSX with your changes. Few comments from the 2.0.5 patch. # Tests are not able to load the native library. TestNativeIO fails with Unable to load native-hadoop library for your platform... using builtin-java classes where applicable. # The working directory should be restored before returning to avoid changing the callers cwd. I think it would be best to use the mkdirat and openat functions except on OS X via conditional compilation. {code} -if (mkdirat(cwd, token, perm) != 0) { +if(fchdir(cwd) 0) { {code} # The iterative approach to replace fcloseall looks somewhat inefficient. At the very least it should be restricted to OS X and fcloseall() used everywhere else. {code} + for (; rl.rlim_max 2; rl.rlim_max--) { +close(rl.rlim_max); + } {code} # Is this change necessary? I can build without it. {code} -version2.4.0a/version +version2.5.0/version {code} Fix build native library on mac osx --- Key: HADOOP-9648 URL: https://issues.apache.org/jira/browse/HADOOP-9648 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.4, 1.2.0, 1.1.2, 2.0.5-alpha Reporter: Kirill A. Korinskiy Attachments: HADOOP-9648-native-osx.1.0.4.patch, HADOOP-9648-native-osx.1.1.2.patch, HADOOP-9648-native-osx.1.2.0.patch, HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch Some patches for fixing build a hadoop native library on os x 10.7/10.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[ https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu updated HADOOP-9637: -- Attachment: HADOOP-9637-trunk.3.patch Attaching a new patch. Adding Native Fstat for Windows as needed by YARN - Key: HADOOP-9637 URL: https://issues.apache.org/jira/browse/HADOOP-9637 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch, HADOOP-9637-trunk.patch In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, {{SecureIOUtils.openForRead()}} was called to enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this. The JIRA try to provide a Windows implementation of {{NativeIO.Posix.getFstat()}}. TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message Owner ... for path ... did not match expected owner The check on file owner happens at {{AggregatedLogFormat.write()}} method. The method calls {{SecureIOUtils.openForRead()}} to read the log files before writing out to the OutputStream. {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the file owner and group. We don't have {{NativeIO.Posix.getFstat()}} implementation on Windows; thus, the failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xi Fang updated HADOOP-9624: Attachment: HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name --- Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[ https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687153#comment-13687153 ] Hadoop QA commented on HADOOP-9651: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588436/HADOOP-9651.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2664//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2664//console This message is automatically generated. Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists - Key: HADOOP-9651 URL: https://issues.apache.org/jira/browse/HADOOP-9651 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Steve Loughran Priority: Minor Attachments: HADOOP-9651.patch While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if you try to create a file that exists and you have set {{overwrite=false}}, {{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it impossible to distinguish a create operation failing from a fixable problem (the file is there) and something more fundamental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xi Fang updated HADOOP-9624: Attachment: HADOOP-9624.branch-1.patch TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name --- Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9622) bzip2 codec can drop records when reading data in splits
[ https://issues.apache.org/jira/browse/HADOOP-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687158#comment-13687158 ] Nathan Roberts commented on HADOOP-9622: Thanks for adding the detailed comment. Well explained! +1 for the patch. bzip2 codec can drop records when reading data in splits Key: HADOOP-9622 URL: https://issues.apache.org/jira/browse/HADOOP-9622 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.4-alpha, 0.23.8 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: blockEndingInCRThenLF.txt.bz2, blockEndingInCR.txt.bz2, HADOOP-9622-2.patch, HADOOP-9622.patch, HADOOP-9622-testcase.patch Bzip2Codec.BZip2CompressionInputStream can cause records to be dropped when reading them in splits based on where record delimiters occur relative to compression block boundaries. Thanks to [~knoguchi] for discovering this problem while working on PIG-3251. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xi Fang updated HADOOP-9624: Attachment: HAOOP-9624.branch-1.patch TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name --- Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xi Fang updated HADOOP-9624: Attachment: (was: HAOOP-9624.branch-1.patch) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name --- Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687160#comment-13687160 ] Hadoop QA commented on HADOOP-9624: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588447/HADOOP-9624.branch-1.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2667//console This message is automatically generated. TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name --- Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687162#comment-13687162 ] Xi Fang commented on HADOOP-9624: - Thanks Chris. Two patches have been attached. TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name --- Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[ https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687168#comment-13687168 ] Hadoop QA commented on HADOOP-9637: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588442/HADOOP-9637-trunk.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2665//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2665//console This message is automatically generated. Adding Native Fstat for Windows as needed by YARN - Key: HADOOP-9637 URL: https://issues.apache.org/jira/browse/HADOOP-9637 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch, HADOOP-9637-trunk.patch In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, {{SecureIOUtils.openForRead()}} was called to enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this. The JIRA try to provide a Windows implementation of {{NativeIO.Posix.getFstat()}}. TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message Owner ... for path ... did not match expected owner The check on file owner happens at {{AggregatedLogFormat.write()}} method. The method calls {{SecureIOUtils.openForRead()}} to read the log files before writing out to the OutputStream. {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the file owner and group. We don't have {{NativeIO.Posix.getFstat()}} implementation on Windows; thus, the failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687184#comment-13687184 ] Hadoop QA commented on HADOOP-9624: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588443/HADOOP-9624.trunk.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2666//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2666//console This message is automatically generated. TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name --- Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687233#comment-13687233 ] Daryn Sharp commented on HADOOP-9421: - Per Sanjay's request, here's a quick summary: This patch does not direct address deficiencies in negotiation process. It merely provides the flexibility to fix them in the near future. The basic changes are: * SASL protobufs * SASL protobufs wrapped with RPC header * Server advertised SASL auth methods - currently ignored by client Wrapping SASL negotiation in a RPC header avoids decoding errors. Ex. A secure client currently tries to decode a RPC exception from an incompatible connection header as a SASL message. The client also can't handle a RPC exception mid-stream during SASL negotiation. The RPC header allows the server and client to correctly decode the payload. Using the RPC headers, with the addition of a streamId, will also allow the _future potential_ to multiplex multiple UGIs over the same connection. Services like the NN can be overwhelmed by a stampede of connections. The NM may be an ideal candidate for aggregation of connections, or even a per-node multiplexor for tasks. Server advertisement of auth methods aims to address the current limitation of allowing 1 pre-determined auth or simple. Additional consideration is needed for a client-side whitelist to avoid server unexpectedly requesting weak authentication. In the current patch, the client ignores the server and blindly attempts the negotiation as it does today. +Other future features unlocked by this design+ * Brings us much closer to pluggable auth methods w/o changing server and client code * Client may select a server advertised auth method to: *# Support services and/or clusters with heterogenous auth methods (ex. thinking of knox, rhino, etc) *# *Critical to Y!*: IP failover, especially for HA with kerberos. The client needs the active NN's host to acquire a service ticket *# Simplify token selection by using an opaque identifier supplied by the server - will eliminate use_ip, and even the complex HA token logic *# Support accessing multi-interface hosts on all interfaces *# Support accessing services via any of their hostnames, ips, or CNAMEs * Ability to greatly reduce complexity of client/server auth code, and cleanly decouple SASL logic * As mentioned before, multiplexing of different UGIs over a shared connection Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687268#comment-13687268 ] Hudson commented on HADOOP-9582: Integrated in Hadoop-trunk-Commit #3970 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3970/]) HADOOP-9582. Non-existent file to hadoop fs -conf doesn't throw error. Contributed by Ashwin Shankar (Revision 1494331) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1494331 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShell.java Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582-4.txt, HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-9582: --- Resolution: Fixed Fix Version/s: 2.3.0 0.23.9 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Ashwin! I committed this to trunk, branch-2, and branch-0.23. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Fix For: 3.0.0, 0.23.9, 2.3.0 Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582-4.txt, HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[ https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9637: -- Hadoop Flags: Reviewed +1 for the patch. I verified on Mac (without native libs), Linux (with native libs), and Windows (with native libs). Chuan, thank you for incorporating all of the feedback. I'll commit this. Adding Native Fstat for Windows as needed by YARN - Key: HADOOP-9637 URL: https://issues.apache.org/jira/browse/HADOOP-9637 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch, HADOOP-9637-trunk.patch In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, {{SecureIOUtils.openForRead()}} was called to enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this. The JIRA try to provide a Windows implementation of {{NativeIO.Posix.getFstat()}}. TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message Owner ... for path ... did not match expected owner The check on file owner happens at {{AggregatedLogFormat.write()}} method. The method calls {{SecureIOUtils.openForRead()}} to read the log files before writing out to the OutputStream. {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the file owner and group. We don't have {{NativeIO.Posix.getFstat()}} implementation on Windows; thus, the failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8040) Add symlink support to FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-8040: Attachment: hadoop-8040-6.patch Thanks for the review Colin. I'm attaching just a consolidated patch again since the changes I made were minor; I wanted to follow up with some of your review comments before posting the patch split. bq. Why are these static methods rather than instance methods? I refactored these out of FileContext and decided to stick them in Path. Seems kind of weird to have a check instance method that throws an exception like this, so I'd mildly prefer to leave them static. bq. package-private (i.e. no qualifier.). fixed bq. a function whose name begins with getPath... should return a path. fixed, this was also a refactor but good point bq. Debug printout left in. fixed bq. Symlinks do not necessarily have the same owner as the file they point to. Here's any example: bq. Symlinks have a file mode (aka permission bits) which are independent of the files they point to... I copied this bit from the FileContext implementation, RawLocalFS. It just uses the target's status which, as you correctly noted, can be different from the link's status. Semantics for all the local filesystems is kinda fuzzy, but I agree that this feels incorrect. I'd prefer to fix both of these classes at once though in a follow-on JIRA (especially if JNI is potentially involved). {code} final public FileStatus makeQualified(URI defaultUri, Path path) { {code} bq. Are you sure it wouldn't be easier to just have a method to change the path of the object? Mutating feels wrong to me based on how makeQualified is used. The path is often passed into a method which then tries to qualify it (e.g. Hdfs#getFileStatus). Mutating in place means the caller gets back a qualified path, which is probably not what they want. We could either mutate and make copies of the original path in all these places, or just leave it as it is. bq. In fact, you seem to have left a field out here yourself-- fileId. Hmm, interesting comment, good eye. The issue here is that since symlinks can point to other filesystems, we have to return a FileStatus, not an HdfsFileStatus. Plain old FileStatus doesn't have fileId, so we have to leave it out. I think this is okay from a user perspective, since FileSystem methods only return FileStatus, and HdfsFileStatus isn't a subclass of FileStatus anyway. bq. FSLinkResolver: it seems like you only need one of these per functionJust make a static FSLinkResolverContentSummary and use that... I think this is as intended. The new inner class sometimes needs to wrap final parameters of the containing method. Since the params are different each time (and different per call), I think it needs to do this at runtime. bq. should MAX_PATH_LINKS be configurable? I think the intent here was to just pick a reasonable upper bound. I doubt any real user has 32 links to links, and I don't think there's a reason to tune it down either. Add symlink support to FileSystem - Key: HADOOP-8040 URL: https://issues.apache.org/jira/browse/HADOOP-8040 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha Reporter: Eli Collins Assignee: Andrew Wang Attachments: hadoop-8040-1.patch, hadoop-8040-2.patch, hadoop-8040-3.patch, hadoop-8040-4.patch, hadoop-8040-5.patch, hadoop-8040-6.patch HADOOP-6421 added symbolic links to FileContext. Resolving symlinks is done on the client-side, and therefore requires client support. An HDFS symlink (created by FileContext) when accessed by FileSystem will result in an unhandled UnresolvedLinkException. Because not all users will migrate from FileSystem to FileContext in lock step, and we want users of FileSystem to be able to access all paths created by FileContext, we need to support symlink resolution in FileSystem as well, to facilitate migration to FileContext. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[ https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687292#comment-13687292 ] Hudson commented on HADOOP-9637: Integrated in Hadoop-trunk-Commit #3971 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3971/]) HADOOP-9637. Adding Native Fstat for Windows as needed by YARN. Contributed by Chuan Liu. (Revision 1494341) Result = SUCCESS cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1494341 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chmod.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chown.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/ls.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java Adding Native Fstat for Windows as needed by YARN - Key: HADOOP-9637 URL: https://issues.apache.org/jira/browse/HADOOP-9637 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch, HADOOP-9637-trunk.patch In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, {{SecureIOUtils.openForRead()}} was called to enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this. The JIRA try to provide a Windows implementation of {{NativeIO.Posix.getFstat()}}. TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message Owner ... for path ... did not match expected owner The check on file owner happens at {{AggregatedLogFormat.write()}} method. The method calls {{SecureIOUtils.openForRead()}} to read the log files before writing out to the OutputStream. {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the file owner and group. We don't have {{NativeIO.Posix.getFstat()}} implementation on Windows; thus, the failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[ https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9637: -- Resolution: Fixed Fix Version/s: 2.1.0-beta 3.0.0 Target Version/s: 3.0.0, 2.1.0-beta Status: Resolved (was: Patch Available) I committed this to trunk, branch-2, and branch-2.1-beta. Chuan, thank you for the contribution. Adding Native Fstat for Windows as needed by YARN - Key: HADOOP-9637 URL: https://issues.apache.org/jira/browse/HADOOP-9637 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 3.0.0, 2.1.0-beta Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch, HADOOP-9637-trunk.patch In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, {{SecureIOUtils.openForRead()}} was called to enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this. The JIRA try to provide a Windows implementation of {{NativeIO.Posix.getFstat()}}. TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message Owner ... for path ... did not match expected owner The check on file owner happens at {{AggregatedLogFormat.write()}} method. The method calls {{SecureIOUtils.openForRead()}} to read the log files before writing out to the OutputStream. {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the file owner and group. We don't have {{NativeIO.Posix.getFstat()}} implementation on Windows; thus, the failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687295#comment-13687295 ] Todd Lipcon commented on HADOOP-9439: - - seems like {{pw_lock_object}} is declared in two .c files... one should probably be marked extern - The following code is smelly to me: {code} int i, ret, nvalid, pw_lock_locked = 0; {code} because only the last variable is actually initialized. I don't _think_ you end up using any of the uninitialized variables, so it's not a true bug, but I think it's worth separating them out into their own lines. Do we not allow C99 style declaration in the middle of a function in our JNI code? I've always liked that better than the original C style of declaring all at the top. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687314#comment-13687314 ] Colin Patrick McCabe commented on HADOOP-9439: -- I don't like C99 style declarations because they tend to introduce bugs. When you have some cleanup label at the end of the function, if your goto passes over the variable declaration, the variable will be treated as undefined. So, for example, if you have: int i = 4; JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687317#comment-13687317 ] Colin Patrick McCabe commented on HADOOP-9439: -- ok, these keyboard shortcuts defeated me again. I wish I could turn that off. Anyway, if you have: {code} goto cleanup: struct foo *foo= calloc(1, sizeof(*foo)); ... cleanup: free(foo); {code} then you're freeing an uninitialized pointer. Not good. I also don't like declarations in the middle because of a philosophical reason: it tends to lead to excessively long functions, rather than encouraging modularity. As to the extern thing, yeah, I agree. It's rather odd that the linker does that kind of symbol merging, but apparently it does. I don't see the point of putting all variable declarations on separate lines. Java allows declaration of variables on the same line as well. C++0x allows things like vector to be initialized with a reasonable syntax, so you are finally allowed to admit that this is a good thing, you C++ advocate, you. :) JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9624: -- Description: TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. was: TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9624: -- Summary: TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name (was: TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that once the enlistment root path has X in its name, TEST_ROOT_DIR will also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9624: -- Attachment: HADOOP-9624.branch-1.2.patch I'm attaching a very minor update to the branch-1 patch, changing and to or in the comment, again just for consistency between the 2 branches. +1 for both patches. I'll commit these. TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.2.patch, HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9624: -- Target Version/s: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 (was: 1-win) Affects Version/s: 1.3.0 2.1.0-beta 3.0.0 Hadoop Flags: Reviewed TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.2.patch, HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9652) FileContext#getFileLinkStatus does not fill in the link owner and mode
Colin Patrick McCabe created HADOOP-9652: Summary: FileContext#getFileLinkStatus does not fill in the link owner and mode Key: HADOOP-9652 URL: https://issues.apache.org/jira/browse/HADOOP-9652 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe {{FileContext#getFileLinkStatus}} does not actually get the owner and mode of the symlink, but instead uses the owner and mode of the symlink target. If the target can't be found, it fills in bogus values (the empty string and FsPermission.getDefault) for these. Symlinks have an owner distinct from who created them, and getFileLinkStatus ought to expose this. In some operating systems, symlinks can have a permission other than 0777. We ought to expose this in RawLocalFilesystem and other places, although we don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687347#comment-13687347 ] Hadoop QA commented on HADOOP-9624: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588475/HADOOP-9624.branch-1.2.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2669//console This message is automatically generated. TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.2.patch, HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687359#comment-13687359 ] Hudson commented on HADOOP-9624: Integrated in Hadoop-trunk-Commit #3972 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3972/]) HADOOP-9624. TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name. Contributed by Xi Fang. (Revision 1494363) Result = SUCCESS cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1494363 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.2.patch, HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9652) FileContext#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HADOOP-9652: --- Assignee: Andrew Wang FileContext#getFileLinkStatus does not fill in the link owner and mode -- Key: HADOOP-9652 URL: https://issues.apache.org/jira/browse/HADOOP-9652 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Andrew Wang {{FileContext#getFileLinkStatus}} does not actually get the owner and mode of the symlink, but instead uses the owner and mode of the symlink target. If the target can't be found, it fills in bogus values (the empty string and FsPermission.getDefault) for these. Symlinks have an owner distinct from who created them, and getFileLinkStatus ought to expose this. In some operating systems, symlinks can have a permission other than 0777. We ought to expose this in RawLocalFilesystem and other places, although we don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687375#comment-13687375 ] Arun C Murthy commented on HADOOP-9421: --- Is there any reason to not commit this and possibly fix performance issues later? Will it change the protocols incompatibly to improve perf? Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9624: -- Resolution: Fixed Fix Version/s: 1.3.0 2.1.0-beta 1-win 3.0.0 Status: Resolved (was: Patch Available) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Fix For: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Attachments: HADOOP-9624.branch-1.2.patch, HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687378#comment-13687378 ] Chris Nauroth commented on HADOOP-9624: --- I committed the trunk patch to trunk, branch-2, and branch-2.1-beta. I committed the branch-1 patch to branch-1 and branch-1-win. Xi, thank you for contributing this test fix! TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Attachments: HADOOP-9624.branch-1.2.patch, HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[ https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687389#comment-13687389 ] Xi Fang commented on HADOOP-9624: - Chris, thanks for your review and comments. TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name - Key: HADOOP-9624 URL: https://issues.apache.org/jira/browse/HADOOP-9624 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Labels: test Fix For: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 Attachments: HADOOP-9624.branch-1.2.patch, HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest. PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and X in its name. {code} final private static PathFilter TEST_X_FILTER = new PathFilter() { public boolean accept(Path file) { if(file.getName().contains(x) || file.toString().contains(X)) return true; else return false; {code} Some of the test cases construct a path by combining path TEST_ROOT_DIR with a customized partial path. The problem is that TEST_ROOT_DIR may also has X in its name. The path check will pass even if the customized partial path doesn't have X. However, for this case the path filter is supposed to reject this path. An easy fix is to change file.toString().contains(X) to file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName() only returns the final component of this path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8040) Add symlink support to FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687390#comment-13687390 ] Colin Patrick McCabe commented on HADOOP-8040: -- bq. I refactored these out of FileContext and decided to stick them in Path. Seems kind of weird to have a check instance method that throws an exception like this, so I'd mildly prefer to leave them static. I'll leave it up to you, but I really prefer instance methods in this case. bq. Semantics for all the local filesystems is kinda fuzzy, but I agree that this feels incorrect. I'd prefer to fix both of these classes at once though in a follow-on JIRA (especially if JNI is potentially involved). OK. I filed HADOOP-9652 for this. bq. Hmm, interesting comment, good eye. The issue here is that since symlinks can point to other filesystems, we have to return a FileStatus, not an HdfsFileStatus. Plain old FileStatus doesn't have fileId, so we have to leave it out. I think this is okay from a user perspective, since FileSystem methods only return FileStatus, and HdfsFileStatus isn't a subclass of FileStatus anyway. Yeah, given that it has to be copied anyway, this is probably fine. Can you add a comment? bq. I think this is as intended. The new inner class sometimes needs to wrap final parameters of the containing method. Since the params are different each time (and different per call), I think it needs to do this at runtime. OK. bq. I think the intent here was to just pick a reasonable upper bound. I doubt any real user has 32 links to links, and I don't think there's a reason to tune it down either. Looks like there was some discussion on this earlier: https://issues.apache.org/jira/browse/HDFS-245?focusedCommentId=12642970page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12642970 32 is probably fine for now. If we ever need it to be configurable, we can do that in a separate JIRA. Add symlink support to FileSystem - Key: HADOOP-8040 URL: https://issues.apache.org/jira/browse/HADOOP-8040 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha Reporter: Eli Collins Assignee: Andrew Wang Attachments: hadoop-8040-1.patch, hadoop-8040-2.patch, hadoop-8040-3.patch, hadoop-8040-4.patch, hadoop-8040-5.patch, hadoop-8040-6.patch HADOOP-6421 added symbolic links to FileContext. Resolving symlinks is done on the client-side, and therefore requires client support. An HDFS symlink (created by FileContext) when accessed by FileSystem will result in an unhandled UnresolvedLinkException. Because not all users will migrate from FileSystem to FileContext in lock step, and we want users of FileSystem to be able to access all paths created by FileContext, we need to support symlink resolution in FileSystem as well, to facilitate migration to FileContext. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687398#comment-13687398 ] Todd Lipcon commented on HADOOP-9439: - bq. I don't see the point of putting all variable declarations on separate lines. Java allows declaration of variables on the same line as well The point is that when a lot of people see {{int a, b, c = 0;}} they think that all three variables are getting set to 0, but in fact it's only {{c}} that is set. Putting on separate lines makes it more obvious -- probably not important if you expect mostly veteran C programmers to read the code, but given that a lot of folks in the Hadoop community rarely look at C, I figured it would be more obvious this way. In Java, you always get a compile time error for accessing an uninitialized variable, whereas in C it's just a warning, which I'm certain we'll miss in the compile spew of a Hadoop build. So, probably better to explicitly initialize all the variables, whether on one line or multiple. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Attachment: HADOOP-9631.trunk.3.patch Another try at this patch. Problem I was facing was passing down getServerDefaults() call from InternalViewOfDir class. So, in ViewFs::getServerDeaults() did a match of mountpoints and if there exists a matching mountpoint return the filesystem, else return LocalConfig as earlier. This was needed so that we can handle case of FileNotFound, invalid dir/file creation and such. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Patch Available (was: Open) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9586) unit test failure: org.apache.hadoop.hdfs.TestFileCreation.testFileCreationSetLocalInterface
[ https://issues.apache.org/jira/browse/HADOOP-9586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687404#comment-13687404 ] Leo Leung commented on HADOOP-9586: --- Could be related to DFSClient::getRandomLocalInterfaceAddr() picking up an IPv6 IP for interface lo on a dual stack system. Changes seems to be from [HDFS-3148] h3. The following fails: 2013-06-18 16:34:30,141 DEBUG hdfs.DFSClient (DFSClient.java:init(372)) - Using local interfaces [lo] with addresses {color:red}[/0:0:0:0:0:0:0:1%1:0,/127.0.0.1:0]{color} [...] 2013-06-18 16:34:30,289 DEBUG hdfs.DFSClient (DFSClient.java:createBlockOutputStream(3852)) - Connecting to 127.0.0.1:51117 2013-06-18 16:34:30,289 DEBUG hdfs.DFSClient (DFSClient.java:getRandomLocalInterfaceAddr(1204)) - Using local interface {color:red}*/0:0:0:0:0:0:0:1%1:0*{color} 2013-06-18 16:34:30,289 INFO hdfs.DFSClient (DFSClient.java:createBlockOutputStream(3904)) - Exception in createBlockOutputStream 127.0.0.1:51117 java.net.SocketException: Network is unreachable 2013-06-18 16:34:30,290 INFO hdfs.DFSClient (DFSClient.java:nextBlockOutputStream(3810)) - Abandoning blk_7035064663379527570_1001 2013-06-18 16:34:30,292 INFO hdfs.DFSClient (DFSClient.java:nextBlockOutputStream(3814)) - Excluding datanode 127.0.0.1:51117 2013-06-18 16:34:30,294 WARN namenode.FSNamesystem (BlockPlacementPolicyDefault.java:chooseTarget(209)) - Not able to place enough replicas, still in need of 1 to reach 1 Not able to place enough replicas 2013-06-18 16:34:30,294 ERROR security.UserGroupInformation (UserGroupInformation.java:doAs(1235)) - PriviledgedActionException as:ddn cause:java.io.IOException: File /user/ddn/filestatus.dat could only be replicated to 0 nodes, instead of 1 2013-06-18 16:34:30,294 INFO ipc.Server (Server.java:run(1463)) - IPC Server handler 8 on 41922, call addBlock(/user/ddn/filestatus.dat, DFSClient_NONMAPREDUCE_-97075681_1, [Lorg.apache.hadoop.hdfs.protocol.DatanodeInfo;@b04bce3) from 127.0.0.1:48545: error: java.io.IOException: File /user/ddn/filestatus.dat could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /user/ddn/filestatus.dat could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1983) h3. The following works: [root@host] echo 1 /proc/sys/net/ipv6/conf/all/disable_ipv6 2013-06-18 16:52:41,890 DEBUG hdfs.DFSClient (DFSClient.java:init(372)) - Using local interfaces [lo] with addresses [/127.0.0.1:0] 2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient (DFSClient.java:createBlockOutputStream(3852)) - Connecting to 127.0.0.1:45078 2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient (DFSClient.java:getRandomLocalInterfaceAddr(1204)) - Using local interface /127.0.0.1:0 2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient (DFSClient.java:createBlockOutputStream(3856)) - Send buf size 131072 2013-06-18 16:52:42,030 INFO datanode.DataNode (DataXceiver.java:writeBlock(249)) - Receiving blk_4193380550170530321_1001 src: /127.0.0.1:60010 dest: /127.0.0.1:45078 2013-06-18 16:52:42,032 DEBUG hdfs.DFSClient (DFSClient.java:run(3277)) - DataStreamer block blk_4193380550170530321_1001 wrote packet seqno:0 size:8281 offsetInBlock:0 lastPacketInBlock:true h3. Summary/Proposal: Need a fix for DFSClient to understand and support (or not) IPv6 unit test failure: org.apache.hadoop.hdfs.TestFileCreation.testFileCreationSetLocalInterface Key: HADOOP-9586 URL: https://issues.apache.org/jira/browse/HADOOP-9586 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 1.3.0 Reporter: Giridharan Kesavan https://builds.apache.org/job/Hadoop-branch1/lastCompletedBuild/testReport/org.apache.hadoop.hdfs/TestFileCreation/testFileCreationSetLocalInterface/ org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/jenkins/filestatus.dat could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at
[jira] [Updated] (HADOOP-9652) FileContext#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9652: - Description: {{FileContext#getFileLinkStatus}} does not actually get the owner and mode of the symlink, but instead uses the owner and mode of the symlink target. If the target can't be found, it fills in bogus values (the empty string and FsPermission.getDefault) for these. Symlinks have an owner distinct from the owner of the target they point to, and getFileLinkStatus ought to expose this. In some operating systems, symlinks can have a permission other than 0777. We ought to expose this in RawLocalFilesystem and other places, although we don't necessarily have to support this behavior in HDFS. was: {{FileContext#getFileLinkStatus}} does not actually get the owner and mode of the symlink, but instead uses the owner and mode of the symlink target. If the target can't be found, it fills in bogus values (the empty string and FsPermission.getDefault) for these. Symlinks have an owner distinct from who created them, and getFileLinkStatus ought to expose this. In some operating systems, symlinks can have a permission other than 0777. We ought to expose this in RawLocalFilesystem and other places, although we don't necessarily have to support this behavior in HDFS. FileContext#getFileLinkStatus does not fill in the link owner and mode -- Key: HADOOP-9652 URL: https://issues.apache.org/jira/browse/HADOOP-9652 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Andrew Wang {{FileContext#getFileLinkStatus}} does not actually get the owner and mode of the symlink, but instead uses the owner and mode of the symlink target. If the target can't be found, it fills in bogus values (the empty string and FsPermission.getDefault) for these. Symlinks have an owner distinct from the owner of the target they point to, and getFileLinkStatus ought to expose this. In some operating systems, symlinks can have a permission other than 0777. We ought to expose this in RawLocalFilesystem and other places, although we don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8040) Add symlink support to FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687448#comment-13687448 ] Hadoop QA commented on HADOOP-8040: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588466/hadoop-8040-6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 13 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2668//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2668//console This message is automatically generated. Add symlink support to FileSystem - Key: HADOOP-8040 URL: https://issues.apache.org/jira/browse/HADOOP-8040 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha Reporter: Eli Collins Assignee: Andrew Wang Attachments: hadoop-8040-1.patch, hadoop-8040-2.patch, hadoop-8040-3.patch, hadoop-8040-4.patch, hadoop-8040-5.patch, hadoop-8040-6.patch HADOOP-6421 added symbolic links to FileContext. Resolving symlinks is done on the client-side, and therefore requires client support. An HDFS symlink (created by FileContext) when accessed by FileSystem will result in an unhandled UnresolvedLinkException. Because not all users will migrate from FileSystem to FileContext in lock step, and we want users of FileSystem to be able to access all paths created by FileContext, we need to support symlink resolution in FileSystem as well, to facilitate migration to FileContext. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9439: - Attachment: HADOOP-9439.006.patch * declare pw_lock_object as extern * put declaration of pw_lock_locked on a separate line JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687524#comment-13687524 ] Hadoop QA commented on HADOOP-9439: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588505/HADOOP-9439.006.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2671//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2671//console This message is automatically generated. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687530#comment-13687530 ] Hadoop QA commented on HADOOP-9631: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588487/HADOOP-9631.trunk.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2670//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2670//console This message is automatically generated. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6837) Support for LZMA compression
[ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687627#comment-13687627 ] Harsh J commented on HADOOP-6837: - FB's hadoop-0.20 seems to have a working implementation of this, although I do not know results of its stability yet: https://github.com/facebook/hadoop-20/blob/master/src/core/org/apache/hadoop/io/compress/LzmaCodec.java (and others). Support for LZMA compression Key: HADOOP-6837 URL: https://issues.apache.org/jira/browse/HADOOP-6837 Project: Hadoop Common Issue Type: Improvement Components: io Reporter: Nicholas Carlini Assignee: Nicholas Carlini Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687643#comment-13687643 ] Sanjay Radia commented on HADOOP-9421: -- bq. Client today does send/(send/read)+, and now it's send/read/(send/read)+. Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira