[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686432#comment-13686432
]
Hadoop QA commented on HADOOP-9637:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588295/HADOOP-9637-trunk.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2663//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2663//console
This message is automatically generated.
Adding Native Fstat for Windows as needed by YARN
-
Key: HADOOP-9637
URL: https://issues.apache.org/jira/browse/HADOOP-9637
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.patch
In the YARN, nodemanager need to enforce the log file can only be accessed by
the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
{{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
Windows. The YARN unit tests
TestAggregatedLogFormat.testContainerLogsFileAccess and
TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
this.
The JIRA try to provide a Windows implementation of
{{NativeIO.Posix.getFstat()}}.
TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
Windows. The test case try to simulate a situation where first log file is
owned by different user (probably symlink) and second one by the user itself.
In this situation, the attempt to try to aggregate the logs should fail with
the error message Owner ... for path ... did not match expected owner
The check on file owner happens at {{AggregatedLogFormat.write()}} method.
The method calls {{SecureIOUtils.openForRead()}} to read the log files before
writing out to the OutputStream.
{{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9515) Add general interface for NFS and Mount
[ https://issues.apache.org/jira/browse/HADOOP-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686579#comment-13686579 ] Hudson commented on HADOOP-9515: Integrated in Hadoop-Yarn-trunk #244 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/244/]) CHANGES.txt change for HADOOP-9515 (Revision 1493929) HADOOP-9515. Add general interface for NFS and Mount. Contributed by Brandon Li (Revision 1493924) Result = SUCCESS brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493929 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493924 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountEntry.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountInterface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsFileType.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsTime.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3 * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/FileHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3FileAttributes.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Interface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Status.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/ACCESS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/COMMIT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/CREATE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSINFO3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSSTAT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/GETATTR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/LOOKUP3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/MKDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/PATHCONF3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READ3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIRPLUS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READLINK3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/REMOVE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RENAME3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RMDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RequestWithHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/SETATTR3Request.java *
[jira] [Commented] (HADOOP-9515) Add general interface for NFS and Mount
[ https://issues.apache.org/jira/browse/HADOOP-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686675#comment-13686675 ] Hudson commented on HADOOP-9515: Integrated in Hadoop-Hdfs-trunk #1434 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1434/]) CHANGES.txt change for HADOOP-9515 (Revision 1493929) HADOOP-9515. Add general interface for NFS and Mount. Contributed by Brandon Li (Revision 1493924) Result = FAILURE brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493929 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493924 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountEntry.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountInterface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsFileType.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsTime.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3 * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/FileHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3FileAttributes.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Interface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Status.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/ACCESS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/COMMIT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/CREATE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSINFO3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSSTAT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/GETATTR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/LOOKUP3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/MKDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/PATHCONF3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READ3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIRPLUS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READLINK3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/REMOVE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RENAME3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RMDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RequestWithHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/SETATTR3Request.java *
[jira] [Commented] (HADOOP-9515) Add general interface for NFS and Mount
[ https://issues.apache.org/jira/browse/HADOOP-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686746#comment-13686746 ] Hudson commented on HADOOP-9515: Integrated in Hadoop-Mapreduce-trunk #1461 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1461/]) CHANGES.txt change for HADOOP-9515 (Revision 1493929) HADOOP-9515. Add general interface for NFS and Mount. Contributed by Brandon Li (Revision 1493924) Result = SUCCESS brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493929 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493924 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountEntry.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountInterface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsFileType.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsTime.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3 * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/FileHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3FileAttributes.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Interface.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Status.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/ACCESS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/COMMIT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/CREATE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSINFO3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/FSSTAT3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/GETATTR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/LOOKUP3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/MKDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/PATHCONF3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READ3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READDIRPLUS3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/READLINK3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/REMOVE3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RENAME3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RMDIR3Request.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/RequestWithHandle.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/SETATTR3Request.java *
[jira] [Created] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
Steve Loughran created HADOOP-9651:
--
Summary: Filesystems to throw FileAlreadyExistsException in
createFile(path, overwrite=false) when the file exists
Key: HADOOP-9651
URL: https://issues.apache.org/jira/browse/HADOOP-9651
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if you
try to create a file that exists and you have set {{overwrite=false}},
{{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it impossible
to distinguish a create operation failing from a fixable problem (the file is
there) and something more fundamental.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[
https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686763#comment-13686763
]
Steve Loughran commented on HADOOP-9651:
Here I propose
# saying MUST throw {{FileAlreadyExistsException}}
# fixing all the hadoop filesystems to do this where they do not today.
# making sure the FS tests check for this.
HDFS already has the correct behaviour, as do the blobstore(s).
Filesystems to throw FileAlreadyExistsException in createFile(path,
overwrite=false) when the file exists
-
Key: HADOOP-9651
URL: https://issues.apache.org/jira/browse/HADOOP-9651
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if
you try to create a file that exists and you have set {{overwrite=false}},
{{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it
impossible to distinguish a create operation failing from a fixable problem
(the file is there) and something more fundamental.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[
https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686765#comment-13686765
]
Steve Loughran commented on HADOOP-9651:
stack
{code}
ava.io.IOException: File already exists: file:/Users/stevel/tmp/buffer.json
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:276)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:269)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.init(ChecksumFileSystem.java:382)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:430)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:411)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:888)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:869)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:768)
{code}
Filesystems to throw FileAlreadyExistsException in createFile(path,
overwrite=false) when the file exists
-
Key: HADOOP-9651
URL: https://issues.apache.org/jira/browse/HADOOP-9651
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if
you try to create a file that exists and you have set {{overwrite=false}},
{{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it
impossible to distinguish a create operation failing from a fixable problem
(the file is there) and something more fundamental.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686983#comment-13686983
]
Chris Nauroth commented on HADOOP-9637:
---
Thanks for addressing my questions. It all makes sense. I have just one more
comment about the new patch.
{code}
if (Shell.WINDOWS) {
int i = owner.indexOf('\\');
if (i != -1)
this.owner = owner.substring(i + 1);
}
{code}
This code is identical for username and group, so let's refactor it behind a
helper function. We also can add comments stating that the reason for this is
to remove domain.
Adding Native Fstat for Windows as needed by YARN
-
Key: HADOOP-9637
URL: https://issues.apache.org/jira/browse/HADOOP-9637
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.patch
In the YARN, nodemanager need to enforce the log file can only be accessed by
the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
{{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
Windows. The YARN unit tests
TestAggregatedLogFormat.testContainerLogsFileAccess and
TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
this.
The JIRA try to provide a Windows implementation of
{{NativeIO.Posix.getFstat()}}.
TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
Windows. The test case try to simulate a situation where first log file is
owned by different user (probably symlink) and second one by the user itself.
In this situation, the attempt to try to aggregate the logs should fail with
the error message Owner ... for path ... did not match expected owner
The check on file owner happens at {{AggregatedLogFormat.write()}} method.
The method calls {{SecureIOUtils.openForRead()}} to read the log files before
writing out to the OutputStream.
{{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687022#comment-13687022 ] Jason Lowe commented on HADOOP-9582: bq. I did do this in one my previous patch,however Steve(on 22/May/13 06:01) recommended that raising AssertionFailedError would be better for diagnostics. Ah, yes, that makes sense for better diagnostics. I'm +1 on the latest patch and will commit later today. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582-4.txt, HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[
https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Baptiste Onofré updated HADOOP-9651:
-
Attachment: HADOOP-9651.patch
Patch for the RawLocalFileSystem (including unit tests)
Filesystems to throw FileAlreadyExistsException in createFile(path,
overwrite=false) when the file exists
-
Key: HADOOP-9651
URL: https://issues.apache.org/jira/browse/HADOOP-9651
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
Attachments: HADOOP-9651.patch
While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if
you try to create a file that exists and you have set {{overwrite=false}},
{{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it
impossible to distinguish a create operation failing from a fixable problem
(the file is there) and something more fundamental.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[
https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Baptiste Onofré updated HADOOP-9651:
-
Status: Patch Available (was: Open)
This patch updates the RawLocalFileSystem to throw FileAlreadyExistsException.
Filesystems to throw FileAlreadyExistsException in createFile(path,
overwrite=false) when the file exists
-
Key: HADOOP-9651
URL: https://issues.apache.org/jira/browse/HADOOP-9651
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
Attachments: HADOOP-9651.patch
While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if
you try to create a file that exists and you have set {{overwrite=false}},
{{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it
impossible to distinguish a create operation failing from a fixable problem
(the file is there) and something more fundamental.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[
https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687119#comment-13687119
]
Jean-Baptiste Onofré commented on HADOOP-9651:
--
I'm reviewing the other filesystems to see if some updates are required to
correctly throw FileAlreadyExistsException.
Filesystems to throw FileAlreadyExistsException in createFile(path,
overwrite=false) when the file exists
-
Key: HADOOP-9651
URL: https://issues.apache.org/jira/browse/HADOOP-9651
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
Attachments: HADOOP-9651.patch
While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if
you try to create a file that exists and you have set {{overwrite=false}},
{{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it
impossible to distinguish a create operation failing from a fixable problem
(the file is there) and something more fundamental.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8040) Add symlink support to FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687131#comment-13687131
]
Colin Patrick McCabe commented on HADOOP-8040:
--
{code}
protected static void checkNotSchemeWithRelative(final Path path)
protected static void checkNotRelative(final Path path)
public static String getPathWithoutSchemeAndAuthority(Path path)
{code}
Why are these static methods rather than instance methods? They seem to just
operate on a single path, and don't do anything smart when you pass null, so I
can't see why you'd want them to be static. I also don't see why you would
want {{protected}} here, rather than package-private (i.e. no qualifier.).
Also, a function whose name begins with getPath... should return a path. Let
the caller call toString if he wants to.
{code}
System.out.println(RLFS wd: + workingDir);
{code}
Debug printout left in.
{code}
private FileStatus getFileLinkStatusInternal(final Path f) throws IOException
{
String target = readLink(f);
try {
FileStatus fs = getFileStatus(f);
// If f refers to a regular file or directory
if (target.isEmpty()) {
return fs;
}
// Otherwise f refers to a symlink
return new FileStatus(fs.getLen(),
false,
fs.getReplication(),
fs.getBlockSize(),
fs.getModificationTime(),
fs.getAccessTime(),
fs.getPermission(),
fs.getOwner(),
fs.getGroup(),
new Path(target),
f);
{code}
Symlinks do not necessarily have the same owner as the file they point to.
Here's any example:
{code}
cmccabe@keter:~/hadoop2 touch a
cmccabe@keter:~/hadoop2 ln -s a b
cmccabe@keter:~/hadoop2 sudo chown bob b
root's password:
cmccabe@keter:~/hadoop2 ls -l a b
-rw-r--r-- 1 bob users 0 Jun 18 12:14 a
lrwxrwxrwx 1 cmccabe users 1 Jun 18 12:14 b - a
{code}
Symlinks have a file mode (aka permission bits) which are independent of the
files they point to. On certain UNIXes (but not Linux), symlinks can have
permissions other than 0777. I'm not sure if you can get access to lstat from
JDK6, or if you'll need JNI for this.
{code}
final public FileStatus makeQualified(URI defaultUri, Path path) {
{code}
This is basically a copy constructor that also modifies
{{HdfsFileStatus#path}}. Are you sure it wouldn't be easier to just have a
method to change the path of the object? As new fields are added to
HdfsFileStatus, it's going to be very, very easy for people to forget to change
this function to copy them. In fact, you seem to have left a field out here
yourself-- fileId. This is especially an issue when subclasses start coming
into play. All the problems that plague clone() plague this method too. I
also feel like having a mutator like setParentPath(URI defaultURI, Path parent)
would be more efficient, as well as more maintainable.
FSLinkResolver: it seems like you only need one of these per function. Since
they are stateless, there isn't any point in creating garbage that you'll have
to collect later. Just make a static FSLinkResolverContentSummary and use
that in getContentSummary. etc. etc. Also, should MAX_PATH_LINKS be
configurable?
Add symlink support to FileSystem
-
Key: HADOOP-8040
URL: https://issues.apache.org/jira/browse/HADOOP-8040
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha
Reporter: Eli Collins
Assignee: Andrew Wang
Attachments: hadoop-8040-1.patch, hadoop-8040-2.patch,
hadoop-8040-3.patch, hadoop-8040-4.patch, hadoop-8040-5.patch
HADOOP-6421 added symbolic links to FileContext. Resolving symlinks is done
on the client-side, and therefore requires client support. An HDFS symlink
(created by FileContext) when accessed by FileSystem will result in an
unhandled UnresolvedLinkException. Because not all users will migrate from
FileSystem to FileContext in lock step, and we want users of FileSystem to be
able to access all paths created by FileContext, we need to support symlink
resolution in FileSystem as well, to facilitate migration to FileContext.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HADOOP-9421: -- Priority: Blocker (was: Major) Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9648) Fix build native library on mac osx
[
https://issues.apache.org/jira/browse/HADOOP-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687136#comment-13687136
]
Arpit Agarwal commented on HADOOP-9648:
---
Thanks for taking this up Kirill. I verified that the native library builds on
OSX with your changes.
Few comments from the 2.0.5 patch.
# Tests are not able to load the native library. TestNativeIO fails with
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable.
# The working directory should be restored before returning to avoid changing
the callers cwd. I think it would be best to use the mkdirat and openat
functions except on OS X via conditional compilation.
{code}
-if (mkdirat(cwd, token, perm) != 0) {
+if(fchdir(cwd) 0) {
{code}
# The iterative approach to replace fcloseall looks somewhat inefficient. At
the very least it should be restricted to OS X and fcloseall() used everywhere
else.
{code}
+ for (; rl.rlim_max 2; rl.rlim_max--) {
+close(rl.rlim_max);
+ }
{code}
# Is this change necessary? I can build without it.
{code}
-version2.4.0a/version
+version2.5.0/version
{code}
Fix build native library on mac osx
---
Key: HADOOP-9648
URL: https://issues.apache.org/jira/browse/HADOOP-9648
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1.0.4, 1.2.0, 1.1.2, 2.0.5-alpha
Reporter: Kirill A. Korinskiy
Attachments: HADOOP-9648-native-osx.1.0.4.patch,
HADOOP-9648-native-osx.1.1.2.patch, HADOOP-9648-native-osx.1.2.0.patch,
HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch
Some patches for fixing build a hadoop native library on os x 10.7/10.8.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chuan Liu updated HADOOP-9637:
--
Attachment: HADOOP-9637-trunk.3.patch
Attaching a new patch.
Adding Native Fstat for Windows as needed by YARN
-
Key: HADOOP-9637
URL: https://issues.apache.org/jira/browse/HADOOP-9637
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
HADOOP-9637-trunk.patch
In the YARN, nodemanager need to enforce the log file can only be accessed by
the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
{{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
Windows. The YARN unit tests
TestAggregatedLogFormat.testContainerLogsFileAccess and
TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
this.
The JIRA try to provide a Windows implementation of
{{NativeIO.Posix.getFstat()}}.
TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
Windows. The test case try to simulate a situation where first log file is
owned by different user (probably symlink) and second one by the user itself.
In this situation, the attempt to try to aggregate the logs should fail with
the error message Owner ... for path ... did not match expected owner
The check on file owner happens at {{AggregatedLogFormat.write()}} method.
The method calls {{SecureIOUtils.openForRead()}} to read the log files before
writing out to the OutputStream.
{{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Fang updated HADOOP-9624:
Attachment: HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root
path has X in its name
---
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.patch, HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even
if the customized partial path doesn't have X. However, for this case the
path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9651) Filesystems to throw FileAlreadyExistsException in createFile(path, overwrite=false) when the file exists
[
https://issues.apache.org/jira/browse/HADOOP-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687153#comment-13687153
]
Hadoop QA commented on HADOOP-9651:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588436/HADOOP-9651.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2664//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2664//console
This message is automatically generated.
Filesystems to throw FileAlreadyExistsException in createFile(path,
overwrite=false) when the file exists
-
Key: HADOOP-9651
URL: https://issues.apache.org/jira/browse/HADOOP-9651
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
Attachments: HADOOP-9651.patch
While HDFS and other filesystems throw a {{FileAlreadyExistsException}} if
you try to create a file that exists and you have set {{overwrite=false}},
{{RawLocalFileSystem}} throws a plain {{IOException}}. This makes it
impossible to distinguish a create operation failing from a fixable problem
(the file is there) and something more fundamental.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Fang updated HADOOP-9624:
Attachment: HADOOP-9624.branch-1.patch
TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root
path has X in its name
---
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch,
HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even
if the customized partial path doesn't have X. However, for this case the
path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9622) bzip2 codec can drop records when reading data in splits
[ https://issues.apache.org/jira/browse/HADOOP-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687158#comment-13687158 ] Nathan Roberts commented on HADOOP-9622: Thanks for adding the detailed comment. Well explained! +1 for the patch. bzip2 codec can drop records when reading data in splits Key: HADOOP-9622 URL: https://issues.apache.org/jira/browse/HADOOP-9622 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.4-alpha, 0.23.8 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: blockEndingInCRThenLF.txt.bz2, blockEndingInCR.txt.bz2, HADOOP-9622-2.patch, HADOOP-9622.patch, HADOOP-9622-testcase.patch Bzip2Codec.BZip2CompressionInputStream can cause records to be dropped when reading them in splits based on where record delimiters occur relative to compression block boundaries. Thanks to [~knoguchi] for discovering this problem while working on PIG-3251. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Fang updated HADOOP-9624:
Attachment: HAOOP-9624.branch-1.patch
TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root
path has X in its name
---
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch,
HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even
if the customized partial path doesn't have X. However, for this case the
path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Fang updated HADOOP-9624:
Attachment: (was: HAOOP-9624.branch-1.patch)
TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root
path has X in its name
---
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch,
HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even
if the customized partial path doesn't have X. However, for this case the
path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687160#comment-13687160
]
Hadoop QA commented on HADOOP-9624:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588447/HADOOP-9624.branch-1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2667//console
This message is automatically generated.
TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root
path has X in its name
---
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch,
HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even
if the customized partial path doesn't have X. However, for this case the
path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687162#comment-13687162
]
Xi Fang commented on HADOOP-9624:
-
Thanks Chris. Two patches have been attached.
TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root
path has X in its name
---
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch,
HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even
if the customized partial path doesn't have X. However, for this case the
path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687168#comment-13687168
]
Hadoop QA commented on HADOOP-9637:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588442/HADOOP-9637-trunk.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2665//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2665//console
This message is automatically generated.
Adding Native Fstat for Windows as needed by YARN
-
Key: HADOOP-9637
URL: https://issues.apache.org/jira/browse/HADOOP-9637
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
HADOOP-9637-trunk.patch
In the YARN, nodemanager need to enforce the log file can only be accessed by
the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
{{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
Windows. The YARN unit tests
TestAggregatedLogFormat.testContainerLogsFileAccess and
TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
this.
The JIRA try to provide a Windows implementation of
{{NativeIO.Posix.getFstat()}}.
TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
Windows. The test case try to simulate a situation where first log file is
owned by different user (probably symlink) and second one by the user itself.
In this situation, the attempt to try to aggregate the logs should fail with
the error message Owner ... for path ... did not match expected owner
The check on file owner happens at {{AggregatedLogFormat.write()}} method.
The method calls {{SecureIOUtils.openForRead()}} to read the log files before
writing out to the OutputStream.
{{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687184#comment-13687184
]
Hadoop QA commented on HADOOP-9624:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588443/HADOOP-9624.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2666//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2666//console
This message is automatically generated.
TestFSMainOperationsLocalFileSystem failed when the Hadoop enlistment root
path has X in its name
---
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch,
HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even
if the customized partial path doesn't have X. However, for this case the
path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687233#comment-13687233 ] Daryn Sharp commented on HADOOP-9421: - Per Sanjay's request, here's a quick summary: This patch does not direct address deficiencies in negotiation process. It merely provides the flexibility to fix them in the near future. The basic changes are: * SASL protobufs * SASL protobufs wrapped with RPC header * Server advertised SASL auth methods - currently ignored by client Wrapping SASL negotiation in a RPC header avoids decoding errors. Ex. A secure client currently tries to decode a RPC exception from an incompatible connection header as a SASL message. The client also can't handle a RPC exception mid-stream during SASL negotiation. The RPC header allows the server and client to correctly decode the payload. Using the RPC headers, with the addition of a streamId, will also allow the _future potential_ to multiplex multiple UGIs over the same connection. Services like the NN can be overwhelmed by a stampede of connections. The NM may be an ideal candidate for aggregation of connections, or even a per-node multiplexor for tasks. Server advertisement of auth methods aims to address the current limitation of allowing 1 pre-determined auth or simple. Additional consideration is needed for a client-side whitelist to avoid server unexpectedly requesting weak authentication. In the current patch, the client ignores the server and blindly attempts the negotiation as it does today. +Other future features unlocked by this design+ * Brings us much closer to pluggable auth methods w/o changing server and client code * Client may select a server advertised auth method to: *# Support services and/or clusters with heterogenous auth methods (ex. thinking of knox, rhino, etc) *# *Critical to Y!*: IP failover, especially for HA with kerberos. The client needs the active NN's host to acquire a service ticket *# Simplify token selection by using an opaque identifier supplied by the server - will eliminate use_ip, and even the complex HA token logic *# Support accessing multi-interface hosts on all interfaces *# Support accessing services via any of their hostnames, ips, or CNAMEs * Ability to greatly reduce complexity of client/server auth code, and cleanly decouple SASL logic * As mentioned before, multiplexing of different UGIs over a shared connection Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687268#comment-13687268 ] Hudson commented on HADOOP-9582: Integrated in Hadoop-trunk-Commit #3970 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3970/]) HADOOP-9582. Non-existent file to hadoop fs -conf doesn't throw error. Contributed by Ashwin Shankar (Revision 1494331) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1494331 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShell.java Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582-4.txt, HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-9582: --- Resolution: Fixed Fix Version/s: 2.3.0 0.23.9 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Ashwin! I committed this to trunk, branch-2, and branch-0.23. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Assignee: Ashwin Shankar Fix For: 3.0.0, 0.23.9, 2.3.0 Attachments: HADOOP-9582-4-b23.txt, HADOOP-9582-4.txt, HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9637:
--
Hadoop Flags: Reviewed
+1 for the patch. I verified on Mac (without native libs), Linux (with native
libs), and Windows (with native libs). Chuan, thank you for incorporating all
of the feedback. I'll commit this.
Adding Native Fstat for Windows as needed by YARN
-
Key: HADOOP-9637
URL: https://issues.apache.org/jira/browse/HADOOP-9637
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
HADOOP-9637-trunk.patch
In the YARN, nodemanager need to enforce the log file can only be accessed by
the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
{{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
Windows. The YARN unit tests
TestAggregatedLogFormat.testContainerLogsFileAccess and
TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
this.
The JIRA try to provide a Windows implementation of
{{NativeIO.Posix.getFstat()}}.
TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
Windows. The test case try to simulate a situation where first log file is
owned by different user (probably symlink) and second one by the user itself.
In this situation, the attempt to try to aggregate the logs should fail with
the error message Owner ... for path ... did not match expected owner
The check on file owner happens at {{AggregatedLogFormat.write()}} method.
The method calls {{SecureIOUtils.openForRead()}} to read the log files before
writing out to the OutputStream.
{{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8040) Add symlink support to FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-8040:
Attachment: hadoop-8040-6.patch
Thanks for the review Colin. I'm attaching just a consolidated patch again
since the changes I made were minor; I wanted to follow up with some of your
review comments before posting the patch split.
bq. Why are these static methods rather than instance methods?
I refactored these out of FileContext and decided to stick them in Path. Seems
kind of weird to have a check instance method that throws an exception like
this, so I'd mildly prefer to leave them static.
bq. package-private (i.e. no qualifier.).
fixed
bq. a function whose name begins with getPath... should return a path.
fixed, this was also a refactor but good point
bq. Debug printout left in.
fixed
bq. Symlinks do not necessarily have the same owner as the file they point to.
Here's any example:
bq. Symlinks have a file mode (aka permission bits) which are independent of
the files they point to...
I copied this bit from the FileContext implementation, RawLocalFS. It just uses
the target's status which, as you correctly noted, can be different from the
link's status. Semantics for all the local filesystems is kinda fuzzy, but I
agree that this feels incorrect. I'd prefer to fix both of these classes at
once though in a follow-on JIRA (especially if JNI is potentially involved).
{code}
final public FileStatus makeQualified(URI defaultUri, Path path) {
{code}
bq. Are you sure it wouldn't be easier to just have a method to change the path
of the object?
Mutating feels wrong to me based on how makeQualified is used. The path is
often passed into a method which then tries to qualify it (e.g.
Hdfs#getFileStatus). Mutating in place means the caller gets back a qualified
path, which is probably not what they want.
We could either mutate and make copies of the original path in all these
places, or just leave it as it is.
bq. In fact, you seem to have left a field out here yourself-- fileId.
Hmm, interesting comment, good eye. The issue here is that since symlinks can
point to other filesystems, we have to return a FileStatus, not an
HdfsFileStatus. Plain old FileStatus doesn't have fileId, so we have to leave
it out. I think this is okay from a user perspective, since FileSystem methods
only return FileStatus, and HdfsFileStatus isn't a subclass of FileStatus
anyway.
bq. FSLinkResolver: it seems like you only need one of these per
functionJust make a static FSLinkResolverContentSummary and use that...
I think this is as intended. The new inner class sometimes needs to wrap final
parameters of the containing method. Since the params are different each time
(and different per call), I think it needs to do this at runtime.
bq. should MAX_PATH_LINKS be configurable?
I think the intent here was to just pick a reasonable upper bound. I doubt any
real user has 32 links to links, and I don't think there's a reason to tune it
down either.
Add symlink support to FileSystem
-
Key: HADOOP-8040
URL: https://issues.apache.org/jira/browse/HADOOP-8040
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha
Reporter: Eli Collins
Assignee: Andrew Wang
Attachments: hadoop-8040-1.patch, hadoop-8040-2.patch,
hadoop-8040-3.patch, hadoop-8040-4.patch, hadoop-8040-5.patch,
hadoop-8040-6.patch
HADOOP-6421 added symbolic links to FileContext. Resolving symlinks is done
on the client-side, and therefore requires client support. An HDFS symlink
(created by FileContext) when accessed by FileSystem will result in an
unhandled UnresolvedLinkException. Because not all users will migrate from
FileSystem to FileContext in lock step, and we want users of FileSystem to be
able to access all paths created by FileContext, we need to support symlink
resolution in FileSystem as well, to facilitate migration to FileContext.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687292#comment-13687292
]
Hudson commented on HADOOP-9637:
Integrated in Hadoop-trunk-Commit #3971 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3971/])
HADOOP-9637. Adding Native Fstat for Windows as needed by YARN. Contributed
by Chuan Liu. (Revision 1494341)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1494341
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chmod.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chown.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/ls.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java
Adding Native Fstat for Windows as needed by YARN
-
Key: HADOOP-9637
URL: https://issues.apache.org/jira/browse/HADOOP-9637
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
HADOOP-9637-trunk.patch
In the YARN, nodemanager need to enforce the log file can only be accessed by
the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
{{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
Windows. The YARN unit tests
TestAggregatedLogFormat.testContainerLogsFileAccess and
TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
this.
The JIRA try to provide a Windows implementation of
{{NativeIO.Posix.getFstat()}}.
TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
Windows. The test case try to simulate a situation where first log file is
owned by different user (probably symlink) and second one by the user itself.
In this situation, the attempt to try to aggregate the logs should fail with
the error message Owner ... for path ... did not match expected owner
The check on file owner happens at {{AggregatedLogFormat.write()}} method.
The method calls {{SecureIOUtils.openForRead()}} to read the log files before
writing out to the OutputStream.
{{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9637:
--
Resolution: Fixed
Fix Version/s: 2.1.0-beta
3.0.0
Target Version/s: 3.0.0, 2.1.0-beta
Status: Resolved (was: Patch Available)
I committed this to trunk, branch-2, and branch-2.1-beta. Chuan, thank you for
the contribution.
Adding Native Fstat for Windows as needed by YARN
-
Key: HADOOP-9637
URL: https://issues.apache.org/jira/browse/HADOOP-9637
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 3.0.0, 2.1.0-beta
Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
HADOOP-9637-trunk.patch
In the YARN, nodemanager need to enforce the log file can only be accessed by
the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
{{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
Windows. The YARN unit tests
TestAggregatedLogFormat.testContainerLogsFileAccess and
TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
this.
The JIRA try to provide a Windows implementation of
{{NativeIO.Posix.getFstat()}}.
TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
Windows. The test case try to simulate a situation where first log file is
owned by different user (probably symlink) and second one by the user itself.
In this situation, the attempt to try to aggregate the logs should fail with
the error message Owner ... for path ... did not match expected owner
The check on file owner happens at {{AggregatedLogFormat.write()}} method.
The method calls {{SecureIOUtils.openForRead()}} to read the log files before
writing out to the OutputStream.
{{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687295#comment-13687295
]
Todd Lipcon commented on HADOOP-9439:
-
- seems like {{pw_lock_object}} is declared in two .c files... one should
probably be marked extern
- The following code is smelly to me:
{code}
int i, ret, nvalid, pw_lock_locked = 0;
{code}
because only the last variable is actually initialized. I don't _think_ you end
up using any of the uninitialized variables, so it's not a true bug, but I
think it's worth separating them out into their own lines. Do we not allow C99
style declaration in the middle of a function in our JNI code? I've always
liked that better than the original C style of declaring all at the top.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687314#comment-13687314 ] Colin Patrick McCabe commented on HADOOP-9439: -- I don't like C99 style declarations because they tend to introduce bugs. When you have some cleanup label at the end of the function, if your goto passes over the variable declaration, the variable will be treated as undefined. So, for example, if you have: int i = 4; JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687317#comment-13687317
]
Colin Patrick McCabe commented on HADOOP-9439:
--
ok, these keyboard shortcuts defeated me again. I wish I could turn that off.
Anyway, if you have:
{code}
goto cleanup:
struct foo *foo= calloc(1, sizeof(*foo));
...
cleanup:
free(foo);
{code}
then you're freeing an uninitialized pointer. Not good.
I also don't like declarations in the middle because of a philosophical reason:
it tends to lead to excessively long functions, rather than encouraging
modularity.
As to the extern thing, yeah, I agree. It's rather odd that the linker does
that kind of symbol merging, but apparently it does.
I don't see the point of putting all variable declarations on separate lines.
Java allows declaration of variables on the same line as well. C++0x allows
things like vector to be initialized with a reasonable syntax, so you are
finally allowed to admit that this is a good thing, you C++ advocate, you. :)
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9624:
--
Description:
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and
X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR with
a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X. However,
for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
was:
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x and
X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR with
a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even if
the customized partial path doesn't have X. However, for this case the path
filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch,
HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X.
However, for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9624:
--
Summary: TestFSMainOperationsLocalFileSystem failed when the Hadoop test
root path has X in its name (was: TestFSMainOperationsLocalFileSystem failed
when the Hadoop enlistment root path has X in its name)
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.patch, HADOOP-9624.patch,
HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that once the enlistment root path has X in its name,
TEST_ROOT_DIR will also has X in its name. The path check will pass even
if the customized partial path doesn't have X. However, for this case the
path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9624:
--
Attachment: HADOOP-9624.branch-1.2.patch
I'm attaching a very minor update to the branch-1 patch, changing and to or
in the comment, again just for consistency between the 2 branches.
+1 for both patches. I'll commit these.
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.2.patch,
HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X.
However, for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9624:
--
Target Version/s: 3.0.0, 1-win, 2.1.0-beta, 1.3.0 (was: 1-win)
Affects Version/s: 1.3.0
2.1.0-beta
3.0.0
Hadoop Flags: Reviewed
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.2.patch,
HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X.
However, for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9652) FileContext#getFileLinkStatus does not fill in the link owner and mode
Colin Patrick McCabe created HADOOP-9652:
Summary: FileContext#getFileLinkStatus does not fill in the link
owner and mode
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
{{FileContext#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If the
target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from who created them, and getFileLinkStatus
ought to expose this.
In some operating systems, symlinks can have a permission other than 0777. We
ought to expose this in RawLocalFilesystem and other places, although we don't
necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687347#comment-13687347
]
Hadoop QA commented on HADOOP-9624:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588475/HADOOP-9624.branch-1.2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2669//console
This message is automatically generated.
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.2.patch,
HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X.
However, for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687359#comment-13687359
]
Hudson commented on HADOOP-9624:
Integrated in Hadoop-trunk-Commit #3972 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3972/])
HADOOP-9624. TestFSMainOperationsLocalFileSystem failed when the Hadoop
test root path has X in its name. Contributed by Xi Fang. (Revision 1494363)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1494363
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.2.patch,
HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X.
However, for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9652) FileContext#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reassigned HADOOP-9652:
---
Assignee: Andrew Wang
FileContext#getFileLinkStatus does not fill in the link owner and mode
--
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
{{FileContext#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from who created them, and getFileLinkStatus
ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687375#comment-13687375 ] Arun C Murthy commented on HADOOP-9421: --- Is there any reason to not commit this and possibly fix performance issues later? Will it change the protocols incompatibly to improve perf? Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9624:
--
Resolution: Fixed
Fix Version/s: 1.3.0
2.1.0-beta
1-win
3.0.0
Status: Resolved (was: Patch Available)
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Fix For: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Attachments: HADOOP-9624.branch-1.2.patch,
HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X.
However, for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687378#comment-13687378
]
Chris Nauroth commented on HADOOP-9624:
---
I committed the trunk patch to trunk, branch-2, and branch-2.1-beta. I
committed the branch-1 patch to branch-1 and branch-1-win. Xi, thank you for
contributing this test fix!
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Attachments: HADOOP-9624.branch-1.2.patch,
HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X.
However, for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9624) TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has X in its name
[
https://issues.apache.org/jira/browse/HADOOP-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687389#comment-13687389
]
Xi Fang commented on HADOOP-9624:
-
Chris, thanks for your review and comments.
TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has
X in its name
-
Key: HADOOP-9624
URL: https://issues.apache.org/jira/browse/HADOOP-9624
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Labels: test
Fix For: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Attachments: HADOOP-9624.branch-1.2.patch,
HADOOP-9624.branch-1.patch, HADOOP-9624.patch, HADOOP-9624.trunk.patch
TestFSMainOperationsLocalFileSystem extends Class FSMainOperationsBaseTest.
PathFilter FSMainOperationsBaseTest#TEST_X_FILTER checks if a path has x
and X in its name.
{code}
final private static PathFilter TEST_X_FILTER = new PathFilter() {
public boolean accept(Path file) {
if(file.getName().contains(x) || file.toString().contains(X))
return true;
else
return false;
{code}
Some of the test cases construct a path by combining path TEST_ROOT_DIR
with a customized partial path.
The problem is that TEST_ROOT_DIR may also has X in its name. The path
check will pass even if the customized partial path doesn't have X.
However, for this case the path filter is supposed to reject this path.
An easy fix is to change file.toString().contains(X) to
file.getName().contains(X). Note that org.apache.hadoop.fs.Path.getName()
only returns the final component of this path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8040) Add symlink support to FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687390#comment-13687390 ] Colin Patrick McCabe commented on HADOOP-8040: -- bq. I refactored these out of FileContext and decided to stick them in Path. Seems kind of weird to have a check instance method that throws an exception like this, so I'd mildly prefer to leave them static. I'll leave it up to you, but I really prefer instance methods in this case. bq. Semantics for all the local filesystems is kinda fuzzy, but I agree that this feels incorrect. I'd prefer to fix both of these classes at once though in a follow-on JIRA (especially if JNI is potentially involved). OK. I filed HADOOP-9652 for this. bq. Hmm, interesting comment, good eye. The issue here is that since symlinks can point to other filesystems, we have to return a FileStatus, not an HdfsFileStatus. Plain old FileStatus doesn't have fileId, so we have to leave it out. I think this is okay from a user perspective, since FileSystem methods only return FileStatus, and HdfsFileStatus isn't a subclass of FileStatus anyway. Yeah, given that it has to be copied anyway, this is probably fine. Can you add a comment? bq. I think this is as intended. The new inner class sometimes needs to wrap final parameters of the containing method. Since the params are different each time (and different per call), I think it needs to do this at runtime. OK. bq. I think the intent here was to just pick a reasonable upper bound. I doubt any real user has 32 links to links, and I don't think there's a reason to tune it down either. Looks like there was some discussion on this earlier: https://issues.apache.org/jira/browse/HDFS-245?focusedCommentId=12642970page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12642970 32 is probably fine for now. If we ever need it to be configurable, we can do that in a separate JIRA. Add symlink support to FileSystem - Key: HADOOP-8040 URL: https://issues.apache.org/jira/browse/HADOOP-8040 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha Reporter: Eli Collins Assignee: Andrew Wang Attachments: hadoop-8040-1.patch, hadoop-8040-2.patch, hadoop-8040-3.patch, hadoop-8040-4.patch, hadoop-8040-5.patch, hadoop-8040-6.patch HADOOP-6421 added symbolic links to FileContext. Resolving symlinks is done on the client-side, and therefore requires client support. An HDFS symlink (created by FileContext) when accessed by FileSystem will result in an unhandled UnresolvedLinkException. Because not all users will migrate from FileSystem to FileContext in lock step, and we want users of FileSystem to be able to access all paths created by FileContext, we need to support symlink resolution in FileSystem as well, to facilitate migration to FileContext. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687398#comment-13687398
]
Todd Lipcon commented on HADOOP-9439:
-
bq. I don't see the point of putting all variable declarations on separate
lines. Java allows declaration of variables on the same line as well
The point is that when a lot of people see {{int a, b, c = 0;}} they think that
all three variables are getting set to 0, but in fact it's only {{c}} that is
set. Putting on separate lines makes it more obvious -- probably not important
if you expect mostly veteran C programmers to read the code, but given that a
lot of folks in the Hadoop community rarely look at C, I figured it would be
more obvious this way.
In Java, you always get a compile time error for accessing an uninitialized
variable, whereas in C it's just a warning, which I'm certain we'll miss in the
compile spew of a Hadoop build. So, probably better to explicitly initialize
all the variables, whether on one line or multiple.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Attachment: HADOOP-9631.trunk.3.patch Another try at this patch. Problem I was facing was passing down getServerDefaults() call from InternalViewOfDir class. So, in ViewFs::getServerDeaults() did a match of mountpoints and if there exists a matching mountpoint return the filesystem, else return LocalConfig as earlier. This was needed so that we can handle case of FileNotFound, invalid dir/file creation and such. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Patch Available (was: Open) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9586) unit test failure: org.apache.hadoop.hdfs.TestFileCreation.testFileCreationSetLocalInterface
[
https://issues.apache.org/jira/browse/HADOOP-9586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687404#comment-13687404
]
Leo Leung commented on HADOOP-9586:
---
Could be related to DFSClient::getRandomLocalInterfaceAddr() picking up an IPv6
IP for interface lo on a dual stack system. Changes seems to be from
[HDFS-3148]
h3. The following fails:
2013-06-18 16:34:30,141 DEBUG hdfs.DFSClient (DFSClient.java:init(372)) -
Using local interfaces [lo] with addresses
{color:red}[/0:0:0:0:0:0:0:1%1:0,/127.0.0.1:0]{color}
[...]
2013-06-18 16:34:30,289 DEBUG hdfs.DFSClient
(DFSClient.java:createBlockOutputStream(3852)) - Connecting to 127.0.0.1:51117
2013-06-18 16:34:30,289 DEBUG hdfs.DFSClient
(DFSClient.java:getRandomLocalInterfaceAddr(1204)) - Using local interface
{color:red}*/0:0:0:0:0:0:0:1%1:0*{color}
2013-06-18 16:34:30,289 INFO hdfs.DFSClient
(DFSClient.java:createBlockOutputStream(3904)) - Exception in
createBlockOutputStream 127.0.0.1:51117 java.net.SocketException: Network is
unreachable
2013-06-18 16:34:30,290 INFO hdfs.DFSClient
(DFSClient.java:nextBlockOutputStream(3810)) - Abandoning
blk_7035064663379527570_1001
2013-06-18 16:34:30,292 INFO hdfs.DFSClient
(DFSClient.java:nextBlockOutputStream(3814)) - Excluding datanode
127.0.0.1:51117
2013-06-18 16:34:30,294 WARN namenode.FSNamesystem
(BlockPlacementPolicyDefault.java:chooseTarget(209)) - Not able to place enough
replicas, still in need of 1 to reach 1
Not able to place enough replicas
2013-06-18 16:34:30,294 ERROR security.UserGroupInformation
(UserGroupInformation.java:doAs(1235)) - PriviledgedActionException as:ddn
cause:java.io.IOException: File /user/ddn/filestatus.dat could only be
replicated to 0 nodes, instead of 1
2013-06-18 16:34:30,294 INFO ipc.Server (Server.java:run(1463)) - IPC Server
handler 8 on 41922, call addBlock(/user/ddn/filestatus.dat,
DFSClient_NONMAPREDUCE_-97075681_1,
[Lorg.apache.hadoop.hdfs.protocol.DatanodeInfo;@b04bce3) from 127.0.0.1:48545:
error: java.io.IOException: File /user/ddn/filestatus.dat could only be
replicated to 0 nodes, instead of 1
java.io.IOException: File /user/ddn/filestatus.dat could only be replicated to
0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1983)
h3. The following works:
[root@host] echo 1 /proc/sys/net/ipv6/conf/all/disable_ipv6
2013-06-18 16:52:41,890 DEBUG hdfs.DFSClient (DFSClient.java:init(372)) -
Using local interfaces [lo] with addresses [/127.0.0.1:0]
2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient
(DFSClient.java:createBlockOutputStream(3852)) - Connecting to 127.0.0.1:45078
2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient
(DFSClient.java:getRandomLocalInterfaceAddr(1204)) - Using local interface
/127.0.0.1:0
2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient
(DFSClient.java:createBlockOutputStream(3856)) - Send buf size 131072
2013-06-18 16:52:42,030 INFO datanode.DataNode
(DataXceiver.java:writeBlock(249)) - Receiving blk_4193380550170530321_1001
src: /127.0.0.1:60010 dest: /127.0.0.1:45078
2013-06-18 16:52:42,032 DEBUG hdfs.DFSClient (DFSClient.java:run(3277)) -
DataStreamer block blk_4193380550170530321_1001 wrote packet seqno:0 size:8281
offsetInBlock:0 lastPacketInBlock:true
h3. Summary/Proposal:
Need a fix for DFSClient to understand and support (or not) IPv6
unit test failure:
org.apache.hadoop.hdfs.TestFileCreation.testFileCreationSetLocalInterface
Key: HADOOP-9586
URL: https://issues.apache.org/jira/browse/HADOOP-9586
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 1.3.0
Reporter: Giridharan Kesavan
https://builds.apache.org/job/Hadoop-branch1/lastCompletedBuild/testReport/org.apache.hadoop.hdfs/TestFileCreation/testFileCreationSetLocalInterface/
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/jenkins/filestatus.dat could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at
[jira] [Updated] (HADOOP-9652) FileContext#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-9652:
-
Description:
{{FileContext#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If the
target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to, and
getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777. We
ought to expose this in RawLocalFilesystem and other places, although we don't
necessarily have to support this behavior in HDFS.
was:
{{FileContext#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If the
target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from who created them, and getFileLinkStatus
ought to expose this.
In some operating systems, symlinks can have a permission other than 0777. We
ought to expose this in RawLocalFilesystem and other places, although we don't
necessarily have to support this behavior in HDFS.
FileContext#getFileLinkStatus does not fill in the link owner and mode
--
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
{{FileContext#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8040) Add symlink support to FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687448#comment-13687448
]
Hadoop QA commented on HADOOP-8040:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588466/hadoop-8040-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2668//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2668//console
This message is automatically generated.
Add symlink support to FileSystem
-
Key: HADOOP-8040
URL: https://issues.apache.org/jira/browse/HADOOP-8040
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha
Reporter: Eli Collins
Assignee: Andrew Wang
Attachments: hadoop-8040-1.patch, hadoop-8040-2.patch,
hadoop-8040-3.patch, hadoop-8040-4.patch, hadoop-8040-5.patch,
hadoop-8040-6.patch
HADOOP-6421 added symbolic links to FileContext. Resolving symlinks is done
on the client-side, and therefore requires client support. An HDFS symlink
(created by FileContext) when accessed by FileSystem will result in an
unhandled UnresolvedLinkException. Because not all users will migrate from
FileSystem to FileContext in lock step, and we want users of FileSystem to be
able to access all paths created by FileContext, we need to support symlink
resolution in FileSystem as well, to facilitate migration to FileContext.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9439: - Attachment: HADOOP-9439.006.patch * declare pw_lock_object as extern * put declaration of pw_lock_locked on a separate line JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687524#comment-13687524
]
Hadoop QA commented on HADOOP-9439:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588505/HADOOP-9439.006.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2671//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2671//console
This message is automatically generated.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HADOOP-9439.006.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[
https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687530#comment-13687530
]
Hadoop QA commented on HADOOP-9631:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588487/HADOOP-9631.trunk.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2670//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2670//console
This message is automatically generated.
ViewFs should use underlying FileSystem's server side defaults
--
Key: HADOOP-9631
URL: https://issues.apache.org/jira/browse/HADOOP-9631
Project: Hadoop Common
Issue Type: Bug
Components: fs, viewfs
Affects Versions: 2.0.4-alpha
Reporter: Lohit Vijayarenu
Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch,
HADOOP-9631.trunk.3.patch, TestFileContext.java
On a cluster with ViewFS as default FileSystem, creating files using
FileContext will always result with replication factor of 1, instead of
underlying filesystem default (like HDFS)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6837) Support for LZMA compression
[ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687627#comment-13687627 ] Harsh J commented on HADOOP-6837: - FB's hadoop-0.20 seems to have a working implementation of this, although I do not know results of its stability yet: https://github.com/facebook/hadoop-20/blob/master/src/core/org/apache/hadoop/io/compress/LzmaCodec.java (and others). Support for LZMA compression Key: HADOOP-6837 URL: https://issues.apache.org/jira/browse/HADOOP-6837 Project: Hadoop Common Issue Type: Improvement Components: io Reporter: Nicholas Carlini Assignee: Nicholas Carlini Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687643#comment-13687643 ] Sanjay Radia commented on HADOOP-9421: -- bq. Client today does send/(send/read)+, and now it's send/read/(send/read)+. Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
