[jira] [Created] (HADOOP-11163) MetricsSystemImpl may miss a registered source
Chuan Liu created HADOOP-11163:
--
Summary: MetricsSystemImpl may miss a registered source
Key: HADOOP-11163
URL: https://issues.apache.org/jira/browse/HADOOP-11163
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Calling to {{MetricsSystemImpl#register(String name, String desc, T source)}}
could miss when the name is {{null}}. This is related to my previous change in
HADOOP-11105.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11105) MetricsSystemImpl could leak memory in registered callbacks
Chuan Liu created HADOOP-11105:
--
Summary: MetricsSystemImpl could leak memory in registered
callbacks
Key: HADOOP-11105
URL: https://issues.apache.org/jira/browse/HADOOP-11105
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
In {{MetricsSystemImpl.register(final String name, final String description,
final T sink)}} method, we will also add a callback function to the
{{callbacks}} array list. However, upon unregistering, the callback function is
not removed from the list in existing code. As a result, the memory allocated
for the callback function could accumulate in the {{MetricsSystemImpl}} list.
In one of our cluster, we have seen a case where there are about 874 MB memory
retained by {{MetricsSystemImpl}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-10175) Har files system authority should preserve userinfo
Chuan Liu created HADOOP-10175: -- Summary: Har files system authority should preserve userinfo Key: HADOOP-10175 URL: https://issues.apache.org/jira/browse/HADOOP-10175 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Chuan Liu Assignee: Chuan Liu When Har file system parse the URI get the authority at initialization, the userinfo is not preserved. This may lead to failures if the underlying file system relies on the userinfo to work properly. E.g. har://file-user:passwd@localhost:80/test.har will be parsed to har://file-localhost:80/test.har, where user:passwd is lost in the processing. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HADOOP-10110) hadoop-auth has a build break due to missing dependency
Chuan Liu created HADOOP-10110:
--
Summary: hadoop-auth has a build break due to missing dependency
Key: HADOOP-10110
URL: https://issues.apache.org/jira/browse/HADOOP-10110
Project: Hadoop Common
Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Blocker
We have a build break in hadoop-auth if build with maven cache cleaned. The
error looks like the follows. The problem exists on both Windows and Linux. If
you have old jetty jars in your maven cache, you won't see the error.
{noformat}
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 1:29.469s
[INFO] Finished at: Mon Nov 18 12:30:36 PST 2013
[INFO] Final Memory: 37M/120M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hadoop-auth: Compilation failure: Compilation
failure:
[ERROR]
/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]
cannot access org.mortbay.component.AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
[ERROR] server = new Server(0);
[ERROR]
/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[94,29]
cannot access org.mortbay.component.LifeCycle
[ERROR] class file for org.mortbay.component.LifeCycle not found
[ERROR] server.getConnectors()[0].setHost(host);
[ERROR]
/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,10]
cannot find symbol
[ERROR] symbol : method start()
[ERROR] location: class org.mortbay.jetty.Server
[ERROR]
/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[102,12]
cannot find symbol
[ERROR] symbol : method stop()
[ERROR] location: class org.mortbay.jetty.Server
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-auth
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HADOOP-10082) TestHDFSCLI fails on Windows due to difference in line endings
Chuan Liu created HADOOP-10082: -- Summary: TestHDFSCLI fails on Windows due to difference in line endings Key: HADOOP-10082 URL: https://issues.apache.org/jira/browse/HADOOP-10082 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0 Reporter: Chuan Liu Priority: Minor This is similar issue with HADOOP-8657. git appended /r by default on Windows at checkout. When TestHDFSCLI runs, the verification will fail on file size for files like data15bytes, data30bytes, etc. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-10031) Investigate PathData.expandAsGlob() support for Windows path
Chuan Liu created HADOOP-10031: -- Summary: Investigate PathData.expandAsGlob() support for Windows path Key: HADOOP-10031 URL: https://issues.apache.org/jira/browse/HADOOP-10031 Project: Hadoop Common Issue Type: Bug Reporter: Chuan Liu Assignee: Chuan Liu The current PathData.expandAsGlob() does not support Windows path of the pattern "C:\some\dir". This method is used by many fs shell command to expand the paths from command line arguments. As a result, running some fs shell command with Windows local path as arguments will fail. This is a regression with 1.0. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-10030) FsShell -put/copyFromLocal should support Windows local path
Chuan Liu created HADOOP-10030:
--
Summary: FsShell -put/copyFromLocal should support Windows local
path
Key: HADOOP-10030
URL: https://issues.apache.org/jira/browse/HADOOP-10030
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
In current trunk code, running 'hadoop fs -put c:\windows\path /hdfs/path' will
lead to an error {{put: unexpected URISyntaxException}}. This is a regression
to the 1.0 fs shell commands. FsShell put and copyFromLocal should support
Windows local path formats as localSrc argument to the two commands.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HADOOP-9948) Add a config value to CLITestHelper to skip tests on Windows
Chuan Liu created HADOOP-9948:
-
Summary: Add a config value to CLITestHelper to skip tests on
Windows
Key: HADOOP-9948
URL: https://issues.apache.org/jira/browse/HADOOP-9948
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
We want to add a configuration to CLITestHelper so we can skip some tests on
Windows. We want this feature because some test cases in TestHDFSCLI should not
be tested on Windows. Or more specifically, the globbing test cases. The
differences are explained in HDFS-4632. The proposed syntax looks like follows.
{noformat}
ls: Negative test for quoted /*/* globbing
false
{noformat}
When Windows set to false, we will skip running the test on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
Chuan Liu created HADOOP-9865:
-
Summary: FileContext.globStatus() has a regression with respect to
relative path
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Reporter: Chuan Liu
Assignee: Chuan Liu
I discovered the problem when running unit test TestMRJobClient. The cause is
indirect in this case. In the unit test, we try to launch a job and list its
status. The job failed, and caused the list command get a result of 0, which
triggered the unit test assert. From the log and debug, the job failed because
we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This step failed
because we passed in a relative path to {{FileContext.globStatus()}} in
{{FileUtil.createJarWithClassPath}}. The relevant log looks like the following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0] launcher.ContainerLaunch
(ContainerLaunch.java:call(270)) - Failed to launch container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9759) Add support for NativeCodeLoader#getLibraryName on Windows
Chuan Liu created HADOOP-9759:
-
Summary: Add support for NativeCodeLoader#getLibraryName on Windows
Key: HADOOP-9759
URL: https://issues.apache.org/jira/browse/HADOOP-9759
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Critical
HADOOP-9164 introduced a new native API to get the library path for the Hadoop
native libraries and other third party native libraries. There is no equivalent
support of this native API in the original patch. This JIRA is created to track
the work to support {{NativeCodeLoader#getLibraryName}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9681) FileUtil.unTarUsingJava() should close the InputStream upon finishing
Chuan Liu created HADOOP-9681:
-
Summary: FileUtil.unTarUsingJava() should close the InputStream
upon finishing
Key: HADOOP-9681
URL: https://issues.apache.org/jira/browse/HADOOP-9681
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
In {{FileUtil.unTarUsingJava()}} method, we did not close input steams
explicitly upon finish. This could lead to a file handle leak on Windows.
I discovered this when investigating the unit test case failure of
{{TestFSDownload.testDownloadArchive()}}. FSDownload class will use
{{FileUtil.unTarUsingJava()}} to unpack some temporary archive file. Later, the
temporary file should be deleted. Because of the file handle leak, the
{{File.delete()}} method fails. The test case then fails because it assert the
temporary file should not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8440) HarFileSystem.decodeHarURI fails for URIs whose host contains numbers
[ https://issues.apache.org/jira/browse/HADOOP-8440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu reopened HADOOP-8440: --- We also need this fix in trunk and branch-2. > HarFileSystem.decodeHarURI fails for URIs whose host contains numbers > - > > Key: HADOOP-8440 > URL: https://issues.apache.org/jira/browse/HADOOP-8440 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 1.0.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic >Priority: Minor > Attachments: HADOOP-8440-2-branch-1-win.patch, > HADOOP-8440-branch-1-win.2.patch, HADOOP-8440-branch-1-win.patch, > HADOOP-8440-branch-1-win.patch, HADOOP-8440-trunk.patch, > HADOOP-8440-trunk.patch > > > For example, HarFileSystem.decodeHarURI will fail for the following URI: > har://hdfs-127.0.0.1:51040/user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9656) Gridmix unit tests fail on Windows and Linux
Chuan Liu created HADOOP-9656:
-
Summary: Gridmix unit tests fail on Windows and Linux
Key: HADOOP-9656
URL: https://issues.apache.org/jira/browse/HADOOP-9656
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
The following three Gridmix unit tests fail on both Windows and Linux:
*TestGridmixSubmission
*TestLoadJob
*TestSleepJob
One common cause of failure for both Windows and Linux is that in
{{GridmixJob.configureHighRamProperties()}} method -1 was passed in to
{{scaleConfigParameter}} as default per task memory request.
In additional to the memory setting issue, Windows also have a path issue. In
{{CommonJobTest.doSubmission()}} method, "root" path is an HDFS path, however,
it is initialized as a local file path. This lead to later failure to create
"root" on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9526) TestShellCommandFencer and TestShell fail on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-9526. --- Resolution: Fixed Resolving this. The open problem should be tracked by HADOOP-9632 > TestShellCommandFencer and TestShell fail on Windows > > > Key: HADOOP-9526 > URL: https://issues.apache.org/jira/browse/HADOOP-9526 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HADOOP-9526.001.patch, HADOOP-9526.002.patch, > HADOOP-9526-host-fix.patch > > > The following TestShellCommandFencer tests fail on Windows. > # testTargetAsEnvironment > # testConfAsEnvironment > # testTargetAsEnvironment > TestShell#testInterval also fails. > All failures look like test issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9637) TestAggregatedLogFormat fails on Windows
Chuan Liu created HADOOP-9637:
-
Summary: TestAggregatedLogFormat fails on Windows
Key: HADOOP-9637
URL: https://issues.apache.org/jira/browse/HADOOP-9637
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows.
The test case try to simulate a situation where first log file is owned by
different user (probably symlink) and second one by the user itself. In this
situation, the attempt to try to aggregate the logs should fail with the error
message "Owner ... for path ... did not match expected owner ...".
The check on file owner happens at {{AggregatedLogFormat.write()}} method. The
method calls {{SecureIOUtils.openForRead()}} to read the log files before
writing out to the OutputStream.
{{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9632) TestShellCommandFencer will fail if there is a 'host' machine in the network
Chuan Liu created HADOOP-9632: - Summary: TestShellCommandFencer will fail if there is a 'host' machine in the network Key: HADOOP-9632 URL: https://issues.apache.org/jira/browse/HADOOP-9632 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor TestShellCommandFencer will fail if there is a machine named ‘host’ in the network. The %target_address% environment variable used in the test was from result of InetSocketAddress.getAddress(). The method will return 'host/ip' instead of only 'host' when the host actually exists in the network. When the test comparing the log output, it assumes there is no ip in the address. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-9526) TestShellCommandFencer and TestShell fail on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu reopened HADOOP-9526: --- TestShellCommandFencer will fail if there is a machine named ‘host’ in the network. The %target_address% environment variable used in the test was from result of InetSocketAddress.getAddress(). The method will return host/ip when the host actually exists in the network. When the test comparing the log output, it assumes there is no ip in the address. > TestShellCommandFencer and TestShell fail on Windows > > > Key: HADOOP-9526 > URL: https://issues.apache.org/jira/browse/HADOOP-9526 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HADOOP-9526.001.patch, HADOOP-9526.002.patch > > > The following TestShellCommandFencer tests fail on Windows. > # testTargetAsEnvironment > # testConfAsEnvironment > # testTargetAsEnvironment > TestShell#testInterval also fails. > All failures look like test issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu reopened HADOOP-8958: --- Reopen for porting to branch-2. The original patch should apply. > ViewFs:Non absolute mount name failures when running multiple tests on Windows > -- > > Key: HADOOP-8958 > URL: https://issues.apache.org/jira/browse/HADOOP-8958 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0 > > Attachments: HADOOP-8958.2.patch, HADOOP-8958.patch > > > This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-9131) TestLocalFileSystem#testListStatusWithColons cannot run on Windows
[
https://issues.apache.org/jira/browse/HADOOP-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chuan Liu reopened HADOOP-9131:
---
Reopen for porting to branch-2.
> TestLocalFileSystem#testListStatusWithColons cannot run on Windows
> --
>
> Key: HADOOP-9131
> URL: https://issues.apache.org/jira/browse/HADOOP-9131
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 3.0.0
>
> Attachments: HADOOP-9131.1.patch
>
>
> HADOOP-8962 added a new test,
> {{TestLocalFileSystem#testListStatusWithColons}}, covering the case of files
> that contain ':'. This test cannot pass on Windows, because on Windows, the
> local file system does not support ':' in file names.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8957) AbstractFileSystem#IsValidName should be overridden for embedded file systems like ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-8957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu reopened HADOOP-8957: --- Reopen for porting to branch-2. > AbstractFileSystem#IsValidName should be overridden for embedded file systems > like ViewFs > - > > Key: HADOOP-8957 > URL: https://issues.apache.org/jira/browse/HADOOP-8957 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0 > > Attachments: HADOOP-8957-branch-trunk-win.2.patch, > HADOOP-8957-branch-trunk-win.3.patch, HADOOP-8957-branch-trunk-win.4.patch, > HADOOP-8957.patch, HADOOP-8957.patch, HADOOP-8957-trunk.4.patch > > > This appears to be a problem with parsing a Windows-specific path, ultimately > throwing InvocationTargetException from AbstractFileSystem.newInstance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9620) Stabilize branch-2 on Windows
Chuan Liu created HADOOP-9620: - Summary: Stabilize branch-2 on Windows Key: HADOOP-9620 URL: https://issues.apache.org/jira/browse/HADOOP-9620 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Chuan Liu After merging trunk change into branch-2, we are seeing a new set of unit test failures in branch-2 on Windows. This an umbrella jira to track the remaining issues in branch-2 on Windows with the goal to stabilize branch-2 on Windows and achieve 100% unit test pass rate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9609) Remove sh dependency of bin-package target
Chuan Liu created HADOOP-9609: - Summary: Remove sh dependency of bin-package target Key: HADOOP-9609 URL: https://issues.apache.org/jira/browse/HADOOP-9609 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor In Ant package target, we no longer use packageBinNativeHadoop.sh to place native library binaries. However, the same script is still present in the bin-package target. We should remove bin-package target's dependency on sh to keep it consistent with package target. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9443) Port winutils static code analysis change to trunk
Chuan Liu created HADOOP-9443: - Summary: Port winutils static code analysis change to trunk Key: HADOOP-9443 URL: https://issues.apache.org/jira/browse/HADOOP-9443 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Chuan Liu Assignee: Chuan Liu We hit a problem in winutils when running tests on Windows. The static code analysis change will fix the problem. More specifically, the old code always assumes the security descriptor get from GetSecurityDescriptorControl() is relative, and will make an absolute security descriptor out of it. The new absolute security descriptor will then pass to SetSecurityDescriptorDacl() to set permissions on the file. If the security descriptor is absolute, the new absolute security descriptor will be NULL, and we will run into the problem. This is what happened exactly in our case. The fix from static code analysis will solve the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8601) Extend TestShell to cover Windows shell commands
[ https://issues.apache.org/jira/browse/HADOOP-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8601. --- Resolution: Implemented Fix Version/s: 1-win With HADOOP-8972, we will test Windows shell commands with a separate test: TestWinUtils, and this JIRA is no longer needed. > Extend TestShell to cover Windows shell commands > > > Key: HADOOP-8601 > URL: https://issues.apache.org/jira/browse/HADOOP-8601 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 1-win > > > The existing unit test only covers Linux shell commands. Since we begin to > support Windows and use completely different commands on Windows, it make > sense to extend TestShell to cover Windows use cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9007) TestJobStatusPersistency.testJobStoreDisablingWithInvalidPath failure
Chuan Liu created HADOOP-9007:
-
Summary:
TestJobStatusPersistency.testJobStoreDisablingWithInvalidPath failure
Key: HADOOP-9007
URL: https://issues.apache.org/jira/browse/HADOOP-9007
Project: Hadoop Common
Issue Type: Bug
Reporter: Chuan Liu
Priority: Minor
We have the unit test failure.
org.apache.hadoop.mapred.TestJobStatusPersistency.testJobStoreDisablingWithInvalidPath
Error Message
{noformat}
Cannot chmod parent!
Stacktrace
junit.framework.AssertionFailedError: Cannot chmod parent!
at
org.apache.hadoop.mapred.TestJobStatusPersistency.testJobStoreDisablingWithInvalidPath(TestJobStatusPersistency.java:161)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9006) Winutils should keep Administrators privileges intact
Chuan Liu created HADOOP-9006:
-
Summary: Winutils should keep Administrators privileges intact
Key: HADOOP-9006
URL: https://issues.apache.org/jira/browse/HADOOP-9006
Project: Hadoop Common
Issue Type: Bug
Reporter: Chuan Liu
Priority: Minor
This issue was originally discovered by [~ivanmi]. Cite his words as follows.
{quuote}
Current by design behavior is for winutils to ACL the folders only for the user
passed in thru chmod/chown. This causes some un-natural side effects in cases
where Hadoop services run in the context of a non-admin user. For example,
Administrators on the box will no longer be able to:
- delete files created in the context of Hadoop services (other users)
- check the size of the folder where HDFS blocks are stored
{quote}
In my opinion, it is natural for some special accounts on Windows to be able to
access all the folders, including Hadoop folders. This is similar to Linux in
the way root users on Linux can always access any directories regardless the
permissions set the those directories.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8683) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8683. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8683 > URL: https://issues.apache.org/jira/browse/HADOOP-8683 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 1-win > > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings, Local Policies, User Rights Assignment, Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8682) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8682. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8682 > URL: https://issues.apache.org/jira/browse/HADOOP-8682 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 1-win > > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings, Local Policies, User Rights Assignment, Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8681) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8681. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8681 > URL: https://issues.apache.org/jira/browse/HADOOP-8681 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 1-win > > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings, Local Policies, User Rights Assignment, Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8680) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8680. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8680 > URL: https://issues.apache.org/jira/browse/HADOOP-8680 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 1-win > > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings, Local Policies, User Rights Assignment, Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8679) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8679. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8679 > URL: https://issues.apache.org/jira/browse/HADOOP-8679 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings, Local Policies, User Rights Assignment, Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8678) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8678. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8678 > URL: https://issues.apache.org/jira/browse/HADOOP-8678 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings, Local Policies, User Rights Assignment, Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8677) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8677. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8677 > URL: https://issues.apache.org/jira/browse/HADOOP-8677 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8676) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8676. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8676 > URL: https://issues.apache.org/jira/browse/HADOOP-8676 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8675) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8675. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8675 > URL: https://issues.apache.org/jira/browse/HADOOP-8675 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8674) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8674. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8674 > URL: https://issues.apache.org/jira/browse/HADOOP-8674 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8672) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8672. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8672 > URL: https://issues.apache.org/jira/browse/HADOOP-8672 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. > > In Hadoop, symlinks is mostly used to for DistributedCache and attempted > logs. We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. > This JIRA is created to tracking symlink support on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8671) Add symbolic link support on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8671. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Add symbolic link support on Windows > > > Key: HADOOP-8671 > URL: https://issues.apache.org/jira/browse/HADOOP-8671 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. > > In Hadoop, symlinks is mostly used to for DistributedCache and attempted > logs. We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. > This JIRA is created to tracking symlink support on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8670) Add symbolic link support on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8670. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Add symbolic link support on Windows > > > Key: HADOOP-8670 > URL: https://issues.apache.org/jira/browse/HADOOP-8670 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. > > In Hadoop, symlinks is mostly used to for DistributedCache and attempted > logs. We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. > This JIRA is created to tracking symlink support on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8673) Create true symbolic links on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8673. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Create true symbolic links on Windows > - > > Key: HADOOP-8673 > URL: https://issues.apache.org/jira/browse/HADOOP-8673 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. In > Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. > We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. This JIRA is created to tracking symlink support on > Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8669) Add symbolic link support on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu resolved HADOOP-8669. --- Resolution: Duplicate Duplicate of HADOOP-8694 > Add symbolic link support on Windows > > > Key: HADOOP-8669 > URL: https://issues.apache.org/jira/browse/HADOOP-8669 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu > > In branch-1-win, we currently copy files for symbolic links in Hadoop on > Windows. We have talked to [~davidlao] who made the original fix, and did > some investigation on Windows. Windows began to support symbolic links > (symlinks) since Vista/Server 2008. The original reason to copy files instead > of creating actual symlinks is that only Administrators has the privilege to > create symlinks on Windows _by default_. After talking to NTFS folks, we knew > the reason for that is mostly due to security, and this default behavior may > not be changed in near future. Though this behavior can be changed via the > Local Security Policy management console, i.e. secpol.msc, under Security > Settings\Local Policies\User Rights Assignment\Create symbolic links. > > In Hadoop, symlinks is mostly used to for DistributedCache and attempted > logs. We felt the usages are important enough for us to provide true symlinks > support, and users need to have the symlink creation privilege enabled on > Windows to use Hadoop. > This JIRA is created to tracking symlink support on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8936) Local users should be able to query a domain user's groups on Windows
Chuan Liu created HADOOP-8936: - Summary: Local users should be able to query a domain user's groups on Windows Key: HADOOP-8936 URL: https://issues.apache.org/jira/browse/HADOOP-8936 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor Fix For: 1-win When Hadoop run by a local user, and a domain user submit a job, Hadoop will need to get the local groups for the domain user. This fails in 'winutils' now because we tried to query domain controller for domain users and local users does not have the permission to do so. We should fix the problem so that local users should be able to query a domain user's local groups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8935) Make 'winutils ls' show the SID if the owner does not exist on the system
Chuan Liu created HADOOP-8935: - Summary: Make 'winutils ls' show the SID if the owner does not exist on the system Key: HADOOP-8935 URL: https://issues.apache.org/jira/browse/HADOOP-8935 Project: Hadoop Common Issue Type: Bug Reporter: Chuan Liu Priority: Minor Right now, 'winutils ls' will fail if the file belongs to a user SID that does not exist on the system. E.g. the user is deleted. Previously, this is only a hypothesis scenario. However, we have seen some failures in the Azure deployment where the OS is re-imaged, and renders the old SID invalid. [~jgordon] proposed to display the SID itself in the invalid SID case similar to the situation on Linux. This JIRA is created to track this proposal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8908) 'winutils.exe' code refactory
Chuan Liu created HADOOP-8908: - Summary: 'winutils.exe' code refactory Key: HADOOP-8908 URL: https://issues.apache.org/jira/browse/HADOOP-8908 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1-win Reporter: Chuan Liu Priority: Minor Fix For: 1-win We want to split the existing 'winutil.exe' code into a library project and a executable project. The library project will be able to be linked and used in future projects. It is also good software engineering practice to introduce the modularity into the project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8763) Set group owner on Windows failed
Chuan Liu created HADOOP-8763: - Summary: Set group owner on Windows failed Key: HADOOP-8763 URL: https://issues.apache.org/jira/browse/HADOOP-8763 Project: Hadoop Common Issue Type: Bug Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor Fix For: 1-win RawLocalFileSystem.setOwner() method may incorrectly set the group owner of a file on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8694) Create true symbolic links on Windows
Chuan Liu created HADOOP-8694: - Summary: Create true symbolic links on Windows Key: HADOOP-8694 URL: https://issues.apache.org/jira/browse/HADOOP-8694 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 1-win In branch-1-win, we currently copy files for symbolic links in Hadoop on Windows. We have talked to [~davidlao] who made the original fix, and did some investigation on Windows. Windows began to support symbolic links (symlinks) since Vista/Server 2008. The original reason to copy files instead of creating actual symlinks is that only Administrators has the privilege to create symlinks on Windows _by default_. After talking to NTFS folks, we knew the reason for that is mostly due to security, and this default behavior may not be changed in near future. Though this behavior can be changed via the Local Security Policy management console, i.e. secpol.msc, under Security Settings\Local Policies\User Rights Assignment\Create symbolic links. In Hadoop, symlinks is mostly used to for DistributedCache and attempted logs. We felt the usages are important enough for us to provide true symlinks support, and users need to have the symlink creation privilege enabled on Windows to use Hadoop. This JIRA is created to tracking symlink support on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8601) Extend TestShell to cover Windows shell commands
Chuan Liu created HADOOP-8601: - Summary: Extend TestShell to cover Windows shell commands Key: HADOOP-8601 URL: https://issues.apache.org/jira/browse/HADOOP-8601 Project: Hadoop Common Issue Type: Test Affects Versions: 1-win Reporter: Chuan Liu Assignee: Chuan Liu The existing unit test only covers Linux shell commands. Since we begin to support Windows and use completely different commands on Windows, it make sense to extend TestShell to cover Windows use cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8564) Create a Windows native InputStream class to address datanode concurrent reading and writing issue
Chuan Liu created HADOOP-8564:
-
Summary: Create a Windows native InputStream class to address
datanode concurrent reading and writing issue
Key: HADOOP-8564
URL: https://issues.apache.org/jira/browse/HADOOP-8564
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
HDFS files are made up of blocks. First, let’s look at writing. When the data
is written to datanode, an active or temporary file is created to receive
packets. After the last packet for the block is received, we will finalize the
block. One step during finalization is to rename the block file to a new
directory. The relevant code can be found via the call sequence:
FSDataSet.finalizeBlockInternal -> FSDir.addBlock.
{code}
if ( ! metaData.renameTo( newmeta ) ||
! src.renameTo( dest ) ) {
throw new IOException( "could not move files for " + b +
" from tmp to " +
dest.getAbsolutePath() );
}
{code}
Let’s then switch to reading. On HDFS, it is expected the client can also read
these unfinished blocks. So when the read calls from client reach datanode, the
datanode will open an input stream on the unfinished block file.
The problem comes in when the file is opened for reading while the datanode
receives last packet from client and try to rename the finished block file.
This operation will succeed on Linux, but not on Windows . The behavior can be
modified on Windows to open the file with FILE_SHARE_DELETE flag on, i.e.
sharing the delete (including renaming) permission with other processes while
opening the file. There is also a Java bug ([id
6357433|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6357433]) reported a
while back on this. However, since this behavior exists for Java on Windows
since JDK 1.0, the Java developers do not want to break the backward
compatibility on this behavior. Instead, a new file system API is proposed in
JDK 7.
As outlined in the [Java forum|http://www.java.net/node/645421] by the Java
developer (kbr), there are three ways to fix the problem:
# Use different mechanism in the application in dealing with files.
# Create a new implementation of InputStream abstract class using Windows
native code.
# Patch JDK with a private patch that alters FileInputStream behavior.
For the third option, it cannot fix the problem for users using Oracle JDK.
We discussed some options for the first approach. For example one option is to
use two phase renaming, i.e. first hardlink; then remove the old hardlink when
read is finished. This option was thought to be rather pervasive. Another
option discussed is to change the HDFS behavior on Windows by not allowing
client reading unfinished blocks. However this behavior change is thought to be
problematic and may affect other application build on top of HDFS.
For all the reasons discussed above, we will use the second approach to address
the problem.
If there are better options to fix the problem, we would also like to hear
about them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8544) Move an assertion location in 'winutils chmod'
Chuan Liu created HADOOP-8544: - Summary: Move an assertion location in 'winutils chmod' Key: HADOOP-8544 URL: https://issues.apache.org/jira/browse/HADOOP-8544 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Chuan Liu Assignee: Chuan Liu Priority: Trivial We have an assertion in chmod that will be triggered in case of permission change without giving a permission or a reference, e.g. 'chmod + [FILE]'. Such operations are valid, and will trigger the assertion for winutils debug build. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8457) Address file ownership issue for users in Administrators group on Windows.
Chuan Liu created HADOOP-8457: - Summary: Address file ownership issue for users in Administrators group on Windows. Key: HADOOP-8457 URL: https://issues.apache.org/jira/browse/HADOOP-8457 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.1.0, 0.24.0 Reporter: Chuan Liu Priority: Minor On Linux, the initial file owners are the creators. (I think this is true in general. If there are exceptions, please let me know.) On Windows, the file created by a user in the Administrators group has the initial owner ‘Administrators’, i.e. the the Administrators group is the initial owner of the file. As a result, this leads to an exception when we check file ownership in SecureIOUtils .checkStat() method. As a result, this method is disabled right now. We need to address this problem and enable the method on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8456) Support spaces in user names and group names.
Chuan Liu created HADOOP-8456: - Summary: Support spaces in user names and group names. Key: HADOOP-8456 URL: https://issues.apache.org/jira/browse/HADOOP-8456 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.1.0, 0.24.0 Reporter: Chuan Liu Priority: Minor When parsing results returned by ‘ls’, we made implicit assumption that user and group names cannot contain spaces. On Linux, spaces are not allowed in user names and group names. This is not the case for Windows. We need to find a way to fix the problem for Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8454) Fix the ‘chmod =[perm]’ bug in winutils
Chuan Liu created HADOOP-8454: - Summary: Fix the ‘chmod =[perm]’ bug in winutils Key: HADOOP-8454 URL: https://issues.apache.org/jira/browse/HADOOP-8454 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 1.1.0, 0.24.0 Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor The original patch for [Hadoop-8235|https://issues.apache.org/jira/browse/HADOOP-8235] contained a bug for ‘chmod’ implantation. The logic to compute new access mask when ‘chmod’ mode string has ‘=’ in them is incorrect. For example, ‘winutils chmod o=g foo’ will result wrong permission settings for the file ‘foo’. The Jira is created to track the bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8455) Address user name format on domain joined Windows machines
Chuan Liu created HADOOP-8455: - Summary: Address user name format on domain joined Windows machines Key: HADOOP-8455 URL: https://issues.apache.org/jira/browse/HADOOP-8455 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.1.0, 0.24.0 Reporter: Chuan Liu Assignee: Ivan Mitic Priority: Minor For a domain joined Windows machine, user name along is not a unique identifier. User name plus domain name is need in order to unique identify the user. For example, we can have both ‘Win1\Alex’ and ‘Redmond\Alex’ on a computer named Win1 that joins Redmond domain. In order to avoid ambiguity, ‘whoami’ on Windows and the new ‘winutils’ created in [Hadoop-8235|https://issues.apache.org/jira/browse/HADOOP-8235] both return [domain]\[username] as the username. In Hadoop, we only use user name right now. This may lead to some inconsistency, and production bugs if users of the same name exist on the machine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8453) Add unit tests for winutils
Chuan Liu created HADOOP-8453: - Summary: Add unit tests for winutils Key: HADOOP-8453 URL: https://issues.apache.org/jira/browse/HADOOP-8453 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 1.1.0, 0.24.0 Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor In [Hadoop-8235|https://issues.apache.org/jira/browse/HADOOP-8235], we created a Windows console program, named ‘winutils’, to emulate some Linux command line utilities used by Hadoop. However no tests are provided in the original patch. As this code is quite complicated, and the complexity may even grow up in the future. We think unit tests are necessary to ensure code quality, as well as smooth future development. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
