[jira] Created: (HADOOP-7006) hadoop fs -getmerge does not work using codebase from trunk.
hadoop fs -getmerge does not work using codebase from trunk. Key: HADOOP-7006 URL: https://issues.apache.org/jira/browse/HADOOP-7006 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.22.0 Reporter: Chris Nauroth Fix For: 0.22.0 Running the codebase from trunk, the hadoop fs -getmerge command does not work. As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files. It would merge all files into a single file on the local file system. Running the same command using the codebase from trunk, it looks like nothing happens. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] [Created] (HADOOP-12172) FsShell mkdir -p makes an unnecessary check for the existence of the parent.
Chris Nauroth created HADOOP-12172:
--
Summary: FsShell mkdir -p makes an unnecessary check for the
existence of the parent.
Key: HADOOP-12172
URL: https://issues.apache.org/jira/browse/HADOOP-12172
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
The {{mkdir}} command in {{FsShell}} checks for the existence of the parent of
the directory and returns an error if it doesn't exist. The {{-p}} option
suppresses the error and allows the directory creation to continue, implicitly
creating all missing intermediate directories. However, the existence check
still runs even with {{-p}} specified, and its result is ignored. Depending on
the file system, this is a wasteful RPC call (HDFS) or HTTP request
(WebHDFS/S3/Azure) imposing extra latency for the client and extra load for the
server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12204) Use consistent indentation in releasedocmaker.py.
Chris Nauroth created HADOOP-12204: -- Summary: Use consistent indentation in releasedocmaker.py. Key: HADOOP-12204 URL: https://issues.apache.org/jira/browse/HADOOP-12204 Project: Hadoop Common Issue Type: Sub-task Reporter: Chris Nauroth Priority: Minor releasedocmaker.py currently uses a mix of 2-space and 4-space indentation. It would be good to make this consistent for readability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12232) Upgrade Tomcat dependency to 6.0.44.
Chris Nauroth created HADOOP-12232: -- Summary: Upgrade Tomcat dependency to 6.0.44. Key: HADOOP-12232 URL: https://issues.apache.org/jira/browse/HADOOP-12232 Project: Hadoop Common Issue Type: Improvement Components: build Reporter: Chris Nauroth Assignee: Chris Nauroth The Hadoop distro currently bundles Tomcat version 6.0.41 by default. The current Tomcat 6 version is 6.0.44, which includes a few incremental bug fixes. Let's update our default version so that our users get the latest bug fixes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12304) Applications using FileContext fail with the default file system configured to be wasb/s3/etc.
Chris Nauroth created HADOOP-12304:
--
Summary: Applications using FileContext fail with the default file
system configured to be wasb/s3/etc.
Key: HADOOP-12304
URL: https://issues.apache.org/jira/browse/HADOOP-12304
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.7.1, 2.7.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Blocker
HADOOP-11618 fixed a bug with {{DelegateToFileSystem}} using the wrong default
port. As a side effect of this patch, file path URLs that previously had no
port now insert :0 for the port, as per the default implementation of
{{FileSystem#getDefaultPort}}. At runtime, this can cause an application to
erroneously try contacting port 0 for a remote blob store service. The
connection fails. Ultimately, this renders wasb, s3, and probably custom file
system implementations outside the Hadoop source tree completely unusable as
the default file system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12390) Enhance FsShell file put to support selectively preserving individual file attributes.
Chris Nauroth created HADOOP-12390:
--
Summary: Enhance FsShell file put to support selectively
preserving individual file attributes.
Key: HADOOP-12390
URL: https://issues.apache.org/jira/browse/HADOOP-12390
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Chris Nauroth
{{hadoop fs -put}} currently supports the {{-p}} option. When used, this
option preserves the source file's access time, modification time, ownership
and mode. If the destination is HDFS, then this effectively means HDFS must be
configured to use modification time. If the HDFS deployment chooses to disable
modification time by setting {{dfs.namenode.accesstime.precision}} to 0, then
attempts to use the {{-p}} flag all fail with "Access time for hdfs is not
configured." This issue proposes to introduce separate options for preserving
just ownership, just mode, or just times. For backwards compatibility, the
behavior of a bare {{-p}} must continue to be preserving all 3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11742) mkdir by file system shell fails on an empty bucket
[
https://issues.apache.org/jira/browse/HADOOP-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-11742.
Resolution: Duplicate
I'm resolving this as a duplicate of HADOOP-11918. Reading through the history
on both issues, it appears that there is more recent activity on HADOOP-11918,
and the patch there is closer to acceptance. (If I'm mistaken, then please
feel free to reopen this.)
> mkdir by file system shell fails on an empty bucket
> ---
>
> Key: HADOOP-11742
> URL: https://issues.apache.org/jira/browse/HADOOP-11742
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.0
> Environment: CentOS 7
>Reporter: Takenori Sato
>Assignee: Takenori Sato
>Priority: Minor
> Attachments: HADOOP-11742-branch-2.7.001.patch,
> HADOOP-11742-branch-2.7.002.patch, HADOOP-11742-branch-2.7.003-1.patch,
> HADOOP-11742-branch-2.7.003-2.patch
>
>
> I have built the latest 2.7, and tried S3AFileSystem.
> Then found that _mkdir_ fails on an empty bucket, named *s3a* here, as
> follows:
> {code}
> # hadoop-2.7.0-SNAPSHOT/bin/hdfs dfs -mkdir s3a://s3a/foo
> 15/03/24 03:49:35 DEBUG s3a.S3AFileSystem: Getting path status for
> s3a://s3a/foo (foo)
> 15/03/24 03:49:36 DEBUG s3a.S3AFileSystem: Not Found: s3a://s3a/foo
> 15/03/24 03:49:36 DEBUG s3a.S3AFileSystem: Getting path status for s3a://s3a/
> ()
> 15/03/24 03:49:36 DEBUG s3a.S3AFileSystem: Not Found: s3a://s3a/
> mkdir: `s3a://s3a/foo': No such file or directory
> {code}
> So does _ls_.
> {code}
> # hadoop-2.7.0-SNAPSHOT/bin/hdfs dfs -ls s3a://s3a/
> 15/03/24 03:47:48 DEBUG s3a.S3AFileSystem: Getting path status for s3a://s3a/
> ()
> 15/03/24 03:47:48 DEBUG s3a.S3AFileSystem: Not Found: s3a://s3a/
> ls: `s3a://s3a/': No such file or directory
> {code}
> This is how it works via s3n.
> {code}
> # hadoop-2.7.0-SNAPSHOT/bin/hdfs dfs -ls s3n://s3n/
> # hadoop-2.7.0-SNAPSHOT/bin/hdfs dfs -mkdir s3n://s3n/foo
> # hadoop-2.7.0-SNAPSHOT/bin/hdfs dfs -ls s3n://s3n/
> Found 1 items
> drwxrwxrwx - 0 1970-01-01 00:00 s3n://s3n/foo
> {code}
> The snapshot is the following:
> {quote}
> \# git branch
> \* branch-2.7
> trunk
> \# git log
> commit 929b04ce3a4fe419dece49ed68d4f6228be214c1
> Author: Harsh J
> Date: Sun Mar 22 10:18:32 2015 +0530
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HADOOP-10761) Provide an NFS gateway option to get groups from local name service for easy deployment in cloud
[ https://issues.apache.org/jira/browse/HADOOP-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reopened HADOOP-10761: I'm thinking this was closed by mistake. Typically, we resolve the issue only after work is done and patches are reviewed and committed. This patch has not been committed. > Provide an NFS gateway option to get groups from local name service for easy > deployment in cloud > > > Key: HADOOP-10761 > URL: https://issues.apache.org/jira/browse/HADOOP-10761 > Project: Hadoop Common > Issue Type: New Feature > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li > Attachments: RpcProgramNfs3.java, SysSecurityHandler.java, patch.txt > > > In cloud deployment, client machines and Hadoop Cluster are sometimes in > different DMZ and LDAP in some other. Integration client machines with LDAP > might require lot of firewall tickets, which can be a pain especially when > users keep on adding new client machines. > When the client is not integrated with LDAP for the group mapping, NFS should > not use the group list in the request. Instead, NFS gateway can get the group > list from local name service. > This JIRA is to track the effort to provide a configuration option to get > list of GID from LDAP/OS for making the deployment at Cloud easier. Also this > can remove the limitation of 15 GID in the NFS request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12438) TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.
Chris Nauroth created HADOOP-12438:
--
Summary: TestLocalFileSystem tests can fail on Windows after
HDFS-8767 fix for handling pipe.
Key: HADOOP-12438
URL: https://issues.apache.org/jira/browse/HADOOP-12438
Project: Hadoop Common
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Trivial
A test written as part of HDFS-8767 uses Mockito's
{{Whitebox#setInternalState}} to force the static
{{RawLocalFileSystem#useDeprecatedFileStatus}} to {{false}}. On systems that
don't implement a {{stat}} call (notably Windows), this leaves the class in a
state where it will attempt the {{stat}} call anyway during subsequent tests.
This ultimately causes tests to fail with {{UnsupportedOperationException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12479) ProtocMojo does not log the reason for a protoc compilation failure.
Chris Nauroth created HADOOP-12479:
--
Summary: ProtocMojo does not log the reason for a protoc
compilation failure.
Key: HADOOP-12479
URL: https://issues.apache.org/jira/browse/HADOOP-12479
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
If protoc fails with a compilation error in the proto files, our Maven plugin
won't print the details. The only way to figure it out is to repeat running
the {{protoc}} command manually from outside the Hadoop build. This is because
our {{ProtocMojo}} only captures stdout from the {{protoc}} command, and
compilation errors get written to {{stderr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-12390) Enhance FsShell file put to support selectively preserving individual file attributes.
[
https://issues.apache.org/jira/browse/HADOOP-12390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-12390.
Resolution: Duplicate
[~jagadesh.kiran], yes, thank you for the reminder. I'm resolving this as
duplicate now.
> Enhance FsShell file put to support selectively preserving individual file
> attributes.
> --
>
> Key: HADOOP-12390
> URL: https://issues.apache.org/jira/browse/HADOOP-12390
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Jagadesh Kiran N
>
> {{hadoop fs -put}} currently supports the {{-p}} option. When used, this
> option preserves the source file's access time, modification time, ownership
> and mode. If the destination is HDFS, then this effectively means HDFS must
> be configured to use modification time. If the HDFS deployment chooses to
> disable access time by setting {{dfs.namenode.accesstime.precision}} to 0,
> then attempts to use the {{-p}} flag all fail with "Access time for hdfs is
> not configured." This issue proposes to introduce separate options for
> preserving just ownership, just mode, or just times. For backwards
> compatibility, the behavior of a bare {{-p}} must continue to be preserving
> all 3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12519) hadoop-azure tests create a metrics configuration file in the module root directory.
Chris Nauroth created HADOOP-12519:
--
Summary: hadoop-azure tests create a metrics configuration file in
the module root directory.
Key: HADOOP-12519
URL: https://issues.apache.org/jira/browse/HADOOP-12519
Project: Hadoop Common
Issue Type: Bug
Components: azure, test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
The hadoop-azure JUnit tests create a metrics configuration file. This file
gets saved in the root directory of the hadoop-azure module. This dirties the
git workspace and won't get removed by {{mvn clean}}, because it's outside of
the build target directory. It also can cause the pre-commit license check
step to fail, because this ends up looking like the patch added a new file
without the Apache license header.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12520) Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests.
Chris Nauroth created HADOOP-12520: -- Summary: Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests. Key: HADOOP-12520 URL: https://issues.apache.org/jira/browse/HADOOP-12520 Project: Hadoop Common Issue Type: Improvement Components: azure, test Reporter: Chris Nauroth Assignee: Chris Nauroth The hadoop-azure tests support execution against the live Azure Storage service if the developer specifies the key to an Azure Storage account. The configuration works by overwriting the src/test/resources/azure-test.xml file. This can be an error-prone process. The azure-test.xml file is checked into revision control to show an example. There is a risk that the tester could overwrite azure-test.xml containing the keys and then accidentally commit the keys to revision control. This would leak the keys to the world for potential use by an attacker. This issue proposes to use XInclude to isolate the keys into a separate file, ignored by git, which will never be committed to revision control. This is very similar to the setup already used by hadoop-aws for integration testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
Chris Nauroth created HADOOP-12535:
--
Summary: Run FileSystem contract tests with hadoop-azure.
Key: HADOOP-12535
URL: https://issues.apache.org/jira/browse/HADOOP-12535
Project: Hadoop Common
Issue Type: Bug
Components: azure, test
Reporter: Chris Nauroth
Assignee: Duo Xu
This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
hadoop-azure/WASB. The contract tests define the expected semantics of the
{{FileSystem}}, so running these for hadoop-azure is likely to catch potential
problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12540) TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.
Chris Nauroth created HADOOP-12540:
--
Summary: TestAzureFileSystemInstrumentation#testClientErrorMetrics
fails intermittently due to assumption that a lease error will be thrown.
Key: HADOOP-12540
URL: https://issues.apache.org/jira/browse/HADOOP-12540
Project: Hadoop Common
Issue Type: Bug
Components: azure, test
Reporter: Chris Nauroth
Assignee: Gaurav Kanade
HADOOP-12508 changed the behavior of an Azure Storage lease violation during
deletes. It appears that
{{TestAzureFileSystemInstrumentation#testClientErrorMetrics}} is partly
dependent on the old behavior for simulating an error to be tracked by the
metrics system. I am seeing intermittent failures in this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12542) TestDNS fails on Windows after HADOOP-12437.
Chris Nauroth created HADOOP-12542: -- Summary: TestDNS fails on Windows after HADOOP-12437. Key: HADOOP-12542 URL: https://issues.apache.org/jira/browse/HADOOP-12542 Project: Hadoop Common Issue Type: Bug Components: net Reporter: Chris Nauroth Assignee: Chris Nauroth HADOOP-12437 added several new tests covering functionality of resolving host names based on an alternate network interface. These tests are failing on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12550) NativeIO#renameTo on Windows cannot replace an existing file at the destination.
Chris Nauroth created HADOOP-12550:
--
Summary: NativeIO#renameTo on Windows cannot replace an existing
file at the destination.
Key: HADOOP-12550
URL: https://issues.apache.org/jira/browse/HADOOP-12550
Project: Hadoop Common
Issue Type: Bug
Components: native
Environment: Windows
Reporter: Chris Nauroth
Assignee: Chris Nauroth
{{NativeIO#renameTo}} currently has different semantics on Linux vs. Windows if
a file already exists at the destination. On Linux, it's a passthrough to the
[rename|http://linux.die.net/man/2/rename] syscall, which will replace an
existing file at the destination. On Windows, it's a passthrough to
[MoveFile|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365239%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396],
which cannot replace an existing file at the destination and instead triggers
an error. The easiest way to observe this difference is to run the HDFS test
{{TestRollingUpgrade#testRollback}}. This fails on Windows due to a block
recovery after truncate trying to replace a block at an existing destination
path. This issue proposes to use
[MoveFileEx|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240(v=vs.85).aspx]
on Windows with the {{MOVEFILE_REPLACE_EXISTING}} flag.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12555) WASB to read credentials from a credential provider
Chris Nauroth created HADOOP-12555: -- Summary: WASB to read credentials from a credential provider Key: HADOOP-12555 URL: https://issues.apache.org/jira/browse/HADOOP-12555 Project: Hadoop Common Issue Type: Improvement Components: azure Reporter: Chris Nauroth Priority: Minor As HADOOP-12548 is going to do for s3, WASB should be able to read a password from a credential provider. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12557) Add information in BUILDING.txt about the need for the FINDBUGS_HOME environment variable for docs builds.
Chris Nauroth created HADOOP-12557:
--
Summary: Add information in BUILDING.txt about the need for the
FINDBUGS_HOME environment variable for docs builds.
Key: HADOOP-12557
URL: https://issues.apache.org/jira/browse/HADOOP-12557
Project: Hadoop Common
Issue Type: Improvement
Components: documentation
Reporter: Chris Nauroth
Priority: Minor
BUILDING.txt mentions Findbugs 1.3.9 as a requirement, but it doesn't mention
that the {{FINDBUGS_HOME}} environment variable must point to the base
directory of the Findbugs installation when running builds with {{-Pdocs}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-12574) Multiple Xmx Parameters to Java Process
[
https://issues.apache.org/jira/browse/HADOOP-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-12574.
Resolution: Duplicate
Resolving as duplicate of HADOOP-9902.
> Multiple Xmx Parameters to Java Process
> ---
>
> Key: HADOOP-12574
> URL: https://issues.apache.org/jira/browse/HADOOP-12574
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Jonathan Ellithorpe
>
> When launching hadoop, the setting of environment variable HADOOP_HEAPMAX
> results in multiple Xmx parameters being passed to the java application
> launcher. This is because a default Xmx setting included in the
> HADOOP_CLIENT_OPTS in etc/hadoop/hadoop-env.sh, line 61:
> {noformat}
> # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
> export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
> {noformat}
> is merged into the variable HADOOP_OPTS in bin/hadoop, line 160:
> {noformat}
> # Always respect HADOOP_OPTS and HADOOP_CLIENT_OPTS
> HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
> {noformat}
> Orthogonally, the HADOOP_HEAPMAX environment variable triggers the setting of
> JAVA_HEAP_MAX in hadoop-config.sh, line 192:
> {noformat}
> # check envvars which might override default args
> if [ "$HADOOP_HEAPSIZE" != "" ]; then
> #echo "run with heapsize $HADOOP_HEAPSIZE"
> JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
> #echo $JAVA_HEAP_MAX
> fi
> {noformat}
> Which is ultimately passed to the java application launcher in bin/hadoop,
> line 166:
> {noformat}
> exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"
> {noformat}
> The result: two Xmx settings passed to the Java application launcher. Which
> one is respected depends on the JVM, but for HotSpot the rightmost setting is
> respected, and so the HADOOP_HEAPMAX environment variable has no effect and
> the default setting in HADOOP_CLIENT_OPTS wins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HADOOP-12572) Update Hadoop's lz4 to r131
[
https://issues.apache.org/jira/browse/HADOOP-12572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reopened HADOOP-12572:
I consistently see segfaults when running {{TestLz4CompressorDecompressor}} on
both Linux and Windows after this patch. I'd like to revert it later today
while we figure out what is wrong. It looks like the tests didn't run during
pre-commit, so we might also need to check what Yetus is doing with native-only
patches. Maybe we need to update the logic in the personality file.
> Update Hadoop's lz4 to r131
> ---
>
> Key: HADOOP-12572
> URL: https://issues.apache.org/jira/browse/HADOOP-12572
> Project: Hadoop Common
> Issue Type: Improvement
> Components: native
>Reporter: Kevin Bowling
>Assignee: Kevin Bowling
> Fix For: 2.8.0
>
> Attachments: HADOOP-12572.001.patch
>
>
> Update hadoop's native lz4 copy to r131 versus the current r123 copy.
> Release notes for the versions are at https://github.com/Cyan4973/lz4/releases
> Noteworthy changes:
> * 30% performance improvement for clang
> * GCC 4.9+ bug fixes
> * New 32/64 bits, little/big endian and strict/efficient align detection
> routines (internal)
> * Small decompression speed improvement
> This is my first Hadoop patch, review/feedback appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12600) FileContext should be annotated as a Stable interface.
Chris Nauroth created HADOOP-12600:
--
Summary: FileContext should be annotated as a Stable interface.
Key: HADOOP-12600
URL: https://issues.apache.org/jira/browse/HADOOP-12600
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Chris Nauroth
Priority: Trivial
The {{FileContext}} class currently is annotated as {{Evolving}}. However, at
this point we really need to treat it as a {{Stable}} interface.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HADOOP-11505) Various native parts use bswap incorrectly and unportably
[ https://issues.apache.org/jira/browse/HADOOP-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reopened HADOOP-11505: I reverted this, because it broke the Windows build. The hadoop.dll build does not use CMake, so there was no hadoop_endian.h header present on Windows at compile time. I'll review this patch more closely and give a recommendation on what to do for Windows. > Various native parts use bswap incorrectly and unportably > - > > Key: HADOOP-11505 > URL: https://issues.apache.org/jira/browse/HADOOP-11505 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: HADOOP-11505.001.patch, HADOOP-11505.003.patch, > HADOOP-11505.004.patch, HADOOP-11505.005.patch, HADOOP-11505.006.patch > > > hadoop-mapreduce-client-nativetask fails to use x86 optimizations in some > cases. Also, on some alternate, non-x86, non-ARM architectures the generated > code is incorrect. Thanks to Steve Loughran and Edward Nevill for finding > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12643) Allow getmerge command to skip writing the CRC file.
Chris Nauroth created HADOOP-12643:
--
Summary: Allow getmerge command to skip writing the CRC file.
Key: HADOOP-12643
URL: https://issues.apache.org/jira/browse/HADOOP-12643
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Chris Nauroth
The {{-getmerge}} shell command always write a local CRC file. It's prefixed
with a '.', so it's a hidden file and often goes unnoticed. This behavior
differs from {{-get}}, which skips writing the CRC file by default, but allows
the user to opt in to writing a CRC file by passing the {{-crc}} argument.
This issue proposes adding support to {{-getmerge}} to skip writing the CRC
file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-12707) key of FileSystem inner class Cache contains UGI.hascode which uses the defualt hascode method, leading to the memory leak
[
https://issues.apache.org/jira/browse/HADOOP-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-12707.
Resolution: Won't Fix
Hello, [~haitao-tony]. Thank you for filing the issue, but the current
implementation of {{UserGroupInformation#hashCode}} and
{{UserGroupInformation#equals}} is by design. Please see issue HADOOP-6670 if
you're interested in more background.
I have 2 suggestions to offer for the memory leak problem that you noticed:
# Consider using the {{FileSystem#closeAllForUGI}} method, which would close
all cached instances for a user.
# If it's not possible to change code, then consider disabling the cache by
setting configuration property {{fs.hdfs.impl.disable.cache}} to {{true}} in
core-site.xml. This would bypass the cache completely, which might be
appropriate depending on your application.
> key of FileSystem inner class Cache contains UGI.hascode which uses the
> defualt hascode method, leading to the memory leak
> --
>
> Key: HADOOP-12707
> URL: https://issues.apache.org/jira/browse/HADOOP-12707
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1
>Reporter: sunhaitao
>Assignee: sunhaitao
>
> FileSystem.get(conf) method,By default it will get the fs object from
> CACHE,But the key of the CACHE constains ugi.hashCode, which uses the
> default hascode method of subject instead of the hascode method overwritten
> by subject.
>@Override
> public int hashCode() {
> return (scheme + authority).hashCode() + ugi.hashCode() + (int)unique;
> }
> In this case, even if same user, if the calll FileSystem.get(conf) twice, two
> different key will be created. In long duartion, this will lead to memory
> leak.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-12740) Hadoop copyFromLocal command fails with NoSuchMethodError on startCopyFromBlob
[ https://issues.apache.org/jira/browse/HADOOP-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-12740. Resolution: Not A Problem Thank you, Maik. I'll close this issue. > Hadoop copyFromLocal command fails with NoSuchMethodError on startCopyFromBlob > -- > > Key: HADOOP-12740 > URL: https://issues.apache.org/jira/browse/HADOOP-12740 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Maik Groenewegen > > There was a change in the azure-storage version from 3.x to 4.x where the > method startCopyFromBlob was changed to startCopy. > https://github.com/Azure/azure-storage-java/blob/d108c110ea042df5c898a461007b91818fa98f15/BreakingChanges.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12795) KMS does not log detailed stack trace for unexpected errors.
Chris Nauroth created HADOOP-12795: -- Summary: KMS does not log detailed stack trace for unexpected errors. Key: HADOOP-12795 URL: https://issues.apache.org/jira/browse/HADOOP-12795 Project: Hadoop Common Issue Type: Bug Components: kms Reporter: Chris Nauroth If the KMS server encounters an unexpected error resulting in an HTTP 500 response, it does not log the stack trace. This makes it difficult to troubleshoot. The client side exception cannot provide further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12801) TestS3Credentials#noSecretShouldThrow and TestS3Credentials#noAccessIdShouldThrow fail.
Chris Nauroth created HADOOP-12801:
--
Summary: TestS3Credentials#noSecretShouldThrow and
TestS3Credentials#noAccessIdShouldThrow fail.
Key: HADOOP-12801
URL: https://issues.apache.org/jira/browse/HADOOP-12801
Project: Hadoop Common
Issue Type: Bug
Reporter: Chris Nauroth
While reviewing another patch, I noticed that
{{TestS3Credentials#noSecretShouldThrow}} and
{{TestS3Credentials#noAccessIdShouldThrow}} are failing. See comments for full
stack traces.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12815) TestS3ContractRootDir#testRmEmptyRootDirNonRecursive and TestS3ContractRootDir#testRmRootRecursive fail on branch-2.
Chris Nauroth created HADOOP-12815: -- Summary: TestS3ContractRootDir#testRmEmptyRootDirNonRecursive and TestS3ContractRootDir#testRmRootRecursive fail on branch-2. Key: HADOOP-12815 URL: https://issues.apache.org/jira/browse/HADOOP-12815 Project: Hadoop Common Issue Type: Bug Reporter: Chris Nauroth TestS3ContractRootDir#testRmEmptyRootDirNonRecursive and TestS3ContractRootDir#testRmRootRecursive fail on branch-2. The tests pass on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12836) Change hadoop-azure to support full semantics required for the root directory.
Chris Nauroth created HADOOP-12836:
--
Summary: Change hadoop-azure to support full semantics required
for the root directory.
Key: HADOOP-12836
URL: https://issues.apache.org/jira/browse/HADOOP-12836
Project: Hadoop Common
Issue Type: Improvement
Components: azure, test
Reporter: Chris Nauroth
HADOOP-12535 introduced usage of the file system contract tests in
hadoop-azure. During development of that patch, an attempt was made to execute
the root directory tests defined in {{AbstractContractRootDirectoryTest}}, but
some of the tests failed. This issue tracks the work of reattempting to add
those tests and implementing any fixes required in hadoop-azure to make those
tests pass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12899) External dist-layout-stitching script does not work correctly on Windows.
Chris Nauroth created HADOOP-12899: -- Summary: External dist-layout-stitching script does not work correctly on Windows. Key: HADOOP-12899 URL: https://issues.apache.org/jira/browse/HADOOP-12899 Project: Hadoop Common Issue Type: Bug Components: build Environment: Windows Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Blocker In HADOOP-12850, we pulled the dist-layout-stitching script out of hadoop-dist/pom.xml and into an external script. It appears this change is not working correctly on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12905) Clean up CHANGES.txt RAT exclusions from pom.xml files.
Chris Nauroth created HADOOP-12905: -- Summary: Clean up CHANGES.txt RAT exclusions from pom.xml files. Key: HADOOP-12905 URL: https://issues.apache.org/jira/browse/HADOOP-12905 Project: Hadoop Common Issue Type: Improvement Components: build Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Trivial Now that we have stopped tracking CHANGES.txt files, we can remove the RAT exclusions for those files from the pom.xml files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12915) shelldocs and releasedocmaker build steps do not work correctly on Windows.
Chris Nauroth created HADOOP-12915: -- Summary: shelldocs and releasedocmaker build steps do not work correctly on Windows. Key: HADOOP-12915 URL: https://issues.apache.org/jira/browse/HADOOP-12915 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Chris Nauroth In the pom.xml files, the calls to shelldocs and releasedocmaker use the exec-maven-plugin to invoke the scripts directly. On *nix, this works fine in cooperation with the bang lines in the scripts that tell it the script interpreter to use. The bang lines don't work on Windows though. Instead, exec-maven-plugin needs to specify bash as the executable and pass the script as the first argument. Beyond that, there seem to be further as-yet-undiagnosed compatibility issues within the scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-8848) hadoop-mapreduce-client-core fails compilation in Eclipse due to missing Avro-generated classes
Chris Nauroth created HADOOP-8848: - Summary: hadoop-mapreduce-client-core fails compilation in Eclipse due to missing Avro-generated classes Key: HADOOP-8848 URL: https://issues.apache.org/jira/browse/HADOOP-8848 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth After importing all of hadoop-common trunk into Eclipse with the m2e plugin, the Avro-generated classes in hadoop-mapreduce-client-core don't show up on Eclipse's classpath. This causes compilation errors for anything that depends on those classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8858) backport branch-trunk-win: HADOOP-8234 Enable user group mappings on Windows
Chris Nauroth created HADOOP-8858: - Summary: backport branch-trunk-win: HADOOP-8234 Enable user group mappings on Windows Key: HADOOP-8858 URL: https://issues.apache.org/jira/browse/HADOOP-8858 Project: Hadoop Common Issue Type: Task Components: security Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Backport the code for HADOOP-8234 to enable user group mappings on Windows. This code had been committed to branch-1-win. This issue tracks backporting to branch-trunk-win, in preparation for merging all the way to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8871) FileUtil.symLink: if shell command fails, logging message does not correctly print command attempted
Chris Nauroth created HADOOP-8871: - Summary: FileUtil.symLink: if shell command fails, logging message does not correctly print command attempted Key: HADOOP-8871 URL: https://issues.apache.org/jira/browse/HADOOP-8871 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1-win Reporter: Chris Nauroth When the shell command call to create a symlink fails, we attempt to log the details. We log cmd, which is a String[], and Java prints it using the default Object.toString implementation. We need to change this to print the contents of the array so that we see the actual command executed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8858) backport branch-trunk-win: HADOOP-8234 Enable user group mappings on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-8858. --- Resolution: Invalid Discarding this Jira, because this functionality will go to trunk in a different form. > backport branch-trunk-win: HADOOP-8234 Enable user group mappings on Windows > > > Key: HADOOP-8858 > URL: https://issues.apache.org/jira/browse/HADOOP-8858 > Project: Hadoop Common > Issue Type: Task > Components: security >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-8858-branch-trunk-win.patch > > > Backport the code for HADOOP-8234 to enable user group mappings on Windows. > This code had been committed to branch-1-win. This issue tracks backporting > to branch-trunk-win, in preparation for merging all the way to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8909) Maven protoc calls must not depend on external sh script
Chris Nauroth created HADOOP-8909: - Summary: Maven protoc calls must not depend on external sh script Key: HADOOP-8909 URL: https://issues.apache.org/jira/browse/HADOOP-8909 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Currently, several pom.xml files rely on external shell scripting to call protoc. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8924) Hadoop Common creating version annotation must not depend on sh
Chris Nauroth created HADOOP-8924: - Summary: Hadoop Common creating version annotation must not depend on sh Key: HADOOP-8924 URL: https://issues.apache.org/jira/browse/HADOOP-8924 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8933) test-patch.sh fails erroneously on platforms that can't build native
Chris Nauroth created HADOOP-8933: - Summary: test-patch.sh fails erroneously on platforms that can't build native Key: HADOOP-8933 URL: https://issues.apache.org/jira/browse/HADOOP-8933 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth If a developer is working on a platform that can't build native (like OS X right now), then test-patch.sh will report the patch as a failure due to "The patch appears to cause the build to fail." This is incorrect, because the developer's patch didn't cause the build to fail. Adding an extra optional flag to test-patch.sh would help developers on these platforms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8945) winutils: merge codebase from branch-1-win to branch-trunk-win
Chris Nauroth created HADOOP-8945: - Summary: winutils: merge codebase from branch-1-win to branch-trunk-win Key: HADOOP-8945 URL: https://issues.apache.org/jira/browse/HADOOP-8945 Project: Hadoop Common Issue Type: Task Components: native Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth The winutils codebase on branch-1-win provides Windows compatibility with external shell command calls. This codebase needs to be merged to branch-trunk-win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8946) winutils: compile codebase during Maven build on branch-trunk-win
Chris Nauroth created HADOOP-8946: - Summary: winutils: compile codebase during Maven build on branch-trunk-win Key: HADOOP-8946 URL: https://issues.apache.org/jira/browse/HADOOP-8946 Project: Hadoop Common Issue Type: Task Components: build, native Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth The winutils codebase builds on branch-1-win through the Ant build process. We need to port that into the Maven build process on branch-trunk-win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8947) need basic merge of branch-1-win to branch-trunk-win to enable initial test pass
Chris Nauroth created HADOOP-8947: - Summary: need basic merge of branch-1-win to branch-trunk-win to enable initial test pass Key: HADOOP-8947 URL: https://issues.apache.org/jira/browse/HADOOP-8947 Project: Hadoop Common Issue Type: Task Components: fs, util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth We need a basic merge of past work contributed on branch-1-win to branch-trunk-win to enable further testing on a trunk-derived version of the codebase on Windows. Primarily, this involves merging in the changes to FileUtil and Shell to enable calls to winutils. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8948) TestFileUtil.testGetDU fails on Windows due to incorrect assumption of line separator
Chris Nauroth created HADOOP-8948: - Summary: TestFileUtil.testGetDU fails on Windows due to incorrect assumption of line separator Key: HADOOP-8948 URL: https://issues.apache.org/jira/browse/HADOOP-8948 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0, trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth The test asserts 8 bytes of disk usage, based on 2 files, each containing a single line of 3 characters followed by line terminator. This is fine for Unix line endings (1 additional byte) but incorrect for Windows line endings (2 additional bytes). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8949) FileUtil.CygPathCommand is dead code on branch-1-win and branch-trunk-win
Chris Nauroth created HADOOP-8949: - Summary: FileUtil.CygPathCommand is dead code on branch-1-win and branch-trunk-win Key: HADOOP-8949 URL: https://issues.apache.org/jira/browse/HADOOP-8949 Project: Hadoop Common Issue Type: Task Components: fs Affects Versions: 1-win, trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth We are converting to use winutils for Windows compatibility on external shell commands. With those changes in place, the FileUtil.CygPathCommand inner class is no longer called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8953) Shell PathData parsing failures on Windows
Chris Nauroth created HADOOP-8953: - Summary: Shell PathData parsing failures on Windows Key: HADOOP-8953 URL: https://issues.apache.org/jira/browse/HADOOP-8953 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Several test suites fail on Windows, apparently due to Windows-specific path parsing bugs in PathData. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8954) "stat" executable not found on Windows
Chris Nauroth created HADOOP-8954: - Summary: "stat" executable not found on Windows Key: HADOOP-8954 URL: https://issues.apache.org/jira/browse/HADOOP-8954 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Hadoop Common attempts to call the "stat" executable, which isn't available on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8955) "chmod" executable not found on Windows
Chris Nauroth created HADOOP-8955: - Summary: "chmod" executable not found on Windows Key: HADOOP-8955 URL: https://issues.apache.org/jira/browse/HADOOP-8955 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Hadoop Common attempts to call the "chmod" executable, which isn't available on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8956) FileSystem.primitiveMkdir failures on Windows cause multiple test suites to fail
Chris Nauroth created HADOOP-8956: - Summary: FileSystem.primitiveMkdir failures on Windows cause multiple test suites to fail Key: HADOOP-8956 URL: https://issues.apache.org/jira/browse/HADOOP-8956 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Multiple test suites fail on Windows in calls to FileSystem.primitiveMkdir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8957) FileContext.getFileContext causes multiple test failures on Windows due to InvocationTargetException
Chris Nauroth created HADOOP-8957: - Summary: FileContext.getFileContext causes multiple test failures on Windows due to InvocationTargetException Key: HADOOP-8957 URL: https://issues.apache.org/jira/browse/HADOOP-8957 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth This appears to be a problem with parsing a Windows-specific path, ultimately throwing InvocationTargetException from AbstractFileSystem.newInstance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
Chris Nauroth created HADOOP-8958: - Summary: ViewFs:Non absolute mount name failures when running multiple tests on Windows Key: HADOOP-8958 URL: https://issues.apache.org/jira/browse/HADOOP-8958 Project: Hadoop Common Issue Type: Bug Components: viewfs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8959) TestUserGroupInformation fails on Windows due to "id" executable not found
Chris Nauroth created HADOOP-8959: - Summary: TestUserGroupInformation fails on Windows due to "id" executable not found Key: HADOOP-8959 URL: https://issues.apache.org/jira/browse/HADOOP-8959 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth TestUserGroupInformation.testGetServerSideGroups contains a hard-coded call to "id", which isn't present on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8960) TestMetricsServlet fails on Windows
Chris Nauroth created HADOOP-8960: - Summary: TestMetricsServlet fails on Windows Key: HADOOP-8960 URL: https://issues.apache.org/jira/browse/HADOOP-8960 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth This appears to be caused by an incorrect assumption of line terminator in an assertion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8961) GenericOptionsParser URI parsing failure on Windows
Chris Nauroth created HADOOP-8961: - Summary: GenericOptionsParser URI parsing failure on Windows Key: HADOOP-8961 URL: https://issues.apache.org/jira/browse/HADOOP-8961 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth This appears to be a mishandling of a path that starts with a drive specification on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
Chris Nauroth created HADOOP-8973: - Summary: DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8974) TestDFVariations fails on Windows
Chris Nauroth created HADOOP-8974: - Summary: TestDFVariations fails on Windows Key: HADOOP-8974 URL: https://issues.apache.org/jira/browse/HADOOP-8974 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth The test fails on Windows. This may be related to code ported in to DF.java from branch-1-win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8975) TestFileContextResolveAfs fails on Windows
Chris Nauroth created HADOOP-8975: - Summary: TestFileContextResolveAfs fails on Windows Key: HADOOP-8975 URL: https://issues.apache.org/jira/browse/HADOOP-8975 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth This appears to be a Windows-specific path parsing issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8976) TestFSMainOperationsLocalFileSystem fails on Windows
Chris Nauroth created HADOOP-8976: - Summary: TestFSMainOperationsLocalFileSystem fails on Windows Key: HADOOP-8976 URL: https://issues.apache.org/jira/browse/HADOOP-8976 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth There are multiple failures in this test suite, and there appear to be many different reasons. This needs further investigation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8977) multiple FsShell test failures on Windows
Chris Nauroth created HADOOP-8977: - Summary: multiple FsShell test failures on Windows Key: HADOOP-8977 URL: https://issues.apache.org/jira/browse/HADOOP-8977 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Multiple FsShell-related tests fail on Windows. Commands are returning non-zero exit status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8978) TestTrash fails on Windows
Chris Nauroth created HADOOP-8978: - Summary: TestTrash fails on Windows Key: HADOOP-8978 URL: https://issues.apache.org/jira/browse/HADOOP-8978 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth The tests assert that a file is found in trash after deleting, but it's not found when run on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8979) TestHttpServer fails on Windows
Chris Nauroth created HADOOP-8979: - Summary: TestHttpServer fails on Windows Key: HADOOP-8979 URL: https://issues.apache.org/jira/browse/HADOOP-8979 Project: Hadoop Common Issue Type: Bug Components: net Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth The test asserts status of binding an HTTP server to a port. Perhaps there is some platform-specific difference in handling this on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8980) TestRPC fails on Windows
Chris Nauroth created HADOOP-8980: - Summary: TestRPC fails on Windows Key: HADOOP-8980 URL: https://issues.apache.org/jira/browse/HADOOP-8980 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth This failure may indicate a difference in socket handling on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
Chris Nauroth created HADOOP-8981: - Summary: TestMetricsSystemImpl fails on Windows Key: HADOOP-8981 URL: https://issues.apache.org/jira/browse/HADOOP-8981 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8982) TestSocketIOWithTimeout fails on Windows
Chris Nauroth created HADOOP-8982: - Summary: TestSocketIOWithTimeout fails on Windows Key: HADOOP-8982 URL: https://issues.apache.org/jira/browse/HADOOP-8982 Project: Hadoop Common Issue Type: Bug Components: net Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth This is a possible race condition or difference in socket handling on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9011) saveVersion.py does not include branch in version annotation
Chris Nauroth created HADOOP-9011: - Summary: saveVersion.py does not include branch in version annotation Key: HADOOP-9011 URL: https://issues.apache.org/jira/browse/HADOOP-9011 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth HADOOP-8924 created saveVersion.py on branch-trunk-win. Unlike saveVersion.sh on trunk, it did not include the branch attribute in the version annotation. This causes errors at runtime for anything that tries to read the annotation via VersionInfo. This also causes a unit test failure in TestYarnVersionInfo. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9043) winutils can create unusable symlinks
Chris Nauroth created HADOOP-9043: - Summary: winutils can create unusable symlinks Key: HADOOP-9043 URL: https://issues.apache.org/jira/browse/HADOOP-9043 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 1-win, trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth In general, the winutils symlink command rejects attempts to create symlinks targeting a destination file that does not exist. However, if given a symlink destination with forward slashes pointing at a file that does exist, then it creates the symlink with the forward slashes, and then attempts to open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9081) Port TestWinUtils from branch-1-win to branch-trunk-win
Chris Nauroth created HADOOP-9081: - Summary: Port TestWinUtils from branch-1-win to branch-trunk-win Key: HADOOP-9081 URL: https://issues.apache.org/jira/browse/HADOOP-9081 Project: Hadoop Common Issue Type: Improvement Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth branch-1-win has a test suite named TestWinUtils to cover winutils.exe functionality. We need to port this to branch-trunk-win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9102) winutils task isAlive does not return a non-zero exit code if the requested task is not alive
Chris Nauroth created HADOOP-9102: - Summary: winutils task isAlive does not return a non-zero exit code if the requested task is not alive Key: HADOOP-9102 URL: https://issues.apache.org/jira/browse/HADOOP-9102 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Work on YARN-233 noted that winutils task isAlive does not return a non-zero exit code if the requested task is not alive. This is inconsistent with the equivalent check on Unix, kill -0, which uses a non-zero exit code to communicate status. By changing winutils task isAlive to return a non-zero exit code, we can make the error handling code on the Java side consistent, avoiding the need for additional if (WINDOWS) checks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9110) winutils ls off-by-one error indexing MONTHS array can cause access violation
Chris Nauroth created HADOOP-9110: - Summary: winutils ls off-by-one error indexing MONTHS array can cause access violation Key: HADOOP-9110 URL: https://issues.apache.org/jira/browse/HADOOP-9110 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 1-win, trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth In ls.c, the LsPrintLine function uses the wMonth field of a SYSTEMTIME struct to index into MONTHS, an array of 12 elements containing string representations of the months. The wMonth field is 1-based (1=January), but indexing into an array is zero-based. This gives the wrong month, and since we just crossed into month 12, it also has started causing an access violation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9129) ViewFs does not validate internal names in the mount table
Chris Nauroth created HADOOP-9129:
-
Summary: ViewFs does not validate internal names in the mount table
Key: HADOOP-9129
URL: https://issues.apache.org/jira/browse/HADOOP-9129
Project: Hadoop Common
Issue Type: Bug
Components: viewfs
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Currently, there is no explicit validation of {{ViewFs}} internal names in the
mount table during initialization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9130) TestMapReduceChildJVM fails in branch-trunk-win
Chris Nauroth created HADOOP-9130:
-
Summary: TestMapReduceChildJVM fails in branch-trunk-win
Key: HADOOP-9130
URL: https://issues.apache.org/jira/browse/HADOOP-9130
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
The YARN-233 patch for getting YARN working on Windows forgot to include a
corresponding change in {{TestMapReduceChildJVM}}, so the test is failing now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9131) TestLocalFileSystem#testListStatusWithColons cannot run on Windows
Chris Nauroth created HADOOP-9131:
-
Summary: TestLocalFileSystem#testListStatusWithColons cannot run
on Windows
Key: HADOOP-9131
URL: https://issues.apache.org/jira/browse/HADOOP-9131
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
HADOOP-8962 added a new test, {{TestLocalFileSystem#testListStatusWithColons}},
covering the case of files that contain ':'. This test cannot pass on Windows,
because on Windows, the local file system does not support ':' in file names.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9144) FindBugs reports new warnings in branch-trunk-win
Chris Nauroth created HADOOP-9144: - Summary: FindBugs reports new warnings in branch-trunk-win Key: HADOOP-9144 URL: https://issues.apache.org/jira/browse/HADOOP-9144 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Testing the merge from branch-trunk-win to trunk, we saw some new FindBugs warnings that need to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8976) TestFSMainOperationsLocalFileSystem fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-8976. --- Resolution: Fixed Fix Version/s: trunk-win This test has been resolved by the other issues mentioned earlier. > TestFSMainOperationsLocalFileSystem fails on Windows > > > Key: HADOOP-8976 > URL: https://issues.apache.org/jira/browse/HADOOP-8976 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: trunk-win > > > There are multiple failures in this test suite, and there appear to be many > different reasons. This needs further investigation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9202) test-patch.sh fails during mvn eclipse:eclipse if patch adds a new module to the build
Chris Nauroth created HADOOP-9202: - Summary: test-patch.sh fails during mvn eclipse:eclipse if patch adds a new module to the build Key: HADOOP-9202 URL: https://issues.apache.org/jira/browse/HADOOP-9202 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth test-patch.sh tries running mvn eclipse:eclipse after applying the patch. It runs this before running mvn install. The mvn eclipse:eclipse command doesn't actually build the code, so if the patch in question is adding a whole new module, then any other modules dependent on finding it in the reactor will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9207) version info source checksum does not include all source files
Chris Nauroth created HADOOP-9207: - Summary: version info source checksum does not include all source files Key: HADOOP-9207 URL: https://issues.apache.org/jira/browse/HADOOP-9207 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Reporter: Chris Nauroth The build process takes an MD5 checksum of the source files in Common and YARN. The HDFS version info command prints the checksum from Common. The YARN version info command prints the checksum from YARN. This is incomplete in that the HDFS source code is never included in the checksum, and 2 different YARN builds with the same YARN code but different Common code would have the same checksum. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9232) JniBasedUnixGroupsMappingWithFallback fails on Windows with UnsatisfiedLinkError
Chris Nauroth created HADOOP-9232:
-
Summary: JniBasedUnixGroupsMappingWithFallback fails on Windows
with UnsatisfiedLinkError
Key: HADOOP-9232
URL: https://issues.apache.org/jira/browse/HADOOP-9232
Project: Hadoop Common
Issue Type: Bug
Components: native, security
Affects Versions: trunk-win
Reporter: Chris Nauroth
{{JniBasedUnixGroupsMapping}} calls native code which isn't implemented
properly for Windows, causing {{UnsatisfiedLinkError}}. The fallback logic in
{{JniBasedUnixGroupsMappingWithFallback}} works by checking if the native code
is loaded during startup. In this case, hadoop.dll is present and loaded, but
it doesn't contain the right code. There will be no attempt to fallback to
{{ShellBasedUnixGroupsMapping}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9249) hadoop-maven-plugins version-info goal causes build failure when running with Clover
Chris Nauroth created HADOOP-9249: - Summary: hadoop-maven-plugins version-info goal causes build failure when running with Clover Key: HADOOP-9249 URL: https://issues.apache.org/jira/browse/HADOOP-9249 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Running Maven with the -Pclover option for code coverage causes the build to fail because of not finding a Clover class while running hadoop-maven-plugins version-info. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9263) findbugs reports extra warnings on branch-1
Chris Nauroth created HADOOP-9263:
-
Summary: findbugs reports extra warnings on branch-1
Key: HADOOP-9263
URL: https://issues.apache.org/jira/browse/HADOOP-9263
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 1.1.2
Reporter: Chris Nauroth
Currently on branch-1, src/test/test-patch.properties defines
{{OK_FINDBUGS_WARNINGS=211}}, but findbugs actually finds 225 warnings. This
means that any patch will get -1 from ant test-patch due to 14 new findbugs
warnings, even though the warnings are unrelated to that patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9264) port change to use Java untar API on Windows from branch-1-win to trunk
Chris Nauroth created HADOOP-9264: - Summary: port change to use Java untar API on Windows from branch-1-win to trunk Key: HADOOP-9264 URL: https://issues.apache.org/jira/browse/HADOOP-9264 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 3.0.0 Reporter: Chris Nauroth HADOOP-8847 originally introduced this change on branch-1-win. HADOOP-9081 ported the change to branch-trunk-win. This should be simple to port to trunk, which would simplify the merge and test activity happening on HADOOP-8562. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9266) correct release audit warnings on branch-trunk-win
Chris Nauroth created HADOOP-9266: - Summary: correct release audit warnings on branch-trunk-win Key: HADOOP-9266 URL: https://issues.apache.org/jira/browse/HADOOP-9266 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth There are several new release audit warnings reported on branch-trunk-win for files introduced during the Windows compatibility work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9270) FileUtil#unTarUsingTar on branch-trunk-win contains an incorrect comment
Chris Nauroth created HADOOP-9270: - Summary: FileUtil#unTarUsingTar on branch-trunk-win contains an incorrect comment Key: HADOOP-9270 URL: https://issues.apache.org/jira/browse/HADOOP-9270 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth HADOOP-9081 changed Windows handling of untar to use pure Java. There had been a comment in that code describing why we need to pass a particular flag to the external tar command on Windows. Now that Windows doesn't call the external tar command, the comment is no longer relevant. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9271) revert Python build scripts from branch-trunk-win
Chris Nauroth created HADOOP-9271: - Summary: revert Python build scripts from branch-trunk-win Key: HADOOP-9271 URL: https://issues.apache.org/jira/browse/HADOOP-9271 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth On branch-trunk-win, some work was committed to convert custom build scripting to Python as a cross-platform solution to support builds on both Linux and Windows. The community has since rejected this solution. This issue tracks removal of the Python build code from branch-trunk-win. This will restore branch-trunk-win to the same state as trunk, which uses shell for custom build scripting. This means that developers working on branch-trunk-win will once again need sh.exe on the PATH to build the distribution. This will be a temporary state. A subsequent jira will add build scripts specific to Windows, using a language more common in a Windows development environment. The community has approved minimal use of multiple platform-specific scripting languages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9278) TestHarFileSystemBasics fails on Windows due to invalid HAR URI and file handle leak
Chris Nauroth created HADOOP-9278: - Summary: TestHarFileSystemBasics fails on Windows due to invalid HAR URI and file handle leak Key: HADOOP-9278 URL: https://issues.apache.org/jira/browse/HADOOP-9278 Project: Hadoop Common Issue Type: Bug Components: fs, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth We need to change the tests to use valid HAR URIs and fix the file handle leak. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9294) GetGroupsTestBase assumes output with a platform-specific line ending, which causes test failures on Windows
Chris Nauroth created HADOOP-9294: - Summary: GetGroupsTestBase assumes output with a platform-specific line ending, which causes test failures on Windows Key: HADOOP-9294 URL: https://issues.apache.org/jira/browse/HADOOP-9294 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9309) test failures on Windows due UnsatisfiedLinkError in NativeCodeLoader#buildSupportsSnappy
Chris Nauroth created HADOOP-9309:
-
Summary: test failures on Windows due UnsatisfiedLinkError in
NativeCodeLoader#buildSupportsSnappy
Key: HADOOP-9309
URL: https://issues.apache.org/jira/browse/HADOOP-9309
Project: Hadoop Common
Issue Type: Bug
Components: native, util
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Checking for Snappy support calls native method
{{NativeCodeLoader#buildSupportsSnappy}}. This method has not been implemented
for Windows in hadoop.dll, so it throws {{UnsatisfiedLinkError}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9312) JniBasedUnixGroupsMapping#getGroupForUser can potentially leak memory
Chris Nauroth created HADOOP-9312: - Summary: JniBasedUnixGroupsMapping#getGroupForUser can potentially leak memory Key: HADOOP-9312 URL: https://issues.apache.org/jira/browse/HADOOP-9312 Project: Hadoop Common Issue Type: Bug Components: native, security Affects Versions: 3.0.0, trunk-win Reporter: Chris Nauroth This method lazily initializes a static variable to contain an empty array of groups. If multiple threads call the method before the variable has been initialized, then there is potential for a race condition. This would cause multiple allocations of the array and leaked memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9347) add instructions to BUILDING.txt describing how to build on Windows
Chris Nauroth created HADOOP-9347: - Summary: add instructions to BUILDING.txt describing how to build on Windows Key: HADOOP-9347 URL: https://issues.apache.org/jira/browse/HADOOP-9347 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Add documentation to BUILDING.txt describing dependencies and instructions for building on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9348) address TODO in winutils to add more command line usage and examples
Chris Nauroth created HADOOP-9348: - Summary: address TODO in winutils to add more command line usage and examples Key: HADOOP-9348 URL: https://issues.apache.org/jira/browse/HADOOP-9348 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: trunk-win Reporter: Chris Nauroth A TODO was marked in hadoop-common-project/hadoop-common/src/main/winutils/main.c to add more command line usage and examples. This issue tracks resolution of this TODO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9351) Hadoop daemon startup scripts cause duplication of command line arguments
Chris Nauroth created HADOOP-9351: - Summary: Hadoop daemon startup scripts cause duplication of command line arguments Key: HADOOP-9351 URL: https://issues.apache.org/jira/browse/HADOOP-9351 Project: Hadoop Common Issue Type: Bug Components: scripts Reporter: Chris Nauroth Fix For: 3.0.0 Command line arguments such as -Dhadoop.log.dir are appearing multiple times when launching Hadoop daemons. This can cause confusion for an operator looking at the process table, especially if there are different values for multiple occurrences of the same argument. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9354) native.sln missing license header
Chris Nauroth created HADOOP-9354: - Summary: native.sln missing license header Key: HADOOP-9354 URL: https://issues.apache.org/jira/browse/HADOOP-9354 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Trivial We need to add the license header to native.sln. winutils.sln already has it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9356) remove remaining references to cygwin/cygpath from scripts
Chris Nauroth created HADOOP-9356: - Summary: remove remaining references to cygwin/cygpath from scripts Key: HADOOP-9356 URL: https://issues.apache.org/jira/browse/HADOOP-9356 Project: Hadoop Common Issue Type: Improvement Components: build, scripts Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth branch-trunk-win still contains a few references to Cygwin and the cygpath command that need to be removed now that they are no longer needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9390) Tests fail when run as root.
Chris Nauroth created HADOOP-9390:
-
Summary: Tests fail when run as root.
Key: HADOOP-9390
URL: https://issues.apache.org/jira/browse/HADOOP-9390
Project: Hadoop Common
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Fix For: 3.0.0
There is at least one test, {{TestDiskChecker}}, that fails when running as
root. The test assumes that setting file permissions can make a file
inaccessible, without considering the possibility that root can access
everything.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9391) Refactor Shell methods to accept File instead of String arguments.
Chris Nauroth created HADOOP-9391:
-
Summary: Refactor Shell methods to accept File instead of String
arguments.
Key: HADOOP-9391
URL: https://issues.apache.org/jira/browse/HADOOP-9391
Project: Hadoop Common
Issue Type: Improvement
Reporter: Chris Nauroth
Assignee: Ivan Mitic
Changing these methods to accept {{File}} instead of {{String}} for file paths
would reduce the likelihood of subtle string manipulation bugs when handling
file paths in calling code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9390) Tests fail when run as root/Administrator.
[
https://issues.apache.org/jira/browse/HADOOP-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-9390.
---
Resolution: Won't Fix
Assignee: Chris Nauroth
There has been no additional feedback, so I'm resolving this as won't fix.
> Tests fail when run as root/Administrator.
> --
>
> Key: HADOOP-9390
> URL: https://issues.apache.org/jira/browse/HADOOP-9390
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 3.0.0
>
>
> There is at least one test, {{TestDiskChecker}}, that fails when running as
> root on Linux or Administrator on Windows. The test assumes that setting
> file permissions can make a file inaccessible, without considering the
> possibility that root can access everything.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9431) TestSecurityUtil#testLocalHostNameForNullOrWild on systems where hostname contains capital letters
Chris Nauroth created HADOOP-9431:
-
Summary: TestSecurityUtil#testLocalHostNameForNullOrWild on
systems where hostname contains capital letters
Key: HADOOP-9431
URL: https://issues.apache.org/jira/browse/HADOOP-9431
Project: Hadoop Common
Issue Type: Bug
Components: security, test
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
{{TestSecurityUtil#testLocalHostNameForNullOrWild}} contains assertions about
the principal returned for a null or wildcard hostname. The logic in
{{SecurityUtil#getServerPrincipal}} converts the local hostname to lower-case.
The test method does not perform the same conversion though, so the test passes
on systems with a hostname containing no upper-case letters, but it fails if
the hostname does contain upper-case letters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9433) TestLocalFileSystem#testHasFileDescriptor leaks file handle
Chris Nauroth created HADOOP-9433: - Summary: TestLocalFileSystem#testHasFileDescriptor leaks file handle Key: HADOOP-9433 URL: https://issues.apache.org/jira/browse/HADOOP-9433 Project: Hadoop Common Issue Type: Bug Components: fs, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth The test does not close a file handle. This can cause subsequent tests in the suite to fail due to failure to delete the test working directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9437) TestNativeIO#testRenameTo fails on Windows due to assumption that POSIX errno is embedded in NativeIOException
Chris Nauroth created HADOOP-9437:
-
Summary: TestNativeIO#testRenameTo fails on Windows due to
assumption that POSIX errno is embedded in NativeIOException
Key: HADOOP-9437
URL: https://issues.apache.org/jira/browse/HADOOP-9437
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
HDFS-4428 added a detailed error message for failures to rename files by
embedding the POSIX errno in the {{NativeIOException}}. On Windows, the
mapping of errno is not performed, so the errno enum value will not be present
in the {{NativeIOException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9467) Metrics2 record filtering (.record.filter.include/exclude) does not filter by name
Chris Nauroth created HADOOP-9467: - Summary: Metrics2 record filtering (.record.filter.include/exclude) does not filter by name Key: HADOOP-9467 URL: https://issues.apache.org/jira/browse/HADOOP-9467 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 3.0.0, 1-win, 2.0.4-alpha Reporter: Chris Nauroth Assignee: Chris Nauroth Filtering by record considers only the record's tag for filtering and not the record's name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9483) winutils support fo readlink command
Chris Nauroth created HADOOP-9483: - Summary: winutils support fo readlink command Key: HADOOP-9483 URL: https://issues.apache.org/jira/browse/HADOOP-9483 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 3.0.0 Reporter: Chris Nauroth The current codebase relies on the Unix readlink command to determine the target of a symlink on the local file system. winutils currently does not support this functionality on Windows. Adding the command to winutils will prevent the need to use GnuWin32 or Cygwin for readlink support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
