[jira] [Created] (HADOOP-12169) ListStatus on empty dir in S3A lists itself instead of returning an empty list
Pieter Reuse created HADOOP-12169:
-
Summary: ListStatus on empty dir in S3A lists itself instead of
returning an empty list
Key: HADOOP-12169
URL: https://issues.apache.org/jira/browse/HADOOP-12169
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Reporter: Pieter Reuse
Assignee: Pieter Reuse
Upon testing the patch for HADOOP-11918, I stumbled upon a weird behaviour this
introduces to the S3AFileSystem-class. Calling ListStatus() on an empty bucket
returns an empty list, while doing the same on an empty directory, returns an
array of length 1 containing only this directory itself.
The bugfix is quite simple. In the line of code {code}...if
(keyPath.equals(f)...{code} (S3AFileSystem:758), keyPath is qualified wrt. the
fs and f is not. Therefore, this returns false while it shouldn't. The bugfix
to make f qualified in this line of code.
More formally: accoring to the formal definition of [The Hadoop FileSystem API
Definition|https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/filesystem/],
more specifically FileSystem.listStatus, only child elements of a directory
should be returned upon a listStatus()-call.
In detail:
{code}
elif isDir(FS, p): result [getFileStatus(c) for c in children(FS, p) where f(c)
== True]
{code}
and
{code}
def children(FS, p) = {q for q in paths(FS) where parent(q) == p}
{code}
Which translates to the result of listStatus on an empty directory being an
empty list. This is the same behaviour as ls has in Unix, which is what someone
would expect from a FileSystem.
Note: it seemed appropriate to add the test of this patch to the same file as
the test for HADOOP-11918, but as a result, one of the two will have to be
rebased wrt. the other before being applied to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12170) hadoop-common's JNIFlags.cmake is redundant and can be removed
Alan Burlison created HADOOP-12170: -- Summary: hadoop-common's JNIFlags.cmake is redundant and can be removed Key: HADOOP-12170 URL: https://issues.apache.org/jira/browse/HADOOP-12170 Project: Hadoop Common Issue Type: Sub-task Components: native Reporter: Alan Burlison Assignee: Alan Burlison With the integration of: * HADOOP-12036 Consolidate all of the cmake extensions in one *directory * HADOOP-12104 Migrate Hadoop Pipes native build to new CMake * HDFS-8635 Migrate HDFS native build to new CMake framework * MAPREDUCE-6407 Migrate MAPREDUCE native build to new CMake YARN-3827 Migrate YARN native build to new CMake framework hadoop-common-project/hadoop-common/src/JNIFlags.cmake is now redundant and can be removed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: Hadoop-common-trunk-Java8 #245
See https://builds.apache.org/job/Hadoop-common-trunk-Java8/245/changes Changes: [vinayakumarb] HDFS-8579. Update HDFS usage with missing options (Contributed by J.Andreina) [vinayakumarb] HADOOP-12154. FileSystem#getUsed() returns the file length only from root '/' (Contributed by J.Andreina) [vinayakumarb] HDFS-8627. NPE thrown if unable to fetch token from Namenode (Contributed by J.Andreina) [stevel] Revert HADOOP-12009 Clarify FileSystem.listStatus() sorting order fix FileSystemContractBaseTest:testListStatus. (J.Andreina via stevel) [aw] HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) [jlowe] MAPREDUCE-6384. Add the last reporting reducer info for too many fetch failure diagnostics. Contributed by Chang Li [gera] YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) [wang] HADOOP-12158. Improve error message in TestCryptoStreamsWithOpensslAesCtrCryptoCodec when OpenSSL is not installed. [cmccabe] HDFS-8635. Migrate HDFS native build to new CMake framework (Alan Burlison via Colin P. McCabe) [cmccabe] MAPREDUCE-6407. Migrate MAPREDUCE native build to new CMake framework (Alan Burlison via Colin P. McCabe) [cmccabe] YARN-3827. Migrate YARN native build to new CMake framework (Alan Burlison via Colin P. McCabe) [cmccabe] HADOOP-10798. globStatus() should always return a sorted list of files (cmccabe) [cmccabe] HADOOP-12124. Add HTrace support for FsShell (cmccabe) [rkanter] HADOOP-12159. Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces (rchiang via rkanter) [rkanter] MAPREDUCE-6121. JobResourceUpdater#compareFs() doesn't handle HA namespaces (rchiang via rkanter) [cnauroth] HADOOP-12116. Fix unrecommended syntax usages in hadoop/hdfs/yarn script for cygwin in branch-2. Contributed by Li Lu. [lei] HADOOP-12164. Fix TestMove and TestFsShellReturnCode failed to get command name using reflection. (Lei Xu) [devaraj] YARN-3823. Fix mismatch in default values for -- [...truncated 5601 lines...] Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.TestSequenceFileAppend Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.761 sec - in org.apache.hadoop.io.TestSequenceFileAppend Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.TestMapFile Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.349 sec - in org.apache.hadoop.io.TestMapFile Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.TestWritableName Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.177 sec - in org.apache.hadoop.io.TestWritableName Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.TestSortedMapWritable Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.208 sec - in org.apache.hadoop.io.TestSortedMapWritable Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.TestSequenceFileSync Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.051 sec - in org.apache.hadoop.io.TestSequenceFileSync Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.retry.TestFailoverProxy Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.066 sec - in org.apache.hadoop.io.retry.TestFailoverProxy Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.retry.TestDefaultRetryPolicy Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.375 sec - in org.apache.hadoop.io.retry.TestDefaultRetryPolicy Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.retry.TestRetryProxy Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.254 sec - in org.apache.hadoop.io.retry.TestRetryProxy Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.TestDefaultStringifier Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.374 sec - in org.apache.hadoop.io.TestDefaultStringifier Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.io.TestBloomMapFile Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.393 sec - in org.apache.hadoop.io.TestBloomMapFile Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running
Re: [VOTE] Release Apache Hadoop 2.7.1 RC0
+1 (binding) - Verified signatures and digests - Successfully performed a native build from source- Deployed a single-node cluster- Ran sample MapReduce jobs Jason From: Vinod Kumar Vavilapalli vino...@apache.org To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Cc: vino...@apache.org Sent: Monday, June 29, 2015 3:45 AM Subject: [VOTE] Release Apache Hadoop 2.7.1 RC0 Hi all, I've created a release candidate RC0 for Apache Hadoop 2.7.1. As discussed before, this is the next stable release to follow up 2.6.0, and the first stable one in the 2.7.x line. The RC is available for validation at: *http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/ http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/* The RC tag in git is: release-2.7.1-RC0 The maven artifacts are available via repository.apache.org at *https://repository.apache.org/content/repositories/orgapachehadoop-1019/ https://repository.apache.org/content/repositories/orgapachehadoop-1019/* Please try the release and vote; the vote will run for the usual 5 days. Thanks, Vinod PS: It took 2 months instead of the planned [1] 2 weeks in getting this release out: post-mortem in a separate thread. [1]: A 2.7.1 release to follow up 2.7.0 http://markmail.org/thread/zwzze6cqqgwq4rmw
[jira] [Created] (HADOOP-12171) Shorten overly-long htrace span names for server
Colin Patrick McCabe created HADOOP-12171:
-
Summary: Shorten overly-long htrace span names for server
Key: HADOOP-12171
URL: https://issues.apache.org/jira/browse/HADOOP-12171
Project: Hadoop Common
Issue Type: Bug
Components: tracing
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Shorten overly-long htrace span names for the server. For example,
{{org.apache.hadoop.hdfs.protocol.ClientProtocol.create}} should be
{{ClientProtocol#create}} instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
RE: [VOTE] Release Apache Hadoop 2.7.1 RC0
+1 (non-binding) Build from source deployed in 4 nodes cluster for Secure Mode and Non-Secure Mode. Tested with applications spark and MapReduce for RM HA, RM workPreservingRestat, NM work preserving restart. - Rohith Sharma K S -Original Message- From: Mit Desai [mailto:mitdesa...@gmail.com] Sent: 30 June 2015 23:33 To: hdfs-...@hadoop.apache.org Cc: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 2.7.1 RC0 +1 (non-binding) + Built from source + Verified signatures + Deployed on a single node cluster + Ran some sample jobs to successful completion Thanks for driving the release Vinod! -Mit Desai On Tue, Jun 30, 2015 at 12:51 PM, Varun Vasudev vvasu...@apache.org wrote: +1 (non-binding) Built from source, deployed in a single node cluster and ran some test jobs. -Varun On 6/30/15, 9:58 AM, Zhijie Shen zs...@hortonworks.com wrote: +1 (binding) Built from source, deployed a single node cluster and tried some MR jobs. - Zhijie From: Devaraj K deva...@apache.org Sent: Monday, June 29, 2015 9:24 PM To: common-dev@hadoop.apache.org Cc: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 2.7.1 RC0 +1 (non-binding) Deployed in a 3 node cluster and ran some Yarn Apps and MR examples, works fine. On Tue, Jun 30, 2015 at 1:46 AM, Xuan Gong xg...@hortonworks.com wrote: +1 (non-binding) Compiled and deployed a single node cluster, ran all the tests. Xuan Gong On 6/29/15, 1:03 PM, Arpit Gupta ar...@hortonworks.com wrote: +1 (non binding) We have been testing rolling upgrades and downgrades from 2.6 to this release and have had successful runs. -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Jun 29, 2015, at 12:45 PM, Lei Xu l...@cloudera.com wrote: +1 binding Downloaded src and bin distribution, verified md5, sha1 and sha256 checksums of both tar files. Built src using mvn package. Ran a pseudo HDFS cluster Ran dfs -put some files, and checked files on NN's web interface. On Mon, Jun 29, 2015 at 11:54 AM, Wangda Tan wheele...@gmail.com wrote: +1 (non-binding) Compiled and deployed a single node cluster, tried to change node labels and run distributed_shell with node label specified. On Mon, Jun 29, 2015 at 10:30 AM, Ted Yu yuzhih...@gmail.com wrote: +1 (non-binding) Compiled hbase branch-1 with Java 1.8.0_45 Ran unit test suite which passed. On Mon, Jun 29, 2015 at 7:22 AM, Steve Loughran ste...@hortonworks.com wrote: +1 binding from me. Tests: Rebuild slider with Hadoop.version=2.7.1; ran all the tests including against a secure cluster. Repeated for windows running Java 8. All tests passed On 29 Jun 2015, at 09:45, Vinod Kumar Vavilapalli vino...@apache.org wrote: Hi all, I've created a release candidate RC0 for Apache Hadoop 2.7.1. As discussed before, this is the next stable release to follow up 2.6.0, and the first stable one in the 2.7.x line. The RC is available for validation at: *http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/ http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/* The RC tag in git is: release-2.7.1-RC0 The maven artifacts are available via repository.apache.org at * https://repository.apache.org/content/repositories/orgapachehadoop- 101 9/ https://repository.apache.org/content/repositories/orgapachehadoop- 101 9/ * Please try the release and vote; the vote will run for the usual 5 days. Thanks, Vinod PS: It took 2 months instead of the planned [1] 2 weeks in getting this release out: post-mortem in a separate thread. [1]: A 2.7.1 release to follow up 2.7.0 http://markmail.org/thread/zwzze6cqqgwq4rmw -- Lei (Eddy) Xu Software Engineer, Cloudera -- Thanks Devaraj K
[jira] [Created] (HADOOP-12174) compute a code quality index
Allen Wittenauer created HADOOP-12174: - Summary: compute a code quality index Key: HADOOP-12174 URL: https://issues.apache.org/jira/browse/HADOOP-12174 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HADOOP-12111 Reporter: Allen Wittenauer Priority: Minor Just an idea: Compute a code quality index based upon checkstyle, findbugs, and shellcheck but with all exclusions and disables turned off. Generate a number pre-patch and post-patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12172) FsShell mkdir -p makes an unnecessary check for the existence of the parent.
Chris Nauroth created HADOOP-12172:
--
Summary: FsShell mkdir -p makes an unnecessary check for the
existence of the parent.
Key: HADOOP-12172
URL: https://issues.apache.org/jira/browse/HADOOP-12172
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
The {{mkdir}} command in {{FsShell}} checks for the existence of the parent of
the directory and returns an error if it doesn't exist. The {{-p}} option
suppresses the error and allows the directory creation to continue, implicitly
creating all missing intermediate directories. However, the existence check
still runs even with {{-p}} specified, and its result is ignored. Depending on
the file system, this is a wasteful RPC call (HDFS) or HTTP request
(WebHDFS/S3/Azure) imposing extra latency for the client and extra load for the
server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12173) NetworkTopology#add calls NetworkTopology#toString always
Inigo Goiri created HADOOP-12173: Summary: NetworkTopology#add calls NetworkTopology#toString always Key: HADOOP-12173 URL: https://issues.apache.org/jira/browse/HADOOP-12173 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.7.0 Reporter: Inigo Goiri Fix For: 2.7.1 It always does a toString of the whole topology but this is not required when there are no errors. This is adding a very big overhead to large clusters as it's walking the whole tree every time we add a node to the cluster. HADOOP-10953 did some fix in that area but the errors is still there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release Apache Hadoop 2.7.1 RC0
Vinod, thanks for putting together this release. +1 (binding) - Verified signatures - Installed binary release on Centos 6 pseudo cluster * Copied files in and out of HDFS using the shell * Mounted HDFS via NFS and copied a 10GB file in and out over NFS * Ran example MapReduce jobs - Deployed pseudo cluster from sources on Centos 6, verified native bits - Deployed pseudo cluster from sources on Windows 2008 R2, verified native bits and ran example MR jobs Arpit On 6/29/15, 1:45 AM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Hi all, I've created a release candidate RC0 for Apache Hadoop 2.7.1. As discussed before, this is the next stable release to follow up 2.6.0, and the first stable one in the 2.7.x line. The RC is available for validation at: *http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/ http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/* The RC tag in git is: release-2.7.1-RC0 The maven artifacts are available via repository.apache.org at *https://repository.apache.org/content/repositories/orgapachehadoop-1019/ https://repository.apache.org/content/repositories/orgapachehadoop-1019/* Please try the release and vote; the vote will run for the usual 5 days. Thanks, Vinod PS: It took 2 months instead of the planned [1] 2 weeks in getting this release out: post-mortem in a separate thread. [1]: A 2.7.1 release to follow up 2.7.0 http://markmail.org/thread/zwzze6cqqgwq4rmw
