Re: Thinking ahead to 2.4

Hi, Sorry ignore my inputs. Just keep sub tasks as blocker, because the whole Umbrella tasks may not block release. On Fri, Mar 7, 2014 at 9:44 AM, Azuryy Yu wrote: > Hi Arun, > > I just advice remove some sub tasks from blockers, then add Umbrella > tasks, such as: > > RM HA - YARN-149 > YAR

Re: Thinking ahead to 2.4

Hi Arun, I just advice remove some sub tasks from blockers, then add Umbrella tasks, such as: RM HA - YARN-149 YARN Generic Application Timeline - YARN-1530 Generic application history service- YARN-321 Rolling upgrade(will be ready, no blockers currently, I just mention it) - HDFS-5535 n

Re: In-Memory Reference FS implementations

Thanks steve. So i guess the conclusion is 1) Wait on HADOOP-9361. 2) There definitively cannot be a strict contract for a single HCFS, based on your examples shown. In the meantime ill audit existing test coverage, and let me know if i can lend a hand in the cleanup process. On Thu, Mar 6,

[jira] [Created] (HDFS-6074) Under replicated blocks with ONE replica should get replication priority over blocks with more than one replica.

Mike George created HDFS-6074: - Summary: Under replicated blocks with ONE replica should get replication priority over blocks with more than one replica. Key: HDFS-6074 URL: https://issues.apache.org/jira/browse/HDFS-

[jira] [Created] (HDFS-6073) NameNodeResourceChecker prints 'null' mount point to the log

Akira AJISAKA created HDFS-6073: --- Summary: NameNodeResourceChecker prints 'null' mount point to the log Key: HDFS-6073 URL: https://issues.apache.org/jira/browse/HDFS-6073 Project: Hadoop HDFS

[jira] [Created] (HDFS-6072) Clean up dead code of FSImage

Haohui Mai created HDFS-6072: Summary: Clean up dead code of FSImage Key: HDFS-6072 URL: https://issues.apache.org/jira/browse/HDFS-6072 Project: Hadoop HDFS Issue Type: Improvement R

[jira] [Created] (HDFS-6071) BlockReaderLocal doesn't return -1 on EOF when doing a zero-length read on a short file

Colin Patrick McCabe created HDFS-6071: -- Summary: BlockReaderLocal doesn't return -1 on EOF when doing a zero-length read on a short file Key: HDFS-6071 URL: https://issues.apache.org/jira/browse/HDFS-6071

[jira] [Created] (HDFS-6070) Cleanup use of ReadStatistics in DFSInputStream

Andrew Wang created HDFS-6070: - Summary: Cleanup use of ReadStatistics in DFSInputStream Key: HDFS-6070 URL: https://issues.apache.org/jira/browse/HDFS-6070 Project: Hadoop HDFS Issue Type: Impro

[jira] [Created] (HDFS-6069) Quash stack traces when ACLs are disabled

Andrew Wang created HDFS-6069: - Summary: Quash stack traces when ACLs are disabled Key: HDFS-6069 URL: https://issues.apache.org/jira/browse/HDFS-6069 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Created] (HDFS-6068) Disallow snapshot names that are also invalid directory names

Andrew Wang created HDFS-6068: - Summary: Disallow snapshot names that are also invalid directory names Key: HDFS-6068 URL: https://issues.apache.org/jira/browse/HDFS-6068 Project: Hadoop HDFS Is

Re: Thinking ahead to 2.4

Gang, Most of the big-ticket items are already in, awesome! I'm thinking we could roll out a 2.4 RC in the next 2-3 weeks after we get through the list of blockers. Here is a handy link: http://s.apache.org/hadoop-2.4-blockers If you find more, please set Target Version to 2.4.0 and mark it

[jira] [Reopened] (HDFS-6063) TestAclCLI fails intermittently when running test 24: copyFromLocal

[ https://issues.apache.org/jira/browse/HDFS-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reopened HDFS-6063: - Assignee: Chris Nauroth I'm going to reopen this. I'm still seeing a few failures that look to

Re: In-Memory Reference FS implementations

Lets get the HADOOP-9361 stuff in (it lives alongside FileSystemContractBaseTest) and you can work off that. On 6 March 2014 18:57, Jay Vyas wrote: > Thanks Colin: that's a good example of why we want To unify the hcfs test > profile. So how can hcfs implementations use current hadoop-common

Re: In-Memory Reference FS implementations

EMR's S3 does extra things, which is why netflix used injection tricks to add theirs on top. For blobstores, key use cases are 1. -general source of low-rate-of-change artifacts 2. -input for analysis jobs 3. -output from them 4. -chained operations 5. storage of data to outlive th

Re: In-Memory Reference FS implementations

Thanks Colin: that's a good example of why we want To unify the hcfs test profile. So how can hcfs implementations use current hadoop-common tests? In mind there are three ways. - one solution is to manually cobble together and copy tests , running them one by one and seeing which ones apply

Re: In-Memory Reference FS implementations

NetFlix's Apache-licensed S3mper system provides consistency for an S3-backed store. http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html It would be nice to see this or something like it integrated with Hadoop. I fear that a lot of applications are not ready for eventual consiste

Re: In-Memory Reference FS implementations

do you consider that native S3 FS a real "reference implementation" for blob stores? or just something that , by mere chance, we are able to use as a ref. impl.

[jira] [Created] (HDFS-6067) TestPread.testMaxOutHedgedReadPool is flaky

Colin Patrick McCabe created HDFS-6067: -- Summary: TestPread.testMaxOutHedgedReadPool is flaky Key: HDFS-6067 URL: https://issues.apache.org/jira/browse/HDFS-6067 Project: Hadoop HDFS Iss

Re: FileSystem and FileContext Janitor, at your service !

I think that is the purpose of the bigtop smoke tests, not the filesystem smoke tests. right? On Thu, Mar 6, 2014 at 12:51 PM, Steve Loughran wrote: > I was thinking to test YARN-hosted apps like MapReduce, we need to see how > they handle filesystems with different consistency/atomicity models,

Re: In-Memory Reference FS implementations

On 6 March 2014 16:37, Jay Vyas wrote: > As part of HADOOP-9361, im visioning this. > > 1) - We create In Memory FS implementation of different Reference > FileSystems, each of which specifies appropriate tests , and passes those > tests , i.e. > >InMemStrictlyConsistentFS (i.e. hdfs) > HDFS

Re: FileSystem and FileContext Janitor, at your service !

I was thinking to test YARN-hosted apps like MapReduce, we need to see how they handle filesystems with different consistency/atomicity models, and YARN -even MiniYARNCluster -forks things off. If the MR commit logic is isolated, that could be tested in the JUnit JVM. But for other applications -e

In-Memory Reference FS implementations

As part of HADOOP-9361, im visioning this. 1) - We create In Memory FS implementation of different Reference FileSystems, each of which specifies appropriate tests , and passes those tests , i.e. InMemStrictlyConsistentFS (i.e. hdfs) InMemEventuallyConsistentFS (blob stores) InMemMinmalF

Re: FileSystem and FileContext Janitor, at your service !

steve you mentioned: >> but to test YARN it has to be visible across processes. What do you mean by "test yarn"? I think for the FileSystem APIs unit testing, we dont care about YARN, do we? On Thu, Mar 6, 2014 at 6:02 AM, Steve Loughran wrote: > On 5 March 2014 19:07, Jay Vyas wrote: >

Hadoop-Hdfs-trunk - Build # 1693 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1693/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7915 lines...] [Hadoop-Hdfs-trunk] $ /bin/bash -x /tmp/hu

Build failed in Jenkins: Hadoop-Hdfs-trunk #1693

See Changes: [cmccabe] HDFS-6061. Allow dfs.datanode.shared.file.descriptor.path to contain multiple entries and fall back when needed (cmccabe) [wheat9] HDFS-6058. Fix TestHDFSCLI failures after HADOOP-8691 change. Contributed by

Re: FileSystem and FileContext Janitor, at your service !

On 5 March 2014 19:07, Jay Vyas wrote: > Hi HCFS Community :) > > This is Jay... Some of you know me I hack on a broad range of file > system and hadoop ecosystem interoperability stuff. I just wanted to > introduce myself and let you folks know im going to be working to help > clean up the

[jira] [Created] (HDFS-6066) logGenerationStamp is not needed to reduce editlog size

chenping created HDFS-6066: -- Summary: logGenerationStamp is not needed to reduce editlog size Key: HDFS-6066 URL: https://issues.apache.org/jira/browse/HDFS-6066 Project: Hadoop HDFS Issue Type: Imp

[jira] [Created] (HDFS-6065) HDFS zero-copy reads should return empty buffer on EOF

Colin Patrick McCabe created HDFS-6065: -- Summary: HDFS zero-copy reads should return empty buffer on EOF Key: HDFS-6065 URL: https://issues.apache.org/jira/browse/HDFS-6065 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-6063) TestAclCLI fails intermittently when running test 24: copyFromLocal

[ https://issues.apache.org/jira/browse/HDFS-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai resolved HDFS-6063. -- Resolution: Duplicate This jira should be a duplicate of HDFS-6058. > TestAclCLI fails intermittently w

[jira] [Created] (HDFS-6064) DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_DEFAULT is not updated with latest block report interval of 6 hrs

Vinayakumar B created HDFS-6064: --- Summary: DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_DEFAULT is not updated with latest block report interval of 6 hrs Key: HDFS-6064 URL: https://issues.apache.org/jira/browse/HDFS

[jira] [Created] (HDFS-6063) TestAclCLI fails intermittently

Colin Patrick McCabe created HDFS-6063: -- Summary: TestAclCLI fails intermittently Key: HDFS-6063 URL: https://issues.apache.org/jira/browse/HDFS-6063 Project: Hadoop HDFS Issue Type: Bug