[jira] [Created] (HDFS-9358) TestNodeCount#testNodeCount timed out

[Wei-Chiu Chuang (JIRA)]

Wei-Chiu Chuang created HDFS-9358:
-

 Summary: TestNodeCount#testNodeCount timed out
 Key: HDFS-9358
 URL: https://issues.apache.org/jira/browse/HDFS-9358
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Wei-Chiu Chuang


I have seen this test failure occurred a few times in trunk:

Error Message

Timeout: excess replica count not equal to 2 for block blk_1073741825_1001 
after 2 msec.  Last counts: live = 2, excess = 0, corrupt = 0

Stacktrace

java.util.concurrent.TimeoutException: Timeout: excess replica count not equal 
to 2 for block blk_1073741825_1001 after 2 msec.  Last counts: live = 2, 
excess = 0, corrupt = 0
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount.checkTimeout(TestNodeCount.java:152)
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount.checkTimeout(TestNodeCount.java:146)
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount.__CLR4_0_39bdgm666uf(TestNodeCount.java:130)
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount.testNodeCount(TestNodeCount.java:54)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9359) Enable libhdfs++ to use existing libhdfs CLI tests

[James Clampffer (JIRA)]

James Clampffer created HDFS-9359:
-

 Summary: Enable libhdfs++ to use existing libhdfs CLI tests
 Key: HDFS-9359
 URL: https://issues.apache.org/jira/browse/HDFS-9359
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: James Clampffer
Assignee: James Clampffer






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

continuing releases on Apache Hadoop 2.6.x

[Sangjin Lee]

As you may have seen, 2.6.2 is out
. I have also retargeted all
open issues that were targeted for 2.6.2 to 2.6.3.

Continuing the discussion in the email thread here
, I'd like us to maintain the
cadence of monthly point releases in the 2.6.x line. It would be great if
we can have 2.6.3 released before the year-end holidays.

If you have any bugfixes and improvements that are targeted for 2.7.x (or
2.8) that you think are applicable to 2.6.x, please *set the target version
to 2.6.3* and merge them to branch-2.6. Please use your judgment in terms
of the applicability and quality of the changes so that we can ensure each
point release is consistently better quality than the previous one. Thanks
everyone!

Regards,
Sangjin

Re: continuing releases on Apache Hadoop 2.6.x

[Vinod Vavilapalli]

Just to stress on the following, it is very important that any critical 
bug-fixes that we push into 2.8.0 or even trunk, we should consider them for 
2.6.3 and 2.7.3 if it makes sense. This is the only way we can avoid extremely 
long release cycles like that of 2.6.1.

Also, to clarify a little, use Target-version if you want a discussion of the 
backport, but if you do end up backporting patches after that, you should set 
the fix-version to be 2.6.1.

Thanks
+Vinod


> On Nov 2, 2015, at 11:29 AM, Sangjin Lee  wrote:
> 
> As you may have seen, 2.6.2 is out
> . I have also retargeted all
> open issues that were targeted for 2.6.2 to 2.6.3.
> 
> Continuing the discussion in the email thread here
> , I'd like us to maintain the
> cadence of monthly point releases in the 2.6.x line. It would be great if
> we can have 2.6.3 released before the year-end holidays.
> 
> If you have any bugfixes and improvements that are targeted for 2.7.x (or
> 2.8) that you think are applicable to 2.6.x, please *set the target version
> to 2.6.3* and merge them to branch-2.6. Please use your judgment in terms
> of the applicability and quality of the changes so that we can ensure each
> point release is consistently better quality than the previous one. Thanks
> everyone!
> 
> Regards,
> Sangjin

Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Vinod Vavilapalli]

Forking the thread. Started looking at the 2.8 list, various features’ status 
and arrived here.

While I understand the pervasive nature of EC and a need for a significant 
bake-in, moving this to a 3.x release is not a good idea. We will surely get a 
2.8 out this year and, as needed, I can even spend time getting started on a 
2.9. OTOH, 3.x is long ways off, and given all the incompatibilities there, it 
would be a while before users can get their hands on EC if it were to be only 
on 3.x. At best, this may force sites that want EC to backport the entire EC 
feature to older releases, at worst this will be repeat the mess of 0.20 
security release forks.

If we think adding this to 2.8 (even if it switched off) is too much risk per 
our original plan, let’s move this to 2.9, there by leaving enough time for 
stability, integration testing and bake-in, and a realistic chance of having it 
end up on users’ clusters soonish.

+Vinod

> On Oct 19, 2015, at 1:44 PM, Andrew Wang  wrote:
> 
> I think our plan thus far has been to target this for 3.0. I'm okay with
> putting it in branch-2 if we've given a hard look at compatibility, but
> I'll note though that 2.8 is already looking like quite a large release,
> and our release bandwidth has been focused on the 2.6 and 2.7 maintenance
> releases. Adding another multi-hundred JIRAs to 2.8 might make it too
> unwieldy to get out the door. If we bump EC past that, 3.0 might very well
> be our next release vehicle. I do plan to revive the 3.0 schedule some time
> next year. With EC and JDK8 in a good spot, the only big feature remaining
> is classpath isolation.
> 
> EC is also a pretty fundamental change to HDFS. Even if it's compatible, in
> terms of size and impact it might best belong in a new major release.
> 
> Best,
> Andrew
> 
> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> vinayakumarb.apa...@gmail.com> wrote:
> 
>> Is anyone else also thinks that feature is ready to goto branch-2  as well?
>> 
>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since then and
>> ready to go in branch-2.
>> 
>> -Vinay
>> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
>> 
>>> Thanks Vinay for capturing the issue and Uma for offering the help.
>>> 
>>> ---
>>> Zhe Zhang
>>> 
>>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
>> uma.ganguma...@intel.com
 
>>> wrote:
>>> 
 Vinay,
 
 
 I would merge them as part of HDFS-9182.
 
 Thanks,
 Uma
 
 
 
 On 10/5/15, 12:48 AM, "Vinayakumar B"  wrote:
 
> Hi Andrew,
> I see CHANGES.txt entries not yet merged from
>> CHANGES-HDFS-EC-7285.txt.
> 
> Was this intentional?
> 
> Regards,
> Vinay
> 
> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
>> andrew.w...@cloudera.com>
> wrote:
> 
>> Branch has been merged to trunk, thanks again to everyone who worked
>>> on
>> the
>> feature!
>> 
>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang 
>> wrote:
>> 
>>> Thanks everyone who has participated in this discussion.
>>> 
>>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote
>> has
>> passed.
>>> I will do a final 'git merge' with trunk and work with Andrew to
>>> merge
>> the
>>> branch to trunk. I'll update on this thread when the merge is
>> done.
>>> 
>>> ---
>>> Zhe Zhang
>>> 
>>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A 
>> wrote:
>>> 
 (Change it to binding.)
 
 +1
 I have been involved in the development and code review on the
>> feature
 branch. It's a great feature and I think it's ready to merge it
>>> into
>>> trunk.
 
 Thanks all for the contribution.
 
 Regards,
 Yi Liu
 
 
 -Original Message-
 From: Liu, Yi A
 Sent: Friday, September 25, 2015 1:51 PM
 To: hdfs-dev@hadoop.apache.org
 Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to
>>> trunk
 
 +1 (non-binding)
 I have been involved in the development and code review on the
>> feature
 branch. It's a great feature and I think it's ready to merge it
>>> into
>>> trunk.
 
 Thanks all for the contribution.
 
 Regards,
 Yi Liu
 
 
 -Original Message-
 From: Vinayakumar B [mailto:vinayakum...@apache.org]
 Sent: Friday, September 25, 2015 12:21 PM
 To: hdfs-dev@hadoop.apache.org
 Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to
>>> trunk
 
 +1,
 
 I've been involved starting from design and development of
>> ErasureCoding.
 I think phase 1 of this development is ready to be merged to
>>> trunk.
 

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Andrew Wang]

Thanks for forking the thread Vinod,

SGTM, though I really do recommend waiting for 2.9 given the current size
of 2.8. I'm not a fan of an "off by default" half-measure, since it doesn't
change our compatibility requirements, and there's some major NN surgery
that can't really be disabled.

If we do find a major user who's backported this to their own branch-2
fork, I agree that's motivation to get it in an upstream release quicker. I
haven't heard anything along these lines though.

On Mon, Nov 2, 2015 at 11:49 AM, Vinod Vavilapalli 
wrote:

> Forking the thread. Started looking at the 2.8 list, various features’
> status and arrived here.
>
> While I understand the pervasive nature of EC and a need for a significant
> bake-in, moving this to a 3.x release is not a good idea. We will surely
> get a 2.8 out this year and, as needed, I can even spend time getting
> started on a 2.9. OTOH, 3.x is long ways off, and given all the
> incompatibilities there, it would be a while before users can get their
> hands on EC if it were to be only on 3.x. At best, this may force sites
> that want EC to backport the entire EC feature to older releases, at worst
> this will be repeat the mess of 0.20 security release forks.
>
> If we think adding this to 2.8 (even if it switched off) is too much risk
> per our original plan, let’s move this to 2.9, there by leaving enough time
> for stability, integration testing and bake-in, and a realistic chance of
> having it end up on users’ clusters soonish.
>
> +Vinod
>
> > On Oct 19, 2015, at 1:44 PM, Andrew Wang 
> wrote:
> >
> > I think our plan thus far has been to target this for 3.0. I'm okay with
> > putting it in branch-2 if we've given a hard look at compatibility, but
> > I'll note though that 2.8 is already looking like quite a large release,
> > and our release bandwidth has been focused on the 2.6 and 2.7 maintenance
> > releases. Adding another multi-hundred JIRAs to 2.8 might make it too
> > unwieldy to get out the door. If we bump EC past that, 3.0 might very
> well
> > be our next release vehicle. I do plan to revive the 3.0 schedule some
> time
> > next year. With EC and JDK8 in a good spot, the only big feature
> remaining
> > is classpath isolation.
> >
> > EC is also a pretty fundamental change to HDFS. Even if it's compatible,
> in
> > terms of size and impact it might best belong in a new major release.
> >
> > Best,
> > Andrew
> >
> > On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> > vinayakumarb.apa...@gmail.com> wrote:
> >
> >> Is anyone else also thinks that feature is ready to goto branch-2  as
> well?
> >>
> >> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since then
> and
> >> ready to go in branch-2.
> >>
> >> -Vinay
> >> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
> >>
> >>> Thanks Vinay for capturing the issue and Uma for offering the help.
> >>>
> >>> ---
> >>> Zhe Zhang
> >>>
> >>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
> >> uma.ganguma...@intel.com
> 
> >>> wrote:
> >>>
>  Vinay,
> 
> 
>  I would merge them as part of HDFS-9182.
> 
>  Thanks,
>  Uma
> 
> 
> 
>  On 10/5/15, 12:48 AM, "Vinayakumar B" 
> wrote:
> 
> > Hi Andrew,
> > I see CHANGES.txt entries not yet merged from
> >> CHANGES-HDFS-EC-7285.txt.
> >
> > Was this intentional?
> >
> > Regards,
> > Vinay
> >
> > On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
> >> andrew.w...@cloudera.com>
> > wrote:
> >
> >> Branch has been merged to trunk, thanks again to everyone who worked
> >>> on
> >> the
> >> feature!
> >>
> >> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang 
> >> wrote:
> >>
> >>> Thanks everyone who has participated in this discussion.
> >>>
> >>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote
> >> has
> >> passed.
> >>> I will do a final 'git merge' with trunk and work with Andrew to
> >>> merge
> >> the
> >>> branch to trunk. I'll update on this thread when the merge is
> >> done.
> >>>
> >>> ---
> >>> Zhe Zhang
> >>>
> >>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A 
> >> wrote:
> >>>
>  (Change it to binding.)
> 
>  +1
>  I have been involved in the development and code review on the
> >> feature
>  branch. It's a great feature and I think it's ready to merge it
> >>> into
> >>> trunk.
> 
>  Thanks all for the contribution.
> 
>  Regards,
>  Yi Liu
> 
> 
>  -Original Message-
>  From: Liu, Yi A
>  Sent: Friday, September 25, 2015 1:51 PM
>  To: hdfs-dev@hadoop.apache.org
>  Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to
> >>> trunk
> 
> 

Jenkins build is back to normal : Hadoop-Hdfs-trunk-Java8 #563

[Apache Jenkins Server]

See 

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #565

[Apache Jenkins Server]

See 

Changes:

[zhz] HDFS-9339. Extend full test of KMS ACLs. Contributed by Daniel

[lei] HDFS-9312. Fix TestReplication to be FsDataset-agnostic. (lei)

[yliu] HDFS-9275. Wait previous ErasureCodingWork to finish before schedule

[lei] HDFS-9308. Add truncateMeta() and deleteMeta() to MiniDFSCluster. (Tony

--
[...truncated 6488 lines...]
Running org.apache.hadoop.hdfs.web.TestWebHDFSAcl
Tests run: 64, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 32.071 sec - 
in org.apache.hadoop.hdfs.web.TestWebHDFSAcl
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.web.TestWebHdfsTokens
Killed
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.web.TestWebHDFS
Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.481 sec - 
in org.apache.hadoop.hdfs.web.TestWebHDFS
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.web.resources.TestParam
Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.756 sec - in 
org.apache.hadoop.hdfs.web.resources.TestParam
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.738 sec - in 
org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.309 sec - 
in org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestAppendSnapshotTruncate
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.145 sec - in 
org.apache.hadoop.hdfs.TestAppendSnapshotTruncate
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestFileCreationClient
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.816 sec - in 
org.apache.hadoop.hdfs.TestFileCreationClient
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestBalancerBandwidth
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.113 sec - in 
org.apache.hadoop.hdfs.TestBalancerBandwidth
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestHDFSServerPorts
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.319 sec - in 
org.apache.hadoop.hdfs.TestHDFSServerPorts
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestBlockStoragePolicy
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.864 sec - 
in org.apache.hadoop.hdfs.TestBlockStoragePolicy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.235 sec - in 
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.qjournal.server.TestJournal
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.224 sec - in 
org.apache.hadoop.hdfs.qjournal.server.TestJournal
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.qjournal.server.TestJournalNodeMXBean
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.578 sec - in 
org.apache.hadoop.hdfs.qjournal.server.TestJournalNodeMXBean
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.qjournal.server.TestJournalNode
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.583 sec - in 
org.apache.hadoop.hdfs.qjournal.server.TestJournalNode
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.qjournal.client.TestSegmentRecoveryComparator
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.269 sec - in 
org.apache.hadoop.hdfs.qjournal.client.TestSegmentRecoveryComparator
Java HotSpot(TM) 64-Bit Server VM warning: ignoring 

Hadoop-Hdfs-trunk-Java8 - Build # 565 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/565/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 6681 lines...]
main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS Client . SUCCESS [04:16 min]
[INFO] Apache Hadoop HDFS  FAILURE [  03:10 h]
[INFO] Apache Hadoop HDFS Native Client .. SKIPPED
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  0.073 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 03:15 h
[INFO] Finished at: 2015-11-03T05:38:24+00:00
[INFO] Final Memory: 72M/827M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-hdfs: ExecutionException: java.lang.RuntimeException: The forked 
VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs
 && /home/jenkins/tools/java/jdk1.8.0/jre/bin/java -Xmx2048m 
-XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/surefire/surefirebooter2356637940303460489.jar
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/surefire/surefire7007908682028126133tmp
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/surefire/surefire_3653076470516256072854tmp
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-hdfs
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating HDFS-9308
Updating HDFS-9275
Updating HDFS-9312
Updating HDFS-9339
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics.testDataNodeTimeSpend

Error Message:
null

Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics.testDataNodeTimeSpend(TestDataNodeMetrics.java:289)

Re: 2.7.2 release plan

[Wangda Tan]

HI Naga and Vinod/Tsuyoshi/Karthik,

Looked at this list, IIRC, some of them are 70k+ patch, I'm afraid the
changes number is too many and risky for a minor release. Issues besides
YARN-3136 are more or less change web UI / REST API, and they look more
like enhancements instead of bug fixes.

I marked YARN-3136 to 2.7.2-candidate, and I suggest to delay other changes
to 2.8.0 release.

Thoughts?

Regards,
Wangda


On Wed, Oct 28, 2015 at 11:56 PM, Naganarasimha G R (Naga) <
garlanaganarasi...@huawei.com> wrote:

> Thanks for sharing this important viewpoint.
>
> This sub list of NodeLabels jiras what i have selected is doing minimal
> modifications to the core code but tries to increase the usability of
> NodeLabels and fix some bugs or add missing necessary features
> Other additional features which  were done for NodeLabels like Distributed
> Scheduling or Delegated Centralized are all not included.
> May be Wangda could be better judge to further scrutinize the list and
> select from them or even add to them
> My intention here is to only make the Node Labels more usable as there has
> been long delay for 2.8 and not heard of any approximate dates for it.
>
> Regards,
> + Naga
>
>
> 
> From: Karthik Kambatla [ka...@cloudera.com]
> Sent: Thursday, October 29, 2015 04:04
> To: yarn-...@hadoop.apache.org
> Cc: Tsuyoshi Ozawa; Vinod Vavilapalli; hdfs-dev@hadoop.apache.org;
> common-...@hadoop.apache.org; Vinod Kumar Vavilapalli; Wangda Tan
> Subject: Re: 2.7.2 release plan
>
> I would like for us to make sure later maintenance releases are more stable
> than previous ones. IMO, increasing stability is more important than the
> timing of a release.
>
> Will adding all the patches in 2.7.3 reduce the stability going from 2.7.2
> to 2.7.3? If yes, can we just leave them for 2.8.0?
>
> On Wed, Oct 28, 2015 at 12:07 PM, Naganarasimha G R (Naga) <
> garlanaganarasi...@huawei.com> wrote:
>
> > Yes Vinod & Tsuyoshi,
> >
> > Within a week merging them would be difficult. I can start backporting
> > them after 2.7.2 so that it can be ported to 2.7.3 faster, also shall i
> > apply  2.7.3-candidate labels to them ?
> >
> > + Naga
> > __
> > From: Tsuyoshi Ozawa [oz...@apache.org]
> > Sent: Wednesday, October 28, 2015 23:13
> > To: Vinod Vavilapalli
> > Cc: yarn-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> > common-...@hadoop.apache.org; Vinod Kumar Vavilapalli; Wangda Tan;
> > Tsuyoshi Ozawa; Naganarasimha G R (Naga)
> > Subject: Re: 2.7.2 release plan
> >
> > Vinod,
> >
> > Thank you for taking care of this. I've checked the list of changes.
> > As a result, I agree that we don't have enough time to backport these
> > changes into 2.7.2 by this weekend. For a fast move, it's better
> > suggestion to me to backport these tickets into 2.7.3.
> >
> > Best,
> > - Tsuyoshi
> >
> > On Thu, Oct 29, 2015 at 2:19 AM, Vinod Vavilapalli
> >  wrote:
> > > Tsuyoshi / Wangda / Naga,
> > >
> > > This looks too big of a list to me if we have to cut an RC by this
> > weekend per my plan.
> > >
> > > I’d suggest a fast move on things you think are low risk enough and
> punt
> > everything else for next release.
> > >
> > > Thanks
> > > +Vinod
> > >
> > >> On Oct 28, 2015, at 3:08 AM, Naganarasimha G R (Naga) <
> > garlanaganarasi...@huawei.com> wrote:
> > >>
> > >> Thanks Tsuyoshi,
> > >> If required even i can pitch in  :)
> > >> Additional to this we added the support in Mapreduce for labels in
> > MAPREDUCE-6304,
> > >>
> > >> Regards,
> > >> + Naga
> > >> 
> > >> From: Tsuyoshi Ozawa [oz...@apache.org]
> > >> Sent: Wednesday, October 28, 2015 14:28
> > >> To: yarn-...@hadoop.apache.org
> > >> Cc: hdfs-dev@hadoop.apache.org; common-...@hadoop.apache.org; Vinod
> > Kumar Vavilapalli; Wangda tan
> > >> Subject: Re: 2.7.2 release plan
> > >>
> > >> Thank you for reporting, Naganarasimha.
> > >> Vinod and Wangda, I will help you to backport these changes.
> > >>
> > >> Best,
> > >> - Tsuyoshi
> > >>
> > >> On Wed, Oct 28, 2015 at 2:57 PM, Naganarasimha G R (Naga)
> > >>  wrote:
> > >>> Hi Vinod, & Wangda
> > >>>
> > >>> I think it would be good to backport, following jira's related to
> > NodeLabels as it will improve debug ability and usability of NodeLabels
> > >>> 
> > >>> Key Summary
> > >>> 
> > >>> YARN-4215   YARN-2492 RMNodeLabels Manager Need to verify and
> > replace node labels for the only modified Node Label Mappings in the
> request
> > >>> YARN-4162   YARN-2492 CapacityScheduler: Add resource usage by
> > partition and queue capacity by partition to REST API
> > >>> YARN-4140   YARN-2492 RM container allocation delayed incase of
> > app submitted to Nodelabel partition
> > >>> YARN-3717   YARN-2492 Expose app/am/queue's node-label-expression
> > to RM 

[jira] [Created] (HDFS-9360) Storage type usage isn't updated properly after file deletion

[Ming Ma (JIRA)]

Ming Ma created HDFS-9360:
-

 Summary: Storage type usage isn't updated properly after file 
deletion
 Key: HDFS-9360
 URL: https://issues.apache.org/jira/browse/HDFS-9360
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ming Ma


For a directory that doesn't have any storage policy defined, its quota usage 
is deducted when a file is deleted. This means incorrect value for storage 
quota usage. Later when applications set the storage type, it can exceed its 
storage quota.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Jing Zhao]

Thanks for the discussion, Vinod and Andrew. Backporting EC to 2.9 sounds
good to me.

On Mon, Nov 2, 2015 at 12:06 PM, Andrew Wang 
wrote:

> Thanks for forking the thread Vinod,
>
> SGTM, though I really do recommend waiting for 2.9 given the current size
> of 2.8. I'm not a fan of an "off by default" half-measure, since it doesn't
> change our compatibility requirements, and there's some major NN surgery
> that can't really be disabled.
>
> If we do find a major user who's backported this to their own branch-2
> fork, I agree that's motivation to get it in an upstream release quicker. I
> haven't heard anything along these lines though.
>
> On Mon, Nov 2, 2015 at 11:49 AM, Vinod Vavilapalli <
> vino...@hortonworks.com>
> wrote:
>
> > Forking the thread. Started looking at the 2.8 list, various features’
> > status and arrived here.
> >
> > While I understand the pervasive nature of EC and a need for a
> significant
> > bake-in, moving this to a 3.x release is not a good idea. We will surely
> > get a 2.8 out this year and, as needed, I can even spend time getting
> > started on a 2.9. OTOH, 3.x is long ways off, and given all the
> > incompatibilities there, it would be a while before users can get their
> > hands on EC if it were to be only on 3.x. At best, this may force sites
> > that want EC to backport the entire EC feature to older releases, at
> worst
> > this will be repeat the mess of 0.20 security release forks.
> >
> > If we think adding this to 2.8 (even if it switched off) is too much risk
> > per our original plan, let’s move this to 2.9, there by leaving enough
> time
> > for stability, integration testing and bake-in, and a realistic chance of
> > having it end up on users’ clusters soonish.
> >
> > +Vinod
> >
> > > On Oct 19, 2015, at 1:44 PM, Andrew Wang 
> > wrote:
> > >
> > > I think our plan thus far has been to target this for 3.0. I'm okay
> with
> > > putting it in branch-2 if we've given a hard look at compatibility, but
> > > I'll note though that 2.8 is already looking like quite a large
> release,
> > > and our release bandwidth has been focused on the 2.6 and 2.7
> maintenance
> > > releases. Adding another multi-hundred JIRAs to 2.8 might make it too
> > > unwieldy to get out the door. If we bump EC past that, 3.0 might very
> > well
> > > be our next release vehicle. I do plan to revive the 3.0 schedule some
> > time
> > > next year. With EC and JDK8 in a good spot, the only big feature
> > remaining
> > > is classpath isolation.
> > >
> > > EC is also a pretty fundamental change to HDFS. Even if it's
> compatible,
> > in
> > > terms of size and impact it might best belong in a new major release.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> > > vinayakumarb.apa...@gmail.com> wrote:
> > >
> > >> Is anyone else also thinks that feature is ready to goto branch-2  as
> > well?
> > >>
> > >> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since
> then
> > and
> > >> ready to go in branch-2.
> > >>
> > >> -Vinay
> > >> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
> > >>
> > >>> Thanks Vinay for capturing the issue and Uma for offering the help.
> > >>>
> > >>> ---
> > >>> Zhe Zhang
> > >>>
> > >>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
> > >> uma.ganguma...@intel.com
> > 
> > >>> wrote:
> > >>>
> >  Vinay,
> > 
> > 
> >  I would merge them as part of HDFS-9182.
> > 
> >  Thanks,
> >  Uma
> > 
> > 
> > 
> >  On 10/5/15, 12:48 AM, "Vinayakumar B" 
> > wrote:
> > 
> > > Hi Andrew,
> > > I see CHANGES.txt entries not yet merged from
> > >> CHANGES-HDFS-EC-7285.txt.
> > >
> > > Was this intentional?
> > >
> > > Regards,
> > > Vinay
> > >
> > > On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
> > >> andrew.w...@cloudera.com>
> > > wrote:
> > >
> > >> Branch has been merged to trunk, thanks again to everyone who
> worked
> > >>> on
> > >> the
> > >> feature!
> > >>
> > >> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang <
> zhezh...@cloudera.com>
> > >> wrote:
> > >>
> > >>> Thanks everyone who has participated in this discussion.
> > >>>
> > >>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote
> > >> has
> > >> passed.
> > >>> I will do a final 'git merge' with trunk and work with Andrew to
> > >>> merge
> > >> the
> > >>> branch to trunk. I'll update on this thread when the merge is
> > >> done.
> > >>>
> > >>> ---
> > >>> Zhe Zhang
> > >>>
> > >>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A 
> > >> wrote:
> > >>>
> >  (Change it to binding.)
> > 
> >  +1
> >  I have been involved in the development and code review on the
> > >> feature
> >  branch. It's a great 

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Gangumalla, Uma]

+1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan to
have 2.8 and 2.9 releases.

Regards,
Uma

On 11/2/15, 11:49 AM, "Vinod Vavilapalli"  wrote:

>Forking the thread. Started looking at the 2.8 list, various features¹
>status and arrived here.
>
>While I understand the pervasive nature of EC and a need for a
>significant bake-in, moving this to a 3.x release is not a good idea. We
>will surely get a 2.8 out this year and, as needed, I can even spend time
>getting started on a 2.9. OTOH, 3.x is long ways off, and given all the
>incompatibilities there, it would be a while before users can get their
>hands on EC if it were to be only on 3.x. At best, this may force sites
>that want EC to backport the entire EC feature to older releases, at
>worst this will be repeat the mess of 0.20 security release forks.
>
>If we think adding this to 2.8 (even if it switched off) is too much risk
>per our original plan, let¹s move this to 2.9, there by leaving enough
>time for stability, integration testing and bake-in, and a realistic
>chance of having it end up on users¹ clusters soonish.
>
>+Vinod
>
>> On Oct 19, 2015, at 1:44 PM, Andrew Wang 
>>wrote:
>> 
>> I think our plan thus far has been to target this for 3.0. I'm okay with
>> putting it in branch-2 if we've given a hard look at compatibility, but
>> I'll note though that 2.8 is already looking like quite a large release,
>> and our release bandwidth has been focused on the 2.6 and 2.7
>>maintenance
>> releases. Adding another multi-hundred JIRAs to 2.8 might make it too
>> unwieldy to get out the door. If we bump EC past that, 3.0 might very
>>well
>> be our next release vehicle. I do plan to revive the 3.0 schedule some
>>time
>> next year. With EC and JDK8 in a good spot, the only big feature
>>remaining
>> is classpath isolation.
>> 
>> EC is also a pretty fundamental change to HDFS. Even if it's
>>compatible, in
>> terms of size and impact it might best belong in a new major release.
>> 
>> Best,
>> Andrew
>> 
>> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
>> vinayakumarb.apa...@gmail.com> wrote:
>> 
>>> Is anyone else also thinks that feature is ready to goto branch-2  as
>>>well?
>>> 
>>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since
>>>then and
>>> ready to go in branch-2.
>>> 
>>> -Vinay
>>> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
>>> 
 Thanks Vinay for capturing the issue and Uma for offering the help.
 
 ---
 Zhe Zhang
 
 On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
>>> uma.ganguma...@intel.com
> 
 wrote:
 
> Vinay,
> 
> 
> I would merge them as part of HDFS-9182.
> 
> Thanks,
> Uma
> 
> 
> 
> On 10/5/15, 12:48 AM, "Vinayakumar B" 
>wrote:
> 
>> Hi Andrew,
>> I see CHANGES.txt entries not yet merged from
>>> CHANGES-HDFS-EC-7285.txt.
>> 
>> Was this intentional?
>> 
>> Regards,
>> Vinay
>> 
>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
>>> andrew.w...@cloudera.com>
>> wrote:
>> 
>>> Branch has been merged to trunk, thanks again to everyone who
>>>worked
 on
>>> the
>>> feature!
>>> 
>>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang 
>>> wrote:
>>> 
 Thanks everyone who has participated in this discussion.
 
 With 7 +1's (5 binding and 2 non-binding), and no -1, this vote
>>> has
>>> passed.
 I will do a final 'git merge' with trunk and work with Andrew to
 merge
>>> the
 branch to trunk. I'll update on this thread when the merge is
>>> done.
 
 ---
 Zhe Zhang
 
 On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A 
>>> wrote:
 
> (Change it to binding.)
> 
> +1
> I have been involved in the development and code review on the
>>> feature
> branch. It's a great feature and I think it's ready to merge it
 into
 trunk.
> 
> Thanks all for the contribution.
> 
> Regards,
> Yi Liu
> 
> 
> -Original Message-
> From: Liu, Yi A
> Sent: Friday, September 25, 2015 1:51 PM
> To: hdfs-dev@hadoop.apache.org
> Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to
 trunk
> 
> +1 (non-binding)
> I have been involved in the development and code review on the
>>> feature
> branch. It's a great feature and I think it's ready to merge it
 into
 trunk.
> 
> Thanks all for the contribution.
> 
> Regards,
> Yi Liu
> 
> 
> -Original Message-
> From: Vinayakumar B [mailto:vinayakum...@apache.org]
> Sent: Friday, 

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Zhe Zhang]

Thanks Vinod for the proposal and Andrew/Jing for the comments. 2.9 sounds
a good plan. As Andrew pointed out, 2.8 is already quite big. And even when
disabled, EC logic has been baked in NN pretty deeply.

Do we have a tentative date or estimate for 2.9?

---
Zhe Zhang

On Mon, Nov 2, 2015 at 1:22 PM, Jing Zhao  wrote:

> Thanks for the discussion, Vinod and Andrew. Backporting EC to 2.9 sounds
> good to me.
>
> On Mon, Nov 2, 2015 at 12:06 PM, Andrew Wang 
> wrote:
>
> > Thanks for forking the thread Vinod,
> >
> > SGTM, though I really do recommend waiting for 2.9 given the current size
> > of 2.8. I'm not a fan of an "off by default" half-measure, since it
> doesn't
> > change our compatibility requirements, and there's some major NN surgery
> > that can't really be disabled.
> >
> > If we do find a major user who's backported this to their own branch-2
> > fork, I agree that's motivation to get it in an upstream release
> quicker. I
> > haven't heard anything along these lines though.
> >
> > On Mon, Nov 2, 2015 at 11:49 AM, Vinod Vavilapalli <
> > vino...@hortonworks.com>
> > wrote:
> >
> > > Forking the thread. Started looking at the 2.8 list, various features’
> > > status and arrived here.
> > >
> > > While I understand the pervasive nature of EC and a need for a
> > significant
> > > bake-in, moving this to a 3.x release is not a good idea. We will
> surely
> > > get a 2.8 out this year and, as needed, I can even spend time getting
> > > started on a 2.9. OTOH, 3.x is long ways off, and given all the
> > > incompatibilities there, it would be a while before users can get their
> > > hands on EC if it were to be only on 3.x. At best, this may force sites
> > > that want EC to backport the entire EC feature to older releases, at
> > worst
> > > this will be repeat the mess of 0.20 security release forks.
> > >
> > > If we think adding this to 2.8 (even if it switched off) is too much
> risk
> > > per our original plan, let’s move this to 2.9, there by leaving enough
> > time
> > > for stability, integration testing and bake-in, and a realistic chance
> of
> > > having it end up on users’ clusters soonish.
> > >
> > > +Vinod
> > >
> > > > On Oct 19, 2015, at 1:44 PM, Andrew Wang 
> > > wrote:
> > > >
> > > > I think our plan thus far has been to target this for 3.0. I'm okay
> > with
> > > > putting it in branch-2 if we've given a hard look at compatibility,
> but
> > > > I'll note though that 2.8 is already looking like quite a large
> > release,
> > > > and our release bandwidth has been focused on the 2.6 and 2.7
> > maintenance
> > > > releases. Adding another multi-hundred JIRAs to 2.8 might make it too
> > > > unwieldy to get out the door. If we bump EC past that, 3.0 might very
> > > well
> > > > be our next release vehicle. I do plan to revive the 3.0 schedule
> some
> > > time
> > > > next year. With EC and JDK8 in a good spot, the only big feature
> > > remaining
> > > > is classpath isolation.
> > > >
> > > > EC is also a pretty fundamental change to HDFS. Even if it's
> > compatible,
> > > in
> > > > terms of size and impact it might best belong in a new major release.
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> > > > vinayakumarb.apa...@gmail.com> wrote:
> > > >
> > > >> Is anyone else also thinks that feature is ready to goto branch-2
> as
> > > well?
> > > >>
> > > >> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since
> > then
> > > and
> > > >> ready to go in branch-2.
> > > >>
> > > >> -Vinay
> > > >> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
> > > >>
> > > >>> Thanks Vinay for capturing the issue and Uma for offering the help.
> > > >>>
> > > >>> ---
> > > >>> Zhe Zhang
> > > >>>
> > > >>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
> > > >> uma.ganguma...@intel.com
> > > 
> > > >>> wrote:
> > > >>>
> > >  Vinay,
> > > 
> > > 
> > >  I would merge them as part of HDFS-9182.
> > > 
> > >  Thanks,
> > >  Uma
> > > 
> > > 
> > > 
> > >  On 10/5/15, 12:48 AM, "Vinayakumar B" 
> > > wrote:
> > > 
> > > > Hi Andrew,
> > > > I see CHANGES.txt entries not yet merged from
> > > >> CHANGES-HDFS-EC-7285.txt.
> > > >
> > > > Was this intentional?
> > > >
> > > > Regards,
> > > > Vinay
> > > >
> > > > On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
> > > >> andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Branch has been merged to trunk, thanks again to everyone who
> > worked
> > > >>> on
> > > >> the
> > > >> feature!
> > > >>
> > > >> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang <
> > zhezh...@cloudera.com>
> > > >> wrote:
> > > >>
> > > >>> Thanks everyone who has participated in this discussion.
> > > >>>
> > > >>> With 7 +1's (5 binding and 2 

Re: Planning Apache Hadoop 2.7.2

[Vinod Vavilapalli]

We are down to 4 now (https://issues.apache.org/jira/issues/?filter=12332867), 
appreciate help moving forward with them.

Thanks
+Vinod

On Oct 26, 2015, at 11:34 AM, Vinod Kumar Vavilapalli 
> wrote:

Got swamped again.

We already have about 112 patches in 2.7.2 already.

There are 15 open tickets in progress. I’ll push progress on them for an RC 
towards end of this week.

Thanks
+Vinod

On Sep 25, 2015, at 3:27 PM, Vinod Kumar Vavilapalli 
> wrote:

Hi all,

We released 2.7.1 nearly 2.5 months ago. I got caught up with a very long 
release process for 2.6.1 so couldn't make progress on a 2.7.2. Now is the time!

Things to do

 (#1) Branch
-- Branch 2.7 has been open to 2.7.2 commits for a while.
-- In order to converge on a release, I will branch out 2.7.2 soon.

  (#2) Patches
-- 2.7.2 already has a boat load [1] of fixes.
-- The list of open blocker / critical tickets [2] is not small. I'll start 
triaging and see what can make it in a week or so of time.

  (#3) Release
-- Even if we can get half of the blocker / critical tickets in, the full 
list [3] will be big enough for us to start voting on an RC in no less than a 
week.
-- Leaving aside some buffer time, I plan to start RC process by end of 
first week of October.

Thoughts?

Appreciate help in moving open tickets [2] forward.

A general note:  Please consider putting any critical / blocker tickets on 2.8 
into 2.6.2 and 2.7.2 releases.

Thanks
+Vinod

[1] 2.7.2 Fixed Tickets: https://issues.apache.org/jira/issues/?filter=12333473
[2] 2.7.2 Open Blockers / Critical Tickets: 
https://issues.apache.org/jira/issues/?filter=12332867
[3] 2.7.2 Release Tickets: 
https://issues.apache.org/jira/issues/?filter=12333461

[jira] [Created] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.

[Chris Nauroth (JIRA)]

Chris Nauroth created HDFS-9362:
---

 Summary: TestAuditLogger#testAuditLoggerWithCallContext assumes 
Unix line endings, fails on Windows.
 Key: HDFS-9362
 URL: https://issues.apache.org/jira/browse/HDFS-9362
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor


{{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to 
exercise the new audit logging with caller context functionality.  The tests 
assume Unix line endings by hard-coding "\n" in asserts.  These tests fail on 
Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Zheng, Kai]

Sounds good to me. When it's determined to include EC in 2.9 release, it may be 
good to have a rough release date as Zhe asked, so accordingly the scope of EC 
can be discussed out. We still have quite a few of things as Phase I follow-on 
tasks to do before EC can be deployed in a production system. Phase II to 
develop non-striping EC for cold data would possibly be started after that. We 
might consider to include only Phase I and leave Phase II for next release 
according to the rough release date.

Regards,
Kai

-Original Message-
From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] 
Sent: Tuesday, November 03, 2015 5:41 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 
(erasure coding) branch to trunk]

+1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan 
+to
have 2.8 and 2.9 releases.

Regards,
Uma

On 11/2/15, 11:49 AM, "Vinod Vavilapalli"  wrote:

>Forking the thread. Started looking at the 2.8 list, various features¹ 
>status and arrived here.
>
>While I understand the pervasive nature of EC and a need for a 
>significant bake-in, moving this to a 3.x release is not a good idea. 
>We will surely get a 2.8 out this year and, as needed, I can even spend 
>time getting started on a 2.9. OTOH, 3.x is long ways off, and given 
>all the incompatibilities there, it would be a while before users can 
>get their hands on EC if it were to be only on 3.x. At best, this may 
>force sites that want EC to backport the entire EC feature to older 
>releases, at worst this will be repeat the mess of 0.20 security release forks.
>
>If we think adding this to 2.8 (even if it switched off) is too much 
>risk per our original plan, let¹s move this to 2.9, there by leaving 
>enough time for stability, integration testing and bake-in, and a 
>realistic chance of having it end up on users¹ clusters soonish.
>
>+Vinod
>
>> On Oct 19, 2015, at 1:44 PM, Andrew Wang 
>>wrote:
>> 
>> I think our plan thus far has been to target this for 3.0. I'm okay 
>>with  putting it in branch-2 if we've given a hard look at 
>>compatibility, but  I'll note though that 2.8 is already looking like 
>>quite a large release,  and our release bandwidth has been focused on 
>>the 2.6 and 2.7 maintenance  releases. Adding another multi-hundred 
>>JIRAs to 2.8 might make it too  unwieldy to get out the door. If we 
>>bump EC past that, 3.0 might very well  be our next release vehicle. I 
>>do plan to revive the 3.0 schedule some time  next year. With EC and 
>>JDK8 in a good spot, the only big feature remaining  is classpath 
>>isolation.
>> 
>> EC is also a pretty fundamental change to HDFS. Even if it's 
>>compatible, in  terms of size and impact it might best belong in a new 
>>major release.
>> 
>> Best,
>> Andrew
>> 
>> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < 
>> vinayakumarb.apa...@gmail.com> wrote:
>> 
>>> Is anyone else also thinks that feature is ready to goto branch-2  
>>>as well?
>>> 
>>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since 
>>>then and  ready to go in branch-2.
>>> 
>>> -Vinay
>>> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
>>> 
 Thanks Vinay for capturing the issue and Uma for offering the help.
 
 ---
 Zhe Zhang
 
 On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
>>> uma.ganguma...@intel.com
> 
 wrote:
 
> Vinay,
> 
> 
> I would merge them as part of HDFS-9182.
> 
> Thanks,
> Uma
> 
> 
> 
> On 10/5/15, 12:48 AM, "Vinayakumar B" 
>wrote:
> 
>> Hi Andrew,
>> I see CHANGES.txt entries not yet merged from
>>> CHANGES-HDFS-EC-7285.txt.
>> 
>> Was this intentional?
>> 
>> Regards,
>> Vinay
>> 
>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
>>> andrew.w...@cloudera.com>
>> wrote:
>> 
>>> Branch has been merged to trunk, thanks again to everyone who 
>>>worked
 on
>>> the
>>> feature!
>>> 
>>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang 
>>> 
>>> wrote:
>>> 
 Thanks everyone who has participated in this discussion.
 
 With 7 +1's (5 binding and 2 non-binding), and no -1, this vote
>>> has
>>> passed.
 I will do a final 'git merge' with trunk and work with Andrew 
 to
 merge
>>> the
 branch to trunk. I'll update on this thread when the merge is
>>> done.
 
 ---
 Zhe Zhang
 
 On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A 
 
>>> wrote:
 
> (Change it to binding.)
> 
> +1
> I have been involved in the development and code review on the
>>> feature
> branch. It's a great feature and I think it's ready to merge 
> it
 into
 

Hadoop-Hdfs-trunk - Build # 2500 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2500/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 9357 lines...]
[INFO] Deleting 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Skipping javadoc generation
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS Client . SUCCESS [06:30 min]
[INFO] Apache Hadoop HDFS  FAILURE [  06:32 h]
[INFO] Apache Hadoop HDFS Native Client .. SKIPPED
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  0.195 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 06:38 h
[INFO] Finished at: 2015-11-02T22:36:22+00:00
[INFO] Final Memory: 68M/589M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-hdfs: There was a timeout or other error in the fork -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-hdfs
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating HDFS-8777
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
16 tests failed.
FAILED:  
org.apache.hadoop.hdfs.TestDFSClientRetries.testIdempotentAllocateBlockAndClose

Error Message:
Not replicated yet: /testIdempotentAllocateBlock

Stack Trace:
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not 
replicated yet: /testIdempotentAllocateBlock
at 
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:190)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2445)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:796)
at 
org.apache.hadoop.hdfs.DFSOutputStream.addBlock(DFSOutputStream.java:911)
at 
org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1684)
at 
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1494)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:594)


FAILED:  org.apache.hadoop.hdfs.TestEncryptionZones.testStartFileRetry

Error Message:
test timed out after 12 milliseconds

Stack Trace:
java.lang.Exception: test timed out after 12 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at 

RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Vinayakumar B]

+1 for the idea.
On Nov 3, 2015 07:22, "Zheng, Kai"  wrote:

> Sounds good to me. When it's determined to include EC in 2.9 release, it
> may be good to have a rough release date as Zhe asked, so accordingly the
> scope of EC can be discussed out. We still have quite a few of things as
> Phase I follow-on tasks to do before EC can be deployed in a production
> system. Phase II to develop non-striping EC for cold data would possibly be
> started after that. We might consider to include only Phase I and leave
> Phase II for next release according to the rough release date.
>
> Regards,
> Kai
>
> -Original Message-
> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com]
> Sent: Tuesday, November 03, 2015 5:41 AM
> To: hdfs-dev@hadoop.apache.org
> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285
> (erasure coding) branch to trunk]
>
> +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan
> +to
> have 2.8 and 2.9 releases.
>
> Regards,
> Uma
>
> On 11/2/15, 11:49 AM, "Vinod Vavilapalli"  wrote:
>
> >Forking the thread. Started looking at the 2.8 list, various features¹
> >status and arrived here.
> >
> >While I understand the pervasive nature of EC and a need for a
> >significant bake-in, moving this to a 3.x release is not a good idea.
> >We will surely get a 2.8 out this year and, as needed, I can even spend
> >time getting started on a 2.9. OTOH, 3.x is long ways off, and given
> >all the incompatibilities there, it would be a while before users can
> >get their hands on EC if it were to be only on 3.x. At best, this may
> >force sites that want EC to backport the entire EC feature to older
> >releases, at worst this will be repeat the mess of 0.20 security release
> forks.
> >
> >If we think adding this to 2.8 (even if it switched off) is too much
> >risk per our original plan, let¹s move this to 2.9, there by leaving
> >enough time for stability, integration testing and bake-in, and a
> >realistic chance of having it end up on users¹ clusters soonish.
> >
> >+Vinod
> >
> >> On Oct 19, 2015, at 1:44 PM, Andrew Wang 
> >>wrote:
> >>
> >> I think our plan thus far has been to target this for 3.0. I'm okay
> >>with  putting it in branch-2 if we've given a hard look at
> >>compatibility, but  I'll note though that 2.8 is already looking like
> >>quite a large release,  and our release bandwidth has been focused on
> >>the 2.6 and 2.7 maintenance  releases. Adding another multi-hundred
> >>JIRAs to 2.8 might make it too  unwieldy to get out the door. If we
> >>bump EC past that, 3.0 might very well  be our next release vehicle. I
> >>do plan to revive the 3.0 schedule some time  next year. With EC and
> >>JDK8 in a good spot, the only big feature remaining  is classpath
> >>isolation.
> >>
> >> EC is also a pretty fundamental change to HDFS. Even if it's
> >>compatible, in  terms of size and impact it might best belong in a new
> >>major release.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> >> vinayakumarb.apa...@gmail.com> wrote:
> >>
> >>> Is anyone else also thinks that feature is ready to goto branch-2
> >>>as well?
> >>>
> >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since
> >>>then and  ready to go in branch-2.
> >>>
> >>> -Vinay
> >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
> >>>
>  Thanks Vinay for capturing the issue and Uma for offering the help.
> 
>  ---
>  Zhe Zhang
> 
>  On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
> >>> uma.ganguma...@intel.com
> >
>  wrote:
> 
> > Vinay,
> >
> >
> > I would merge them as part of HDFS-9182.
> >
> > Thanks,
> > Uma
> >
> >
> >
> > On 10/5/15, 12:48 AM, "Vinayakumar B" 
> >wrote:
> >
> >> Hi Andrew,
> >> I see CHANGES.txt entries not yet merged from
> >>> CHANGES-HDFS-EC-7285.txt.
> >>
> >> Was this intentional?
> >>
> >> Regards,
> >> Vinay
> >>
> >> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
> >>> andrew.w...@cloudera.com>
> >> wrote:
> >>
> >>> Branch has been merged to trunk, thanks again to everyone who
> >>>worked
>  on
> >>> the
> >>> feature!
> >>>
> >>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang
> >>> 
> >>> wrote:
> >>>
>  Thanks everyone who has participated in this discussion.
> 
>  With 7 +1's (5 binding and 2 non-binding), and no -1, this vote
> >>> has
> >>> passed.
>  I will do a final 'git merge' with trunk and work with Andrew
>  to
>  merge
> >>> the
>  branch to trunk. I'll update on this thread when the merge is
> >>> done.
> 
>  ---
>  Zhe Zhang
> 
>  On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A
>  

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Jing Zhao]

+1 for the plan about Phase I & II.

BTW, maybe out of the scope of this thread, just want to mention we should
either move the jira under HDFS-8031 or update the jira component as
"erasure-coding" when making further improvement or fixing bugs in EC. In
this way it will be easier for later backporting EC to 2.9.

On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B  wrote:

> +1 for the idea.
> On Nov 3, 2015 07:22, "Zheng, Kai"  wrote:
>
> > Sounds good to me. When it's determined to include EC in 2.9 release, it
> > may be good to have a rough release date as Zhe asked, so accordingly the
> > scope of EC can be discussed out. We still have quite a few of things as
> > Phase I follow-on tasks to do before EC can be deployed in a production
> > system. Phase II to develop non-striping EC for cold data would possibly
> be
> > started after that. We might consider to include only Phase I and leave
> > Phase II for next release according to the rough release date.
> >
> > Regards,
> > Kai
> >
> > -Original Message-
> > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com]
> > Sent: Tuesday, November 03, 2015 5:41 AM
> > To: hdfs-dev@hadoop.apache.org
> > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285
> > (erasure coding) branch to trunk]
> >
> > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan
> > +to
> > have 2.8 and 2.9 releases.
> >
> > Regards,
> > Uma
> >
> > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" 
> wrote:
> >
> > >Forking the thread. Started looking at the 2.8 list, various features¹
> > >status and arrived here.
> > >
> > >While I understand the pervasive nature of EC and a need for a
> > >significant bake-in, moving this to a 3.x release is not a good idea.
> > >We will surely get a 2.8 out this year and, as needed, I can even spend
> > >time getting started on a 2.9. OTOH, 3.x is long ways off, and given
> > >all the incompatibilities there, it would be a while before users can
> > >get their hands on EC if it were to be only on 3.x. At best, this may
> > >force sites that want EC to backport the entire EC feature to older
> > >releases, at worst this will be repeat the mess of 0.20 security release
> > forks.
> > >
> > >If we think adding this to 2.8 (even if it switched off) is too much
> > >risk per our original plan, let¹s move this to 2.9, there by leaving
> > >enough time for stability, integration testing and bake-in, and a
> > >realistic chance of having it end up on users¹ clusters soonish.
> > >
> > >+Vinod
> > >
> > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang 
> > >>wrote:
> > >>
> > >> I think our plan thus far has been to target this for 3.0. I'm okay
> > >>with  putting it in branch-2 if we've given a hard look at
> > >>compatibility, but  I'll note though that 2.8 is already looking like
> > >>quite a large release,  and our release bandwidth has been focused on
> > >>the 2.6 and 2.7 maintenance  releases. Adding another multi-hundred
> > >>JIRAs to 2.8 might make it too  unwieldy to get out the door. If we
> > >>bump EC past that, 3.0 might very well  be our next release vehicle. I
> > >>do plan to revive the 3.0 schedule some time  next year. With EC and
> > >>JDK8 in a good spot, the only big feature remaining  is classpath
> > >>isolation.
> > >>
> > >> EC is also a pretty fundamental change to HDFS. Even if it's
> > >>compatible, in  terms of size and impact it might best belong in a new
> > >>major release.
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> > >> vinayakumarb.apa...@gmail.com> wrote:
> > >>
> > >>> Is anyone else also thinks that feature is ready to goto branch-2
> > >>>as well?
> > >>>
> > >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since
> > >>>then and  ready to go in branch-2.
> > >>>
> > >>> -Vinay
> > >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
> > >>>
> >  Thanks Vinay for capturing the issue and Uma for offering the help.
> > 
> >  ---
> >  Zhe Zhang
> > 
> >  On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
> > >>> uma.ganguma...@intel.com
> > >
> >  wrote:
> > 
> > > Vinay,
> > >
> > >
> > > I would merge them as part of HDFS-9182.
> > >
> > > Thanks,
> > > Uma
> > >
> > >
> > >
> > > On 10/5/15, 12:48 AM, "Vinayakumar B" 
> > >wrote:
> > >
> > >> Hi Andrew,
> > >> I see CHANGES.txt entries not yet merged from
> > >>> CHANGES-HDFS-EC-7285.txt.
> > >>
> > >> Was this intentional?
> > >>
> > >> Regards,
> > >> Vinay
> > >>
> > >> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
> > >>> andrew.w...@cloudera.com>
> > >> wrote:
> > >>
> > >>> Branch has been merged to trunk, thanks again to everyone who
> > >>>worked
> >  on
> > >>> the

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Haohui Mai]

+1 on putting EC on 2.9.

Is it a good time to start the discussion on the issues of releasing 2.8?

~Haohui

On Mon, Nov 2, 2015 at 1:40 PM, Gangumalla, Uma
 wrote:
> +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan to
> have 2.8 and 2.9 releases.
>
> Regards,
> Uma
>
> On 11/2/15, 11:49 AM, "Vinod Vavilapalli"  wrote:
>
>>Forking the thread. Started looking at the 2.8 list, various features¹
>>status and arrived here.
>>
>>While I understand the pervasive nature of EC and a need for a
>>significant bake-in, moving this to a 3.x release is not a good idea. We
>>will surely get a 2.8 out this year and, as needed, I can even spend time
>>getting started on a 2.9. OTOH, 3.x is long ways off, and given all the
>>incompatibilities there, it would be a while before users can get their
>>hands on EC if it were to be only on 3.x. At best, this may force sites
>>that want EC to backport the entire EC feature to older releases, at
>>worst this will be repeat the mess of 0.20 security release forks.
>>
>>If we think adding this to 2.8 (even if it switched off) is too much risk
>>per our original plan, let¹s move this to 2.9, there by leaving enough
>>time for stability, integration testing and bake-in, and a realistic
>>chance of having it end up on users¹ clusters soonish.
>>
>>+Vinod
>>
>>> On Oct 19, 2015, at 1:44 PM, Andrew Wang 
>>>wrote:
>>>
>>> I think our plan thus far has been to target this for 3.0. I'm okay with
>>> putting it in branch-2 if we've given a hard look at compatibility, but
>>> I'll note though that 2.8 is already looking like quite a large release,
>>> and our release bandwidth has been focused on the 2.6 and 2.7
>>>maintenance
>>> releases. Adding another multi-hundred JIRAs to 2.8 might make it too
>>> unwieldy to get out the door. If we bump EC past that, 3.0 might very
>>>well
>>> be our next release vehicle. I do plan to revive the 3.0 schedule some
>>>time
>>> next year. With EC and JDK8 in a good spot, the only big feature
>>>remaining
>>> is classpath isolation.
>>>
>>> EC is also a pretty fundamental change to HDFS. Even if it's
>>>compatible, in
>>> terms of size and impact it might best belong in a new major release.
>>>
>>> Best,
>>> Andrew
>>>
>>> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
>>> vinayakumarb.apa...@gmail.com> wrote:
>>>
 Is anyone else also thinks that feature is ready to goto branch-2  as
well?

 Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since
then and
 ready to go in branch-2.

 -Vinay
 On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:

> Thanks Vinay for capturing the issue and Uma for offering the help.
>
> ---
> Zhe Zhang
>
> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
 uma.ganguma...@intel.com
>>
> wrote:
>
>> Vinay,
>>
>>
>> I would merge them as part of HDFS-9182.
>>
>> Thanks,
>> Uma
>>
>>
>>
>> On 10/5/15, 12:48 AM, "Vinayakumar B" 
>>wrote:
>>
>>> Hi Andrew,
>>> I see CHANGES.txt entries not yet merged from
 CHANGES-HDFS-EC-7285.txt.
>>>
>>> Was this intentional?
>>>
>>> Regards,
>>> Vinay
>>>
>>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang <
 andrew.w...@cloudera.com>
>>> wrote:
>>>
 Branch has been merged to trunk, thanks again to everyone who
worked
> on
 the
 feature!

 On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang 
 wrote:

> Thanks everyone who has participated in this discussion.
>
> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote
 has
 passed.
> I will do a final 'git merge' with trunk and work with Andrew to
> merge
 the
> branch to trunk. I'll update on this thread when the merge is
 done.
>
> ---
> Zhe Zhang
>
> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A 
 wrote:
>
>> (Change it to binding.)
>>
>> +1
>> I have been involved in the development and code review on the
 feature
>> branch. It's a great feature and I think it's ready to merge it
> into
> trunk.
>>
>> Thanks all for the contribution.
>>
>> Regards,
>> Yi Liu
>>
>>
>> -Original Message-
>> From: Liu, Yi A
>> Sent: Friday, September 25, 2015 1:51 PM
>> To: hdfs-dev@hadoop.apache.org
>> Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to
> trunk
>>
>> +1 (non-binding)
>> I have been involved in the development and code review on the
 feature
>> branch. It's a great 

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Vinod Vavilapalli]

Yes, I’ve already started looking at 2.8.0, that is exactly how I ended up with 
this discussion on the state of EC.

+Vinod


On Nov 2, 2015, at 3:02 PM, Haohui Mai 
> wrote:

Is it a good time to start the discussion on the issues of releasing 2.8?

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Andrew Wang]

If we use an umbrella JIRA to categorize all the ongoing EC work, that will
let us more easily change the target version later. For instance, if we
decide to bump Phase II out of 2.9, then we just need to change the target
version of the Phase II umbrella rather than all the subtasks.

On Mon, Nov 2, 2015 at 4:26 PM, Zheng, Kai  wrote:

> Yeah, so for the issues we recently resolved on trunk and are addressing
> as follow-on tasks in Phase I, we would label them with "erasure coding"
> and maybe also set the target version as "2.9" for the convenience?
>
> -Original Message-
> From: Jing Zhao [mailto:ji...@apache.org]
> Sent: Tuesday, November 03, 2015 8:04 AM
> To: hdfs-dev@hadoop.apache.org
> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285
> (erasure coding) branch to trunk]
>
> +1 for the plan about Phase I & II.
>
> BTW, maybe out of the scope of this thread, just want to mention we should
> either move the jira under HDFS-8031 or update the jira component as
> "erasure-coding" when making further improvement or fixing bugs in EC. In
> this way it will be easier for later backporting EC to 2.9.
>
> On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B <
> vinayakumarb.apa...@gmail.com
> > wrote:
>
> > +1 for the idea.
> > On Nov 3, 2015 07:22, "Zheng, Kai"  wrote:
> >
> > > Sounds good to me. When it's determined to include EC in 2.9
> > > release, it may be good to have a rough release date as Zhe asked,
> > > so accordingly the scope of EC can be discussed out. We still have
> > > quite a few of things as Phase I follow-on tasks to do before EC can
> > > be deployed in a production system. Phase II to develop non-striping
> > > EC for cold data would possibly
> > be
> > > started after that. We might consider to include only Phase I and
> > > leave Phase II for next release according to the rough release date.
> > >
> > > Regards,
> > > Kai
> > >
> > > -Original Message-
> > > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com]
> > > Sent: Tuesday, November 03, 2015 5:41 AM
> > > To: hdfs-dev@hadoop.apache.org
> > > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge
> > > HDFS-7285 (erasure coding) branch to trunk]
> > >
> > > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we
> > > +plan to
> > > have 2.8 and 2.9 releases.
> > >
> > > Regards,
> > > Uma
> > >
> > > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" 
> > wrote:
> > >
> > > >Forking the thread. Started looking at the 2.8 list, various
> > > >features¹ status and arrived here.
> > > >
> > > >While I understand the pervasive nature of EC and a need for a
> > > >significant bake-in, moving this to a 3.x release is not a good idea.
> > > >We will surely get a 2.8 out this year and, as needed, I can even
> > > >spend time getting started on a 2.9. OTOH, 3.x is long ways off,
> > > >and given all the incompatibilities there, it would be a while
> > > >before users can get their hands on EC if it were to be only on
> > > >3.x. At best, this may force sites that want EC to backport the
> > > >entire EC feature to older releases, at worst this will be repeat
> > > >the mess of 0.20 security release
> > > forks.
> > > >
> > > >If we think adding this to 2.8 (even if it switched off) is too
> > > >much risk per our original plan, let¹s move this to 2.9, there by
> > > >leaving enough time for stability, integration testing and bake-in,
> > > >and a realistic chance of having it end up on users¹ clusters soonish.
> > > >
> > > >+Vinod
> > > >
> > > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang
> > > >>
> > > >>wrote:
> > > >>
> > > >> I think our plan thus far has been to target this for 3.0. I'm
> > > >>okay with  putting it in branch-2 if we've given a hard look at
> > > >>compatibility, but  I'll note though that 2.8 is already looking
> > > >>like quite a large release,  and our release bandwidth has been
> > > >>focused on the 2.6 and 2.7 maintenance  releases. Adding another
> > > >>multi-hundred JIRAs to 2.8 might make it too  unwieldy to get out
> > > >>the door. If we bump EC past that, 3.0 might very well  be our
> > > >>next release vehicle. I do plan to revive the 3.0 schedule some
> > > >>time  next year. With EC and
> > > >>JDK8 in a good spot, the only big feature remaining  is classpath
> > > >>isolation.
> > > >>
> > > >> EC is also a pretty fundamental change to HDFS. Even if it's
> > > >>compatible, in  terms of size and impact it might best belong in a
> > > >>new major release.
> > > >>
> > > >> Best,
> > > >> Andrew
> > > >>
> > > >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B <
> > > >> vinayakumarb.apa...@gmail.com> wrote:
> > > >>
> > > >>> Is anyone else also thinks that feature is ready to goto
> > > >>>branch-2 as well?
> > > >>>
> > > >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable
> > > >>>since then and  ready to go in branch-2.
> > > >>>
> > > >>> 

[jira] [Created] (HDFS-9363) Add fetchReplica() to FsDatasetTestUtils()

[Tony Wu (JIRA)]

Tony Wu created HDFS-9363:
-

 Summary: Add fetchReplica() to FsDatasetTestUtils()
 Key: HDFS-9363
 URL: https://issues.apache.org/jira/browse/HDFS-9363
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: HDFS, test
Affects Versions: 2.7.1
Reporter: Tony Wu
Assignee: Tony Wu
Priority: Minor


{{FsDatasetTestUtils()}} abstracts away the details in {{FsDataset}} to allow 
writing generic tests regardless of underlying {{FsDataset}} implementations. 
We can add a {{fetchReplica()}} method to allow some HDFS tests to avoid using 
{{FsDatasetTestUtil#fetchReplicaInfo()}}, which assumes FsDatasetImpl is the 
only implementation of FsDataset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #564

[Apache Jenkins Server]

See 

Changes:

[zhz] HDFS-8777. Erasure Coding: add tests for taking snapshots on EC files.

[aajisaka] MAPREDUCE-6525. Fix test failure of 
TestMiniMRClientCluster.testRestart.

[zhz] HDFS-9329. TestBootstrapStandby#testRateThrottling is flaky because

[cnauroth] HADOOP-12533. Introduce FileNotFoundException in WASB for read and 
seek

[cnauroth] HADOOP-12508. delete fails with exception when lease is held on blob.

--
[...truncated 6840 lines...]
Running org.apache.hadoop.hdfs.TestLease
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.353 sec - in 
org.apache.hadoop.hdfs.TestLease
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.682 sec - in 
org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestHFlush
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.284 sec - 
in org.apache.hadoop.hdfs.TestHFlush
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestErasureCodingPolicies
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.44 sec - in 
org.apache.hadoop.hdfs.TestErasureCodingPolicies
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestRemoteBlockReader
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.835 sec - in 
org.apache.hadoop.hdfs.TestRemoteBlockReader
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestHdfsAdmin
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.912 sec - in 
org.apache.hadoop.hdfs.TestHdfsAdmin
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestDistributedFileSystem
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.38 sec - in 
org.apache.hadoop.hdfs.TestDistributedFileSystem
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestRollingUpgradeRollback
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.847 sec - in 
org.apache.hadoop.hdfs.TestRollingUpgradeRollback
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestRollingUpgrade
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 143.92 sec - 
in org.apache.hadoop.hdfs.TestRollingUpgrade
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestDatanodeDeath
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.337 sec - in 
org.apache.hadoop.hdfs.TestDatanodeDeath
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestCrcCorruption
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.558 sec - in 
org.apache.hadoop.hdfs.TestCrcCorruption
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestFsShellPermission
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.191 sec - in 
org.apache.hadoop.hdfs.TestFsShellPermission
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.TestLocatedBlock
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.241 sec - in 
org.apache.hadoop.hdfs.protocol.TestLocatedBlock
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.TestLayoutVersion
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.294 sec - in 
org.apache.hadoop.hdfs.protocol.TestLayoutVersion
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.118 sec - in 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.datatransfer.TestPacketReceiver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.41 sec - 

Hadoop-Hdfs-trunk-Java8 - Build # 564 - Failure

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/564/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 7033 lines...]
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS Client . SUCCESS [04:35 min]
[INFO] Apache Hadoop HDFS  FAILURE [  04:13 h]
[INFO] Apache Hadoop HDFS Native Client .. SKIPPED
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  0.059 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 04:17 h
[INFO] Finished at: 2015-11-03T01:26:30+00:00
[INFO] Final Memory: 55M/516M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-hdfs
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating HDFS-9329
Updating HADOOP-12508
Updating HADOOP-12533
Updating MAPREDUCE-6525
Updating HDFS-8777
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
3 tests failed.
FAILED:  
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics.testDataNodeTimeSpend

Error Message:
null

Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics.testDataNodeTimeSpend(TestDataNodeMetrics.java:289)


FAILED:  
org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes.testCircularLinkedListWrites

Error Message:
Some writers didn't complete in expected runtime! Current writer 
state:[Circular Writer:
  directory: /test-0
  target length: 50
  current item: 25
  done: false
, Circular Writer:
  directory: /test-1
  target length: 50
  current item: 30
  done: false
] expected:<0> but was:<2>

Stack Trace:
java.lang.AssertionError: Some writers didn't complete in expected runtime! 
Current writer state:[Circular Writer:
 directory: /test-0
 target length: 50
 current item: 25
 done: false
, Circular Writer:
 directory: /test-1
 target length: 50
 current item: 30
 done: false
] expected:<0> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 

RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

[Zheng, Kai]

Yeah, so for the issues we recently resolved on trunk and are addressing as 
follow-on tasks in Phase I, we would label them with "erasure coding" and maybe 
also set the target version as "2.9" for the convenience?

-Original Message-
From: Jing Zhao [mailto:ji...@apache.org] 
Sent: Tuesday, November 03, 2015 8:04 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 
(erasure coding) branch to trunk]

+1 for the plan about Phase I & II.

BTW, maybe out of the scope of this thread, just want to mention we should 
either move the jira under HDFS-8031 or update the jira component as 
"erasure-coding" when making further improvement or fixing bugs in EC. In this 
way it will be easier for later backporting EC to 2.9.

On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B  wrote:

> +1 for the idea.
> On Nov 3, 2015 07:22, "Zheng, Kai"  wrote:
>
> > Sounds good to me. When it's determined to include EC in 2.9 
> > release, it may be good to have a rough release date as Zhe asked, 
> > so accordingly the scope of EC can be discussed out. We still have 
> > quite a few of things as Phase I follow-on tasks to do before EC can 
> > be deployed in a production system. Phase II to develop non-striping 
> > EC for cold data would possibly
> be
> > started after that. We might consider to include only Phase I and 
> > leave Phase II for next release according to the rough release date.
> >
> > Regards,
> > Kai
> >
> > -Original Message-
> > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com]
> > Sent: Tuesday, November 03, 2015 5:41 AM
> > To: hdfs-dev@hadoop.apache.org
> > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge 
> > HDFS-7285 (erasure coding) branch to trunk]
> >
> > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we 
> > +plan to
> > have 2.8 and 2.9 releases.
> >
> > Regards,
> > Uma
> >
> > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" 
> wrote:
> >
> > >Forking the thread. Started looking at the 2.8 list, various 
> > >features¹ status and arrived here.
> > >
> > >While I understand the pervasive nature of EC and a need for a 
> > >significant bake-in, moving this to a 3.x release is not a good idea.
> > >We will surely get a 2.8 out this year and, as needed, I can even 
> > >spend time getting started on a 2.9. OTOH, 3.x is long ways off, 
> > >and given all the incompatibilities there, it would be a while 
> > >before users can get their hands on EC if it were to be only on 
> > >3.x. At best, this may force sites that want EC to backport the 
> > >entire EC feature to older releases, at worst this will be repeat 
> > >the mess of 0.20 security release
> > forks.
> > >
> > >If we think adding this to 2.8 (even if it switched off) is too 
> > >much risk per our original plan, let¹s move this to 2.9, there by 
> > >leaving enough time for stability, integration testing and bake-in, 
> > >and a realistic chance of having it end up on users¹ clusters soonish.
> > >
> > >+Vinod
> > >
> > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang 
> > >>
> > >>wrote:
> > >>
> > >> I think our plan thus far has been to target this for 3.0. I'm 
> > >>okay with  putting it in branch-2 if we've given a hard look at 
> > >>compatibility, but  I'll note though that 2.8 is already looking 
> > >>like quite a large release,  and our release bandwidth has been 
> > >>focused on the 2.6 and 2.7 maintenance  releases. Adding another 
> > >>multi-hundred JIRAs to 2.8 might make it too  unwieldy to get out 
> > >>the door. If we bump EC past that, 3.0 might very well  be our 
> > >>next release vehicle. I do plan to revive the 3.0 schedule some 
> > >>time  next year. With EC and
> > >>JDK8 in a good spot, the only big feature remaining  is classpath 
> > >>isolation.
> > >>
> > >> EC is also a pretty fundamental change to HDFS. Even if it's 
> > >>compatible, in  terms of size and impact it might best belong in a 
> > >>new major release.
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < 
> > >> vinayakumarb.apa...@gmail.com> wrote:
> > >>
> > >>> Is anyone else also thinks that feature is ready to goto 
> > >>>branch-2 as well?
> > >>>
> > >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable 
> > >>>since then and  ready to go in branch-2.
> > >>>
> > >>> -Vinay
> > >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang"  wrote:
> > >>>
> >  Thanks Vinay for capturing the issue and Uma for offering the help.
> > 
> >  ---
> >  Zhe Zhang
> > 
> >  On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma <
> > >>> uma.ganguma...@intel.com
> > >
> >  wrote:
> > 
> > > Vinay,
> > >
> > >
> > > I would merge them as part of HDFS-9182.
> > >
> > > Thanks,
> > > Uma
> > >
> > >
> > >
> > > On 10/5/15, 12:48 AM, 

[jira] [Created] (HDFS-9364) Unnecessary DNS resolution attempts when NameNodeProxies

[Xiao Chen (JIRA)]

Xiao Chen created HDFS-9364:
---

 Summary: Unnecessary DNS resolution attempts when NameNodeProxies
 Key: HDFS-9364
 URL: https://issues.apache.org/jira/browse/HDFS-9364
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Xiao Chen
Assignee: Xiao Chen


When creating NameNodeProxies, we always try to DNS-resolve namenode URIs. This 
is unnecessary if the URI is logical, and may be significantly slow if the DNS 
is having problems. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9365) Balaner should call getNNServiceRpcAddressesForCluster after HDFS-6376

[Tsz Wo Nicholas Sze (JIRA)]

Tsz Wo Nicholas Sze created HDFS-9365:
-

 Summary: Balaner should call getNNServiceRpcAddressesForCluster 
after HDFS-6376
 Key: HDFS-9365
 URL: https://issues.apache.org/jira/browse/HDFS-9365
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer & mover
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze


HDFS-6376 added support for DistCp between two HA clusters.  After the change, 
Balaner will use all the NN from both the local and the remote clusters.  It 
should call getNNServiceRpcAddressesForCluster and only use the local cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hadoop-Hdfs-trunk - Build # 2501 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2501/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 9563 lines...]
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Skipping javadoc generation
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS Client . SUCCESS [04:00 min]
[INFO] Apache Hadoop HDFS  FAILURE [  03:18 h]
[INFO] Apache Hadoop HDFS Native Client .. SKIPPED
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  0.096 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 03:22 h
[INFO] Finished at: 2015-11-03T03:04:40+00:00
[INFO] Final Memory: 57M/755M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-hdfs
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating HDFS-9329
Updating HADOOP-12508
Updating HADOOP-12533
Updating HDFS-9339
Updating MAPREDUCE-6525
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
23 tests failed.
FAILED:  org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt

Error Message:
test timed out after 5 milliseconds

Stack Trace:
java.lang.Exception: test timed out after 5 milliseconds
at java.lang.Object.wait(Native Method)
at 
org.apache.hadoop.hdfs.DataStreamer.waitForAckedSeqno(DataStreamer.java:763)
at 
org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:700)
at 
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:774)
at 
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:753)
at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at 
org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt(TestCrcCorruption.java:134)


FAILED:  org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testDelegationToken

Error Message:
org.apache.hadoop.security.authentication.client.AuthenticationException: 
Authentication failed, status: 500, message: 
org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter$KMSResponse

Stack Trace:
java.io.IOException: 
org.apache.hadoop.security.authentication.client.AuthenticationException: 
Authentication failed, status: 500, message: 
org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter$KMSResponse
at 
org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
  

Build failed in Jenkins: Hadoop-Hdfs-trunk #2501

[Apache Jenkins Server]

See 

Changes:

[aajisaka] MAPREDUCE-6525. Fix test failure of 
TestMiniMRClientCluster.testRestart.

[zhz] HDFS-9329. TestBootstrapStandby#testRateThrottling is flaky because

[cnauroth] HADOOP-12533. Introduce FileNotFoundException in WASB for read and 
seek

[cnauroth] HADOOP-12508. delete fails with exception when lease is held on blob.

[zhz] HDFS-9339. Extend full test of KMS ACLs. Contributed by Daniel

--
[...truncated 9370 lines...]
Running org.apache.hadoop.hdfs.TestMissingBlocksAlert
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.975 sec - in 
org.apache.hadoop.hdfs.TestMissingBlocksAlert
Running org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.853 sec - in 
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
Running org.apache.hadoop.hdfs.TestFileStatus
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.966 sec - in 
org.apache.hadoop.hdfs.TestFileStatus
Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.509 sec - 
in org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
Running org.apache.hadoop.hdfs.TestBalancerBandwidth
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.222 sec - in 
org.apache.hadoop.hdfs.TestBalancerBandwidth
Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.416 sec - 
in org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
Running org.apache.hadoop.hdfs.TestSetTimes
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.821 sec - in 
org.apache.hadoop.hdfs.TestSetTimes
Running org.apache.hadoop.TestGenericRefresh
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.536 sec - in 
org.apache.hadoop.TestGenericRefresh
Running org.apache.hadoop.tracing.TestTracing
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.763 sec - in 
org.apache.hadoop.tracing.TestTracing
Running org.apache.hadoop.tracing.TestTracingShortCircuitLocalRead
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.988 sec - in 
org.apache.hadoop.tracing.TestTracingShortCircuitLocalRead
Running org.apache.hadoop.tracing.TestTraceAdmin
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.564 sec - in 
org.apache.hadoop.tracing.TestTraceAdmin
Running org.apache.hadoop.security.TestPermission
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.355 sec - in 
org.apache.hadoop.security.TestPermission
Running org.apache.hadoop.security.TestPermissionSymlinks
Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.892 sec - in 
org.apache.hadoop.security.TestPermissionSymlinks
Running org.apache.hadoop.security.TestRefreshUserMappings
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.999 sec - in 
org.apache.hadoop.security.TestRefreshUserMappings
Running org.apache.hadoop.fs.TestFcHdfsSetUMask
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.119 sec - in 
org.apache.hadoop.fs.TestFcHdfsSetUMask
Running org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
Tests run: 74, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 10.259 sec - 
in org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
Running org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.014 sec - in 
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.422 sec - in 
org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRename
Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.224 sec - in 
org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete
Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.496 sec - in 
org.apache.hadoop.fs.contract.hdfs.TestHDFSContractAppend
Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.21 sec - in 
org.apache.hadoop.fs.contract.hdfs.TestHDFSContractOpen
Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.367 sec - in 
org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractConcat
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.171 sec - in 
org.apache.hadoop.fs.contract.hdfs.TestHDFSContractConcat
Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: