Re: CHANGES.TXT

2015-09-23 Thread Chris Douglas 
On Mon, Sep 14, 2015 at 1:09 PM, Andrew Wang  wrote:
> I put some time into this a little while back, doing releasedocmaker
> backports from the Yetus branch. IIRC from Allen, we still need to get lint
> mode working before it's good to go. Probably a good bit of JIRA cleanup
> will be required to get a clean lint run.

The lint tests will evolve as we discover new inconsistencies. Does it
need to be a prerequisite?

We've had this discussion before. Using JIRA is at least as accurate
as CHANGES.txt, we should just use Allen's script (now part of Yetus?)
to generate the release notes, and stop rehashing this. If someone
wants to cross-reference other sources to improve accuracy, that would
be fine followup. -C

> On Sun, Sep 13, 2015 at 4:38 AM, Steve Loughran 
> wrote:
>
>>
>> > On 12 Sep 2015, at 20:26, Allen Wittenauer  wrote:
>> >
>> >
>> > On Sep 12, 2015, at 12:25 PM, Allen Wittenauer  wrote:
>> >
>> >>
>> >> On Sep 12, 2015, at 11:03 AM, Steve Loughran 
>> wrote:
>> >>
>> >>>
>> >>> 2. go to JIRA-generated change logs. Though for that to be reliable,
>> those fix-version fields have to be 100% accurate too.
>> >
>> >
>> > P.S.,
>> https://github.com/aw-altiscale/eco-release-metadata/blob/master/HADOOP/README.md
>> >
>> >
>>
>> nice. I'd prefer this, as long as we keep up on the versioning. In
>> particular, any cherry picking back to a point release, 2.7.2 etc will need
>> the entry being re-edited
>>
>> in the meantime, I celebrate Haohui's volunteering -though need to warn it
>> was the thought of making the 2.7.x and 2.6.x changes consistent which
>> worried me there -you end up having to re-arrange entries across four files
>>
>> -steve
>>

Build failed in Jenkins: Hadoop-common-trunk-Java8 #446

2015-09-23 Thread Apache Jenkins Server 
See 

Changes:

[wheat9] HADOOP-12438. Reset RawLocalFileSystem.useDeprecatedFileStatus in 
TestLocalFileSystem. Contributed by Chris Nauroth.

[wheat9] HDFS-9128. TestWebHdfsFileContextMainOperations and 
TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows. 
Contributed by Chris Nauroth.

--
[...truncated 5603 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 sec - in 
org.apache.hadoop.io.TestTextNonUTF8
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestMD5Hash
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.191 sec - in 
org.apache.hadoop.io.TestMD5Hash
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSequenceFileAppend
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.738 sec - in 
org.apache.hadoop.io.TestSequenceFileAppend
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestMapFile
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.166 sec - in 
org.apache.hadoop.io.TestMapFile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestWritableName
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.153 sec - in 
org.apache.hadoop.io.TestWritableName
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSortedMapWritable
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.218 sec - in 
org.apache.hadoop.io.TestSortedMapWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSequenceFileSync
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.795 sec - in 
org.apache.hadoop.io.TestSequenceFileSync
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestFailoverProxy
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.077 sec - in 
org.apache.hadoop.io.retry.TestFailoverProxy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestDefaultRetryPolicy
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.352 sec - in 
org.apache.hadoop.io.retry.TestDefaultRetryPolicy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestRetryProxy
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.217 sec - in 
org.apache.hadoop.io.retry.TestRetryProxy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestDefaultStringifier
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.336 sec - in 
org.apache.hadoop.io.TestDefaultStringifier
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestBloomMapFile
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.333 sec - in 
org.apache.hadoop.io.TestBloomMapFile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestBytesWritable
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.187 sec - in 
org.apache.hadoop.io.TestBytesWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestWritableUtils
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.14 sec - in 
org.apache.hadoop.io.TestWritableUtils
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestBooleanWritable
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.16 sec - in 
org.apache.hadoop.io.TestBooleanWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestDataByteBuffers
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.332 sec - in 
org.apache.hadoop.io.TestDataByteBuffers
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestVersionedWritable
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.158 sec - in 
org.apache.hadoop.io.TestVersionedWritable
Java HotSpot(TM) 64-Bit

Re: [RESULT][VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-23 Thread Vinod Vavilapalli 
Done with all the release activities. Waiting for the bits to propagate to all 
the mirrors.

Thanks
+Vinod

On Sep 23, 2015, at 1:18 PM, Vinod Kumar Vavilapalli 
mailto:vino...@hortonworks.com>> wrote:

Here’s my +1 to conclude the vote.

With 5 binding +1s (Xuan, Jian, Junping, Karthik, myself), 8 non-binding +1s 
(Wangda, Sanjin, Rohith, Brahma, Mit, Akira, Tsuyoshi, Masatake ), 4 
non-committal yes’s (Ted, Eric, Chang, Kuhu), this vote passes.

I’ll push out the bits.

Thanks to everyone who voted on this release!

Thanks
+Vinod


On Sep 16, 2015, at 7:10 PM, Vinod Kumar Vavilapalli 
mailto:vino...@apache.org>> wrote:

Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and 
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release 
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes for 
the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779, YARN-3248), 
additional fix to avoid incompatibility (YARN-3740), other UI bugs (YARN-1884, 
YARN-3544) and the MiniYARNCluster issue (right patch for YARN-2890) that Jeff 
Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via 
repository.apache.org at 
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
 -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] - 
non-committed but desired patches. 2.6.1 is already big as is and is late by 
any standard, we can definitely include them in the next release.
 - The 2.6.1 wiki page [3] captures some (but not all) of the context of the 
patches that we pushed in.
 - Given the number of fixes pushed [4] in, we had to make a bunch of changes 
to our original plan - we added a few improvements that helped us backport 
patches easier (or in many cases made backports possible), and we dropped a few 
that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676, HDFS-7611, HDFS-7843, 
HDFS-8850).
 - I ran all the unit tests which (surprisingly?) passed. (Except for one, 
which pointed out a missing fix HDFS-7552).

As discussed before [5]
 - This release is the first point release after 2.6.0
 - I’d like to use this as a starting release for 2.6.2 in a few weeks and then 
follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread: 
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets: 
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes
[4] List of 2.6.1 patches pushed: 
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release: http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
 - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these 
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off 2.6.1.
 - The additional patches in RC1 that got into 2.6.1 all the way from 2.8 are 
NOT in 2.7.2 yet, this will be done as a followup.

[jira] [Created] (HADOOP-12438) TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.

2015-09-23 Thread Chris Nauroth (JIRA) 
Chris Nauroth created HADOOP-12438:
--

 Summary: TestLocalFileSystem tests can fail on Windows after 
HDFS-8767 fix for handling pipe.
 Key: HADOOP-12438
 URL: https://issues.apache.org/jira/browse/HADOOP-12438
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Trivial


A test written as part of HDFS-8767 uses Mockito's 
{{Whitebox#setInternalState}} to force the static 
{{RawLocalFileSystem#useDeprecatedFileStatus}} to {{false}}.  On systems that 
don't implement a {{stat}} call (notably Windows), this leaves the class in a 
state where it will attempt the {{stat}} call anyway during subsequent tests.  
This ultimately causes tests to fail with {{UnsupportedOperationException}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: CHANGES.TXT

2015-09-23 Thread Colin P. McCabe 
I don't understand your negative tone.  What point specifically did I
and other people in the conversation miss?

Colin

On Tue, Sep 22, 2015 at 6:21 PM, Allen Wittenauer  wrote:
>
> I don’t whether your ability to completely miss my point every time we 
> communicate with each other, regardless of the issue, is intentional or just 
> a special talent.
>
> On Sep 22, 2015, at 8:52 AM, Colin P. McCabe  wrote:
>
>> I think it's extremely unrealistic to expect Hadoop to ever follow a
>> branchless development model.  In fact, the recent trend has been in
>> the opposite direction-- prominent members of the community have
>> pointed out that we don't have enough long-running, well-tested and
>> well-supported branches.  Producing such a branch was the goal of the
>> ongoing 2.6.1 release effort.  Even if we did somehow switch to a
>> branchless development model, we have numerous people backporting
>> patches to their own repositories-- both Hadoop vendors and large
>> organizations that run Hadoop internally and have their own branches.
>>
>> Branchless development especially doesn't make sense for HDFS, since
>> it would force people to do time-consuming and potentially risky
>> layout versions just to get small bugfixes.  Very few cluster
>> operators want to update the version of their data on-disk just to get
>> this month's urgent bugfix.  There are similar issues in other parts
>> of the stack such as YARN.
>>
>> Anyway, as Steve pointed out in his original post, merge conflicts in
>> CHANGES.txt are not the only problem caused by that file.  It's simply
>> very inaccurate and misleading, since it must be manually updated.  In
>> more than 3 years of working with Hadoop, I've never found CHANGES.txt
>> useful for anything.  git log and JIRA tell you everything you need to
>> know.  CHANGES.txt is a burden to update, misleading to operators, and
>> a relic that should have been removed years ago.
>>
>> I really hope this CHANGES.txt thread doesn't peter out like the rest
>> of them.  Please, let's fix this, finally.  Autogenerate this file.
>>
>> best,
>> Colin
>>
>> On Mon, Sep 14, 2015 at 7:10 PM, Allen Wittenauer  wrote:
>>>
>>> On Sep 14, 2015, at 5:15 PM, Colin P. McCabe  wrote:
>>>
 Let's stay focused on the title of the thread-- CHANGES.txt-- and
 discuss issues surrounding releasing trunk in a separate thread.
>>>
>>>
>>>It directly addresses the thread:  if one isn’t cherry picking 
>>> patches because there aren’t multiple primary branches in development, the 
>>> changes.txt conflicts effectively go away.
>

[RESULT][VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-23 Thread Vinod Vavilapalli 
Here’s my +1 to conclude the vote.

With 5 binding +1s (Xuan, Jian, Junping, Karthik, myself), 8 non-binding +1s 
(Wangda, Sanjin, Rohith, Brahma, Mit, Akira, Tsuyoshi, Masatake ), 4 
non-committal yes’s (Ted, Eric, Chang, Kuhu), this vote passes.

I’ll push out the bits.

Thanks to everyone who voted on this release!

Thanks
+Vinod


On Sep 16, 2015, at 7:10 PM, Vinod Kumar Vavilapalli 
mailto:vino...@apache.org>> wrote:

Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and 
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release 
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes for 
the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779, YARN-3248), 
additional fix to avoid incompatibility (YARN-3740), other UI bugs (YARN-1884, 
YARN-3544) and the MiniYARNCluster issue (right patch for YARN-2890) that Jeff 
Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via 
repository.apache.org at 
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
 -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] - 
non-committed but desired patches. 2.6.1 is already big as is and is late by 
any standard, we can definitely include them in the next release.
 - The 2.6.1 wiki page [3] captures some (but not all) of the context of the 
patches that we pushed in.
 - Given the number of fixes pushed [4] in, we had to make a bunch of changes 
to our original plan - we added a few improvements that helped us backport 
patches easier (or in many cases made backports possible), and we dropped a few 
that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676, HDFS-7611, HDFS-7843, 
HDFS-8850).
 - I ran all the unit tests which (surprisingly?) passed. (Except for one, 
which pointed out a missing fix HDFS-7552).

As discussed before [5]
 - This release is the first point release after 2.6.0
 - I’d like to use this as a starting release for 2.6.2 in a few weeks and then 
follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread: 
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets: 
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes
[4] List of 2.6.1 patches pushed: 
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release: http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
 - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these 
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off 2.6.1.
 - The additional patches in RC1 that got into 2.6.1 all the way from 2.8 are 
NOT in 2.7.2 yet, this will be done as a followup.

Re: Local repo sharing for maven builds

2015-09-23 Thread Vinayakumar B 
In case if we are going to have separate repo for each executor,

I have checked, each jenkins node is allocated 2 executors. so we only need
to create one more replica.

Regards,
Vinay

On Wed, Sep 23, 2015 at 7:33 PM, Steve Loughran 
wrote:

>
> > On 22 Sep 2015, at 16:39, Colin P. McCabe  wrote:
> >
> >> ANNOUNCEMENT: new patches which contain hard-coded ports in test runs
> will henceforth be reverted. Jenkins matters more than the 30s of your time
> it takes to use the free port finder methods. Same for any hard code paths
> in filesystems.
> >
> > +1.  Can you add this to HowToContribute on the wiki?  Or should we
> > vote on it first?
>
> I don't think we need to vote on it: hard code ports should be something
> we veto on patches anyway.
>
> In https://issues.apache.org/jira/browse/HADOOP-12143 I propose having a
> better style guide in the docs.
>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-23 Thread Vinod Vavilapalli 
I’ve been out sick for a while, so couldn’t close this in time. Doing so now.

+Vinod

On Sep 16, 2015, at 7:10 PM, Vinod Kumar Vavilapalli 
mailto:vino...@apache.org>> wrote:

Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and 
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release 
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes for 
the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779, YARN-3248), 
additional fix to avoid incompatibility (YARN-3740), other UI bugs (YARN-1884, 
YARN-3544) and the MiniYARNCluster issue (right patch for YARN-2890) that Jeff 
Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via 
repository.apache.org at 
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
 -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] - 
non-committed but desired patches. 2.6.1 is already big as is and is late by 
any standard, we can definitely include them in the next release.
 - The 2.6.1 wiki page [3] captures some (but not all) of the context of the 
patches that we pushed in.
 - Given the number of fixes pushed [4] in, we had to make a bunch of changes 
to our original plan - we added a few improvements that helped us backport 
patches easier (or in many cases made backports possible), and we dropped a few 
that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676, HDFS-7611, HDFS-7843, 
HDFS-8850).
 - I ran all the unit tests which (surprisingly?) passed. (Except for one, 
which pointed out a missing fix HDFS-7552).

As discussed before [5]
 - This release is the first point release after 2.6.0
 - I’d like to use this as a starting release for 2.6.2 in a few weeks and then 
follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread: 
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets: 
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes
[4] List of 2.6.1 patches pushed: 
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release: http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
 - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these 
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off 2.6.1.
 - The additional patches in RC1 that got into 2.6.1 all the way from 2.8 are 
NOT in 2.7.2 yet, this will be done as a followup.

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-23 Thread Vinod Vavilapalli 
Tx Eric, given the long cycle for 2.6.1, I’ll put MAPREDUCE-6334 into 2.6.2.

Thanks
+Vinod

On Sep 21, 2015, at 11:19 AM, Eric Payne 
mailto:eric.payne1...@yahoo.com>> wrote:

Hi Vinod and everyone else who helped on this release!

Thank you very much for going through the work and effort to put this release 
together.

While doing my testing for Hadoop 2.6.1 RC1, I encountered the following issue:
[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon IOException 
during InMemoryMapOutput shuffle handler - ASF 
JIRA












[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon IOException 
during ...
We are seeing this happen when an NM's disk goes bad during the creation of map 
output(s) the reducer's fetcher can read the shuffle header and reserve the 
memory


View on issues.apache.org

Preview by Yahoo





This is actually something we ran across frequently when running 2.6. You may 
want to consider pulling that in to 2.6.

Other than that, it looks fine. I did the following manual testing on a 
one-node cluster:
- successfully downloaded and compiled the code
- successfully ran streaming jobs
- successfully ran distributed shell jobs
-- as part of the distributed shell testing, I started the jobs with the 
'-keep_containers_across_application_attempts' property set, killed the AM 
container, and verified that it would in fact keep the containers running 
across AM restarts.
- sucessfully ran wordcount jobs.

Thank you,
-Eric Payne


From: Vinod Kumar Vavilapalli mailto:vino...@apache.org>>
To: common-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org
Cc: vino...@apache.org; Sangjin Lee 
mailto:sjl...@gmail.com>>; Akira AJISAKA 
mailto:ajisa...@oss.nttdata.co.jp>>
Sent: Wednesday, September 16, 2015 9:10 PM
Subject: [VOTE] Release Apache Hadoop 2.6.1 RC1

Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes
for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI
bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for
YARN-2890) that Jeff Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via 
repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
-  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
non-committed but desired patches. 2.6.1 is already big as is and is late
by any standard, we can definitely include them in the next release.
- The 2.6.1 wiki page [3] captures some (but not all) of the context of
the patches that we pushed in.
- Given the number of fixes pushed [4] in, we had to make a bunch of
changes to our original plan - we added a few improvements that helped us
backport patches easier (or in many cases made backports possible), and we
dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
HDFS-7611, HDFS-7843, HDFS-8850).
- I ran all the unit tests which (surprisingly?) passed. (Except for one,
which pointed out a missing fix HDFS-7552).

As discussed before [5]
- This release is the first point release after 2.6.0
- I’d like to use this as a starting release for 2.6.2 in a few weeks and
then follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread:
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets:
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
-Working-Notes
[4] List of 2.6.1 patches pushed:
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release:
http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
- Note that branch-2.6 which will be the base for 2.6.2 doesn't have these
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off
2.6.1.
- The additional patches in RC1 that got into 2.6.1 all the way from 2.8
are NOT in 2.7.2 yet, this will be done as a followup.

[jira] [Created] (HADOOP-12436) GlobPattern regex library has performance issues with wildcard characters

2015-09-23 Thread Matthew (JIRA) 
Matthew created HADOOP-12436:


 Summary: GlobPattern regex library has performance issues with 
wildcard characters
 Key: HADOOP-12436
 URL: https://issues.apache.org/jira/browse/HADOOP-12436
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.7.1, 2.2.0
Reporter: Matthew
Assignee: Matthew


java.util.regex classes have performance problems with certain wildcard 
patterns.  Namely, consecutive * characters in a file name (not properly 
escaped as literals) will cause commands such as "hadoop fs -ls file**name" 
to consume 100% CPU and probably never return in a reasonable time (time scales 
with number of *'s). 

Here is an example:

{noformat}
hdfs dfs -ls 
'/tmp/job_1429571161900_4222-1430338332599-tda%2D%2D+**+++...%270%27%28Stage-1430338580443-39-2000-SUCCEEDED-production%2Dhigh-1430338340360.jhist'
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk

2015-09-23 Thread Jing Zhao 
+1

I've been involved in both development and review on the branch, and I
believe it's now ready to get merged into trunk. Many thanks to all the
contributors and reviewers!

Thanks,
-Jing

On Tue, Sep 22, 2015 at 6:17 PM, Zheng, Kai  wrote:

> Non-binding +1
>
> According to our extensive performance tests, striping + ISA-L coder based
> erasure coding not only can save storage, but also can increase the
> throughput of a client or a cluster. It will be a great addition to HDFS
> and its users. Based on the latest branch codes, we also observed it's very
> reliable in the concurrent tests. We'll provide the perf test report after
> it's sorted out and hope it helps.
> Thanks!
>
> Regards,
> Kai
>
> -Original Message-
> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com]
> Sent: Wednesday, September 23, 2015 8:50 AM
> To: hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org
> Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk
>
> +1
>
> Great addition to HDFS. Thanks all contributors for the nice work.
>
> Regards,
> Uma
>
> On 9/22/15, 3:40 PM, "Zhe Zhang"  wrote:
>
> >Hi,
> >
> >I'd like to propose a vote to merge the HDFS-7285 feature branch back
> >to trunk. Since November 2014 we have been designing and developing
> >this feature under the umbrella JIRAs HDFS-7285 and HADOOP-11264, and
> >have committed approximately 210 patches.
> >
> >The HDFS-7285 feature branch was created to support the first phase of
> >HDFS erasure coding (HDFS-EC). The objective of HDFS-EC is to
> >significantly reduce storage space usage in HDFS clusters. Instead of
> >always creating 3 replicas of each block with 200% storage space
> >overhead, HDFS-EC provides data durability through parity data blocks.
> >With most EC configurations, the storage overhead is no more than 50%.
> >Based on profiling results of production clusters, we decided to
> >support EC with the striped block layout in the first phase, so that
> >small files can be better handled. This means dividing each logical
> >HDFS file block into smaller units (striping cells) and spreading them
> >on a set of DataNodes in round-robin fashion. Parity cells are
> >generated for each stripe of original data cells. We have made changes
> >to NameNode, client, and DataNode to generalize the block concept and
> >handle the mapping between a logical file block and its internal
> >storage blocks. For further details please see the design doc on
> >HDFS-7285.
> >HADOOP-11264 focuses on providing flexible and high-performance codec
> >calculation support.
> >
> >The nightly Jenkins job of the branch has reported several successful
> >runs, and doesn't show new flaky tests compared with trunk. We have
> >posted several versions of the test plan including both unit testing
> >and cluster testing, and have executed most tests in the plan. The most
> >basic functionalities have been extensively tested and verified in
> >several real clusters with different hardware configurations; results
> >have been very stable. We have created follow-on tasks for more
> >advanced error handling and optimization under the umbrella HDFS-8031.
> >We also plan to implement or harden the integration of EC with existing
> >features such as WebHDFS, snapshot, append, truncate, hflush, hsync,
> >and so forth.
> >
> >Development of this feature has been a collaboration across many
> >companies and institutions. I'd like to thank J. Andreina, Takanobu
> >Asanuma, Vinayakumar B, Li Bo, Takuya Fukudome, Uma Maheswara Rao G,
> >Rui Li, Yi Liu, Colin McCabe, Xinwei Qin, Rakesh R, Gao Rui, Kai
> >Sasaki, Walter Su, Tsz Wo Nicholas Sze, Andrew Wang, Yong Zhang, Jing
> >Zhao, Hui Zheng and Kai Zheng for their code contributions and reviews.
> >Andrew and Kai Zheng also made fundamental contributions to the initial
> >design. Rui Li, Gao Rui, Kai Sasaki, Kai Zheng and many other
> >contributors have made great efforts in system testing. Many thanks go
> >to Weihua Jiang for proposing the JIRA, and ATM, Todd Lipcon, Silvius
> >Rus, Suresh, as well as many others for providing helpful feedbacks.
> >
> >Following the community convention, this vote will last for 7 days
> >(ending September 29th). Votes from Hadoop committers are binding but
> >non-binding votes are very welcome as well. And here's my non-binding +1.
> >
> >Thanks,
> >---
> >Zhe Zhang
>
>

[jira] [Created] (HADOOP-12435) Findbugs is broken on trunk

2015-09-23 Thread Mit Desai (JIRA) 
Mit Desai created HADOOP-12435:
--

 Summary: Findbugs is broken on trunk
 Key: HADOOP-12435
 URL: https://issues.apache.org/jira/browse/HADOOP-12435
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Mit Desai


Here is the stacktrace of the failure.
{noformat}


 Pre-patch trunk Java verification




Compiling /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build
/home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests 
-DhadoopPatchProcess -Ptest-patch > 
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkJavacWarnings.txt
 2>&1
Javadoc'ing /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build
/home/jenkins/tools/maven/latest/bin/mvn clean test javadoc:javadoc -DskipTests 
-Pdocs -DhadoopPatchProcess > 
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkJavadocWarnings.txt
 2>&1
Patch does not appear to need site tests.
findbugs baseline for /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build
  Running findbugs in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
/home/jenkins/tools/maven/latest/bin/mvn clean test findbugs:findbugs 
-DskipTests -DhadoopPatchProcess > 
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkFindBugsOutputhadoop-yarn-client.txt
 2>&1
Exception in thread "main" java.io.FileNotFoundException: 
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkFindbugsWarningshadoop-yarn-client.xml
 (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:146)
at 
edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231)
at 
edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308)
at 
edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295)
at edu.umd.cs.findbugs.workflow.Filter.main(Filter.java:712)
  Running findbugs in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
/home/jenkins/tools/maven/latest/bin/mvn clean test findbugs:findbugs 
-DskipTests -DhadoopPatchProcess > 
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkFindBugsOutputhadoop-yarn-server-web-proxy.txt
 2>&1
Pre-patch trunk findbugs is broken?

Elapsed time:  16m  1s
{noformat}

Link to the console output of the run. Seen in multiple places.
1. https://builds.apache.org/job/PreCommit-YARN-Build/9243/console
2. https://builds.apache.org/job/PreCommit-YARN-Build/9241/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Local repo sharing for maven builds

2015-09-23 Thread Steve Loughran 

> On 22 Sep 2015, at 16:39, Colin P. McCabe  wrote:
> 
>> ANNOUNCEMENT: new patches which contain hard-coded ports in test runs will 
>> henceforth be reverted. Jenkins matters more than the 30s of your time it 
>> takes to use the free port finder methods. Same for any hard code paths in 
>> filesystems.
> 
> +1.  Can you add this to HowToContribute on the wiki?  Or should we
> vote on it first?

I don't think we need to vote on it: hard code ports should be something we 
veto on patches anyway. 

In https://issues.apache.org/jira/browse/HADOOP-12143 I propose having a better 
style guide in the docs.

Jenkins build is back to normal : Hadoop-common-trunk-Java8 #441

2015-09-23 Thread Apache Jenkins Server 
See