Build failed in Jenkins: Hadoop-common-trunk-Java8 #435

2015-09-21 Thread Apache Jenkins Server 
See 

Changes:

[wheat9] HDFS-9111. Move hdfs-client protobuf convert methods from PBHelper to 
PBHelperClient. Contributed by Mingliang Liu.

[ozawa] HADOOP-12428. Fix inconsistency between log-level guards and 
statements. Contributed by Jagadesh Kiran N and Jackie Chang.

--
[...truncated 3759 lines...]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-minikdc 
---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ 
hadoop-minikdc ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hadoop-minikdc ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-minikdc ---
[INFO] Surefire report directory: 


---
 T E S T S
---

---
 T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.246 sec - in 
org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestMiniKdc
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.738 sec - in 
org.apache.hadoop.minikdc.TestMiniKdc

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-minikdc 
---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-minikdc ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minikdc ---
[INFO] 
Loading source files for package org.apache.hadoop.minikdc...
Constructing Javadoc information...
Standard Doclet version 1.8.0
Building tree for all the packages and classes...
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating

huge editlog segment size make standby start failure

2015-09-21 Thread jiang yu 
Hi everyone,
My SNN failed two days ago, and it stopped to trigger ANN roll edit, so the 
editlog can be 10G large. After i restart the SNN, it failed to fetcher the 
editlog, because it is too large, the log is below :
015-09-22 00:23:07,338 ERROR 
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Got error reading 
edit log input stream 
http://**:8480/getJournal?jid=ns1&segmentTxId=19034359098&storageInfo=-56%3A200185119%3A1401352022932%3ACID-3c312573-1381-44f2-9e8b-fa2529f043d7&ugi=hadoop;
 failing over to edit log 
http://***:8480/getJournal?jid=ns1&segmentTxId=19034359098&storageInfo=-56%3A200185119%3A1401352022932%3ACID-3c312573-1381-44f2-9e8b-fa2529f043d7&ugi=hadoop
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at 
sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2707)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader$PositionTrackingInputStream.read(FSEditLogLoader.java:1105)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:42)

I don’t think it is good idea to set connection timeout in URLFactory, which is 
1 min default.
For now, i can’t restart the SNN, so ANN roll edit per day, and the edit size 
is too large  making SNN impossible to restart.
I am currently developing some utility to resolve this problem.
1. using RPC to ask ANN roll editlog like Editlog Tailer doing
2. Copy all the meta data from SNN to ANN, and read the newest FSImage file and 
read the editlog file on local file system then apply to FSNamesystem, after 
that save namespace to form a new FSImage file
3.After that restart SNN and hope everything goes well


Any idea? i appreciate to get your reply, thank you.

[jira] [Created] (HADOOP-12427) Upgrade Mockito version to 1.10.19

2015-09-21 Thread Giovanni Matteo Fumarola (JIRA) 
Giovanni Matteo Fumarola created HADOOP-12427:
-

 Summary: Upgrade Mockito version to 1.10.19
 Key: HADOOP-12427
 URL: https://issues.apache.org/jira/browse/HADOOP-12427
 Project: Hadoop Common
  Issue Type: Wish
  Components: build
Reporter: Giovanni Matteo Fumarola
Assignee: Giovanni Matteo Fumarola
Priority: Minor


The current version is 1.8.5 - inserted in 2011.
The new version works better with PowerMockito - Mock for Static methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-21 Thread Tsuyoshi Ozawa 
+1 (non-binding)

1. Built from source.
2. Verified signature of bin tar ball and source tar ball.
3. Running some examples.
4. Running Spark on YARN with spark-shell. It works well.

I think 4 verifies compatibility is preserved.
Thanks everyone for working to release 2.6.1.

Thanks,
- Tsuyoshi

On Thu, Sep 17, 2015 at 11:10 AM, Vinod Kumar Vavilapalli
 wrote:
> Hi all,
>
> After a nearly month long [1] toil, with loads of help from Sangjin Lee and
> Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release
> candidate RC1 for hadoop-2.6.1.
>
> RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes
> for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
> YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI
> bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for
> YARN-2890) that Jeff Zhang raised.
>
> The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/
>
> The RC tag in git is: release-2.6.1-RC1
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1021
>
> Some notes from our release process
>  -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
> non-committed but desired patches. 2.6.1 is already big as is and is late
> by any standard, we can definitely include them in the next release.
>  - The 2.6.1 wiki page [3] captures some (but not all) of the context of
> the patches that we pushed in.
>  - Given the number of fixes pushed [4] in, we had to make a bunch of
> changes to our original plan - we added a few improvements that helped us
> backport patches easier (or in many cases made backports possible), and we
> dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
> HDFS-7611, HDFS-7843, HDFS-8850).
>  - I ran all the unit tests which (surprisingly?) passed. (Except for one,
> which pointed out a missing fix HDFS-7552).
>
> As discussed before [5]
>  - This release is the first point release after 2.6.0
>  - I’d like to use this as a starting release for 2.6.2 in a few weeks and
> then follow up with more of these.
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
> [1] Hadoop 2.6.1 Release process thread:
> http://markmail.org/thread/wkbgkxkhntx5tlux
> [2] 2.6.1 Pending tickets:
> https://issues.apache.org/jira/issues/?filter=12331711
> [3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
> -Working-Notes
> [4] List of 2.6.1 patches pushed:
> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
> %20and%20labels%20%3D%20%222.6.1-candidate%22
> [5] Planning Hadoop 2.6.1 release:
> http://markmail.org/thread/sbykjn5xgnksh6wg
>
> PS:
>  - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these
> fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off
> 2.6.1.
>  - The additional patches in RC1 that got into 2.6.1 all the way from 2.8
> are NOT in 2.7.2 yet, this will be done as a followup.

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-21 Thread Kuhu Shukla 
Thanks everyone! I checked the following things and did not find any issues.* 
Compilation* Running unit tests under org.apache.hadoop.yarn.server package* 
Bringing up a single node cluster* Running  a WordCount job
Regards,Kuhu Shukla

 On Monday, September 21, 2015 2:55 PM, Chang Li  
wrote:
   

 Thanks everyone who helped on this release!
Have run compilation and run various job on single node cluster. Have also
test my contribution for YARN-3267
 and verified that all
related unit tests pass.

Thank you,
Chang Li

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-21 Thread Chang Li 
Thanks everyone who helped on this release!
Have run compilation and run various job on single node cluster. Have also
test my contribution for YARN-3267
 and verified that all
related unit tests pass.

Thank you,
Chang Li

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-21 Thread Akira AJISAKA 

+1 (non-binding) if MAPREDUCE-6334 is addressed.

- Downloaded and verified signatures and checksums
- Deployed a single node cluster
- Built Tez 0.7.0 with Hadoop 2.6.1 pom
- Built Hive 1.2.1 with Hadoop 2.6.1 pom
- Ran some Hive on Tez queries

Thanks everyone who worked on this release.

-Akira

On 9/22/15 03:19, Eric Payne wrote:

Hi Vinod and everyone else who helped on this release!

Thank you very much for going through the work and effort to put this release 
together.
While doing my testing for Hadoop 2.6.1 RC1, I encountered the following 
issue:[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon 
IOException during InMemoryMapOutput shuffle handler - ASF JIRA

|   |
|   |   |   |   |   |
| [MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon IOException 
during ...We are seeing this happen when an NM's disk goes bad during the 
creation of map output(s) the reducer's fetcher can read the shuffle header and 
reserve the memory  |
|  |
| View on issues.apache.org | Preview by Yahoo |
|  |
|   |


This is actually something we ran across frequently when running 2.6. You may 
want to consider pulling that in to 2.6.
Other than that, it looks fine. I did the following manual testing on a 
one-node cluster:
- successfully downloaded and compiled the code- successfully ran streaming 
jobs- successfully ran distributed shell jobs-- as part of the distributed 
shell testing, I started the jobs with the 
'-keep_containers_across_application_attempts' property set, killed the AM 
container, and verified that it would in fact keep the containers running 
across AM restarts.- sucessfully ran wordcount jobs.
Thank you,
-Eric Payne

   From: Vinod Kumar Vavilapalli 
  To: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Cc: vino...@apache.org; Sangjin Lee ; Akira AJISAKA 

  Sent: Wednesday, September 16, 2015 9:10 PM
  Subject: [VOTE] Release Apache Hadoop 2.6.1 RC1

Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes
for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI
bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for
YARN-2890) that Jeff Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
  -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
non-committed but desired patches. 2.6.1 is already big as is and is late
by any standard, we can definitely include them in the next release.
  - The 2.6.1 wiki page [3] captures some (but not all) of the context of
the patches that we pushed in.
  - Given the number of fixes pushed [4] in, we had to make a bunch of
changes to our original plan - we added a few improvements that helped us
backport patches easier (or in many cases made backports possible), and we
dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
HDFS-7611, HDFS-7843, HDFS-8850).
  - I ran all the unit tests which (surprisingly?) passed. (Except for one,
which pointed out a missing fix HDFS-7552).

As discussed before [5]
  - This release is the first point release after 2.6.0
  - I’d like to use this as a starting release for 2.6.2 in a few weeks and
then follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread:
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets:
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
-Working-Notes
[4] List of 2.6.1 patches pushed:
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release:
http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
  - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off
2.6.1.
  - The additional patches in RC1 that got into 2.6.1 all the way from 2.8
are NOT in 2.7.2 yet, this will be done as a followup.

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-21 Thread Mit Desai 
+1 (non-binding)

# Verified signatures
# Built from source
# Deployed a single node cluster.
# Run example jobs on single-node cluster.

Thanks Vinod and all others for the effort of putting the release together.

Thanks,
Mit Desai

On Mon, Sep 21, 2015 at 1:19 PM, Eric Payne <
eric.payne1...@yahoo.com.invalid> wrote:

> Hi Vinod and everyone else who helped on this release!
>
> Thank you very much for going through the work and effort to put this
> release together.
> While doing my testing for Hadoop 2.6.1 RC1, I encountered the following
> issue:[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon
> IOException during InMemoryMapOutput shuffle handler - ASF JIRA
>
> |   |
> |   |   |   |   |   |
> | [MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon
> IOException during ...We are seeing this happen when an NM's disk goes bad
> during the creation of map output(s) the reducer's fetcher can read the
> shuffle header and reserve the memory  |
> |  |
> | View on issues.apache.org | Preview by Yahoo |
> |  |
> |   |
>
>
> This is actually something we ran across frequently when running 2.6. You
> may want to consider pulling that in to 2.6.
> Other than that, it looks fine. I did the following manual testing on a
> one-node cluster:
> - successfully downloaded and compiled the code- successfully ran
> streaming jobs- successfully ran distributed shell jobs-- as part of the
> distributed shell testing, I started the jobs with the
> '-keep_containers_across_application_attempts' property set, killed the AM
> container, and verified that it would in fact keep the containers running
> across AM restarts.- sucessfully ran wordcount jobs.
> Thank you,
> -Eric Payne
>
>   From: Vinod Kumar Vavilapalli 
>  To: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Cc: vino...@apache.org; Sangjin Lee ; Akira AJISAKA <
> ajisa...@oss.nttdata.co.jp>
>  Sent: Wednesday, September 16, 2015 9:10 PM
>  Subject: [VOTE] Release Apache Hadoop 2.6.1 RC1
>
> Hi all,
>
> After a nearly month long [1] toil, with loads of help from Sangjin Lee and
> Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release
> candidate RC1 for hadoop-2.6.1.
>
> RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes
> for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
> YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI
> bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for
> YARN-2890) that Jeff Zhang raised.
>
> The RC is available at:
> http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/
>
> The RC tag in git is: release-2.6.1-RC1
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1021
>
> Some notes from our release process
>  -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
> non-committed but desired patches. 2.6.1 is already big as is and is late
> by any standard, we can definitely include them in the next release.
>  - The 2.6.1 wiki page [3] captures some (but not all) of the context of
> the patches that we pushed in.
>  - Given the number of fixes pushed [4] in, we had to make a bunch of
> changes to our original plan - we added a few improvements that helped us
> backport patches easier (or in many cases made backports possible), and we
> dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
> HDFS-7611, HDFS-7843, HDFS-8850).
>  - I ran all the unit tests which (surprisingly?) passed. (Except for one,
> which pointed out a missing fix HDFS-7552).
>
> As discussed before [5]
>  - This release is the first point release after 2.6.0
>  - I’d like to use this as a starting release for 2.6.2 in a few weeks and
> then follow up with more of these.
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
> [1] Hadoop 2.6.1 Release process thread:
> http://markmail.org/thread/wkbgkxkhntx5tlux
> [2] 2.6.1 Pending tickets:
> https://issues.apache.org/jira/issues/?filter=12331711
> [3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
> -Working-Notes
> [4] List of 2.6.1 patches pushed:
> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
> %20and%20labels%20%3D%20%222.6.1-candidate%22
> [5] Planning Hadoop 2.6.1 release:
> http://markmail.org/thread/sbykjn5xgnksh6wg
>
> PS:
>  - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these
> fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off
> 2.6.1.
>  - The additional patches in RC1 that got into 2.6.1 all the way from 2.8
> are NOT in 2.7.2 yet, this will be done as a followup.
>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-21 Thread Eric Payne 
Hi Vinod and everyone else who helped on this release!

Thank you very much for going through the work and effort to put this release 
together.
While doing my testing for Hadoop 2.6.1 RC1, I encountered the following 
issue:[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon 
IOException during InMemoryMapOutput shuffle handler - ASF JIRA

|   |
|   |   |   |   |   |
| [MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon IOException 
during ...We are seeing this happen when an NM's disk goes bad during the 
creation of map output(s) the reducer's fetcher can read the shuffle header and 
reserve the memory  |
|  |
| View on issues.apache.org | Preview by Yahoo |
|  |
|   |


This is actually something we ran across frequently when running 2.6. You may 
want to consider pulling that in to 2.6.
Other than that, it looks fine. I did the following manual testing on a 
one-node cluster:
- successfully downloaded and compiled the code- successfully ran streaming 
jobs- successfully ran distributed shell jobs-- as part of the distributed 
shell testing, I started the jobs with the 
'-keep_containers_across_application_attempts' property set, killed the AM 
container, and verified that it would in fact keep the containers running 
across AM restarts.- sucessfully ran wordcount jobs.
Thank you,
-Eric Payne
 
  From: Vinod Kumar Vavilapalli 
 To: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org 
Cc: vino...@apache.org; Sangjin Lee ; Akira AJISAKA 
 
 Sent: Wednesday, September 16, 2015 9:10 PM
 Subject: [VOTE] Release Apache Hadoop 2.6.1 RC1
   
Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes
for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI
bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for
YARN-2890) that Jeff Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
 -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
non-committed but desired patches. 2.6.1 is already big as is and is late
by any standard, we can definitely include them in the next release.
 - The 2.6.1 wiki page [3] captures some (but not all) of the context of
the patches that we pushed in.
 - Given the number of fixes pushed [4] in, we had to make a bunch of
changes to our original plan - we added a few improvements that helped us
backport patches easier (or in many cases made backports possible), and we
dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
HDFS-7611, HDFS-7843, HDFS-8850).
 - I ran all the unit tests which (surprisingly?) passed. (Except for one,
which pointed out a missing fix HDFS-7552).

As discussed before [5]
 - This release is the first point release after 2.6.0
 - I’d like to use this as a starting release for 2.6.2 in a few weeks and
then follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread:
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets:
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
-Working-Notes
[4] List of 2.6.1 patches pushed:
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release:
http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
 - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off
2.6.1.
 - The additional patches in RC1 that got into 2.6.1 all the way from 2.8
are NOT in 2.7.2 yet, this will be done as a followup.

RE: I want to start contributing to Apache Hadoop project

2015-09-21 Thread Brahma Reddy Battula 

Thanks for the interest, Suchith Javagal. 

Have you seen http://wiki.apache.org/hadoop/HowToContribute from the Apache
Hadoop wiki? I think it is a good starting point if you want to become
familiar with the project.


Also, the wiki page has a very handy link to all
the JIRAs that are "easy" fixes that you can do and get you started
contributing to the project:
https://issues.apache.org/jira/issues/?filter=12331506


I recommend doing the following:

1) Look at unit test JIRAs, both open and closed.  Tons of benefits:
they'll help you understand some part of the API, some could use some
comments and enhancements, etc.

2) Pick a component (ResourceManager, NodeManager, MR APIs, DataNode,
NameNode, etc.) and focus on that to start.  It's easier than trying to
understand it all at once.

3) Participate in some code reviews.  It will be non-binding, but testing
out other code can help you understand some part.




Thanks & Regards
 Brahma Reddy Battula





From: Suchith Javagal [suchithj...@gmail.com]
Sent: Monday, September 21, 2015 6:11 PM
To: common-dev@hadoop.apache.org
Subject: I want to start contributing to Apache Hadoop project

Hello everyone,

I'm a newbie who wants to contribute to the Apache Hadoop project. My
skills include Java,C,C++ (Intermediate) and Scala(Basic..I can manage
though). I cloned the repo today. How do I get started?

Regards,
Suchith J N

I want to start contributing to Apache Hadoop project

2015-09-21 Thread Suchith Javagal 
Hello everyone,

I'm a newbie who wants to contribute to the Apache Hadoop project. My
skills include Java,C,C++ (Intermediate) and Scala(Basic..I can manage
though). I cloned the repo today. How do I get started?

Regards,
Suchith J N

[jira] [Created] (HADOOP-12426) Add Entry point for Kerberos health check

2015-09-21 Thread Steve Loughran (JIRA) 
Steve Loughran created HADOOP-12426:
---

 Summary: Add Entry point for Kerberos health check
 Key: HADOOP-12426
 URL: https://issues.apache.org/jira/browse/HADOOP-12426
 Project: Hadoop Common
  Issue Type: New Feature
  Components: security
Affects Versions: 3.0.0
Reporter: Steve Loughran
Priority: Minor


If we a little command line entry point for testing kerberos settings, 
including some automated diagnostics checks, we could simplify fielding the 
client-side support calls.

Specifically
* check JRE for having java crypto extensions at full key length.
* network checks: do you know your own name?
* Is the user kinited in?
* if a tgt is specified, does it exist?
* are hadoop security options consistent?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Local repo sharing for maven builds

2015-09-21 Thread Steve Loughran 

> On 19 Sep 2015, at 04:42, Allen Wittenauer  wrote:
> 
> a) Multi-module patches are always troublesome because it makes the test 
> system do significantly more work.  For Yetus, we've pared it down as far as 
> we can go to get *some* speed increases, but if a patch does something like 
> hit every pom.xml file, there's nothing that can be done to make it better 
> other than splitting up the patch.
> 
> b) It's worth noting that it happens more often to HDFS patches because HDFS 
> unit tests take too damn long.  Some individual tests take 10 minutes! They 
> invariably collide with the various full builds (NOT pre commit! Those other 
> things that Steve pointed out that we're ignoring).  While Yetus has support 
> for running unit tests in parallel, Hadoop does not.  


I think the main thing I've been complaining about is how we ignore failing 
scheduled Jenkins runs; its been so unreliable that we all ignore the constant 
background noise of jenkins failures. That's compounded by how some test runs 
(hello Yarn-precommit!) send jenkins mails to the dev- list. (I've turned that 
off now: if you get jenkins failures on yarn-dev then its from the regular ones)

> 
> c) mvn install is pretty much required for a not insignificant amount of 
> multi-module patches, esp if they hit hadoop-common.  For a large chunk of 
> "oh just make it one patch", it's effectively a death sentence on the Jenkins 
> side.

The race conditions have existed for a long, long time. It only surfaces when 
you have a patch that spans artifacts which is one of: (1) incompatible across 
builds (2) needs to be synced across builds to work. If things still linked up, 
you'd have the race *but you wouldn't notice*. It's only the artifact-spanning 
patches which surface.

YARN has had this for a while, but it's builds are shorter, it's HDFS that's 
the problem for the reasons AW's noted
-theres' now >1 JAR
-it takes a long time to build and test, host conflict is inevitable.


There is one tactic not yet looked at: every build to set a hadoop version, e.g 
instead of all precommits being hadoop-3.0.0-SNAPSHOT, they could be 
hadoop-3.0.0-JIRA-4313-SNAPSHOT. No conflict, just the need to schedule a run 
that cleans up the m2 repo every night. If timestamped version numbers are used 
hadoop-3.0.0-2015-09-21-11:38 then the job can make better decisions about what 
to purge. Test runs could even rm their own artifacts after, perhaps.

I think this would be the best way to isolate —no need for private repos, with 
the followon need to download the entire repo on every run, 100% isolation.

The other issue with race conditions is port assignments, too much code with 
hard coded ports. —there's been slow work on that, with Brahma Reddy Battula 
deserving special mention here. But its almost a losing battle, chasing where 
the next hard-coded port goes in, and again, leads to unreliable test runs that 
everyone ignores.


ANNOUNCEMENT: new patches which contain hard-coded ports in test runs will 
henceforth be reverted. Jenkins matters more than the 30s of your time it takes 
to use the free port finder methods. Same for any hard code paths in 
filesystems.


> 
> d) I'm a big fan of d. 
> 
> e) File a bug against Yetus and we'll add the ability to set ant/gradle/maven 
> args from the command line.  I thought I had it in there when I rewrote the 
> support for multiple build tools, gradle, etc, but I clearly dropped it on 
> the floor.

people won't do that. Switching to per-run hadoop version numbers should 
suffice for artifact dependencies, leaving only ports and paths.
> 
> f) Any time you "give the option to the patch submitter", you generate a not 
> insignificant amount of work on the test infrastructure to determine intent 
> because it effectively means implementing some parsing of a comment.  It's 
> not particularly easy because humans rarely follow the rules.  Just see how 
> well we are at following the Hadoop Compatibility Guidelines. Har har.  No 
> really: people still struggle with filling in JIRA headers correctly and 
> naming patches to trigger the appropriate branch for the test.

where's that documented BTW? I did try looking for it at the weekend..


> 
> g) It's worth noting that Hadoop trunk is *not* using the latest test-patch 
> code.  So there are some significant improvements on the way as soon as we 
> get a release out the door.
> 
> 


well get on with it then :)

I'm going to be at apachecon Data EU next week -who else will be. Maybe we 
could make it a goal of the conference to come out of the week with jenkins 
building reliably. I've been looking at it at weekends but don't have time in 
the week.

RE: [VOTE] Release Apache Hadoop 2.6.1 RC1

2015-09-21 Thread Brahma Reddy Battula 
+1 (non-binding) 
--Build from source code
--Set up a pseudo-distributed cluster and Distributed HA Cluster
--Ran Several jobs Slive,Terasort and pi.
--All are working fine.


The footer of the Web UI "Hadoop, 2014" is old. ( I think,we can include 
HDFS-8149)


Thanks & Regards
 Brahma Reddy Battula

From: Junping Du [j...@hortonworks.com]
Sent: Monday, September 21, 2015 12:12 PM
To: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

+1 (binding) with:
- Download and verify signatures.
- Build and deploy a single node cluster.
- Run some example jobs (PI, etc.) on deployed cluster.

Thanks,

Junping

From: Rohith Sharma K S 
Sent: Monday, September 21, 2015 6:53 AM
To: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: RE: [VOTE] Release Apache Hadoop 2.6.1 RC1

+1 (non-binding)

Downloaded source, built package and installed 5 node cluster.
1. Verified for RMHA/RMRestart/RMWorkpreservingRestart cluster in 
secure/non-secure mode.
2. Attached JCarder tool to cluster for identifying deadlock cycles, No cycles 
found.
3. Verified cluster for high availability.
4. Ran sample MR applications with RM HA enabled.
5. Run sanity test cases and working fine.


Thanks & Regards
Rohith Sharma K S

-Original Message-
From: sjl...@gmail.com [mailto:sjl...@gmail.com] On Behalf Of Sangjin Lee
Sent: 18 September 2015 09:50
To: yarn-...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

+1 (non-binding)

Verified the signatures, set up a pseudo-distributed cluster, ran several test 
jobs, and ran an uber job. Also verified that the UI issue I saw on
RC0 is now gone. Thanks Vinod!

Sangjin

On Thu, Sep 17, 2015 at 7:24 PM, Jian He  wrote:

> +1 (binding)
>
> Build from source code.
> Deployed a local cluster.
> Validated sample jobs passed.
>
> Jian
>
> > On Sep 18, 2015, at 7:34 AM, Wangda Tan  wrote:
> >
> > Deployed a local cluster, verified configured cluster with node
> > labels,
> run
> > jobs with/without node labels.
> >
> > +1 (non-binding)
> >
> > Thanks!
> >
> > On Thu, Sep 17, 2015 at 2:40 PM, Xuan Gong 
> wrote:
> >
> >> Update my vote from +1 (non-binding) to +1 binding
> >>
> >> Thanks
> >>
> >> Xuan Gong
> >>
> >>> On Sep 17, 2015, at 2:05 PM, Xuan Gong  wrote:
> >>>
> >>> +1 (non-binding)
> >>> Download and compile the source code, run several MR jobs.
> >>>
> >>> Xuan Gong
> >>>
>  On Sep 16, 2015, at 7:10 PM, Vinod Kumar Vavilapalli <
> >> vino...@apache.org> wrote:
> 
>  Hi all,
> 
>  After a nearly month long [1] toil, with loads of help from
>  Sangjin
> Lee
> >> and
>  Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a
> >> release
>  candidate RC1 for hadoop-2.6.1.
> 
>  RC1 is RC0 [0] (for which I opened and closed a vote last week) +
>  UI
> >> fixes
>  for the issue Sangjin raised (YARN-3171 and the dependencies
> YARN-3779,
>  YARN-3248), additional fix to avoid incompatibility (YARN-3740),
>  other
> >> UI
>  bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right
>  patch
> >> for
>  YARN-2890) that Jeff Zhang raised.
> 
>  The RC is available at:
> >> http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/
> 
>  The RC tag in git is: release-2.6.1-RC1
> 
>  The maven artifacts are available via repository.apache.org at
> 
> https://repository.apache.org/content/repositories/orgapachehadoop-102
> 1
> 
>  Some notes from our release process
>  -  - Sangjin and I moved out a bunch of items pending from 2.6.1
>  [2] - non-committed but desired patches. 2.6.1 is already big as
>  is and is
> >> late
>  by any standard, we can definitely include them in the next release.
>  - The 2.6.1 wiki page [3] captures some (but not all) of the
>  context
> of
>  the patches that we pushed in.
>  - Given the number of fixes pushed [4] in, we had to make a bunch
>  of changes to our original plan - we added a few improvements
>  that helped
> >> us
>  backport patches easier (or in many cases made backports
>  possible),
> and
> >> we
>  dropped a few that didn't make sense (HDFS-7831, HDFS-7926,
>  HDFS-7676, HDFS-7611, HDFS-7843, HDFS-8850).
>  - I ran all the unit tests which (surprisingly?) passed. (Except
>  for
> >> one,
>  which pointed out a missing fix HDFS-7552).
> 
>  As discussed before [5]
>  - This release is the first point release after 2.6.0
>  - I’d like to use this as a starting release for 2.6.2 in a few
>  weeks
> >> and
>  then follow up with more of these.
> 
>  Please