Re: committing HADOOP-11746 test-patch improvements

[Allen Wittenauer]

Some status:

* So far, HADOOP-11627 was filed which is luckily an extremely easy bug to fix.

* There have been a few runs which seems to indicate that *something* is 
destroying the artifact directory in the middle of  runs…. which is very very 
odd and something I hadn’t seen in any of my testing.  In any case, I clearly 
need to add some safety code here to report back that something went awry and 
report back which node, console, etc this happened on.  Someone more familiar 
with the Jenkins setup might be able to shed some light on why that might 
happen. All of these runs appear to be on H3, so might be related? Impacted 
issues with this have been:

- HDFS-8200 (https://builds.apache.org/job/PreCommit-HDFS-Build/10335/)
- HDFS-8147 (https://builds.apache.org/job/PreCommit-HDFS-Build/10338/)
- YARN-3301 (https://builds.apache.org/job/PreCommit-YARN-Build/7441/) 

* Some sort of good news! The new mvn site test does appear to be capable of 
catching issues!  YARN-3410 was committed just prior to the new test code and 
has markdown-to-html syntax issues.  HADOOP-11627 (for example) shows the 
results. :D

On Apr 21, 2015, at 10:48 PM, Allen Wittenauer  wrote:

> 
> FYI, MAPREDUCE-6324 is the first JIRA to be tested with the new code in place 
> if someone hasn’t seen the new output. 
> 
> On Apr 21, 2015, at 9:06 PM, Allen Wittenauer  wrote:
> 
>> 
>> 
>>  Just a heads up that I’ll be committing this to trunk and branch-2 here 
>> in a bit. I’ll watch it over the next hour or two before heading off to bed. 
>>  (I’m in London at the moment.)  If there are any issues, revert and drop a 
>> note in HADOOP-11746 with any issues.
>> 
>> Despite what will hopefully be obvious changes, a few things to keep your 
>> eyes out for:
>> 
>>  * shellcheck won’t work until shellcheck actually gets installed on the 
>> jenkins nodes.
>>  * checkstyle might give spurious errors until HADOOP-11778 is 
>> committed, since it currently NPEs, at least in trunk.
>> 
>> 
>> On Apr 16, 2015, at 7:38 PM, Chris Nauroth  wrote:
>> 
>>> I'd like to thank Allen Wittenauer for his work on HADOOP-11746 to rewrite
>>> test-patch.sh.  There is a lot of nice new functionality in there.  My
>>> favorite part is that some patches will execute much faster, so I expect
>>> this will make the project more efficient overall at moving patches
>>> through the pre-commit process.
>>> 
>>> I have +1'd the patch, but since this is a tool that we all use
>>> frequently, I'd like to delay a day before committing.  Please comment on
>>> the jira if you have any additional feedback.  We're aiming to commit on
>>> Friday, 4/17.
>>> 
>>> Chris Nauroth
>>> Hortonworks
>>> http://hortonworks.com/
>>> 
>> 
> 

Re: committing HADOOP-11746 test-patch improvements

[Allen Wittenauer]

Err, first jira mentioned should be HADOOP-11861.

On Apr 22, 2015, at 8:10 AM, Allen Wittenauer  wrote:

> 
> Some status:
> 
> * So far, HADOOP-11627 was filed which is luckily an extremely easy bug to 
> fix.
> 
> * There have been a few runs which seems to indicate that *something* is 
> destroying the artifact directory in the middle of  runs…. which is very very 
> odd and something I hadn’t seen in any of my testing.  In any case, I clearly 
> need to add some safety code here to report back that something went awry and 
> report back which node, console, etc this happened on.  Someone more familiar 
> with the Jenkins setup might be able to shed some light on why that might 
> happen. All of these runs appear to be on H3, so might be related? Impacted 
> issues with this have been:
> 
> - HDFS-8200 (https://builds.apache.org/job/PreCommit-HDFS-Build/10335/)
> - HDFS-8147 (https://builds.apache.org/job/PreCommit-HDFS-Build/10338/)
> - YARN-3301 (https://builds.apache.org/job/PreCommit-YARN-Build/7441/) 
> 
> * Some sort of good news! The new mvn site test does appear to be capable of 
> catching issues!  YARN-3410 was committed just prior to the new test code and 
> has markdown-to-html syntax issues.  HADOOP-11627 (for example) shows the 
> results. :D
> 
> On Apr 21, 2015, at 10:48 PM, Allen Wittenauer  wrote:
> 
>> 
>> FYI, MAPREDUCE-6324 is the first JIRA to be tested with the new code in 
>> place if someone hasn’t seen the new output. 
>> 
>> On Apr 21, 2015, at 9:06 PM, Allen Wittenauer  wrote:
>> 
>>> 
>>> 
>>> Just a heads up that I’ll be committing this to trunk and branch-2 here 
>>> in a bit. I’ll watch it over the next hour or two before heading off to 
>>> bed.  (I’m in London at the moment.)  If there are any issues, revert and 
>>> drop a note in HADOOP-11746 with any issues.
>>> 
>>> Despite what will hopefully be obvious changes, a few things to keep your 
>>> eyes out for:
>>> 
>>> * shellcheck won’t work until shellcheck actually gets installed on the 
>>> jenkins nodes.
>>> * checkstyle might give spurious errors until HADOOP-11778 is 
>>> committed, since it currently NPEs, at least in trunk.
>>> 
>>> 
>>> On Apr 16, 2015, at 7:38 PM, Chris Nauroth  wrote:
>>> 
 I'd like to thank Allen Wittenauer for his work on HADOOP-11746 to rewrite
 test-patch.sh.  There is a lot of nice new functionality in there.  My
 favorite part is that some patches will execute much faster, so I expect
 this will make the project more efficient overall at moving patches
 through the pre-commit process.
 
 I have +1'd the patch, but since this is a tool that we all use
 frequently, I'd like to delay a day before committing.  Please comment on
 the jira if you have any additional feedback.  We're aiming to commit on
 Friday, 4/17.
 
 Chris Nauroth
 Hortonworks
 http://hortonworks.com/
 
>>> 
>> 
> 

Which newbie Jira's need work?

[Darrell Taylor]

Hi All,

First off I've submitted a patch for HADOOP-11813 as a starter for 10, this
was an easy change and starts to get me familiar with where things are.

Now I've been looking through the newbie Jira's to see what else I can do
to help, but I'm struggling to see the wood for the trees, which I suspect
is down to my lack of knowledge around how things hang together.

I figured I'd go for some easy stuff to start with, so had a look at
HADOOP-10861.  Most of the html I can find
under /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps seems to be valid.
After further digging I found HDFS-274 which appears to be a patch for the
pages when they were .jsp instead of html, so does this just need closing?

So essentially this boils down to two mains questions:

* How do I know if a newbie Jira is even still valid?
* Does anybody have any suggestions for a suitable newbie ticket to pick up?

Thanks
Darrell

Re: Which newbie Jira's need work?

[Tsuyoshi Ozawa]

Hi Darrell,

Thank you for having interest for contributions to Hadoop project!

> * How do I know if a newbie Jira is even still valid?

Please ping committers whether the issue is valid when the issue looks
to have been resolved already.

> * Does anybody have any suggestions for a suitable newbie ticket to pick up?

One suggestion from me is to use JQL on JIRA to find issues for newbie.
If you run following query, you will find fresh and open issues.

project = "Hadoop Common" and labels = "newbie" and status = open
ORDER BY updatedDate

Also I think one good starting point is to fix documentation, typos in
code, trivial changes, or something.

Thanks,
- Tsuyoshi


On Wed, Apr 22, 2015 at 4:47 PM, Darrell Taylor
 wrote:
> Hi All,
>
> First off I've submitted a patch for HADOOP-11813 as a starter for 10, this
> was an easy change and starts to get me familiar with where things are.
>
> Now I've been looking through the newbie Jira's to see what else I can do
> to help, but I'm struggling to see the wood for the trees, which I suspect
> is down to my lack of knowledge around how things hang together.
>
> I figured I'd go for some easy stuff to start with, so had a look at
> HADOOP-10861.  Most of the html I can find
> under /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps seems to be valid.
> After further digging I found HDFS-274 which appears to be a patch for the
> pages when they were .jsp instead of html, so does this just need closing?
>
> So essentially this boils down to two mains questions:
>
> * How do I know if a newbie Jira is even still valid?
> * Does anybody have any suggestions for a suitable newbie ticket to pick up?
>
> Thanks
> Darrell

Build failed in Jenkins: Hadoop-common-trunk-Java8 #173

[Apache Jenkins Server]

See 

Changes:

[vinayakumarb] HDFS-7993. Provide each Replica details in fsck (Contributed by 
J.Andreina)

[arp] HDFS-8163. Using monotonicNow for block report scheduling causes test 
failures on recently restarted systems. (Arpit Agarwal)

[Arun Suresh] HADOOP-11704. DelegationTokenAuthenticationFilter must pass 
ipaddress instead of hostname to ProxyUsers#authorize (Anubhav Dhoot via 
asuresh)

[cmccabe] HDFS-8133. Improve readability of deleted block check (Daryn Sharp 
via Colin P. McCabe)

[aw] HADOOP-11746. rewrite test-patch.sh (aw)

[ozawa] YARN-3495. Confusing log generated by FairScheduler. Contributed by 
Brahma Reddy Battula.

[gera] MAPREDUCE-6293. Set job classloader on uber-job's LocalContainerLauncher 
event thread. (Sangjin Lee via gera)

[gera] HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up 
split calculation (gera)

[gera] MAPREDUCE-6297. Task Id of the failed task in diagnostics should link to 
the task page. (Siqi Li via gera)

[stevel] HADOOP-11846 TestCertificateUtil.testCorruptPEM failing on Jenkins 
JDK8. (Larry McCay via stevel)

[raviprak] HADOOP-11827. Speed-up distcp buildListing() using threadpool (Zoran 
Dimitrijevic via raviprak)

[wangda] YARN-3410. YARN admin should be able to remove individual application 
records from RMStateStore. (Rohith Sharmaks via wangda)

[jianhe] YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. 
Contributed by Rohith Sharmaks

[jianhe] YARN-3503. Expose disk utilization percentage and bad local and log 
dir counts in NM metrics. Contributed by Varun Vasudev

[wheat9] HDFS-8185. Separate client related routines in HAUtil into a new 
class. Contributed by Haohui Mai.

[aajisaka] YARN-3410. Addendum fix for compilation error. Contributed by Rohith.

--
[...truncated 4183 lines...]

Command line was: /home/jenkins/tools/java/jdk1.8.0/jre/../bin/javadoc @options 
@packages

Refer to the generated Javadoc files in 
'
 dir.

at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: 
MavenReportException: Error while creating archive: 
Exit code: 1 - 
:41:
 warning: no @throws for javax.servlet.ServletException
  public static RSAPublicKey parseRSAPublicKey(String pem) throws 
ServletException {
 ^
:37:
 warning: no description for @throws
   * @throws Exception
 ^
:64:
 warning: no description for @throws
   * @throws Exception
 ^

[jira] [Created] (HADOOP-11864) JWTRedirectAuthenticationHandler breaks java8 javadocs

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-11864:
---

 Summary: JWTRedirectAuthenticationHandler breaks java8 javadocs
 Key: HADOOP-11864
 URL: https://issues.apache.org/jira/browse/HADOOP-11864
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 3.0.0
 Environment: Jenkins on Java8
Reporter: Steve Loughran
Assignee: Larry McCay


Jenkins on Java8 is failing as {{JWTRedirectAuthenticationHandler}} has 
{{}} tags in it, something javadoc on java8 considers illegal





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Set minimum version of Hadoop 3 to JDK 8

[Steve Loughran]

> On 21 Apr 2015, at 23:31, Andrew Wang  wrote:
> 
> Hey Robert,
> 
> As a first step, could we try switching all our precommit and nightly
> builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe
> to do in any case since it'll still target 1.7.

+1

we have jenkin builds running on Java 8 -its just they have enough longstanding 
and intermittent test failures that they get ignored

I've been trying to keep hadoop-common somewhat under control, but 

-YARN is failing due to to hard coded ports in tests - this needs to be fixed 
urgently: YARN-3528 
-HDFS build process itself is failing with some config problem checkstyle


> 
> I'll note that HADOOP-10530 details the pain Steve went through switching
> us to JDK7. Might be some lessons learned about how to do this transition
> more smoothly.
> 


The key thing is to have Jenkins happy before trying to change anything. And 
make sure that "happy" means that jenkins really is building and testing 
against the version of Java that you think it is. I spent about 15 minutes/hour 
of my weekend kicking off builds and tuning them.  -I volunteered Colin Patrick 
McCabe to to that bit of the switch, though of course he doesn't have to accept 
that opportunuty ( 
https://issues.apache.org/jira/browse/HADOOP-10530?focusedCommentId=14239250&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14239250
 

Someone needs to take ownership of the "Jenkins builds are broken" problem 
-ideally someone in each subproject. That's a prerequisite to any major changes 
in the build process, but should also be best practice anyway. Ask your QE 
teams "would you like the ASF projects to care whether their jenkins test runs 
succeed" and see what they say.

as a reminder, all the jenkins builds are here
https://builds.apache.org/view/H-L/view/Hadoop/

please look at them and try to fix them —without even waiting for a switch to 
Java 8

[jira] [Created] (HADOOP-11865) Incorrect path mentioned in document for accessing script files

[J.Andreina (JIRA)]

J.Andreina created HADOOP-11865:
---

 Summary: Incorrect path mentioned in document for accessing script 
files
 Key: HADOOP-11865
 URL: https://issues.apache.org/jira/browse/HADOOP-11865
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: J.Andreina
Assignee: J.Andreina


Instead of "sbin" it is mentioned as "bin" in path to access script files.

1. 
  To start/stop KMS use KMS's *bin/kms.sh* script. For example:

2.
Run the new version with -upgrade option *(bin/start-dfs.sh -upgrade).*

3. start the cluster with rollback option. *(bin/start-dfs.sh -rollback).*

4.

*/bin/hadoop-daemon.sh* --config $HADOOP_CONF_DIR --script "$bin"/hdfs start 
balancer [-policy ]

5.*bin/distributed-exclude.sh* 

6.*bin/refresh-namenodes.sh*

Need to check for any other occurrences and fix the same



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: committing HADOOP-11746 test-patch improvements

[Sean Busbey]

On Wed, Apr 22, 2015 at 2:10 AM, Allen Wittenauer  wrote:

>
>
> * There have been a few runs which seems to indicate that *something* is
> destroying the artifact directory in the middle of  runs…. which is very
> very odd and something I hadn’t seen in any of my testing.  In any case, I
> clearly need to add some safety code here to report back that something
> went awry and report back which node, console, etc this happened on.
> Someone more familiar with the Jenkins setup might be able to shed some
> light on why that might happen. All of these runs appear to be on H3, so
> might be related? Impacted issues with this have been:
>
> - HDFS-8200 (https://builds.apache.org/job/PreCommit-HDFS-Build/10335/)
> - HDFS-8147 (https://builds.apache.org/job/PreCommit-HDFS-Build/10338/)
> - YARN-3301 (https://builds.apache.org/job/PreCommit-YARN-Build/7441/)
>
>
>From the HDFS precommit build:

>
> PATCHPROCESS=${WORKSPACE}/../patchprocess
> mkdir -p ${PATCHPROCESS}
>

Working on directories outside of the workspace for the job is not good,
though I'm not sure if that's the source of the issue. Do I need to
coordinate fixing this with anyone?

-- 
Sean

[jira] [Created] (HADOOP-11866) increase readability of the white space script

[Naganarasimha G R (JIRA)]

Naganarasimha G R created HADOOP-11866:
--

 Summary: increase readability of the white space script
 Key: HADOOP-11866
 URL: https://issues.apache.org/jira/browse/HADOOP-11866
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Naganarasimha G R
Priority: Minor


HADOOP-11845 supports listing of the lines which has trailing white spaces but 
doesn't inform patch line number. Without this report output will not be of 
much help as in most cases it reports blank lines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11867) FS API: Add a high-performance vectored Read to FSDataInputStream API

[Gopal V (JIRA)]

Gopal V created HADOOP-11867:


 Summary: FS API: Add a high-performance vectored Read to 
FSDataInputStream API
 Key: HADOOP-11867
 URL: https://issues.apache.org/jira/browse/HADOOP-11867
 Project: Hadoop Common
  Issue Type: New Feature
Affects Versions: 2.8.0
Reporter: Gopal V


The most significant way to read from a filesystem in an efficient way is to 
let the FileSystem implementation handle the seek behaviour underneath the API 
to be the most efficient as possible.

A better approach to the seek problem is to provide a sequence of read 
locations as part of a single call, while letting the system schedule/plan the 
reads ahead of time.

This is exceedingly useful for seek-heavy readers on HDFS, since this allows 
for potentially optimizing away the seek-gaps within the FSDataInputStream 
implementation.

For seek+read systems with even more latency than locally-attached disks, 
something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would take 
of the seeks internally while reading chunk.remaining() bytes into each chunk 
(which may be {{slice()}}ed off a bigger buffer).

The base implementation can stub in this as a sequence of seeks + read() into 
ByteBuffers, without forcing each FS implementation to override this in any way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

IMPORTANT: testing patches for branches

[Allen Wittenauer]

Hey gang, 

Just so everyone is aware, if you are working on a patch for either a 
feature branch or a major branch, if you name the patch with the branch name 
following the spec in HowToContribute (and a few other ways… test-patch tries 
to figure it out!), test-patch.sh *should* be switching the repo over to that 
branch for testing. 

For example,  naming a patch foo-branch-2.01.patch should get tested on 
branch-2.  Naming a patch foo-HDFS-7285.00.patch should get tested on the 
HDFS-7285 branch.

This hopefully means that there should really be no more ‘blind’ +1’s 
to patches that go to branches.  The “we only test against trunk” argument is 
no longer valid. :)

Re: IMPORTANT: testing patches for branches

[Vinod Kumar Vavilapalli]

Does this mean HADOOP-7435 is no longer needed / closeable as dup?

Thanks
+Vinod

On Apr 22, 2015, at 12:34 PM, Allen Wittenauer  wrote:

>   
> Hey gang, 
> 
>   Just so everyone is aware, if you are working on a patch for either a 
> feature branch or a major branch, if you name the patch with the branch name 
> following the spec in HowToContribute (and a few other ways… test-patch tries 
> to figure it out!), test-patch.sh *should* be switching the repo over to that 
> branch for testing. 
> 
>   For example,  naming a patch foo-branch-2.01.patch should get tested on 
> branch-2.  Naming a patch foo-HDFS-7285.00.patch should get tested on the 
> HDFS-7285 branch.
> 
>   This hopefully means that there should really be no more ‘blind’ +1’s 
> to patches that go to branches.  The “we only test against trunk” argument is 
> no longer valid. :)
> 
> 

Re: IMPORTANT: testing patches for branches

[Allen Wittenauer]

More than likely. It probably needs more testing (esp under Jenkins).  

It should be noted that the code in test-patch.sh has lots of problems with 
branch-0, minor, and micro releases.  But for major releases, it seems to work 
well for me. :)  

On Apr 22, 2015, at 8:45 PM, Vinod Kumar Vavilapalli  
wrote:

> Does this mean HADOOP-7435 is no longer needed / closeable as dup?
> 
> Thanks
> +Vinod
> 
> On Apr 22, 2015, at 12:34 PM, Allen Wittenauer  wrote:
> 
>>  
>> Hey gang, 
>> 
>>  Just so everyone is aware, if you are working on a patch for either a 
>> feature branch or a major branch, if you name the patch with the branch name 
>> following the spec in HowToContribute (and a few other ways… test-patch 
>> tries to figure it out!), test-patch.sh *should* be switching the repo over 
>> to that branch for testing. 
>> 
>>  For example,  naming a patch foo-branch-2.01.patch should get tested on 
>> branch-2.  Naming a patch foo-HDFS-7285.00.patch should get tested on the 
>> HDFS-7285 branch.
>> 
>>  This hopefully means that there should really be no more ‘blind’ +1’s 
>> to patches that go to branches.  The “we only test against trunk” argument 
>> is no longer valid. :)
>> 
>> 
> 

Re: IMPORTANT: testing patches for branches

[Allen Wittenauer]

Oh, this is also in the release notes, but one can use a git reference # as 
well. :) (with kudos to OOM for the idea.)

On Apr 22, 2015, at 8:57 PM, Allen Wittenauer  wrote:

> 
> More than likely. It probably needs more testing (esp under Jenkins).  
> 
> It should be noted that the code in test-patch.sh has lots of problems with 
> branch-0, minor, and micro releases.  But for major releases, it seems to 
> work well for me. :)  
> 
> On Apr 22, 2015, at 8:45 PM, Vinod Kumar Vavilapalli 
>  wrote:
> 
>> Does this mean HADOOP-7435 is no longer needed / closeable as dup?
>> 
>> Thanks
>> +Vinod
>> 
>> On Apr 22, 2015, at 12:34 PM, Allen Wittenauer  wrote:
>> 
>>> 
>>> Hey gang, 
>>> 
>>> Just so everyone is aware, if you are working on a patch for either a 
>>> feature branch or a major branch, if you name the patch with the branch 
>>> name following the spec in HowToContribute (and a few other ways… 
>>> test-patch tries to figure it out!), test-patch.sh *should* be switching 
>>> the repo over to that branch for testing. 
>>> 
>>> For example,  naming a patch foo-branch-2.01.patch should get tested on 
>>> branch-2.  Naming a patch foo-HDFS-7285.00.patch should get tested on the 
>>> HDFS-7285 branch.
>>> 
>>> This hopefully means that there should really be no more ‘blind’ +1’s 
>>> to patches that go to branches.  The “we only test against trunk” argument 
>>> is no longer valid. :)
>>> 
>>> 
>> 
> 

Re: [DISCUSS] Looking to a 2.8.0 release

[Vinod Kumar Vavilapalli]

+dev lists.

Forgot about that, sure. I added this and my initial list to the Roadmap wiki.

Thank
+Vinod

On Apr 21, 2015, at 9:34 PM, Rohith Sharma K S  
wrote:

> Dear Vinod
> 
>   Regarding the road map of Hadoop-2.8.0, Can basic Application priority 
> working model includes in next release?
> 
> Thanks & Regards
> Rohith Sharma K S
> 
> -Original Message-
> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] 
> Sent: 22 April 2015 03:09
> To: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Cc: vino...@apache.org
> Subject: [DISCUSS] Looking to a 2.8.0 release
> 
> With 2.7.0 out of the way, and with more maintenance releases to stabilize 
> it, I propose we start thinking about 2.8.0.
> 
> Here's my first cut of the proposal, will update the Roadmap wiki.
> - Support *both* JDK7 and JDK8 runtimes: HADOOP-11090
> - Compatibility tools to catch backwards, forwards compatibility issues at 
> patch submission, release times. Some of it is captured at YARN-3292. This 
> also involves resurrecting jdiff (HADOOP-11776/YARN-3426/MAPREDUCE-6310)
> and/or investing in new tools.
> - HADOOP-11656 Classpath isolation for downstream clients
> - Support for Erasure Codes in HDFS HDFS-7285
> - Early work for disk and network isolation in YARN: YARN-2139, YARN-2140
> - YARN Timeline Service Next generation: YARN-2928. At least branch-merge
> + early peek.
> - Supporting non-exclusive node-labels: YARN-3214
> 
> I'm experimenting with more agile 2.7.x releases and would like to continue 
> the same by volunteering as the RM for 2.8.x too.
> 
> Given the long time we took with 2.7.0, the timeline I am looking at is
> 8-12 weeks. We can pick as many features as they finish along and make a more 
> predictable releases instead of holding up releases for ever.
> 
> Thoughts?
> 
> Thanks
> +Vinod

[DISCUSS] Release numbering for stable 2.8 and beyond

[Vinod Kumar Vavilapalli]

Forking the thread.

In the previous 2.7.1 thread [1], there were enough yays to my proposal to wait 
for a bug-fix release or two before calling a 2.x release stable. There were 
some concerns about the naming.

We have two options, taking 2.8 as an example
 (1) Release 2.8.0, call it as an alpha in documentation and release notes, 
wait for a 2.8.1/2.8.2 reasonably stable enough to be called as the first 
stable release of 2.8.
 (2) Release 2.8.0-alpha, 2.8.0-beta etc before culminating in a 2.8.0 stable 
release.

(1) is what I preferred first up. This is what HBase used to do, and far 
beyond, in the linux kernel releases. It helps in scenarios where we are forced 
to downgrade a release, say due to major issues. We can simply announce it as 
not stable retroactively, change the pointers on our website and move on.

Thoughts?

Thanks,
+Vinod

[1] http://markmail.org/thread/ogzk4phj6wsdpssu

On Apr 21, 2015, at 4:59 PM, Vinod Kumar Vavilapalli  
wrote:

> 
> Sure, I agree it's better to have clear guidelines and scheme. Let me fork 
> this thread about that.
> 
> Re 2.7.0, I just forgot about the naming initially though I was clear in the 
> discussion/voting. I so had to end up calling it alpha outside of the release 
> artifact naming.
> 
> Thanks
> +Vinod
> 
> On Apr 21, 2015, at 4:26 PM, Andrew Wang  wrote:
> 
>> I would also like to support Karthik's proposal on the release thread about
>> version numbering. 2.7.0 being "alpha" is pretty confusing since all of the
>> other 2.x releases since GA have been stable. I think users would prefer a
>> version like "2.8.0-alpha1" instead, which is very clear and similar to
>> what we did for 2.0 and 2.1. Then we release 2.8.0 when we're actually
>> stable.
>> 
>> I don't know if it's retroactively possible to do this for 2.7.0, but it's
>> something to consider for the next 2.7 alpha or beta or whatever.
>> 

Re: [DISCUSS] Release numbering for stable 2.8 and beyond

[Andrew Wang]

Thanks for forking this Vinod,

Linux used to do the odd/even minor versions for unstable/stable, but that
went away when 2.6 lasted forever. With the 3.x and 4.x I think it's just
always stable. The odd/even though was at least a convention everyone knew
about.

Stack can comment better than I about HBase versioning, but my impression
is that they do something similar. Evens are stable, odds are not.

I'd be okay with an even/odd convention, but it would still be our third
versioning convention in as many years.

I like (2) since it's similar to what we did with 2.0 and 2.1. Our contract
since 2.2 has been that everything is compatible / stable. Adding a tag
makes it very clear that we are now doing unstable releases. This also has
the pleasing property that we culminate in a 2.x.0, rather than stable
starting at 2.x.4 or whatever. Much simpler than having to consult a
website.

Re (2), we should add a number (e.g. alpha2) too in case there's more than
one alpha or beta too.

Best,
Andrew

On Wed, Apr 22, 2015 at 2:17 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com> wrote:

> Forking the thread.
>
> In the previous 2.7.1 thread [1], there were enough yays to my proposal to
> wait for a bug-fix release or two before calling a 2.x release stable.
> There were some concerns about the naming.
>
> We have two options, taking 2.8 as an example
>  (1) Release 2.8.0, call it as an alpha in documentation and release
> notes, wait for a 2.8.1/2.8.2 reasonably stable enough to be called as the
> first stable release of 2.8.
>  (2) Release 2.8.0-alpha, 2.8.0-beta etc before culminating in a 2.8.0
> stable release.
>
> (1) is what I preferred first up. This is what HBase used to do, and far
> beyond, in the linux kernel releases. It helps in scenarios where we are
> forced to downgrade a release, say due to major issues. We can simply
> announce it as not stable retroactively, change the pointers on our website
> and move on.
>
> Thoughts?
>
> Thanks,
> +Vinod
>
> [1] http://markmail.org/thread/ogzk4phj6wsdpssu
>
> On Apr 21, 2015, at 4:59 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
>
> >
> > Sure, I agree it's better to have clear guidelines and scheme. Let me
> fork this thread about that.
> >
> > Re 2.7.0, I just forgot about the naming initially though I was clear in
> the discussion/voting. I so had to end up calling it alpha outside of the
> release artifact naming.
> >
> > Thanks
> > +Vinod
> >
> > On Apr 21, 2015, at 4:26 PM, Andrew Wang 
> wrote:
> >
> >> I would also like to support Karthik's proposal on the release thread
> about
> >> version numbering. 2.7.0 being "alpha" is pretty confusing since all of
> the
> >> other 2.x releases since GA have been stable. I think users would
> prefer a
> >> version like "2.8.0-alpha1" instead, which is very clear and similar to
> >> what we did for 2.0 and 2.1. Then we release 2.8.0 when we're actually
> >> stable.
> >>
> >> I don't know if it's retroactively possible to do this for 2.7.0, but
> it's
> >> something to consider for the next 2.7 alpha or beta or whatever.
> >>
>
>

Re: [DISCUSS] Release numbering for stable 2.8 and beyond

[Karthik Kambatla]

Approach (1) seems like a good way to handle stability concerns people
might have. If we explicitly distinguish between current and stable (i.e.,
not set them both to the latest release). It would be nice to do a VOTE for
calling a release stable.

I would use approach (2) for compatibility concerns. Ideally, I wouldn't
ship anything in a release unless we are sure we can preserve its
compatibility. If we really think a feature is not ready, we could always
guard them with private configs that devs or beta-testers could enable to
use. In the past, there have been proposals about labeling specific
features alpha and beta. I am open to considering it provided we define
what those terms mean, ideally in our compat guidelines.

On Wed, Apr 22, 2015 at 2:46 PM, Andrew Wang 
wrote:

> Thanks for forking this Vinod,
>
> Linux used to do the odd/even minor versions for unstable/stable, but that
> went away when 2.6 lasted forever. With the 3.x and 4.x I think it's just
> always stable. The odd/even though was at least a convention everyone knew
> about.
>
> Stack can comment better than I about HBase versioning, but my impression
> is that they do something similar. Evens are stable, odds are not.
>
> I'd be okay with an even/odd convention, but it would still be our third
> versioning convention in as many years.
>
> I like (2) since it's similar to what we did with 2.0 and 2.1. Our contract
> since 2.2 has been that everything is compatible / stable. Adding a tag
> makes it very clear that we are now doing unstable releases. This also has
> the pleasing property that we culminate in a 2.x.0, rather than stable
> starting at 2.x.4 or whatever. Much simpler than having to consult a
> website.
>
> Re (2), we should add a number (e.g. alpha2) too in case there's more than
> one alpha or beta too.
>
> Best,
> Andrew
>
> On Wed, Apr 22, 2015 at 2:17 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
>
> > Forking the thread.
> >
> > In the previous 2.7.1 thread [1], there were enough yays to my proposal
> to
> > wait for a bug-fix release or two before calling a 2.x release stable.
> > There were some concerns about the naming.
> >
> > We have two options, taking 2.8 as an example
> >  (1) Release 2.8.0, call it as an alpha in documentation and release
> > notes, wait for a 2.8.1/2.8.2 reasonably stable enough to be called as
> the
> > first stable release of 2.8.
> >  (2) Release 2.8.0-alpha, 2.8.0-beta etc before culminating in a 2.8.0
> > stable release.
> >
> > (1) is what I preferred first up. This is what HBase used to do, and far
> > beyond, in the linux kernel releases. It helps in scenarios where we are
> > forced to downgrade a release, say due to major issues. We can simply
> > announce it as not stable retroactively, change the pointers on our
> website
> > and move on.
> >
> > Thoughts?
> >
> > Thanks,
> > +Vinod
> >
> > [1] http://markmail.org/thread/ogzk4phj6wsdpssu
> >
> > On Apr 21, 2015, at 4:59 PM, Vinod Kumar Vavilapalli <
> > vino...@hortonworks.com> wrote:
> >
> > >
> > > Sure, I agree it's better to have clear guidelines and scheme. Let me
> > fork this thread about that.
> > >
> > > Re 2.7.0, I just forgot about the naming initially though I was clear
> in
> > the discussion/voting. I so had to end up calling it alpha outside of the
> > release artifact naming.
> > >
> > > Thanks
> > > +Vinod
> > >
> > > On Apr 21, 2015, at 4:26 PM, Andrew Wang 
> > wrote:
> > >
> > >> I would also like to support Karthik's proposal on the release thread
> > about
> > >> version numbering. 2.7.0 being "alpha" is pretty confusing since all
> of
> > the
> > >> other 2.x releases since GA have been stable. I think users would
> > prefer a
> > >> version like "2.8.0-alpha1" instead, which is very clear and similar
> to
> > >> what we did for 2.0 and 2.1. Then we release 2.8.0 when we're actually
> > >> stable.
> > >>
> > >> I don't know if it's retroactively possible to do this for 2.7.0, but
> > it's
> > >> something to consider for the next 2.7 alpha or beta or whatever.
> > >>
> >
> >
>



-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.

http://five.sentenc.es

Re: IMPORTANT: testing patches for branches

[Niels Basjes]

Perhaps a script is due that creates the patch file with -exactly- the
right name.
Something like what HBase has as dev-support/make_patch.sh perhaps?

On Wed, Apr 22, 2015 at 10:30 PM, Allen Wittenauer  wrote:

>
> Oh, this is also in the release notes, but one can use a git reference #
> as well. :) (with kudos to OOM for the idea.)
>
> On Apr 22, 2015, at 8:57 PM, Allen Wittenauer  wrote:
>
> >
> > More than likely. It probably needs more testing (esp under Jenkins).
> >
> > It should be noted that the code in test-patch.sh has lots of problems
> with branch-0, minor, and micro releases.  But for major releases, it seems
> to work well for me. :)
> >
> > On Apr 22, 2015, at 8:45 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
> >
> >> Does this mean HADOOP-7435 is no longer needed / closeable as dup?
> >>
> >> Thanks
> >> +Vinod
> >>
> >> On Apr 22, 2015, at 12:34 PM, Allen Wittenauer 
> wrote:
> >>
> >>>
> >>> Hey gang,
> >>>
> >>> Just so everyone is aware, if you are working on a patch for
> either a feature branch or a major branch, if you name the patch with the
> branch name following the spec in HowToContribute (and a few other ways…
> test-patch tries to figure it out!), test-patch.sh *should* be switching
> the repo over to that branch for testing.
> >>>
> >>> For example,  naming a patch foo-branch-2.01.patch should get
> tested on branch-2.  Naming a patch foo-HDFS-7285.00.patch should get
> tested on the HDFS-7285 branch.
> >>>
> >>> This hopefully means that there should really be no more ‘blind’
> +1’s to patches that go to branches.  The “we only test against trunk”
> argument is no longer valid. :)
> >>>
> >>>
> >>
> >
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

RE: IMPORTANT: testing patches for branches

[Zheng, Kai]

Hi Allen,

This sounds great. 

>> Naming a patch foo-HDFS-7285.00.patch should get tested on the HDFS-7285 
>> branch.
Does it happen locally in developer's machine when running test-patch.sh, or 
also mean something in Hadoop Jenkins building when a JIRA becoming patch 
available? Thanks.

Regards,
Kai

-Original Message-
From: Allen Wittenauer [mailto:a...@altiscale.com] 
Sent: Thursday, April 23, 2015 3:35 AM
To: common-dev@hadoop.apache.org
Cc: yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org
Subject: IMPORTANT: testing patches for branches


Hey gang, 

Just so everyone is aware, if you are working on a patch for either a 
feature branch or a major branch, if you name the patch with the branch name 
following the spec in HowToContribute (and a few other ways... test-patch tries 
to figure it out!), test-patch.sh *should* be switching the repo over to that 
branch for testing. 

For example,  naming a patch foo-branch-2.01.patch should get tested on 
branch-2.  Naming a patch foo-HDFS-7285.00.patch should get tested on the 
HDFS-7285 branch.

This hopefully means that there should really be no more 'blind' +1's 
to patches that go to branches.  The "we only test against trunk" argument is 
no longer valid. :)

Re: Set minimum version of Hadoop 3 to JDK 8

[Robert Kanter]

You're absolutely right Steve.  We should get the jobs under control.

At the very least, before we move to JDK 8 in Jenkins, we should get the
Java8 nightly builds working.  I didn't realize we had these, and I had
thought all compiling and most test issues with JDK8 were already fixed
based on what HADOOP-11090 looks like; but I guess there's more.

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-common-trunk-Java8/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Java8/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/

I'll take a deeper look at these and file JIRAs as necessary later today.




On Wed, Apr 22, 2015 at 3:13 AM, Steve Loughran 
wrote:

>
> > On 21 Apr 2015, at 23:31, Andrew Wang  wrote:
> >
> > Hey Robert,
> >
> > As a first step, could we try switching all our precommit and nightly
> > builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe
> > to do in any case since it'll still target 1.7.
>
> +1
>
> we have jenkin builds running on Java 8 -its just they have enough
> longstanding and intermittent test failures that they get ignored
>
> I've been trying to keep hadoop-common somewhat under control, but
>
> -YARN is failing due to to hard coded ports in tests - this needs to be
> fixed urgently: YARN-3528
> -HDFS build process itself is failing with some config problem checkstyle
>
>
> >
> > I'll note that HADOOP-10530 details the pain Steve went through switching
> > us to JDK7. Might be some lessons learned about how to do this transition
> > more smoothly.
> >
>
>
> The key thing is to have Jenkins happy before trying to change anything.
> And make sure that "happy" means that jenkins really is building and
> testing against the version of Java that you think it is. I spent about 15
> minutes/hour of my weekend kicking off builds and tuning them.  -I
> volunteered Colin Patrick McCabe to to that bit of the switch, though of
> course he doesn't have to accept that opportunuty (
> https://issues.apache.org/jira/browse/HADOOP-10530?focusedCommentId=14239250&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14239250
>
> Someone needs to take ownership of the "Jenkins builds are broken" problem
> -ideally someone in each subproject. That's a prerequisite to any major
> changes in the build process, but should also be best practice anyway. Ask
> your QE teams "would you like the ASF projects to care whether their
> jenkins test runs succeed" and see what they say.
>
> as a reminder, all the jenkins builds are here
> https://builds.apache.org/view/H-L/view/Hadoop/
>
> please look at them and try to fix them —without even waiting for a switch
> to Java 8

Re: [RESULT][VOTE] Release Apache Hadoop 2.7.0 RC0

[Vinod Kumar Vavilapalli]

It took a while for the artifact distribution to come around, I initially had 
trouble pushing them from home too.

Anyways, I just updated the website too. Sending an announcement now..

Thanks,
+Vinod

On Apr 20, 2015, at 8:06 PM, Vinod Kumar Vavilapalli  
wrote:

> With 22 +1s (7 binding), one +0 and no -1s the vote passes.
> 
> Thanks for everyone who tried the release and voted.
> 
> I'll push the bits and send out an announcement.
> 
> Thanks
> +Vinod
> 
> On Apr 10, 2015, at 4:44 PM, Vinod Kumar Vavilapalli  
> wrote:
> 
>> Hi all,
>> 
>> I've created a release candidate RC0 for Apache Hadoop 2.7.0.
>> 
>> The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.7.0-RC0/
>> 
>> The RC tag in git is: release-2.7.0-RC0
>> 
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1017/
>> 
>> As discussed before
>> - This release will only work with JDK 1.7 and above
>> - I’d like to use this as a starting release for 2.7.x [1], depending on
>> how it goes, get it stabilized and potentially use a 2.7.1 in a few
>> weeks as the stable release.
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1]: A 2.7.1 release to follow up 2.7.0
>> http://markmail.org/thread/zwzze6cqqgwq4rmw
> 

[jira] [Created] (HADOOP-11869) checkstyle rules/script need re-visiting

[Sidharta Seethana (JIRA)]

Sidharta Seethana created HADOOP-11869:
--

 Summary: checkstyle rules/script need re-visiting
 Key: HADOOP-11869
 URL: https://issues.apache.org/jira/browse/HADOOP-11869
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sidharta Seethana


There seem to be a lot of arcane errors being caused by the checkstyle 
rules/script. Real issues tend to be buried in this noise. Some examples :

1. "Line is longer than 80 characters" - this shows up even for cases like 
import statements, package names
2. "Missing a Javadoc comment." - for every private member including cases like 
"Configuration conf". 

Having rules like these will result in a large number of pre-commit job 
failures. We should fine tune the rules used for checkstyle. 


 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11870) [JDK8] AuthenticationFilter, CertificateUtil, SignerSecretProviders, KeyAuthorizationKeyProvider Javadoc issues

[Robert Kanter (JIRA)]

Robert Kanter created HADOOP-11870:
--

 Summary: [JDK8] AuthenticationFilter, CertificateUtil, 
SignerSecretProviders, KeyAuthorizationKeyProvider Javadoc issues
 Key: HADOOP-11870
 URL: https://issues.apache.org/jira/browse/HADOOP-11870
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 3.0.0
 Environment: Jenkins on Java8
Reporter: Robert Kanter
Assignee: Robert Kanter


Jenkins on Java8 is failing due to a number of Javadoc violations that are now 
considered ERRORs in the following classes:
- AuthenticationFilter.java
- CertificateUtil.java
- RolloverSignerSecretProvider.java
- SignerSecretProvider.java
- ZKSignerSecretProvider.java
- KeyAuthorizationKeyProvider.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11871) Improve RSRawErasureDecoder avoiding unnecessary compuating and recovering

[Kai Zheng (JIRA)]

Kai Zheng created HADOOP-11871:
--

 Summary: Improve RSRawErasureDecoder avoiding unnecessary 
compuating and recovering
 Key: HADOOP-11871
 URL: https://issues.apache.org/jira/browse/HADOOP-11871
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng


As discussed in HADOOP-11847, due to and inherited from the limitation of 
HDFS-RAID codes, {{RSErasureDecoder}} has to compute and recover unnecessarily 
some units even they're not really erased, but just not to read.
This is to resolve the limiation and improve {{RSErasureDecoder}} for better 
performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)