Build failed in Jenkins: Hadoop-Common-0.23-Build #1073

2014-09-15 Thread Apache Jenkins Server 
See 

--
[...truncated 8263 lines...]
Running org.apache.hadoop.io.TestBloomMapFile
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.882 sec
Running org.apache.hadoop.io.TestObjectWritableProtos
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.321 sec
Running org.apache.hadoop.io.TestTextNonUTF8
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 sec
Running org.apache.hadoop.io.nativeio.TestNativeIO
Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 0.159 sec
Running org.apache.hadoop.io.TestSortedMapWritable
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.193 sec
Running org.apache.hadoop.io.TestMapFile
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.655 sec
Running org.apache.hadoop.io.TestUTF8
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.312 sec
Running org.apache.hadoop.io.TestBoundedByteArrayOutputStream
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.042 sec
Running org.apache.hadoop.io.retry.TestRetryProxy
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.209 sec
Running org.apache.hadoop.io.retry.TestFailoverProxy
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.193 sec
Running org.apache.hadoop.io.TestSetFile
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.085 sec
Running org.apache.hadoop.io.serializer.TestWritableSerialization
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.316 sec
Running org.apache.hadoop.io.serializer.TestSerializationFactory
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.283 sec
Running org.apache.hadoop.io.serializer.avro.TestAvroSerialization
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.537 sec
Running org.apache.hadoop.util.TestGenericOptionsParser
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.691 sec
Running org.apache.hadoop.util.TestReflectionUtils
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.514 sec
Running org.apache.hadoop.util.TestJarFinder
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.782 sec
Running org.apache.hadoop.util.TestPureJavaCrc32
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.288 sec
Running org.apache.hadoop.util.TestHostsFileReader
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.184 sec
Running org.apache.hadoop.util.TestShutdownHookManager
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.144 sec
Running org.apache.hadoop.util.TestDiskChecker
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.491 sec
Running org.apache.hadoop.util.TestStringUtils
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.136 sec
Running org.apache.hadoop.util.TestGenericsUtil
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec
Running org.apache.hadoop.util.TestAsyncDiskService
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.126 sec
Running org.apache.hadoop.util.TestProtoUtil
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec
Running org.apache.hadoop.util.TestDataChecksum
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.185 sec
Running org.apache.hadoop.util.TestRunJar
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.127 sec
Running org.apache.hadoop.util.TestOptions
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.08 sec
Running org.apache.hadoop.util.TestShell
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.195 sec
Running org.apache.hadoop.util.TestIndexedSort
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.614 sec
Running org.apache.hadoop.util.TestStringInterner
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec
Running org.apache.hadoop.record.TestRecordVersioning
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.161 sec
Running org.apache.hadoop.record.TestBuffer
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.055 sec
Running org.apache.hadoop.record.TestRecordIO
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.18 sec
Running org.apache.hadoop.security.TestGroupFallback
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 sec
Running org.apache.hadoop.security.TestGroupsCaching
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.278 sec
Running org.apache.hadoop.security.TestProxyUserFromEnv
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.356 sec
Running org.apache.hadoop.security.TestUserGroupInformation
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.676 sec
Running org.apache.hadoop.security.TestJNIGroupsMapping
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.142 sec
Runni

mapreduce precommit build does not show test results

2014-09-15 Thread Kihwal Lee 
Ever since build #8430 in 8/29, test results are not showing up.  I see there 
was build config changes around that time. Anyone with the right permission 
care to take a look? 

Thanks,Kihwal

In hindsight... Re: Thinking ahead to hadoop-2.6

2014-09-15 Thread Allen Wittenauer 

It’s now September.  With the passage of time, I have a lot of doubts 
about this plan and where that trajectory takes us.

* The list of changes that are already in branch-2 scare the crap out of any 
risk adverse person (Hello to my fellow operations people!). Not only are the 
number of changes extremely high, but in addition there are a lot of major, 
blockbuster features in what is supposed to be a minor release.  Combined with 
the fact that we’ve had to do some micro releases, it seems to hint that 
branch-2 is getting less stable over time.

*  One of the plans talked about was rolling a 2.7 release that drops JDK6 and 
makes JDK7 the standard.  If 2.7 comes after 2.6 in October, date wise  makes 
it somewhere around January 2015.  JDK7 EOL’s in April 2015.  So we’ll have a 
viable JDK7 release for exactly 3 months.  Frankly, it is too late for us to 
talk about JDK7 and we need to start thinking about JDK8.  

* trunk is currently sitting at 3 years old.  There is a lot of stuff that has 
been hanging around that really needs to get into people hands so that we can 
start stabilizing it for a “real” release.  


To me this all says one thing:

Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha 
with JDK8 as the minimum.  2.5.1 becomes the base for all sustaining work.  
This gives the rest of the community time to move to JDK8 if they haven’t 
already.  For downstream vendors, it gives a roadmap for their customers who 
will be asking about JDK8 sooner rather than later.  By the time 3.0 
stabilizes, we’re probably looking at April, which is perfect timing.

One of the issues I’ve heard mention is that 3.0 doesn’t have anything 
“compelling” in it.  Well, dropping 2.6 makes the feature list the carrot, JDK8 
support is obviously the stick.

Thoughts?




On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan  wrote:

> Thanks for initiating the thread Arun.
> 
> Can we add YARN-1051  to
> the list? We have most of the patches for the sub-JIRAs under review and
> have committed a couple.
> 
> -Subru
> 
> -- Forwarded message --
> 
> From: Arun C Murthy 
> 
> Date: Tue, Aug 12, 2014 at 1:34 PM
> 
> Subject: Thinking ahead to hadoop-2.6
> 
> To: "common-dev@hadoop.apache.org" , "
> hdfs-...@hadoop.apache.org" , "
> mapreduce-...@hadoop.apache.org" ,
> 
> "yarn-...@hadoop.apache.org" 
> 
> 
> 
> 
> 
> Folks,
> 
> 
> 
> With hadoop-2.5 nearly done, it's time to start thinking ahead to
> hadoop-2.6.
> 
> 
> 
> Currently, here is the Roadmap per the wiki:
> 
> 
> 
>• HADOOP
> 
>• Credential provider HADOOP-10607
> 
>• HDFS
> 
>• Heterogeneous storage (Phase 2) - Support APIs for using
> storage tiers by the applications HDFS-5682
> 
>• Memory as storage tier HDFS-5851
> 
>• YARN
> 
>• Dynamic Resource Configuration YARN-291
> 
>• NodeManager Restart YARN-1336
> 
>• ResourceManager HA Phase 2 YARN-556
> 
>• Support for admin-specified labels in YARN YARN-796
> 
>• Support for automatic, shared cache for YARN application
> artifacts YARN-1492
> 
>• Support NodeGroup layer topology on YARN YARN-18
> 
>• Support for Docker containers in YARN YARN-1964
> 
>• YARN service registry YARN-913
> 
> 
> 
> My suspicion is, as is normal, some will make the cut and some won't.
> 
> Please do add/subtract from the list as appropriate. Ideally, it would be
> good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence.
> 
> 
> 
> More importantly, as we discussed previously, we'd like hadoop-2.6 to be
> the *last* Apache Hadoop 2.x release which support JDK6. I'll start a
> discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see
> how they feel about this.
> 
> 
> 
> thanks,
> 
> Arun
> 
> 
> 
> 
> 
> --
> 
> CONFIDENTIALITY NOTICE
> 
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

2014-09-15 Thread Colin McCabe 
On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer  wrote:
>
> It’s now September.  With the passage of time, I have a lot of doubts 
> about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of any 
> risk adverse person (Hello to my fellow operations people!). Not only are the 
> number of changes extremely high, but in addition there are a lot of major, 
> blockbuster features in what is supposed to be a minor release.  Combined 
> with the fact that we’ve had to do some micro releases, it seems to hint that 
> branch-2 is getting less stable over time.

I don't see what is so scary about 2.6, can you be more concrete?  It
seems like a pretty normal release to me and most of the new features
are optional.

I also don't see why you think that "branch-2 is getting less stable
over time."  Actually, I think that branch-2 has gotten more stable
over time as people have finally gotten around to upgrading from 1.x
or earlier, and contributed their efforts to addressing regressions in
branch-2.

> *  One of the plans talked about was rolling a 2.7 release that drops JDK6 
> and makes JDK7 the standard.  If 2.7 comes after 2.6 in October, date wise  
> makes it somewhere around January 2015.  JDK7 EOL’s in April 2015.  So we’ll 
> have a viable JDK7 release for exactly 3 months.  Frankly, it is too late for 
> us to talk about JDK7 and we need to start thinking about JDK8.
>
> * trunk is currently sitting at 3 years old.  There is a lot of stuff that 
> has been hanging around that really needs to get into people hands so that we 
> can start stabilizing it for a “real” release.

We have been pretty careful about minimizing trunk's divergence from
branch-2.  I can't think of an example of anything in trunk that
"really needs to get into people's hands"-- did I forget something?

>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha 
> with JDK8 as the minimum.  2.5.1 becomes the base for all sustaining work.  
> This gives the rest of the community time to move to JDK8 if they haven’t 
> already.  For downstream vendors, it gives a roadmap for their customers who 
> will be asking about JDK8 sooner rather than later.  By the time 3.0 
> stabilizes, we’re probably looking at April, which is perfect timing.
>
> One of the issues I’ve heard mention is that 3.0 doesn’t have 
> anything “compelling” in it.  Well, dropping 2.6 makes the feature list the 
> carrot, JDK8 support is obviously the stick.
>
> Thoughts?

As we've discussed before, supporting JDK8 is very different from
forcing people to use JDK8.  branch-2 and Hadoop 2.6 most certainly
should support JDK8, and most certainly NOT force people to use JDK8.
Cloudera has been using JDK7 internally for a long time, and
recommending it to customers too.  Some developers are using JDK8 as
well.  It works fine (although I'm sure there will be bugs and
workarounds that get reported and fixed as more people migrate).  I
don't see this particular issue as a reason to change the schedule.

best,
Colin


>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan  wrote:
>
>> Thanks for initiating the thread Arun.
>>
>> Can we add YARN-1051  to
>> the list? We have most of the patches for the sub-JIRAs under review and
>> have committed a couple.
>>
>> -Subru
>>
>> -- Forwarded message --
>>
>> From: Arun C Murthy 
>>
>> Date: Tue, Aug 12, 2014 at 1:34 PM
>>
>> Subject: Thinking ahead to hadoop-2.6
>>
>> To: "common-dev@hadoop.apache.org" , "
>> hdfs-...@hadoop.apache.org" , "
>> mapreduce-...@hadoop.apache.org" ,
>>
>> "yarn-...@hadoop.apache.org" 
>>
>>
>>
>>
>>
>> Folks,
>>
>>
>>
>> With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> hadoop-2.6.
>>
>>
>>
>> Currently, here is the Roadmap per the wiki:
>>
>>
>>
>>• HADOOP
>>
>>• Credential provider HADOOP-10607
>>
>>• HDFS
>>
>>• Heterogeneous storage (Phase 2) - Support APIs for using
>> storage tiers by the applications HDFS-5682
>>
>>• Memory as storage tier HDFS-5851
>>
>>• YARN
>>
>>• Dynamic Resource Configuration YARN-291
>>
>>• NodeManager Restart YARN-1336
>>
>>• ResourceManager HA Phase 2 YARN-556
>>
>>• Support for admin-specified labels in YARN YARN-796
>>
>>• Support for automatic, shared cache for YARN application
>> artifacts YARN-1492
>>
>>• Support NodeGroup layer topology on YARN YARN-18
>>
>>• Support for Docker containers in YARN YARN-1964
>>
>>• YARN service registry YARN-913
>>
>>
>>
>> My suspicion is, as is normal, some will make the cut and some won't.
>>
>> Please do add/subtract from the list as appropriate. Ideally, it would be
>> good to ship hadoop-2.6 in

Re: Git repo ready to use

2014-09-15 Thread Todd Lipcon 
Hey all,

For those of you who like to see the entire history of a file going back to
2006, I found I had to add a new graft to .git/info/grafts:

# Project un-split in new writable git repo
a196766ea07775f18ded69bd9e8d239f8cfd3ccc
928d485e2743115fe37f9d123ce9a635c5afb91a
cd66945f62635f589ff93468e94c0039684a8b6d
77f628ff5925c25ba2ee4ce14590789eb2e7b85b

FWIW, my entire file now contains:

# Project split
5128a9a453d64bfe1ed978cf9ffed27985eeef36
6c16dc8cf2b28818c852e95302920a278d07ad0c
6a3ac690e493c7da45bbf2ae2054768c427fd0e1
6c16dc8cf2b28818c852e95302920a278d07ad0c
546d96754ffee3142bcbbf4563c624c053d0ed0d
6c16dc8cf2b28818c852e95302920a278d07ad0c
4e569e629a98a4ef5326e5d25a84c7d57b5a8f7a
c78078dd2283e2890018ff0e87d751c86163f99f

# Project un-split in new writable git repo
a196766ea07775f18ded69bd9e8d239f8cfd3ccc
928d485e2743115fe37f9d123ce9a635c5afb91a
cd66945f62635f589ff93468e94c0039684a8b6d
77f628ff5925c25ba2ee4ce14590789eb2e7b85b

which seems to do a good job for me (not sure if the first few lines are
necessary anymore in the latest world)

-Todd



On Fri, Sep 12, 2014 at 11:31 AM, Colin McCabe 
wrote:

> It's an issue with test-patch.sh.  See
> https://issues.apache.org/jira/browse/HADOOP-11084
>
> best,
> Colin
>
> On Mon, Sep 8, 2014 at 3:38 PM, Andrew Wang 
> wrote:
> > We're still not seeing findbugs results show up on precommit runs. I see
> > that we're archiving "../patchprocess/*", and Ted thinks that since it's
> > not in $WORKSPACE it's not getting picked up. Can we get confirmation of
> > this issue? If so, we could just add "patchprocess" to the toplevel
> > .gitignore.
> >
> > On Thu, Sep 4, 2014 at 8:54 AM, Sangjin Lee  wrote:
> >
> >> That's good to know. Thanks.
> >>
> >>
> >> On Wed, Sep 3, 2014 at 11:15 PM, Vinayakumar B  >
> >> wrote:
> >>
> >> > I think its still pointing to old svn repository which is just read
> only
> >> > now.
> >> >
> >> > You can use latest mirror:
> >> > https://github.com/apache/hadoop
> >> >
> >> > Regards,
> >> > Vinay
> >> > On Sep 4, 2014 11:37 AM, "Sangjin Lee"  wrote:
> >> >
> >> > > It seems like the github mirror at
> >> > https://github.com/apache/hadoop-common
> >> > > has stopped getting updates as of 8/22. Could this mirror have been
> >> > broken
> >> > > by the git transition?
> >> > >
> >> > > Thanks,
> >> > > Sangjin
> >> > >
> >> > >
> >> > > On Fri, Aug 29, 2014 at 11:51 AM, Ted Yu 
> wrote:
> >> > >
> >> > > > From https://builds.apache.org/job/Hadoop-hdfs-trunk/1854/console
> :
> >> > > >
> >> > > > ERROR: No artifacts found that match the file pattern
> >> > > > "trunk/hadoop-hdfs-project/*/target/*.tar.gz". Configuration
> >> > > > error?ERROR  >:
> >> > > > ?trunk/hadoop-hdfs-project/*/target/*.tar.gz? doesn?t match
> anything,
> >> > > > but ?hadoop-hdfs-project/*/target/*.tar.gz? does. Perhaps that?s
> what
> >> > > > you mean?
> >> > > >
> >> > > >
> >> > > > I corrected the path to hdfs tar ball.
> >> > > >
> >> > > >
> >> > > > FYI
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, Aug 29, 2014 at 8:48 AM, Alejandro Abdelnur <
> >> t...@cloudera.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > it seems we missed updating the HADOOP precommit job to use
> Git, it
> >> > was
> >> > > > > still using SVN. I've just updated it.
> >> > > > >
> >> > > > > thx
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Aug 28, 2014 at 9:26 PM, Ted Yu 
> >> wrote:
> >> > > > >
> >> > > > > > Currently patchprocess/ (contents shown below) is one level
> >> higher
> >> > > than
> >> > > > > > ${WORKSPACE}
> >> > > > > >
> >> > > > > > diffJavadocWarnings.txt
> >> > > >  newPatchFindbugsWarningshadoop-hdfs.html
> >> > > > > >  patchFindBugsOutputhadoop-hdfs.txt
> >> patchReleaseAuditOutput.txt
> >> > > > > >  trunkJavadocWarnings.txt
> >> > > > > > filteredPatchJavacWarnings.txt
> >> > > newPatchFindbugsWarningshadoop-hdfs.xml
> >> > > > > > patchFindbugsWarningshadoop-hdfs.xml
> >> patchReleaseAuditWarnings.txt
> >> > > > > > filteredTrunkJavacWarnings.txt  patch
> >> > > > > > patchJavacWarnings.txttestrun_hadoop-hdfs.txt
> >> > > > > > jirapatchEclipseOutput.txt
> >> > > > > >  patchJavadocWarnings.txt  trunkJavacWarnings.txt
> >> > > > > >
> >> > > > > > Under Files to archive input box of
> >> > PreCommit-HDFS-Build/configure, I
> >> > > > > saw:
> >> > > > > >
> >> > > > > > '../patchprocess/*' doesn't match anything, but '*' does.
> Perhaps
> >> > > > that's
> >> > > > > > what you mean?
> >> > > > > >
> >> > > > > > I guess once patchprocess is moved back under ${WORKSPACE}, a
> lot
> >> > of
> >> > > > > things
> >> > > > > > would be back to normal.
> >> > > > > >
> >> > > > > > Cheers
> >> > > > > >
> >> > > > > > On Thu, Aug 28, 2014 at 9:16 PM, Alejandro Abdelnur <
> >> > > t...@cloudera.com
> >> > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > i'm also seeing broken links for javadocs warnings.
> >> > > > > > >
>

[jira] [Created] (HADOOP-11093) HADOOP_DAEMON_ROOT_LOGGER is used inconsistently

2014-09-15 Thread Allen Wittenauer (JIRA) 
Allen Wittenauer created HADOOP-11093:
-

 Summary: HADOOP_DAEMON_ROOT_LOGGER is used inconsistently
 Key: HADOOP-11093
 URL: https://issues.apache.org/jira/browse/HADOOP-11093
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Allen Wittenauer






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11094) HADOOP_AUDIT_LOGGER and HADOOP_SECURITY_LOGGER are poorly defined

2014-09-15 Thread Allen Wittenauer (JIRA) 
Allen Wittenauer created HADOOP-11094:
-

 Summary: HADOOP_AUDIT_LOGGER and HADOOP_SECURITY_LOGGER  are 
poorly defined
 Key: HADOOP-11094
 URL: https://issues.apache.org/jira/browse/HADOOP-11094
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Allen Wittenauer


The semantics around HADOOP_AUDIT_LOGGER and HADOOP_SECURITY_LOGGER  appear to 
be poorly defined. 

See comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.5.1 RC0

2014-09-15 Thread Chen He 
+1 non-binding

download source code, build from scratch, deploy to a single node, and run
sleep job succeed.

Regards!

Chen

On Thu, Sep 11, 2014 at 9:53 PM, Akira AJISAKA 
wrote:

> Thanks Karthik for updating the wiki. Now I'm +1 (non-binding).
>
> Regards,
> Akira
>
>
> (2014/09/11 2:10), Karthik Kambatla wrote:
>
>> Thanks for reporting the mistake in the documentation, Akira. While it is
>> good to fix it, I am not sure it is big enough to warrant another RC,
>> particularly because 2.5.1 is very much 2.5.0 done right.
>>
>> I just updated the how-to-release wiki to capture this step in the release
>> process, so we don't miss it in the future.
>>
>> On Mon, Sep 8, 2014 at 11:37 PM, Akira AJISAKA <
>> ajisa...@oss.nttdata.co.jp>
>> wrote:
>>
>>  -0 (non-binding)
>>>
>>> In the document, "Apache Hadoop 2.5.1 is a minor release in the 2.x.y
>>> release line, buliding upon the previous stable release 2.4.1."
>>>
>>> Hadoop 2.5.1 is a point release. Filed HADOOP-11078 to track this.
>>>
>>> Regards,
>>> Akira
>>>
>>>
>>> (2014/09/09 0:51), Karthik Kambatla wrote:
>>>
>>>  +1 (non-binding)

 Built the source tarball, brought up a pseudo-distributed cluster and
 ran
 a
 few MR jobs. Verified documentation and size of the binary tarball.

 On Fri, Sep 5, 2014 at 5:18 PM, Karthik Kambatla 
 wrote:

   Hi folks,

>
> I have put together a release candidate (RC0) for Hadoop 2.5.1.
>
> The RC is available at: http://people.apache.org/~
> kasha/hadoop-2.5.1-RC0/
> The RC git tag is release-2.5.1-RC0
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1010/
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the now usual 5
> days.
>
> Thanks
> Karthik
>
>
>

>>>
>>
>

[jira] [Resolved] (HADOOP-10675) Add server-side encryption functionality to s3a

2014-09-15 Thread David S. Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang resolved HADOOP-10675.

   Resolution: Fixed
Fix Version/s: 2.6.0

This was fixed as part of HADOOP-10400.

> Add server-side encryption functionality to s3a
> ---
>
> Key: HADOOP-10675
> URL: https://issues.apache.org/jira/browse/HADOOP-10675
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Affects Versions: 2.4.0
>Reporter: David S. Wang
>Assignee: David S. Wang
> Fix For: 2.6.0
>
> Attachments: HADOOP-10675-1.patch, HADOOP-10675-2.patch
>
>
> The current patch for s3a in HADOOP-10400 does not have the capability to 
> specify server-side encryption.  This JIRA will track the addition of such 
> functionality to HADOOP-10400, similar to what was done in HADOOP-10568 for 
> s3n.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-10676) S3AOutputStream not reading new config knobs for multipart configs

2014-09-15 Thread David S. Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang resolved HADOOP-10676.

   Resolution: Fixed
Fix Version/s: 2.6.0

This was fixed as part of HADOOP-10400.

> S3AOutputStream not reading new config knobs for multipart configs
> --
>
> Key: HADOOP-10676
> URL: https://issues.apache.org/jira/browse/HADOOP-10676
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Affects Versions: 2.4.0
>Reporter: David S. Wang
>Assignee: David S. Wang
> Fix For: 2.6.0
>
> Attachments: HADOOP-10676-1.patch
>
>
> S3AOutputStream.java does not have the code to read the new config knobs for 
> multipart configs.  This patch will add that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-10677) ExportSnapshot fails on kerberized cluster using s3a

2014-09-15 Thread David S. Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang resolved HADOOP-10677.

   Resolution: Fixed
Fix Version/s: 2.6.0

This was fixed as part of HADOOP-10400.

> ExportSnapshot fails on kerberized cluster using s3a
> 
>
> Key: HADOOP-10677
> URL: https://issues.apache.org/jira/browse/HADOOP-10677
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Affects Versions: 2.4.0
>Reporter: David S. Wang
>Assignee: David S. Wang
> Fix For: 2.6.0
>
> Attachments: HADOOP-10677-1.patch
>
>
> When using HBase ExportSnapshot on a kerberized cluster, exporting to s3a 
> using HADOOP-10400, we see the following problem:
> Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: 
> patch283two
>   at 
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
> The problem seems to be that the patch in HADOOP-10400 does not have 
> getCanonicalServiceName().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

2014-09-15 Thread Allen Wittenauer 

On Sep 15, 2014, at 11:17 AM, Colin McCabe  wrote:

> On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer  wrote:
>> 
>>It’s now September.  With the passage of time, I have a lot of doubts 
>> about this plan and where that trajectory takes us.
>> 
>> * The list of changes that are already in branch-2 scare the crap out of any 
>> risk adverse person (Hello to my fellow operations people!). Not only are 
>> the number of changes extremely high, but in addition there are a lot of 
>> major, blockbuster features in what is supposed to be a minor release.  
>> Combined with the fact that we’ve had to do some micro releases, it seems to 
>> hint that branch-2 is getting less stable over time.
> 
> I don't see what is so scary about 2.6, can you be more concrete?  It
> seems like a pretty normal release to me and most of the new features
> are optional.
> 
> I also don't see why you think that "branch-2 is getting less stable
> over time."  Actually, I think that branch-2 has gotten more stable
> over time as people have finally gotten around to upgrading from 1.x
> or earlier, and contributed their efforts to addressing regressions in
> branch-2.
> 

Please re-read what I wrote above.


>> *  One of the plans talked about was rolling a 2.7 release that drops JDK6 
>> and makes JDK7 the standard.  If 2.7 comes after 2.6 in October, date wise  
>> makes it somewhere around January 2015.  JDK7 EOL’s in April 2015.  So we’ll 
>> have a viable JDK7 release for exactly 3 months.  Frankly, it is too late 
>> for us to talk about JDK7 and we need to start thinking about JDK8.
>> 
>> * trunk is currently sitting at 3 years old.  There is a lot of stuff that 
>> has been hanging around that really needs to get into people hands so that 
>> we can start stabilizing it for a “real” release.
> 
> We have been pretty careful about minimizing trunk's divergence from
> branch-2.  I can't think of an example of anything in trunk that
> "really needs to get into people's hands"-- did I forget something?

There isn't anything in *any* of the recent branch-2 releases that 
qualifies as "really needs to get into people's hands" either, so I'm not sure 
what your point here is.

>> To me this all says one thing:
>> 
>>Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha 
>> with JDK8 as the minimum.  2.5.1 becomes the base for all sustaining work.  
>> This gives the rest of the community time to move to JDK8 if they haven’t 
>> already.  For downstream vendors, it gives a roadmap for their customers who 
>> will be asking about JDK8 sooner rather than later.  By the time 3.0 
>> stabilizes, we’re probably looking at April, which is perfect timing.
>> 
>>One of the issues I’ve heard mention is that 3.0 doesn’t have 
>> anything “compelling” in it.  Well, dropping 2.6 makes the feature list the 
>> carrot, JDK8 support is obviously the stick.
>> 
>>Thoughts?
> 
> As we've discussed before, supporting JDK8 is very different from
> forcing people to use JDK8.  branch-2 and Hadoop 2.6 most certainly
> should support JDK8, and most certainly NOT force people to use JDK8.

We aren't the ones forcing the JDK8 issue: Oracle is.  Users who want a 
viable, supported JDK will be moving to 8 by April. 

> Cloudera has been using JDK7 internally for a long time, and
> recommending it to customers too.

Thank you for using your vendor hat to support my point:  JDK7 works 
fine with Hadoop 2 now, so those users who can't upgrade to JDK8 for whatever 
reason still have a viable option if they want to use Hadoop.  After all, again 
as you point out above, all of the new bits are optional features and could get 
pushed to a later release.  We can continue to make security and critical bug 
fixes against 2.5.x and start pushing those optional features into a newly 
viable 3.x release.  It also supports our major.minor.micro nomenclature as 
well as puts us in line with a lot of other Apache projects. (as pointed out to 
me today: http://t.co/Ry8Zu6yZkd )

> Some developers are using JDK8 as
> well.  It works fine (although I'm sure there will be bugs and
> workarounds that get reported and fixed as more people migrate).

All of the people that I know that are doing JDK8 are applying patches 
to make it work.  (See HADOOP-11090)

>  I don't see this particular issue as a reason to change the schedule.

Look at this from a long-term perspective.  If we make the move to 
support JDK7 as the base line in the next few months, we're essentially saying 
that we're willing to keep JDK7 working for the next 3-4 years (given our 
normal JVM update process).  This isn't a viable strategy given that Oracle 
(you know, the people that provide the JVM) is about to drop support/provide 
updates in 7 months and JDK9 in early access with an expected ship date in 
2016.  JDK7 will be ancient at that point, even worse that JDK6 feels now.