Re: Switching to Java 7
On 8 December 2014 at 19:58, Colin McCabe wrote: > It would be nice if we could have a separate .m2 directory per test > executor. > > It seems like that would eliminate these race conditions once and for > all, at the cost of storing a few extra jars (proportional to the # of > simultaneous executors) > > best, > Colin > all we should really need to do is to force a unique build ID on every build so that it's not the 3.0.0-SNAPSHOT that is needed. We'd have the problem of repository-cruft-bloat instead. Returning to the build, hadoop-hdfs is going OOM on test runs even on java 7; there is a patch which should fix this https://issues.apache.org/jira/browse/HADOOP-11363 Jenkins is down right now & can't verify —could someone have a look @ this and apply if they are happy. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Switching to Java 7
On Mon, Dec 8, 2014 at 7:46 AM, Steve Loughran wrote: > On 8 December 2014 at 14:58, Ted Yu wrote: > >> Looks like there was still OutOfMemoryError : >> >> >> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/ >> > > Well, I'm going to ignore that for now as it's a java 8 problem, surfacing > this weekend once the builds were actually switched to Java 8. memory size > tuning can continue. > > I have now committed the Java 7+ only patch to branch-2 and up: new code > does not have to worry about java 6 compatibility unless they plan to > backport to Java 2.6 or earlier. Having written some Java 7 code, the <> > constructor for typed classes are a convenience, the multiple-catch entries > more useful, as they eliminate duplicate code in exception handling. > > Getting this patch in has revealed that the Jenkins builds of hadoop are > (a) a bit of a mess and (b) prone to race conditions related to the m2 > repository if >1 project builds simultaneously. The way the nightly builds > are staggered means this doesn't usually surface, but it may show up during > precommit/postcommit builds. It would be nice if we could have a separate .m2 directory per test executor. It seems like that would eliminate these race conditions once and for all, at the cost of storing a few extra jars (proportional to the # of simultaneous executors) best, Colin > > The switch to Java 7 as the underlying JDK appears to be triggering > failures, these are things that the projects themselves are going to have > to look at. > > > This then, is where we are with builds right now. This is not a consequence > of the changes to the POM; this list predates that patch. This is Jenkins > running Hadoop builds and tests with Java 7u55 > > > *Working: * > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-branch2/ > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk/ > > *failing tests* > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-2-Build/ > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk/ > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/ > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/ > > *failing tests on Java 8 (may include OOM)* > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-common-trunk-Java8/ > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/ > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Java8/ > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/ > > > *failing with maven internal dependency problems* > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/ > > > *failing even though it appears to work in the logs* > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-trunk/ > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
Re: Switching to Java 7
On 8 December 2014 at 14:58, Ted Yu wrote: > Looks like there was still OutOfMemoryError : > > > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/ > > FYI > we need to set the surefire memory usage explicitly; it doesn't pick up the maven JVM values itself. I've created a JIRA for this. https://issues.apache.org/jira/browse/HADOOP-11363 I'm not going to do this myself as I've had enough of Hadoop POMs and jenkins build configs for the rest of 2014. I'll review patches though -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Switching to Java 7
Ps. I made app.py and stuff...
On 8 Dec 2014 02:59, "Ted Yu" wrote:
> Looking at the test failures of
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/ which uses jdk 1.7:
>
> e.g.
>
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameFileAndDeleteSnapshot/
>
> java.lang.OutOfMemoryError: Java heap space
> at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:120)
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:68)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> at
> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
> at
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
>
>
> Should more heap be given to the tests ?
>
>
> Cheers
>
>
> On Sun, Dec 7, 2014 at 2:09 PM, Steve Loughran
> wrote:
>
> > The latest migration status:
> >
> > if the jenkins builds are happy then the patch will go in -I do that
> > monday morning 10:00 UTC
> >
> > https://builds.apache.org/view/H-L/view/Hadoop/
> >
> > Getting jenkins to work has been "surprisingly difficult"...it turns out
> > that those builds which we thought were java7 or java8 weren't, as
> setting
> > export JAVA_HOME=${TOOLS_HOME}/java/latest
> >
> > meant that they picked up a java 6 machine
> >
> > Now the trunk precommit/postcommit and scheduled branches should have
> > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.7.0_55
> >
> > the Java 8 builds have more changes
> >
> > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.8.0
> > export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> > and -Dmaven.javadoc.skip=true on the mvn builds
> >
> > without these javadocs fails and test runs OOM.
> >
> > We need to have something resembling the nightly build env setup again,
> > git/Svn stored file with something for java8 alongside the normal env
> vars.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
Re: Switching to Java 7
Re g cloud, as far as I can tell what they want is a writeup for existing
services. So we get healthstat running and then write up what we can do in
terms of data flow managment and visualisation then that should cut it.
It would be nice to put the calculator on there as software as a service,
but it needs much more polish
On 8 Dec 2014 02:59, "Ted Yu" wrote:
> Looking at the test failures of
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/ which uses jdk 1.7:
>
> e.g.
>
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameFileAndDeleteSnapshot/
>
> java.lang.OutOfMemoryError: Java heap space
> at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:120)
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:68)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> at
> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
> at
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
>
>
> Should more heap be given to the tests ?
>
>
> Cheers
>
>
> On Sun, Dec 7, 2014 at 2:09 PM, Steve Loughran
> wrote:
>
> > The latest migration status:
> >
> > if the jenkins builds are happy then the patch will go in -I do that
> > monday morning 10:00 UTC
> >
> > https://builds.apache.org/view/H-L/view/Hadoop/
> >
> > Getting jenkins to work has been "surprisingly difficult"...it turns out
> > that those builds which we thought were java7 or java8 weren't, as
> setting
> > export JAVA_HOME=${TOOLS_HOME}/java/latest
> >
> > meant that they picked up a java 6 machine
> >
> > Now the trunk precommit/postcommit and scheduled branches should have
> > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.7.0_55
> >
> > the Java 8 builds have more changes
> >
> > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.8.0
> > export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> > and -Dmaven.javadoc.skip=true on the mvn builds
> >
> > without these javadocs fails and test runs OOM.
> >
> > We need to have something resembling the nightly build env setup again,
> > git/Svn stored file with something for java8 alongside the normal env
> vars.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
Re: Switching to Java 7
On 8 December 2014 at 14:58, Ted Yu wrote: > Looks like there was still OutOfMemoryError : > > > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/ > Well, I'm going to ignore that for now as it's a java 8 problem, surfacing this weekend once the builds were actually switched to Java 8. memory size tuning can continue. I have now committed the Java 7+ only patch to branch-2 and up: new code does not have to worry about java 6 compatibility unless they plan to backport to Java 2.6 or earlier. Having written some Java 7 code, the <> constructor for typed classes are a convenience, the multiple-catch entries more useful, as they eliminate duplicate code in exception handling. Getting this patch in has revealed that the Jenkins builds of hadoop are (a) a bit of a mess and (b) prone to race conditions related to the m2 repository if >1 project builds simultaneously. The way the nightly builds are staggered means this doesn't usually surface, but it may show up during precommit/postcommit builds. The switch to Java 7 as the underlying JDK appears to be triggering failures, these are things that the projects themselves are going to have to look at. This then, is where we are with builds right now. This is not a consequence of the changes to the POM; this list predates that patch. This is Jenkins running Hadoop builds and tests with Java 7u55 *Working: * https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-branch2/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk/ *failing tests* https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-2-Build/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk/ https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/ *failing tests on Java 8 (may include OOM)* https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-common-trunk-Java8/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Java8/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/ *failing with maven internal dependency problems* https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/ *failing even though it appears to work in the logs* https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-trunk/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Switching to Java 7
Looks like there was still OutOfMemoryError :
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/
FYI
On Mon, Dec 8, 2014 at 2:42 AM, Steve Loughran
wrote:
> yes, bumped them up to
>
> export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> export ANT_OPTS=$MAVEN_OPTS
>
> also extended test runs times.
>
>
>
> On 8 December 2014 at 00:58, Ted Yu wrote:
>
> > Looking at the test failures of
> > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/ which uses jdk
> 1.7:
> >
> > e.g.
> >
> >
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameFileAndDeleteSnapshot/
> >
> > java.lang.OutOfMemoryError: Java heap space
> > at
> sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:120)
> > at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:68)
> > at
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> > at
> > io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> > at
> io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
> > at
> >
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> > at
> >
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
> >
> >
> > Should more heap be given to the tests ?
> >
> >
> > Cheers
> >
> >
> > On Sun, Dec 7, 2014 at 2:09 PM, Steve Loughran
> > wrote:
> >
> > > The latest migration status:
> > >
> > > if the jenkins builds are happy then the patch will go in -I do that
> > > monday morning 10:00 UTC
> > >
> > > https://builds.apache.org/view/H-L/view/Hadoop/
> > >
> > > Getting jenkins to work has been "surprisingly difficult"...it turns
> out
> > > that those builds which we thought were java7 or java8 weren't, as
> > setting
> > > export JAVA_HOME=${TOOLS_HOME}/java/latest
> > >
> > > meant that they picked up a java 6 machine
> > >
> > > Now the trunk precommit/postcommit and scheduled branches should have
> > > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.7.0_55
> > >
> > > the Java 8 builds have more changes
> > >
> > > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.8.0
> > > export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> > > and -Dmaven.javadoc.skip=true on the mvn builds
> > >
> > > without these javadocs fails and test runs OOM.
> > >
> > > We need to have something resembling the nightly build env setup again,
> > > git/Svn stored file with something for java8 alongside the normal env
> > vars.
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Switching to Java 7
yes, bumped them up to
export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
export ANT_OPTS=$MAVEN_OPTS
also extended test runs times.
On 8 December 2014 at 00:58, Ted Yu wrote:
> Looking at the test failures of
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/ which uses jdk 1.7:
>
> e.g.
>
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameFileAndDeleteSnapshot/
>
> java.lang.OutOfMemoryError: Java heap space
> at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:120)
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:68)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> at
> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
> at
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
>
>
> Should more heap be given to the tests ?
>
>
> Cheers
>
>
> On Sun, Dec 7, 2014 at 2:09 PM, Steve Loughran
> wrote:
>
> > The latest migration status:
> >
> > if the jenkins builds are happy then the patch will go in -I do that
> > monday morning 10:00 UTC
> >
> > https://builds.apache.org/view/H-L/view/Hadoop/
> >
> > Getting jenkins to work has been "surprisingly difficult"...it turns out
> > that those builds which we thought were java7 or java8 weren't, as
> setting
> > export JAVA_HOME=${TOOLS_HOME}/java/latest
> >
> > meant that they picked up a java 6 machine
> >
> > Now the trunk precommit/postcommit and scheduled branches should have
> > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.7.0_55
> >
> > the Java 8 builds have more changes
> >
> > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.8.0
> > export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> > and -Dmaven.javadoc.skip=true on the mvn builds
> >
> > without these javadocs fails and test runs OOM.
> >
> > We need to have something resembling the nightly build env setup again,
> > git/Svn stored file with something for java8 alongside the normal env
> vars.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Switching to Java 7
Looking at the test failures of
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/ which uses jdk 1.7:
e.g.
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameFileAndDeleteSnapshot/
java.lang.OutOfMemoryError: Java heap space
at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:120)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:68)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
at
io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
at
io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
Should more heap be given to the tests ?
Cheers
On Sun, Dec 7, 2014 at 2:09 PM, Steve Loughran
wrote:
> The latest migration status:
>
> if the jenkins builds are happy then the patch will go in -I do that
> monday morning 10:00 UTC
>
> https://builds.apache.org/view/H-L/view/Hadoop/
>
> Getting jenkins to work has been "surprisingly difficult"...it turns out
> that those builds which we thought were java7 or java8 weren't, as setting
> export JAVA_HOME=${TOOLS_HOME}/java/latest
>
> meant that they picked up a java 6 machine
>
> Now the trunk precommit/postcommit and scheduled branches should have
> export JAVA_HOME=${TOOLS_HOME}/java/jdk1.7.0_55
>
> the Java 8 builds have more changes
>
> export JAVA_HOME=${TOOLS_HOME}/java/jdk1.8.0
> export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> and -Dmaven.javadoc.skip=true on the mvn builds
>
> without these javadocs fails and test runs OOM.
>
> We need to have something resembling the nightly build env setup again,
> git/Svn stored file with something for java8 alongside the normal env vars.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Switching to Java 7
The latest migration status:
if the jenkins builds are happy then the patch will go in -I do that
monday morning 10:00 UTC
https://builds.apache.org/view/H-L/view/Hadoop/
Getting jenkins to work has been "surprisingly difficult"...it turns out
that those builds which we thought were java7 or java8 weren't, as setting
export JAVA_HOME=${TOOLS_HOME}/java/latest
meant that they picked up a java 6 machine
Now the trunk precommit/postcommit and scheduled branches should have
export JAVA_HOME=${TOOLS_HOME}/java/jdk1.7.0_55
the Java 8 builds have more changes
export JAVA_HOME=${TOOLS_HOME}/java/jdk1.8.0
export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
and -Dmaven.javadoc.skip=true on the mvn builds
without these javadocs fails and test runs OOM.
We need to have something resembling the nightly build env setup again,
git/Svn stored file with something for java8 alongside the normal env vars.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Switching to Java 7
yeah, I'm trying to set some of the common ones up first it's a bit confusing making sense and isolating JVM updates with other test failures, especially as some of the failures seem intermittent, and some of the test runs (hadoop-hfds-trunk) don't even collect all the test results —you see a "build fail" message without any failure tracking. expect more "jenkins failed" messages as I'm going to be triggering more test runs than usual (UK time) On 1 December 2014 at 17:56, Haohui Mai wrote: > Hi Steve, > > I think the pre-commit Jenkins are running Java 6, they need to be > switched to Java 7 as well. > > Haohui > > > On Dec 1, 2014, at 5:41 AM, Steve Loughran > wrote: > > > > I'm planning to flip the Javac language & JVM settings to java 7 this > week > > > > https://issues.apache.org/jira/browse/HADOOP-10530 > > > > the latest patch also has a profile that sets the language to java8, for > > the curious; one bit of code will need patching to compile there. > > > > The plan for the change ASF-side is: > > > > 1 -switch jenkins patch/regular commits to java7 > > 2 -apply the HADOOP-10530 patch > > > > locally, anyone who runs Jenkins with Java 6 will have to upgrade/switch > > JVM after (2), and anyone with JAVA_HOME set to a jdk 6 JDK is going to > > have to edit their environment variables. > > > > Is there anything else we need to do before the big Java 7 switch? > > > > -Steve > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Switching to Java 7
Hi Steve, I think the pre-commit Jenkins are running Java 6, they need to be switched to Java 7 as well. Haohui > On Dec 1, 2014, at 5:41 AM, Steve Loughran wrote: > > I'm planning to flip the Javac language & JVM settings to java 7 this week > > https://issues.apache.org/jira/browse/HADOOP-10530 > > the latest patch also has a profile that sets the language to java8, for > the curious; one bit of code will need patching to compile there. > > The plan for the change ASF-side is: > > 1 -switch jenkins patch/regular commits to java7 > 2 -apply the HADOOP-10530 patch > > locally, anyone who runs Jenkins with Java 6 will have to upgrade/switch > JVM after (2), and anyone with JAVA_HOME set to a jdk 6 JDK is going to > have to edit their environment variables. > > Is there anything else we need to do before the big Java 7 switch? > > -Steve > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Switching to Java 7
I'm planning to flip the Javac language & JVM settings to java 7 this week https://issues.apache.org/jira/browse/HADOOP-10530 the latest patch also has a profile that sets the language to java8, for the curious; one bit of code will need patching to compile there. The plan for the change ASF-side is: 1 -switch jenkins patch/regular commits to java7 2 -apply the HADOOP-10530 patch locally, anyone who runs Jenkins with Java 6 will have to upgrade/switch JVM after (2), and anyone with JAVA_HOME set to a jdk 6 JDK is going to have to edit their environment variables. Is there anything else we need to do before the big Java 7 switch? -Steve -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
