I did several tests by adding additional console output. The build was always running on H11 and sometimes it was working, while in other cases not.
Inspecting the byte code did not show anything unusual, it looks correct. Maybe it is related to jacoco, as it adds a javaagent which manipulates bytecode during the execution. I will do further tests to see if there is a connection. Thomas On Tue, Jan 13, 2015 at 2:50 AM, sebb <seb...@gmail.com> wrote: > On 13 January 2015 at 01:12, sebb <seb...@gmail.com> wrote: > > On 13 January 2015 at 00:53, Phil Steitz <phil.ste...@gmail.com> wrote: > >> On 1/12/15 5:44 PM, sebb wrote: > >>> On 12 January 2015 at 22:21, Thomas Neidhart < > thomas.neidh...@gmail.com> wrote: > >>>> On 01/12/2015 11:17 PM, Phil Steitz wrote: > >>>>> On 1/12/15 2:30 PM, Thomas Neidhart wrote: > >>>>>> On 01/12/2015 10:26 PM, Thomas Neidhart wrote: > >>>>>>> On 01/12/2015 08:09 PM, Phil Steitz wrote: > >>>>>>>> On 1/12/15 11:37 AM, sebb wrote: > >>>>>>>>> On 12 January 2015 at 18:11, Phil Steitz <phil.ste...@gmail.com> > wrote: > >>>>>>>>>> On 1/12/15 10:50 AM, sebb wrote: > >>>>>>>>>>> On 11 January 2015 at 22:10, Phil Steitz < > phil.ste...@gmail.com> wrote: > >>>>>>>>>>>> On 1/11/15 11:19 AM, Phil Steitz wrote: > >>>>>>>>>>>>> On 1/10/15 10:49 PM, Phil Steitz wrote: > >>>>>>>>>>>>>> On 1/9/15 6:09 PM, sebb wrote: > >>>>>>>>>>>>>>> On 10 January 2015 at 01:01, Phil Steitz < > phil.ste...@gmail.com> wrote: > >>>>>>>>>>>>>>>> On 1/9/15 5:32 PM, sebb wrote: > >>>>>>>>>>>>>>>>> On 9 January 2015 at 23:48, sebb <seb...@gmail.com> > wrote: > >>>>>>>>>>>>>>>>>> Of the last 6 runs, only 1 had a problem with unit test > failures. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> All the builds ran on ubuntu3, apart from the failure > which ran on H10. > >>>>>>>>>>>>>>>>>> This may have some bearing on the result; I don't yet > know. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I had a quick look at 2 tests that failed: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> SimpleRegressionTest.testPerfect > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> SimpleRegressionTest.testPerfectNegative > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Although the test case has some instance data, these > particular tests > >>>>>>>>>>>>>>>>>> do not use any, so it does not look like a concurrency > issue in the > >>>>>>>>>>>>>>>>>> unit test itself. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> The SimpleRegression class has mutable instance data, > but the test > >>>>>>>>>>>>>>>>>> cases create their own instance. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I don't know anything about the math functions > involved, but it looks > >>>>>>>>>>>>>>>>>> as though Infinity might result from getSignificance() > if > >>>>>>>>>>>>>>>>>> getSlopeStdErr() returns 0, as the latter is used as a > divisor. Or if > >>>>>>>>>>>>>>>>>> the field sumXX is 0 because that is also used as a > divisor. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Maybe the H10 host has different floating point > hardware? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I'll try running some more tests on H10. > >>>>>>>>>>>>>>>>> the build failed again on H10; exactly the same tests > failed as before: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> This test: > >>>>>>>>>>>>>>>>> > https://builds.apache.org/job/Commons%20Math%20H10/1/console > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Previous failure: > >>>>>>>>>>>>>>>>> https://builds.apache.org/job/Commons%20Math/14/console > >>>>>>>>>>>>>>>> This is actually a bug. Thanks, sebb (and Jenkins)! > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Has been here since 1.x. What is going on is that the > data sets > >>>>>>>>>>>>>>>> used in the test cases are set up to be perfect linear > >>>>>>>>>>>>>>>> relationships, which should in fact lead to mean square > error (and > >>>>>>>>>>>>>>>> hence slope standard error) equal to 0. The Jenkins box > must be > >>>>>>>>>>>>>>>> getting exact 0. The funny thing is the test is there to > validate > >>>>>>>>>>>>>>>> correct performance for models like this. Its success > unfortunately > >>>>>>>>>>>>>>>> depends on poor precision. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I will open a JIRA for this. I don't think it is a > release blocker > >>>>>>>>>>>>>>>> for 3.4.1, as I am sure you would get the same thing in > any earlier > >>>>>>>>>>>>>>>> version of [math]. > >>>>>>>>>>>>>>> OK good to know. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'll leave the H10 Jenkins job for now to make it easy to > retest. > >>>>>>>>>>>>>> My first guess here was wrong. The infinities are being > handled > >>>>>>>>>>>>>> correctly for the JDKs I have. Something must be going > awry in the > >>>>>>>>>>>>>> t distribution cumulative probability computation for +INF > on the > >>>>>>>>>>>>>> box that is failing. Is there a way to find out exactly > what JDK > >>>>>>>>>>>>>> and OS version are being used? > >>>>>>>>>>>>> I just committed a test that tests the t distribution > computations > >>>>>>>>>>>>> directly. It seems to have run clean; but the other test > ran clean > >>>>>>>>>>>>> too. Is there any way to force the build to use the host > that fails? > >>>>>>>>>>>> I can't make any sense of what is going on with the Jenkins > builds. > >>>>>>>>>>>> Clean runs and then lots of errors. This one explains the > >>>>>>>>>>>> SimpleRegression "problem" (which is not a problem with that > class > >>>>>>>>>>>> at least) > >>>>>>>>>>>> > >>>>>>>>>>>> > testCumulativeProbablilityExtremes(org.apache.commons.math3.distribution.TDistributionTest) > Time elapsed: 0.001 sec <<< FAILURE! > >>>>>>>>>>>> java.lang.AssertionError: expected:<1.0> but was:<-Infinity> > >>>>>>>>>>>> at org.junit.Assert.fail(Assert.java:88) > >>>>>>>>>>>> at org.junit.Assert.failNotEquals(Assert.java:743) > >>>>>>>>>>>> at org.junit.Assert.assertEquals(Assert.java:494) > >>>>>>>>>>>> at org.junit.Assert.assertEquals(Assert.java:592) > >>>>>>>>>>>> at > org.apache.commons.math3.distribution.TDistributionTest.testCumulativeProbablilityExtremes(TDistributionTest.java:109) > >>>>>>>>>>>> > >>>>>>>>>>>> Earlier runs this ran clean. There is nothing > non-deterministic about this test (or quite a few of the others that > randomly seem to fail). > >>>>>>>>>>>> > >>>>>>>>>>>> I wonder if we have a bad cpu or something somewhere. > >>>>>>>>>>> AFAICS all the failed builds ran on H10. > >>>>>>>>>>> > >>>>>>>>>>> IMO it is consistent; the apparent randomness comes from the > fact the > >>>>>>>>>>> there are several Ubuntu hosts, including H10. > >>>>>>>>>> Am I reading it / looking at the wrong one, or did this one > succeed? > >>>>>>>>>> > >>>>>>>>>> https://builds.apache.org/view/All/job/Commons%20Math%20H10/6/ > >>>>>>>>>> > >>>>>>>>>> That one was right after I added tests confirming that the t > >>>>>>>>>> distribution cum prob handles INFs correctly. > >>>>>>>>> That did run on H10 and did succeed; I'd not noticed that one > before. > >>>>>>>>> > >>>>>>>>> I think it is still true that the failures have only occurred on > H10. > >>>>>>>>> > >>>>>>>>> However, the latest one is failing: > >>>>>>>>> > >>>>>>>>> https://builds.apache.org/job/Commons%20Math/24/console > >>>>>>>>> > >>>>>>>>> This is on H11 - I think that's the first time H11 has been used. > >>>>>>>>> > >>>>>>>>> I suppose it's possible that H10 and H11 have a common failing, > but it > >>>>>>>>> seems less likely. > >>>>>>>>> > >>>>>>>>> I added a bit more debug - showing the value of sumXX - but that > seems > >>>>>>>>> OK on H11. > >>>>>>>>> > >>>>>>>>> I just added a bit more debug. > >>>>>>>> I am pretty sure the SimpleRegressionTest failure is actually > cause > >>>>>>>> by the same thing causing the t-distribution test to fail (the > >>>>>>>> reason I added that one). > >>>>>>>> > >>>>>>>> One that is more straightforward to chase is this one, which fails > >>>>>>>> pretty consistently when "bad things happen" > >>>>>>>> > >>>>>>>> testExpInf(org.apache.commons.math3.complex.ComplexTest) Time > elapsed: 0.001 sec <<< FAILURE! > >>>>>>>> java.lang.AssertionError: expected:<0.0> but was:<Infinity> > >>>>>>>> at org.junit.Assert.fail(Assert.java:88) > >>>>>>>> at org.junit.Assert.failNotEquals(Assert.java:743) > >>>>>>>> at org.junit.Assert.assertEquals(Assert.java:494) > >>>>>>>> at org.junit.Assert.assertEquals(Assert.java:592) > >>>>>>>> at > org.apache.commons.math3.TestUtils.assertSame(TestUtils.java:76) > >>>>>>>> at > org.apache.commons.math3.TestUtils.assertSame(TestUtils.java:84) > >>>>>>>> at > org.apache.commons.math3.complex.ComplexTest.testExpInf(ComplexTest.java:788) > >>>>>>>> > >>>>>>>> I would wager that what is going on here is 0.0 * -INF = INF. > >>>>>>> The output returned by the debug statements added by sebb is: > >>>>>>> > >>>>>>> expReal=Infinity > >>>>>>> cosImag=0.5403023058681398 > >>>>>>> sinImag=0.8414709848078965 > >>>>>>> result=(Infinity, Infinity) > >>>>>>> > >>>>>>> while expReal should be -Infinity. > >>>>>>> > >>>>>>> of course, Math.exp(Infinity) = Infinity. > >>>>>> oh stupid mistake, please forget my last post. > >>>>>> I messed up expReal with the actual real value. > >>>>> But it should be 0, since expReal should be exp(-INF) > >>>> just added a few more debug output to the test and the result is: > >>>> > >>>> real=-Infinity > >>>> -real=2147483647 > >>>> expReal=Infinity > >>>> > >>>> according to FastMath.exp(), with these values, the code path should > be > >>>> as follows: > >>>> > >>>> if (x < 0.0) { > >>>> intVal = (int) -x; > >>>> > >>>> if (intVal > 746) { > >>>> if (hiPrec != null) { > >>>> hiPrec[0] = 0.0; > >>>> hiPrec[1] = 0.0; > >>>> } > >>>> --> return 0.0; > >>>> } > >>>> > >>>> > >>>> but obviously it doesn't do this. I guess we can only inspect the > >>>> generated class files for a potential compiler bug. > >>> That suggests there should be some additional FastMath tests to show > >>> the underlying error. > > > > Actually there is such a test and it also fails. > > > >>> Perhaps compare with the basic Math versions where relevant. > >> > >> What Thomas is pointing out is that the code is not executing > >> correctly, unless we are missing something. This has nothing to do > >> with FastMath vs. Math. > > > > I was just suggesting that it would be worth checking Math to make > > sure it does not behave the same way. > > > >> Did we ever find out what the JDK is? > > > > No, but could probably add debug to show the java system variables. > > I've added a new test with that, plus a test of Math.exp() > > >> Given the sporadic nature of the failures (different tests failing > >> different times), > > > > Are we sure different tests are failing? > > I've not checked that. > > > > The same hosts fail each time. > > Oops, no they don't always fail. > > >> I wonder if there is some kind of storage / > >> filesystem or cpu corruption going on. Do the Jenkins slaves share > >> a common file system or disk array? Are they virtual hosts? > > > > No idea. > > I just think its suspicious that the failures seem to be repeatable. > > If a host fails once, it always fails. > > Oops - not so, they sometimes work... > > > Hardware failures tend to be more random, and given the test > > environment it's unlikely that the same memory will be used each time. > > >> Phil > >>> > >>>> Thomas > >>>> > >>>>> Phil > >>>>>> Thomas > >>>>>> > >>>>>>> Thomas > >>>>>>> > >>>>>>>>> I can perhaps change the H10 job to additionally run on H11. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Phi > >>>>>>>>>>>> Phil > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> Phil > >>>>>>>>>>>>>> Phil > >>>>>>>>>>>>>>>> Phil > >>>>>>>>>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>>>>>>>> To unsubscribe, e-mail: > dev-unsubscr...@commons.apache.org > >>>>>>>>>>>>>>>>> For additional commands, e-mail: > dev-h...@commons.apache.org > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>>>>>>> To unsubscribe, e-mail: > dev-unsubscr...@commons.apache.org > >>>>>>>>>>>>>>>> For additional commands, e-mail: > dev-h...@commons.apache.org > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>>>>>>>>>>>>> For additional commands, e-mail: > dev-h...@commons.apache.org > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org > >>>>>>>>>>>> > >>>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org > >>>>>>>>>> > >>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > --------------------------------------------------------------------- > >>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org > >>>>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>>>> For additional commands, e-mail: dev-h...@commons.apache.org > >>>>>> > >>>>>> > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>>> For additional commands, e-mail: dev-h...@commons.apache.org > >>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>>> For additional commands, e-mail: dev-h...@commons.apache.org > >>>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>> For additional commands, e-mail: dev-h...@commons.apache.org > >>> > >>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >> For additional commands, e-mail: dev-h...@commons.apache.org > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
