It seems to me that the closure cleaner fails to clean up something. The failed test case defines a serializable class inside the test case, and the class doesn't refer to anything in the outer class. Ideally it can be serialized after cleaning up the closure.
This is somehow a very weird way to define a class, so I'm not sure how serious the problem is. On Mon, Aug 6, 2018 at 3:41 AM Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > Makes sense, not sure if closure cleaning is related to the last one for > example or others. The last one is a bit weird, unless I am missing > something about the LegacyAccumulatorWrapper logic. > > Stavros > > On Sun, Aug 5, 2018 at 10:23 PM, Sean Owen <sro...@gmail.com> wrote: > >> Yep that's what I did. There are more failures with different >> resolutions. I'll open a JIRA and PR and ping you, to make sure that the >> changes are all reasonable, and not an artifact of missing something about >> closure cleaning in 2.12. >> >> In the meantime having a 2.12 build up and running for master will just >> help catch these things. >> >> On Sun, Aug 5, 2018 at 2:16 PM Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com> wrote: >> >>> Hi Sean, >>> >>> I run a quick build so the failing tests seem to be: >>> >>> - SPARK-17644: After one stage is aborted for too many failed attempts, >>> subsequent stagesstill behave correctly on fetch failures *** FAILED *** >>> A job with one fetch failure should eventually succeed >>> (DAGSchedulerSuite.scala:2422) >>> >>> >>> - LegacyAccumulatorWrapper with AccumulatorParam that has no >>> equals/hashCode *** FAILED *** >>> java.io.NotSerializableException: >>> org.scalatest.Assertions$AssertionsHelper >>> Serialization stack: >>> - object not serializable (class: >>> org.scalatest.Assertions$AssertionsHelper, value: >>> org.scalatest.Assertions$AssertionsHelper@3bc5fc8f) >>> >>> >>> The last one can be fixed easily if you set class `MyData(val i: Int) >>> extends Serializable `outside of the test suite. For some reason outers >>> (not removed) are capturing >>> the Scalatest stuff in 2.12. >>> >>> Let me know if we see the same failures. >>> >>> Stavros >>> >>> On Sun, Aug 5, 2018 at 5:10 PM, Sean Owen <sro...@gmail.com> wrote: >>> >>>> Shane et al - could we get a test job in Jenkins to test the Scala 2.12 >>>> build? I don't think I have the access or expertise for it, though I could >>>> probably copy and paste a job. I think we just need to clone the, say, >>>> master Maven Hadoop 2.7 job, and add two steps: run >>>> "./dev/change-scala-version.sh 2.12" first, then add "-Pscala-2.12" to the >>>> profiles that are enabled. >>>> >>>> I can already see two test failures for the 2.12 build right now and >>>> will try to fix those, but this should help verify whether the failures are >>>> 'real' and detect them going forward. >>>> >>>> >>>> >>> > > >
