Looks like there is an issue with Maven (likely just the test itself though). We should look into it.
On Wed, Nov 2, 2016 at 11:32 AM, Dongjoon Hyun <dongj...@apache.org> wrote: > Hi, Sean. > > The same failure blocks me, too. > > - SPARK-18189: Fix serialization issue in KeyValueGroupedDataset *** > FAILED *** > > I used `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive -Phive-thriftserver > -Dsparkr` on CentOS 7 / OpenJDK1.8.0_111. > > Dongjoon. > > On 2016-11-02 10:44 (-0700), Sean Owen <so...@cloudera.com> wrote: > > Sigs, license, etc are OK. There are no Blockers for 2.0.2, though here > are > > the 4 issues still open: > > > > SPARK-14387 Enable Hive-1.x ORC compatibility with > > spark.sql.hive.convertMetastoreOrc > > SPARK-17957 Calling outer join and na.fill(0) and then inner join will > miss > > rows > > SPARK-17981 Incorrectly Set Nullability to False in FilterExec > > SPARK-18160 spark.files & spark.jars should not be passed to driver in > yarn > > mode > > > > Running with Java 8, -Pyarn -Phive -Phive-thriftserver -Phadoop-2.7 on > > Ubuntu 16, I am seeing consistent failures in this test below. I think we > > very recently changed this so it could be legitimate. But does anyone > else > > see something like this? I have seen other failures in this test due to > OOM > > but my MAVEN_OPTS allows 6g of heap, which ought to be plenty. > > > > > > - SPARK-18189: Fix serialization issue in KeyValueGroupedDataset *** > FAILED > > *** > > isContain was true Interpreter output contained 'Exception': > > Welcome to > > ____ __ > > / __/__ ___ _____/ /__ > > _\ \/ _ \/ _ `/ __/ '_/ > > /___/ .__/\_,_/_/ /_/\_\ version 2.0.2 > > /_/ > > > > Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_102) > > Type in expressions to have them evaluated. > > Type :help for more information. > > > > scala> > > scala> keyValueGrouped: > > org.apache.spark.sql.KeyValueGroupedDataset[Int,(Int, Int)] = > > org.apache.spark.sql.KeyValueGroupedDataset@70c30f72 > > > > scala> mapGroups: org.apache.spark.sql.Dataset[(Int, Int)] = [_1: int, > > _2: int] > > > > scala> broadcasted: org.apache.spark.broadcast.Broadcast[Int] = > > Broadcast(0) > > > > scala> > > scala> > > scala> dataset: org.apache.spark.sql.Dataset[Int] = [value: int] > > > > scala> org.apache.spark.SparkException: Job aborted due to stage > failure: > > Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in > > stage 0.0 (TID 0, localhost): > > com.google.common.util.concurrent.ExecutionError: > > java.lang.ClassCircularityError: > > io/netty/util/internal/__matchers__/org/apache/spark/network/protocol/ > MessageMatcher > > at com.google.common.cache.LocalCache$Segment.get( > LocalCache.java:2261) > > at com.google.common.cache.LocalCache.get(LocalCache.java:4000) > > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004) > > at > > com.google.common.cache.LocalCache$LocalLoadingCache. > get(LocalCache.java:4874) > > at > > org.apache.spark.sql.catalyst.expressions.codegen. > CodeGenerator$.compile(CodeGenerator.scala:841) > > at > > org.apache.spark.sql.catalyst.expressions.codegen. > GenerateSafeProjection$.create(GenerateSafeProjection.scala:188) > > at > > org.apache.spark.sql.catalyst.expressions.codegen. > GenerateSafeProjection$.create(GenerateSafeProjection.scala:36) > > at > > org.apache.spark.sql.catalyst.expressions.codegen. > CodeGenerator.generate(CodeGenerator.scala:825) > > at > > org.apache.spark.sql.catalyst.expressions.codegen. > CodeGenerator.generate(CodeGenerator.scala:822) > > at > > org.apache.spark.sql.execution.ObjectOperator$.deserializeRowToObject( > objects.scala:137) > > at > > org.apache.spark.sql.execution.AppendColumnsExec$$ > anonfun$9.apply(objects.scala:251) > > at > > org.apache.spark.sql.execution.AppendColumnsExec$$ > anonfun$9.apply(objects.scala:250) > > at > > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$ > 1$$anonfun$apply$24.apply(RDD.scala:803) > > at > > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$ > 1$$anonfun$apply$24.apply(RDD.scala:803) > > at > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > > at > > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > > at > > org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:79) > > at > > org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:47) > > at org.apache.spark.scheduler.Task.run(Task.scala:86) > > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:274) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.ClassCircularityError: > > io/netty/util/internal/__matchers__/org/apache/spark/network/protocol/ > MessageMatcher > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:348) > > at > > io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate( > JavassistTypeParameterMatcherGenerator.java:62) > > at > > io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate( > JavassistTypeParameterMatcherGenerator.java:54) > > at > > io.netty.util.internal.TypeParameterMatcher.get( > TypeParameterMatcher.java:42) > > at > > io.netty.util.internal.TypeParameterMatcher.find( > TypeParameterMatcher.java:78) > > at > > io.netty.handler.codec.MessageToMessageEncoder.<init> > (MessageToMessageEncoder.java:60) > > at > > org.apache.spark.network.protocol.MessageEncoder.<init> > (MessageEncoder.java:34) > > at > > org.apache.spark.network.TransportContext.<init>( > TransportContext.java:78) > > at > > org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient( > NettyRpcEnv.scala:354) > > at > > org.apache.spark.rpc.netty.NettyRpcEnv.openChannel( > NettyRpcEnv.scala:324) > > at org.apache.spark.repl.ExecutorClassLoader.org > > $apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSpa > rkRPC(ExecutorClassLoader.scala:90) > > at > > org.apache.spark.repl.ExecutorClassLoader$$anonfun$ > 1.apply(ExecutorClassLoader.scala:57) > > at > > org.apache.spark.repl.ExecutorClassLoader$$anonfun$ > 1.apply(ExecutorClassLoader.scala:57) > > at > > org.apache.spark.repl.ExecutorClassLoader.findClassLocally( > ExecutorClassLoader.scala:161) > > at > > org.apache.spark.repl.ExecutorClassLoader.findClass( > ExecutorClassLoader.scala:80) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:348) > > at > > io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate( > JavassistTypeParameterMatcherGenerator.java:62) > > at > > io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate( > JavassistTypeParameterMatcherGenerator.java:54) > > at > > io.netty.util.internal.TypeParameterMatcher.get( > TypeParameterMatcher.java:42) > > at > > io.netty.util.internal.TypeParameterMatcher.find( > TypeParameterMatcher.java:78) > > at > > io.netty.handler.codec.MessageToMessageEncoder.<init> > (MessageToMessageEncoder.java:60) > > at > > org.apache.spark.network.protocol.MessageEncoder.<init> > (MessageEncoder.java:34) > > at > > org.apache.spark.network.TransportContext.<init>( > TransportContext.java:78) > > at > > org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient( > NettyRpcEnv.scala:354) > > at > > org.apache.spark.rpc.netty.NettyRpcEnv.openChannel( > NettyRpcEnv.scala:324) > > at org.apache.spark.repl.ExecutorClassLoader.org > > $apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSpa > rkRPC(ExecutorClassLoader.scala:90) > > at > > org.apache.spark.repl.ExecutorClassLoader$$anonfun$ > 1.apply(ExecutorClassLoader.scala:57) > > at > > org.apache.spark.repl.ExecutorClassLoader$$anonfun$ > 1.apply(ExecutorClassLoader.scala:57) > > at > > org.apache.spark.repl.ExecutorClassLoader.findClassLocally( > ExecutorClassLoader.scala:161) > > at > > org.apache.spark.repl.ExecutorClassLoader.findClass( > ExecutorClassLoader.scala:80) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:411) > > at > > org.apache.spark.util.ParentClassLoader.loadClass( > ParentClassLoader.scala:34) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > at > > org.apache.spark.util.ParentClassLoader.loadClass( > ParentClassLoader.scala:30) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:348) > > at > > org.codehaus.janino.ClassLoaderIClassLoader.findIClass( > ClassLoaderIClassLoader.java:78) > > at org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:254) > > at org.codehaus.janino.UnitCompiler.findTypeByName( > UnitCompiler.java:6893) > > at > > org.codehaus.janino.UnitCompiler.getReferenceType( > UnitCompiler.java:5331) > > at > > org.codehaus.janino.UnitCompiler.getReferenceType( > UnitCompiler.java:5207) > > at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5188) > > at org.codehaus.janino.UnitCompiler.access$12600( > UnitCompiler.java:185) > > at > > org.codehaus.janino.UnitCompiler$16.visitReferenceType( > UnitCompiler.java:5119) > > at org.codehaus.janino.Java$ReferenceType.accept(Java.java:2880) > > at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5159) > > at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5414) > > at org.codehaus.janino.UnitCompiler.access$12400( > UnitCompiler.java:185) > > at > > org.codehaus.janino.UnitCompiler$16.visitArrayType(UnitCompiler. > java:5117) > > at org.codehaus.janino.Java$ArrayType.accept(Java.java:2954) > > at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5159) > > at org.codehaus.janino.UnitCompiler.access$16700( > UnitCompiler.java:185) > > at > > org.codehaus.janino.UnitCompiler$31.getParameterTypes2( > UnitCompiler.java:8533) > > at > > org.codehaus.janino.IClass$IInvocable.getParameterTypes(IClass.java:835) > > at org.codehaus.janino.IClass$IMethod.getDescriptor2(IClass.java:1063) > > at org.codehaus.janino.IClass$IInvocable.getDescriptor( > IClass.java:849) > > at org.codehaus.janino.IClass.getIMethods(IClass.java:211) > > at org.codehaus.janino.IClass.getIMethods(IClass.java:199) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:409) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:393) > > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:185) > > at > > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclara > tion(UnitCompiler.java:347) > > at > > org.codehaus.janino.Java$PackageMemberClassDeclaration. > accept(Java.java:1139) > > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354) > > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:322) > > at > > org.codehaus.janino.SimpleCompiler.compileToClassLoader( > SimpleCompiler.java:383) > > at > > org.codehaus.janino.ClassBodyEvaluator.compileToClass( > ClassBodyEvaluator.java:315) > > at > > org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:233) > > at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:192) > > at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:84) > > at > > org.apache.spark.sql.catalyst.expressions.codegen. > CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$ > CodeGenerator$$doCompile(CodeGenerator.scala:887) > > at > > org.apache.spark.sql.catalyst.expressions.codegen. > CodeGenerator$$anon$1.load(CodeGenerator.scala:950) > > at > > org.apache.spark.sql.catalyst.expressions.codegen. > CodeGenerator$$anon$1.load(CodeGenerator.scala:947) > > at > > com.google.common.cache.LocalCache$LoadingValueReference. > loadFuture(LocalCache.java:3599) > > at > > com.google.common.cache.LocalCache$Segment.loadSync( > LocalCache.java:2379) > > at > > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache. > java:2342) > > at com.google.common.cache.LocalCache$Segment.get( > LocalCache.java:2257) > > ... 26 more > > > > > > > > On Wed, Nov 2, 2016 at 4:52 AM Reynold Xin <r...@databricks.com> wrote: > > > > > Please vote on releasing the following candidate as Apache Spark > version > > > 2.0.2. The vote is open until Fri, Nov 4, 2016 at 22:00 PDT and passes > if a > > > majority of at least 3+1 PMC votes are cast. > > > > > > [ ] +1 Release this package as Apache Spark 2.0.2 > > > [ ] -1 Do not release this package because ... > > > > > > > > > The tag to be voted on is v2.0.2-rc2 > > > (a6abe1ee22141931614bf27a4f371c46d8379e33) > > > > > > This release candidate resolves 84 issues: > > > https://s.apache.org/spark-2.0.2-jira > > > > > > The release files, including signatures, digests, etc. can be found at: > > > http://people.apache.org/~pwendell/spark-releases/spark-2.0.2-rc2-bin/ > > > > > > Release artifacts are signed with the following key: > > > https://people.apache.org/keys/committer/pwendell.asc > > > > > > The staging repository for this release can be found at: > > > https://repository.apache.org/content/repositories/ > orgapachespark-1210/ > > > > > > The documentation corresponding to this release can be found at: > > > http://people.apache.org/~pwendell/spark-releases/spark- > 2.0.2-rc2-docs/ > > > > > > > > > Q: How can I help test this release? > > > A: If you are a Spark user, you can help us test this release by > taking an > > > existing Spark workload and running on this release candidate, then > > > reporting any regressions from 2.0.1. > > > > > > Q: What justifies a -1 vote for this release? > > > A: This is a maintenance release in the 2.0.x series. Bugs already > present > > > in 2.0.1, missing features, or bugs related to new features will not > > > necessarily block this release. > > > > > > Q: What fix version should I use for patches merging into branch-2.0 > from > > > now on? > > > A: Please mark the fix version as 2.0.3, rather than 2.0.2. If a new RC > > > (i.e. RC3) is cut, I will change the fix version of those patches to > 2.0.2. > > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
