Hi, Sean. The same failure blocks me, too.
- SPARK-18189: Fix serialization issue in KeyValueGroupedDataset *** FAILED *** I used `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive -Phive-thriftserver -Dsparkr` on CentOS 7 / OpenJDK1.8.0_111. Dongjoon. On 2016-11-02 10:44 (-0700), Sean Owen <so...@cloudera.com> wrote: > Sigs, license, etc are OK. There are no Blockers for 2.0.2, though here are > the 4 issues still open: > > SPARK-14387 Enable Hive-1.x ORC compatibility with > spark.sql.hive.convertMetastoreOrc > SPARK-17957 Calling outer join and na.fill(0) and then inner join will miss > rows > SPARK-17981 Incorrectly Set Nullability to False in FilterExec > SPARK-18160 spark.files & spark.jars should not be passed to driver in yarn > mode > > Running with Java 8, -Pyarn -Phive -Phive-thriftserver -Phadoop-2.7 on > Ubuntu 16, I am seeing consistent failures in this test below. I think we > very recently changed this so it could be legitimate. But does anyone else > see something like this? I have seen other failures in this test due to OOM > but my MAVEN_OPTS allows 6g of heap, which ought to be plenty. > > > - SPARK-18189: Fix serialization issue in KeyValueGroupedDataset *** FAILED > *** > isContain was true Interpreter output contained 'Exception': > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 2.0.2 > /_/ > > Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_102) > Type in expressions to have them evaluated. > Type :help for more information. > > scala> > scala> keyValueGrouped: > org.apache.spark.sql.KeyValueGroupedDataset[Int,(Int, Int)] = > org.apache.spark.sql.KeyValueGroupedDataset@70c30f72 > > scala> mapGroups: org.apache.spark.sql.Dataset[(Int, Int)] = [_1: int, > _2: int] > > scala> broadcasted: org.apache.spark.broadcast.Broadcast[Int] = > Broadcast(0) > > scala> > scala> > scala> dataset: org.apache.spark.sql.Dataset[Int] = [value: int] > > scala> org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in > stage 0.0 (TID 0, localhost): > com.google.common.util.concurrent.ExecutionError: > java.lang.ClassCircularityError: > io/netty/util/internal/__matchers__/org/apache/spark/network/protocol/MessageMatcher > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261) > at com.google.common.cache.LocalCache.get(LocalCache.java:4000) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:841) > at > org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:188) > at > org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:36) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:825) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:822) > at > org.apache.spark.sql.execution.ObjectOperator$.deserializeRowToObject(objects.scala:137) > at > org.apache.spark.sql.execution.AppendColumnsExec$$anonfun$9.apply(objects.scala:251) > at > org.apache.spark.sql.execution.AppendColumnsExec$$anonfun$9.apply(objects.scala:250) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassCircularityError: > io/netty/util/internal/__matchers__/org/apache/spark/network/protocol/MessageMatcher > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:62) > at > io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:54) > at > io.netty.util.internal.TypeParameterMatcher.get(TypeParameterMatcher.java:42) > at > io.netty.util.internal.TypeParameterMatcher.find(TypeParameterMatcher.java:78) > at > io.netty.handler.codec.MessageToMessageEncoder.<init>(MessageToMessageEncoder.java:60) > at > org.apache.spark.network.protocol.MessageEncoder.<init>(MessageEncoder.java:34) > at > org.apache.spark.network.TransportContext.<init>(TransportContext.java:78) > at > org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:354) > at > org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324) > at org.apache.spark.repl.ExecutorClassLoader.org > $apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSparkRPC(ExecutorClassLoader.scala:90) > at > org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:57) > at > org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:57) > at > org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:161) > at > org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:80) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:62) > at > io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:54) > at > io.netty.util.internal.TypeParameterMatcher.get(TypeParameterMatcher.java:42) > at > io.netty.util.internal.TypeParameterMatcher.find(TypeParameterMatcher.java:78) > at > io.netty.handler.codec.MessageToMessageEncoder.<init>(MessageToMessageEncoder.java:60) > at > org.apache.spark.network.protocol.MessageEncoder.<init>(MessageEncoder.java:34) > at > org.apache.spark.network.TransportContext.<init>(TransportContext.java:78) > at > org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:354) > at > org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324) > at org.apache.spark.repl.ExecutorClassLoader.org > $apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSparkRPC(ExecutorClassLoader.scala:90) > at > org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:57) > at > org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:57) > at > org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:161) > at > org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:80) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:411) > at > org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:34) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:30) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:78) > at org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:254) > at org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:6893) > at > org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5331) > at > org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5207) > at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5188) > at org.codehaus.janino.UnitCompiler.access$12600(UnitCompiler.java:185) > at > org.codehaus.janino.UnitCompiler$16.visitReferenceType(UnitCompiler.java:5119) > at org.codehaus.janino.Java$ReferenceType.accept(Java.java:2880) > at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5159) > at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5414) > at org.codehaus.janino.UnitCompiler.access$12400(UnitCompiler.java:185) > at > org.codehaus.janino.UnitCompiler$16.visitArrayType(UnitCompiler.java:5117) > at org.codehaus.janino.Java$ArrayType.accept(Java.java:2954) > at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5159) > at org.codehaus.janino.UnitCompiler.access$16700(UnitCompiler.java:185) > at > org.codehaus.janino.UnitCompiler$31.getParameterTypes2(UnitCompiler.java:8533) > at > org.codehaus.janino.IClass$IInvocable.getParameterTypes(IClass.java:835) > at org.codehaus.janino.IClass$IMethod.getDescriptor2(IClass.java:1063) > at org.codehaus.janino.IClass$IInvocable.getDescriptor(IClass.java:849) > at org.codehaus.janino.IClass.getIMethods(IClass.java:211) > at org.codehaus.janino.IClass.getIMethods(IClass.java:199) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:409) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:393) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:185) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:347) > at > org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1139) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354) > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:322) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:383) > at > org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:315) > at > org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:233) > at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:192) > at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:84) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:887) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:950) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:947) > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257) > ... 26 more > > > > On Wed, Nov 2, 2016 at 4:52 AM Reynold Xin <r...@databricks.com> wrote: > > > Please vote on releasing the following candidate as Apache Spark version > > 2.0.2. The vote is open until Fri, Nov 4, 2016 at 22:00 PDT and passes if a > > majority of at least 3+1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Spark 2.0.2 > > [ ] -1 Do not release this package because ... > > > > > > The tag to be voted on is v2.0.2-rc2 > > (a6abe1ee22141931614bf27a4f371c46d8379e33) > > > > This release candidate resolves 84 issues: > > https://s.apache.org/spark-2.0.2-jira > > > > The release files, including signatures, digests, etc. can be found at: > > http://people.apache.org/~pwendell/spark-releases/spark-2.0.2-rc2-bin/ > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/pwendell.asc > > > > The staging repository for this release can be found at: > > https://repository.apache.org/content/repositories/orgapachespark-1210/ > > > > The documentation corresponding to this release can be found at: > > http://people.apache.org/~pwendell/spark-releases/spark-2.0.2-rc2-docs/ > > > > > > Q: How can I help test this release? > > A: If you are a Spark user, you can help us test this release by taking an > > existing Spark workload and running on this release candidate, then > > reporting any regressions from 2.0.1. > > > > Q: What justifies a -1 vote for this release? > > A: This is a maintenance release in the 2.0.x series. Bugs already present > > in 2.0.1, missing features, or bugs related to new features will not > > necessarily block this release. > > > > Q: What fix version should I use for patches merging into branch-2.0 from > > now on? > > A: Please mark the fix version as 2.0.3, rather than 2.0.2. If a new RC > > (i.e. RC3) is cut, I will change the fix version of those patches to 2.0.2. > > > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
