[ https://issues.apache.org/jira/browse/SPARK-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
jay vyas updated SPARK-5487: ---------------------------- Description: Building spark's custom shaed akka version is tricky. The code is in https://github.com/pwendell/akka/ (branch = 2.2.3-shaded-proto) , however, when attempting to build, I receive some strange errors. I've attempted to fork off of a Dockerfile for {{SBT 0.12.4}}, which I'll attach in a snippet just as an example of what we might want to facilitate building the spark specific akka until SPARK-5293 is completed. {noformat} [info] Compiling 6 Scala sources and 1 Java source to /tmp/akka/akka-multi-node-testkit/target/classes... [warn] Class com.google.protobuf.MessageLite not found - continuing with a stub. [error] error while loading ProtobufDecoder, class file '/root/.ivy2/cache/io.netty/netty/bundles/netty-3.6.6.Final.jar(org/jboss/netty/handler/codec/protobuf/ProtobufDecoder.class)' is broken [error] (class java.lang.NullPointerException/null) [error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testconductor/RemoteConnection.scala:24: org.jboss.netty.handler.codec.protobuf.ProtobufDecoder does not have a constructor [error] val proto = List(new ProtobufEncoder, new ProtobufDecoder(TestConductorProtocol.Wrapper.getDefaultInstance)) [error] ^ [error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testkit/MultiNodeSpec.scala:267: value await is not a member of scala.concurrent.Future[Iterable[akka.remote.testconductor.RoleName]] [error] Note: implicit method awaitHelper is not applicable here because it comes after the application point and it lacks an explicit result type [error] testConductor.getNodes.await.filterNot(_ == myself).isEmpty [error] ^ [error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testkit/MultiNodeSpec.scala:354: value await is not a member of scala.concurrent.Future[akka.actor.Address] [error] Note: implicit method awaitHelper is not applicable here because it comes after the application point and it lacks an explicit result type [error] def node(role: RoleName): ActorPath = RootActorPath(testConductor.getAddressFor(role).await) [error] ^ [warn] one warning found [error] four errors found [info] Updating {file:/tmp/akka/}akka-docs... [info] Done updating. [info] Updating {file:/tmp/akka/}akka-contrib... [info] Done updating. [info] Updating {file:/tmp/akka/}akka-sample-osgi-dining-hakkers-core... [info] Done updating. [info] Compiling 17 Scala sources to /tmp/akka/akka-cluster/target/classes... [error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:59: type mismatch; [error] found : akka.cluster.protobuf.msg.GossipEnvelope [error] required: com.google.protobuf_spark.MessageLite [error] case m: GossipEnvelope ? compress(gossipEnvelopeToProto(m)) [error] ^ [error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:61: type mismatch; [error] found : akka.cluster.protobuf.msg.MetricsGossipEnvelope [error] required: com.google.protobuf_spark.MessageLite [error] case m: MetricsGossipEnvelope ? compress(metricsGossipEnvelopeToProto(m)) [error] ^ [error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:63: type mismatch; [error] found : akka.cluster.protobuf.msg.Welcome [error] required: com.google.protobuf_spark.MessageLite [error] case InternalClusterAction.Welcome(from, gossip) ? compress(msg.Welcome(uniqueAddressToProto(from), gossipToProto(gossip))) [error] ^ [error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:257: type mismatch; [error] found : com.google.protobuf_spark.ByteString [error] required: com.google.protobuf.ByteString [error] msg.NodeMetrics.Number(msg.NodeMetrics.NumberType.Serialized, None, None, Some(ByteString.copyFrom(bos.toByteArray))) [error] ^ [error] four errors found [info] Updating {file:/tmp/akka/}akka-sample-cluster... [info] Done updating. [info] Updating {file:/tmp/akka/}akka-sample-osgi-dining-hakkers-integration... [info] Done updating. [error] (akka-multi-node-testkit/compile:compile) Compilation failed [error] (akka-cluster/compile:compile) Compilation failed [error] Total time: 21 s, completed Jan 29, 2015 5:51:14 PM root@c938961f45f1:/tmp/akka# {noformat} was: Building spark's custom shaed akka version is tricky. The code is in https://github.com/pwendell/akka/ (branch = 2.2.3-shaded-proto) , however, when attempting to build, I receive some strange errors. I've attempted to fork off of a Dockerfile for {{SBT 0.12.4}}, which I'll attach in a snippet just as an example of what we might want to facilitate building the spark specific akka until SPARK-5293 is completed. {noformat} [info] Compiling 6 Scala sources and 1 Java source to /tmp/akka/akka-multi-node-testkit/target/classes... [warn] Class com.google.protobuf.MessageLite not found - continuing with a stub. [error] error while loading ProtobufDecoder, class file '/root/.ivy2/cache/io.netty/netty/bundles/netty-3.6.6.Final.jar(org/jboss/netty/handler/codec/protobuf/ProtobufDecoder.class)' is broken [error] (class java.lang.NullPointerException/null) [error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testconductor/RemoteConnection.scala:24: org.jboss.netty.handler.codec.protobuf.ProtobufDecoder does not have a constructor [error] val proto = List(new ProtobufEncoder, new ProtobufDecoder(TestConductorProtocol.Wrapper.getDefaultInstance)) [error] ^ [error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testkit/MultiNodeSpec.scala:267: value await is not a member of scala.concurrent.Future[Iterable[akka.remote.testconductor.RoleName]] [error] Note: implicit method awaitHelper is not applicable here because it comes after the application point and it lacks an explicit result type [error] testConductor.getNodes.await.filterNot(_ == myself).isEmpty [error] ^ [error] /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testkit/MultiNodeSpec.scala:354: value await is not a member of scala.concurrent.Future[akka.actor.Address] [error] Note: implicit method awaitHelper is not applicable here because it comes after the application point and it lacks an explicit result type [error] def node(role: RoleName): ActorPath = RootActorPath(testConductor.getAddressFor(role).await) [error] ^ [warn] one warning found [error] four errors found [info] Updating {file:/tmp/akka/}akka-docs... [info] Done updating. [info] Updating {file:/tmp/akka/}akka-contrib... [info] Done updating. [info] Updating {file:/tmp/akka/}akka-sample-osgi-dining-hakkers-core... [info] Done updating. [info] Compiling 17 Scala sources to /tmp/akka/akka-cluster/target/classes... [error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:59: type mismatch; [error] found : akka.cluster.protobuf.msg.GossipEnvelope [error] required: com.google.protobuf_spark.MessageLite [error] case m: GossipEnvelope ? compress(gossipEnvelopeToProto(m)) [error] ^ [error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:61: type mismatch; [error] found : akka.cluster.protobuf.msg.MetricsGossipEnvelope [error] required: com.google.protobuf_spark.MessageLite [error] case m: MetricsGossipEnvelope ? compress(metricsGossipEnvelopeToProto(m)) [error] ^ [error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:63: type mismatch; [error] found : akka.cluster.protobuf.msg.Welcome [error] required: com.google.protobuf_spark.MessageLite [error] case InternalClusterAction.Welcome(from, gossip) ? compress(msg.Welcome(uniqueAddressToProto(from), gossipToProto(gossip))) [error] ^ [error] /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:257: type mismatch; [error] found : com.google.protobuf_spark.ByteString [error] required: com.google.protobuf.ByteString [error] msg.NodeMetrics.Number(msg.NodeMetrics.NumberType.Serialized, None, None, Some(ByteString.copyFrom(bos.toByteArray))) [error] ^ [error] four errors found [info] Updating {file:/tmp/akka/}akka-sample-cluster... [info] Done updating. [info] Updating {file:/tmp/akka/}akka-sample-osgi-dining-hakkers-integration... [info] Done updating. [error] (akka-multi-node-testkit/compile:compile) Compilation failed [error] (akka-cluster/compile:compile) Compilation failed [error] Total time: 21 s, completed Jan 29, 2015 5:51:14 PM root@c938961f45f1:/tmp/akka# > Dockerfile to build spark's custom akka. > ---------------------------------------- > > Key: SPARK-5487 > URL: https://issues.apache.org/jira/browse/SPARK-5487 > Project: Spark > Issue Type: Bug > Components: Build > Affects Versions: 1.2.0 > Reporter: jay vyas > > Building spark's custom shaed akka version is tricky. The code is in > https://github.com/pwendell/akka/ (branch = 2.2.3-shaded-proto) , however, > when attempting to build, I receive some strange errors. > I've attempted to fork off of a Dockerfile for {{SBT 0.12.4}}, which I'll > attach in a snippet just as an example of what we might want to facilitate > building the spark specific akka until SPARK-5293 is completed. > {noformat} > [info] Compiling 6 Scala sources and 1 Java source to > /tmp/akka/akka-multi-node-testkit/target/classes... > [warn] Class com.google.protobuf.MessageLite not found - continuing with a > stub. > [error] error while loading ProtobufDecoder, class file > '/root/.ivy2/cache/io.netty/netty/bundles/netty-3.6.6.Final.jar(org/jboss/netty/handler/codec/protobuf/ProtobufDecoder.class)' > is broken > [error] (class java.lang.NullPointerException/null) > [error] > /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testconductor/RemoteConnection.scala:24: > org.jboss.netty.handler.codec.protobuf.ProtobufDecoder does not have a > constructor > [error] val proto = List(new ProtobufEncoder, new > ProtobufDecoder(TestConductorProtocol.Wrapper.getDefaultInstance)) > [error] ^ > [error] > /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testkit/MultiNodeSpec.scala:267: > value await is not a member of > scala.concurrent.Future[Iterable[akka.remote.testconductor.RoleName]] > [error] Note: implicit method awaitHelper is not applicable here because it > comes after the application point and it lacks an explicit result type > [error] testConductor.getNodes.await.filterNot(_ == myself).isEmpty > [error] ^ > [error] > /tmp/akka/akka-multi-node-testkit/src/main/scala/akka/remote/testkit/MultiNodeSpec.scala:354: > value await is not a member of scala.concurrent.Future[akka.actor.Address] > [error] Note: implicit method awaitHelper is not applicable here because it > comes after the application point and it lacks an explicit result type > [error] def node(role: RoleName): ActorPath = > RootActorPath(testConductor.getAddressFor(role).await) > [error] > ^ > [warn] one warning found > [error] four errors found > [info] Updating {file:/tmp/akka/}akka-docs... > [info] Done updating. > [info] Updating {file:/tmp/akka/}akka-contrib... > [info] Done updating. > [info] Updating {file:/tmp/akka/}akka-sample-osgi-dining-hakkers-core... > [info] Done updating. > [info] Compiling 17 Scala sources to /tmp/akka/akka-cluster/target/classes... > [error] > /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:59: > type mismatch; > [error] found : akka.cluster.protobuf.msg.GossipEnvelope > [error] required: com.google.protobuf_spark.MessageLite > [error] case m: GossipEnvelope ? compress(gossipEnvelopeToProto(m)) > [error] ^ > [error] > /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:61: > type mismatch; > [error] found : akka.cluster.protobuf.msg.MetricsGossipEnvelope > [error] required: com.google.protobuf_spark.MessageLite > [error] case m: MetricsGossipEnvelope ? > compress(metricsGossipEnvelopeToProto(m)) > [error] > ^ > [error] > /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:63: > type mismatch; > [error] found : akka.cluster.protobuf.msg.Welcome > [error] required: com.google.protobuf_spark.MessageLite > [error] case InternalClusterAction.Welcome(from, gossip) ? > compress(msg.Welcome(uniqueAddressToProto(from), gossipToProto(gossip))) > [error] > ^ > [error] > /tmp/akka/akka-cluster/src/main/scala/akka/cluster/protobuf/ClusterMessageSerializer.scala:257: > type mismatch; > [error] found : com.google.protobuf_spark.ByteString > [error] required: com.google.protobuf.ByteString > [error] > msg.NodeMetrics.Number(msg.NodeMetrics.NumberType.Serialized, None, None, > Some(ByteString.copyFrom(bos.toByteArray))) > [error] > ^ > [error] four errors found > [info] Updating {file:/tmp/akka/}akka-sample-cluster... > [info] Done updating. > [info] Updating > {file:/tmp/akka/}akka-sample-osgi-dining-hakkers-integration... > [info] Done updating. > [error] (akka-multi-node-testkit/compile:compile) Compilation failed > [error] (akka-cluster/compile:compile) Compilation failed > [error] Total time: 21 s, completed Jan 29, 2015 5:51:14 PM > root@c938961f45f1:/tmp/akka# > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org