Re: Scala 2.11 and Akka 2.4.0
Hi Manas, Thanks for the reply. I've done that. The problem lies with Spark + akka 2.4.0 build. Seems the maven shader plugin is altering some class files and breaking the Akka runtime. Seems the Spark build on Scala 2.11 using SBT is broken. I'm getting build errors using sbt due to the issues found in the below thread in July of this year. https://mail-archives.apache.org/mod_mbox/spark-dev/201507.mbox/%3CCA+3qhFSJGmZToGmBU1=ivy7kr6eb7k8t6dpz+ibkstihryw...@mail.gmail.com%3E So I went back to maven and decided to risk building Spark on akka 2.3.11 and force the akka 2.4.0 jars onto the server's classpath. I find this a temporary solution while I cannot have a proper akka 2.4.0 runable build. If anyone has managed to get it working, please let me know. tnks, Rod -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Scala-2-11-and-Akka-2-4-0-tp25535p25618.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Scala 2.11 and Akka 2.4.0
There are steps to build spark using scala 2.11 in the spark docs. the first step is /dev/change-scala-version.sh 2.11 which changes the scala version to 2.11. I have not tried compiling spark with akka 2.4.0. ..Manas -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Scala-2-11-and-Akka-2-4-0-tp25535p25587.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Scala 2.11 and Akka 2.4.0
As a I mentioned on the akka mailing list, in case others are following this thread: the issue isn't with dependencies. It's a bug in the maven-shade-plugin. It breaks classfiles when creating the assembly jar (it seems to do some constant propagation). `sbt assembly` doesn't suffer from this issue, probably because it uses another library for jar merging. iulian On Tue, Dec 1, 2015 at 7:21 PM, Boavida, Rodrigo <rodrigo.boav...@aspect.com > wrote: > HI Jacek, > > Yes I was told that as well but no one gave me release schedules, and I > have the immediate need to have Spark Applications communicating with Akka > clusters based on latest version. I'm aware there is an ongoing effort to > change to the low level netty implementation but AFAIK it's not available > yet. > > Any suggestions are very welcomed. > > Tnks, > Rod > > -Original Message- > From: Jacek Laskowski [mailto:ja...@japila.pl] > Sent: 01 December 2015 18:17 > To: Boavida, Rodrigo <rodrigo.boav...@aspect.com> > Cc: user <user@spark.apache.org> > Subject: Re: Scala 2.11 and Akka 2.4.0 > > On Tue, Dec 1, 2015 at 2:32 PM, RodrigoB <rodrigo.boav...@aspect.com> > wrote: > > > I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0. > > Why? AFAIK Spark's leaving Akka's boat and joins Netty's. > > Jacek > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- -- Iulian Dragos -- Reactive Apps on the JVM www.typesafe.com
Re: Scala 2.11 and Akka 2.4.0
>From the dependency tree, akka 2.4.0 was in effect. Maybe check the classpath of master to see if there is older version of akka. Cheers
Re: Scala 2.11 and Akka 2.4.0
On Tue, Dec 1, 2015 at 2:32 PM, RodrigoBwrote: > I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0. Why? AFAIK Spark's leaving Akka's boat and joins Netty's. Jacek - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
RE: Scala 2.11 and Akka 2.4.0
HI Jacek, Yes I was told that as well but no one gave me release schedules, and I have the immediate need to have Spark Applications communicating with Akka clusters based on latest version. I'm aware there is an ongoing effort to change to the low level netty implementation but AFAIK it's not available yet. Any suggestions are very welcomed. Tnks, Rod -Original Message- From: Jacek Laskowski [mailto:ja...@japila.pl] Sent: 01 December 2015 18:17 To: Boavida, Rodrigo <rodrigo.boav...@aspect.com> Cc: user <user@spark.apache.org> Subject: Re: Scala 2.11 and Akka 2.4.0 On Tue, Dec 1, 2015 at 2:32 PM, RodrigoB <rodrigo.boav...@aspect.com> wrote: > I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0. Why? AFAIK Spark's leaving Akka's boat and joins Netty's. Jacek This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Scala 2.11 and Akka 2.4.0
I don't see 2.4.0 release under: http://mvnrepository.com/artifact/com.typesafe.akka/akka-remote_2.10 Probably that was the cause for the 'Could not find artifact' error. On Tue, Dec 1, 2015 at 7:03 AM, Boavida, Rodrigo <rodrigo.boav...@aspect.com > wrote: > Hi Ted, > > Thanks for getting back to me and for the suggestion. > > Running a 'mvn dependency:tree' I get the following: > > [ERROR] Failed to execute goal on project spark-core_2.11: Could not > resolve dependencies for project > org.apache.spark:spark-core_2.11:jar:1.5.2: The following artifacts could > not be resolved: com.typesafe.akka:akka-remote_2.10:jar:2.4.0, > com.typesafe.akka:akka-slf4j_2.10:jar:2.4.0, > com.typesafe.akka:akka-testkit_2.10:jar:2.4.0: Could not find artifact > com.typesafe.akka:akka-remote_2.10:jar:2.4.0 in central ( > https://repo1.maven.org/maven2) -> [Help 1] > > So it seems somehow it's still pulling some 2.10 dependencies. Do you > think this could be the cause for the observed problem? > > tnks, > Rod > > -Original Message- > From: Ted Yu [mailto:yuzhih...@gmail.com] > Sent: 01 December 2015 14:13 > To: Boavida, Rodrigo <rodrigo.boav...@aspect.com> > Cc: user@spark.apache.org > Subject: Re: Scala 2.11 and Akka 2.4.0 > > Have you run 'mvn dependency:tree' and examined the output ? > > There should be some hint. > > Cheers > > > On Dec 1, 2015, at 5:32 AM, RodrigoB <rodrigo.boav...@aspect.com> wrote: > > > > Hi, > > > > I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0. > > I've changed the main pom.xml files to corresponding akka version and > > am getting the following exception when starting the master on > standalone: > > > > Exception Details: > > Location: > >akka/dispatch/Mailbox.processAllSystemMessages()V @152: getstatic > > Reason: > >Type top (current frame, locals[9]) is not assignable to > > 'akka/dispatch/sysmsg/SystemMessage' (stack map, locals[9]) Current > > Frame: > >bci: @131 > >flags: { } > >locals: { 'akka/dispatch/Mailbox', > > 'java/lang/InterruptedException', > > 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox', > 'java/lang/Throwable', 'java/lang/Throwable' } > >stack: { integer } > > Stackmap Frame: > >bci: @152 > >flags: { } > >locals: { 'akka/dispatch/Mailbox', > > 'java/lang/InterruptedException', > > 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox', > > 'java/lang/Throwable', 'java/lang/Throwable', top, top, > 'akka/dispatch/sysmsg/SystemMessage' } > >stack: { } > > Bytecode: > >0x000: 014c 2ab2 0132 b601 35b6 0139 4db2 013e > >0x010: 2cb6 0142 9900 522a b600 c69a 004b 2c4e > >0x020: b201 3e2c b601 454d 2db9 0148 0100 2ab6 > >0x030: 0052 2db6 014b b801 0999 000e bb00 e759 > >0x040: 1301 4db7 010f 4cb2 013e 2cb6 0150 99ff > >0x050: bf2a b600 c69a ffb8 2ab2 0132 b601 35b6 > >0x060: 0139 4da7 ffaa 2ab6 0052 b600 56b6 0154 > >0x070: b601 5a3a 04a7 0091 3a05 1905 3a06 1906 > >0x080: c100 e799 0015 1906 c000 e73a 0719 074c > >0x090: b200 f63a 08a7 0071 b201 5f19 06b6 0163 > >0x0a0: 3a0a 190a b601 6899 0006 1905 bf19 0ab6 > >0x0b0: 016c c000 df3a 0b2a b600 52b6 0170 b601 > >0x0c0: 76bb 000f 5919 0b2a b600 52b6 017a b601 > >0x0d0: 80b6 0186 2ab6 018a bb01 8c59 b701 8e13 > >0x0e0: 0190 b601 9419 09b6 0194 1301 96b6 0194 > >0x0f0: 190b b601 99b6 0194 b601 9ab7 019d b601 > >0x100: a3b2 00f6 3a08 b201 3e2c b601 4299 0026 > >0x110: 2c3a 09b2 013e 2cb6 0145 4d19 09b9 0148 > >0x120: 0100 1904 2ab6 0052 b601 7a19 09b6 01a7 > >0x130: a7ff d62b c600 09b8 0109 572b bfb1 Exception Handler > > Table: > >bci [290, 307] => handler: 120 > > Stackmap Table: > >append_frame(@13,Object[#231],Object[#177]) > >append_frame(@71,Object[#177]) > >chop_frame(@102,1) > > > > full_frame(@120,{Object[#2],Object[#231],Object[#177],Top,Object[#2],O > > bject[#177]},{Object[#223]}) > > > > > full_frame(@152,{Object[#2],Object[#231],Object[#177],Top,Object[#2],Object[#223],Object[#223],Top,Top,Object[#177]},{}) > >append_frame(@173,Object[#357]) > > > > full_frame(@262,{Object[#2],Object[#231],Object[#177],Top,Object[#2]},{}) > >same_frame(@307) > >same_frame(@317) > > at akka.dispatch.Mailboxes.(Mailboxes.scala:33) > >at akka.actor.ActorSystemImpl.(ActorSystem.scala:63
Re: Scala 2.11 and Akka 2.4.0
Please specify the following in your maven commands: -Dscala-2.11 Cheers
RE: Scala 2.11 and Akka 2.4.0
Thanks that worked! I let you know the results. Tnks, Rod From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: 01 December 2015 15:36 To: Boavida, Rodrigo <rodrigo.boav...@aspect.com> Cc: user@spark.apache.org Subject: Re: Scala 2.11 and Akka 2.4.0 Please specify the following in your maven commands: -Dscala-2.11 Cheers This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments.
RE: Scala 2.11 and Akka 2.4.0
Hi Ted, Thanks for getting back to me and for the suggestion. Running a 'mvn dependency:tree' I get the following: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.5.2: The following artifacts could not be resolved: com.typesafe.akka:akka-remote_2.10:jar:2.4.0, com.typesafe.akka:akka-slf4j_2.10:jar:2.4.0, com.typesafe.akka:akka-testkit_2.10:jar:2.4.0: Could not find artifact com.typesafe.akka:akka-remote_2.10:jar:2.4.0 in central (https://repo1.maven.org/maven2) -> [Help 1] So it seems somehow it's still pulling some 2.10 dependencies. Do you think this could be the cause for the observed problem? tnks, Rod -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: 01 December 2015 14:13 To: Boavida, Rodrigo <rodrigo.boav...@aspect.com> Cc: user@spark.apache.org Subject: Re: Scala 2.11 and Akka 2.4.0 Have you run 'mvn dependency:tree' and examined the output ? There should be some hint. Cheers > On Dec 1, 2015, at 5:32 AM, RodrigoB <rodrigo.boav...@aspect.com> wrote: > > Hi, > > I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0. > I've changed the main pom.xml files to corresponding akka version and > am getting the following exception when starting the master on standalone: > > Exception Details: > Location: >akka/dispatch/Mailbox.processAllSystemMessages()V @152: getstatic > Reason: >Type top (current frame, locals[9]) is not assignable to > 'akka/dispatch/sysmsg/SystemMessage' (stack map, locals[9]) Current > Frame: >bci: @131 >flags: { } >locals: { 'akka/dispatch/Mailbox', > 'java/lang/InterruptedException', > 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox', > 'java/lang/Throwable', 'java/lang/Throwable' } >stack: { integer } > Stackmap Frame: >bci: @152 >flags: { } >locals: { 'akka/dispatch/Mailbox', > 'java/lang/InterruptedException', > 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox', > 'java/lang/Throwable', 'java/lang/Throwable', top, top, > 'akka/dispatch/sysmsg/SystemMessage' } >stack: { } > Bytecode: >0x000: 014c 2ab2 0132 b601 35b6 0139 4db2 013e >0x010: 2cb6 0142 9900 522a b600 c69a 004b 2c4e >0x020: b201 3e2c b601 454d 2db9 0148 0100 2ab6 >0x030: 0052 2db6 014b b801 0999 000e bb00 e759 >0x040: 1301 4db7 010f 4cb2 013e 2cb6 0150 99ff >0x050: bf2a b600 c69a ffb8 2ab2 0132 b601 35b6 >0x060: 0139 4da7 ffaa 2ab6 0052 b600 56b6 0154 >0x070: b601 5a3a 04a7 0091 3a05 1905 3a06 1906 >0x080: c100 e799 0015 1906 c000 e73a 0719 074c >0x090: b200 f63a 08a7 0071 b201 5f19 06b6 0163 >0x0a0: 3a0a 190a b601 6899 0006 1905 bf19 0ab6 >0x0b0: 016c c000 df3a 0b2a b600 52b6 0170 b601 >0x0c0: 76bb 000f 5919 0b2a b600 52b6 017a b601 >0x0d0: 80b6 0186 2ab6 018a bb01 8c59 b701 8e13 >0x0e0: 0190 b601 9419 09b6 0194 1301 96b6 0194 >0x0f0: 190b b601 99b6 0194 b601 9ab7 019d b601 >0x100: a3b2 00f6 3a08 b201 3e2c b601 4299 0026 >0x110: 2c3a 09b2 013e 2cb6 0145 4d19 09b9 0148 >0x120: 0100 1904 2ab6 0052 b601 7a19 09b6 01a7 >0x130: a7ff d62b c600 09b8 0109 572b bfb1 Exception Handler > Table: >bci [290, 307] => handler: 120 > Stackmap Table: >append_frame(@13,Object[#231],Object[#177]) >append_frame(@71,Object[#177]) >chop_frame(@102,1) > > full_frame(@120,{Object[#2],Object[#231],Object[#177],Top,Object[#2],O > bject[#177]},{Object[#223]}) > > full_frame(@152,{Object[#2],Object[#231],Object[#177],Top,Object[#2],Object[#223],Object[#223],Top,Top,Object[#177]},{}) >append_frame(@173,Object[#357]) > > full_frame(@262,{Object[#2],Object[#231],Object[#177],Top,Object[#2]},{}) >same_frame(@307) >same_frame(@317) > at akka.dispatch.Mailboxes.(Mailboxes.scala:33) >at akka.actor.ActorSystemImpl.(ActorSystem.scala:635) >at akka.actor.ActorSystem$.apply(ActorSystem.scala:143) >at akka.actor.ActorSystem$.apply(ActorSystem.scala:120) >at > org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) >at > org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) >at > org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:52) >at > org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920) >at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:166) >at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911) >at > org.apache.spark.util.AkkaUtils$.createActorSy
Re: Scala 2.11 and Akka 2.4.0
Have you run 'mvn dependency:tree' and examined the output ? There should be some hint. Cheers > On Dec 1, 2015, at 5:32 AM, RodrigoBwrote: > > Hi, > > I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0. > I've changed the main pom.xml files to corresponding akka version and am > getting the following exception when starting the master on standalone: > > Exception Details: > Location: >akka/dispatch/Mailbox.processAllSystemMessages()V @152: getstatic > Reason: >Type top (current frame, locals[9]) is not assignable to > 'akka/dispatch/sysmsg/SystemMessage' (stack map, locals[9]) > Current Frame: >bci: @131 >flags: { } >locals: { 'akka/dispatch/Mailbox', 'java/lang/InterruptedException', > 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox', > 'java/lang/Throwable', 'java/lang/Throwable' } >stack: { integer } > Stackmap Frame: >bci: @152 >flags: { } >locals: { 'akka/dispatch/Mailbox', 'java/lang/InterruptedException', > 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox', > 'java/lang/Throwable', 'java/lang/Throwable', top, top, > 'akka/dispatch/sysmsg/SystemMessage' } >stack: { } > Bytecode: >0x000: 014c 2ab2 0132 b601 35b6 0139 4db2 013e >0x010: 2cb6 0142 9900 522a b600 c69a 004b 2c4e >0x020: b201 3e2c b601 454d 2db9 0148 0100 2ab6 >0x030: 0052 2db6 014b b801 0999 000e bb00 e759 >0x040: 1301 4db7 010f 4cb2 013e 2cb6 0150 99ff >0x050: bf2a b600 c69a ffb8 2ab2 0132 b601 35b6 >0x060: 0139 4da7 ffaa 2ab6 0052 b600 56b6 0154 >0x070: b601 5a3a 04a7 0091 3a05 1905 3a06 1906 >0x080: c100 e799 0015 1906 c000 e73a 0719 074c >0x090: b200 f63a 08a7 0071 b201 5f19 06b6 0163 >0x0a0: 3a0a 190a b601 6899 0006 1905 bf19 0ab6 >0x0b0: 016c c000 df3a 0b2a b600 52b6 0170 b601 >0x0c0: 76bb 000f 5919 0b2a b600 52b6 017a b601 >0x0d0: 80b6 0186 2ab6 018a bb01 8c59 b701 8e13 >0x0e0: 0190 b601 9419 09b6 0194 1301 96b6 0194 >0x0f0: 190b b601 99b6 0194 b601 9ab7 019d b601 >0x100: a3b2 00f6 3a08 b201 3e2c b601 4299 0026 >0x110: 2c3a 09b2 013e 2cb6 0145 4d19 09b9 0148 >0x120: 0100 1904 2ab6 0052 b601 7a19 09b6 01a7 >0x130: a7ff d62b c600 09b8 0109 572b bfb1 > Exception Handler Table: >bci [290, 307] => handler: 120 > Stackmap Table: >append_frame(@13,Object[#231],Object[#177]) >append_frame(@71,Object[#177]) >chop_frame(@102,1) > > full_frame(@120,{Object[#2],Object[#231],Object[#177],Top,Object[#2],Object[#177]},{Object[#223]}) > > full_frame(@152,{Object[#2],Object[#231],Object[#177],Top,Object[#2],Object[#223],Object[#223],Top,Top,Object[#177]},{}) >append_frame(@173,Object[#357]) > > full_frame(@262,{Object[#2],Object[#231],Object[#177],Top,Object[#2]},{}) >same_frame(@307) >same_frame(@317) > at akka.dispatch.Mailboxes.(Mailboxes.scala:33) >at akka.actor.ActorSystemImpl.(ActorSystem.scala:635) >at akka.actor.ActorSystem$.apply(ActorSystem.scala:143) >at akka.actor.ActorSystem$.apply(ActorSystem.scala:120) >at > org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) >at > org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) >at > org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:52) >at > org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920) >at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:166) >at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911) >at > org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:55) >at > org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:253) >at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:53) >at > org.apache.spark.deploy.master.Master$.startRpcEnvAndEndpoint(Master.scala:1074) >at org.apache.spark.deploy.master.Master$.main(Master.scala:1058) >at org.apache.spark.deploy.master.Master.main(Master.scala) > > --- > > Has anyone encountered this problem before? Seems to be related with a > version mismatch at some level with the Akka mailbox. I would very much > appreciate any comments. > > tnks, > Rod > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Scala-2-11-and-Akka-2-4-0-tp25535.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: