you need to merge reference.conf files and its no longer an issue. see the Build for for spark itself: case "reference.conf" => MergeStrategy.concat
On Tue, Apr 29, 2014 at 3:32 PM, Shivani Rao <raoshiv...@gmail.com> wrote: > Hello folks, > > I was going to post this question to spark user group as well. If you have > any leads on how to solve this issue please let me know: > > I am trying to build a basic spark project (spark depends on akka) and I > am trying to create a fatjar using sbt assembly. The goal is to run the > fatjar via commandline as follows: > java -cp "path to my spark fatjar" mainclassname > > I encountered deduplication errors in the following akka libraries during > sbt assembly > akka-remote_2.10-2.2.3.jar with akka-remote_2.10-2.2.3-shaded-protobuf.jar > akka-actor_2.10-2.2.3.jar with akka-actor_2.10-2.2.3-shaded-protobuf.jar > > I resolved them by using MergeStrategy.first and that helped with a > successful compilation of the sbt assembly command. But for some or the > other configuration parameter in the akka kept throwing up with the > following message > > "Exception in thread "main" com.typesafe.config.ConfigException$Missing: > No configuration setting found for key" > > I then used MergeStrategy.concat for "reference.conf" and I started > getting this repeated error > > Exception in thread "main" com.typesafe.config.ConfigException$Missing: No > configuration setting found for key 'akka.version'. > > I noticed that akka.version is only in the akka-actor jars and not in the > akka-remote. The resulting reference.conf (in my final fat jar) does not > contain akka.version either. So the strategy is not working. > > There are several things I could try > > a) Use the following dependency https://github.com/sbt/sbt-proguard > b) Write a build.scala to handle merging of reference.conf > > https://spark-project.atlassian.net/browse/SPARK-395 > http://letitcrash.com/post/21025950392/howto-sbt-assembly-vs-reference-conf > > c) Create a reference.conf by merging all akka configurations and then > passing it in my java -cp command as shown below > > java -cp <jar-name> -DConfig.file=<config> > > The main issue is that if I run the spark jar as "sbt run" there are no > errors in accessing any of the akka configuration parameters. It is only > when I run it via command line (java -cp <jar-name> classname) that I > encounter the error. > > Which of these is a long term fix to akka issues? For now, I removed the > akka dependencies and that solved the problem, but I know that is not a > long term solution > > Regards, > Shivani > > -- > Software Engineer > Analytics Engineering Team@ Box > Mountain View, CA >
