Well, recent experience tells me that at least SBT does not behave this way. Given a `test` dependency on scalatest 3.0.0, my tests were still being compiled against 2.2.6, which caused a combination of compile errors and runtime errors (given that apparently 3.0.0 was still present at runtime during the test run.) Actually, it could be that they were being compiled against 3.0.0 but 2.2.6 was present at runtime. I'm not sure which is accurate, but adding an exclusion rule when bringing in Spark in order to exclude both scalatest and scalactic resolved the problem.
This may not be a huge issue in real practice, since people will typically be marking spark in its entirety as `provided,test`. In any case, it doesn't seem like it's intentional that scalatest makes its way into compile scope when using any spark dependency in SBT. It's not clear to me whether this is an issue with SBT or with the spark build, but it's a thing that does happen when using SBT (which I imagine a great many people do when building Spark applications). Jeremy On Sat, Oct 29, 2016 at 2:50 AM, Sean Owen <so...@cloudera.com> wrote: > Declare your scalatest dependency as test scope (which is correct anyway). > That would override it I think as desired? > > > On Fri, Oct 28, 2016, 21:22 Shixiong(Ryan) Zhu <shixi...@databricks.com> > wrote: > >> This is my test pom: >> >> <project> >> <modelVersion>4.0.0</modelVersion> >> <artifactId>foo</artifactId> >> <groupId>bar</groupId> >> <version>1.0</version> >> <dependencies> >> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-core_2.10</artifactId> >> <version>2.0.1</version> >> </dependency> >> </dependencies> >> >> </project> >> >> scalatest is in the compile scope: >> >> [INFO] bar:foo:jar:1.0 >> [INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile >> [INFO] +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile >> [INFO] | +- org.apache.avro:avro-ipc:jar:1.7.7:compile >> [INFO] | | \- org.apache.avro:avro:jar:1.7.7:compile >> [INFO] | +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile >> [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile >> [INFO] | \- org.codehaus.jackson:jackson- >> mapper-asl:jar:1.9.13:compile >> [INFO] +- com.twitter:chill_2.10:jar:0.8.0:compile >> [INFO] | \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile >> [INFO] | +- com.esotericsoftware:minlog:jar:1.3.0:compile >> [INFO] | \- org.objenesis:objenesis:jar:2.1:compile >> [INFO] +- com.twitter:chill-java:jar:0.8.0:compile >> [INFO] +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile >> [INFO] +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile >> [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile >> [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile >> [INFO] | | +- org.apache.commons:commons-math:jar:2.1:compile >> [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile >> [INFO] | | +- commons-io:commons-io:jar:2.1:compile >> [INFO] | | +- commons-lang:commons-lang:jar:2.5:compile >> [INFO] | | +- commons-configuration:commons- >> configuration:jar:1.6:compile >> [INFO] | | | +- commons-collections:commons- >> collections:jar:3.2.1:compile >> [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile >> [INFO] | | | | \- commons-beanutils:commons- >> beanutils:jar:1.7.0:compile >> [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0: >> compile >> [INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile >> [INFO] | | +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile >> [INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile >> [INFO] | | \- org.tukaani:xz:jar:1.0:compile >> [INFO] | +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile >> [INFO] | | \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile >> [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2. >> 0:compile >> [INFO] | | +- org.apache.hadoop:hadoop- >> mapreduce-client-common:jar:2.2.0:compile >> [INFO] | | | +- org.apache.hadoop:hadoop-yarn- >> client:jar:2.2.0:compile >> [INFO] | | | | \- com.google.inject:guice:jar:3.0:compile >> [INFO] | | | | +- javax.inject:javax.inject:jar:1:compile >> [INFO] | | | | \- aopalliance:aopalliance:jar:1.0:compile >> [INFO] | | | \- org.apache.hadoop:hadoop-yarn- >> server-common:jar:2.2.0:compile >> [INFO] | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar: >> 2.2.0:compile >> [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile >> [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2. >> 0:compile >> [INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile >> [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient: >> jar:2.2.0:compile >> [INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile >> [INFO] +- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile >> [INFO] +- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile >> [INFO] | +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile >> [INFO] | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5: >> compile >> [INFO] +- org.apache.spark:spark-network-shuffle_2.10:jar:2.0. >> 1:compile >> [INFO] +- org.apache.spark:spark-unsafe_2.10:jar:2.0.1:compile >> [INFO] +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile >> [INFO] | +- commons-codec:commons-codec:jar:1.3:compile >> [INFO] | \- commons-httpclient:commons-httpclient:jar:3.1:compile >> [INFO] +- org.apache.curator:curator-recipes:jar:2.4.0:compile >> [INFO] | +- org.apache.curator:curator-framework:jar:2.4.0:compile >> [INFO] | | \- org.apache.curator:curator-client:jar:2.4.0:compile >> [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile >> [INFO] | \- com.google.guava:guava:jar:14.0.1:compile >> [INFO] +- javax.servlet:javax.servlet-api:jar:3.1.0:compile >> [INFO] +- org.apache.commons:commons-lang3:jar:3.3.2:compile >> [INFO] +- org.apache.commons:commons-math3:jar:3.4.1:compile >> [INFO] +- com.google.code.findbugs:jsr305:jar:1.3.9:compile >> [INFO] +- org.slf4j:slf4j-api:jar:1.7.16:compile >> [INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.16:compile >> [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile >> [INFO] +- log4j:log4j:jar:1.2.17:compile >> [INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile >> [INFO] +- com.ning:compress-lzf:jar:1.0.3:compile >> [INFO] +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile >> [INFO] +- net.jpountz.lz4:lz4:jar:1.3.0:compile >> [INFO] +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile >> [INFO] +- commons-net:commons-net:jar:2.2:compile >> [INFO] +- org.scala-lang:scala-library:jar:2.10.6:compile >> [INFO] +- org.json4s:json4s-jackson_2.10:jar:3.2.11:compile >> [INFO] | \- org.json4s:json4s-core_2.10:jar:3.2.11:compile >> [INFO] | +- org.json4s:json4s-ast_2.10:jar:3.2.11:compile >> [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.6:compile >> [INFO] | \- org.scala-lang:scalap:jar:2.10.0:compile >> [INFO] | \- org.scala-lang:scala-compiler:jar:2.10.0:compile >> [INFO] +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:compile >> [INFO] | +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile >> [INFO] | +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:compile >> [INFO] | | +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:compile >> [INFO] | | \- org.glassfish.hk2.external: >> aopalliance-repackaged:jar:2.4.0-b34:compile >> [INFO] | +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34: >> compile >> [INFO] | \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:compile >> [INFO] | \- org.javassist:javassist:jar:3.18.1-GA:compile >> [INFO] +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:compile >> [INFO] | +- javax.annotation:javax.annotation-api:jar:1.2:compile >> [INFO] | +- org.glassfish.jersey.bundles. >> repackaged:jersey-guava:jar:2.22.2:compile >> [INFO] | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile >> [INFO] +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:compile >> [INFO] | +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2: >> compile >> [INFO] | \- javax.validation:validation-api:jar:1.1.0.Final:compile >> [INFO] +- org.glassfish.jersey.containers:jersey-container- >> servlet:jar:2.22.2:compile >> [INFO] +- org.glassfish.jersey.containers:jersey-container- >> servlet-core:jar:2.22.2:compile >> [INFO] +- org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1:compile >> [INFO] +- io.netty:netty-all:jar:4.0.29.Final:compile >> [INFO] +- io.netty:netty:jar:3.8.0.Final:compile >> [INFO] +- com.clearspring.analytics:stream:jar:2.7.0:compile >> [INFO] +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile >> [INFO] +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile >> [INFO] +- io.dropwizard.metrics:metrics-json:jar:3.1.2:compile >> [INFO] +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:compile >> [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5: >> compile >> [INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:compile >> [INFO] +- com.fasterxml.jackson.module:jackson-module-scala_2.10:jar: >> 2.6.5:compile >> [INFO] | +- org.scala-lang:scala-reflect:jar:2.10.6:compile >> [INFO] | \- com.fasterxml.jackson.module: >> jackson-module-paranamer:jar:2.6.5:compile >> [INFO] +- org.apache.ivy:ivy:jar:2.4.0:compile >> [INFO] +- oro:oro:jar:2.0.8:compile >> [INFO] +- net.razorvine:pyrolite:jar:4.9:compile >> [INFO] +- net.sf.py4j:py4j:jar:0.10.3:compile >> [INFO] +- org.apache.spark:spark-tags_2.10:jar:2.0.1:compile >> [INFO] | \- org.scalatest:scalatest_2.10:jar:2.2.6:compile >> [INFO] \- org.spark-project.spark:unused:jar:1.0.0:compile >> >> On Fri, Oct 28, 2016 at 1:04 PM, Sean Owen <so...@cloudera.com> wrote: >> >> Yes, but scalatest doesn't end up in compile scope, says Maven? >> >> ... >> >> [INFO] +- org.apache.spark:spark-tags_2.11:jar:2.1.0-SNAPSHOT:compile >> >> [INFO] | +- (org.scalatest:scalatest_2.11:jar:2.2.6:test - scope >> managed from compile; omitted for duplicate) >> >> [INFO] | \- (org.spark-project.spark:unused:jar:1.0.0:compile - omitted >> for duplicate) >> >> [INFO] +- org.apache.commons:commons-crypto:jar:1.0.0:compile >> >> [INFO] +- org.spark-project.spark:unused:jar:1.0.0:compile >> >> [INFO] +- org.scalatest:scalatest_2.11:jar:2.2.6:test >> >> ... >> >> On Fri, Oct 28, 2016 at 8:52 PM Jeremy Smith <jeremy.sm...@acorns.com> >> wrote: >> >> spark-core depends on spark-launcher (compile) >> spark-launcher depends on spark-tags (compile) >> spark-tags depends on scalatest (compile) >> >> To be honest I'm not all that familiar with the project structure - >> should I just exclude spark-launcher if I'm not using it? >> >> On Fri, Oct 28, 2016 at 12:27 PM, Sean Owen <so...@cloudera.com> wrote: >> >> It's required because the tags module uses it to define annotations for >> tests. I don't see it in compile scope for anything but the tags module, >> which is then in test scope for other modules. What are you seeing that >> makes you say it's in compile scope? >> >> On Fri, Oct 28, 2016 at 8:19 PM Jeremy Smith <jeremy.sm...@acorns.com> >> wrote: >> >> Hey everybody, >> >> Just a heads up that currently Spark 2.0.1 has a compile dependency on >> Scalatest 2.2.6. It comes from spark-core's dependency on spark-launcher, >> which has a transitive dependency on spark-tags, which has a compile >> dependency on Scalatest. >> >> This makes it impossible to use any other version of Scalatest for >> testing your app if you declare a dependency on any Spark 2.0.1 module; >> you'll get a bunch of runtime errors during testing (unless you figure out >> the reason like I did and explicitly exclude Scalatest from the spark >> dependency). >> >> I think that dependency should probably be moved to a test dependency >> instead. >> >> Thanks, >> Jeremy >> >> >> >>
