[
https://issues.apache.org/jira/browse/TOREE-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15540697#comment-15540697
]
Hollin Wilkins edited comment on TOREE-345 at 10/2/16 5:30 PM:
---
Found out why the error happens, slf4j as a dependency to spark-avro causes it
for some reason. When including spark-avro as a spark package via
__TOREE_SPARK_OPTS__ in the kernel.json file, the error goes away because Spark
Packages will automatically evict the slf4j dependency.
Output from including Spark-Avro via Spark Packages
{code}
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.databricks#spark-avro_2.11;3.0.1 in list
found org.slf4j#slf4j-api;1.7.5 in list
found org.apache.avro#avro;1.7.6 in list
found org.codehaus.jackson#jackson-core-asl;1.9.13 in list
found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in list
found com.thoughtworks.paranamer#paranamer;2.3 in list
found org.xerial.snappy#snappy-java;1.0.5 in list
found org.apache.commons#commons-compress;1.4.1 in list
found org.tukaani#xz;1.0 in list
:: resolution report :: resolve 341ms :: artifacts dl 5ms
:: modules in use:
com.databricks#spark-avro_2.11;3.0.1 from list in [default]
com.thoughtworks.paranamer#paranamer;2.3 from list in [default]
org.apache.avro#avro;1.7.6 from list in [default]
org.apache.commons#commons-compress;1.4.1 from list in [default]
org.codehaus.jackson#jackson-core-asl;1.9.13 from list in [default]
org.codehaus.jackson#jackson-mapper-asl;1.9.13 from list in [default]
org.slf4j#slf4j-api;1.7.5 from list in [default]
org.tukaani#xz;1.0 from list in [default]
org.xerial.snappy#snappy-java;1.0.5 from list in [default]
:: evicted modules:
org.slf4j#slf4j-api;1.6.4 by [org.slf4j#slf4j-api;1.7.5] in [default]
{code}
was (Author: hollinwilkins):
Found out why the error happens, slf4j as a dependency to spark-avro causes it
for some reason. When including spark-avro as a spark package via Spark options
in the kernel.json file, the error goes away because Spark Packages will
automatically evict the dlf4j dependency.
Output from including Spark-Avro via Spark Packages
{code}
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.databricks#spark-avro_2.11;3.0.1 in list
found org.slf4j#slf4j-api;1.7.5 in list
found org.apache.avro#avro;1.7.6 in list
found org.codehaus.jackson#jackson-core-asl;1.9.13 in list
found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in list
found com.thoughtworks.paranamer#paranamer;2.3 in list
found org.xerial.snappy#snappy-java;1.0.5 in list
found org.apache.commons#commons-compress;1.4.1 in list
found org.tukaani#xz;1.0 in list
:: resolution report :: resolve 341ms :: artifacts dl 5ms
:: modules in use:
com.databricks#spark-avro_2.11;3.0.1 from list in [default]
com.thoughtworks.paranamer#paranamer;2.3 from list in [default]
org.apache.avro#avro;1.7.6 from list in [default]
org.apache.commons#commons-compress;1.4.1 from list in [default]
org.codehaus.jackson#jackson-core-asl;1.9.13 from list in [default]
org.codehaus.jackson#jackson-mapper-asl;1.9.13 from list in [default]
org.slf4j#slf4j-api;1.7.5 from list in [default]
org.tukaani#xz;1.0 from list in [default]
org.xerial.snappy#snappy-java;1.0.5 from list in [default]
:: evicted modules:
org.slf4j#slf4j-api;1.6.4 by [org.slf4j#slf4j-api;1.7.5] in [default]
{code}
> Spark 2.0.0 Fails when trying to use spark-avro
> ---
>
> Key: TOREE-345
> URL: https://issues.apache.org/jira/browse/TOREE-345
> Project: TOREE
> Issue Type: Bug
> Environment: Spark 2.0.0, Scala 2.11, Hadoop 2.7, Toree Git SHA:
> 7c1bfb6df7130477c558e69bbb518b0af364e06a
>Reporter: Hollin Wilkins
>
> When trying to use the spark-avro project to load Avro files from Jupyter, we
> get errors.
> First:
> {code}
> %AddDeps com.databricks spark-avro_2.11 3.0.1 --transitive --trace
> {code}
> Then try to load an Avro file and show it:
> {code}
> spark.sqlContext.read.format("com.databricks.spark.avro").load("/tmp/test.avro").show()
> {code}
> And we get an error. I will attach the trace as a file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
