-user-list.1001560.n3.nabble.com/How-can-I-read-this-avro-file-using-spark-scala-tp19400p21601.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr
this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-can-I-read-this-avro-file-using-spark-scala-tp19400p21601.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
sqlContext.avroFile similarly to jsonFile but this
does not work for me, nor do I see this in the Scala docs.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-can-I-read-this-avro-file-using-spark-scala-tp19400p21601.html
Sent from the Apache Spark User List mailing
Thanks for the feedback, I filed a couple of issues:
https://github.com/databricks/spark-avro/issues
On Fri, Nov 21, 2014 at 5:04 AM, thomas j beanb...@googlemail.com wrote:
I've been able to load a different avro file based on GenericRecord with:
val person =
Thanks for the pointer Michael.
I've downloaded spark 1.2.0 from
https://people.apache.org/~pwendell/spark-1.2.0-snapshot1/ and clone and
built the spark-avro repo you linked to.
When I run it against the example avro file linked to in the documentation
it works. However, when I try to load my
I've been able to load a different avro file based on GenericRecord with:
val person = sqlContext.avroFile(/tmp/person.avro)
When I try to call `first()` on it, I get NotSerializableException
exceptions again:
person.first()
...
14/11/21 12:59:17 ERROR Executor: Exception in task 0.0 in stage
I have also been struggling with reading avro. Very glad to hear that there
is a new avro library coming in Spark 1.2 (which by the way, seems to have
a lot of other very useful improvements).
In the meanwhile, I have been able to piece together several snippets/tips
that I found from various
I've read several posts of people struggling to read avro in spark. The
examples I've tried don't work. When I try this solution (
https://stackoverflow.com/questions/23944615/how-can-i-load-avros-in-spark-using-the-schema-on-board-the-avro-files)
I get errors:
spark
One option (starting with Spark 1.2, which is currently in preview) is to
use the Avro library for Spark SQL. This is very new, but we would love to
get feedback: https://github.com/databricks/spark-avro
On Thu, Nov 20, 2014 at 10:19 AM, al b beanb...@googlemail.com wrote:
I've read several
