I am confused as to whether avro support was merged into Spark 1.2 or it is
still an independent library.
I see some people writing sqlContext.avroFile similarly to jsonFile but this
does not work for me, nor do I see this in the Scala docs.
--
View this message in context:
Check this link.
https://github.com/databricks/spark-avro
Home page for Spark-avro project.
Thanks,
Vishnu
On Wed, Feb 11, 2015 at 10:19 PM, Todd bit1...@163.com wrote:
Databricks provides a sample code on its website...but i can't find it for
now.
At 2015-02-12 00:43:07, captainfranz
Thanks for the feedback, I filed a couple of issues:
https://github.com/databricks/spark-avro/issues
On Fri, Nov 21, 2014 at 5:04 AM, thomas j beanb...@googlemail.com wrote:
I've been able to load a different avro file based on GenericRecord with:
val person =
Thanks for the pointer Michael.
I've downloaded spark 1.2.0 from
https://people.apache.org/~pwendell/spark-1.2.0-snapshot1/ and clone and
built the spark-avro repo you linked to.
When I run it against the example avro file linked to in the documentation
it works. However, when I try to load my
I've been able to load a different avro file based on GenericRecord with:
val person = sqlContext.avroFile(/tmp/person.avro)
When I try to call `first()` on it, I get NotSerializableException
exceptions again:
person.first()
...
14/11/21 12:59:17 ERROR Executor: Exception in task 0.0 in stage
I have also been struggling with reading avro. Very glad to hear that there
is a new avro library coming in Spark 1.2 (which by the way, seems to have
a lot of other very useful improvements).
In the meanwhile, I have been able to piece together several snippets/tips
that I found from various
One option (starting with Spark 1.2, which is currently in preview) is to
use the Avro library for Spark SQL. This is very new, but we would love to
get feedback: https://github.com/databricks/spark-avro
On Thu, Nov 20, 2014 at 10:19 AM, al b beanb...@googlemail.com wrote:
I've read several