Re: spark-avro 2.0.1 generates strange schema (spark-avro 1.0.0 is fine)
Here you go: https://github.com/databricks/spark-avro/issues/92 Thanks. On Wed, Oct 14, 2015 at 4:41 PM, Josh Rosen wrote: > Can you report this as an issue at > https://github.com/databricks/spark-avro/issues so that it's easier to > track? Thanks! > > On Wed, Oct 14, 2015 at 1:38 PM, Alex Nastetsky < > alex.nastet...@vervemobile.com> wrote: > >> I save my dataframe to avro with spark-avro 1.0.0 and it looks like this >> (using avro-tools tojson): >> >> {"field1":"value1","field2":976200} >> {"field1":"value2","field2":976200} >> {"field1":"value3","field2":614100} >> >> But when I use spark-avro 2.0.1, it looks like this: >> >> {"field1":{"string":"value1"},"field2":{"long":976200}} >> {"field1":{"string":"value2"},"field2":{"long":976200}} >> {"field1":{"string":"value3"},"field2":{"long":614100}} >> >> At this point I'd be happy to use spark-avro 1.0.0, except that it >> doesn't seem to support specifying a compression codec (I want deflate). >> > >
Re: spark-avro 2.0.1 generates strange schema (spark-avro 1.0.0 is fine)
Can you report this as an issue at https://github.com/databricks/spark-avro/issues so that it's easier to track? Thanks! On Wed, Oct 14, 2015 at 1:38 PM, Alex Nastetsky < alex.nastet...@vervemobile.com> wrote: > I save my dataframe to avro with spark-avro 1.0.0 and it looks like this > (using avro-tools tojson): > > {"field1":"value1","field2":976200} > {"field1":"value2","field2":976200} > {"field1":"value3","field2":614100} > > But when I use spark-avro 2.0.1, it looks like this: > > {"field1":{"string":"value1"},"field2":{"long":976200}} > {"field1":{"string":"value2"},"field2":{"long":976200}} > {"field1":{"string":"value3"},"field2":{"long":614100}} > > At this point I'd be happy to use spark-avro 1.0.0, except that it doesn't > seem to support specifying a compression codec (I want deflate). >
spark-avro 2.0.1 generates strange schema (spark-avro 1.0.0 is fine)
I save my dataframe to avro with spark-avro 1.0.0 and it looks like this (using avro-tools tojson): {"field1":"value1","field2":976200} {"field1":"value2","field2":976200} {"field1":"value3","field2":614100} But when I use spark-avro 2.0.1, it looks like this: {"field1":{"string":"value1"},"field2":{"long":976200}} {"field1":{"string":"value2"},"field2":{"long":976200}} {"field1":{"string":"value3"},"field2":{"long":614100}} At this point I'd be happy to use spark-avro 1.0.0, except that it doesn't seem to support specifying a compression codec (I want deflate).