You can override the schema inference by passing a schema as the second
argument to jsonRDD, however thats not a super elegant solution. We are
considering one option to make this easier here:
https://issues.apache.org/jira/browse/SPARK-4476
On Tue, Nov 18, 2014 at 11:06 PM, Akhil Das
Thank you Michael
I will try it out tomorrow
Daniel
On 19 בנוב׳ 2014, at 21:07, Michael Armbrust mich...@databricks.com wrote:
You can override the schema inference by passing a schema as the second
argument to jsonRDD, however thats not a super elegant solution. We are
considering one
Oh, actually, we do not support MapType provided by the schema given to
jsonRDD at the moment (my bad..). Daniel, you need to wait for the patch of
4476 (I should have one soon).
Thanks,
Yin
On Wed, Nov 19, 2014 at 2:32 PM, Daniel Haviv danielru...@gmail.com wrote:
Thank you Michael
I will
Hi,
I'm loading a json file into a RDD and then save that RDD as parquet.
One of the fields is a map of keys and values but it is being translated and
stored as a struct.
How can I convert the field into a map?
Thanks,
Daniel
Something like this?
val map_rdd = json_rdd.map(json = {
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
val myMap = mapper.readValue[Map[String,String]](json)
myMap
})
Thanks
Best Regards
On Wed, Nov 19, 2014 at
