Thanks guys both those answers are really helpful. Really appreciate this. On 25 Feb 2016 12:05 a.m., "Michał Zieliński" <zielinski.mich...@gmail.com> wrote:
> Hi Anthony, > > Hopefully that's what you wanted :) > > case class Example(id:String, myMap: Map[String,Int]) >> val myDF = sqlContext.createDataFrame( >> Seq( >> Example("a02d1fa5d87dce6a7", Map("Vans" -> 1, "Versace" ->2)) >> ) >> ) >> myDF. >> explode("myMap","tempSeq")((x: Map[String,Int]) => x.toSeq). >> withColumn("brand_key",col("tempSeq")("_1")). >> withColumn("brand_count",col("tempSeq")("_2")). >> drop("tempSeq"). >> show(false) > > > This gives us: > > +-----------------+----------------------------+---------+-----------+ >> |id |myMap |brand_key|brand_count| >> +-----------------+----------------------------+---------+-----------+ >> |a02d1fa5d87dce6a7|Map(Vans -> 1, Versace -> 2)|Vans |1 | >> |a02d1fa5d87dce6a7|Map(Vans -> 1, Versace -> 2)|Versace |2 | >> +-----------------+----------------------------+---------+-----------+ > > > > Cheers, > Michael > > On 25 February 2016 at 00:06, Anthony Brew <atb...@gmail.com> wrote: > >> Hi, >> I have a Dataframe containing a column with a map Map[A,B] with >> multiple values. I want to explode the key,value pairs in the map into a >> new column, actually planing to create 2 new cols. >> >> My plan had been >> >> - explode "input": Map[K,V] to "temp":Iterable[Map[K,V]] >> - new col temp to temp.key >> - new col temp to temp.value >> - drop temp >> >> But I am failing at the first hurdle. >> >> For example my data looks a bit like like.... >> >> scala> test.show() >> +----------------------------+------------------------------------------+ >> | id | brand >> | >> +----------------------------+------------------------------------------+ >> |a02d1fa5d87dce6a7...|Map(Vans -> 1, Versace ->2, ...| >> >> >> but I want to get to >> >> scala> test.show() >> +-----------------------------+------------------------------------------+ >> | id | brand_key | brand_count | >> +-----------------------------+------------------------------------------+ >> | a02d1fa5d87dce6a7...| Vans | 1 | >> | a02d1fa5d87dce6a7...| Versace | 2 | >> >> >> Any suggestions would be appreciated. >> >> Thanks, >> Anthony >> >> >> >