Hi Jacek, Can you please share example how can I access broacasted map val pltStnMapBrdcst = sc.broadcast(keyvalueMap ) val df_replacekeys = df_input.withColumn("map_values", pltStnMapBrdcst.value.get("key"))))
Is the above the right way to access the broadcasted map ? Thanks, Divya On 18 July 2016 at 23:06, Jacek Laskowski <ja...@japila.pl> wrote: > See broadcast variable. > > Or (just a thought) do join between DataFrames. > > Jacek > > On 18 Jul 2016 9:24 a.m., "Divya Gehlot" <divya.htco...@gmail.com> wrote: > >> Hi, >> >> I have created a map by reading a text file >> val keyValueMap = file_read.map(t => t.getString(0) -> >> t.getString(4)).collect().toMap >> >> Now I have another dataframe where I need to dynamically replace all the >> keys of Map with values >> val df_input = reading the file as dataframe >> val df_replacekeys = >> df_input.withColumn("map_values",lit(keyValueMap (col("key")))) >> >> Would really appreciate the help . >> >> >> Thanks, >> Divya >> >> >>