Re: Dynamically get value based on Map key in Spark Dataframe
Hi Divya, That's the right way to access a value for a key in a broadcast map. I'm pretty sure tough that you could do the same (or similar) with higher-level broadcast Datasets. Try it out! Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Tue, Jul 19, 2016 at 4:31 AM, Divya Gehlotwrote: > Hi Jacek, > > Can you please share example how can I access broacasted map > val pltStnMapBrdcst = sc.broadcast(keyvalueMap ) > val df_replacekeys = > df_input.withColumn("map_values",pltStnMapBrdcst.value.get("key" > > Is the above the right way to access the broadcasted map ? > > > > Thanks, > Divya > > > On 18 July 2016 at 23:06, Jacek Laskowski wrote: >> >> See broadcast variable. >> >> Or (just a thought) do join between DataFrames. >> >> Jacek >> >> >> On 18 Jul 2016 9:24 a.m., "Divya Gehlot" wrote: >>> >>> Hi, >>> >>> I have created a map by reading a text file >>> val keyValueMap = file_read.map(t => t.getString(0) -> >>> t.getString(4)).collect().toMap >>> >>> Now I have another dataframe where I need to dynamically replace all the >>> keys of Map with values >>> val df_input = reading the file as dataframe >>> val df_replacekeys = df_input.withColumn("map_values",lit(keyValueMap >>> (col("key" >>> >>> Would really appreciate the help . >>> >>> >>> Thanks, >>> Divya >>> >>> > - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Dynamically get value based on Map key in Spark Dataframe
Hi Jacek, Can you please share example how can I access broacasted map val pltStnMapBrdcst = sc.broadcast(keyvalueMap ) val df_replacekeys = df_input.withColumn("map_values", pltStnMapBrdcst.value.get("key" Is the above the right way to access the broadcasted map ? Thanks, Divya On 18 July 2016 at 23:06, Jacek Laskowskiwrote: > See broadcast variable. > > Or (just a thought) do join between DataFrames. > > Jacek > > On 18 Jul 2016 9:24 a.m., "Divya Gehlot" wrote: > >> Hi, >> >> I have created a map by reading a text file >> val keyValueMap = file_read.map(t => t.getString(0) -> >> t.getString(4)).collect().toMap >> >> Now I have another dataframe where I need to dynamically replace all the >> keys of Map with values >> val df_input = reading the file as dataframe >> val df_replacekeys = >> df_input.withColumn("map_values",lit(keyValueMap (col("key" >> >> Would really appreciate the help . >> >> >> Thanks, >> Divya >> >> >>
Re: Dynamically get value based on Map key in Spark Dataframe
See broadcast variable. Or (just a thought) do join between DataFrames. Jacek On 18 Jul 2016 9:24 a.m., "Divya Gehlot"wrote: > Hi, > > I have created a map by reading a text file > val keyValueMap = file_read.map(t => t.getString(0) -> > t.getString(4)).collect().toMap > > Now I have another dataframe where I need to dynamically replace all the > keys of Map with values > val df_input = reading the file as dataframe > val df_replacekeys = > df_input.withColumn("map_values",lit(keyValueMap (col("key" > > Would really appreciate the help . > > > Thanks, > Divya > > >
Dynamically get value based on Map key in Spark Dataframe
Hi, I have created a map by reading a text file val keyValueMap = file_read.map(t => t.getString(0) -> t.getString(4)).collect().toMap Now I have another dataframe where I need to dynamically replace all the keys of Map with values val df_input = reading the file as dataframe val df_replacekeys = df_input.withColumn("map_values",lit(keyValueMap (col("key" Would really appreciate the help . Thanks, Divya