Re: Dynamically get value based on Map key in Spark Dataframe

Hi Divya, That's the right way to access a value for a key in a broadcast map. I'm pretty sure tough that you could do the same (or similar) with higher-level broadcast Datasets. Try it out! Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark

Re: Dynamically get value based on Map key in Spark Dataframe

Hi Jacek, Can you please share example how can I access broacasted map val pltStnMapBrdcst = sc.broadcast(keyvalueMap ) val df_replacekeys = df_input.withColumn("map_values", pltStnMapBrdcst.value.get("key" Is the above the right way to access the broadcasted map ? Thanks, Divya On 18

Re: Dynamically get value based on Map key in Spark Dataframe

See broadcast variable. Or (just a thought) do join between DataFrames. Jacek On 18 Jul 2016 9:24 a.m., "Divya Gehlot" wrote: > Hi, > > I have created a map by reading a text file > val keyValueMap = file_read.map(t => t.getString(0) -> >

Dynamically get value based on Map key in Spark Dataframe

Hi, I have created a map by reading a text file val keyValueMap = file_read.map(t => t.getString(0) -> t.getString(4)).collect().toMap Now I have another dataframe where I need to dynamically replace all the keys of Map with values val df_input = reading the file as dataframe val df_replacekeys