Re: get host from rdd map

1. You can call any api that returns you the hostname in your map function. Here's a simplified example, You would generally use mapPartitions as it will save the overhead of retrieving hostname multiple times 2. 3. import scala.sys.process._ 4. val distinctHosts =

get host from rdd map

in rdd map function, is there a way i can know the list of host names where the map runs ? any code sample would be appreciated ? thx, Weide

Re: get host from rdd map

Can you outline your use case a bit more ? Do you want to know all the hosts which would run the map ? Cheers On Fri, Oct 23, 2015 at 5:16 PM, weoccc wrote: > in rdd map function, is there a way i can know the list of host names > where the map runs ? any code sample would

Re: get host from rdd map

yea, my use cases is that i want to have some external communications where rdd is being run in map. The external communication might be handled separately transparent to spark. What will be the hacky way and nonhacky way to do that ? :) Weide On Fri, Oct 23, 2015 at 5:32 PM, Ted Yu