to find Difference of locations in Spark Dataframe rows

2022-06-07 Thread Chetan Khatri
Hi Dear Spark Users, It has been many years that I have worked on Spark, Please help me. Thanks much I have different cities and their co-ordinates in DataFrame[Row], I want to find distance in KMs and then show only those records /cities which are 10 KMs far. I have a function created that can

Re: How the data is distributed

2022-06-07 Thread Sid
Thank you for the information. On Tue, 7 Jun 2022, 03:21 Sean Owen, wrote: > Data is not distributed to executors by anything. If you are processing > data with Spark. Spark spawns tasks on executors to read chunks of data > from wherever they are (S3, HDFS, etc). > > > On Mon, Jun 6, 2022 at