You can use the evil "group by key" and use a conventional method to compare against each row with in that iterable. If your similarity function is a n-1 iterable results for n input then you can use a flatmap to do all that stuff on worker side. spark also has cartesian product that might help in your case. Though for 500 M Record it won't be performant.
..Manas -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/DataFrame-Compare-each-row-to-every-other-row-tp25639p25676.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
