Yes. The HIVE UDF and distribute by both supported by Spark SQL.
If you are using Spark 1.4, you can try Hive analytics windows function
(https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics),most
of which are already supported in Spark 1.4, so you don't need the
Did you take a look at the excellent write up by Yin Huai and Michael
Armbrust? It appears that rank is supported in the 1.4.x release.
https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html
Snippet from above article for your convenience:
To answer the first
