Re: Hive on Spark vs Spark on Hive(HiveContext)

2021-07-01 Thread Mich Talebzadeh
Hi Pralabh, You need to check the latest compatibility between Spark version that can successfully work as Hive execution engine This is my old file alluding to spark-1.3.1 as the execution engine set spark.home=/data6/hduser/spark-1.3.1-bin-hadoop2.6; --set

Re: Hive on Spark vs Spark on Hive(HiveContext)

2021-07-01 Thread Pralabh Kumar
Hi mich Thx for replying.your answer really helps. The comparison was done in 2016. I would like to know the latest comparison with spark 3.0 Also what you are suggesting is to migrate queries to Spark ,which is hivecontxt or hive on spark, which is what Facebook also did . Is that understanding

Re: Hive on Spark vs Spark on Hive(HiveContext)

2021-07-01 Thread Mich Talebzadeh
Hi Prahabh, This question has been asked before :) Few years ago (late 2016), I made a presentation on running Hive Queries on the Spark execution engine for Hortonworks. https://www.slideshare.net/MichTalebzadeh1/query-engines-for-hive-mr-spark-tez-with-llap-considerations The issue you will

Hive on Spark vs Spark on Hive(HiveContext)

2021-07-01 Thread Pralabh Kumar
Hi Dev I am having thousands of legacy hive queries . As a plan to move to Spark , we are planning to migrate Hive queries on Spark . Now there are two approaches 1. One is Hive on Spark , which is similar to changing the execution engine in hive queries like TEZ. 2. Another one is