Re: Spark-SQL - Query Hanging, How To Troubleshoot

Hi Patrick, There is not anything wrong with Hive On-premise it is the best data warehouse there is Hive handles both ORC and Parquet formal well. They are both columnar implementations of relational model. What you are seeing is the Spark API to Hive which prefers Parquet. I found out a few

Re: Spark-SQL - Query Hanging, How To Troubleshoot

Thanks for the reply Stephen and Mich. Stephen, you're right, it feels like Spark is waiting for something, but I'm not sure what. I'm the only user on the cluster and there are plenty of resources (+60 cores, +250GB RAM). I even tried restarting Hadoop, Spark and the host servers to make sure

Re: unsubscribe

To unsubscribe e-mail: user-unsubscr...@spark.apache.org Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at

unsubscribe

unsubscribe

Re: Extracting Logical Plan

Hello Winston, I looked into the suggested code snippet. But I am getting the following error ``` value listenerManager is not a member of org.apache.spark.sql.SparkSession ``` Although I can see it is available in the API.

Re: Spark-SQL - Query Hanging, How To Troubleshoot

Steve may have a valid point. You raised an issue with concurrent writes before, if I recall correctly. Since this limitation may be due to Hive metastore. By default Spark uses Apache Derby for its database persistence. *However it is limited to only one Spark session at any time for the purposes