Task not serializable problem in the multi-thread SQL query

2015-02-12 Thread lihu
I try to use the multi-thread to use the Spark SQL query. some sample code just like this: val sqlContext = new SqlContext(sc) val rdd_query = sc.parallelize(data, part) rdd_query.registerTempTable(MyTable) sqlContext.cacheTable(MyTable) val serverPool = Executors.newFixedThreadPool(3) val

Re: Task not serializable problem in the multi-thread SQL query

2015-02-12 Thread Michael Armbrust
It looks to me like perhaps your SparkContext has shut down due to too many failures. I'd look in the logs of your executors for more information. On Thu, Feb 12, 2015 at 2:34 AM, lihu lihu...@gmail.com wrote: I try to use the multi-thread to use the Spark SQL query. some sample code just

Re: Task not serializable problem in the multi-thread SQL query

2015-02-12 Thread lihu
Thanks very much, you're right. I called the sc.stop() before the execute pool shutdown. On Fri, Feb 13, 2015 at 7:04 AM, Michael Armbrust mich...@databricks.com wrote: It looks to me like perhaps your SparkContext has shut down due to too many failures. I'd look in the logs of your