Would you mind to provide executor output so that we can check the
reason why executors died?
And you may run EXPLAIN EXTENDED to find out the physical plan of your
query, something like:
|0: jdbc:hive2://localhost:1 explain extended select * from foo;
Here is the physical plan.
Also attaching the executor log from one of the executors. You can see that
memory consumption is slowly rising and then it is reaching around 10.5 GB.
There it is staying for around 5 minutes 06-50-36 to 06-55-00. Then this
executor is getting killed. ExecutorMemory
Seems that Spark SQL can't retrieve table size statistics and doesn't
enable broadcast join in your case. Would you please try `ANALYZE TABLE
table-name` for both tables to generated table statistics information?
Cheng
On 6/10/15 10:26 PM, Sourav Mazumder wrote:
Here is the physical plan.
Is it the large result set return from the Thrift Server? And can you paste the
SQL and physical plan?
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Tuesday, June 9, 2015 12:01 PM
To: Sourav Mazumder
Cc: user
Subject: Re: Spark SQL with Thrift Server is very very slow and finally failing
From log file I noticed that the ExecutorLostFailure happens after the
memory used by Executor becomes more than the Executor memory value.
However, even if I increase the value of Executor Memory the Executor fails
- only that it takes longer time.
I'm wondering that for joining 2 Hive tables,
Which Spark release are you using ?
Can you pastebin the stack trace w.r.t. ExecutorLostFailure ?
Thanks
On Mon, Jun 8, 2015 at 8:52 PM, Sourav Mazumder sourav.mazumde...@gmail.com
wrote:
Hi,
I am trying to run a SQL form a JDBC driver using Spark's Thrift Server.
I'm doing a join
