Re: [DISCUSS] Adaptive execution in Spark SQL

Hi, I would like to share some experience when using AE in eBay’s data warehouse. 1. Saving many manual setting and tuning effort. Setting shuffle.partition one by one query is annoy, with AE, we just need set a big number for all queries. 2. Saving memory. With AE, we can start less

Re: Time for 2.3.2?

+1. We are evaluating 2.3.1, please release Spark 2.3.2 ASAP. Thanks, Yucai

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations Jerry!! Well deserved! Thanks, Yucai On 29/08/2017, 12:06, "Cheng, Hao" wrote: Congratulations!! Jerry, you really deserve it. Hao -Original Message- From: Mridul Muralidharan [mailto:mri...@gmail.com] Sent: Tuesday,

RE: Spark Sql on large number of files (~500Megs each) fails after couple of hours

, Yucai From: Yash Sharma [mailto:yash...@gmail.com] Sent: Monday, April 11, 2016 11:51 AM To: Yu, Yucai <yucai...@intel.com> Cc: dev@spark.apache.org Subject: Re: Spark Sql on large number of files (~500Megs each) fails after couple of hours Hi Yucai, Thanks for the info. I have ex

RE: Spark Sql on large number of files (~500Megs each) fails after couple of hours

Hi Yash, How about checking the executor(yarn container) log? Most of time, it shows more details, we are using CDH, the log is at: [yucai@sr483 container_1457699919227_0094_01_14]$ pwd /mnt/DP_disk1/yucai/yarn/logs/application_1457699919227_0094/container_1457699919227_0094_01_14