Re: Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

if you only use it in the executors sometimes using lazy works On Thu, Nov 29, 2018 at 9:45 AM James Starks wrote: > This is not problem directly caused by Spark, but it's related; thus > asking here. I use spark to read data from parquet and processing some http > call with sttp

Spark and Zookeeper HA failures

I am trying to set up a Spark cluster with multi-master HA. I have 3 spark nodes connecting to a single zookeeper node running on a separate server. When running in this configuration, Over the course of 1-2 hours each node ends its session because it is not receving any messages from the server.

Spark 2.4.0 worker can't find work/app/folderNo directory for logs

I'm running a 10 node standalone cluster and I'm having issues with a stage completing - it keeps hanging somewhere between 196 and 199/200 blocks completed, but never errors and doesn't move forward. If I look at the

Spark 2.4.0 worker can't find work/app/folderNo directory for logs

I'm running a 10 node standalone cluster and I'm having issues with a stage completing - it keeps hanging somewhere between 196 and 199/200 blocks completed, but never errors and doesn't move forward. If I look at the

Re: Job hangs in blocked task in final parquet write stage

If not, try running a coalesce. Your data may have grown and is defaulting to a number of partitions that causing unnecessary overhead On Thu, Nov 29, 2018 at 3:02 AM Conrad Lee wrote: > Thanks, I'll try using 5.17.0. > > For anyone trying to debug this problem in the future: In other jobs that

Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

This is not problem directly caused by Spark, but it's related; thus asking here. I use spark to read data from parquet and processing some http call with sttp (https://github.com/softwaremill/sttp). However, spark throws Caused by: java.io.NotSerializableException:

Re: Job hangs in blocked task in final parquet write stage

Thanks, I'll try using 5.17.0. For anyone trying to debug this problem in the future: In other jobs that hang in the same manner, the thread dump didn't have any blocked threads, so that might be a red herring. On Wed, Nov 28, 2018 at 4:34 PM Christopher Petrino < christopher.petr...@gmail.com>