Re: Blobserver dying mid-application

2020-10-02 Thread Till Rohrmann
t; $ sysctl -a | grep net.core.somaxconn > > net.core.somaxconn = 1024 > > > > *// *ah > > > > *From:* Chesnay Schepler > *Sent:* Thursday, October 1, 2020 1:41 PM > *To:* Hailu, Andreas [Engineering] ; Till > Rohrmann > *Cc:* user@flink.apache.org; N

RE: Blobserver dying mid-application

2020-10-01 Thread Hailu, Andreas
From: Chesnay Schepler Sent: Thursday, October 1, 2020 1:41 PM To: Hailu, Andreas [Engineering] ; Till Rohrmann Cc: user@flink.apache.org; Nico Kruber Subject: Re: Blobserver dying mid-application All jobs running in a Flink session cluster talk to the same blob server. The time when tasks are submi

Re: Blobserver dying mid-application

2020-10-01 Thread Chesnay Schepler
** *From:*Chesnay Schepler *Sent:* Thursday, October 1, 2020 5:42 AM *To:* Till Rohrmann ; Hailu, Andreas [Engineering] *Cc:* user@flink.apache.org *Subject:* Re: Blobserver dying mid-application It would also be good to know how many slots you have on each task executor. On 10/1/2020 11:21

Re: Blobserver dying mid-application

2020-10-01 Thread Chesnay Schepler
It would also be good to know how many slots you have on each task executor. On 10/1/2020 11:21 AM, Till Rohrmann wrote: Hi Andreas, do the logs of the JM contain any information? Theoretically, each task submission to a `TaskExecutor` can trigger a new connection to the BlobServer. This

Re: Blobserver dying mid-application

2020-10-01 Thread Till Rohrmann
Hi Andreas, do the logs of the JM contain any information? Theoretically, each task submission to a `TaskExecutor` can trigger a new connection to the BlobServer. This depends a bit on how large your TaskInformation is and whether this information is being offloaded to the BlobServer. What you

Blobserver dying mid-application

2020-09-30 Thread Hailu, Andreas
Hello folks, I'm seeing application failures where our Blobserver is refusing connections mid application: 2020-09-30 13:56:06,227 INFO [flink-akka.actor.default-dispatcher-18] org.apache.flink.runtime.taskexecutor.TaskExecutor- Un-registering task and sending final execution