[beginner][StructuredStreaming] Null pointer exception - possible serialization errors.

I am getting a null pointer exception when trying to implement a connection pooling mechanism in Apache Spark. Any help appreciated. https://stackoverflow.com/questions/50205650/spark-connection-pooling-is-this-the-right-approach -- Sent from:

Re: Advice on multiple streaming job

Hi Susan, Thanks for your response. Will try configuration as suggested. But still i am looking for answer does Spark support running multiple jobs on the same port? On Sun, May 6, 2018, 20:27 Susan X. Huynh wrote: > Hi Dhaval, > > Not sure if you have considered this:

Re: Advice on multiple streaming job

Hi vincent, Thanks for your response. We are using YARN, and CNI may not be possible. Thanks & Regards, Dhaval On Sun, May 6, 2018, 13:02 vincent gromakowski < vincent.gromakow...@gmail.com> wrote: > Use a scheduler that abstract the network away with a CNI for instance or > other mécanismes

Re: Advice on multiple streaming job

Hi Dhaval, Not sure if you have considered this: the port 4040 sounds like a driver UI port. By default it will try up to 4056, but you can increase that number with "spark.port.maxRetries". ( https://spark.apache.org/docs/latest/configuration.html) Try setting it to "32". This would help if the

stage blocked sometimes

Hi Sparks: I had encountered a problem, when I run spark application on yarn, sometimes a stage will be blocked that no task of that stage has running and some tasks of that stage has failed, it seems that the stage is still active and not failed on its own. -- Sent from:

Re: Advice on multiple streaming job

Use a scheduler that abstract the network away with a CNI for instance or other mécanismes (mesos, kubernetes, yarn). The CNI will allow to always bind on the same ports because each container will have its own IP. Some other solution like mesos and marathon can work without CNI , with host IP