I am getting a null pointer exception when trying to implement a connection
pooling mechanism in Apache Spark. Any help appreciated.
https://stackoverflow.com/questions/50205650/spark-connection-pooling-is-this-the-right-approach
--
Sent from:
Hi Susan,
Thanks for your response.
Will try configuration as suggested.
But still i am looking for answer does Spark support running multiple jobs
on the same port?
On Sun, May 6, 2018, 20:27 Susan X. Huynh wrote:
> Hi Dhaval,
>
> Not sure if you have considered this:
Hi vincent,
Thanks for your response.
We are using YARN, and CNI may not be possible.
Thanks & Regards,
Dhaval
On Sun, May 6, 2018, 13:02 vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:
> Use a scheduler that abstract the network away with a CNI for instance or
> other mécanismes
Hi Dhaval,
Not sure if you have considered this: the port 4040 sounds like a driver UI
port. By default it will try up to 4056, but you can increase that number
with "spark.port.maxRetries". (
https://spark.apache.org/docs/latest/configuration.html) Try setting it to
"32". This would help if the
Hi Sparks:
I had encountered a problem, when I run spark application on yarn,
sometimes a stage will be blocked that no task of that stage has running and
some tasks of that stage has failed, it seems that the stage is still
active and not failed on its own.
--
Sent from:
Use a scheduler that abstract the network away with a CNI for instance or
other mécanismes (mesos, kubernetes, yarn). The CNI will allow to always
bind on the same ports because each container will have its own IP. Some
other solution like mesos and marathon can work without CNI , with host IP