Wouldn't you need to run Spark on Hadoop in order to use YARN? I
believe that YARN only manages Hadoop nodes, not Spark workers
directly. Besides, what I read was that you would need some extra
plug-ins to be able to get nodes managed dynamically.
Our use case would be like this:
1. A Spark cluster is launched with some fixed number of initial nodes
at the beginning.
2. As work load reaches max capacity (e.g. no more executors), a job
submission is rejected or has to wait in the queue.
3. A new worker node is then instantiated (e.g., a pre-configured
container hosting a worker node is created and started) to take the
extra work load so new jobs can be submitted.
4. Optional: If some worker nodes have been idle for a while, they can
be stopped or removed from the cluster.
I guess an external Spark monitor or manager would be needed to keep an
eye on the work load of the cluster and submission status to be able to
launch/remove new nodes. This shouldn't be difficult to do instead of
dealing with complex frameworks like k8s which isn't really designed for
small scale, on-prem use of Spark and requires dedicated admin resources.
On 10/26/22 3:20 PM, Holden Karau wrote:
So Spark can dynamically scale on YARN, but standalone mode becomes a
bit complicated — where do you envision Spark gets the extra resources
from?
On Wed, Oct 26, 2022 at 12:18 PM Artemis User <arte...@dtechspace.com>
wrote:
Has anyone tried to make a Spark cluster dynamically scalable, i.e.,
adding a new worker node automatically to the cluster when no more
executors are available upon a new job submitted? We need to make
the
whole cluster on-prem and really lightweight, so standalone mode is
preferred and no k8s if possible. Any suggestion? Thanks in
advance!
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
