Thanks Marcelo, Sounds like cluster mode is more resilient than the client-mode.
Does it also depend on the number of Spark nodes involved in choosing which way to go? Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 22 June 2016 at 21:27, Marcelo Vanzin <van...@cloudera.com> wrote: > Trying to keep the answer short and simple... > > On Wed, Jun 22, 2016 at 1:19 PM, Michael Segel > <msegel_had...@hotmail.com> wrote: > > But this gets to the question… what are the real differences between > client > > and cluster modes? > > What are the pros/cons and use cases where one has advantages over the > > other? > > - client mode requires the process that launched the app remain alive. > Meaning the host where it lives has to stay alive, and it may not be > super-friendly to ssh sessions dying, for example, unless you use > nohup. > > - client mode driver logs are printed to stderr by default. yes you > can change that, but in cluster mode, they're all collected by yarn > without any user intervention. > > - if your edge node (from where the app is launched) isn't really part > of the cluster (e.g., lives in an outside network with firewalls or > higher latency), you may run into issues. > > - in cluster mode, your driver's cpu / memory usage is accounted for > in YARN; this matters if your edge node is part of the cluster (and > could be running yarn containers), since in client mode your driver > will potentially use a lot of memory / cpu. > > - finally, in cluster mode YARN can restart your application without > user interference. this is useful for things that need to stay up > (think a long running streaming job, for example). > > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
