How Spark handles dead machines during a job.

Hello, Say, that I'm doing a simple rdd.map followed by collect. Say, also, that one of the executors finish all of its tasks, but there are still other executors running. If the machine that hosted the finished executor gets terminated, does the master still have the results from the finished

Re: Executor shutdown hooks?

ave interruptOnCancel set, then you > can catch the interrupt exception within the Task. > > On Wed, Apr 6, 2016 at 12:24 PM, Sung Hwan Chung <coded...@gmail.com> > wrote: > >> Hi, >> >> I'm looking for ways to add shutdown hooks to executors : i.e

Executor shutdown hooks?

Hi, I'm looking for ways to add shutdown hooks to executors : i.e., when a Job is forcefully terminated before it finishes. The scenario goes likes this : executors are running a long running job within a 'map' function. The user decides to terminate the job, then the mappers should perform some

Re: Contributing to MLlib on GLM

Well, as you said, MLLib already supports GLM in a sense. Except they only support two link functions - identity (linear regression) and logit (logistic regression). It should not be too hard to add other link functions, as all you have to do is add a different gradient function for Poisson/Gamma,