ProcessJobFactory parent process

2015-05-28 Thread Lukas Steiblys
Hello, I’m running Samza tasks using ProcessJobFactory and after I start the job, the initial process spawns a new process that is the actual process where the code is run. The problem is that the parent process stays active even after the job is started and that messes with the way I deploy Sa

Re: ProcessJobFactory parent process

2015-05-28 Thread Yan Fang
Hi Lukas, The parent process is used to manage the lifecycle of the actual process. I am curious how much memory the parent process takes? Thanks, Fang, Yan yanfang...@gmail.com On Thu, May 28, 2015 at 2:30 PM, Lukas Steiblys wrote: > Hello, > > I’m running Samza tasks using ProcessJobFactory

Re: ProcessJobFactory parent process

2015-05-29 Thread Lukas Steiblys
Hi Yan, The memory usage is not very high, but I'm trying to cut the usage any way I can. The bigger problem is when the job crashes and the parent process stays active preventing an auto restart by the Docker supervisor. Lukas On Thursday, May 28, 2015, Yan Fang wrote: > Hi Lukas, > > The pa

Re: ProcessJobFactory parent process

2015-05-29 Thread Yan Fang
Hi Lukas, This sounds like a bug to me. Can you file a JIRA for this? We will have a look at it. Thanks, Fang, Yan yanfang...@gmail.com On Fri, May 29, 2015 at 8:39 AM, Lukas Steiblys wrote: > Hi Yan, > > The memory usage is not very high, but I'm trying to cut the usage any way > I can. > >

Re: ProcessJobFactory parent process

2015-05-29 Thread Yi Pan
Hi, Lukas, I assume that when you say "the job crashes", you were referring to the child process running the container, not the parent process? If yes, we were actually talking about adding container health-check/failure-detection in the JobCoordinator. SAMZA-680 would be the good place to start t

Re: ProcessJobFactory parent process

2015-05-29 Thread Lukas Steiblys
apache.org Subject: Re: ProcessJobFactory parent process Hi, Lukas, I assume that when you say "the job crashes", you were referring to the child process running the container, not the parent process? If yes, we were actually talking about adding container health-check/failure-detection i

Re: ProcessJobFactory parent process

2015-05-29 Thread Yi Pan
From: Yi Pan > Sent: Friday, May 29, 2015 12:47 PM > To: dev@samza.apache.org > Subject: Re: ProcessJobFactory parent process > > > Hi, Lukas, > > I assume that when you say "the job crashes", you were referring to the > child process running the container

Re: ProcessJobFactory parent process

2015-05-29 Thread Lukas Steiblys
: ProcessJobFactory parent process Hi, Lukas, Yes. That's exactly part of the feature to allow health-check/failure-detection of containers. Another short-term option is trying to use ThreadJobFactory, which has the JobCoordinator and containers in the same process. Does that work for your use case? -Yi O

Re: ProcessJobFactory parent process

2015-05-30 Thread Michael Ravits
why I switched to ProcessJobFactory earlier no longer hold true. > > Thanks. > > Lukas > > -Original Message- From: Yi Pan > Sent: Friday, May 29, 2015 2:05 PM > > To: dev@samza.apache.org > Subject: Re: ProcessJobFactory parent process > > Hi, Lukas, >

Re: ProcessJobFactory parent process

2015-06-01 Thread Lukas Steiblys
> > Lukas > > > > -Original Message- From: Yi Pan > > Sent: Friday, May 29, 2015 2:05 PM > > > > To: dev@samza.apache.org > > Subject: Re: ProcessJobFactory parent process > > > > Hi, Lukas, > > > > Yes. That's exactly p

Re: ProcessJobFactory parent process

2015-06-01 Thread Michael Ravits
t; wrote: > > > > > Yes, I think switching to ThreadJobFactory is a good solution. I think > > the > > > reasons why I switched to ProcessJobFactory earlier no longer hold > true. > > > > > > Thanks. > > > > > > Lukas > >

Re: ProcessJobFactory parent process

2015-06-01 Thread Yi Pan
ed to ProcessJobFactory earlier no longer hold > true. > > > > > > Thanks. > > > > > > Lukas > > > > > > -Original Message- From: Yi Pan > > > Sent: Friday, May 29, 2015 2:05 PM > > > > > > To: dev@samza.apac