Hi all, I just added the updated patch <https://issues.apache.org/jira/secure/attachment/12791615/SAMZA-41.0-updated.patch> to SAMZA-41 <https://issues.apache.org/jira/browse/SAMZA-41>.
We have been testing all week in our SAMZA Jobs and is looking good. Hope is helpful and can be committed to the project soon, let me know if there is any concern, happy to help. @Jagadish, I'll be checking SAMZA-881 and add our inputs and what we learned after including the patch. -Best, Jose Luis Barrueta On Thu, Mar 3, 2016 at 2:07 PM Yi Pan <nickpa...@gmail.com> wrote: > Hi, Jose, > > Great! I am looking forward to the patch! Could you open an RB to the JIRA? > > Thanks a lot! > > -Yi > > On Wed, Mar 2, 2016 at 2:31 PM, José Barrueta <j...@stormpath.com> wrote: > > > Hi guys, > > > > At Stormpath, we made a custom samza 10 version merging SAMZA-41 into it, > > it's working well, so we are thinking to update that patch later this > week > > so it can be added to the main project. > > > > HTH, > > > > Jose Luis Barrueta > > > > On Wed, Mar 2, 2016 at 2:11 PM, Yi Pan <nickpa...@gmail.com> wrote: > > > > > Hi, Robert, > > > > > > The main reason that ThreadJobFactory and ProcessJobFactory are not > > > considered "production-ready" is that there is only one container for > the > > > job and all tasks are assigned to the single container. Hence, it is > not > > > easy to scale out of a single host. > > > > > > As Rick mentioned, Netflix has put up a patch in SAMZA-41 based on > 0.9.1 > > o > > > allow static assignment of a subset of partitions to a single > ProcessJob, > > > which allows to launch multiple ProcessJobs in different hosts. We > > planned > > > to merge it to 0.10. But it turns out that too much changes have gone > > into > > > 0.10 and it became difficult to merge the patch. At this point, we can > > > still try the following two options: > > > 1) We can attempt to merge SAMZA-41 to 0.10.1 again, it may take some > > > effort but would give a stop-gap solution. > > > 2) We are working on a standalone Samza model (SAMZA-516, SAMZA-881) to > > > allow users to run Samza w/o depending on YarnJobFactory. This is a > > > long-term effort and will take some time to flesh out. Please join the > > > discussion there s.t. we can be more aligned in our effort. > > > > > > Hope the above gives you an overall picture on where we are going. > > > > > > Thanks a lot! > > > > > > -Yi > > > > > > On Wed, Mar 2, 2016 at 1:28 PM, Rick Mangi <r...@chartbeat.com> wrote: > > > > > > > There was an interesting thread a while back from I believe the > netflix > > > > guys about running ThreadJobFactory in production. > > > > > > > > > > > > > On Mar 2, 2016, at 4:20 PM, Robert Crim <rjc...@gmail.com> wrote: > > > > > > > > > > Hi, > > > > > > > > > > We're currently working on a solution that allows us to run Samza > > jobs > > > on > > > > > Mesos. This seems to be going well, and something we'd like to move > > > away > > > > > from when native Mesos support is added to Samza. > > > > > > > > > > While we're developing and testing our scheduler, I'm wondering > about > > > the > > > > > implications of running tasks with the ThreadJobFactory in > > > "production". > > > > > The documentation advise against this, but it's not clear why. > > > > > > > > > > If we were using the ThreadJobFactory inside of a docker container > on > > > > Mesos > > > > > with Marathon for production, would be our main problem? These are > > not > > > > > particularly high-load tasks. Aside from not be able to get > > > find-grained > > > > > resource scheduling per-task, it seems like the main issue the not > > > being > > > > to > > > > > easily tell when a job stops due to error / exception. > > > > > > > > > > In other words, what would be stop-stopping reasons to not use the > > > > > TreadJobFactory in production? > > > > > > > > > > Thanks, > > > > > Rob > > > > > > > > > > > > > >
