That sounds really good to me. I believe our use case is more suited to the jobs. We provision a new job in some cases per user. In some cases the user would use the multi-tenant instance of our service which scale as needed itself . In other cases that user might need a dedicated instance of our service that can also scale out as needed.
It seems that both cases don't seem like a problem since your usage of ZK is limited to mostly service discovery. We run most Mesos components on baremetal with SSDs so in theory we should be good to go. -----Zameer Manji <zma...@apache.org> wrote: -----To: dev@aurora.apache.org From: Zameer Manji <zma...@apache.org> Date: 03/18/2016 01:24PM Subject: Re: aurora job scalability I would like to chime in and just say from experience Aurora scales to the number of instances across all jobs. From a scale perspective there isn't much difference between having a thousand 1 instance jobs and a single job that has 1k instances as both cases roughly take up the same amount of memory. As Josh mentioned Twitter is running thousands of jobs and these jobs span hundreds of thousands of instances. On Fri, Mar 18, 2016 at 9:27 AM, Joshua Cohen <jco...@apache.org> wrote: > Hi Christopher, > > I think you already got an answer from Stephan in IRC, but just wanted to > follow up for the sake of posterity (in case anyone in the future has a > similar question and finds this thread). The only limit on the number of > jobs that Aurora can run would currently be the amount of memory available > to the Scheduler. Suffice it to say that at Twitter we're running thousands > of jobs with no issues. > > Let us know if you have any follow up questions. > > Cheers, > > Joshua > > On Fri, Mar 18, 2016 at 9:12 AM, Christopher M Luciano < > cmluci...@us.ibm.com > > wrote: > > > Hi all. It seems that we may be outgrowing Marathon. We have a problem > > with the amount of application that we are using, causing us to not > exactly > > be "compliant" with Marathon goals. It seems that the unit for Aurora is > a > > job+instance of that job. Does job map to a Marathon application? If > > similar is there a known limitation to how many jobs one can have? > > > > What we discovered for Marathon more application+bigger env_vars = > bigger > > zknode size and we come dangerously close to hitting the 1 MB default of > > the zk zknode size. I'm wondering if this type of a thing has potentially > > been fixed already in Aurora. > > > > > > > > > > > > > > Christopher M Luciano > > > > Staff Software Engineer, Platform Services > > > > IBM Watson Core Technology > > > > > > > > > > > > -- > Zameer Manji > >
