Thanks Jagadish, just wanted to verify that this wasn't an obvious thing I was missing. I'm also setting the AM's heap size explicitly with the yarn.am.opts configuration parameter; in this case, I'm allowing 512MB for the JVM on top of my heap for both the AM and the container (ie, yarn.am.container.memory.mb=1536, yarn.am.opts=-Xmx1024m); does that sound reasonable?
Cheers, Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this message is prohibited. The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If you have received this message in error, please notify the sender immediately and delete the original message. Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this message is prohibited. The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If you have received this message in error, please notify the sender immediately and delete the original message. On Thu, May 23, 2019 at 10:39 PM Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > No, the default setup should be sufficient - the number of tasks should > have no significant impact on AM memory/resources. > > If you run out-of-memory, you can of course increase yarn.am.memory.mb. > > On Thu, May 23, 2019 at 10:45 AM Malcolm McFarland <mmcfarl...@cavulus.com > > > wrote: > > > Hey folks, > > > > Are there any guidelines for how to provision an Application Master in > > relation to the number of StreamTask instances it will be managing? Ie, > are > > there different memory, CPU, and thread-count figures for 100S > StreamTasks > > vs 1000, vs 10000? > > > > Cheers, > > Malcolm McFarland > > Cavulus > > > > > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any > > unauthorized or improper disclosure, copying, distribution, or use of the > > contents of this message is prohibited. The information contained in this > > message is intended only for the personal and confidential use of the > > recipient(s) named above. If you have received this message in error, > > please notify the sender immediately and delete the original message. > > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University >
