Hi Pradeep, This is a great content to add to the documents. These are the common set of errors which might get googled and hence great to get indexed as well.
You can take a look at: https://github.com/apache/apex-core/tree/master/docs The docs for apex reside there in markdown format. Probably its good a create a troubleshooting page where all such common questions can reside. After you have the content ready, you can create a pull request to apex-core repo which can get merged to apex-core and later deployed to the website by committers. -Chinmay. On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <p...@apache.org> wrote: > Container & memory resource allocation has been a common question around > and so I thought it would be good to explain related configuration > parameters. > > Please feel free to let me know your thoughts. > > Also I'm planning to add following set of information under Apex Docs. How > could one add this to Apex Docs? > > =-=-=-= > > "Container is running beyond physical memory limits. Current usage: X GB of > Y GB physical memory used; A GB of B GB virtual memory used. Killing > container." > > This is basically for some better understanding on Application Master's > container requests & Resource Manager's memory resource allocation. Please > note that these are individual container request params. All these > parameters are in MB i.e. 1024 => 1GB. > > - AM's container requests to RM shall contain memory in the multiples of > *yarn.scheduler.minimum-**allocation-mb* & not exceeding > *yarn.scheduler.maximum-**allocation-mb* > - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024 and > container memory requirement is 1025 ( <= 2048 ), container will be > allocated with 2048 memory. > > - With Apex applications, operator memory can be specified by property > *dt.application.<APP_NAME>.operator.<OPERATOR_NAME>.attr.MEMORY_MB* > - Please note this parameter is at Operator level and container memory > is calculated based on number of Operators deployed in a container + > additional memory required depending on physical deployment requirements > e.g. unifier or bufferserver > - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME > > - If container memory is not specified, then AM would request for 1 unit of > *yarn.scheduler.minimum-**allocation-mb*, RM would provision container > taking that into consideration. > > Node Manager monitors memory usage of each of these containers and kills > the ones crossing the configured limit. > > Almost similar stuff is applicable for CPUs. > > --prad >
