[ https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437166#comment-15437166 ]
Gopal V edited comment on YARN-5551 at 8/25/16 4:29 PM: -------------------------------------------------------- bq. The storage behind that mapping will not be freed even though the path has been deleted because this process still has an active mapping against it. That's exactly the point - these are actually not memory pages, these are pages borrowed from the buffer-cache. Some of them are dirty and some of them are clean, which implies that they are not actually memory consumed by the process if there's any memory pressure. The ideal mechanism for YARN to react would be to force a dirty flush for the specific process to reduce its memory footprint instead of always killing the process when the observed memory footprint is larger - killing a process is not the only way to reclaim memory from a process. Operating purely with kill signals is genuinely overkill. This implementation is trying to be more forgiving of a process which has a large number of clean pages in memory backed by a disk cache file, which are available to the process via .read or .map, but the disk buffer pages used by the OS are counted differently by YARN if it uses .map(). The underlying reality is the same even for dirty pages as the writes are being buffered into the buffer cache anyway, except the write() syscall moves it out of the process space faster than an .mmap + msync. was (Author: gopalv): bq. The storage behind that mapping will not be freed even though the path has been deleted because this process still has an active mapping against it. That's exactly the point - these are actually not memory pages, these are pages borrowed from the buffer-cache. Some of them are dirty and some of them are clean, which implies that they are not actually memory consumed by the process if there's any memory pressure. The ideal mechanism for YARN to react would be to force a dirty flush for the specific process to reduce its memory footprint instead of always killing the process when the observed memory footprint is larger - killing a process is not the only way to reclaim memory from a process. Operating purely with kill signals is genuinely overkill. This implementation is trying to be more forgiving of a process which has a large number of clean pages in memory backed by a disk cache file, which are available to the process via .read or .map, but the disk buffer pages used by the OS are counted differently by YARN if it uses .map(). > Ignore deleted file mapping from memory computation when smaps is enabled > ------------------------------------------------------------------------- > > Key: YARN-5551 > URL: https://issues.apache.org/jira/browse/YARN-5551 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Rajesh Balamohan > Assignee: Rajesh Balamohan > Priority: Minor > Attachments: YARN-5551.branch-2.001.patch > > > Currently deleted file mappings are also included in the memory computation > when SMAP is enabled. For e.g > {noformat} > 7f612004a000-7f612004c000 rw-s 00000000 00:10 4201507513 > /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185 > (deleted) > Size: 8 kB > Rss: 4 kB > Pss: 2 kB > Shared_Clean: 0 kB > Shared_Dirty: 4 kB > Private_Clean: 0 kB > Private_Dirty: 0 kB > Referenced: 4 kB > Anonymous: 0 kB > AnonHugePages: 0 kB > Swap: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > 7f6123f99000-7f6163f99000 rw-p 00000000 08:41 211419477 > /grid/4/hadoop/yarn/local/usercache/root/appcache/application_1466700718395_1249/container_e19_1466700718395_1249_01_000003/7389389356021597290.cache > (deleted) > Size: 1048576 kB > Rss: 637292 kB > Pss: 637292 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 637292 kB > Referenced: 637292 kB > Anonymous: 637292 kB > AnonHugePages: 0 kB > Swap: 0 kB > KernelPageSize: 4 kB > {noformat} > It would be good to exclude these from getSmapBasedRssMemorySize() > computation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org