@mwall , here is the logs. The last log is the Before and after the memory crash issue. I hope to find the reason, could be a memory leak in some place. I manually restarted the service at Feb 27 11:07:43.
*root@weather_bridge:~# sudo grep oom /var/log/** grep: /var/log/apache2: Is a directory grep: /var/log/apt: Is a directory /var/log/auth.log:Feb 27 12:51:09 weather_bridge sudo: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/grep oom /var/log/alternatives.log /var/log/apache2 /var/log/apt /var/log/auth.log /var/log/auth.log.1 /var/log/auth.log.2.gz /var/log/auth.log.3.gz /var/log/auth.log.4.gz /var/log/boot.log /var/log/bootstrap.log /var/log/btmp /var/log/daemon.log /var/log/daemon.log.1 /var/log/daemon.log.2.gz /var/log/daemon.log.3.gz /var/log/daemon.log.4.gz /var/log/debug /var/log/debug.1 /var/log/debug.2.gz /var/log/dmesg /var/log/dpkg.log /var/log/faillog /var/log/fsck /var/log/kern.log /var/log/kern.log.1 /var/log/kern.log.2.gz /var/log/kern.log.3.gz /var/log/kern.log.4.gz /var/log/lastlog /var/log/messages /var/log/messages.1 /var/log/messages.2.gz /var/log/messages.3.gz /var/log/messages.4.gz /var/log/ntpstats /var/log/samba /var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz /var/log/syslog.3.gz /var/log/syslog.4.gz /var/log/syslog.5.gz /var/log/syslog.6.gz /var/log/syslog.7.gz /var/log/user.lo grep: /var/log/fsck: Is a directory /var/log/kern.log:Feb 27 06:45:25 weather_bridge kernel: [1878102.857984] mandb invoked oom-killer: gfp_mask=0x26084c0, order=0, oom_score_adj=0 /var/log/kern.log:Feb 27 06:45:25 weather_bridge kernel: [1878102.858428] [<c012a3e4>] (dump_header) from [<c00dcd10>] (oom_kill_process+0x2b8/0x408) /var/log/kern.log:Feb 27 06:45:25 weather_bridge kernel: [1878102.858460] [<c00dcd10>] (oom_kill_process) from [<c00dd1a4>] (out_of_memory+0x2cc/0x324) /var/log/kern.log:Feb 27 06:45:26 weather_bridge kernel: [1878102.859209] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name /var/log/messages:Feb 27 06:45:25 weather_bridge kernel: [1878102.857984] mandb invoked oom-killer: gfp_mask=0x26084c0, order=0, oom_score_adj=0 /var/log/messages:Feb 27 06:45:25 weather_bridge kernel: [1878102.858428] [<c012a3e4>] (dump_header) from [<c00dcd10>] (oom_kill_process+0x2b8/0x408) /var/log/messages:Feb 27 06:45:25 weather_bridge kernel: [1878102.858460] [<c00dcd10>] (oom_kill_process) from [<c00dd1a4>] (out_of_memory+0x2cc/0x324) /var/log/messages:Feb 27 06:45:26 weather_bridge kernel: [1878102.859209] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name grep: /var/log/ntpstats: Is a directory grep: /var/log/samba: Is a directory /var/log/syslog:Feb 27 06:45:25 weather_bridge kernel: [1878102.857984] mandb invoked oom-killer: gfp_mask=0x26084c0, order=0, oom_score_adj=0 /var/log/syslog:Feb 27 06:45:25 weather_bridge kernel: [1878102.858428] [<c012a3e4>] (dump_header) from [<c00dcd10>] (oom_kill_process+0x2b8/0x408) /var/log/syslog:Feb 27 06:45:25 weather_bridge kernel: [1878102.858460] [<c00dcd10>] (oom_kill_process) from [<c00dd1a4>] (out_of_memory+0x2cc/0x324) /var/log/syslog:Feb 27 06:45:26 weather_bridge kernel: [1878102.859209] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name root@weather_bridge:~# *root@weather_bridge:~# sudo grep total_vm /var/log/** grep: /var/log/apache2: Is a directory grep: /var/log/apt: Is a directory /var/log/auth.log:Feb 27 12:52:09 weather_bridge sudo: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/grep total_vm /var/log/alternatives.log /var/log/apache2 /var/log/apt /var/log/auth.log /var/log/auth.log.1 /var/log/auth.log.2.gz /var/log/auth.log.3.gz /var/log/auth.log.4.gz /var/log/boot.log /var/log/bootstrap.log /var/log/btmp /var/log/daemon.log /var/log/daemon.log.1 /var/log/daemon.log.2.gz /var/log/daemon.log.3.gz /var/log/daemon.log.4.gz /var/log/debug /var/log/debug.1 /var/log/debug.2.gz /var/log/dmesg /var/log/dpkg.log /var/log/faillog /var/log/fsck /var/log/kern.log /var/log/kern.log.1 /var/log/kern.log.2.gz /var/log/kern.log.3.gz /var/log/kern.log.4.gz /var/log/lastlog /var/log/messages /var/log/messages.1 /var/log/messages.2.gz /var/log/messages.3.gz /var/log/messages.4.gz /var/log/ntpstats /var/log/samba /var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz /var/log/syslog.3.gz /var/log/syslog.4.gz /var/log/syslog.5.gz /var/log/syslog.6.gz /var/log/syslog.7.gz /var/log/auth.log:Feb 27 12:52:15 weather_bridge sudo: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/grep total_vm /var/log/alternatives.log /var/log/apache2 /var/log/apt /var/log/auth.log /var/log/auth.log.1 /var/log/auth.log.2.gz /var/log/auth.log.3.gz /var/log/auth.log.4.gz /var/log/boot.log /var/log/bootstrap.log /var/log/btmp /var/log/daemon.log /var/log/daemon.log.1 /var/log/daemon.log.2.gz /var/log/daemon.log.3.gz /var/log/daemon.log.4.gz /var/log/debug /var/log/debug.1 /var/log/debug.2.gz /var/log/dmesg /var/log/dpkg.log /var/log/faillog /var/log/fsck /var/log/kern.log /var/log/kern.log.1 /var/log/kern.log.2.gz /var/log/kern.log.3.gz /var/log/kern.log.4.gz /var/log/lastlog /var/log/messages /var/log/messages.1 /var/log/messages.2.gz /var/log/messages.3.gz /var/log/messages.4.gz /var/log/ntpstats /var/log/samba /var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz /var/log/syslog.3.gz /var/log/syslog.4.gz /var/log/syslog.5.gz /var/log/syslog.6.gz /var/log/syslog.7.gz grep: /var/log/fsck: Is a directory /var/log/kern.log:Feb 27 06:45:26 weather_bridge kernel: [1878102.859209] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name /var/log/messages:Feb 27 06:45:26 weather_bridge kernel: [1878102.859209] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name grep: /var/log/ntpstats: Is a directory grep: /var/log/samba: Is a directory /var/log/syslog:Feb 27 06:45:26 weather_bridge kernel: [1878102.859209] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name root@weather_bridge:~# *Log Before and After crash.* Feb 27 06:44:17 weather_bridge rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="390" x-info="http://www.rsyslog.com";] rsyslogd was HUPed Feb 27 06:44:19 weather_bridge rsyslogd0: action 'action 17' resumed (module 'builtin:ompipe') [try http://www.rsyslog.com/e/0 ] Feb 27 06:44:20 weather_bridge rsyslogd-2359: action 'action 17' resumed (module 'builtin:ompipe') [try http://www.rsyslog.com/e/2359 ] Feb 27 06:45:25 weather_bridge kernel: [1878102.857984] mandb invoked oom-killer: gfp_mask=0x26084c0, order=0, oom_score_adj=0 Feb 27 06:45:25 weather_bridge kernel: [1878102.858072] mandb cpuset=/ mems_allowed=0 Feb 27 06:45:25 weather_bridge kernel: [1878102.858118] CPU: 0 PID: 26272 Comm: mandb Not tainted 4.4.34+ #930 Feb 27 06:45:25 weather_bridge kernel: [1878102.858181] Hardware name: BCM2708 Feb 27 06:45:25 weather_bridge kernel: [1878102.858263] [<c0016cfc>] (unwind_backtrace) from [<c0013c20>] (show_stack+0x20/0x24) Feb 27 06:45:25 weather_bridge kernel: [1878102.858337] [<c0013c20>] (show_stack) from [<c02e437c>] (dump_stack+0x20/0x28) Feb 27 06:45:25 weather_bridge kernel: [1878102.858382] [<c02e437c>] (dump_stack) from [<c012a3e4>] (dump_header+0x60/0x18c) Feb 27 06:45:25 weather_bridge kernel: [1878102.858428] [<c012a3e4>] (dump_header) from [<c00dcd10>] (oom_kill_process+0x2b8/0x408) Feb 27 06:45:25 weather_bridge kernel: [1878102.858460] [<c00dcd10>] (oom_kill_process) from [<c00dd1a4>] (out_of_memory+0x2cc/0x324) Feb 27 06:45:25 weather_bridge kernel: [1878102.858494] [<c00dd1a4>] (out_of_memory) from [<c00e210c>] (__alloc_pages_nodemask+0xa14/0xa8c) Feb 27 06:45:25 weather_bridge kernel: [1878102.858526] [<c00e210c>] (__alloc_pages_nodemask) from [<c010355c>] (__pte_alloc+0x30/0x1b4) Feb 27 06:45:25 weather_bridge kernel: [1878102.858555] [<c010355c>] (__pte_alloc) from [<c0105c48>] (handle_mm_fault+0x654/0xf04) Feb 27 06:45:25 weather_bridge kernel: [1878102.858592] [<c0105c48>] (handle_mm_fault) from [<c05765a4>] (do_page_fault+0x340/0x3c4) Feb 27 06:45:25 weather_bridge kernel: [1878102.858621] [<c05765a4>] (do_page_fault) from [<c05766d4>] (do_translation_fault+0xac/0xb4) Feb 27 06:45:25 weather_bridge kernel: [1878102.858651] [<c05766d4>] (do_translation_fault) from [<c0009204>] (do_DataAbort+0x44/0xc4) Feb 27 06:45:25 weather_bridge kernel: [1878102.858678] [<c0009204>] (do_DataAbort) from [<c0575ecc>] (__dabt_usr+0x4c/0x60) Feb 27 06:45:25 weather_bridge kernel: [1878102.858695] Exception stack(0xc0c0dfb0 to 0xc0c0dff8) Feb 27 06:45:25 weather_bridge kernel: [1878102.858715] dfa0: 00000000 00022000 00000003 b6600000 Feb 27 06:45:25 weather_bridge kernel: [1878102.858739] dfc0: b6500000 00022000 00000000 b6e5a6f8 b6b00040 00001000 b68ffc68 b6800000 Feb 27 06:45:25 weather_bridge kernel: [1878102.858760] dfe0: 00000000 bec25828 b6d8af98 b6d8af9c 60000010 ffffffff Feb 27 06:45:25 weather_bridge kernel: [1878102.858772] Mem-Info: Feb 27 06:45:25 weather_bridge kernel: [1878102.858805] active_anon:50169 inactive_anon:50230 isolated_anon:0 Feb 27 06:45:26 weather_bridge kernel: [1878102.858805] active_file:22 inactive_file:25 isolated_file:0 Feb 27 06:45:26 weather_bridge kernel: [1878102.858805] unevictable:0 dirty:20 writeback:0 unstable:0 Feb 27 06:45:26 weather_bridge kernel: [1878102.858805] slab_reclaimable:869 slab_unreclaimable:1265 Feb 27 06:45:26 weather_bridge kernel: [1878102.858805] mapped:428 shmem:5433 pagetables:474 bounce:0 Feb 27 06:45:26 weather_bridge kernel: [1878102.858805] free:5923 free_pcp:0 free_cma:933 Feb 27 06:45:26 weather_bridge kernel: [1878102.858879] Normal free:23692kB min:16384kB low:20480kB high:24576kB active_anon:200676kB inactive_anon:200920kB active_file:88kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:458752kB managed:445088kB mlocked:0kB dirty:80kB writeback:0kB mapped:1712kB shmem:21732kB slab_reclaimable:3476kB slab_unreclaimable:5060kB kernel_stack:1232kB pagetables:1896kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:3732kB writeback_tmp:0kB pages_scanned:1128 all_unreclaimable? yes Feb 27 06:45:26 weather_bridge kernel: [1878102.858900] lowmem_reserve[]: 0 0 Feb 27 06:45:26 weather_bridge kernel: [1878102.858919] Normal: 443*4kB (UMEHC) 218*8kB (UMEHC) 89*16kB (UMEHC) 70*32kB (UMEHC) 24*64kB (UMEC) 15*128kB (UMEHC) 3*256kB (MEC) 2*512kB (ME) 3*1024kB (UEC) 4*2048kB (UEC) 0*4096kB = 23692kB Feb 27 06:45:26 weather_bridge kernel: [1878102.859060] 5554 total pagecache pages Feb 27 06:45:26 weather_bridge kernel: [1878102.859081] 74 pages in swap cache Feb 27 06:45:26 weather_bridge kernel: [1878102.859096] Swap cache stats: add 1615829, delete 1615755, find 759433/1173535 Feb 27 06:45:26 weather_bridge kernel: [1878102.859107] Free swap = 0kB Feb 27 06:45:26 weather_bridge kernel: [1878102.859117] Total swap = 102396kB Feb 27 06:45:26 weather_bridge kernel: [1878102.859130] 114688 pages RAM Feb 27 06:45:26 weather_bridge kernel: [1878102.859172] 0 pages HighMem/MovableOnly Feb 27 06:45:26 weather_bridge kernel: [1878102.859187] 3416 pages reserved Feb 27 06:45:26 weather_bridge kernel: [1878102.859197] 2048 pages cma reserved Feb 27 06:45:26 weather_bridge kernel: [1878102.859209] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Feb 27 06:45:26 weather_bridge kernel: [1878102.859282] [ 112] 0 112 2026 455 9 0 29 0 systemd-journal Feb 27 06:45:26 weather_bridge kernel: [1878102.859309] [ 114] 0 114 2853 0 7 0 97 -1000 systemd-udevd Feb 27 06:45:26 weather_bridge kernel: [1878102.859331] [ 340] 0 340 958 11 5 0 39 0 cron Feb 27 06:45:26 weather_bridge kernel: [1878102.859353] [ 341] 0 341 847 21 6 0 30 0 systemd-logind Feb 27 06:45:26 weather_bridge kernel: [1878102.859377] [ 348] 105 348 999 63 6 0 34 0 avahi-daemon Feb 27 06:45:26 weather_bridge kernel: [1878102.859432] [ 352] 104 352 1370 31 6 0 53 -900 dbus-daemon Feb 27 06:45:26 weather_bridge kernel: [1878102.859628] [ 362] 105 362 966 0 6 0 55 0 avahi-daemon Feb 27 06:45:26 weather_bridge kernel: [1878102.859654] [ 379] 65534 379 570 4 5 0 26 0 thd Feb 27 06:45:26 weather_bridge kernel: [1878102.859676] [ 390] 0 390 7810 113 10 0 138 0 rsyslogd Feb 27 06:45:26 weather_bridge kernel: [1878102.859698] [ 472] 0 472 139066 90779 254 0 23863 0 python Feb 27 06:45:26 weather_bridge kernel: [1878102.859720] [ 561] 0 561 638 33 5 0 29 0 dhcpcd Feb 27 06:45:26 weather_bridge kernel: [1878102.859741] [ 562] 0 562 1964 0 8 0 114 -1000 sshd Feb 27 06:45:26 weather_bridge kernel: [1878102.859764] [ 590] 106 590 1441 45 6 0 66 0 ntpd Feb 27 06:45:26 weather_bridge kernel: [1878102.859786] [ 601] 0 601 703 0 5 0 31 0 agetty Feb 27 06:45:26 weather_bridge kernel: [1878102.859807] [ 602] 0 602 658 0 5 0 31 0 agetty Feb 27 06:45:26 weather_bridge kernel: [1878102.859829] [ 608] 0 608 1803 223 8 0 46 0 apache2 Feb 27 06:45:26 weather_bridge kernel: [1878102.859850] [25045] 33 25045 57747 406 36 0 50 0 apache2 Feb 27 06:45:26 weather_bridge kernel: [1878102.859872] [25046] 33 25046 57755 406 36 0 45 0 apache2 Feb 27 06:45:26 weather_bridge kernel: [1878102.859896] [26096] 0 26096 1158 64 5 0 5 0 cron Feb 27 06:45:26 weather_bridge kernel: [1878102.859918] [26101] 0 26101 475 19 4 0 0 0 sh Feb 27 06:45:26 weather_bridge kernel: [1878102.859940] [26102] 0 26102 452 22 4 0 0 0 run-parts Feb 27 06:45:26 weather_bridge kernel: [1878102.859962] [26194] 0 26194 475 35 5 0 0 0 apache2 Feb 27 06:45:26 weather_bridge kernel: [1878102.859984] [26232] 0 26232 475 21 4 0 0 0 apache2ctl Feb 27 06:45:26 weather_bridge kernel: [1878102.860007] [26256] 0 26256 1562 179 7 0 0 0 apache2 Feb 27 06:45:26 weather_bridge kernel: [1878102.860076] [26269] 0 26269 475 18 5 0 0 0 man-db Feb 27 06:45:26 weather_bridge kernel: [1878102.860100] [26272] 6 26272 3606 2582 10 0 0 0 mandb Feb 27 06:45:26 weather_bridge kernel: [1878102.860119] Out of memory: Kill process 472 (python) score 814 or sacrifice child Feb 27 06:45:26 weather_bridge kernel: [1878102.860239] Killed process 472 (python) total-vm:556264kB, anon-rss:363116kB, file-rss:0kB Feb 27 06:45:27 weather_bridge apache2[26194]: Reloading web server: apache2. Feb 27 06:45:27 weather_bridge CRON[26096]: (CRON) info (No MTA installed, discarding output) Feb 27 07:17:01 weather_bridge CRON[26350]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 27 08:17:01 weather_bridge CRON[26366]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 27 09:17:01 weather_bridge CRON[26382]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 27 10:17:01 weather_bridge CRON[26398]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 27 11:07:42 weather_bridge systemd[1]: Stopping LSB: weewx weather system... Feb 27 11:07:43 weather_bridge weewx[26463]: Stopping weewx weather system: weewx not running.... Feb 27 11:07:43 weather_bridge systemd[1]: Stopped LSB: weewx weather system. Feb 27 11:07:47 weather_bridge systemd[1]: Starting LSB: weewx weather system... Feb 27 11:07:48 weather_bridge weewx[26520]: engine: Initializing weewx version 3.6.2 Feb 27 11:07:48 weather_bridge weewx[26520]: engine: Using Python 2.7.9 (default, Sep 17 2016, 20:26:04) #012[GCC 4.9.2] Feb 27 11:07:48 weather_bridge weewx[26520]: engine: Platform Linux-4.4.34+-armv6l-with-debian-8.0 Feb 27 11:07:48 weather_bridge weewx[26520]: engine: pid file is /var/run/weewx.pid Feb 27 11:07:49 weather_bridge weewx[26510]: Starting weewx weather system: weewx. Feb 27 11:07:49 weather_bridge systemd[1]: Started LSB: weewx weather system. Feb 27 11:07:49 weather_bridge weewx[26524]: engine: Using configuration file /etc/weewx/weewx.conf Feb 27 11:07:49 weather_bridge weewx[26524]: engine: Loading station type Ultimeter (weewx.drivers.ultimeter) Feb 27 11:07:49 weather_bridge weewx[26524]: ultimeter: driver version is 0.16 Feb 27 11:07:49 weather_bridge weewx[26524]: ultimeter: using serial port /dev/ttyUSB0 Feb 27 11:07:49 weather_bridge weewx[26524]: engine: StdConvert target unit is 0x1 On Monday, February 27, 2017 at 11:41:01 AM UTC-5, mwall wrote: > > francesco, > > it would also help if you could post more context about why the > out-of-memory killer was activated. see the answer to this posting: > > > http://unix.stackexchange.com/questions/128642/debug-out-of-memory-with-var-log-messages > > and post any oom/total_vm results you find: > > sudo grep oom /var/log/* > sudo grep total_vm /var/log/* > > m > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
