Hi RT Users, Since yesterday i have a very strange behavior on my RT. I have a cron script starting rt-crontool every 5 minutes.
/opt/rt3ast/bin/rt-crontool --search RT::Search::FromSQL --search-arg "Status='stalled' AND Queue='GAI_BreakBulk' AND ( 'CF.{CIEL-STATUS}' = '1300' OR 'CF.{CIEL-STATUS}' = '1400' )" --action RT::Action::AutoNew --action-arg new Inside this Queue are round about 130 tickets. After few hours the system slows down until it crashes with this error inside the messages: Nov 9 11:24:19 lxfrabort01 kernel: printk: 2731 messages suppressed. Nov 9 11:24:19 lxfrabort01 kernel: rt-crontool invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Nov 9 11:24:19 lxfrabort01 kernel: [<c0452c32>] out_of_memory+0x3b/0x179 Nov 9 11:24:19 lxfrabort01 kernel: [<c0454065>] __alloc_pages+0x1fe/0x27e Nov 9 11:24:19 lxfrabort01 kernel: [<c04552af>] __do_page_cache_readahead+0xc4/0x1c6 Nov 9 11:24:19 lxfrabort01 kernel: [<c044f991>] sync_page+0x0/0x3b Nov 9 11:24:19 lxfrabort01 kernel: [<c044c6f5>] __delayacct_blkio_end+0x32/0x35 Nov 9 11:24:19 lxfrabort01 kernel: [<c05fb55c>] __wait_on_bit_lock+0x4b/0x52 Nov 9 11:24:19 lxfrabort01 kernel: [<f8926b96>] dm_any_congested+0x2f/0x35 [dm_mod] Nov 9 11:24:19 lxfrabort01 kernel: [<c0452268>] filemap_nopage+0x151/0x315 Nov 9 11:24:19 lxfrabort01 kernel: [<c045a8c2>] __handle_mm_fault+0x172/0x87b Nov 9 11:24:19 lxfrabort01 kernel: [<c05fad89>] schedule+0x90d/0x9ba Nov 9 11:24:19 lxfrabort01 kernel: [<c042d63f>] do_sigaction+0x117/0x156 Nov 9 11:25:25 lxfrabort01 kernel: [<c05fd55f>] do_page_fault+0x20a/0x4b8 Nov 9 11:25:25 lxfrabort01 kernel: [<c05fd355>] do_page_fault+0x0/0x4b8 Nov 9 11:25:25 lxfrabort01 kernel: [<c0404a71>] error_code+0x39/0x40 Nov 9 11:25:25 lxfrabort01 kernel: ======================= Nov 9 11:25:25 lxfrabort01 kernel: Mem-info: Nov 9 11:25:25 lxfrabort01 kernel: DMA per-cpu: Nov 9 11:25:25 lxfrabort01 kernel: cpu 0 hot: high 0, batch 1 used:0 Nov 9 11:25:25 lxfrabort01 kernel: cpu 0 cold: high 0, batch 1 used:0 Nov 9 11:25:25 lxfrabort01 kernel: DMA32 per-cpu: empty Nov 9 11:25:25 lxfrabort01 kernel: Normal per-cpu: Nov 9 11:25:25 lxfrabort01 kernel: cpu 0 hot: high 186, batch 31 used:27 Nov 9 11:25:25 lxfrabort01 kernel: cpu 0 cold: high 62, batch 15 used:44 Nov 9 11:25:25 lxfrabort01 kernel: HighMem per-cpu: Nov 9 11:25:25 lxfrabort01 kernel: cpu 0 hot: high 186, batch 31 used:14 Nov 9 11:25:25 lxfrabort01 kernel: cpu 0 cold: high 62, batch 15 used:11 Nov 9 11:25:25 lxfrabort01 kernel: Free pages: 111660kB (1596kB HighMem) Nov 9 11:25:25 lxfrabort01 kernel: Active:470936 inactive:469165 dirty:0 writeback:0 unstable:0 free:27915 slab:1793 mapped:91 pagetables:2289 Nov 9 11:25:25 lxfrabort01 kernel: DMA free:12340kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes Nov 9 11:25:25 lxfrabort01 kernel: lowmem_reserve[]: 0 0 880 3824 Nov 9 11:25:25 lxfrabort01 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Nov 9 11:25:25 lxfrabort01 kernel: lowmem_reserve[]: 0 0 880 3824 Nov 9 11:25:25 lxfrabort01 kernel: Normal free:97724kB min:3756kB low:4692kB high:5632kB active:382784kB inactive:377384kB present:901120kB pages_scanned:1268501 all_unreclaimable? yes Nov 9 11:25:25 lxfrabort01 kernel: lowmem_reserve[]: 0 0 0 23552 Nov 9 11:25:25 lxfrabort01 kernel: HighMem free:1596kB min:512kB low:3656kB high:6800kB active:1500960kB inactive:1499276kB present:3014656kB pages_scanned:4695953 all_unreclaimable? yes Nov 9 11:25:25 lxfrabort01 kernel: lowmem_reserve[]: 0 0 0 0 Nov 9 11:25:25 lxfrabort01 kernel: DMA: 3*4kB 5*8kB 4*16kB 2*32kB 4*64kB 3*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12340kB Nov 9 11:25:25 lxfrabort01 kernel: DMA32: empty Nov 9 11:25:25 lxfrabort01 kernel: Normal: 1*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 23*4096kB = 97724kB Nov 9 11:25:25 lxfrabort01 kernel: HighMem: 271*4kB 16*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1596kB Nov 9 11:25:25 lxfrabort01 kernel: Swap cache: add 3707117, delete 3707077, find 546011/727786, race 0+20 Nov 9 11:25:25 lxfrabort01 kernel: Free swap = 0kB Nov 9 11:25:25 lxfrabort01 kernel: Total swap = 2031608kB Nov 9 11:25:25 lxfrabort01 kernel: Free swap: 0kB Nov 9 11:25:25 lxfrabort01 kernel: 983040 pages of RAM Nov 9 11:25:25 lxfrabort01 kernel: 753664 pages of HIGHMEM Nov 9 11:25:25 lxfrabort01 kernel: 9002 reserved pages Nov 9 11:25:25 lxfrabort01 kernel: 1520 pages shared Nov 9 11:25:25 lxfrabort01 kernel: 40 pages swap cached Nov 9 11:25:25 lxfrabort01 kernel: 0 pages dirty Nov 9 11:25:25 lxfrabort01 kernel: 0 pages writeback Nov 9 11:25:25 lxfrabort01 kernel: 91 pages mapped Nov 9 11:25:25 lxfrabort01 kernel: 1793 pages slab Nov 9 11:25:25 lxfrabort01 kernel: 2289 pages pagetables What i googled so far is, that the oom_killer is a process that checks for process that causes massive ram usage and tries to kill them - without success. This is a vm installation with centos 5. 1 Processor at 3 GHz with 4 GB RAM and 4 GB Swap. Are there any know issues with RT-CRONTOOL and memory problems? Torsten _______________________________________________ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users SAVE THOUSANDS OF DOLLARS ON RT SUPPORT: If you sign up for a new RT support contract before December 31, we'll take up to 20 percent off the price. This sale won't last long, so get in touch today. Email us at [EMAIL PROTECTED] or call us at +1 617 812 0745. Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com