Hmm, oh bugger, it seems the kernel errors are not the issue. The question remains what is. I still see the errors every few minutes. The errors make reference to GELF, and I only have one GELF tcp input from 2 Windows boxes running NXlog. The errors seem to have started with the last graylog-server update.
On Friday, May 1, 2015 at 4:29:49 PM UTC-7, Mark Moorcroft wrote: > > > This morning I was seeing bunches of errors in the server.log. I think I > tracked them to a syslog/tcp input. My rsyslog entry on the client is as > follows. > > # Graylog > $template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% > %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% > %STRUCTURED-DATA% %msg%\n" > *.* @@xxx.xxx.xxx.xxx:12204;GRAYLOGRFC5424 > > It seems the cause was memory errors on a compute node. The question is if > this is a graylog bug or expected behavior. There were a series of these > "com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'xxx': > was expecting ('true', 'false' or 'null') at [Source:" errors. I'm running > the current versions of graylog-server and elasticsearch. The token 'xxx' > is a random character(s) and then a massive bunch of garbage characters > will follow the error. > > From /var/log/messages: > > May 1 13:08:56 compute-0-21 kernel: flush-8:0: page allocation failure. > order:2, mode:0x20 > May 1 13:08:56 compute-0-21 kernel: Pid: 444, comm: flush-8:0 Not tainted > 2.6.32-431.11.2.el6.x86_64 #1 > May 1 13:08:56 compute-0-21 kernel: Call Trace: > May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ? > __alloc_pages_nodemask+0x74a/0x8d0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ? > kmem_getpages+0x62/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ? > fallback_alloc+0x1ba/0x270 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ? > cache_grow+0x2cf/0x320 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ? > ____cache_alloc_node+0x99/0x160 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ? > kmem_cache_alloc_node_trace+0x90/0x200 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ? > __kmalloc_node+0x4d/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ? > __alloc_skb+0x7a/0x180 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ? > dev_alloc_skb+0x1d/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ? > nv_alloc_rx_optimized+0x198/0x270 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025bc76>] ? > nv_rx_process_optimized+0x126/0x2a0 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ? > nv_napi_poll+0x8c/0x610 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ? > scheduler_tick+0xcc/0x260 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ? > net_rx_action+0x103/0x2f0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ? > free_pcppages_bulk+0x392/0x460 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ? > __do_softirq+0xc1/0x1e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ? > handle_IRQ_event+0x60/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ? > call_softirq+0x1c/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ? > do_softirq+0x65/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ? > irq_exit+0x85/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? > do_IRQ+0x75/0xf0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ? > ret_from_intr+0x0/0x11 > May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff811bdd20>] ? > submit_bh+0x60/0x1f0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0598>] ? > __block_write_full_page+0x1c8/0x330 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bf560>] ? > end_buffer_async_write+0x0/0x190 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c07e0>] ? > block_write_full_page_endio+0xe0/0x120 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c4b30>] ? > buffer_unmapped+0x0/0x20 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0835>] ? > block_write_full_page+0x15/0x20 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c56dd>] ? > ext3_ordered_writepage+0x1ed/0x240 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336c7>] ? > __writepage+0x17/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113498d>] ? > write_cache_pages+0x1fd/0x4c0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa0203e28>] ? > __ext4_journal_stop+0x68/0xa0 [ext4] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336b0>] ? > __writepage+0x0/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134c74>] ? > generic_writepages+0x24/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134cb5>] ? > do_writepages+0x35/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b50cd>] ? > writeback_single_inode+0xdd/0x290 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b54cd>] ? > writeback_sb_inodes+0xbd/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b562b>] ? > writeback_inodes_wb+0xab/0x1b0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5a23>] ? > wb_writeback+0x2f3/0x410 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81527f30>] ? > thread_return+0x4e/0x76e > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81084d92>] ? > del_timer_sync+0x22/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5bfb>] ? > wb_do_writeback+0xbb/0x240 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5de3>] ? > bdi_writeback_task+0x63/0x1b0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b117>] ? > bit_waitqueue+0x17/0xd0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ? > bdi_start_fn+0x0/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81143a26>] ? > bdi_start_fn+0x86/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ? > bdi_start_fn+0x0/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109aee6>] ? > kthread+0x96/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c20a>] ? > child_rip+0xa/0x20 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109ae50>] ? > kthread+0x0/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c200>] ? > child_rip+0x0/0x20 > May 1 13:08:56 compute-0-21 kernel: flush-8:0: page allocation failure. > order:2, mode:0x20 > May 1 13:08:56 compute-0-21 kernel: Pid: 444, comm: flush-8:0 Not tainted > 2.6.32-431.11.2.el6.x86_64 #1 > May 1 13:08:56 compute-0-21 kernel: Call Trace: > May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ? > __alloc_pages_nodemask+0x74a/0x8d0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ? > kmem_getpages+0x62/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ? > fallback_alloc+0x1ba/0x270 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ? > cache_grow+0x2cf/0x320 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ? > ____cache_alloc_node+0x99/0x160 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ? > kmem_cache_alloc_node_trace+0x90/0x200 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ? > __kmalloc_node+0x4d/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ? > __alloc_skb+0x7a/0x180 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ? > dev_alloc_skb+0x1d/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ? > nv_alloc_rx_optimized+0x198/0x270 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025bc76>] ? > nv_rx_process_optimized+0x126/0x2a0 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ? > nv_napi_poll+0x8c/0x610 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ? > scheduler_tick+0xcc/0x260 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ? > net_rx_action+0x103/0x2f0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ? > free_pcppages_bulk+0x392/0x460 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ? > __do_softirq+0xc1/0x1e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ? > handle_IRQ_event+0x60/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ? > call_softirq+0x1c/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ? > do_softirq+0x65/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ? > irq_exit+0x85/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? > do_IRQ+0x75/0xf0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ? > ret_from_intr+0x0/0x11 > May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff811bdd20>] ? > submit_bh+0x60/0x1f0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0598>] ? > __block_write_full_page+0x1c8/0x330 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bf560>] ? > end_buffer_async_write+0x0/0x190 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c07e0>] ? > block_write_full_page_endio+0xe0/0x120 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c4b30>] ? > buffer_unmapped+0x0/0x20 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0835>] ? > block_write_full_page+0x15/0x20 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c56dd>] ? > ext3_ordered_writepage+0x1ed/0x240 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336c7>] ? > __writepage+0x17/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113498d>] ? > write_cache_pages+0x1fd/0x4c0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa0203e28>] ? > __ext4_journal_stop+0x68/0xa0 [ext4] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336b0>] ? > __writepage+0x0/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134c74>] ? > generic_writepages+0x24/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134cb5>] ? > do_writepages+0x35/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b50cd>] ? > writeback_single_inode+0xdd/0x290 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b54cd>] ? > writeback_sb_inodes+0xbd/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b562b>] ? > writeback_inodes_wb+0xab/0x1b0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5a23>] ? > wb_writeback+0x2f3/0x410 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81527f30>] ? > thread_return+0x4e/0x76e > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81084d92>] ? > del_timer_sync+0x22/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5bfb>] ? > wb_do_writeback+0xbb/0x240 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5de3>] ? > bdi_writeback_task+0x63/0x1b0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b117>] ? > bit_waitqueue+0x17/0xd0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ? > bdi_start_fn+0x0/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81143a26>] ? > bdi_start_fn+0x86/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ? > bdi_start_fn+0x0/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109aee6>] ? > kthread+0x96/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c20a>] ? > child_rip+0xa/0x20 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109ae50>] ? > kthread+0x0/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c200>] ? > child_rip+0x0/0x20 > May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure. > order:2, mode:0x20 > May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted > 2.6.32-431.11.2.el6.x86_64 #1 > May 1 13:08:56 compute-0-21 kernel: Call Trace: > May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ? > __alloc_pages_nodemask+0x74a/0x8d0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ? > kmem_getpages+0x62/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ? > fallback_alloc+0x1ba/0x270 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ? > cache_grow+0x2cf/0x320 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ? > ____cache_alloc_node+0x99/0x160 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105a360>] ? > check_preempt_wakeup+0x1c0/0x260 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ? > kmem_cache_alloc_node_trace+0x90/0x200 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105572c>] ? > check_preempt_curr+0x7c/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ? > __kmalloc_node+0x4d/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ? > __alloc_skb+0x7a/0x180 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ? > dev_alloc_skb+0x1d/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ? > nv_alloc_rx_optimized+0x198/0x270 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025a08b>] ? > nv_tx_done_optimized+0x5b/0x200 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ? > nv_napi_poll+0x8c/0x610 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ? > scheduler_tick+0xcc/0x260 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ? > net_rx_action+0x103/0x2f0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ? > free_pcppages_bulk+0x392/0x460 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ? > __do_softirq+0xc1/0x1e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ? > handle_IRQ_event+0x60/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ? > call_softirq+0x1c/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ? > do_softirq+0x65/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ? > irq_exit+0x85/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? > do_IRQ+0x75/0xf0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ? > ret_from_intr+0x0/0x11 > May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffffa02c51dd>] ? > ext3_get_branch+0xed/0x140 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c5188>] ? > ext3_get_branch+0x98/0x140 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c721f>] ? > ext3_get_blocks_handle+0xdf/0xe20 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811422c9>] ? > zone_statistics+0x99/0xc0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bd919>] ? > alloc_buffer_head+0x49/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811be416>] ? > alloc_page_buffers+0x86/0xf0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c8027>] ? > ext3_get_block+0xc7/0x120 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0ed3>] ? > __block_prepare_write+0x1d3/0x530 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ? > ext3_get_block+0x0/0x120 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c153c>] ? > block_write_begin_newtrunc+0x5c/0xd0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c1943>] ? > block_write_begin+0x43/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ? > ext3_get_block+0x0/0x120 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c999b>] ? > ext3_write_begin+0x20b/0x2c0 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ? > ext3_get_block+0x0/0x120 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ? > generic_file_buffered_write+0x123/0x2e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ? > __generic_file_aio_write+0x260/0x490 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ? > generic_file_aio_write+0x88/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ? > do_sync_write+0xfa/0x140 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ? > autoremove_wake_function+0x0/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ? > security_file_permission+0x16/0x20 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ? > vfs_write+0xb8/0x1a0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ? > sys_write+0x51/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ? > system_call_fastpath+0x16/0x1b > May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure. > order:2, mode:0x20 > May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted > 2.6.32-431.11.2.el6.x86_64 #1 > May 1 13:08:56 compute-0-21 kernel: Call Trace: > May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ? > __alloc_pages_nodemask+0x74a/0x8d0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ? > kmem_getpages+0x62/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ? > fallback_alloc+0x1ba/0x270 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ? > cache_grow+0x2cf/0x320 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ? > ____cache_alloc_node+0x99/0x160 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105a360>] ? > check_preempt_wakeup+0x1c0/0x260 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ? > kmem_cache_alloc_node_trace+0x90/0x200 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ? > __kmalloc_node+0x4d/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ? > __alloc_skb+0x7a/0x180 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ? > dev_alloc_skb+0x1d/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ? > nv_alloc_rx_optimized+0x198/0x270 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81043780>] ? > gart_map_page+0x0/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ? > nv_napi_poll+0x8c/0x610 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a758>] ? > irq_exit+0x48/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? > do_IRQ+0x75/0xf0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ? > net_rx_action+0x103/0x2f0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ? > __do_softirq+0xc1/0x1e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ? > handle_IRQ_event+0x60/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ? > call_softirq+0x1c/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ? > do_softirq+0x65/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ? > irq_exit+0x85/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? > do_IRQ+0x75/0xf0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ? > ret_from_intr+0x0/0x11 > May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffffa02c51dd>] ? > ext3_get_branch+0xed/0x140 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c5188>] ? > ext3_get_branch+0x98/0x140 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c721f>] ? > ext3_get_blocks_handle+0xdf/0xe20 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811422c9>] ? > zone_statistics+0x99/0xc0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bd919>] ? > alloc_buffer_head+0x49/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811be416>] ? > alloc_page_buffers+0x86/0xf0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c8027>] ? > ext3_get_block+0xc7/0x120 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0ed3>] ? > __block_prepare_write+0x1d3/0x530 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ? > ext3_get_block+0x0/0x120 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c153c>] ? > block_write_begin_newtrunc+0x5c/0xd0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c1943>] ? > block_write_begin+0x43/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ? > ext3_get_block+0x0/0x120 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c999b>] ? > ext3_write_begin+0x20b/0x2c0 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ? > ext3_get_block+0x0/0x120 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ? > generic_file_buffered_write+0x123/0x2e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ? > __generic_file_aio_write+0x260/0x490 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ? > generic_file_aio_write+0x88/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ? > do_sync_write+0xfa/0x140 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ? > autoremove_wake_function+0x0/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ? > security_file_permission+0x16/0x20 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ? > vfs_write+0xb8/0x1a0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ? > sys_write+0x51/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ? > system_call_fastpath+0x16/0x1b > May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure. > order:2, mode:0x20 > May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted > 2.6.32-431.11.2.el6.x86_64 #1 > May 1 13:08:56 compute-0-21 kernel: Call Trace: > May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ? > __alloc_pages_nodemask+0x74a/0x8d0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ? > tick_program_event+0x2a/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ? > kmem_getpages+0x62/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ? > fallback_alloc+0x1ba/0x270 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ? > cache_grow+0x2cf/0x320 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ? > ____cache_alloc_node+0x99/0x160 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ? > kmem_cache_alloc_node_trace+0x90/0x200 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ? > __kmalloc_node+0x4d/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ? > __alloc_skb+0x7a/0x180 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ? > dev_alloc_skb+0x1d/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ? > nv_alloc_rx_optimized+0x198/0x270 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ? > nv_napi_poll+0x8c/0x610 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ? > net_rx_action+0x103/0x2f0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ? > __do_softirq+0xc1/0x1e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ? > tick_program_event+0x2a/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ? > call_softirq+0x1c/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ? > do_softirq+0x65/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ? > irq_exit+0x85/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff815316ca>] ? > smp_apic_timer_interrupt+0x4a/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100bb93>] ? > apic_timer_interrupt+0x13/0x20 > May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff81138fe7>] ? > shrink_page_list.clone.3+0xf7/0x650 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81138fe0>] ? > shrink_page_list.clone.3+0xf0/0x650 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117834b>] ? > mem_cgroup_lru_del_list+0x2b/0xb0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811397d7>] ? > isolate_lru_pages.clone.0+0xd7/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81142b5d>] ? > wait_iff_congested+0x6d/0xe0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81139f23>] ? > shrink_inactive_list+0x343/0x830 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113a7be>] ? > shrink_mem_cgroup_zone+0x3ae/0x610 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117a01d>] ? > mem_cgroup_iter+0xfd/0x280 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113aa83>] ? > shrink_zone+0x63/0xb0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113abe5>] ? > do_try_to_free_pages+0x115/0x610 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112d2dc>] ? > get_page_from_freelist+0x15c/0x870 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113b2b2>] ? > try_to_free_pages+0x92/0x120 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112f70e>] ? > __alloc_pages_nodemask+0x47e/0x8d0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81167aaa>] ? > alloc_pages_current+0xaa/0x110 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fd47>] ? > __page_cache_alloc+0x87/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81120a5e>] ? > grab_cache_page_write_begin+0x8e/0xc0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c981f>] ? > ext3_write_begin+0x8f/0x2c0 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ? > unlock_page+0x27/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ? > generic_file_buffered_write+0x123/0x2e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ? > __generic_file_aio_write+0x260/0x490 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ? > generic_file_aio_write+0x88/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ? > do_sync_write+0xfa/0x140 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ? > autoremove_wake_function+0x0/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ? > security_file_permission+0x16/0x20 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ? > vfs_write+0xb8/0x1a0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ? > sys_write+0x51/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ? > system_call_fastpath+0x16/0x1b > May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure. > order:2, mode:0x20 > May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted > 2.6.32-431.11.2.el6.x86_64 #1 > May 1 13:08:56 compute-0-21 kernel: Call Trace: > May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ? > __alloc_pages_nodemask+0x74a/0x8d0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ? > tick_program_event+0x2a/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ? > kmem_getpages+0x62/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ? > fallback_alloc+0x1ba/0x270 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ? > cache_grow+0x2cf/0x320 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ? > ____cache_alloc_node+0x99/0x160 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ? > kmem_cache_alloc_node_trace+0x90/0x200 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ? > __kmalloc_node+0x4d/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ? > __alloc_skb+0x7a/0x180 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ? > dev_alloc_skb+0x1d/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ? > nv_alloc_rx_optimized+0x198/0x270 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ? > nv_napi_poll+0x8c/0x610 [forcedeth] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ? > net_rx_action+0x103/0x2f0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ? > __do_softirq+0xc1/0x1e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ? > tick_program_event+0x2a/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ? > call_softirq+0x1c/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ? > do_softirq+0x65/0xa0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ? > irq_exit+0x85/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff815316ca>] ? > smp_apic_timer_interrupt+0x4a/0x60 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100bb93>] ? > apic_timer_interrupt+0x13/0x20 > May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff8109b1d4>] ? > __wake_up_bit+0x4/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ? > unlock_page+0x27/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81138fb0>] ? > shrink_page_list.clone.3+0xc0/0x650 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117834b>] ? > mem_cgroup_lru_del_list+0x2b/0xb0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811397d7>] ? > isolate_lru_pages.clone.0+0xd7/0x170 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81142b5d>] ? > wait_iff_congested+0x6d/0xe0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81139f23>] ? > shrink_inactive_list+0x343/0x830 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113a7be>] ? > shrink_mem_cgroup_zone+0x3ae/0x610 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117a01d>] ? > mem_cgroup_iter+0xfd/0x280 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113aa83>] ? > shrink_zone+0x63/0xb0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113abe5>] ? > do_try_to_free_pages+0x115/0x610 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112d2dc>] ? > get_page_from_freelist+0x15c/0x870 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113b2b2>] ? > try_to_free_pages+0x92/0x120 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811411c0>] ? > next_zone+0x30/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112f70e>] ? > __alloc_pages_nodemask+0x47e/0x8d0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81167aaa>] ? > alloc_pages_current+0xaa/0x110 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fd47>] ? > __page_cache_alloc+0x87/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81120a5e>] ? > grab_cache_page_write_begin+0x8e/0xc0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c981f>] ? > ext3_write_begin+0x8f/0x2c0 [ext3] > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ? > unlock_page+0x27/0x30 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ? > generic_file_buffered_write+0x123/0x2e0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ? > __generic_file_aio_write+0x260/0x490 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ? > generic_file_aio_write+0x88/0x100 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ? > do_sync_write+0xfa/0x140 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ? > autoremove_wake_function+0x0/0x40 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ? > security_file_permission+0x16/0x20 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ? > vfs_write+0xb8/0x1a0 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ? > sys_write+0x51/0x90 > May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ? > system_call_fastpath+0x16/0x1b > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.