[graylog2] Re: com.fasterxml.jackson.core.JsonParseException:

2015-05-01 Thread Mark Moorcroft 

I found a year or more old reference to the same error and the solution 
back then was to switch the NXlog ouput from tcp to udp. The same change 
seems to have stopped the errors now. So it seems I can't currently use 
om_tcp in NXlog to send Windows logs. I'm not sure why udp works and tcp 
doesn't.


On Friday, May 1, 2015 at 4:29:49 PM UTC-7, Mark Moorcroft wrote:


 This morning I was seeing bunches of errors in the server.log. I think I 
 tracked them to a syslog/tcp input. My rsyslog entry on the client is as 
 follows.




-- 
You received this message because you are subscribed to the Google Groups 
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: com.fasterxml.jackson.core.JsonParseException:

2015-05-01 Thread Mark Moorcroft 

Hmm, oh bugger, it seems the kernel errors are not the issue. The question 
remains what is. I still see the errors every few minutes. The errors make 
reference to GELF, and I only have one GELF tcp input from 2 Windows boxes 
running NXlog. The errors seem to have started with the last graylog-server 
update.


On Friday, May 1, 2015 at 4:29:49 PM UTC-7, Mark Moorcroft wrote:


 This morning I was seeing bunches of errors in the server.log. I think I 
 tracked them to a syslog/tcp input. My rsyslog entry on the client is as 
 follows.

 # Graylog
 $template GRAYLOGRFC5424,%PRI%%PROTOCOL-VERSION% 
 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% 
 %STRUCTURED-DATA% %msg%\n
 *.* @@xxx.xxx.xxx.xxx:12204;GRAYLOGRFC5424

 It seems the cause was memory errors on a compute node. The question is if 
 this is a graylog bug or expected behavior. There were a series of these 
 com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'xxx': 
 was expecting ('true', 'false' or 'null') at [Source:  errors. I'm running 
 the current versions of graylog-server and elasticsearch. The token 'xxx' 
 is a random character(s) and then a massive bunch of garbage characters 
 will follow the error.

 From /var/log/messages:

 May  1 13:08:56 compute-0-21 kernel: flush-8:0: page allocation failure. 
 order:2, mode:0x20
 May  1 13:08:56 compute-0-21 kernel: Pid: 444, comm: flush-8:0 Not tainted 
 2.6.32-431.11.2.el6.x86_64 #1
 May  1 13:08:56 compute-0-21 kernel: Call Trace:
 May  1 13:08:56 compute-0-21 kernel: IRQ  [8112f9da] ? 
 __alloc_pages_nodemask+0x74a/0x8d0
 May  1 13:08:56 compute-0-21 kernel: [8116e492] ? 
 kmem_getpages+0x62/0x170
 May  1 13:08:56 compute-0-21 kernel: [8116f0aa] ? 
 fallback_alloc+0x1ba/0x270
 May  1 13:08:56 compute-0-21 kernel: [8116eaff] ? 
 cache_grow+0x2cf/0x320
 May  1 13:08:56 compute-0-21 kernel: [8116ee29] ? 
 cache_alloc_node+0x99/0x160
 May  1 13:08:56 compute-0-21 kernel: [8116fff0] ? 
 kmem_cache_alloc_node_trace+0x90/0x200
 May  1 13:08:56 compute-0-21 kernel: [8117020d] ? 
 __kmalloc_node+0x4d/0x60
 May  1 13:08:56 compute-0-21 kernel: [8145033a] ? 
 __alloc_skb+0x7a/0x180
 May  1 13:08:56 compute-0-21 kernel: [8145090d] ? 
 dev_alloc_skb+0x1d/0x40
 May  1 13:08:56 compute-0-21 kernel: [a025c728] ? 
 nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
 May  1 13:08:56 compute-0-21 kernel: [a025bc76] ? 
 nv_rx_process_optimized+0x126/0x2a0 [forcedeth]
 May  1 13:08:56 compute-0-21 kernel: [a025d80c] ? 
 nv_napi_poll+0x8c/0x610 [forcedeth]
 May  1 13:08:56 compute-0-21 kernel: [8105dd5c] ? 
 scheduler_tick+0xcc/0x260
 May  1 13:08:56 compute-0-21 kernel: [81460fb3] ? 
 net_rx_action+0x103/0x2f0
 May  1 13:08:56 compute-0-21 kernel: [8112eef2] ? 
 free_pcppages_bulk+0x392/0x460
 May  1 13:08:56 compute-0-21 kernel: [8107a8e1] ? 
 __do_softirq+0xc1/0x1e0
 May  1 13:08:56 compute-0-21 kernel: [810e6eb0] ? 
 handle_IRQ_event+0x60/0x170
 May  1 13:08:56 compute-0-21 kernel: [8100c30c] ? 
 call_softirq+0x1c/0x30
 May  1 13:08:56 compute-0-21 kernel: [8100fa75] ? 
 do_softirq+0x65/0xa0
 May  1 13:08:56 compute-0-21 kernel: [8107a795] ? 
 irq_exit+0x85/0x90
 May  1 13:08:56 compute-0-21 kernel: [81531605] ? 
 do_IRQ+0x75/0xf0
 May  1 13:08:56 compute-0-21 kernel: [8100b9d3] ? 
 ret_from_intr+0x0/0x11
 May  1 13:08:56 compute-0-21 kernel: EOI  [811bdd20] ? 
 submit_bh+0x60/0x1f0
 May  1 13:08:56 compute-0-21 kernel: [811c0598] ? 
 __block_write_full_page+0x1c8/0x330
 May  1 13:08:56 compute-0-21 kernel: [811bf560] ? 
 end_buffer_async_write+0x0/0x190
 May  1 13:08:56 compute-0-21 kernel: [811c07e0] ? 
 block_write_full_page_endio+0xe0/0x120
 May  1 13:08:56 compute-0-21 kernel: [a02c4b30] ? 
 buffer_unmapped+0x0/0x20 [ext3]
 May  1 13:08:56 compute-0-21 kernel: [811c0835] ? 
 block_write_full_page+0x15/0x20
 May  1 13:08:56 compute-0-21 kernel: [a02c56dd] ? 
 ext3_ordered_writepage+0x1ed/0x240 [ext3]
 May  1 13:08:56 compute-0-21 kernel: [811336c7] ? 
 __writepage+0x17/0x40
 May  1 13:08:56 compute-0-21 kernel: [8113498d] ? 
 write_cache_pages+0x1fd/0x4c0
 May  1 13:08:56 compute-0-21 kernel: [a0203e28] ? 
 __ext4_journal_stop+0x68/0xa0 [ext4]
 May  1 13:08:56 compute-0-21 kernel: [811336b0] ? 
 __writepage+0x0/0x40
 May  1 13:08:56 compute-0-21 kernel: [81134c74] ? 
 generic_writepages+0x24/0x30
 May  1 13:08:56 compute-0-21 kernel: [81134cb5] ? 
 do_writepages+0x35/0x40
 May  1 13:08:56 compute-0-21 kernel: [811b50cd] ? 
 writeback_single_inode+0xdd/0x290
 May  1 13:08:56 compute-0-21 kernel: [811b54cd] ? 
 writeback_sb_inodes+0xbd/0x170
 May  1 13:08:56 compute-0-21 kernel: [811b562b] ? 
 writeback_inodes_wb+0xab/0x1b0
 May  1 13:08:56 compute-0-21 kernel: