I strongly recommend using 8.13 if you push logs to ES. There are many useful patches since 8.10 (some of them might not be in the changelog though).
Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder <micah.yo...@rackspace.com> wrote: > On 10/7/15 12:11 PM, David Lang wrote: > > It looks like json-c is not really thread-safe. specifically when using > > anything that outputs a json structure. Running into problems requires > > that you have multiple threads working on the same message, but if you > > have an action queue for your elasticsearch output, that would create > > the conditions that can trigger this. > > That may be part of it, thanks. I get this in dmesg on the staging > server (8.10): > > rs:action-app_e[45178] general protection ip:7fcc56c91633 > sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000] > rs:action-app_e[3213]: segfault at 0 ip 00007f538356e633 sp > 00007f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000] > > With the changes I made in the dev server (8.13), here are the dmesg > outputs: > > rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 00007f0d9a667f2c sp > 00007f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000] > rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 00007fccf3c67f2c sp > 00007fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000] > rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 00007f473c5e2f2c sp > 00007f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000] > rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 00007fa5a79c1f2c sp > 00007fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000] > rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 00007f53b11f73d8 error 14 > rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 00007f9fd4df63d8 error 14 > > No json-c there. > > Maybe we should just update to 8.13 in staging, putting my other changes > off for a bit. We do kind of need them though - currently messages are > being dropped, we think because the ElasticSearch actions are direct > queues instead of linked lists, and they are causing bottlenecks > elsewhere in the system. > > Will discuss with my team, thanks! > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
