2015-04-14 14:43 GMT+02:00 David Lang <da...@lang.hm>: > On Tue, 14 Apr 2015, chenlin rao wrote: > >> FYI: The pstats show `mainQ.enqueued: 1322, >> action_json2log-es1003.enqueued: 6684` with interval 60s. Not a so large >> flow. > > > > If i'm understanding what's happening from reading this thread, when the > foreach loop is processing a single message into a bunch of different action > calls, it is processing the different array elements as fast as it possibly > can, so for that short amount of wall clock time, the load is as high as it > is possible to be, and that's when the race condition is showing up. the > fact that you have large messages with lots of items in the array that you > are doing foreach over is giving room for the race condition to take place. > > It sounds like what is happening is that foreach is creating a message > object, calling the elasticsearch input, which is doing an async posting of > the messae into ES (due to bulk mode being on), but then the next loop of > foreach is modifying the object before the action is complete, so when the > module goes to access something, it hits garbage.
I am currently too busy with lognorm to step in here, but I think we should see whats the cause and cure. And then think if we have a design issue with "foreach" and evaluate how to carry on from there. Just my 2cts. Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
