2015-12-16 17:00 GMT+01:00 Ciprian Hacman <[email protected]>: > Not sure if you saw, but provided output from Valgrind with bout inotify > and polling. > To me it seems that in both cases the leak can be seen.
sorry, my fault. I mis-read one of your responses. It would probably still be a good idea to open a bug tracker and including the config in it. Rainer > > Thanks, > Ciprian > > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > On Wed, Dec 16, 2015 at 5:23 PM, Rainer Gerhards <[email protected]> > wrote: > >> 2015-12-16 16:21 GMT+01:00 Ciprian Hacman <[email protected]>: >> > Not sure how easy is to reproduce on your side. >> > If you need help narrowing the leak to a certain commit or release, I >> could >> > help. >> >> OK, I thought you could just switch the imfile mode. So I suggest to >> open a bug tracker, I guess I can look into it after the holiday >> period. >> >> Rainer >> > >> > Ciprian >> > >> > -- >> > Performance Monitoring * Log Analytics * Search Analytics >> > Solr & Elasticsearch Support * http://sematext.com/ >> > >> > On Wed, Dec 16, 2015 at 4:22 PM, Rainer Gerhards < >> [email protected]> >> > wrote: >> > >> >> 2015-12-16 15:10 GMT+01:00 Ciprian Hacman <[email protected] >> >: >> >> > Done. Can you check the gist again? >> >> >> >> yup. It now contains better info, but I wonder where these leaks stem >> >> from. One suspect is inotify polling mode, which we didn't really test >> >> well for quite a while... >> >> >> >> Rainer >> >> > >> >> > Thanks, >> >> > Ciprian >> >> > >> >> > -- >> >> > Performance Monitoring * Log Analytics * Search Analytics >> >> > Solr & Elasticsearch Support * http://sematext.com/ >> >> > >> >> > On Wed, Dec 16, 2015 at 3:40 PM, Rainer Gerhards < >> >> [email protected]> >> >> > wrote: >> >> > >> >> >> 2015-12-16 14:16 GMT+01:00 Ciprian Hacman < >> [email protected] >> >> >: >> >> >> > Done! I updated the output in the same place: >> >> >> > https://gist.github.com/hakman/44afddaf4eb67cda28c6 >> >> >> >> >> >> mhhh... less clear than I had hoped... >> >> >> >> >> >> Can you switch imfile to use inotify mode and retry? >> >> >> >> >> >> Rainer >> >> >> > >> >> >> > Thanks for looking into this so fast! >> >> >> > >> >> >> > Ciprian >> >> >> > >> >> >> > -- >> >> >> > Performance Monitoring * Log Analytics * Search Analytics >> >> >> > Solr & Elasticsearch Support * http://sematext.com/ >> >> >> > >> >> >> > On Wed, Dec 16, 2015 at 2:53 PM, Rainer Gerhards < >> >> >> [email protected]> >> >> >> > wrote: >> >> >> > >> >> >> >> 2015-12-16 13:48 GMT+01:00 Ciprian Hacman < >> >> [email protected] >> >> >> >: >> >> >> >> > Hi, >> >> >> >> > >> >> >> >> > I upgraded a server to Rsyslog 8.15 last night and today the >> >> process >> >> >> was >> >> >> >> > using almost 200MB of RAM (raising steadily). >> >> >> >> > Tried running the process in Valgrind to see if I get an idea >> about >> >> >> what >> >> >> >> > happens, but wasn't that much help for me. >> >> >> >> >> >> >> >> That's because debug symbols are unloaded at module unload time. >> This >> >> >> >> makes valgrind stacktraces unusable. Nevertheless, the information >> >> >> >> looks very promising. >> >> >> >> >> >> >> >> Can you build rsyslog yourself for that box? All we need is the >> >> >> >> --enable-valgrind option, which will essentially remove the module >> >> >> >> unloads and make the stacktrace usable. >> >> >> >> >> >> >> >> Rainer >> >> >> >> > >> >> >> >> > If someone has better debugging skills, I pasted the output >> here. >> >> Not >> >> >> >> sure >> >> >> >> > if I let it run enough or leave it running longer. >> >> >> >> > https://gist.github.com/hakman/44afddaf4eb67cda28c6 >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > Ciprian >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > Performance Monitoring * Log Analytics * Search Analytics >> >> >> >> > Solr & Elasticsearch Support * http://sematext.com/ >> >> >> >> > >> >> >> >> > On Tue, Dec 15, 2015 at 2:41 PM, David Lang <[email protected]> >> wrote: >> >> >> >> > >> >> >> >> >> On Tue, 15 Dec 2015, Ciprian Hacman wrote: >> >> >> >> >> >> >> >> >> >> Hi David, >> >> >> >> >>> >> >> >> >> >>> maxMessageSize="10000" >> >> >> >> >>> queue.size="10000" - main queue >> >> >> >> >>> queue.size="10000" - elasticsearch queue >> >> >> >> >>> >> >> >> >> >>> Based on my calculations this brings me to a max of 200MB of >> >> memory, >> >> >> >> maybe >> >> >> >> >>> a little more depending on how maxMessageSize is calculated. >> >> >> >> >>> >> >> >> >> >>> I read logs from a file and push them to Elasticsearch (on the >> >> same >> >> >> >> >>> network), so TCP is the only possibility. This server has a >> very >> >> >> simple >> >> >> >> >>> setup. >> >> >> >> >>> >> >> >> >> >>> If I don't find the reason for this issue, I might have to go >> >> >> implement >> >> >> >> >>> the >> >> >> >> >>> forwarding to a central location and push to Elasticsearch >> from >> >> >> there. >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> There are advantages to sending things to a central server. >> >> >> >> >> >> >> >> >> >> it's one place to queue data, so you can either allocate more >> ram, >> >> >> or go >> >> >> >> >> to disk as needed without impacting other workloads. >> >> >> >> >> >> >> >> >> >> it's more efficient, the central server is more likely to have >> >> larger >> >> >> >> >> batches of data to feed to ES, and ES only needs to be running >> one >> >> >> >> thread >> >> >> >> >> processing inbound data >> >> >> >> >> >> >> >> >> >> while it is one more point to have to look at, I think it >> >> simplifies >> >> >> >> >> troubleshooting as all the communication to ES (and therefor >> all >> >> the >> >> >> >> errors >> >> >> >> >> for such communication) happen in one place instead of >> >> distributed. >> >> >> >> >> >> >> >> >> >> anyway, let's see how things look with 8.15 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> David Lang >> >> >> >> >> _______________________________________________ >> >> >> >> >> rsyslog mailing list >> >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by a >> >> >> myriad >> >> >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST if >> >> >> you >> >> >> >> >> DON'T LIKE THAT. >> >> >> >> >> >> >> >> >> > _______________________________________________ >> >> >> >> > rsyslog mailing list >> >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> > http://www.rsyslog.com/professional-services/ >> >> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> a >> >> >> myriad >> >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >> you >> >> >> >> DON'T LIKE THAT. >> >> >> >> _______________________________________________ >> >> >> >> rsyslog mailing list >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >> >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >> you >> >> >> >> DON'T LIKE THAT. >> >> >> >> >> >> >> > _______________________________________________ >> >> >> > rsyslog mailing list >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> > http://www.rsyslog.com/professional-services/ >> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >> >> DON'T LIKE THAT. >> >> >> _______________________________________________ >> >> >> rsyslog mailing list >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >> >> DON'T LIKE THAT. >> >> >> >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com/professional-services/ >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
