> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Monday, July 23, 2012 11:14 AM > To: rsyslog-users > Subject: Re: [rsyslog] Messages lost and AWS Security Group > > On Mon, 23 Jul 2012, Xavier Fustero wrote: > > > Hi, > > > > first I want to clarify that my initial issue was fixed adding > another > > queue for the second templates. I read queue stuff quickly so I > missed you > > need a queue for every action. Thanks Rainer. > > > > Second, let's discuss this queue a little bit. I replied in-line. > > > > > > On 20 July 2012 18:15, <[email protected]> wrote: > > > >> On Fri, 20 Jul 2012, Rainer Gerhards wrote: > >> > >> $ActionQueueType LinkedList # use asynchronous processing > >>> $ActionQueueFileName srvrfwd # set file name, also enables disk > mode > >>> $ActionResumeRetryCount -1 # infinite retries on insert failure > >>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog > shuts > >>> > >> > >> A question on this. > >> > >> do you really need to use linkedlist to get async processing? or is > it > >> just the fact that you have defined a queue type for the action? > >> > >> > > To be honest I didn't pay attention on the different type of queues. > I > > tested it quickly to check if the queue system worked or not (it > did). My > > scenario is clients that uses two templates to send logs to a server. > I > > want having a queue in front of each template to avoid losing any > message > > in case something breaks in between ( I am using RELP as well). I am > not > > using queue to send data to a database. > > > > After reading the doc more carefully, I read that In-memory queues > don't > > write in disk. It says they don't survice any program abort. As you > know, I > > am using Linkedlist mode (I didn't know it has process overheat). I > did > > some tests writing a loop to send 1000 messages and aborting the > > communication in between by firewall, aborting rsyslog, stunnel in > the > > server to check that messages are delivered at the end. My surprise > was to > > see that my client writes messages in a file while it can't send them > to > > the server. > > > > I am getting confused as I understood In-memory won't write anything > in > > disk but I see it creates a file with all messages queued in the > working > > directory. I though that will do a Disk-assisted queue. Can someone > clarify > > the difference between them? > > > > > > > >> I've seen linkedlist be significantly slower than the default memory > array > >> (especially if there are lots of messages queued), so I'm wondering > if this > >> is an option. > >> > > > > In case I choose a In-memory queue, do you suggest me to go for the > > FixedArray? Is the only problem with this type that we can lose > message in > > case we accumulate more than N pending messages? > > My question was primarily directed at Rainer, I don't know the answers. > > My understanding is that both FixedArray and LinkedList are in-memory > queues, it's only if you add the ActionQueueFileName tag that it shifts > to > being a disk assited queue and overflowing to disk.
Yes, specifying a disk file name also triggers DA mode (why else would you specify a file name...?) > My understanding is that even with LinkedList, you will first allocate > the > max memory for the message queue in RAM before you start putting extra > messages on disk. Yes, that's how it works. I think there is a watermark (97%?) where disk writing starts, but that's a detail (see queue doc if it matters). > If my understanding is correct, you should be able to use FixedArray > for a > disk assisted queue as well. Linked list vs. Fixedarray does not matter in regard to whether the queue is disk assisted or not. In theory, the difference between the two modes is extremely small (FixedArray saves a malloc/free per message), but in practice it may matter. > note that once you start overflowing to disk, your performance drops > significantly, so in that mode, the performance difference between > FixedArray and LinkedList no longer matters. Yes. DA mode is made to cover outages. It will NOT solve a situation where the system is permanently unable to process the messages or handle bursts. The core can probably handle 1000 in-memory messages in the same time a single disk queue message is handled. Note that recent builds run disk and in memory queue in parallel (parallel dequeue and processing), which somewhat reduces the overhead, but still...). HTH Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
