[rsyslog] omfile - create file path hierarchy splitting hostname

[Luigi Tagliamonte via rsyslog]

Hello group,
my current server hostname looks like
---.
Where environment can be: stg,stg2.. etc.; serverType: api,worker.. etc and
server ID is a unique ID.

I'm trying to configure a receiver rsyslog server that is automatically
able to split the hostname and write the log file in a hierarchy structure:

/
< serverType >/
/
- logFile

I read a couple of blog/docs and the only thing I was able to find is a
multi-config setup like:

if ($fromhost contains ’something′ ) then {}
if ($fromhost contains ’someting′) then {}

Since my environment is dynamic and   and  get
created/destroyed I was looking for a way to split the hostname on the fly
and implement the hierarchy structure illustrated before.
Is there a way to do it? Can somebody point me to a doc/example?
Thank you in advance for all the help!
Regards
L.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] liblognorm 2.0.4 released

[Florian Riedl]

Hi all,

today we released liblognorm 2.0.4. This new version mainly provides
new parser support options, like different JSON number formats and
unix timestamps. See the Changelog for details.

Version 2.0.4, 2017-10-04

added support for native JSON number formats supported by parsers:
number, float, hex
added support for creating unix timestamps supported by parsers:
date-rfc3164, date-rfc5424
fixed build problems on Solaris
… but there still seem to be some code issues, manifested in testbench
failures. So use with care!

Download:
http://www.liblognorm.com/download/liblognorm-2-0-4/

As always, feedback is appreciated.

Best regards,
Florian Riedl
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[David Lang]

one last datapoint on this topic. If you are using spinning rust drives, they 
are limited to 1 fsync/rev, so a 5400 rpm drive has a theoretical limit of 90 
fsyncs/sec


If you have multiple fsyncs per log (file, directory, mark completed, etc), it's 
very easy to get down to the low tens of logs/sec. The original syslog daemon 
did one fsync per log (it didn't do the directory, so there was still risk of 
loss), which is why it had a reputation of being way to slow to deal with 
anything important.


SSDs are faster, but some SSDs lie to the system about the write being safe.

battery backed RAID controllers or battery backed ram cards are the fastest way 
to get something in storage that will survive a system crash.


that PCI SSD I mentioned was able to support rsyslog processing a few thousand 
logs/sec (depending on the filesystem in use, it ranged from ~1k to ~10k 
logs/sec)


reliable storage is _hard_, this is why database writes are so incredibly slow 
compared to writing to a file, it's not (primarily) SQL overhead or complexities 
in the database, it's all the steps that the database takes to give you the ACID 
data protection guarantees. All the nosql databases that are so much faster than 
traditional databases relax one or more of the ACID guarantees.


David Lang


 On Wed, 4 Oct 2017, David Lang wrote:


Date: Wed, 4 Oct 2017 01:23:35 -0700 (PDT)
From: David Lang 
Reply-To: rsyslog-users 
To: mostolog--- via rsyslog 
Subject: Re: [rsyslog] Does imfile-to-relp need DA queues?

On Wed, 4 Oct 2017, mostolog--- via rsyslog wrote:

About this: is there any option/planned feature to ensure message has been 
delivered before marking it as processed?


that is what is done, the problem is that it's marked as processed on the 
queue. If the memory is lost (crash, kill -9), the status and even existance 
of messages on the queues are lost.


rsyslog does:

source --> queue --> action (where --> can be a single message or a batch of 
messages)


unless the queue mode is direct, the source and action are asynchronous from 
each other.


If you want each message processed individually, everything slows to a crawl.

With a disk based queue (and some very paranoid settings) you can force each 
message read from the source to be committed to disk and fsync called (on the 
file and the directory it's in) before it's marked as read on the input. And 
then the message can be read from the disk and processed to the output (such 
as RELP) before it's marked as delivered on disk.


This is safe, but as I said, it resulted in a ~1000x slowdown several years 
ago (and disks are not much faster than that $5000 PCIe SSD I was using 
then), there are just a huge amount of system calls and disk I/O to do (and 
don't use ext3, it resulted in a futher slowdown, it doesn't do fsync well)


By default, rsyslog is not that paranoid, keeping logs in RAM and flushing 
them to disk at shutdown is considered pretty severe protection of the logs. 
Normally we don't do fsync calls when writing things to disk, we assume that 
the system isn't going to crash before the OS writes things.


Rsyslog isn't intended to handle things like financial transactions that must 
not be lost at any cost. It's designed to handle system and application logs 
in a best effort fashion (that best effort is pretty darn good, but we don't 
insist on perfection)


Does the application that writes the logs that you are intending to read do a 
fsync on the log file and the directory that it's in every time it writes a 
log to the file? if not, you are at risk of loosing logs in a system crash 
just from that. when it creates a new file does it check if it needs to sync 
the parent directory of the directory it's in (in case adding the file cause 
the directory to grow larger)? And are you writing this to a RAID array? Does 
your controller card buffer any writes? Does it have a battery backup?


It's very easy to say that the requirement is zero logs lost, but when you 
really dive into the details of what that requires, it gets very messy (and 
expensive)


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[David Lang]

by the way, there was a RELP related queue fix in the last week or so, check the 
git logs. It was marked as fixed just before you started posting your issue, so 
I don't know if they are at all related.


David Lang

 On Wed, 4 Oct 2017, mosto...@gmail.com 
wrote:



Date: Wed, 4 Oct 2017 10:33:36 +0200
From: "mosto...@gmail.com" 
To: David Lang ,
mostolog--- via rsyslog 
Subject: Re: [rsyslog] Does imfile-to-relp need DA queues?

Clear as water.

The idea which was in our minds due to rsyslog issues with RELP (mention on 
another thread) is, by these and Rainer words completely banned since now.


Thanks a lot


On 04/10/17 10:23, David Lang wrote:

On Wed, 4 Oct 2017, mostolog--- via rsyslog wrote:

About this: is there any option/planned feature to ensure message has been 
delivered before marking it as processed?


that is what is done, the problem is that it's marked as processed on the 
queue. If the memory is lost (crash, kill -9), the status and even 
existance of messages on the queues are lost.


rsyslog does:

source --> queue --> action (where --> can be a single message or a batch 
of messages)


unless the queue mode is direct, the source and action are asynchronous 
from each other.


If you want each message processed individually, everything slows to a 
crawl.


With a disk based queue (and some very paranoid settings) you can force 
each message read from the source to be committed to disk and fsync called 
(on the file and the directory it's in) before it's marked as read on the 
input. And then the message can be read from the disk and processed to the 
output (such as RELP) before it's marked as delivered on disk.


This is safe, but as I said, it resulted in a ~1000x slowdown several years 
ago (and disks are not much faster than that $5000 PCIe SSD I was using 
then), there are just a huge amount of system calls and disk I/O to do (and 
don't use ext3, it resulted in a futher slowdown, it doesn't do fsync well)


By default, rsyslog is not that paranoid, keeping logs in RAM and flushing 
them to disk at shutdown is considered pretty severe protection of the 
logs. Normally we don't do fsync calls when writing things to disk, we 
assume that the system isn't going to crash before the OS writes things.


Rsyslog isn't intended to handle things like financial transactions that 
must not be lost at any cost. It's designed to handle system and 
application logs in a best effort fashion (that best effort is pretty darn 
good, but we don't insist on perfection)


Does the application that writes the logs that you are intending to read do 
a fsync on the log file and the directory that it's in every time it writes 
a log to the file? if not, you are at risk of loosing logs in a system 
crash just from that. when it creates a new file does it check if it needs 
to sync the parent directory of the directory it's in (in case adding the 
file cause the directory to grow larger)? And are you writing this to a 
RAID array? Does your controller card buffer any writes? Does it have a 
battery backup?


It's very easy to say that the requirement is zero logs lost, but when you 
really dive into the details of what that requires, it gets very messy (and 
expensive)


David Lang




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

Clear as water.

The idea which was in our minds due to rsyslog issues with RELP (mention 
on another thread) is, by these and Rainer words completely banned since 
now.


Thanks a lot


On 04/10/17 10:23, David Lang wrote:

On Wed, 4 Oct 2017, mostolog--- via rsyslog wrote:

About this: is there any option/planned feature to ensure message has 
been delivered before marking it as processed?


that is what is done, the problem is that it's marked as processed on 
the queue. If the memory is lost (crash, kill -9), the status and even 
existance of messages on the queues are lost.


rsyslog does:

source --> queue --> action (where --> can be a single message or a 
batch of messages)


unless the queue mode is direct, the source and action are 
asynchronous from each other.


If you want each message processed individually, everything slows to a 
crawl.


With a disk based queue (and some very paranoid settings) you can 
force each message read from the source to be committed to disk and 
fsync called (on the file and the directory it's in) before it's 
marked as read on the input. And then the message can be read from the 
disk and processed to the output (such as RELP) before it's marked as 
delivered on disk.


This is safe, but as I said, it resulted in a ~1000x slowdown several 
years ago (and disks are not much faster than that $5000 PCIe SSD I 
was using then), there are just a huge amount of system calls and disk 
I/O to do (and don't use ext3, it resulted in a futher slowdown, it 
doesn't do fsync well)


By default, rsyslog is not that paranoid, keeping logs in RAM and 
flushing them to disk at shutdown is considered pretty severe 
protection of the logs. Normally we don't do fsync calls when writing 
things to disk, we assume that the system isn't going to crash before 
the OS writes things.


Rsyslog isn't intended to handle things like financial transactions 
that must not be lost at any cost. It's designed to handle system and 
application logs in a best effort fashion (that best effort is pretty 
darn good, but we don't insist on perfection)


Does the application that writes the logs that you are intending to 
read do a fsync on the log file and the directory that it's in every 
time it writes a log to the file? if not, you are at risk of loosing 
logs in a system crash just from that. when it creates a new file does 
it check if it needs to sync the parent directory of the directory 
it's in (in case adding the file cause the directory to grow larger)? 
And are you writing this to a RAID array? Does your controller card 
buffer any writes? Does it have a battery backup?


It's very easy to say that the requirement is zero logs lost, but when 
you really dive into the details of what that requires, it gets very 
messy (and expensive)


David Lang


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[David Lang]

On Wed, 4 Oct 2017, mostolog--- via rsyslog wrote:

About this: is there any option/planned feature to ensure message has 
been delivered before marking it as processed?


that is what is done, the problem is that it's marked as processed on the queue. 
If the memory is lost (crash, kill -9), the status and even existance of 
messages on the queues are lost.


rsyslog does:

source --> queue --> action (where --> can be a single message or a batch of 
messages)


unless the queue mode is direct, the source and action are asynchronous from 
each other.


If you want each message processed individually, everything slows to a crawl.

With a disk based queue (and some very paranoid settings) you can force each 
message read from the source to be committed to disk and fsync called (on the 
file and the directory it's in) before it's marked as read on the input. And 
then the message can be read from the disk and processed to the output (such as 
RELP) before it's marked as delivered on disk.


This is safe, but as I said, it resulted in a ~1000x slowdown several years ago 
(and disks are not much faster than that $5000 PCIe SSD I was using then), there 
are just a huge amount of system calls and disk I/O to do (and don't use ext3, 
it resulted in a futher slowdown, it doesn't do fsync well)


By default, rsyslog is not that paranoid, keeping logs in RAM and flushing them 
to disk at shutdown is considered pretty severe protection of the logs. Normally 
we don't do fsync calls when writing things to disk, we assume that the system 
isn't going to crash before the OS writes things.


Rsyslog isn't intended to handle things like financial transactions that must 
not be lost at any cost. It's designed to handle system and application logs in 
a best effort fashion (that best effort is pretty darn good, but we don't insist 
on perfection)


Does the application that writes the logs that you are intending to read do a 
fsync on the log file and the directory that it's in every time it writes a log 
to the file? if not, you are at risk of loosing logs in a system crash just from 
that. when it creates a new file does it check if it needs to sync the parent 
directory of the directory it's in (in case adding the file cause the directory 
to grow larger)? And are you writing this to a RAID array? Does your controller 
card buffer any writes? Does it have a battery backup?


It's very easy to say that the requirement is zero logs lost, but when you 
really dive into the details of what that requires, it gets very messy (and 
expensive)


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[Rainer Gerhards]

You either need to use all direct queues or do e very considerate
architecture change to rsyslog core (a couple of month worth of work,
I think). But in the end, the architecture change would do exactly
what queue direct mode does, so I do not even see any argument in
favor of it.

Rainer

2017-10-04 9:43 GMT+02:00 mosto...@gmail.com :
> As Kafka is a specific platform, which we wouldn't like "clients" to use, we
> were looking for something like this.
>
> source reads file from current position
> source sends to rsyslog (relp is standarized)
> rsyslogs doesn't ack and forwards to kafka
> kafka acks the message once properly saved
> rsyslogs ack's to source the message
> source current position++
> repeat from #1
>
> Of course, kafka could be replaced with any other reliable target
>
> I don't know if this makes sense for you.
>
>
>
> On 04/10/17 09:36, Rainer Gerhards wrote:
>
> 2017-10-04 9:34 GMT+02:00 mosto...@gmail.com :
>
> I was trying to make use of kafka redundancy
>
> As Kafka implements a safe mechanisms (copying the message into multiple
> nodes), I was wondering if theres a relayed-delivery mode
> ie: rsyslog acks the message once kafka has ack'ed the message.
>
> omkafka does this, but to do so it maintains a separate file of
> yet-unacked messages.
>
> Rainer
>
>
>
> On 04/10/17 09:31, Rainer Gerhards wrote:
>
> 2017-10-04 9:29 GMT+02:00 mostolog--- via rsyslog
> :
>
> Thanks Rainer and David
>
> About this: is there any option/planned feature to ensure message has
> been
> delivered before marking it as processed?
>
> eg: tell source message has been acknowledged when target has stored the
> event
>  source--->relp--->rsyslog--->kafka
>
> This would be "as easy as" not moving the reading cursor from files for
> imfile status. (devil in details)
>
> as David said, you can achieve this by using the "direct" queue mode.
> Of course, performance is bad, but that's always the case if you go to
> half-duplex operations.
>
> HTH
> Rainer
>
>
>
>
> On 03/10/17 19:38, David Lang wrote:
>
> and if the system crashes, or issues a kill -9 to rsyslog, there will
> not
> be a chance to save the memory queue and some messages will be lost.
>
> you would have to use a disk queue, or disable queues entirely (direct
> mode) to avoid any chance of log loss. This will cripple your
> performance
> (In a test several years ago, with a high-end PCI SSD, this resulted in
> a
> 1000x slowdown)
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> LIKE THAT.
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[mostolog--- via rsyslog]

No matter if we configure it differently, this parameter is "updated" 
internally.


stats shows that it's stuck at fullDelay and nothing changes from that 
point.


debug doesn't seem to show anything useful.

considering this is our scenario:

    source--->relp--->us--->kafka

which log would you find useful? all? stats? debug?




On 04/10/17 09:38, Rainer Gerhards wrote:

I'll update the doc, but basically this is so that we do not high
highwatermark and leave room for non-delayable messages. You can of
course configure it differently. Nevertheless, the queue should empty,
so this shall not cause anything to stay around...

Rainer

2017-10-03 10:52 GMT+02:00 mosto...@gmail.com :

Full delay mark is set t0 50% of highwatermark, without apparent/documented
reason

https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373


On 03/10/17 10:50, Rainer Gerhards wrote:

What exactly do you mean?

Rainer

Sent from phone, thus brief.

Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog"
:

Hmm...wondering if theres is a bug here as:

main_queue(
 queue.filename="main"
 queue.maxdiskspace="4G"
 queue.saveonshutdown="on"
 queue.lowwatermark="3"
 queue.highwatermark="6"
 queue.size="10"
 queue.type="LinkedList"
 queue.fulldelaymark="9"
 queue.lightdelaymark="75000"
 #queue.discardmark="9750"
 #queue.discardseverity="8"
)

results in:

8544.970496200:main thread: main Q: queue.c: params: type 1, enq-only
0, disk assisted 1, spoolDir '/data', maxFileSz 16777216, maxQSize 10,
lqsize 0, pqsize 0, child 0, full delay 3, light delay 75000, deq batch
size 1024, high wtrmrk 6, low wtrmrk 3, discardmrk 98000, max wrkr
1, min msgs f. wrkr 10

8544.970615922:main thread: main Q[DA]: queue.c: params: type 2,
enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz 16777216, maxQSize
0, lqsize 0, pqsize 0, child 1, full delay -1, light delay -1, deq batch
size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f.
wrkr 0




On 02/10/17 11:00, mosto...@gmail.com wrote:

Hi

Some time ago we reported issues with RELP /hanging/ and not forwarding
messages until restart.

Last week we were able to replicate and still figuring out *how rsyslog
delay system works*.

According to documentation,

 /queue.fulldelaymark//
 //Number of messages when the queue should block delayable
 messages. Messages are NO LONGER PROCESSED until the queue has
 sufficient space again. If a message is delayable depends on the
 input. For example, messages received via imtcp are delayable
 (because TCP can push back), but those received via imudp are not
 (as UDP does not permit a push back). The intent behind this
 setting is to leave some space in an almost-full queue for
 non-delayable messages, which would be lost if the queue runs out
 of space. Please note that if you use a DA queue, setting the
 fulldelaymark BELOW the highwatermark makes the queue never
 activate disk mode for delayable inputs. So this is probably not
 what you want//.

 queue.lightdelaymark number./

It's our understanding that

   * TCP ack's would be delayed _/a bit/_ if queue.size=lightdelaymark.
   * TCP ack's would be nack's if queue.size=fulldelaymark//.
 //

*Is that correct? Doesit behaves as low/highwatermark instead?* (full
delay when size=fulldelaymark, stops delaying when size is back to
lightdelaymark)


The following configuration:

main_queue(
 queue.filename="main"
 queue.maxdiskspace="1G"
 queue.saveonshutdown="on"
 queue.lowwatermark="15"
 queue.highwatermark="18"
 queue.size="20"
 queue.type="LinkedList"
 queue.fulldelaymark="-1"
 queue.lightdelaymark="-1"
)

Resulted in this debug trace:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main thread:
main Q: queue.c: params: type 1, enq-only 0, disk assisted 1, spoolDir
'/data', maxFileSz 16777216, maxQSize 20, lqsize 0, pqsize 0, child 0,
*full delay 9, light delay 14*, deq batch size 1024, high wtrmrk
18, low wtrmrk 15, discardmrk 196000, max wrkr 1, min msgs f. wrkr
20

*Why full is lower than light? *

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main thread:
main Q[DA]: queue.c: params: type 2, enq-only 0, disk assisted 0, spoolDir
'/data', maxFileSz 16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, full
delay -1, light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1,
discardmrk 0, max wrkr 1, min msgs f. wrkr 0


This other configuration,

 queue.filename="relp"
 queue.maxdiskspace="1G"
 queue.saveonshutdown="on"
 queue.lowwatermark="10"
 queue.highwatermark="60"
 queue.size="100"
 queue.type="LinkedList"
 queue.fulldelaymark="500"
 

Re: [rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

As Kafka is a specific platform, which we wouldn't like "clients" to 
use, we were looking for something like this.


1. source reads file from current position
2. source sends to rsyslog *(relp is standarized)*
3. rsyslogs doesn't ack and forwards to kafka
4. kafka acks the message once properly saved
5. rsyslogs ack's to source the message
6. source current position++
7. repeat from #1

Of course, kafka could be replaced with any other reliable target

I don't know if this makes sense for you.


On 04/10/17 09:36, Rainer Gerhards wrote:

2017-10-04 9:34 GMT+02:00 mosto...@gmail.com :

I was trying to make use of kafka redundancy

As Kafka implements a safe mechanisms (copying the message into multiple
nodes), I was wondering if theres a relayed-delivery mode
ie: rsyslog acks the message once kafka has ack'ed the message.

omkafka does this, but to do so it maintains a separate file of
yet-unacked messages.

Rainer




On 04/10/17 09:31, Rainer Gerhards wrote:

2017-10-04 9:29 GMT+02:00 mostolog--- via rsyslog
:

Thanks Rainer and David

About this: is there any option/planned feature to ensure message has
been
delivered before marking it as processed?

eg: tell source message has been acknowledged when target has stored the
event
  source--->relp--->rsyslog--->kafka

This would be "as easy as" not moving the reading cursor from files for
imfile status. (devil in details)

as David said, you can achieve this by using the "direct" queue mode.
Of course, performance is bad, but that's always the case if you go to
half-duplex operations.

HTH
Rainer





On 03/10/17 19:38, David Lang wrote:

and if the system crashes, or issues a kill -9 to rsyslog, there will
not
be a chance to save the memory queue and some messages will be lost.

you would have to use a disk queue, or disable queues entirely (direct
mode) to avoid any chance of log loss. This will cripple your
performance
(In a test several years ago, with a high-end PCI SSD, this resulted in
a
1000x slowdown)

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T
LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[Rainer Gerhards]

2017-10-04 9:34 GMT+02:00 mosto...@gmail.com :
> Do you want me to open an issue?

Not sure if this is helpful or not. We need to find out what causes
the trouble for you. Either here or on github is equally fine with me.
But it is not yet confirmed there is a bug, so an issue tracker might
not make that much sense...

The question is why relp hangs. pstats might help clear up the vision
a bit more.

Rainer
>
>
> On 03/10/17 10:56, mosto...@gmail.com wrote:
>
> Even more...assuming that a full queue will trigger full delays...when it
> would be this delay disabled again?...Cause in our scenario, seems the
> process is stuck forever, until a rsyslog restart.
>
>
>
> On 03/10/17 10:52, mosto...@gmail.com wrote:
>
> Full delay mark is set t0 50% of highwatermark, without apparent/documented
> reason
>
> https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373
>
>
> On 03/10/17 10:50, Rainer Gerhards wrote:
>
> What exactly do you mean?
>
> Rainer
>
> Sent from phone, thus brief.
>
> Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog"
> :
>>
>> Hmm...wondering if theres is a bug here as:
>>
>> main_queue(
>> queue.filename="main"
>> queue.maxdiskspace="4G"
>> queue.saveonshutdown="on"
>> queue.lowwatermark="3"
>> queue.highwatermark="6"
>> queue.size="10"
>> queue.type="LinkedList"
>> queue.fulldelaymark="9"
>> queue.lightdelaymark="75000"
>> #queue.discardmark="9750"
>> #queue.discardseverity="8"
>> )
>>
>> results in:
>>
>> 8544.970496200:main thread: main Q: queue.c: params: type 1, enq-only
>> 0, disk assisted 1, spoolDir '/data', maxFileSz 16777216, maxQSize 10,
>> lqsize 0, pqsize 0, child 0, full delay 3, light delay 75000, deq batch
>> size 1024, high wtrmrk 6, low wtrmrk 3, discardmrk 98000, max wrkr
>> 1, min msgs f. wrkr 10
>>
>> 8544.970615922:main thread: main Q[DA]: queue.c: params: type 2,
>> enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz 16777216, maxQSize
>> 0, lqsize 0, pqsize 0, child 1, full delay -1, light delay -1, deq batch
>> size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f.
>> wrkr 0
>>
>>
>>
>>
>> On 02/10/17 11:00, mosto...@gmail.com wrote:
>>>
>>> Hi
>>>
>>> Some time ago we reported issues with RELP /hanging/ and not forwarding
>>> messages until restart.
>>>
>>> Last week we were able to replicate and still figuring out *how rsyslog
>>> delay system works*.
>>>
>>> According to documentation,
>>>
>>> /queue.fulldelaymark//
>>> //Number of messages when the queue should block delayable
>>> messages. Messages are NO LONGER PROCESSED until the queue has
>>> sufficient space again. If a message is delayable depends on the
>>> input. For example, messages received via imtcp are delayable
>>> (because TCP can push back), but those received via imudp are not
>>> (as UDP does not permit a push back). The intent behind this
>>> setting is to leave some space in an almost-full queue for
>>> non-delayable messages, which would be lost if the queue runs out
>>> of space. Please note that if you use a DA queue, setting the
>>> fulldelaymark BELOW the highwatermark makes the queue never
>>> activate disk mode for delayable inputs. So this is probably not
>>> what you want//.
>>>
>>> queue.lightdelaymark number./
>>>
>>> It's our understanding that
>>>
>>>   * TCP ack's would be delayed _/a bit/_ if queue.size=lightdelaymark.
>>>   * TCP ack's would be nack's if queue.size=fulldelaymark//.
>>> //
>>>
>>> *Is that correct? Doesit behaves as low/highwatermark instead?* (full
>>> delay when size=fulldelaymark, stops delaying when size is back to
>>> lightdelaymark)
>>>
>>>
>>> The following configuration:
>>>
>>> main_queue(
>>> queue.filename="main"
>>> queue.maxdiskspace="1G"
>>> queue.saveonshutdown="on"
>>> queue.lowwatermark="15"
>>> queue.highwatermark="18"
>>> queue.size="20"
>>> queue.type="LinkedList"
>>> queue.fulldelaymark="-1"
>>> queue.lightdelaymark="-1"
>>> )
>>>
>>> Resulted in this debug trace:
>>>
>>> Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main thread:
>>> main Q: queue.c: params: type 1, enq-only 0, disk assisted 1, spoolDir
>>> '/data', maxFileSz 16777216, maxQSize 20, lqsize 0, pqsize 0, child 0,
>>> *full delay 9, light delay 14*, deq batch size 1024, high wtrmrk
>>> 18, low wtrmrk 15, discardmrk 196000, max wrkr 1, min msgs f. wrkr
>>> 20
>>>
>>> *Why full is lower than light? *
>>>
>>> Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main thread:
>>> main Q[DA]: queue.c: params: type 2, enq-only 0, disk assisted 0, spoolDir
>>> '/data', maxFileSz 16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, full
>>> delay -1, light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1,
>>> discardmrk 0, max 

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[Rainer Gerhards]

I'll update the doc, but basically this is so that we do not high
highwatermark and leave room for non-delayable messages. You can of
course configure it differently. Nevertheless, the queue should empty,
so this shall not cause anything to stay around...

Rainer

2017-10-03 10:52 GMT+02:00 mosto...@gmail.com :
> Full delay mark is set t0 50% of highwatermark, without apparent/documented
> reason
>
> https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373
>
>
> On 03/10/17 10:50, Rainer Gerhards wrote:
>
> What exactly do you mean?
>
> Rainer
>
> Sent from phone, thus brief.
>
> Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog"
> :
>>
>> Hmm...wondering if theres is a bug here as:
>>
>> main_queue(
>> queue.filename="main"
>> queue.maxdiskspace="4G"
>> queue.saveonshutdown="on"
>> queue.lowwatermark="3"
>> queue.highwatermark="6"
>> queue.size="10"
>> queue.type="LinkedList"
>> queue.fulldelaymark="9"
>> queue.lightdelaymark="75000"
>> #queue.discardmark="9750"
>> #queue.discardseverity="8"
>> )
>>
>> results in:
>>
>> 8544.970496200:main thread: main Q: queue.c: params: type 1, enq-only
>> 0, disk assisted 1, spoolDir '/data', maxFileSz 16777216, maxQSize 10,
>> lqsize 0, pqsize 0, child 0, full delay 3, light delay 75000, deq batch
>> size 1024, high wtrmrk 6, low wtrmrk 3, discardmrk 98000, max wrkr
>> 1, min msgs f. wrkr 10
>>
>> 8544.970615922:main thread: main Q[DA]: queue.c: params: type 2,
>> enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz 16777216, maxQSize
>> 0, lqsize 0, pqsize 0, child 1, full delay -1, light delay -1, deq batch
>> size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f.
>> wrkr 0
>>
>>
>>
>>
>> On 02/10/17 11:00, mosto...@gmail.com wrote:
>>>
>>> Hi
>>>
>>> Some time ago we reported issues with RELP /hanging/ and not forwarding
>>> messages until restart.
>>>
>>> Last week we were able to replicate and still figuring out *how rsyslog
>>> delay system works*.
>>>
>>> According to documentation,
>>>
>>> /queue.fulldelaymark//
>>> //Number of messages when the queue should block delayable
>>> messages. Messages are NO LONGER PROCESSED until the queue has
>>> sufficient space again. If a message is delayable depends on the
>>> input. For example, messages received via imtcp are delayable
>>> (because TCP can push back), but those received via imudp are not
>>> (as UDP does not permit a push back). The intent behind this
>>> setting is to leave some space in an almost-full queue for
>>> non-delayable messages, which would be lost if the queue runs out
>>> of space. Please note that if you use a DA queue, setting the
>>> fulldelaymark BELOW the highwatermark makes the queue never
>>> activate disk mode for delayable inputs. So this is probably not
>>> what you want//.
>>>
>>> queue.lightdelaymark number./
>>>
>>> It's our understanding that
>>>
>>>   * TCP ack's would be delayed _/a bit/_ if queue.size=lightdelaymark.
>>>   * TCP ack's would be nack's if queue.size=fulldelaymark//.
>>> //
>>>
>>> *Is that correct? Doesit behaves as low/highwatermark instead?* (full
>>> delay when size=fulldelaymark, stops delaying when size is back to
>>> lightdelaymark)
>>>
>>>
>>> The following configuration:
>>>
>>> main_queue(
>>> queue.filename="main"
>>> queue.maxdiskspace="1G"
>>> queue.saveonshutdown="on"
>>> queue.lowwatermark="15"
>>> queue.highwatermark="18"
>>> queue.size="20"
>>> queue.type="LinkedList"
>>> queue.fulldelaymark="-1"
>>> queue.lightdelaymark="-1"
>>> )
>>>
>>> Resulted in this debug trace:
>>>
>>> Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main thread:
>>> main Q: queue.c: params: type 1, enq-only 0, disk assisted 1, spoolDir
>>> '/data', maxFileSz 16777216, maxQSize 20, lqsize 0, pqsize 0, child 0,
>>> *full delay 9, light delay 14*, deq batch size 1024, high wtrmrk
>>> 18, low wtrmrk 15, discardmrk 196000, max wrkr 1, min msgs f. wrkr
>>> 20
>>>
>>> *Why full is lower than light? *
>>>
>>> Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main thread:
>>> main Q[DA]: queue.c: params: type 2, enq-only 0, disk assisted 0, spoolDir
>>> '/data', maxFileSz 16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, full
>>> delay -1, light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1,
>>> discardmrk 0, max wrkr 1, min msgs f. wrkr 0
>>>
>>>
>>> This other configuration,
>>>
>>> queue.filename="relp"
>>> queue.maxdiskspace="1G"
>>> queue.saveonshutdown="on"
>>> queue.lowwatermark="10"
>>> queue.highwatermark="60"
>>> queue.size="100"
>>> queue.type="LinkedList"
>>> queue.fulldelaymark="500"
>>> 

Re: [rsyslog] Does imfile-to-relp need DA queues?

[Rainer Gerhards]

2017-10-04 9:34 GMT+02:00 mosto...@gmail.com :
> I was trying to make use of kafka redundancy
>
> As Kafka implements a safe mechanisms (copying the message into multiple
> nodes), I was wondering if theres a relayed-delivery mode
> ie: rsyslog acks the message once kafka has ack'ed the message.

omkafka does this, but to do so it maintains a separate file of
yet-unacked messages.

Rainer
>
>
>
>
> On 04/10/17 09:31, Rainer Gerhards wrote:
>>
>> 2017-10-04 9:29 GMT+02:00 mostolog--- via rsyslog
>> :
>>>
>>> Thanks Rainer and David
>>>
>>> About this: is there any option/planned feature to ensure message has
>>> been
>>> delivered before marking it as processed?
>>>
>>> eg: tell source message has been acknowledged when target has stored the
>>> event
>>>  source--->relp--->rsyslog--->kafka
>>>
>>> This would be "as easy as" not moving the reading cursor from files for
>>> imfile status. (devil in details)
>>
>> as David said, you can achieve this by using the "direct" queue mode.
>> Of course, performance is bad, but that's always the case if you go to
>> half-duplex operations.
>>
>> HTH
>> Rainer
>>>
>>>
>>>
>>>
>>>
>>> On 03/10/17 19:38, David Lang wrote:

 and if the system crashes, or issues a kill -9 to rsyslog, there will
 not
 be a chance to save the memory queue and some messages will be lost.

 you would have to use a disk queue, or disable queues entirely (direct
 mode) to avoid any chance of log loss. This will cripple your
 performance
 (In a test several years ago, with a high-end PCI SSD, this resulted in
 a
 1000x slowdown)

 David Lang
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T
 LIKE THAT.
>>>
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[mostolog--- via rsyslog]

Do you want me to open an issue?


On 03/10/17 10:56, mosto...@gmail.com wrote:
Even more...assuming that a full queue will trigger full delays...when 
it would be this delay disabled again?...Cause in our scenario, seems 
the process is stuck forever, until a rsyslog restart.




On 03/10/17 10:52, mosto...@gmail.com wrote:


Full delay mark is set t0 50% of highwatermark, without 
apparent/documented reason


https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373


On 03/10/17 10:50, Rainer Gerhards wrote:

What exactly do you mean?

Rainer

Sent from phone, thus brief.

Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog" 
>:


Hmm...wondering if theres is a bug here as:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="4G"
    queue.saveonshutdown="on"
    queue.lowwatermark="3"
    queue.highwatermark="6"
    queue.size="10"
    queue.type="LinkedList"
    queue.fulldelaymark="9"
    queue.lightdelaymark="75000"
    #queue.discardmark="9750"
    #queue.discardseverity="8"
)

results in:

8544.970496200:main thread    : main Q: queue.c: params: type 1,
enq-only 0, disk assisted 1, spoolDir '/data', maxFileSz
16777216, maxQSize 10, lqsize 0, pqsize 0, child 0, full
delay 3, light delay 75000, deq batch size 1024, high wtrmrk
6, low wtrmrk 3, discardmrk 98000, max wrkr 1, min msgs
f. wrkr 10

8544.970615922:main thread    : main Q[DA]: queue.c: params:
type 2, enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz
16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, full delay
-1, light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk
1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0




On 02/10/17 11:00, mosto...@gmail.com
 wrote:

Hi

Some time ago we reported issues with RELP /hanging/ and not
forwarding messages until restart.

Last week we were able to replicate and still figuring out
*how rsyslog delay system works*.

According to documentation,

    /queue.fulldelaymark//
    //Number of messages when the queue should block delayable
    messages. Messages are NO LONGER PROCESSED until the
queue has
    sufficient space again. If a message is delayable
depends on the
    input. For example, messages received via imtcp are
delayable
    (because TCP can push back), but those received via
imudp are not
    (as UDP does not permit a push back). The intent behind this
    setting is to leave some space in an almost-full queue for
    non-delayable messages, which would be lost if the queue
runs out
    of space. Please note that if you use a DA queue,
setting the
    fulldelaymark BELOW the highwatermark makes the queue never
    activate disk mode for delayable inputs. So this is
probably not
    what you want//.

    queue.lightdelaymark number./

It's our understanding that

  * TCP ack's would be delayed _/a bit/_ if
queue.size=lightdelaymark.
  * TCP ack's would be nack's if queue.size=fulldelaymark//.
    //

*Is that correct? Doesit behaves as low/highwatermark
instead?* (full delay when size=fulldelaymark, stops
delaying when size is back to lightdelaymark)


The following configuration:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="1G"
    queue.saveonshutdown="on"
    queue.lowwatermark="15"
    queue.highwatermark="18"
    queue.size="20"
    queue.type="LinkedList"
    queue.fulldelaymark="-1"
    queue.lightdelaymark="-1"
)

Resulted in this debug trace:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main
thread    : main Q: queue.c: params: type 1, enq-only 0,
disk assisted 1, spoolDir '/data', maxFileSz 16777216,
maxQSize 20, lqsize 0, pqsize 0, child 0, *full delay
9, light delay 14*, deq batch size 1024, high wtrmrk
18, low wtrmrk 15, discardmrk 196000, max wrkr 1,
min msgs f. wrkr 20

*Why full is lower than light? *

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main
thread    : main Q[DA]: queue.c: params: type 2, enq-only 0,
disk assisted 0, spoolDir '/data', maxFileSz 16777216,
maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1,
light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk
1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0


This other configuration,

        queue.filename="relp"
    

Re: [rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

I was trying to make use of kafka redundancy

As Kafka implements a safe mechanisms (copying the message into multiple 
nodes), I was wondering if theres a relayed-delivery mode

ie: rsyslog acks the message once kafka has ack'ed the message.



On 04/10/17 09:31, Rainer Gerhards wrote:

2017-10-04 9:29 GMT+02:00 mostolog--- via rsyslog :

Thanks Rainer and David

About this: is there any option/planned feature to ensure message has been
delivered before marking it as processed?

eg: tell source message has been acknowledged when target has stored the
event
 source--->relp--->rsyslog--->kafka

This would be "as easy as" not moving the reading cursor from files for
imfile status. (devil in details)

as David said, you can achieve this by using the "direct" queue mode.
Of course, performance is bad, but that's always the case if you go to
half-duplex operations.

HTH
Rainer





On 03/10/17 19:38, David Lang wrote:

and if the system crashes, or issues a kill -9 to rsyslog, there will not
be a chance to save the memory queue and some messages will be lost.

you would have to use a disk queue, or disable queues entirely (direct
mode) to avoid any chance of log loss. This will cripple your performance
(In a test several years ago, with a high-end PCI SSD, this resulted in a
1000x slowdown)

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[Rainer Gerhards]

2017-10-04 9:29 GMT+02:00 mostolog--- via rsyslog :
> Thanks Rainer and David
>
> About this: is there any option/planned feature to ensure message has been
> delivered before marking it as processed?
>
> eg: tell source message has been acknowledged when target has stored the
> event
> source--->relp--->rsyslog--->kafka
>
> This would be "as easy as" not moving the reading cursor from files for
> imfile status. (devil in details)

as David said, you can achieve this by using the "direct" queue mode.
Of course, performance is bad, but that's always the case if you go to
half-duplex operations.

HTH
Rainer
>
>
>
>
>
> On 03/10/17 19:38, David Lang wrote:
>>
>> and if the system crashes, or issues a kill -9 to rsyslog, there will not
>> be a chance to save the memory queue and some messages will be lost.
>>
>> you would have to use a disk queue, or disable queues entirely (direct
>> mode) to avoid any chance of log loss. This will cripple your performance
>> (In a test several years ago, with a high-end PCI SSD, this resulted in a
>> 1000x slowdown)
>>
>> David Lang
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

Thanks Rainer and David

About this: is there any option/planned feature to ensure message has 
been delivered before marking it as processed?


eg: tell source message has been acknowledged when target has stored the 
event

    source--->relp--->rsyslog--->kafka

This would be "as easy as" not moving the reading cursor from files for 
imfile status. (devil in details)





On 03/10/17 19:38, David Lang wrote:
and if the system crashes, or issues a kill -9 to rsyslog, there will 
not be a chance to save the memory queue and some messages will be lost.


you would have to use a disk queue, or disable queues entirely (direct 
mode) to avoid any chance of log loss. This will cripple your 
performance (In a test several years ago, with a high-end PCI SSD, 
this resulted in a 1000x slowdown)


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.