from:"Randall Diffenderfer via rsyslog"

[rsyslog] correct way to redirect log messages to STDOUT

2020-08-26 Thread Randall Diffenderfer via rsyslog

working in a container env, the ask is to have a single rsyslog process 
"concentrate" logs from disparate processes and spit them out to STDOUT.

what's the *right way* to do this?

___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] handling oversized messages

2017-10-25 Thread Randall Diffenderfer via rsyslog

i am working with a backlevel version of rsyslogd, so i don't have any
hint of that in there.  oh well...

the remote endpoint is, for all intents and purposes, a black hole; it can
be any number of different SIEM or log transport systems, but the main
limiter is the "default" 8k barrier.  my json records can exceed that
limit without trying very hard.  i don't think i can unilaterally compress
them either...  that would probably make them incomprehensible to a
non-zero subset of my targets!

On 10/25/17, 12:12, "David Lang"  wrote:

>There was a recent config option to imfile to allow you to configure
>between 
>trucating the message and splitting the message to have more of it appear
>in 
>another message
>
>There is no way for rsyslog to combine messages once they have been
>split, it 
>processes messages one at a time.
>
>David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] handling oversized messages

2017-10-25 Thread Randall Diffenderfer via rsyslog

thanks for the followup.  i had come to a similar conclusion.  10# in a 5#
sack is gonna leave some stuff on the floor... :-)

On 10/25/17, 12:09, "David Lang" <da...@lang.hm> wrote:

>you can increase N so that messages you receive/read from file are intact
>within 
>rsyslog, but then you need to figure out what to do with them before you
>send 
>them
>
>you can write them to a local 'oversized' file and send some subset of
>the log 
>with a tag to say that this isn't the full message.
>
>you can try to figure a sane way to split them into 1/n messages and send
>them 
>as multiple messages (proabably through omprog and that program generates
>new 
>messages, but you can also do this in rsyslog with multiple templates and
>sending the 'same' message through the different templates to the same
>destination)
>
>there is no generic answer to the question of how do I put 10k of data
>into a 1k 
>message without loosing anything :-)
>
>David Lang
>
>  On Wed, 25 Oct 2017, Randall Diffenderfer via rsyslog wrote:
>
>> Date: Wed, 25 Oct 2017 18:48:52 +
>> From: Randall Diffenderfer via rsyslog <rsyslog@lists.adiscon.com>
>> To: Rainer Gerhards <rgerha...@hq.adiscon.com>,
>> rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Randall Diffenderfer <rdiffender...@proofpoint.com>
>> Subject: Re: [rsyslog] handling oversized messages
>> 
>> i suppose i deserved thatÂŠ :-)
>>
>> however, i have to interoperate with other folks who can't/won't
>>increase N ÂŠ
>>
>> so, i am pegged at their "N" in order to see no data corruption, as the
>>messages are structured, in that they are json, and busting them up will
>>be a bit problematicÂŠ
>>
>> From: Rainer Gerhards
>><rgerha...@hq.adiscon.com<mailto:rgerha...@hq.adiscon.com>>
>> Date: Wednesday, October 25, 2017 at 11:33
>> To: rsyslog-users
>><rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>
>> Cc: Randall Diffenderfer
>><rdiffender...@proofpoint.com<mailto:rdiffender...@proofpoint.com>>
>> Subject: Re: [rsyslog] handling oversized messages
>>
>> It may sound dumb, but: increase n! That's why this setting exists.
>>
>> Rainer
>>
>> Sent from phone, thus brief.
>>
>> Am 25.10.2017 19:48 schrieb "Randall Diffenderfer via rsyslog"
>><rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>:
>>
>> given the global setting of "maxmessagesize=N",  what is my recourse if
>>i
>> need to process a message > N in imfile?
>>
>> in other i/o modules?  it appears the message is truncated at ~N, and
>>not
>> split (which is what i thought i had seen in the past...)
>>
>> ___
>> rsyslog mailing list
>> 
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mai
>>lman_listinfo_rsyslog=DwID-g=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rX
>>jhg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=Tsk9ggiaNh8XQwMDW_NJS
>>gtFP5Gg9BJUwk_tROvEBGk=RZyuuUrNE1HDXRaBvHQSIeQQqjFf6UaevYcum2vnk6Q=>ttps://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mail
>>man_listinfo_rsyslog=DwMFaQ=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXj
>>hg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=0d7WauHwvHvrFWs0bcAWNN
>>xOtvaCSn7-izMA1oDTedw=Y2aZ4XynKrB4wgg_Y21Tuh5VisfzrJe4cTVcQkFfBow=>
>> 
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_profe
>>ssional-2Dservices_=DwID-g=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjh
>>g=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=Tsk9ggiaNh8XQwMDW_NJSgt
>>FP5Gg9BJUwk_tROvEBGk=GHF7DIY7y7puq6Xueai6-Fe7xPtjstJAzq7Ohp1TXt0=>ps://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professi
>>onal-2Dservices_=DwMFaQ=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg
>>=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=0d7WauHwvHvrFWs0bcAWNNxOtv
>>aCSn7-izMA1oDTedw=iK9krYQbWQ66DQXmXGuvorYG28Ioqr9mIroDehsBFLM=>
>> What's up with rsyslog? Follow
>>https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhard
>>s=DwID-g=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg=9sXs16LkSD1Mzh
>>oJxgyl7R_tait1dEXqvyD3NCT1wEA=Tsk9ggiaNh8XQwMDW_NJSgtFP5Gg9BJUwk_tROvEB
>>Gk=99ftjFUt5naxsc_8bAM_9TFPSrM6B8EpUR9vvnhyBkc=<https://urldefense.pr
>>oofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards=DwMFaQ=Vxt5e0Os
>>vvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD
>>3NCT1wEA=0d7WauHwvHvrFWs0bcAWNNxOtvaCSn7-izMA1oDTedw=0XCzqR0TiMEj5d53
>>jKXtPEmzmNe0C3xpAV8fiGXFyq4=

Re: [rsyslog] handling oversized messages

2017-10-25 Thread Randall Diffenderfer via rsyslog

that's my point — i am trying to figure out what i can do …

at the moment i am trying to reconcile my "cloudy" memory of message splitting. 
 i thought that did occur in certain i/o paths, so i was surprised to see a 
pure truncation in the imfile path.

From: Rainer Gerhards 
<rgerha...@hq.adiscon.com<mailto:rgerha...@hq.adiscon.com>>
Date: Wednesday, October 25, 2017 at 11:57
To: Randall Diffenderfer 
<rdiffender...@proofpoint.com<mailto:rdiffender...@proofpoint.com>>
Cc: rsyslog-users <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>
Subject: Re: [rsyslog] handling oversized messages

Yeah but if they are too large... They are. Especially with json payload, what 
will you do against this?

Rainer

Sent from phone, thus brief.

Am 25.10.2017 20:51 schrieb "Randall Diffenderfer" 
<rdiffender...@proofpoint.com<mailto:rdiffender...@proofpoint.com>>:
i suppose i deserved that… :-)

however, i have to interoperate with other folks who can't/won't increase N …

so, i am pegged at their "N" in order to see no data corruption, as the 
messages are structured, in that they are json, and busting them up will be a 
bit problematic…

From: Rainer Gerhards 
<rgerha...@hq.adiscon.com<mailto:rgerha...@hq.adiscon.com>>
Date: Wednesday, October 25, 2017 at 11:33
To: rsyslog-users <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>
Cc: Randall Diffenderfer 
<rdiffender...@proofpoint.com<mailto:rdiffender...@proofpoint.com>>
Subject: Re: [rsyslog] handling oversized messages

It may sound dumb, but: increase n! That's why this setting exists.

Rainer

Sent from phone, thus brief.

Am 25.10.2017 19:48 schrieb "Randall Diffenderfer via rsyslog" 
<rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>:

given the global setting of "maxmessagesize=N",  what is my recourse if i
need to process a message > N in imfile?

in other i/o modules?  it appears the message is truncated at ~N, and not
split (which is what i thought i had seen in the past...)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog=DwMFaQ=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=0d7WauHwvHvrFWs0bcAWNNxOtvaCSn7-izMA1oDTedw=Y2aZ4XynKrB4wgg_Y21Tuh5VisfzrJe4cTVcQkFfBow=>
http://www.rsyslog.com/professional-services/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_=DwMFaQ=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=0d7WauHwvHvrFWs0bcAWNNxOtvaCSn7-izMA1oDTedw=iK9krYQbWQ66DQXmXGuvorYG28Ioqr9mIroDehsBFLM=>
What's up with rsyslog? Follow 
https://twitter.com/rgerhards<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards=DwMFaQ=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=0d7WauHwvHvrFWs0bcAWNNxOtvaCSn7-izMA1oDTedw=0XCzqR0TiMEj5d53jKXtPEmzmNe0C3xpAV8fiGXFyq4=>
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] handling oversized messages

2017-10-25 Thread Randall Diffenderfer via rsyslog

i suppose i deserved that… :-)

however, i have to interoperate with other folks who can't/won't increase N …

so, i am pegged at their "N" in order to see no data corruption, as the 
messages are structured, in that they are json, and busting them up will be a 
bit problematic…

From: Rainer Gerhards 
<rgerha...@hq.adiscon.com<mailto:rgerha...@hq.adiscon.com>>
Date: Wednesday, October 25, 2017 at 11:33
To: rsyslog-users <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>
Cc: Randall Diffenderfer 
<rdiffender...@proofpoint.com<mailto:rdiffender...@proofpoint.com>>
Subject: Re: [rsyslog] handling oversized messages

It may sound dumb, but: increase n! That's why this setting exists.

Rainer

Sent from phone, thus brief.

Am 25.10.2017 19:48 schrieb "Randall Diffenderfer via rsyslog" 
<rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>:

given the global setting of "maxmessagesize=N",  what is my recourse if i
need to process a message > N in imfile?

in other i/o modules?  it appears the message is truncated at ~N, and not
split (which is what i thought i had seen in the past...)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog=DwMFaQ=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=0d7WauHwvHvrFWs0bcAWNNxOtvaCSn7-izMA1oDTedw=Y2aZ4XynKrB4wgg_Y21Tuh5VisfzrJe4cTVcQkFfBow=>
http://www.rsyslog.com/professional-services/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_=DwMFaQ=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=0d7WauHwvHvrFWs0bcAWNNxOtvaCSn7-izMA1oDTedw=iK9krYQbWQ66DQXmXGuvorYG28Ioqr9mIroDehsBFLM=>
What's up with rsyslog? Follow 
https://twitter.com/rgerhards<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards=DwMFaQ=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhg=9sXs16LkSD1MzhoJxgyl7R_tait1dEXqvyD3NCT1wEA=0d7WauHwvHvrFWs0bcAWNNxOtvaCSn7-izMA1oDTedw=0XCzqR0TiMEj5d53jKXtPEmzmNe0C3xpAV8fiGXFyq4=>
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] handling oversized messages

2017-10-25 Thread Randall Diffenderfer via rsyslog


given the global setting of "maxmessagesize=N",  what is my recourse if i
need to process a message > N in imfile?

in other i/o modules?  it appears the message is truncated at ~N, and not
split (which is what i thought i had seen in the past...)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] correct way to redirect log messages to STDOUT

Re: [rsyslog] handling oversized messages

Re: [rsyslog] handling oversized messages

Re: [rsyslog] handling oversized messages

Re: [rsyslog] handling oversized messages

[rsyslog] handling oversized messages

6 matches

Site Navigation

Mail list logo

Footer information