> if you have
>
> start transaction
> insert
> insert
> insert
> end transaction
>
> that should be almost as fast as
>
> insert into ... values(),(),()
>
> and wildly faster than just
>
> insert
> insert
> insert
> (no transaction defined)
>
> If this isn't true, then we need to modify the o
On Wed, 8 Nov 2017, Steven Libby wrote:
how many messages in a transaction? it should be large. Is there any chance
that MySQL has trouble with that many messages in a transaction?
It's doing 10,000 per transaction, but I think the fact that it's only
inserting one-at-a-time is not very effic
how many messages in a transaction? it should be large. Is there any chance
that MySQL has trouble with that many messages in a transaction?
It's doing 10,000 per transaction, but I think the fact that it's only
inserting one-at-a-time is not very efficient. I assumed that batch
processing meant
look in the debug logs for how many message are processed between calls to
insert them into MySQL.
David Lang
On Wed, 8 Nov 2017, Steven Libby wrote:
Date: Wed, 8 Nov 2017 16:26:08 -0500
From: Steven Libby
To: David Lang
Cc: Steven Libby via rsyslog
Subject: Re: [rsyslog] ommysql database que
ocessed between calls to
> insert them into MySQL.
>
> David Lang
>
> On Wed, 8 Nov 2017, Steven Libby wrote:
>
> Date: Wed, 8 Nov 2017 16:26:08 -0500
>> From: Steven Libby
>> To: David Lang
>> Cc: Steven Libby via rsyslog
>> Subject: Re: [rsys
message are processed between calls to
insert them into MySQL.
David Lang
On Wed, 8 Nov 2017, Steven Libby wrote:
Date: Wed, 8 Nov 2017 16:26:08 -0500
From: Steven Libby
To: David Lang
Cc: Steven Libby via rsyslog
Subject: Re: [rsyslog] ommysql database queue bottleneck
I've converte
queue on the ruleset and the only item in the ruleset
>>>>
>>>> when you have to queues like this, you end up with lots of locking
>>>> overhead and poor batch sizing.
>>>>
>>>> with retrycount=0 you never attempt to re-deliver a message th
forever
>>>
>>> try setting it to -1 and see if logs are getting stuck (delivery stops),
>>> if so, you are getting some insert attempts that are not being accepted
>>> by
>>> mysql and since you have retries disabled, the entire batch is being
>>>
ntire batch is being thrown
away.
On Fri, 3 Nov 2017, Steven Libby via rsyslog wrote:
Date: Fri, 3 Nov 2017 14:53:17 -0400
From: Steven Libby via rsyslog
To: rsyslog@lists.adiscon.com
Cc: Steven Libby
Subject: [rsyslog] ommysql database queue bottleneck
We are running rsyslog 8.21.0 on a RHEL 6.
are getting some insert attempts that are not being accepted by
mysql and since you have retries disabled, the entire batch is being thrown
away.
On Fri, 3 Nov 2017, Steven Libby via rsyslog wrote:
Date: Fri, 3 Nov 2017 14:53:17 -0400
From: Steven Libby via rsyslog
To: rsyslog@lists.adiscon.com
even Libby via rsyslog wrote:
>
> Date: Fri, 3 Nov 2017 14:53:17 -0400
>> From: Steven Libby via rsyslog
>> To: rsyslog@lists.adiscon.com
>> Cc: Steven Libby
>> Subject: [rsyslog] ommysql database queue bottleneck
>>
>>
>> We are running rsyslog 8.21.0 on a RH
2017 14:53:17 -0400
From: Steven Libby via rsyslog
To: rsyslog@lists.adiscon.com
Cc: Steven Libby
Subject: [rsyslog] ommysql database queue bottleneck
We are running rsyslog 8.21.0 on a RHEL 6.7 box with 256GB of RAM and 96
cores, writing to a MySQL instance (Percona Server 5.6.28-76) using Toku
We are running rsyslog 8.21.0 on a RHEL 6.7 box with 256GB of RAM and 96
cores, writing to a MySQL instance (Percona Server 5.6.28-76) using TokuDB
as the storage engine. This is our centralized log server, which receives
messages from numerous network devices at an average rate of 7k messages
per
13 matches
Mail list logo
