I use omkafka heavily - the confParam setting is what you’re looking for, basically all of the standard Kafka producer / consumer related parameters are available there. (see https://kafka.apache.org/documentation/#producerconfigs <https://kafka.apache.org/documentation/#producerconfigs>). What you set and how you set it is heavily dependent on your use case - number of producers, number of consumers, number / size of topics, throughput expectations, etc. But in general for omkafka (producer) you’ll get the most results tuning buffer.memory and batch.size. Those will help you adjust the size and frequency of producing messages.
To add threads out to Kafka? I’m not positive but I’d assume adding multiple rulesets may accomplish that goal - say, multiple listening ports, each with it’s own dedicated ruleset. That would add additional rsyslog queues, and thus multiple output queues. Though in our setup we have 1 instance of rsyslog running per Kafka broker, and have never run in to a case where either rsyslog or Kafka gets bottlenecked Andrew > On Dec 23, 2017, at 8:56 PM, deoren > <rsyslog-users-lists.adiscon....@whyaskwhy.org> wrote: > > > > On 12/22/2017 9:52 AM, Luigi Tagliamonte via rsyslog wrote: >> Hi there! >> What are the tunable parameters for this module, like: >> - an option to increase the number of threads for kafka processing >> - number of messages to process per req. >> - etc.. >> Regards >> L. > > > Module docs: > > * http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.html > * http://www.rsyslog.com/doc/v8-stable/configuration/modules/imkafka.html > > Disclaimer: I know little (if anything) about Apache Kafka, but I found > the following mentioned in the module docs: > > ############################################### > confParam [parameter] > Type: Array > > Default: none > > Permits to specify Kafka options. Rather than offering a myriad of > config settings to match the Kafka parameters, we provide this setting > here as a vehicle to set any Kafka parameter. This has the big advantage > that Kafka parameters that come up in new releases can immediately be used. > > Note that we use librdkafka for the Kafka connection, so the parameters > are actually those that librdkafka supports. As of our understanding, > this is a superset of the native Kafka parameters. > ############################################### > > > That bit is the same for both modules. The omkafka module has this in > addition to the above: > > > ############################################### > topicConfParam [parameter] > Type: Array > > Default: none > > In essence the same as confParam, but for the Kafka topic. > ############################################### > > > I assume that means that you can use the confParam parameter for the > rsyslog imkafka/omkafka modules to pass through native Kafka > parameters/settings. > > Hope that helps. If not, I hope that someone with more knowledge of this > module can chime in and assist. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT.
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
