Finally, I had a chance to test the new -kafka packages on RHEL6.5.
First the quirks/errors. I tried this config but didn't work:
action(type="omkafka" topic="rawunstruct"
broker="test-kafkanode1-1-node.ops.abc.net:9092,
test-kafkanode1-2-node.ops.abc.net:9092,
test-kafkanode1-3-node.ops.abc.net:9092,
test-kafkanode1-4-node.ops.abc.net:9092,
test-kafkanode1-5-node.ops.abc.net:9092"
partitions.number="5"
errorFile="/data/data01/rsyslog/errors/kafka-output-err.log"
confParam=["compression.codec=snappy",
"request.required.acks=1",
"producer.type=async",
"batch.num.messages=10000"
]
queue.filename="toKafka"
queue.size="360000000"
queue.maxdiskspace="360G"
queue.highwatermark="216000000"
queue.discardmark="288000000"
queue.type="LinkedList"
queue.dequeuebatchsize="4096"
queue.timeoutenqueue="0"
queue.maxfilesize="4G"
queue.saveonshutdown="on"
queue.workerThreads="4"
template="rawfwd"
) stop
rsyslogd complained:
2014-12-21T00:31:58.980146+00:00 test-kafkanode1-24-node rsyslogd-1000:
error in kafka parameter 'request.required.acks=1': No such configuration
property: "request.required.acks" [try http://www.rsyslog.com/e/1000 ]
2014-12-21T00:33:23.507670+00:00 test-kafkanode1-24-node rsyslogd: [origin
software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="8265" x-info="
http://www.rsyslog.com";] exiting on signal 15.
2014-12-21T00:33:26.624810+00:00 test-kafkanode1-24-node rsyslogd: [origin
software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="19759" x-info="
http://www.rsyslog.com";] start
2014-12-21T00:33:26.624811+00:00 test-kafkanode1-24-node rsyslogd-1000:
error in kafka parameter 'producer.type=async': No such configuration
property: "producer.type" [try http://www.rsyslog.com/e/1000 ]
2014-12-21T00:33:26.616090+00:00 test-kafkanode1-24-node rsyslogd0: debug
log file is '/data/data01/rsyslog/debug/debug.log', fd 3 [try
http://www.rsyslog.com/e/0 ]
2014-12-21T00:33:53.092950+00:00 test-kafkanode1-24-node rsyslogd-1000:
error in kafka parameter 'producer.type=async': No such configuration
property: "producer.type" [try http://www.rsyslog.com/e/1000 ]
So I trimmed down the config to:
action(type="omkafka" topic="rawunstruct"
broker="test-kafkanode1-1-node.ops.abc.net:9092,
test-kafkanode1-2-node.ops.abc.net:9092,
test-kafkanode1-3-node.ops.abc.net:9092,
test-kafkanode1-4-node.ops.abc.net:9092,
test-kafkanode1-5-node.ops.abc.net:9092"
partitions.number="5"
errorFile="/data/data01/rsyslog/errors/kafka-output-err.log"
confParam=["compression.codec=snappy"]
queue.filename="toKafka"
queue.size="360000000"
queue.maxdiskspace="360G"
queue.highwatermark="216000000"
queue.discardmark="288000000"
queue.type="LinkedList"
queue.dequeuebatchsize="4096"
queue.timeoutenqueue="0"
queue.maxfilesize="4G"
queue.saveonshutdown="on"
queue.workerThreads="4"
template="rawfwd"
) stop
At restart, rsyslogd failed to stop cleanly, throwing these errors in
/var/log/messages:
2014-12-21T00:35:33.314674+00:00 test-kafkanode1-24-node kernel: rs:action
1 que[19764]: segfault at 21 ip 00007f9c329f4689 sp 00007f9c31a78b18 error
4 in libc-2.12.so[7f9c328c9000+18b000]
But start was ok.
Then, there was a crash but abrtd did not keep the crash dump :(
So, I set rsyslogd to run in debug mode but after several tens of Gigs
worth of debug logs, I couldn't re-produce the crash so went back to debug
set to '0'.
Now, so far, haven't had a crash in about half an hour and I would say
about ~40 million messages must have passed through in this time :)
For a first cut, I give this release a 10/10 :D
Hoping to test more and smoke out any bugs. As a current user of
rsyslog->flume->kafka, rsyslog->kafka simply kicks butt!!! I can't wait to
say goodbye to resource hungry and sloooowww flume/java :)
Thanks,
Tim
On Tue, Dec 16, 2014 at 2:11 PM, Rainer Gerhards <rgerha...@hq.adiscon.com>
wrote:
> Andre has now also created some test packages for Centos 6 & 7. If you
> would like to use them, add this package source:
>
> http://rpms.adiscon.com/testing/
>
> Note that this source is obviously not stable, but we will keep it
> available until the official release happens.
>
> Rainer
>
> 2014-12-15 19:06 GMT+01:00 Tait Clarridge <t...@clarridge.ca>:
> >
> > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards
> > <rgerha...@hq.adiscon.com> wrote:
> > > So here it is:
> > >
> > > https://github.com/rsyslog/rsyslog/tree/master-omkafka
> > >
> > > I have deliberately left out the impstats calls as I saw you included
> > them
> > > in your source, so feel free to add them again ;)
> > >
> > > Note that there is a different approach to parameters: my version has
> > only
> > > very few hardcoded ones, but instead permits the user to set actual
> kafka
> > > parameters via the confparam and topicparam parameters. In my
> experience,
> > > this is far better with a rapidly evolving backend, because now any new
> > or
> > > changed parameter is immediately available.
> > >
> > > I'll write up some bare doc later today. So far this is tested on
> Ubuntu
> > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone else)
> > > could test it - especially as I have only very limited Kafka know-how
> and
> > > no real use case myself.
> > >
> > > Thanks,
> > > Rainer
> > >
> >
> > Great! Looks much better than my original version, the "dynamic"
> > configuration for conf and topicconf are definitely a win there
> > (hardcoding is the worst).
> > I had submitted my pull request with the template enabled topics, I
> > had to shuffle some of the items around to make the topics work on
> > each call of writeKafka.
> >
> > Found here:
> > https://github.com/rsyslog/rsyslog/pull/186
> >
> > Tait
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
