Hello,
I want to stream the content of a file on kafka. I have setup following architecture: File -> imfile -> omkafka -> kafka. My configuration looks like following: global (workDirectory="$CUSTOMLOCATION") module (load="imfile") module (load="omkafka") input (type="imfile" File="$PATH_TO_FILE" Tag="" PersistStateInterval="10") action(type="omkafka" topic="mytopic" broker="$KAFKA_BROKER" errorFile="$CUSOMLOCATION/error_file" statsFile="$CUSTOMLOCATION/stat_file" resubmitOnFailure="on" failedMsgFile="$CUSTOMLOCATION/failed_msg" keepFailedMessages="on" TopicConfParam="request.required.acks=all") confParam=["socket.timeout.ms=10", "socket.keepalive.enable=true","max.in.flight=1"]) main_queue(queue.workerthreads="1" queue.dequeueBatchSize="30" "queue.size=50000" queue.minDequeueBatchSize="10" queue.minDequeueBatchSize.timeout="100") I noticed few things with Omkafka module: 1. I don't see errorFile, stat_file, failedMsgFile 2. I had a file with 1,000,000 records. It did not stream all messages to kafka broker. 3. Omkafka module does not send a single message; it sent a batch of message to kafka (that is fine with me; but it should be documented somewhere). 4. There is option to append key in the message. I saw a Jira to make dynamic keys. However, is it possible to append line_number or message number as a key. Also, I am going to stream an important file using omkafka module. I guess I need to tweak my configuration properly but so far, I have not been able to stream entire file properly. I expect some duplicates but as long as the content of messages are sent to kafka broker in orderly manner with/without duplicates is fine. Can you please help me fix my configuration ? Thank you, Jigar Rathod _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
