Re: Kafka - Connect for logs processing

2019-03-15 Thread Pulkit Manchanda
Hi Hans, Thanks for quick response. I am gonna look into it. Thanks Pulkit On Fri, Mar 15, 2019 at 11:39 AM Hans Jespersen wrote: > Take a look at kafka-connect-spooldir and see if it meets your needs. > > https://www.confluent.io/connector/kafka-connect-spooldir/ > > This connector can monitor

Re: Kafka - Connect for logs processing

2019-03-15 Thread Hans Jespersen
Take a look at kafka-connect-spooldir and see if it meets your needs. https://www.confluent.io/connector/kafka-connect-spooldir/ This connector can monitor a directory and pick up any new files that are created. Great for picking up batch files, parsing them, and publishing each line as if it w

Kafka - Connect for logs processing

2019-03-15 Thread Pulkit Manchanda
Hi All, I am building a data pipeline to send logs from one data source to the other node. I am using Kafka Connect standalone for this integration. Everything works fine but the problem is on Day1 the log file is renamed as log_Day0 and a new log file log_Day1 is created. And my Kafka Connect do