Re: Suggestions of pulling local application logs into Kafka

Kafka Connect can definitely be used for this -- it's one of the reasons we designed it with standalone mode ( http://docs.confluent.io/3.0.0/connect/userguide.html#workers). For the specific connector, we include a very simple File connector with Kafka which will just take each line and send it

Re: Suggestions of pulling local application logs into Kafka

On Tue, May 3, 2016 at 1:00 PM Banias H wrote: > I should add Flume is not an option for various reasons. > > On Tue, May 3, 2016 at 2:49 PM, Banias H wrote: > > > We use Kafka (0.9.x) internally in our pipeline and now we would like to > > ingest

Re: Suggestions of pulling local application logs into Kafka

I should add Flume is not an option for various reasons. On Tue, May 3, 2016 at 2:49 PM, Banias H wrote: > We use Kafka (0.9.x) internally in our pipeline and now we would like to > ingest application logs sitting in local file system of servers external to > the Kafka

Suggestions of pulling local application logs into Kafka

We use Kafka (0.9.x) internally in our pipeline and now we would like to ingest application logs sitting in local file system of servers external to the Kafka cluster. We could write a producer program running on the application servers to push files to Kafka. However we wonder if we can leverage