Re: Pulling data from remote server

2015-12-29 Thread Gwen Shapira
We don't have a built-in functionality for this, but I know few places that implemented the following architecture. 1. Install a small (1-3 nodes) Kafka cluster on the remote environments (AWS, Rackspace, etc) 2. Use KafkaConnect or similar to pull logs into the local Kafka 3. Install Kafka + Mirr

Pulling data from remote server

2015-12-29 Thread Bin Wang
Hi there, I have several different production servers running in different cloud providers, like AWS, Rackspace, ..etc. And I have one central big data environment inside company's network. I need to build a NRT(near real time) monitoring system which I need to stream and archive all the logs from