Apache Flume has the necessary pieces. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/
On Wednesday, March 12, 2014 5:01:37 AM UTC-4, Jörg Prante wrote: > > It would also be possible to write a custom Java syslog protocol socket > receiver and index log messages into ES, for example by reusing syslog4j. > Similar to the UDP bulk indexing. > > Jörg > > > On Wed, Mar 12, 2014 at 4:16 AM, Otis Gospodnetic > <otis.gos...@gmail.com<javascript:> > > wrote: > >> Hi, >> >> Is that Logstash instance reading files that are produces by syslog-ng >> servers? Maybe not.... but if yes, have you considered using Rsyslog with >> omelasticsearch instead to simplify the architecture? >> >> Otis >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> On Tuesday, March 4, 2014 10:11:59 AM UTC-5, Eric wrote: >>> >>> Hello, >>> >>> I've been working on a POC for Logstash/ElasticSearch/Kibana for about 2 >>> months now and everything has worked out pretty good and we are ready to >>> move it to production. Before building out the infrastructure, I want to >>> make sure my shard/node/index setup is correct as that is the main part >>> that I'm still a bit fuzzy on. Overall my setup is this: >>> >>> Servers >>> Networking Gear >>> syslog-ng server >>> End Points -----------------> Load Balancer >>> ------------> syslog-ng server --------------> Logs >>> stored in 5 flat files on SAN storage >>> Security Devices >>> syslog-ng server >>> Etc. >>> >>> I have logstash running on one of the syslog-ng servers and is basically >>> reading the input of 5 different files and sending them to ElasticSearch. >>> So within ElasticSearch, I am creating 5 different indexes a day so I can >>> do granular user access control within Kibana. >>> >>> unix-$date >>> windows-$date >>> networking-$date >>> security-$date >>> endpoint-$date >>> >>> My plan is to have 3 ElasticSearch servers with ~10 gig of RAM each on >>> them. For my POC I have 2 and it's working fine for 2,000 events/second. My >>> main concern is how I setup the ElasticSearch servers so they are as >>> efficient as possible. With my 5 different indexes a day, and I plan on >>> keeping ~1 month of logs within ES, is 3 servers enough? Should I have 1 >>> master node and the other 2 be just basic setups that are data and >>> searching? Also, will 1 replica be sufficient for this setup or should I do >>> 2 to be safe? In my POC, I've had a few issues where I ran out of memory or >>> something weird happened and I lost data for a while so wanted to try to >>> limit that as much as possible. We'll also have quite a few users >>> potentially querying the system so I didn't know if I should setup a >>> dedicated search node for one of these. >>> >>> Besides the ES cluster, I think everything else should be fine. I have >>> had a few concerns about logstash keeping up with the amount of entries >>> coming into syslog-ng but haven't seen much in the way of load balancing >>> logstash or verifying if it's able to keep up or not. I've spot checked the >>> files quite a bit and everything seems to be correct but if there is a >>> better way to do this, I'm all ears. >>> >>> I'm going to have my KIbana instance installed on the master ES node, >>> which shouldn't be a big deal. I've played with the idea of putting the ES >>> servers on the syslog-ng servers and just have a separate NIC for the ES >>> traffic but didn't want to bog down the servers a whole lot. >>> >>> Any thoughts or recommendations would be greatly appreciated. >>> >>> Thanks, >>> Eric >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/aa24cb27-6292-4d42-aa09-b13f6688c11f%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/aa24cb27-6292-4d42-aa09-b13f6688c11f%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4292278e-bf8f-4638-a00b-14f59e79851b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
