Re: Basic Question on splitting data sources between 2 or more ES systems
You can run different instances of LS each with its own config file. When you define your file input, just point it to a unique since_db location (that's different for each instance) http://logstash.net/docs/1.3.3/inputs/file#sincedb_path -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5229af0a-cfbd-490d-a703-84b71e7e9d5a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Basic Question on splitting data sources between 2 or more ES systems
Forgive me but when you say feeders do you mean the LS actually processing the log? Can you run multiple LS's on the same log without having them trip over each other or end up with missing data read by the other LS first? On Wednesday, March 12, 2014 3:12:04 PM UTC, Binh Ly wrote: > > Yes it could - although test it to see if it is acceptable to you. If it > becomes a problem, then you can always run multiple LS feeders one per ES > cluster and then just separate the config outputs individually. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/40b5a5f5-42b6-4fa1-baa4-5d27639b4563%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Basic Question on splitting data sources between 2 or more ES systems
Yes it could - although test it to see if it is acceptable to you. If it becomes a problem, then you can always run multiple LS feeders one per ES cluster and then just separate the config outputs individually. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a5995252-e0db-4bbd-8768-3ce1238d9eef%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
RE: Basic Question on splitting data sources between 2 or more ES systems
Could that lead to delays in 1 cluster receiving data though? From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Binh Ly Sent: 12 March 2014 14:15 To: elasticsearch@googlegroups.com Subject: Re: Basic Question on splitting data sources between 2 or more ES systems Oh sorry, if your feeder is Logstash, I'd probably just add to the config output section to feed both ES clusters. :) -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/50OvM5LYU2I/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com<mailto:elasticsearch+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36667CDCAAF70140AE7738BB93CA8C9605B132%40ExMbx1.ul.campus. For more options, visit https://groups.google.com/d/optout.
Re: Basic Question on splitting data sources between 2 or more ES systems
Oh sorry, if your feeder is Logstash, I'd probably just add to the config output section to feed both ES clusters. :) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
RE: Basic Question on splitting data sources between 2 or more ES systems
Thanks for that but it wasn’t so much the kibana side of things I was wondering about as I would expect to have to use separate kibanas anyway. Its more what’s the best way to set things up so that I can have the same data source in 2 different ES clusters so that its available to both but each cluster doesn’t have the extra sources required by the other? From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Binh Ly Sent: 12 March 2014 13:56 To: elasticsearch@googlegroups.com Subject: Re: Basic Question on splitting data sources between 2 or more ES systems Sounds reasonable. Yeah at the moment, Kibana pointing to a single cluster will share all the dashboards among everybody who have access to it. If you don't want the dashboard sharing, you need 2 separate ES clusters for now, each with its own Kibana. However, there are some ideas in this post that you might be able to use to configure a proxy behind 2 Kibanas (for instance) pointing to a single ES cluster: http://www.elasticsearch.org/blog/restricting-users-kibana-filtered-aliases/ -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/50OvM5LYU2I/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com<mailto:elasticsearch+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36667CDCAAF70140AE7738BB93CA8C9605B105%40ExMbx1.ul.campus. For more options, visit https://groups.google.com/d/optout.
Re: Basic Question on splitting data sources between 2 or more ES systems
Sounds reasonable. Yeah at the moment, Kibana pointing to a single cluster will share all the dashboards among everybody who have access to it. If you don't want the dashboard sharing, you need 2 separate ES clusters for now, each with its own Kibana. However, there are some ideas in this post that you might be able to use to configure a proxy behind 2 Kibanas (for instance) pointing to a single ES cluster: http://www.elasticsearch.org/blog/restricting-users-kibana-filtered-aliases/ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Basic Question on splitting data sources between 2 or more ES systems
Its probably down to not using the correct terminology when searching this group but whats the recommended way to handle the situation where I need the same data source to be in 2 different elastic searches? *ES1 data sources* syslogs firewall logs webserver logs *ES2 data sources* Twitter flat file webserver logs (same as ES1) The idea would be that each elastic search would have its own kibana but 1 would be used by sys admins and the other would be used for more business analysis purposes -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/05a3c8d8-d1b1-4bea-8ebf-f70db8010833%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
