Re: Elasticsearch: 2-node cluster with failover
On Thursday, May 28, 2015 at 12:25 CEST, prakhar prakhar.mishra1...@gmail.com wrote: If I add one *master only* node on one of the two existing servers (2 nodes on the same physical server), with *HEAP*, lets say, /100mb/; will that work? Or I have to add another machine for that? That should work, but it obviously means that if the machine with two Elasticsearch nodes is taken out your cluster will be inoperable (but it will survive if the one-node machine dies). -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- Please update your bookmarks! We have moved to https://discuss.elastic.co/ --- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150528110806.GA25289%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Elastic Search configuration
On Wednesday, May 13, 2015 at 08:43 CEST, Hardik Dobariya hardikdobariya1...@gmail.com wrote: we have around 500 indices all sizing to approx 50gb respectively and total size goes around 1tb for all indices and will keep on increasing. The reason behind setting node3 as Master=false,data=false because in configuration file i read this type of node will only work for searching ,aggregation etc. Do i still need to set node3 three as mater and data You'll definitely want all three nodes to be master-eligible (as Mark explained this prevents a split brain situation). Whether it makes sense to dedicate one of the nodes to queries depends a bit on the types of queries you make but I'd say it's unlikely that dedicating a third of your cluster's capacity for queries is the best use of your money. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- Please update your bookmarks! We have moved to https://discuss.elastic.co/ --- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150513071628.GA13466%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Curator 3
On Tuesday, May 12, 2015 at 21:06 CEST, Wendel Ferreira wendelmineir...@gmail.com wrote: Pessoal, boa tarde. Depois que atualizei o elasticsearch para a 1.5 vi que a rotinha no crontab do curator parou de funcionar. This mailing list is in English. The new discussion forum that's replacing this list has a couple of non-English categories but currently none for Portuguese (if that indeed is your preferred language). https://discuss.elastic.co/c/in-your-native-tongue [...] -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- Please update your bookmarks! We have moved to https://discuss.elastic.co/ --- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150512201102.GA2679%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Elastic Search configuration
On Tuesday, May 12, 2015 at 13:24 CEST, Hardik Dobariya hardikdobariya1...@gmail.com wrote: yes i can understand the use of network file system is not good. Actually we do not have any physical machines.we are using virtual environment.This is the reason we are using NAS to store data on network. Sure, but you don't have to use NAS just because you use VMs. Local disks or volumes mounted from a SAN are still preferred to accessing the file system over SMB or NFS. Any suggestion on using virtual environment for elastic search? and yes am already using 3 nodes.2 masters with data and one child Yes, and as Mark says that's a bad idea. You're better off allowing all three nodes to be masters. Unless you have a serious query load you should keep data on all three nodes. Having a third of the cluster's capacity dedicated to processing queries is most likely very wasteful. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- Please update your bookmarks! We have moved to https://discuss.elastic.co/ --- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150512121645.GA806%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: @timestamo and Date in the logs are not matching
On Friday, May 08, 2015 at 09:39 CEST, vurkechud...@gmail.com wrote: The @timestamp field and the actual log generated Date is not matching as shown in the screenshot. Discussion thread already started here: https://discuss.elastic.co/t/timestamp-field-not-matching-with-the-actual-log-field/343 -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- Please update your bookmarks! We moved to https://discuss.elastic.co/ --- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150508074525.GA30221%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Does elasticsearh-curator 3.0.3 support elasticsearch 1.0.1
On Wednesday, May 06, 2015 at 07:09 CEST, sumeet dembra sumeetdem...@gmail.com wrote: That is what I am not getting. Why is it showing 2014.00.28? There is no such index present in my ES setup which has date 2014.00.28.. Looking at the code I suspect you have a snapshot or an index with that name. The exception happens in timestamp_check[0] which is only called from apply_filter[1]. apply_filter is, in turn, called from [2] and [3]. Upping the loglevel to 'debug' will cause the list being filtered to be logged. [0] https://github.com/elastic/curator/blob/v3.0.3/curator/api/filter.py#L285 [1] https://github.com/elastic/curator/blob/v3.0.3/curator/api/filter.py#L127 [2] https://github.com/elastic/curator/blob/v3.0.3/curator/cli/snapshot_selection.py#L74 [3] https://github.com/elastic/curator/blob/v3.0.3/curator/cli/index_selection.py#L80 -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- Please update your bookmarks! We moved to https://discuss.elastic.co/ --- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150506074757.GA16989%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: abnormal file input behavior?
On Friday, May 01, 2015 at 21:04 CEST, Sitka sitkaw...@gmail.com wrote: I have a file of logging records I am using to debug some filter parses. I am using file input and have set starting_position to beginning. So I startup logstash see what I get and killed it and make fixes and go again. I have seen if sometimes it reads the file and sometimes not. I ran some experiments and found that if I delete the file and rewrite it and then start up logstash it reads the file. If I have previously read the file, then when logstash starts it doesn't read the file despite being told to start at beginning. Am I missing something here? Is this intended behavior or a possible bug? This is the intended behavior and documented at http://logstash.net/docs/1.4.2/inputs/file#start_position: This option only modifies 'first contact' situations where a file is new and not seen before. If a file has already been seen before, this option has no effect. If you want to force Logstash to reprocess a file, delete the corresponding sincedb file (or entry within such a file). For testing purposes it's much more convenient to use the stdin input. Next time, please post Logstash questions like this one to the logstash-users list. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150504064649.GA19365%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Please suggest.
On Wednesday, April 15, 2015 at 08:00 CEST, vikas gopal vikas.ha...@gmail.com wrote: thank you for your quick response. I am totally new to this, any document or website to understand nginx or any guide to configure nginx as a reverse proxy on windows server 2012. Have you had a look at the nginx documentation or any of the top hits when you search for nginx windows? If they're not adequate for your needs, perhaps you can phrase a more specific question. Note that there's nothing special about nginx. IIS is capable of doing what you want too. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150416060348.GB25217%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: How to achieve ELK High availability
On Thursday, April 16, 2015 at 13:35 CEST, vikas gopal vikas.ha...@gmail.com wrote: Thank you for the suggestion , yes I am aware and I am done with ES clustering . Now I want the same for LS . Since LS does not have in build feature like ES has , so what would be the best way for LS to make i highly available in windows environment? Let's continue that discussion in the logstash-users thread. https://groups.google.com/d/topic/logstash-users/tQHqrXPSV_w/discussion -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150416140755.GA23695%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: kibana display usernum problem?
On Thursday, April 16, 2015 at 03:34 CEST, way way smalldream...@gmail.com wrote: but i think you did not understand what i want I do not want Y-Axis aggregation log count, I just whant Y -Axis display my logs KV's value the guide all is aggregation about log's count Change the aggregation to show the mean/max/min value (whichever you prefer) instead of the count. Kibana 4 might be capable of plotting raw data points. Which Kibana version are you running? -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150416055639.GA25217%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Installation Guide
On Thursday, April 09, 2015 at 08:31 CEST, Gunasekar gn.sek...@gmail.com wrote: I'm newbie to Elasticsearch, Am little bit confused on the Workflow of ELK stack. Installed ELK stack. LOG-FORWARDER not sending data to Elasticsearch server. Kibana not loading. I think, Once log-forwarder workers kibana will show the graphs. Kindly any one help me with the complete guide. Your question is too broad and open-ended. Why aren't the logs being sent to Elasticsearch? What type of logs are you trying to send? What's your overall setup? Are you using logstash-forwarder or Logstash? Which versions? Kibana 3 or Kibana 4? Logstash has a getting started guide (below) and I'm sure there are many blog posts that explain things with slightly different perspectives. http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150409070104.GA15772%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Monitor incoming data rates
On Thursday, April 09, 2015 at 05:49 CEST, Bernie Carolan bernie.caro...@gmail.com wrote: I have alerts configured on my Logstash - Elasticsearch setup which perform regular queries to see what state the cluster is in etc. Recently I had a situation where Logstash was running OK and ES cluster was in Green state, but there was no data going into ES. Is there a way to monitor this, e.g if the incoming data rate to ES drops below 100 events a minute. I can't do it by Logstash metrics because that side of it was running normally, i.e. no change in data rates. How can Logstash be running normally when it halts the pipeline when one output isn't able to accept messages? Are you saying that Elasticsearch accepts messages but just drops them on the floor? -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150409061248.GA13186%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: 2 Errors with Elasticsearch
On Monday, April 06, 2015 at 17:05 CEST, kelnrluierhfeulne dmch12...@gmail.com wrote: When I open Kibana by searching for my IP in my browser, I get the following 2 errors. Would anyone happen to have any advice on how to fix these errors? I already updated Elasticsearch to its latest version (elasticsearch-1.4.2, kibana-4.0.1-linux-x64, logstash-1.4.2): (1.4.2 isn't the latest version of Elasticsearch, not even in the 1.4.x series.) - Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above. - Error Could not reach http://localhost:9200/_nodes. If you are using a proxy, ensure it is configured correctly Are you *sure* this is Kibana 4? AFAIK that error message only exists in Kibana 3 -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150408071351.GC3078%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: 2 Errors with Elasticsearch
On Wednesday, April 08, 2015 at 09:13 CEST, Magnus Bäck magnus.b...@sonymobile.com wrote: On Monday, April 06, 2015 at 17:05 CEST, kelnrluierhfeulne dmch12...@gmail.com wrote: [...] - Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above. - Error Could not reach http://localhost:9200/_nodes. If you are using a proxy, ensure it is configured correctly Are you *sure* this is Kibana 4? AFAIK that error message only exists in Kibana 3 Oh, and if it indeed is Kibana 3 I responded to the same question on logstash-users yesterday. https://groups.google.com/d/msg/logstash-users/rFtu_WBK3Hk/TwNbAFuV0J0J Please don't post the same question to multiple lists. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150408071918.GA4886%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Reindex from the existing index
On Wednesday, April 01, 2015 at 06:15 CEST, Vladi Feigin vla...@liveperson.com wrote: Is there a way to build a new index from the existing index ? Elasticsearch doesn't ship with a tool for this purpose nor a one-stop-shop API, but there are several wrappers of the bulk reindexing API. I've successfully used the es-reindex tool and the elasticsearch.helpers.reindex() function from the official Python client for ES. https://github.com/geronime/es-reindex http://elasticsearch-py.readthedocs.org/en/master/helpers.html -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150401062112.GC3582%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: How to achieve ELK High availability
On Wednesday, April 01, 2015 at 05:06 CEST, vikas gopal vikas.ha...@gmail.com wrote: Need your valuable suggestions here . I have ELK on a single windows instance and I want to make it high available . I mean if one machine goes down second will take up the whole load, like clustering. Can you suggest how I can achieve this. Since you're saying like clustering, are you aware that Elasticsearch supports clustering natively? To improve the availability, run two or (preferably) at least three Elasticsearch nodes and configure replicas for your shards. If one node goes down all data will still be available. http://www.elastic.co/guide/en/elasticsearch/guide/current/distributed-cluster.html -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150401063316.GD3582%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
elastic.co blog RSS URL missing
The not too widely announced move from elasticsearch.(com|org) to elastic.co the other week seems to have broken the old Elasticsearch blog RSS feed, and I can’t find the RSS URL for the replacement elastic.co blog. Please say there is one. A final post to the old blog referring to the new one would've been nice. I'm probably not the only one who's missed the updates from the last week or two. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150325071127.GA22589%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Why does creating a repository fail?
On Tuesday, March 17, 2015 at 01:37 CET, David Reagan jer...@gmail.com wrote: On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden and...@elastic.co wrote: I'm not that familiar with iSCSI so I hesitate to say for sure, but anytime you are cross-mounting filesystems on Linux you have to take uid/gid consistency into account. If I were manually creating the elasticsearch user, that'd be easy. But I'm relying on apt to do the job for me. So, yeah... You can create the elasticsearch user with the uid you want before the Debian package gets a chance to do it. Hmm... I suppose I could manually create an elasticsearch2 user, then modify the defaults files to use it when running ES. Still seems clunky... No need. If the elasticsearch user exists the package installation will just skip that step. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150318072807.GC15152%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: how to create dashboard
On Tuesday, March 10, 2015 at 10:15 CET, vikas gopal vikas.ha...@gmail.com wrote: Since i am new to this technology , I need your assistance to start building dashboard in ELK. I have downloaded all the 3 tools (E,L,K) . I want to create a dashboard from a syslog file . I don't know how I can get data into ELK and prepare dashboard out of it. Please suggest from where I can start. Logstash's own getting started guide covers syslog. http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash If you need further help I suggest you ask more specific questions. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150311071558.GC5729%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Default Number Of Replicas for new indices
On Wednesday, March 11, 2015 at 10:56 CET, Matt Stibbs mattsti...@gmail.com wrote: How do I change the default number of replicas for new indices in my ES cluster? The index.number_of_replicas setting in elasticsearch.yml controls this, but it can be overridden by the corresponding dynamic setting. On top of that, index templates can override both these options. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html [...] -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150311105820.GC31552%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: How ELK stores data
On Monday, March 09, 2015 at 16:34 CET, vikas gopal vikas.ha...@gmail.com wrote: I am totally new to this tool, so I have couple of basic queries 1) How ELK stores indexed data. Like traditional analytic tools stores data in flat files or in their own database . Elasticsearch is based on Lucene and the data is stored in whatever format Lucene uses. This isn't something you have to care about. 2) How we can perform historical search Using the regular query APIs. Sorry for such a general answer but your question is very general. 3) How license is provided , I mean is it based on data indexed per day ? It's free Apache-licensed software so you don't have to pay anything. If you feel you need a support contract that's being offered at a couple of different levels. I'm sure there are third parties offering similar services. http://www.elasticsearch.com/support/ 4) If I want to start do I need to download 3 tools (ElasticSearch,Logstash, Kibana) If you want the whole stack from log collection to storage to visualization then yes, you need all three. But apart from a dependency from Kibana to Elasticsearch the tools are independent. I suggest you download them and try them out. That's the quickest way to figure out whether the tool stack (or a subset thereof) fits your needs. There are also a number of videos available. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150309161010.GA18116%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: ES - settings/mappings - globally for an index - index: not_analyzed and analyzer:whitespace - new feature or not supported.
On Thursday, March 05, 2015 at 22:56 CET, Mark Walkom markwal...@gmail.com wrote: On 6 March 2015 at 01:39, KaranM [1]karan.mu...@gmail.com wrote: I want to globally set the following for all current string fields for an Index and also for the future(new) string fields on that Index. Can some some one send example or link that has example, I was researching and could not find one. index: not_analyzed, analyzer:whitespace You cannot set it globally, you have to do it for each field. Wait, isn't this what dynamic templates are for? http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html#dynamic-templates -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150309161404.GB18116%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Removing subfield in concrete mapping
I'm using the following index template (derived from the Logstash index
template):
{
template: logstash-*,
...
mappings: {
_default_: {
...
dynamic_templates: [
{
string_fields: {
match: *,
match_mapping_type: string,
mapping: {
type: string,
index: analyzed,
omit_norms: true,
fields: {
raw: {
type: string,
index: not_analyzed,
ignore_above: 256
}
}
}
}
}
]
},
gerrit_event: {
dynamic_templates: [
{
string_fields: {
match: *,
match_mapping_type: string,
mapping: {
type: string,
index: not_analyzed,
omit_norms: true,
fields: {}
}
}
}
]
}
}
...
}
I thought this would've effectively deleted the .raw subfield for
gerrit_event documents, but that doesn't happen:
$ curl --silent -XGET
'http://localhost:9200/logstash-2015.04.25/_mapping/gerrit_event?pretty' | head
-n 22
{
logstash-2015.04.25 : {
mappings : {
gerrit_event : {
dynamic_templates : [ {
string_fields : {
mapping : {
index : not_analyzed,
type : string,
fields : {
raw : {
index : not_analyzed,
ignore_above : 256,
type : string
}
},
omit_norms : true
},
match : *,
match_mapping_type : string
}
} ],
Is this expected? If so, why, and is there another way to remove
accomplish what I want, i.e. have a .raw subfield for dynamically mapped
string fields in all types but not gerrit_event? (Without removing the
subfield from _default_ and adding it in many other places, obviously.)
--
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20150226155719.GA16432%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Re: logstash ..sending files inside a directory to elasticsearch configration
On Wednesday, February 25, 2015 at 07:10 CET, Ch Ravikishore ravikishore.ris...@gmail.com wrote: I am new to ELK stack..my requirement here is ,my logtash should send the files that i place inside a folder to elasticsearch.. Iam looking for the configuration for this task.. This question is more about Logstash than Elasticsearch so I suggest you instead ask the question on the logstash-users mailing list. https://groups.google.com/forum/#!forum/logstash-users Also, there are *many* tutorials that describe the setup of the ELK stack, including one on the Logstash web site (below). I think you should read some of what's already available. With more knowledge you'll be able to ask better and less open-ended questions. http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150225063931.GA25857%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Assistance requried for Logstash filter with GROK
On Wednesday, February 25, 2015 at 05:55 CET,
Bharath Paruchuri bharath.1...@gmail.com wrote:
I'm trying to filter below weblogic log using Logtrash filter GROK.
Please post Logstash question to the logstash-users mailing list.
https://groups.google.com/forum/#!forum/logstash-users
[...]
multiline {
type = SOA1-diagnostic
pattern = ^\[%{TIMESTAMP_ISO8601\]
Couldn't help noticing that there's a } missing here.
[...]
--
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20150225064247.GB25857%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Re: Availability of logstash-forwarder debian packages
On Wednesday, February 11, 2015 at 12:53 CET, Dennis Plöger dploeger2...@gmail.com wrote: Until recently I used the elasticsearch-package repositories (packages.elasticsearch.org) to install logstash-forwarder. However, it now seems as if logstash-forwarder isn't available anymore via deb http://packages.elasticsearch.org/logstashforwarder/debian stable main. Did the structure change on the package server or is the package not available anymore? I found no blog post or something like that. Please see the following GitHub issue: https://github.com/elasticsearch/logstash-forwarder/issues/184 -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150211124816.GA7634%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: backup and restore doubt ??
On Monday, February 09, 2015 at 19:37 CET, Subbarao Kondragunta subbu2perso...@gmail.com wrote: I took backup of all indices by default with snapname 'test'. Can i restore only specific indices from snap 'test' , not all ? Yes, by specifying the indices to restore in the 'indices' key in the JSON document that you POST to initiate the restore operation. http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_restoring_from_a_snapshot.html -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150209193237.GA21999%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Need review for my REST query (template modification)
On Thursday, January 22, 2015 at 11:57 CET,
Aldian aldian...@gmail.com wrote:
I am using the usual ELK stack with the default template
([1]http://pastebin.com/DtYiazVr). In every log message, the date in
stored in field named log_date, which the date filter converts in a
@timestamp. I want to set the log_date field as not_analyzed so
that I can sort it in Kibana without getting weird results.
You're storing the same timestamp in two fields? Why?
I built the following query
curl -XPUT localhost:9200/_template/template_1 -d '
{
template : logstash-*,
properties : {
log_date : {
type : string,
index : not_analyzed
}
}
}
Can you confirm that the request is correct?
It looks okay. You may want to use the 'order' key to make sure these
two matching index templates are applied in a well-defined and obvious
order.
I have doubts about the template name. I thought about calling url
localhost:9200/_template/logstash in order to modify the existing
template rather than creating a new one, but I am afraid of what could
happen the day I restart logstash, so my thinking is that if all works
as intended, both logstash default template and that one will apply.
I disable Logstash's index template handling and maintain my own
template (that started out as a copy of Logstash's).
Also I believe that templates are only about future data. Is there any
way to retro apply it back on existing indexes?
You'll have to reindex the data, e.g. using es-reindex.
https://github.com/geronime/es-reindex
--
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20150129122021.GE30366%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Re: logstash / kibana can't connect to instance
On Thursday, January 29, 2015 at 06:51 CET, ma...@venusgeo.com wrote: Can anyone please look into this. This is a volunteer-based mailing list. If want a 24-hour SLA there are paid options for that. On Wednesday, January 28, 2015 at 5:43:23 AM UTC-8, ma...@venusgeo.com wrote: I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see only thing message come up: [...] Connection Failed Possibility #1: Your elasticsearch server is down or unreachable This can be caused by a network outage, or a failure of the Elasticsearch process. If you have recently run a query that required a terms facet to be executed it is possible the process has run out of memory and stopped. Be sure to check your Elasticsearch logs for any sign of memory pressure. Your browser's developer console (naming varies) probably contains clues about the problem. In Chrome you can access it with e.g. Ctrl+Shift+I. [...] -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150129121307.GD30366%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Kibana - IIS 7.5
On Monday, January 26, 2015 at 14:58 CET, GWired garrettcjohn...@gmail.com wrote: I was able to get Kibana setup on my localhost and did a generic entry to allow everything into the elasticsearch.yml http.cors.allow-origin: /.*/ Now I'm trying to getting it to run on my remote server running IIS 7.5 on port 8080. The page loads but only the top bar loads and nothing else any ideas? Did you also enable CORS by setting http.cors.enabled to true? http://stackoverflow.com/questions/26828099/kibana-returns-connection-failed http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150126144857.GA18395%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Configure Kibana for HTTPS
On Tuesday, January 20, 2015 at 14:54 CET, Karthik M karthik4...@gmail.com wrote: On Tuesday, January 20, 2015 at 2:17:49 AM UTC-5, Magnus Bäck wrote: I want the front end of ES (kibana) to run on SSL but keep the backend connection from Kibana to ES unencrypted since both are running on the same host. I configured Apache2 to accept SSL connections and it works but when Kibana populates the dashboard it get the below error. Any help is very much appreciated. Could not reach http://ec2-XX-XX-XX-XX.compute-1.amazonaws.com/elasticsearch/_ nodes. If you are using a proxy, ensure it is configured correctly Which version of Kibana is this? Kibana version 3 Kibana 3 doesn't have a backend. The connections to Elasticsearch originate from your browser so you'll want to encrypt them as well. Placing just the Kibana files behind HTTPS isn't useful. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150120140106.GA484%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Configure Kibana for HTTPS
On Tuesday, January 20, 2015 at 15:59 CET, Karthik Gmail karthik4...@gmail.com wrote: On Jan 20, 2015, at 9:01 AM, Magnus Bäck magnus.b...@sonymobile.com wrote: Kibana 3 doesn't have a backend. The connections to Elasticsearch originate from your browser so you'll want to encrypt them as well. Placing just the Kibana files behind HTTPS isn't useful. Thanks Magnus. Would be able to point me towards on how to set that up? Googling kibana reverse proxy should yield some useful results, but start with the blog post from the Elasticsearch team. http://www.elasticsearch.org/blog/playing-http-tricks-nginx/ -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150120150330.GA5033%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Configure Kibana for HTTPS
On Monday, January 19, 2015 at 15:45 CET, Karthik M karthik4...@gmail.com wrote: I want the front end of ES (kibana) to run on SSL but keep the backend connection from Kibana to ES unencrypted since both are running on the same host. I configured Apache2 to accept SSL connections and it works but when Kibana populates the dashboard it get the below error. Any help is very much appreciated. Could not reach [1]http://ec2-XX-XX-XX-XX.compute-1.amazonaws.com/elasticsearch/_ nodes. If you are using a proxy, ensure it is configured correctly Which version of Kibana is this? -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150120071740.GA2980%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch logging
On Friday, January 02, 2015 at 19:20 CET, Jesse Redl jr...@vendasta.com wrote: When reviewing the logs generated by elasticsearch (1.4.x), a single log message is being split across multiple lines? Yes, that's normal for Java logs. Is this configurable within logging.yml? My undstanding is that elasticsearch is powered by log4j however, I'm not familiar with this product, nor can I find any decent documentation on the logging.yml file. I suspect the reason you're asking is that you're considering ingesting the Elasticsearch logs with Logstash, in which case you should look into the JSONEventLayoutV1 Log4j layout (https://github.com/logstash/log4j-jsonevent-layout) to get the logs in JSON format. You should be able to drop that jar file along with its dependencies (net.minidev:json-smart:1.1.1 and commons-lang:commons-lang:2.6) into the Elasticsearch lib directory (typically /usr/share/elasticsearch/lib) and adjust logging.yml to use that layout. See also https://github.com/elasticsearch/elasticsearch/issues/8786. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150105095131.GA12719%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: elastic search upgrade issues
On Monday, January 05, 2015 at 10:13 CET, phani.nadimi...@goktree.com wrote: Thank you for the quick reply. i upgraded both nodes in elastic search. the following are the diskspaces in each node node 1 : 3Gb available out of 35 GB node 2 : 4GB available out of 35 GB is this be a problem? yes java and elastic search has same versions on each nodes. java version using : 1.7.0_55 Yes, this is probably your problem. By default Elasticsearch won't allocate shards on nodes with 15% free disk space. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html#disk -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150105095715.GB12719%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: [Kibana] group by request?
On Tuesday, December 16, 2014 at 10:03 CET, stephanos stephan.beh...@gmail.com wrote: we are using Google App Engine to host our SaaS app. Google offers a nice log browser but it is way too slw. So one of my colleagues suggested we pipe our logs to logstash and make them accessible via Kibana. So far so good, we managed to set everything up. But when Kibana was shown to the other team members they weren't really excited. It was much faster, yes. It allowed to make better queries, yes. BUT it broke the pattern they knew from the Google App Engine log browser: /some-request log message 1 log message 2 /another-request log message 3 /yet-another-request log message 4 While Kibana works like this: log message 1/some-request log message 2/some-request log message 3/another-request log message 4/yet-another-request So basically App Engine groups log messages by request. To get my team on board, can we make Kibana do the same? Not out of the box, no. Kibana doesn't have any such contextual understanding of messages and currently can't be configured as such either. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20141222063817.GB11963%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Controlling users to change the Kibana dashboard settings and saving.
On Monday, December 01, 2014 at 14:21 CET, Pillalamarri Kaushik aghamars...@gmail.com wrote: I would like to control the changes that are being made to the Kibana dashboard by users. I would like to authenticate the user doing that by asking for username and password before making changes or before saving the changes made. I am using windows operating system. Kibana and Elasticsearch don't have any access control features (but the upcoming companion product Shield does; it might eventually help you). You'll have to add authentication on top of them, e.g. by configuring your web browser to require authentication for POST, PUT, and/or DELETE requests to the kibana-int index. Depending on how capable your web server is You may have to write a custom reverse proxy for this. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20141222065154.GE11963%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Keeping a value across events
On Monday, December 15, 2014 at 22:18 CET, Pierre Carlson mpc...@gmail.com wrote: Totally forgot to mention that this is a LogStash question. Indeed, and as such https://groups.google.com/forum/#!forum/logstash-users is a better fit. This topic has come up in the past so you may find relevant answers in the archives (the short answer is no, you can't do this without a custom plugin). -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20141216073617.GA9662%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Grox help
On Monday, November 24, 2014 at 20:09 CET, Billy F billyfurl...@gmail.com wrote: arrrg. forgot to escape the |. Excellent! Next time, please keep in mind that the logstash-users list is a better fit for grok questions than the elasticsearch list. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20141124194331.GA8664%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Help receiving syslog data in Logstash
On Wednesday, November 12, 2014 at 16:14 CET,
Andrew Stacey arsta...@gmail.com wrote:
This is probably a very noobish question. I just starting playing
with an ELK stack I have set up on Centos 7. All the core services
seem to be working but I can't seem to get it to receive syslog
messages. I have both selinux and the firewall turned off (just a
local lab right now). Netstat -nlp does not show anything listening
on port 514.
According to the logstash book, I need to add the following syslog
input plugin
syslog {
type = syslog
port = 5514
}
This question would've been a better fit for the Logstash mailing list.
https://groups.google.com/forum/#!forum/logstash-users
[...]
ERROR couldn't connect to tcp socket on 10.1.10.154:514; No connection
could be made because the target machine actively refused it.
nxlog tries to send to port 514 but you've configured Logstash to listen
on port 5514. Either one needs to be adjusted to match the other. Keep
in mind that only root can listen on port 514 (but see below) and
Logstash is typically not run as root.
http://unix.stackexchange.com/questions/10735/linux-allowing-an-user-to-listen-to-a-port-below-1024
[...]
--
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20141112155404.GA25446%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Re: Creating index dynamically in ES.
On Tuesday, November 04, 2014 at 00:57 CET,
Alejandro Alves alejandro.al...@gmail.com wrote:
El miércoles, 19 de febrero de 2014 05:02:40 UTC+13, Binh Ly
escribió:
You can specify the index name in the elasticsearch output:
http://logstash.net/docs/1.3.3/outputs/elasticsearch#index
For example, let's say I have a field named clientip, I can make
indexes named ls-clientip by specifying something like this:
output {
elasticsearch {
host = localhost
index = ls-%{clientip}
}
}
How or where do you declare the variables such as %{clientip}?
They are fields in the message, often populated by a 'grok' filter.
Note that not all strings in Logstash configuration files support
%{variablename} interpolation. Where you can use such references
is unfortunately underdocumented.
--
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20141110075747.GD6370%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Re: Multiple Timezones in Elasticsearch/Kibana
On Thursday, October 16, 2014 at 18:57 CEST, Kellan Strong vaid.kel...@gmail.com wrote: I am having a problem with different timezones sending their information to elasticsearch/kibana. One of the logs that is sending is at UTC time however the elasticsearch box is at local time zone. The message is clearly sent at the time of the event however elasticsearch or kibana is indexing it so that only when its that time that will it show up. Is there a way to allow elasticsearch/kibana to be dynamic and read messages as they come in, rather than later ? More information is needed. How are you sending the messages to Elasticsearch? Is Logstash involved? Kibana relies on the @timestamp field to be UTC. If your logs are in UTC too it sounds like something is interpreting them as local time and adjusting the timestamp accordingly before updating @timestamp. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20141020072815.GA8014%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Error: No config files found: /etc/logstash/conf.d
On Monday, October 06, 2014 at 10:11 CEST, StueckJu j.stuec...@gmx.com wrote: Hey, when i start logstash as a service /etc/init.d/logstash start, i get the Error Message in the topic in file logstash.stdout. I have a logstash config file in the directory /etc/logstash/conf.d/server.conf. So i don't know why logstash doesnt find the file. Let's move this thread to logstash-users. Kept the elasticsearch list cc'd for now. How is Logstash being invoked? Is /etc/logstash/conf.d being passed with the -f (or --config) option? [...] -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20141006114922.GD3541%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: How does logstash chose which timestamped index to use?
On Tuesday, September 30, 2014 at 20:31 CEST, Matt Hughes hughes.m...@gmail.com wrote: I have a logstash-forwarder client sending events to lumberjack - elasticsearch to timestamped logstash indices. How does logstash decide what *day* index to put the document in. Does it look at @timestamp? Yes. @timestamp is just generated when the document is received, correct? So if you logged an event on a client at 11 pm UTC but it didn't make it to elasticsearch until 1am UTC the next day, which index would it go in? Would it go in the day it was created or would it go in the day it got to elasticsearch? If the latter, is there a way to force logstash to respect a date field in the original log event? You should use a 'date' filter to extract the date and time from a field in the log message and populate the @timestamp field. http://logstash.net/docs/1.4.2/filters/date This is really more of a Logstash question, and there's a separate group for that: logstash-us...@googlegroups.com -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140930184741.GA21595%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Copy IT Data to Local Server
On Sunday, September 28, 2014 at 18:48 CEST, naveen gayar navind...@gmail.com wrote: I wish to export the data from remote environment and import into my local server. Look into snapshots. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140929065220.GA31948%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: ES JsonParseException
On Thursday, September 18, 2014 at 12:40 CEST,
Foobar Geez foobarg...@gmail.com wrote:
Thanks. I provided a bad example as I guess I over-simplified it and
also edited it to remove proprietary data (thus, missed }).
The following example exhibits the same issue as described in my
original post.
curl -XPUT '[1]http://localhost:9200/test/test/test' -d '
{
rules: [
{
users : [ mile\kilo ]
}
]
}'
{error:RemoteTransportException[[High-Tech][inet[/X.X.X.X:9300]][ind
ex]]; nested: MapperParsingException[failed to parse [rules.users]];
nested: JsonParseException[Unrecognized character escape 'k' (code
107)\n at [Source: UNKNOWN; line: 5, column: 40]]; ,status:400}
As with many other languages, literal backslashes in string literals
need to be written \\.
curl -XPUT 'http://localhost:9200/test/test/test' -d '
{
rules: [
{
users : [ mile\\kilo ]
}
]
}'
--
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20140919064348.GE21271%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Re: Some indices failing with SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]
On Friday, September 12, 2014 at 08:53 CEST,
Kevin DeLand kevin.del...@gmail.com wrote:
Everything was working fine when all of a sudden some indices started
failing.
GET localhost:9200/logstash-2014.09.11/_search
yields response:
{error:SearchPhaseExecutionException[Failed to execute phase
[query], all shards failed],status:503}
How's the cluster's health? Anything interesting in the Elasticsearch
logs?
--
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20140912065653.GC3212%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch parse failure error
On Thursday, September 11, 2014 at 22:50 CEST,
shriyansh jain shriyanshaj...@gmail.com wrote:
I am using ELK stack and have a cluster of 2 elasticsearch nodes. When
I am querying Elasticsearch from kibana. I am getting the following
log error message in the elasticsearch log file.
http://pastebin.com/sD539SNZ
I am not able to figure out what is causing the error to happen. Any
input will greatly appreciated.
Quoting your gist:
... filtered:{query:{query_string:{query:tags:\sjc-array254\
AND proc\ AND cmd:\pmd\}},filter: ...
So, it looks like you're sending the following query:
tags:sjc-array254 AND proc AND cmd:pmd
There's a quote too many in there.
--
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20140912070341.GE3212%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Re: Discrete value aggregations on a URL field
On Friday, September 12, 2014 at 09:23 CEST, Ali Kheyrollahi alios...@gmail.com wrote: On Friday, 12 September 2014 08:18:19 UTC+1, Ali Kheyrollahi wrote: I am trying to find numbers of discrete value per URL in a day and the result is not what I expect. [...] Result is bizarre, I mean it breaks my URL into its segments and aggregates on that. Do I need to use Hash of the URL (I prefer not to)? OK, it seems that I need to use not_analyzed on the field. Is that correct? Yes. -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140912085425.GA9172%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Balance between number of indices and shards per index
I'm setting up an Elasticsearch-based log cluster and I'm having some doubts about how I should choose the number of indices and shards. By default, Logstash and Kibana use per-day indices and Elasticsearch defaults to five shards per index. I'm worried that this will create an excessive number of shards with a log retension of, say, 100 days. With one replica per shard I'd be facing 1000 shards cluster-wide. With three or four data nodes that's at least 250 shards per node. Whether this is too much obviously depends on the node and perhaps on the size of the daily indices, but regardless it doesn't seem particularly advantageous with such a number of shards. Would it make more sense to use week-based indices or reduce the number of (primary) shards per index to two or three to get the number of shards per node down towards or below 100? Or should I stop worrying? -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140708071656.GA26839%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
Re: Question about time based indexes/rolling indexes and eviction policies?
On Friday, May 23, 2014 at 20:13 CEST, John Smith java.dev@gmail.com wrote: #1 I have been reading around and some people suggest if doing log analytics to split the index based on time. Is this built in into Elastic search or does it mean I have to do it manual? I don't believe Elasticsearch itself understands date-based indices, but Logstash does. If manual PUT http://myhost:9200/myindex-(get-current-date-here)/SomeDoc/Id I'm pulling my data from SQL server and going to either use ETL or JDBC gatherer. I suppose the ETL process needs to consider the date and when it does it's index PUT to check and roll over the date so that a new index gets created? Yes. And my queries need to consider this also so they know that on each day they need to search the new index? Yes, unless you use an index alias like _all to search in all indices but that obviously has performance implication and in part voids the benefits of multiple indices. #2 is there such a thing as eviction policies? Basically is there a way to check if we are running out of diskspace and to either remove entries from the index or in the above case delete/archive indexes older then a few days? If disk space is your limiting factor you should find the curator script useful. You could also set the _ttl value of messages to have them automatically expire after a set time. https://github.com/elasticsearch/curator http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html -- Magnus Bäck| Software Engineer, Development Tools magnus.b...@sonymobile.com | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140526063906.GB16396%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
