Email notification on ERROR in log

Hi frnds, I have configured, Logstash + redis + indexer + elasticsearch + kibana in mysetup for logcollection. Using beaver + rsyslog agent to pushlogs to the logstash. The logs are being collected successfully. What I want to setup now if the email based alert. My requirement was If any spec

Re: Cluster discovery on Amazon EC2 problem - need urgent help

Zoran, good to hear it is working now. It should work pretty well with ec2 auto discovery - unicast is a good starting point but unless you are statically assigning them via cloud formation (or manually?), it may not be worth the trouble (and it stops you from dynamically scaling your cluster) -

Re: Cluster discovery on Amazon EC2 problem - need urgent help

Hi David, Thank you for your advices. It really helped me to solve the issue and make it works. At the end I had to leave these two: discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["10.185.210.54[9300-9400]","10.101.176.236[9300-9400]"] and to remove: network.pu

How to extend TermQuery to ''_all" field

hi all, In my case, fields are too much, I want use "_all" to simplify query parameters. As term query dsl, I want use like this, how to implement it? "term" : { "_all" : "kimchy" } Thanks. Haiwei -- You received this message because you are subscribed to the Google Groups "e

Announcing: A Wireshark dissector for elasticsearch

Hi Guys I have been working on a Wireshark dissector for elasticsearch. This allows you to more intelligently debug elasticsearch problems at the network level. I have been working in my own branch of Wireshark and will be getting it merged in the official distribution as soon as I can get some

Reroute API - Node does not match index include filters

I'm having trouble with my cluster in that it seems only 3 of my nodes (out of 5) are being utilized for primary shards. All replica shards remain unassigned also. In trying to resolve the issue I tried using the Reroute API however it gives me the following error (with explain on): explanatio

Re: GC issue

Thank you. It seems to happen once in 3mins, so it should be ok. thanks, Nara. On Fri, Oct 10, 2014 at 4:19 PM, Bruce Ritchie wrote: > Young generation GC happens a lot normally and it's normally not a concern > as it takes so little time per GC cycle. In my experience it's only a > concern if

Referencing multi-fields on indexing

I am trying to figure out a good way to support a use case where an IP field needs to be both analyzed and indexed for ipv4 range queries (as type 'ip'), and stored as-is (as string) for display purpose. It would seem like multi-field might work for this. But one trick is that it turns out that va

Re: [ANN] Elasticsearch Smart Chinese Analysis plugin 2.4.1 released

Great news, I'm glad that backward compatibility is important :) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To

Re: GC issue

Young generation GC happens a lot normally and it's normally not a concern as it takes so little time per GC cycle. In my experience it's only a concern if it's happening many many times per second which often indicates too small a young generation. > > It's the old generation GC cycles that you

Re: Cluster discovery on Amazon EC2 problem - need urgent help

Not sure but may be related to public/private IP. May be debug logs will give you more insights? -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 10 oct. 2014 à 22:40, Zoran Jeremic a écrit : > > Hi David, > > Thank you for your quick response. That was great guess about

Re: Cluster discovery on Amazon EC2 problem - need urgent help

Hi David, Thank you for your quick response. That was great guess about the space after ":". It was really something that made a problem, so I'm now a step forward. It seems that it's trying to establish the connection, but there are a plenty of exceptions stating that Nework is unreachable. Wh

any timeline on kibana 4.0 become stable release

I am currently on Kibana 3 and ElasticSearch 1.3. I am interested in upgrading to kibana 4.0 and Elastic Search 1.4. My manager is very concern with beta release, is there any estimate when Kibana 4.0 will be out of beta and become stable release? Is there a good documentation for kibana 4

GC issue

i have setup elasticsearch on a windows box and set ES_HEAP_SIZE to 10g out of total 28g. For some reason, GC keeps happening as soon as heap memory reaches around 250mb. Attached couple of images from BigDesk. Please let me know how to avoid this. ES version: 1.3.2 Java Version: VM name: Ja

Re: How many shards is to many shards per server on SSD?

On Friday, October 10, 2014 8:39:43 AM UTC-7, Jörg Prante wrote: > > RAM is 1000x faster than SSD. > I mean this is the big caveat isn't it? If you can fit your whole index in RAM, then great, go for it... but others have loads that can't it all in RAM at which point SSD becomes a decent opti

Re: Marvel license file/order number baked into a container

Boaz, .marvel-kibana now has a 'state-2' file inside it. and obv for now since i didnt restart or do anything of that nature i am not asked for the license details. I wonder if what I see is that .marvel-kibana is only stored with one primary and one replica and when i reload the cluster some

Java API "extended_bounds" Not Working?

Hi Guys, I'm trying to run a Date Histogram aggregation, with extended_bounds that cover (now-7d) -> now, and return 0 doc-count buckets. This works perfectly in Sense, but when running in Java, the response does not include my extended bounds. Strange part is, debugging the program reveals tha

best way to secure Elastic Search in window environment

Can some one help me how to secure elastic search hosted in windows environment like having api key or somethign else. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an em

Re: Kibana upgrade trouble - nor 4.0BETA1 neither 3.11 work now!

After all I've figured (with some help) that Kibana 4 uses node info API to check that ALL nodes in the cluster are at least 1.4.0.Beta1, The trick is that if you have Logstash talking to your ES, chances are it does so using "node" protocol, effectively pretending to be extra nodes in cluster w

GeoDistance search with dynamic distance

I have a product database, where products have a set delivery zones, represented in elasticsearch as an array: [ { distance: double (in miles), location: geohash (source location of product/seller), deliveryprice: double } ] does anyone have any idea how I can filter using the d

Refusal to recover after node rebuild

Hi, I've got a proof of concept cluster with 5 nodes. Several months rsyslog data is in there with 2 replicas per index. I then decided to rebuilt 2 nodes simultaneously. No problem. Cluster reallocated as expected and each of the remaining 3 nodes stored all of the indexes and replicas in

Bulk insert vs Single insert

Hi All, The primary dev managing our ES cluster has made the statement that single document writes to ES will only provide us with roughly 30 / 40 writes a second. Whereas the bulk operations will give us more in the range of a 1,000+. I realize that bulk is always faster (or is generally) an

Re: How many shards is to many shards per server on SSD?

On Thu, Oct 9, 2014 at 6:34 PM, Kevin Burton wrote: > > > On Wednesday, October 8, 2014 12:07:30 AM UTC-7, Jörg Prante wrote: >> >> With ES, you can go up to the bandwidth limit the OS allows for writing >> I/O (if you disable throttling etc.) >> >> This means, if you write to one shard, it can b

Re: How many indexes is too many indexes?

Thank you for the feedback guys, it is greatly appreciated. I had not thought about file descriptors so that gives me another thing to think about. Our daily volume will be pretty high across all of our users, I don't think we have a great estimate, but right now we are at about 50 million docu

completion type mapping

Hi mates, I am trying to create mapping in my auto completion scenario. As shown below, I am using "copy_to" command in order to copy a field value into a completion type which name is "factContent_suggest". The problem I have encountered is that I cannot update weight property by using "factC

Re: How many shards is to many shards per server on SSD?

RAM is 1000x faster than SSD. Elasticsearch loads the index into RAM when needed, so all searches are after loading index files into memory. Unless your index does not fit into virtual memory, you can assume that searching is done on a) RAM b) if not in RAM, you hit the file cache or c) you miss th

Performance with an arbitrary number of indicies

Lets say that I was providing a service to customers that requires customers to sign up to use my service. I'm also using elasticsearch to store analytics data about each customer. The number of documents recorded for each customer is completely arbitrary (some customers may have 1million + docu

Elasticsearch treatment of duplicate Netflow Records

Howdy folks, I have what I hope will be a fairly simple question for you. I'm going to be pumping netflow records into an ES cluster. In order to ensure that I catch all of these messages, I'm going to have multiple netflow destinations configured on the originating devices (routers, switches

Re: Turn on logging in live production

I don't believe there are plans for changing logging level on a specific node using the api. I'd file an issue with pretty much what you said. You might be able to limit the verbosity by just setting TRACE or DEBUG on the logger you need. Thats not as good as per node, but its something. On Fri

Re: Update merge settings pre-1.4 without downtime

Thanks for the response. However, you can't switch merge policies on the fly though if I'm not mistaken; if I remember right that definitely requires closing/reopening an index. From the docs ,

Re: how to update a object within array in elastic search

**I have got my answer, and its solution is** POST /twitter/twit/1/_update { "script": "item_to_remove = nil; foreach (item : ctx._source.list) { if (item['tweet_id'] == tweet_id) { item_to_remove=item; } } if (item_to_remove != nil) ctx._source.list.remove(item_to_remove);",

Re: Update existing values within a list or array in Elastic Search

**I have got my answer, and its solution is** POST /twitter/twit/1/_update { "script": "item_to_remove = nil; foreach (item : ctx._source.list) { if (item['tweet_id'] == tweet_id) { item_to_remove=item; } } if (item_to_remove != nil) ctx._source.list.remove(item_to_remove);",

Re: Sorting by nested fields

I suspect that because you have multiple levels of nesting you need to use the nested path (it's probably choosing marketplaces.prices, and you would want just market

Sorting on long float field doesnt work correctly (ES 1.3.4)

I have a problem with uncorrect sorting based on float type field. I have a float type field in my data called 'gtd_sort_number' (see data example below). Value is a microtime so its quite long with decimal values (as on example below). Now when I do a query like below (2nd example), the result

Re: ES eating all memory despite JVM startup configuration

Hello, I’ve a bit more information about that. My ES process creates thousands of threads and doesn’t seem either to close them nor even use most of them, until it eats all the memory and the process crashes (around 25k active threads). Here’s a dump of the thread_pool state 5 minutes after l

Re: Turn on logging in live production

Hi Nikolas, Thanks for instant reply . will it be supported in coming releases ? Since only few nodes are getting loaded in out es cluster , in that case i need to watch logs on those nodes alone. Also changing log level in entire cluster affects client node too . Enabling TRACE level in cli

Re: Turn on logging in live production

Not without restarting it. On Oct 10, 2014 7:13 AM, "Anantha Govindarajan" < ananthagovindara...@gmail.com> wrote: > Hi Nikolas, > > Is is possible to change log level to specific node ? This will be useful > incase of heavy indexing clusters. > > -- > You received this message because you are sub

Re: Update merge settings pre-1.4 without downtime

You could try switching merge policies and then switching back. I never tried that but I think it might work. Nik On Oct 10, 2014 12:33 AM, "Jonathan Foy" wrote: > Hello > > Is there any way of changing the merge settings of a live index without > downtime in ES versions prior to 1.4 (I'm on 1.1

Re: Turn on logging in live production

Hi Nikolas, Is is possible to change log level to specific node ? This will be useful incase of heavy indexing clusters. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an

Re: Curator issue on ES 1.4.0.Beta1

Hi Tanuj, This has already come up in https://github.com/elasticsearch/curator/issues/183 A fix is already in master: https://github.com/untergeek/expire-logs/commit/685016295be83f14d4cad6ab9577eb1461ad9621 On Friday, October 10, 2014 5:03:35 AM UTC-5, tanuj bansal wrote: > > Hi All: > > I h

Curator issue on ES 1.4.0.Beta1

Hi All: I have elasticsearch-1.4.0.Beta1-1.noarch deployed on my system and while using curator I get the following error Traceback (most recent call last): File "/usr/bin/curator", line 9, in load_entry_point('elasticsearch-curator==1.1.2', 'console_scripts', 'curator')() File "/us

Error when update number_of_replicas

Hi, I am following documentation explained via http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html When I run this command, I got acknowledged= True, followed by error messages below. C:\>curl -XPUT "localhost:9200/qsearch_perf/_settings" -d ' { "i

1 cluster, multiple replicas with different synch settings

Is it feasible to configure a cluster, with multiple nodes, with more than 1 replica, and use the cluster.routing.allocation.awareness.attributes option to configure: a) Configure replica1 across a series of racks within the datacentre b) Configure replica2 as distinct zones within two differen

Re: How to index Office files? *.txt and *.pdf are working...

It was fixed in release 2.3.2 (ES 1.3.x): https://github.com/elasticsearch/elasticsearch-mapper-attachments/issues/82 For 2.2.1 (ES 1.2.x) try to patch and rebuild the plugin: https://github.com/elasticsearch/elasticsearch-mapper-attachments/commit/87b38c54eb0c40185a507a3dea64ee06060e4d2d Am Don

search on particular field gives no results

I have a document indexed whose structure is as follows: { "_index": "yelp", "_type": "user", "_id": "ABC", "_score": 1, "_source": { "yelping_since": "2007-07", "votes": {}, "review_count": 1798, "name": "TEST_USER", "user_id": "123", "friends": [], "fans": 8

Cluster Configuration.

Hi, How can i achieve following. I have 5 nodes with 30GB RAM in it. I want to create Index with 5 shards and 1 replica. I want all primary shards to be in memory and replica on disk. Also, i want to disable search against replica. So, when user search query goes against shards in memory

[ANN] Elasticsearch Servlet Transport plugin 2.4.0 released

Heya, We are pleased to announce the release of the Elasticsearch Servlet Transport plugin, version 2.4.0. The wares transport plugin allows to use the REST interface over servlets.. https://github.com/elasticsearch/elasticsearch-transport-wares/ Release Notes - elasticsearch-transport-wares -