Re: alerts from Kibana/ES

2014-06-02 Thread horst knete
Hi NF, we did also set up alerting with our zabbix-monitoring-system. What we use are simple Linux-Scripts using the command curl to search in given elasticsearch-indices. In the zabbix-system are build triggers that are runs the script on our elasticsearch-server and interpret the output

Re: is it possible to update snapshot with new lucene segment and restore

2014-06-02 Thread Chetana
Could please suggest a best option for merging index data stored in HDFS with the index data stored in ES node On Saturday, May 31, 2014 1:45:13 AM UTC+5:30, Binh Ly wrote: No I don't believe so. The snapshot data is not really a valid Lucene index, per se. It does contain segment files, but

Re: RFC 6902 requires variant type mapping

2014-06-02 Thread joergpra...@gmail.com
You'd have to use a plugin for such kind of operations, because vanilla ES does not support RFC 6902 I'm also interested in supporting HTTP PATCH by Elasticsearch, because this is a must have for modifying resources due to the rules of Linked Data Platform (LDP)

efficient merging of index data stored in heterogeneous repositories

2014-06-02 Thread Chetana
I have some index data stored in hdfs (ES compatible indexing) and rest in ES and also ES and hdfs clusters are different. Is there any other better alternative than below options 1. Bulk index request - but it is unnecessary as the data is already in indexed format 2. Move/copy periodically

Re: Optimizations for nested aggregations

2014-06-02 Thread nilsga
We're using the Java API. I assume that it is using a binary representation of some kind that is more compact than JSON. I just mentioned JSON to illustrate the size of the response. I'll certainly try to monitor disk/network activity. Nils-H On Friday, May 30, 2014 7:16:38 AM UTC+2, Otis

Using AND and OR terms with multi-match ?

2014-06-02 Thread Kristian Rosenvold
Fixing the to be or not to be stop word problem with multi-match, has the very undesirable effect of breaking user-entered AND and OR terms directly in the query string. Our users would really like both; is there a way to accomplish this ? Kristian -- You received this message because you

Kibana not loading certain dashboards after upgrades

2014-06-02 Thread Maarten Ureel
Hello, I recently upgraded all components of our logging system: logstash from 1.3 to 1.4, ES from 0.9 to 1.2 and Kibana from 3 RC1 to 3.1. Since that upgrade however I cannot load certain dashboards anymore. I though it might be because of the name (ID) as they often contain colons (:).

Configuring cross-cloud cluster via REST API

2014-06-02 Thread Martin Harris
Hi Folks, I'm trying to setup a cross-cloud elastic-search cluster. As it's cross-cloud, the usual discovery will not work, so I've started elasticsearch on three nodes with multicast disabled (discovery.zen.ping.multicast.enabled=false) and unicast enabled

Re: Configuring cross-cloud cluster via REST API

2014-06-02 Thread Itamar Syn-Hershko
This will not work, discovery is done on port 9300 (by default) and cannot be done over HTTP I think what you are looking for is tribe node - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-tribe.html You'll then have 2 separate clusters but you can talk to both as

Re: Kibana not loading certain dashboards after upgrades

2014-06-02 Thread Mark Walkom
You might have to edit the dashboards and update them as there is a number of changes between 3 and 3.1. We ran into this a few times, especially moving to 3.1. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 2 June

Re: Configuring cross-cloud cluster via REST API

2014-06-02 Thread Martin Harris
Thanks Itamar, The tribe setup might work, but it may be easier if I can (at runtime) simply give elasticsearch a new IP address to add to the cluster - this will be accessible on port 9300 so the communication will not be over HTTP. Is there any way to do this? Cheers Martin On 2 June 2014

Re: Configuring cross-cloud cluster via REST API

2014-06-02 Thread joergpra...@gmail.com
You have to restart the whole cluster. Switching discovery while running a cluster is not possible. Jörg On Mon, Jun 2, 2014 at 12:49 PM, Martin Harris martin.har...@cloudsoftcorp.com wrote: Hi Folks, I'm trying to setup a cross-cloud elastic-search cluster. As it's cross-cloud, the usual

Re: Configuring cross-cloud cluster via REST API

2014-06-02 Thread Itamar Syn-Hershko
Just enable multicast using the plugin for your cloud provider... -- Itamar Syn-Hershko http://code972.com | @synhershko https://twitter.com/synhershko Freelance Developer Consultant Author of RavenDB in Action http://manning.com/synhershko/ On Mon, Jun 2, 2014 at 2:08 PM, Martin Harris

Re: Kibana not loading certain dashboards after upgrades

2014-06-02 Thread Maarten Ureel
Yes, but how would I do that? Kibana won't display the dashboards.. also, when trying to retrieve the dashboard by it's ID from ElasticSearch, it should at least return a JSON array, right? It's not that Kibana is unable to parse the dashboard info, ES just doesn't return it anymore for certain

Re: Terms aggregation scripts running slower than expected

2014-06-02 Thread Guillermo Arias del Río
Thanks! That is a even a better solution. I have made some tests and it works. The buckets - and their order - are almost always the same. El miércoles, 9 de abril de 2014 21:36:16 UTC+2, Thomas S. escribió: Hi, I am currently exploring the option of using scripts with aggregations and I

Problem indexing historical log data to elasticsearch 1.2.0

2014-06-02 Thread 'Joey Morris' via elasticsearch
Hi All, I'm new to elasticsearch and I've got a datetime/indexing problem. I'm using Logstash 1.4.1, elasticsearch 1.2.0, and rabbitmq 3.3.1 in a Window server environment. My Shipper server - Server A - is setup with Logstash 1.4.1, elasticsearch 1.2.0. and amq Server A input is a simple

Synonym filter behavior for single word / multi words

2014-06-02 Thread Bernhardt Scherer
Hello there, I am currently trying out the synonym filter. Here are my settings: settings: { analysis: { filter: { nGram_filter: { type: nGram, min_gram: 2, max_gram: 15 }, synonym: { type: synonym, synonyms_path : analysis/synonym.txt, ignore_case: true, expand: true } }, analyzer: {

Elasticsearch aggregation script not maintaining document ID integrity

2014-06-02 Thread Benjamin Smith
I have a aggregation script which concatenates 2 field values. The field is a tag, and I am returning the `tag.name` and `tag.id`. Everything (almost) works as expected, and in the correct format. The issue is that the ID does not always match the name. Also, IDs are duplicated within the

Re: timestamp distribution hour of day

2014-06-02 Thread Kevin S
That is correct. Events between 2-3am even on different days would appear in the same bucket On Sunday, June 1, 2014 10:16:33 PM UTC-7, Patrick Proniewski wrote: hello, I don't see how an histogram on _timestamp could be a solution. If I understand correctly, Kevin is looking for a way

Kibana browser compatibility issues

2014-06-02 Thread InquiringMind
My first attempt at Kibana 3.1.0 was a little bumpy due to browser issues. After some reading, I performed the minimal download, unpack, and point the browser at index.html. If both Kibana and ES are on the same machine it will just work. Not quite! Contrary to all of the don't set this to

Sort descending with JavaScript API

2014-06-02 Thread el
This is a syntax question. In my search I want to sort descending on a field named 'no'. I tried things like params.sort = no: {order: 'desc'} or params.sort = [{'no': {'order': 'desc'}}] but nothings works. If I just specify params.sort = no the ascending sort works fine. Can somebody give

can we use different indices in a single elasticsearch

2014-06-02 Thread Ankit Mittal
Dear All, I want to use different indices in a single elasticsearch instance. Thanks, Ankit -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to

Re: timestamp distribution hour of day

2014-06-02 Thread Antonio Augusto Santos
I was looking for this myself the other day. You can get the results you want from ES with the following aggregation: { aggs: { hour: { terms: { script: doc['@timestamp'].date.hourOfDay } } } } For Kibana, you'd probably need a

Re: can we use different indices in a single elasticsearch

2014-06-02 Thread Nikolas Everett
Elasticsearch will run many indexes on the same cluster. The guide has more starting here: http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/distributed-cluster.html On Mon, Jun 2, 2014 at 12:29 PM, Ankit Mittal ankit.lnc...@gmail.com wrote: Dear All, I want to use different

Re: Kibana bettermap widget

2014-06-02 Thread Steven Pisarski
Thanks Binh for the reply, I cannot see any way ES would view the long/lat values as strings and had also tried other geoip standards to no avail. My biggest issue is the terrible or non-existent messages from ES or Kibana that could help shed light on the issue. I guess bettermap is truly

Re: Document found with _search but not with GET

2014-06-02 Thread Neil Andrassy
Hi, I have a feeling we're facing a similar issue (about which I posted separately). The behaviour is like the routing that applied before (in our case explicit on some indexes and just on _id for others) when the document was indexed, doesn't now resolve to the same shard following a cluster

Re: Managing Snapshot Files from Outside ES

2014-06-02 Thread David F. Severski
Thanks for this, Otis. For the archives, it looks like the indices subdirectory at the snapshot repo can be managed outside of ES, while the files at the top level of your repo (including the index and various metadata files) need to be left as is. If you move those top level files to Glacier,

Detective wanted: Green cluster but query timeout unless changing preference to local.

2014-06-02 Thread arthurX
I had debugging this problem for so long time now. I tweaked every relevant setting but just cannot get it to work. I really hope someone with knowledge about Elasticsearch networking can give me some help. So I got two machines in the same data-center, but communicate to each other by public

Re: Kibana bettermap widget

2014-06-02 Thread 'Binh Ly' via elasticsearch
Steve, if you can show a copy of your mapping here, I can take a quick look. curl localhost:9200/your_index/_mapping?pretty It should work fine, the only time it doesn't work for me is when I messed up in the mapping. -- You received this message because you are subscribed to the Google

Re: Kibana browser compatibility issues

2014-06-02 Thread 'Binh Ly' via elasticsearch
If you simply point the browser at the file system index.html, in my experience, that only works in Firefox (and only if you explicitly do http://server:9200;). The Kibana default assumes that you actually run Kibana from a web server (or as an ES site plugin if you prefer) and that ES is

Return related terms

2014-06-02 Thread Antonio Augusto Santos
Hi, Is it possible to do search that returns the terms that occur on a field that matches my search? For example, if I have some docs like: { field: a dog is a great animal } { field : my favorite animal are cats } { field: what a great animal is my dog } { field: my pet is a cat

Migration from Solr to ElasticSearch

2014-06-02 Thread Diego Marchi
Hello all, I'm testing the ES environment to see if a migration from Solr could bring benefits to our system. We are considering a complete renovation of our service, taking it from Java to Python plus a lot of new enhancements. Currently we use Solr for indexing purposes. We store webpages

Re: Elasticsearch aggregation script not maintaining document ID integrity

2014-06-02 Thread 'Binh Ly' via elasticsearch
Is it possible that you have a single document with tag.id = 352 and tag.name = Tag B? And at the same time another document with tag.id = 352 and tag.name = Tag A? I'd query the data just to be sure. -- You received this message because you are subscribed to the Google Groups elasticsearch

Re: Return related terms

2014-06-02 Thread 'Binh Ly' via elasticsearch
One possible way is to augment your search with a terms aggregation query on an analyzed version of the field you want to extract the terms from. Be careful though is this can be very very expensive if you have a field with lots of distinct terms. -- You received this message because you are

Best cluster environment for search

2014-06-02 Thread Marcelo Paes Rech
Hi guys, I'm looking for an article or a guide for the best cluster configuration. I read a lot of articles like change this configuration and you must create X shards per node but I didn't saw nothing like ElasticSearch Official guide for creating a cluster. What I would like to know are

Re: Synonym filter behavior for single word / multi words

2014-06-02 Thread 'Binh Ly' via elasticsearch
It seems to work fine for me (ES 1.2). Can you please post an full reproducible sequence of commands that I can execute to try? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it,

Re: Synonym filter behavior for single word / multi words

2014-06-02 Thread 'Binh Ly' via elasticsearch
What I mean by working: Input: Inbus Output: { tokens : [ { token : inbus, start_offset : 0, end_offset : 5, type : SYNONYM, position : 1 }, { token : innensechskant, start_offset : 0, end_offset : 5, type : SYNONYM, position : 1 }, { token :

Re: alerts from Kibana/ES

2014-06-02 Thread Otis Gospodnetic
Aha. FWIW, such saved queries functionality is being added to Logsene http://sematext.com/logsene/ as I type. Logsene is currently free if you want to play around with it. No such thing in Kibana, but of course there is Percolator. Otis -- Performance Monitoring * Log Analytics * Search

Troubleshooting cluster wide performance slow downs

2014-06-02 Thread Jeff Moyer
Hello, We have a 15 node cluster and have been experiencing intermittent cluster wide performance issues that last for between 1 and 5 minutes. ES 1.1.0 jdk1.7.0_51 We started seeing this after upgrading from .0.9.3 to .1.1.0, although we have also made many other changes around the same

Indexing data from Oracle database tables

2014-06-02 Thread Praison Selvaraj
What is the suggested approach to index records from tables in Oracle database? We are considering indexing the data just when the user logs in so that the data is available for search and do away with the index when the user session expires or logs out. We need to do this for around 15

Re: Migration from Solr to ElasticSearch

2014-06-02 Thread Otis Gospodnetic
Hi, You could migrate from Solr to ES without reindexing because at the end of the day it is Lucene that writes data to index. You'd want to make sure your ES mappings match your Solr schema. You'd want to create the matching number of shards and replicas you had in Solr(Cloud?). You'd manually

search parse exceptions on marvel dashboard

2014-06-02 Thread Nikhil Mulley
Hi, In my elasticsearch cluster, off late on the marvel dashboard there are searchparseexceptions. Any idea what are these below exceptions mean? They seem to be happening very recently and even restart of the elasticsearch service does not seem to help. Environment: ElasticSearch 0.90 +

Re: Mismatch versions ES AWS Plugin?

2014-06-02 Thread David Pilato
Indeed. We need to release it. In the mean time, you can build it by yourself and install it manually. Best -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 3 juin 2014 à 00:57, Ryan Tabora ratab...@gmail.com a écrit : Hey all, I am guessing I am running into a

Delete, rename , alias for types

2014-06-02 Thread srikanth ramineni
Hi, I want to add and modify few field types for a type. Is there any way to create alias to exiting type in index or rename the type in the index. Can any one provide me better way to do this. Scenario: i have two types in one index called contractindex below are two types contract

Configure connections using TransportClient

2014-06-02 Thread Srinath C
Hi, I wasn't able to figure out how to configure the number of connections that are opened up by a TransportClient to an ElasticSearch cluster. There was some information in one of the discussions https://groups.google.com/forum/#!topic/elasticsearch/7-ob2IeYnMI but they don't seem to work