Re: Setting up alerts

Hello Joshua , Percolater is the usual choice here. You can register queries against poercolator and when you index some feed , the peorcolater informs you that such a search query matches against your index. This information is passed when indexing. With logstash , I am not sure how this can be

Re: Wildcard search / query via REST request URI

URI Search is mapped to query string See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-uri-request.html And http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html Although it should work, you should not use that in

Setting up alerts

I am working on a demo using Elasticsearch, Logstash, Kibana and one of the key features that I am looking for is the ability to setup alerts to send out emails. Specifically I want to setup an alert to be emailed when a log is recorded with "severity=error" and "category=category1" occurs more

Re: EsRejectedExecutionException[rejected execution (queue capacity 50)

The queue sizes are defined in the threadpool section of the configuration. IIRC, they cannot be updated dynamically: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html -- Ivan On Fri, May 30, 2014 at 2:56 PM, srikanth ramineni wrote: > Hi , > > I a

Re: Elasticsearch 1.20 and 1.1.2

Jörg thanks for the heads up about XContentRestResponse going away. I've run into that as an issue with a river I help maintain. Do you know what the new recommended alternative is? Thanks, Ben On Thursday, May 22, 2014 3:47:25 PM UTC-7, Mark Walkom wrote: > > Hurray! > > However they are stil

Re: questions on custom routing

1) ES will use your routing value to do indexing and searches and narrow it down directly (specifically helpful for searches) to a single shard. 2) If you produce your own document ID, then ES will just use it without any issues. -- You received this message because you are subscribed to the G

Re: Marvel not showing nodes stats

Are there any marvel errors on one of the data nodes that marvel is getting the data from? Usually you error means marvel could not ship the data over to the monitoring cluster. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe f

Re: How to know the query commands that ES server accepted? Any flag?

If you mean a list of all the search queries executed, there is not currently a clean way to do this. You can kinda fudge it by enabling the query slowlog in the elasticsearch.yml file, but if you have many nodes, you still have to go to each node that contains the shards that were hit to retri

"Sense Sheets", create/use multiple sense pages

Hi guys, I have googled for a while and could not find anything on the subject. I have several projects using Elasticsearch and more often than I would like, my sense becomes HELL. All kinds of tests for all kinds of projects get mixed together. I started separating in json files, but it was a s

Re: Trying to fetch document ids with a geohash_grid aggregation

1) The upcoming top_hits aggregation "may" help here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-top-hits-aggregation.html 2) If you do localhost:9200/_search?search_type=count, the hits will not be returned in the most efficient manner poss

Re: EsRejectedExecutionException[rejected execution (queue capacity 50)

Hi , I am getting below es rejection exception. problem updating content indexing for entity: 85539735340578996965234585294218135410438591031260132420 error: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardRepli

Re: Script field coercion Out of range

Is it possible that you have different JDK versions? I tried this on 1.7u25 and it gave me the error, but not on 1.7u55. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an

Re: Red status unassigned shards help

You can set the replicas for an index using the API (or kopf). As for your upgrade concerns, see http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campai

Re: scroll aggregations

Unfortunately not at the moment. However, you could look into spreading the data around with more shards/nodes (thus lesser memory requirements per node), or add more RAM, or possibly use disk-based fielddata: http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/ -- You rec

Re: Kibana "bettermap" widget

The field has to be a numeric type (for example, double) containing 2 values in lon, lat sequence. Try double checking your mapping to make sure it is not a string or something else. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscr

Re: Improving a slow running Match_All Query

Yes, the match_all keeps taking that time. It hasn't improved after the first few queries. I did not run the Optimize command since we were in the middle of Indexing. I can run it now by setting the max_num_segments to 1. On Friday, May 30, 2014 1:52:55 PM UTC-7, Jörg Prante wrote: > > Is "matc

Re: Run native script on non-data node

I don't believe so. If you access a native script from your search and it's not on your data node, you will get an error. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an

Re: IDF per customer, many customers per index - best practices

Hi Jörg, Thanks for your quick answer. I was not aware of this IDF calculation per shard in regular queries, but it makes sense - one more scatter-gather phase is required for the global stats. I'll probably end up with putting many (if possible similar) customers on a single index to make "ava

Re: Improving a slow running Match_All Query

Is "match_all" always running at that time or is it getting faster after a first run? Did you run an optimize with maximum number of segments? What is your segment count? Jörg On Fri, May 30, 2014 at 9:20 PM, wrote: > *Bump* > > > On Wednesday, May 28, 2014 4:10:26 PM UTC-7, sai...@roblox.com

Re: is it possible to update snapshot with new lucene segment and restore

No I don't believe so. The snapshot data is not really a "valid" Lucene index, per se. It does contain segment files, but they are named and packaged in a specific manner that it would be best not to mess with them. ;) -- You received this message because you are subscribed to the Google Group

Re: Filtering *before* a query

Interesting, when I hit refresh I can see the strategy section. Nevertheless, it seems that when I set the 'strategy' equal to 'leap_frog_filter_first' I achieve better performance. This seems to be what I needed. Thank you! On Fri, May 30, 2014 at 11:45 AM, Ivan Brusic wrote: > For some reaso

Re: What is the difference between common terms query vs match query with cutoff_frequency set

They should produce the same exact query (CommonTermsQuery). The only thing I can think of that is different is the CommonTerms query provides more options other than just the cutoff_frequency (i.e. low_freq_operator, high_freq_operator, etc) which are not available in the Match query. I do not

Re: How to delete all entries based on the contents of two fields

With some tips from whack in the #logstash irc channel, I was able to delete what I wanted with this: curl -XDELETE 'http://node1:9200/logstash-2014.05.27/_query' -d '{ > "query_string" : { > "query" : "path:\"folderLogFileLoadedFrom\"" > } > } > ' > On Thursday, May 29, 20

Re: Improving a slow running Match_All Query

*Bump* On Wednesday, May 28, 2014 4:10:26 PM UTC-7, sai...@roblox.com wrote: > > Hello, > > The queries that we run seem to be very CPU Intensive and cause the > Servers to max out within a short amount of time. On debugging, it looks > like standard queries take too long to respond too. > > We

Need suggestions on type of query to be used for a given analysis for better results?

Hi, I'm using following analyzers for indexing my documents in ES: "analysis" : { "analyzer" : { "str_search_analyzer" : { "tokenizer" : "standard", "filter" : ["lowercase","asciifolding"]

Re: Error "Failed to derive xcontent"

It seems to work for me. I'd try typing this character by character straight into the command line just to make sure nothing "invisible" is interfering with the text. This sometimes happens when I copy-paste from a different app into the shell command line. -- You received this message because

Wildcard search / query via REST request URI

Hi, Is it possible to do a wildcard search via the request URI? For example http://localhost:9200/index1/type1/_search?q=type1.text_ID=* If yes, what is the correct format? I tried the above format and did not get any hit for my data. I'm new to ElasticSearch. So, please pardon me if this qu

Re: ClassCastException on Sort

This only happens when we sort by "Id" field. is there something special with field Name as "Id". On Tuesday, 27 May 2014 14:55:18 UTC-7, VB wrote: > > We are using 90.11 and we have a use case where have following type > >- accountsearch: { > - dynamic: strict > - properties: {

Re: Is it possible to get a bucketed aggregation based on the count of values for a field?

I don't believe this is possible at the moment. If you can pre-process your data and produce this summarization indexed into ES: Sara: 3 Mike: 2 John: 1 Then you can use the range (or filter) agg as you already mentioned. -- You received this message because you are subscribed to the Google Gr

Error "Failed to derive xcontent"

I am a newbie to ElasticSearch and tried this first one via curl curl -XPUT "http://localhost:9200/movies/movie/1"; -d' { "title": "The Godfather", "director": "Francis Ford Coppola", "year": 1972, "genres": ["Crime", "Drama"] }' But I am getting the following error {"error":"Mappe

Is it possible to get a bucketed aggregation based on the count of values for a field?

For example, assume I have the following docs: {user:"Mike"} {user:"John"} {user:"Mike"} {user:"Sara"} {user:"Sara"} {user:"Sara"} I can do a terms agg on user and get: Sara: 3 Mike: 2 John: 1 What if I didn't care about the actual total number of terms per value, and instead just wanted them b

Re: Filtering *before* a query

For some reason, when I viewed that page at work, I was not seeing the strategy section either, but I was able to at home. Try refreshing the page, worked for me. Perhaps I should play around with those settings the next time I fine tune my queries. I use a combination of both pre and post filter

Re: Filtering *before* a query

Hi Adrien, Thanks. This sounds like what I need, however the page you link to does not discuss the 'strategy' parameter that you mention. I see some documentation about a filter strategy with Lucene ( http://lucene.apache.org/core/4_1_0/core/org/apache/lucene/search/FilteredQuery.FilterStrategy.h

Re: Max latency between nodes

We're still seeing node drops, and what is more bizzare is we're seeing this on a test cluster we stood up that actually has no activity on it (no reads or writes going to it). Does anyone have any additional thoughts? Here is the info from the configuration and the logs we're seeing on the d

Re: What is the difference between common terms query vs match query with cutoff_frequency set

Bump On Monday, May 12, 2014 4:22:51 PM UTC-4, Mike wrote: > > I was reading up on the match query and noticed that it has a > cutoff_frequency parameter, which seems to do pretty much what the common > terms query does. > >1. What is the difference between the common and match queries? >

Re: Failing unit tests on a fresh fork

When forking the master, things are expected to be quite volatile, and you should allow the ES core team a few hours or even days to let the dust settle down. Don't worry too much if things are temporarily broken on master, for stable builds, there are tagged releases... Jörg On Fri, May 30, 201

Re: Aggs

Hi Bogdan, This is not possible since aggregations require you to provide the fields on which to execute aggregations explicitely. An option could be to change the way that you model your document to leverage nested documents, so that your documents would like { "properties": [ { "name": "ke

Re: Failing unit tests on a fresh fork

Thanks, just did that. Incidentally, in this particular case the fix was already made while I was opening the issue :-) On Friday, May 30, 2014 3:24:39 PM UTC+2, Nikolas Everett wrote: > > Normally tests aren't broken but with randomized testing it is possible > for things to sneak through. The

Re: [ANN] Elasticsearch experimental highlighter

Hi Bruce, I'm not actually sure it'll work on 0.90.X - I didn't start working on it until 1.1.0. "Its pretty quick" means lots of things, unfortunately. If you configure it to segment the source like the postings highlighter it is typically about 10% slower then the posting highlighter. If you

Re: Adding NGram to language analyzer

On Thu, May 29, 2014 at 4:05 PM, Panagiotis Nikitopoulos < panosbo...@gmail.com> wrote: > I have the exact same problem with greek language. > Have you figured out how to solve it? > Thanks! > > > First build a copy of the greek analyzer as a custom analyzer. Have a look at https://gerrit.wikimed

Re: Red status unassigned shards help

Thanks Mark and pawan, Here is my output from netstat: tcp6 0 0 :::9200 :::* LISTEN 1155/java Mark are you talking about upgrading to the lastest 0.9 or to 1.x.x? Still waiting on a good method to go to the lastest 1.x in ES with out mes

Re: What OS memory does es use other than Java?

On Thu, May 29, 2014 at 8:16 PM, Edward Sargisson wrote: > > Our fix was to configure VMWare to reserve the entire configured memory. > This means that the host doesn't try to take the memory back. It seemed > sensible to reserve all of the configured memory as we want elasticsearch > to keep its

Re: Hide some system fields

Thanks a lot! i'll try this. Also i need _timestamp field into output. I tried several ways, but to no avail. пятница, 30 мая 2014 г., 2:34:47 UTC+4 пользователь Gail Long написал: > > You can use jq to selectively pull JSON values, arrays, and objects from > elasticsearch returns. It lets you

Re: Interpolation of discovery.zen.ping.unicast.hosts

Yes that works. Docs could really use work there though. The same string in elasticsearch.yml is ["one", "two", "three"], so I assumed I'd need to pass in brackets. On Thursday, May 29, 2014 4:09:02 PM UTC-4, InquiringMind wrote: > > I believe that the host names must be comma-separated and

Re: Failing unit tests on a fresh fork

Normally tests aren't broken but with randomized testing it is possible for things to sneak through. The best thing to do in this case is to submit an issue with the stack trace and the failing test. The stack trace will contain the randomized seed that'll be needed to reproduce the failure. Nik

Re: IDF per customer, many customers per index - best practices

IDF is calculated per shard, and only in DFS search types, it is calculated over all nodes in an initial scatter phase. http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_search_options.html#_literal_search_type_literal If you are concerned about IDF in a single multi-user index p

Failing unit tests on a fresh fork

Hi, I forked Elasticsearch in order to test a possible bugfix and followed the instructions here . Unfortunately, running the unit tests of the fresh fork (commit 0e2d33b4a446af033db48284737a62b9cc8c99bb) via ES_TEST_LOCAL=true mvn

is it possible to update snapshot with new lucene segment and restore

I have a requirment where for some data setting the '_source' in indexrequest is strightforward but for some huge amount of data I need to run long running activity to generate data which needs to be indexed. So frequently I plan to take a snapshot from ES to Hadoop and want to add new lucene

Re: Random node disconnects in Azure, no resource issues as near as I can tell

The three nodes are connected by an Azure virtual network. They are all part of a single cloud service, operating in a load balanced set. I am not currently using any kind of FQDN, so the unicast host names are "es-machine-1", "es-machine-2" etc. No domain suffix whatsoever. As far as I know

IDF per customer, many customers per index - best practices

Dear ElasticSearch Community, There are many sources over the internet which recommend putting many customers into one index. One example is the Shay Banon's talk given at Berlin Buzzwords [1]. This approach has many advantages and the alternative - one customer per index seems like a huge over

Re: Reverse nested aggregation parsing error

Hi Stephan, The bug has been fixed, can you try out a new build? Martijn On 30 May 2014 09:45, Martijn v Groningen wrote: > This is indeed a bug, thanks for sharing it! This should be easy to fix. > > > On 27 May 2014 11:35, wrote: > >> Hello, >> I tried the new top_hots aggregation and made

Re: Elasticsearch and Smile encoded JSON

LZF compression is always enabled on the transport layer. http://wiki.fasterxml.com/SmileFormatSpec describes that although compression within SMILE is possible there is no compression scheme included in SMILE. My idea is to disable compression for SMILE / CBOR - both come with a serialization ov

Aggs

Please tell me if it is posible to make dynamic aggregation on all inner proprietes of a field, or nested object, because i have many product every of this product have different attributes, so there is posible to make automatically aggs without knowing before what attributes field are named? -

Re: Calculating sum of nested fields with date_histogram aggregation

Indeed, your aggregation runs in the context of the root document. You need to use a nested aggregation to tell Elasticsearch to use your nested field as a context: "aggs": { "transactions": { "nested": { "path": "associated_transactions" }, "aggs": { "revenue": {

Elasticsearch restart deleted all indices

1. When i restarted a node yesterday, this happened (logs follow)- 2. i lost 96 days worth of my logs. df before and after the restart indicated that i had lost ~260G of indices post install. 3. After this, the node started syncing logs from the very beginning from the rest of the cluster. 4. Spe

Routing instability after upgrading from 1.1.1 to 1.2

Hi, We have mapping templates set up to drive custom routing based on a field in our doc. The index and search seemed to be working fine under 1.1.1, but following an upgrade to 1.2 a percentage of our searches are not returning the expected documents - they actually return nothing as we're fil

Configuring esFactory host dynamically in AngularJs app

I have an AngularJS app using elasticsearch . It works well, but I have a problem deploying to production. I have a service that looks like this: angular.module('components.common.elasticSearchFactory',[ ]) .service('ElasticSearch', function (esFactory, Config) { function getHost(){ return

Re: ElasticSearch, Save data in a file. Thank you!!

I might be wrong but I think I already answered here:  https://groups.google.com/d/msgid/elasticsearch/F6B62E3D-B5FB-49CA-B536-E1330BAF2EC6%40pilato.fr Was that someone else or the same Fran? --  David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 30 mai 2014 à

Implicit Custom Filter?

I am trying to set up document-level security for my index. The documents have fields which will be filtered on to enforce access permissions. My question is: given a query, is it possible to set things up so that ES will invoke a custom script filter on *every* clause in said query without mungin

Search issue with snowball stemmer

Hello everyone, I have follow index mapping: and I posted the next content: When I make this one request: /http://epbyvitw0052:9200/some_content/docs/_search?q=sampling/ I'm getting result: but when I send request without type: /http://epbyvitw0052:9200/some_content/_search?q=sampling/

Run native java script only on non-data node

I can run my native java script on data nodes. (which need to be loaded in each nodes). I tried to load it only on non-data node, but it's not working. So I wonder is it possible to load only on non-data node and make it work. Thanks! -- View this message in context: http://elasticsearch-users.

Calculating sum of nested fields with date_histogram aggregation

Hello, I have a mapping that looks like this: "client" : { // various irrelevant stuff here... "associated_transactions" : { "type" : "nested", "include_in_parent" : true, "properties" : { "amount" : { "type" : "double" }, "effective_at" : { "typ

ElasticSearch, Save data in a file. Thank you!!

Hello everyone, I am new using ElasticSearch and I don`t have enough idea. I will be grateful if someone could tell me how to save the information collected from the twitter river into a file. Than you!, Fran -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/El

Re: Reverse nested aggregation parsing error

This is indeed a bug, thanks for sharing it! This should be easy to fix. On 27 May 2014 11:35, wrote: > Hello, > I tried the new top_hots aggregation and made it work on denormalized data. > > However, when I tries to add a filter I ran into the following exception: > > [2014-05-27 11:32:12,869