Elasticsearch testing question

For testing Elasticsearch, have you ever thought of doing two runs of the junit4 plugin? One to grab the fast unit tests first and one to grab the integration tests? I know most of the tests are integration tests but you might be able to squeeze some speed out of the process by parallelizing the

Re: Need some input/advice on scripting approach

On Thu, Apr 3, 2014 at 4:07 PM, Greg Lappen wrote: > Hello - I am an elasticsearch newbie and after doing a lot of reading, I > think I need to use scripts to accomplish my objectives. Basically, we are > indexing and searching documents already, and now we need to do a > non-trivial financial c

Re: Redistribute shards before shutting down

On Thu, Apr 3, 2014 at 3:36 PM, Mikhail Simin wrote: Hello! > > Is there a way to instruct a single ES node to actively upload its shards > to the rest of the farm? > See the last example here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-cluster.html#allocation

Re: Rolling restart of a cluster?

I'm not sure what is up but my advice is to make sure you read the cluster state from the node you are restarting. That'll make sure it is up in the first place and you'll get that node's view of the cluster. Nik On Wed, Apr 2, 2014 at 2:08 PM, Mike Deeks wrote: > That is exactly what I'm do

Re: Rolling restart of a cluster?

I just used this to upgrade our labs environment a couple of days ago: #!/bin/bash export prefix=deployment-elastic0 export suffix=.eqiad.wmflabs rm -f servers for i in {1..4}; do echo $prefix$i$suffix >> servers done cat << __commands__ > /tmp/commands wget https://download.elasticsearch.or

Re: Sudden 100% CPU usage by ElasticSearch

Threads look like four concurrent scans but I can't be sure what they are up to. Nik On Thu, Mar 27, 2014 at 4:02 PM, Jos Kraaijeveld wrote: > Apologies, the gist link wasn't working properly: > > https://gist.github.com/Kaidence/2b95c207f4e6a79841c5

Re: Confusing highlight result when creating many tokens

Your confusing query is actually broken up into the following query: filtered(((md5:ba0 md5:ba0d md5:ba0d7 md5:ba0d72 md5:ba0d722 md5:ba0d722f md5:ba0d722f0 md5:ba0d722f04 md5:ba0d722f049 md5:ba0d722f0493 md5:ba0d722f0493f md5:ba0d722f0493f9 md5:ba0d722f0493f98 md5:ba0d722f0493f986 md5:ba0d722f049

I'd love an extra pair of eyes on my strategy to ween myself off the Fast Vector Highlighter

I think I've come up with a way to ween myself off of the Fast Vector Highlighter: https://github.com/nik9000/expiremental-highlighter It: * Doesn't need offsets in postings or term enums with offsets but can use either to speed itself up. * Can fragment like the Postings Highlighter, the Fast Vec

Re: Elasticsearch backend?

You might be wondering if it is using Tomcat or Glassfish or something too. The answer is not usually. There is a plugin that will let you install it in a servlet container but most folks just run it as a standalone service. It has an init script and stuff. Nik On Thu, Mar 20, 2014 at 1:02 P

Re: Some questions on wikipedia river and cluster config

The wikipedia river has a really inefficient regex or two and it spends most of its time munging wikitext into text. If you Java then you might want to dig into it. I've never had the time unfortunately. Depending on what you need you can get the job done with bash and perl. If you just need th

Re: precise field matching without defining a not_analyzed extra field possible?

Missed your last email. Ignore my suggestion and use the raw field:) On Thu, Mar 13, 2014 at 8:50 AM, Nikolas Everett wrote: > If you plan to do this frequently then go with the raw field. It'll be > faster. > > If you want to fool around without changing any mappings

Re: precise field matching without defining a not_analyzed extra field possible?

If you plan to do this frequently then go with the raw field. It'll be faster. If you want to fool around without changing any mappings then use a script filter to get the field from the _source. It isn't efficient at all. I'd suggest guarding it with a more efficient filter. Like so: "que

Re: Multilingual Search Plugin

t;> http://wortschatz.uni-leipzig.de/~cbiemann/software/toolbox/ are under >> GPL, so I released the decompound plugin under GPL too. Maybe >> redistribution of ASV binary tree files is possible even under ASL, but I >> am no lawyer. Reimplementation of a training CLI tool u

Re: Multilingual Search Plugin

t open-source. (simply playing > the devil's advocate). > > Cheers, > > Ivan > > > On Wed, Mar 12, 2014 at 10:24 AM, Nikolas Everett wrote: > >> Oh I'd love so much to have a great multi language analyzer! You don't >> happen to release portions

Re: function_score with decay functions and script score

I've had a lot of luck stuffing script scores into the rescore. It helps if one the scripts is slower then you'd like. You do have to rely on the query string being "pretty good" at finding the results that you want though. Since all you want is to multiply you can do it by using a rescore in sc

Re: Multilingual Search Plugin

Oh I'd love so much to have a great multi language analyzer! You don't happen to release portions of the thing open source? I'm obligated to only use open source software. Nik On Wed, Mar 12, 2014 at 10:07 AM, wrote: > If anyone is looking to do multilingual search, take a look at this simpl

Re: Lucene query parser to ES Java API

Yeah, don't try to wrap the query_string in piles of regexes to make sure it never throws errors and blocks features you don't want but lets through features you do. It isn't fun. On Wed, Mar 12, 2014 at 6:36 AM, Clinton Gormley wrote: > You may want to look at using the simple query string que

Re: Rolling back from 1.0 to 0.90.12 - Will data be affected?

There was a lucene upgrade in there so I don't believe you can go back. If you haven't written anything or optimized then maybe? Unfortunately I think your best bet it to fix the code Look at source filtering which will return something more familiar. Sorry, Nik On Fri, Mar 7, 2014 at 10

During upgrade to 1.0.1 secondaries didn't restore from disk

Last night we upgraded from 0.90.10 to 1.0.1 and it went really well except that the secondaries recovered very slowly. I _thought_ that because disabled writes (and reads too) that the secondaries would be able to recover from disk as well. Am I missing something? Is this a peculiarity of cross

Re: Urgent - No shards assigning

Doesn't node.data: false mean don't put anydata on this node? Nik On Wed, Mar 5, 2014 at 10:54 AM, Aaron Troy wrote: > Hi all, > > I have a system set up with 12 nodes, 48 indexes, and 12 shards/index (I > know that number is way too high). > > Anyways, I'm in the process of indexing all the d

Re: [Ann] Elasticsearch Image Plugin 1.0.0 released

Oh man do I have some images So I see something about a hash for large datasets but no reference to what the storage requirements are like either for the hash or for the linear search. Do you have some links? I'm working with on order of ten million images and, while I don't have a mandate t

Re: Elasticsearch database.

On Wed, Mar 5, 2014 at 4:41 AM, san wrote: > what is maximum size of logs that can be stored in Elasticsearch? > Each document can be pretty big. Some of mine are in tens of megabyte dumps from pdf files. Examples: https://commons.wikimedia.org/w/index.php?title=Special%3ASearch&profile=defaul

Re: Elasticsearch Performance Analysis

To give you a sense - we have 16 elasticsearch nodes with "slow" disks and easily push ~5000 docs/second without many optimizations. To get that particular number we use: 1. 10 processes to push the data (PHP so no threads) 2. The bulk api 3. We set the refresh interval to -1 We could go a ton

Re: Question on script scoring

First, .empty is super slow right now: https://github.com/elasticsearch/elasticsearch/issues/5086 Second, be paranoid about order of operations with MVEL. I'd play around with parrens until it worked. Third, multiply is the default boost mode so you can just leave out the _score which might make t

Re: How to get multiple corrections with phrase suggester?

Set max_errors to 2. Numbers greater than two can get quite slow so try to stick with 1 or 2. On Sun, Mar 2, 2014 at 9:28 PM, Alexey Kotlyarov wrote: > I want to correct multiple terms in a phrase at a time using the phrase > suggester. However, ES only suggests changing one _or_ the other, and

Re: Single thread with high CPU usage

Check to see how much GC you are doing when it spikes. If it is high, try to clear the cache: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-clearcache.html I'd try clearing each cache one at a time to see which one helps. If that is the problem you can configure El

Re: Elasticsearch Reverse Suggester Problem

esults. > > > On Tuesday, February 25, 2014 8:00:14 PM UTC, Nikolas Everett wrote: > >> I believe the job of the reverse filter is to efficiently provide >> suggestions that share a suffix with the provided term rather than a >> prefix. You might try removing the pre

Re: scalability and creating 1 index per user

On Tue, Feb 25, 2014 at 4:46 PM, ESUser wrote: > Hi All, > I am exploring elastic search to create one index per user instead of one > big index for all the users. Each index would be about 6G. > I am wondering if anyone has tried it and how would it scale? > > I couldn't find that elastic search

Re: Elasticsearch Reverse Suggester Problem

I believe the job of the reverse filter is to efficiently provide suggestions that share a suffix with the provided term rather than a prefix. You might try removing the pre_filter to see if it handles reversed words. The reason for the reverse index for the suffix is that lucene stores terms in

Re: Zero Downtime Reindexing

Here is how I do it: 1. Have index called foo_1392831890 with alias foo pointing to it 2. Create index called foo_1392841890 with new config 3. Scan/scroll everything from the foo alias into foo_1392841890. 4. Swap alias. Time has now warped backwards. 5. Run script to reindex everything tha

Re: Odd hot MVEL

On Tue, Feb 11, 2014 at 11:26 AM, Ivan Brusic wrote: > Great catch. Which Elasticsearch version and which JDK? > > Thankfully my documents are uniform, so I have been able to skip isEmpty > checks. > I believe I started seeing it on 0.90.6. I'm running 0.90.10 in production now and see it. I v

Re: Odd hot MVEL

is the key here. 3. If most of the documents don't have a foo then this is really slow. Like, two seconds slow. 4. Instead, switch to this (doc['foo'].isEmpty() ? 0 : doc['foo'].value) * doc['bar']. That is faster. .36 seconds or so. Not super speedy, but much

Re: [Java API] Is QueryBuiler thread safe?

None of the builders are thread safe, no. I believe they are all quite light weight though. On Mon, Feb 10, 2014 at 9:25 PM, Arinto Murdopo wrote: > Hi all, > > Is QueryBuilder implementation thread-safe? > I plan to use one static QueryBuilder class that is shared by several > threads. > > Be

Re: ElasticSearch while browsing an index that is updated every couple of minutes

. > > On Friday, February 7, 2014 1:51:49 PM UTC-8, Nikolas Everett wrote: > >> In past lives I was involved in integrating Solr to power all kinds of >> funky little "shelves" on a page. Elasticsearch can handle it. The trick >> is to make sure that you don&

Re: ElasticSearch while browsing an index that is updated every couple of minutes

In past lives I was involved in integrating Solr to power all kinds of funky little "shelves" on a page. Elasticsearch can handle it. The trick is to make sure that you don't do too many slow thing against too many documents. If the shelf is built by matches and sorting (by a field or relevance)

Re: Extending Thai analyzer.

If you don't like the language analyzer you have to rebuild it as a custom analyzer then add what you need to it. { "analyzer": { "thai_with_ngram": { "type": "custom", "tokenizer": "standard", "filters": ["standard", "lowercase", "thai", "thai_stop", "ngram"] } },

Re: calculation of whymatch in elasticsearch

I was thinking of putting together a simple "highlighter" that just returns if a field contains a match or not. This sounds like a nice logic extension to that. It probably wouldn't actually be a "highlighter" but I imagine it'd run during the highlight phase and function similarly. It'd need so

Re: Everything just slows down simetimes

you are monitoring > both of those. > > > On Wed, Feb 5, 2014 at 10:28 AM, Nikolas Everett wrote: > >> I've seen a few cases of "everything" slow down. Like this: >> [image: Elasticsearch cluster eqiad es_query_time] >> I mean, we don't have the

Re: How to make some docs less relevant on search

You could also try a boosting query where the negative clause is a filtered query matching the other field, I believe. Nik On Wed, Feb 5, 2014 at 1:29 PM, Nikolas Everett wrote: > You could use use a function score query with a clause with a filter. > That filter could select things

Re: How to make some docs less relevant on search

You could use use a function score query with a clause with a filter. That filter could select things in the other field and multiply their score by .01 or something. That'd mostly do what you want, I think. Nik On Wed, Feb 5, 2014 at 1:25 PM, Tiago Rodrigues wrote: > Hello all, > > I have so

Everything just slows down simetimes

I've seen a few cases of "everything" slow down. Like this: [image: Elasticsearch cluster eqiad es_query_time] I mean, we don't have the most even service times at the moment, but woah! I don't see anything in Elasticsearch's primary logs. The slow logs abound. They show that things that should

Re: Highlighting - limit number of highlights across fields

No, there isn't a way to do that. Just throwing away the extra results is generally not a big deal unless the field is very expensive to highlight. I've found highlighting to be reasonably cheap most of the time. On Wed, Feb 5, 2014 at 11:38 AM, Michael Stringer < michael.strin...@zestia.com> wr

Re: Scroll repeats

document has been updated, Elasticsearch will returned the old version. > > > On Wed, Feb 5, 2014 at 3:03 PM, Nikolas Everett wrote: > >> Can scroll requests repeat the same document twice? What if the document >> was updated? I know it really isn't the same Lucene document the

Scroll repeats

Can scroll requests repeat the same document twice? What if the document was updated? I know it really isn't the same Lucene document the but I sure think of it as the same thing. Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubs

Re: cluster reroute and potential data loss

If all replicas of a particular shard are unallocated and you allow_primary allocate one then it'll allocate empty. If a node that had some data for that shard comes back it won't be able to use that data because the shard has been allocated empty. On Mon, Feb 3, 2014 at 4:42 PM, Mark Conlin wr

Re: How to stop searching on first match?

4 ..." > > Maybe this is the problema, right? > > > > On Thu, Jan 30, 2014 at 5:33 PM, Nikolas Everett wrote: > >> Try something like this: >> { >>"fields": [], >>"query": { >>

Re: How to stop searching on first match?

06 08 13 19 22 25 28 29 30 31 40 42 44 46 53 55 57 59 > record 2 = 02 05 06 08 11 12 23 25 29 31 34 36 37 43 44 52 54 55 57 58 > record 3 = 04 05 12 14 19 20 24 29 31 35 38 39 40 44 47 53 54 56 57 58 > ... > > They are all unique. Unique combinations. > I usually look for do

Re: How to stop searching on first match?

t the > first match? > Is there any other way to do that? > > May be saying "relax, you job is done" after the first match? > :) > > I don't know if I was clear. I'm trying to make it work faster by reducing > it's work. > > > > On Thu, Ja

Re: How to stop searching on first match?

"size": 1 will only return the top match. Elasticsearch will still count all the other matches and order them though. It doesn't have a search type that says "get me the first N of these you see I don't care about order". This is generally OK because Lucene is really fast at ordering the documen

Postings format questions

Has anyone experimented with changing the postings format much? I haven't but the pulsing format looks like it might be useful because I have a number of fields which have low frequency terms. In this case it is mostly one term per document, sometimes one term to tens or hundreds of documents. N

Re: Multiple nodes on a powerful system?

On Wed, Jan 29, 2014 at 4:50 PM, Josh Harrison wrote: > Other than the resource footprint, is there any reason we should avoid > running multiple node instances of a cluster on the same machine, assuming > all the shard awareness stuff is in place to keep all the copies of a given > shard from be

Re: Optional asciifolding

@synhershko <https://twitter.com/synhershko> > Freelance Developer & Consultant > Author of RavenDB in Action <http://manning.com/synhershko/> > > > On Tue, Jan 28, 2014 at 6:00 PM, Nikolas Everett wrote: > >> I'm looking to make asciifolding optional in my

Optional asciifolding

I'm looking to make asciifolding optional in my (English) index. If the user searches without any high ascii characters then I want to match against the folded tokens. If the user searches with high ascii characters then I only want to match the unfolded tokens. Is this possible with Elasticsearch

Re: Limit number of results sorted

pper limit for the size of this priority queue, is it > correct? I think this would be a great addition! > > On Fri, Jan 24, 2014 at 9:56 AM, Nikolas Everett wrote: > >> Is there a way to issue a query to Elasticsearch using the sort parameter >> that limits the number of re

Limit number of results sorted

Is there a way to issue a query to Elasticsearch using the sort parameter that limits the number of results that are sorted either per shard or during the merge phase? I don't want to be able to accidentally load all the documents into memory but I'm ok with returning less accurate results. Nik

Re: Cluster state yellow

We use mutlicast but our networks are very we're pretty used to multicast for other stuff like varnish cache flushes. I figured unicast vs multicast was a networking decision based on how your data center is rigged. No one has complained about chatter and we've never seen a problem with accidenta

Re: Very open Elasticsearch installation

On Tue, Jan 14, 2014 at 10:28 AM, InquiringMind wrote: > Hi, Nik. > > I seem to be getting very good results and getting them quickly. But I do > notice some results like the following. Note that the query string I > entered is the title of a book that a good friend (and B-24 navigator > during WW

Re: Very open Elasticsearch installation

On Wed, Jan 8, 2014 at 4:01 PM, Nikolas Everett wrote: > > On Wed, Jan 8, 2014 at 3:49 PM, David Pilato wrote: > >> I'm a bit disappointed that this search gives no result: >> https://en.wikisource.org/w/index.php?title=Special%3ASearch&profile=default&se

Is there a kind of query/rescore/similarity magic that lets me know if all the terms in a field are matched?

I'm looking to boost matches that where all the terms in the field match more than I'm getting out of the default similarity. Is there some way to ask Elasticsearch to do that? I'm ok with only checking in some small window of top documents or really anything other than a large performance hit.

Re: Very open Elasticsearch installation

On Wed, Jan 8, 2014 at 4:22 PM, joergpra...@gmail.com wrote: > I can't tell it in other words, but your step to ES is a landmark. > > Thank you, Nik, for making this public, this helps me a lot for spreading > the word for more openness... > > > https://en.wikisource.org/w/index.php?title=Special

Re: Very open Elasticsearch installation

On Wed, Jan 8, 2014 at 3:49 PM, David Pilato wrote: > This is really awesome Nik! > Congrats to your team. > > I'm a bit disappointed that this search gives no result: > https://en.wikisource.org/w/index.php?title=Special%3ASearch&profile=default&search=elasticsearch&fulltext=Search > :-) > > Lo

Very open Elasticsearch installation

I've spent the past six months or so writing and deploying a replacement on site search system and we've finally decided it was time to blog about it. I figure this list might find this useful because everything

Odd hot MVEL

Does anyone know what might be causing MVEL to do this: 100.3% (501.3ms out of 500ms) cpu usage by thread 'elasticsearch[elastic1002][search][T#23]' 9/10 snapshots sharing following 47 elements java.lang.Throwable.fillInStackTrace(Native Method) java.lang.Throwable.fillInStack

Re: Google: ElasticSearch Guide Missing from Index?

I've noticed it as well. On Mon, Jan 6, 2014 at 11:39 AM, Justin Treher wrote: > Did something drastically change that Google has dropped the ES guide from > search results? The correct section of the guide used to be the first > result, now it is buried. For people that are just getting starte

Re: Upgrade from 0.90.5 to 0.90.9

On Sun, Jan 5, 2014 at 12:58 PM, Robin Verlangen wrote: > Hi there, > > We currently run on ES 0.90.5 and are willing to upgrade to the latest > version within a couple of weeks. I have already read the release notes, > and apart from a minor Lucene upgrade to 4.0.6 there's not much that seems >

Re: 0.90.7 to 0.90.9 : new dependencies

>From my perspective plugins are more of a pain then a feature: Compared to installing the deb, plugins are very difficult to work with especially when you are paranoid and won't use the installer. Now you have to build a whole new way to install just this one piece of infrastructure. And many of

Re: Returning substring in a GET request

On Thu, Jan 2, 2014 at 9:36 PM, Adolfo Rodriguez wrote: Hi, I have some quite large documents (about 30k) and I want to get only an *excerpt > of each document*, for example, 250 characters. It really does not matter > if they are the first 250 characters or a chunk in the middle. I would > prefer

Re: Highlight values from one field in another field

You mean you want the terms highlighted in one field to come from the contents of another field rather than the query? If so, then Elasticsearch doesn't support that already but you could write a custom highlighter to do that. Nik On Tue, Dec 31, 2013 at 9:40 AM, mayap wrote: > Hi All, > > We

Re: Compared to Solr (with Solr Cloud), what is the advantage(s) of Elasticsearch?

On Fri, Dec 27, 2013 at 4:30 AM, Daniel Guo wrote: > distributed model > Automatic shard rebalancing works quite well. We're able to do rolling restarts without losing any redundancy. It is useful to keep in mind that some things, like scores and suggestions, come from data that is per shard r

Re: Get matched keywords while searching on an analysed field

It would be possible to implement a "highlighter" that just returned which terms matched which fields but Elasticsearch doesn't have one now. I think you have two options at this point: 1. Do something hacky with highlighting like asking for the text length to be the max(all_strings.map(strlen).m

Re: Compared to Solr (with Solr Cloud), what is the advantage(s) of Elasticsearch?

About six months ago I spent a week porting a prototype from Solr Cloud to Elasticsearch with the intent of evaluating Elasticsearch and either throwing out the port or building off of it. By the third day or so I was convinced I'd stick with Elasticsearch because: 1. I was impressed with http://

Re: 0.90.8 released

On Thu, Dec 19, 2013 at 3:15 PM, joergpra...@gmail.com < joergpra...@gmail.com> wrote: > From the ES version numbering scheme, it is impossible to predict > consequences of updates. > > Whenever a minor release appears, it may or may not break existing code, > especially in plugins. > > So my humb

Re: Node stuck in cluster after it crashed

On Mon, Dec 16, 2013 at 9:22 PM, Nikolas Everett wrote: > One of my nodes crashed today we weren't able to start the machine again. > Sound like hardware problems. Any way, it is still listed in > _cluster/state and shards are trying to relocate to it. Bouncing another > node

<    1   2   3   4