Marvel architecture question

Hi, first of all, congrats for Marvel release. Step in the right direction. I have some questions: If I understand correctly, as of now Marvel has to run in the cluster that it collects metrics from (as a plugin). Let's call this cluster A. It is recommended for production env to have the Marvel

Re: Marvel not yet compatible with elasticsearch-1.0.0.Beta2 ?

Hi Finn, You can use 0.90.10 or 1.0.0.RC1 . Beta2 is indeed not compatible. Cheers, Boaz On Thursday, January 30, 2014 6:21:49 AM UTC+1, Finn Poitier wrote: > > > Hello there, > > I´m running a cluster on elasticsearch-1.0.0.Beta2 and just tried to > install Marvel as plugin. > > But when resta

Query score calculation algorithm for multiple scoring option given in query

Hi All, I have a following query, which I need to understand, How the scoring calculations are taking place here the query is : { "query": { "custom_filters_score": { "query": { "filtered": { "query": { "bool": { "should": [

Marvel not yet compatible with elasticsearch-1.0.0.Beta2 ?

Hello there, I´m running a cluster on elasticsearch-1.0.0.Beta2 and just tried to install Marvel as plugin. But when restarting Elasticsearch, it breaks. I guess Marvel isn´t compatible with Elasticsearch 1.0.0. yet? -- You received this message because you are subscribed to the Google Grou

Re: Can anyone please explain the mapping in Elasticsearch ?

Binh, I want to use conditional analysis for one set of documents not the single field in those documents. So, it's like I'm analyzing my type A documents with a set of tokens and rest of the documents with a different set of tokens and filters. Is there any way by which I can do it ? Thanks On

Re: Multiple nodes on a powerful system?

Thanks Jörg, Mark and Nikolas, some great information here. The 6x6 configuration was something of a worst case example, the farthest we'd probably stretch it would be 3 nodes per host on 16-18 hosts, which should be a little more reasonable. Hopefully we'll be able to do a support contract wit

Re: Marvel and basic_auth

Hi Al, try settings the following in your elasticsearch.yml: marvel.agent.exporter.es.hosts: [ "user:passwd@host:9200" ] Cheers, Boaz On Wednesday, January 29, 2014 3:37:59 PM UTC+1, Al Smith wrote: > > Perhaps a silly question, and yes I've RTFM'd the online documenation - > I'd like to kno

Tweaking BM25 parameters efficiently

Hi all, I am trying to tweak the parameters of the BM25 similarity module. I am evaluating the quality of the results using an NDCG-like metric, and would like to iterate through several values of k1 and b in order to find the ones that maximize this score. Unfortunately, because this is set a

Re: Multiple nodes on a powerful system?

You should consider the RAM to CPU/Disk ratio. On systems with huge memory, CPUs have the tendency to become weak, and the I/O subsystem must push data with higher pressure from RAM to drive (spindle or SSD). Huge RAM helps for caching strategies but also creates headaches, large caches must be lon

Re: Performance issue for search query is taking taking around 5 to 6 secs

Thanks Binh. I had tried with using the Term Filter .I am seeing little performance improvement but not as expected (expected 1sec) Any pointers. Thanks On Wednesday, January 29, 2014 1:29:31 PM UTC-8, Binh Ly wrote: > > It sounds like you don't really need scoring so you might want to use

Re: Trouble understanding mlockall behaviour

The JVM uses reserved memory, buffers, and stacks beside the heap, most can not be controlled by user settings. To reduce memory as much as possible for a Java process, do not use mlockall tricks, use 32bit JVM and reduce maximum allowed stack size with -Xss (this will affect performance of course)

How can I not store data under $ES_HOME/data while I'm using AWS S3

Hi, I'm using elasticsearch on AWS. with EC2 and S3, following guide http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ Everything seems work fine, just one thing, When I put data into elasticsearch, S3 creates datas, but also there are datas created under $ES_HOME/data path. Is there

Re: Best way to search/index the data - with and without whitespace

Thale, I would try edge ngrams (both index and search) and see how that works. I don't see why it wouldn't work for your 2 cases - just make your queries into match queries and use the "AND" operator. Good luck! -- You received this message because you are subscribed to the Google Groups "elas

Re: Trouble understanding mlockall behaviour

Also mlockall may not be supported depending on your OS. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view thi

Re: Problem with shingles as an autocomplete solution

Elliot, you're on the right path. However, the match_phrase_prefix query will then look at the positions of your terms and find matches only when all the analyzed terms are found as well as taking into account the exact sequence of the terms. That explains why you're not matching like you expec

Re: Query Time Analysis: Are field value also analyzed?

Karan, It should work no problem, if you do a query like this, it should match "Harry Greenberg": { "query": { "match": { "person_name": { "query": "harry g", "operator": "AND" } } } } -- You received this message because you are subscribed to the Google

Re: Trouble understanding mlockall behaviour

This is what it does - http://linux.die.net/man/2/mlockall Basically it prevents the memory being moved to swap. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 30 January 2014 00:01, briche arnaud wrote: > Hi, > >

Re: Multiple nodes on a powerful system?

I know some people on IRC are using containers (docker), we went down the virtualisation path instead. Both work fine. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 30 January 2014 09:03, Nikolas Everett wrote: >

Re: Query Time Analysis: Are field value also analyzed?

Thanks your your answer Binh My mapping is: "person_name" : { "type" : "string", "analyzer" : "person_name_analyzer" } >From your explanation looks like ES will analyze both the query string and the stored value in the document. That is exactly what I want. Is

Re: Multiple nodes on a powerful system?

On Wed, Jan 29, 2014 at 4:50 PM, Josh Harrison wrote: > Other than the resource footprint, is there any reason we should avoid > running multiple node instances of a cluster on the same machine, assuming > all the shard awareness stuff is in place to keep all the copies of a given > shard from be

Multiple nodes on a powerful system?

Other than the resource footprint, is there any reason we should avoid running multiple node instances of a cluster on the same machine, assuming all the shard awareness stuff is in place to keep all the copies of a given shard from being stored on those nodes that are all resident on a single

Re: Performance issue for search query is taking taking around 5 to 6 secs

It sounds like you don't really need scoring so you might want to use filters instead of the query_string query. Perhaps you can try the terms filter instead. Maybe something like this: String[] terms = { "sn1", "sn2" }; QueryBuilder q = QueryBuilders.filteredQuery(QueryBuilders.mat

Re: Missing Indexes After upgrade

These have both been confirmed. We are running Java 1.7.0_15. On Wed, Jan 29, 2014 at 12:24 PM, Binh Ly wrote: > Ryan, > > Please check and and verify that every single node in your cluster has: > > 1) Same ES version > > 2) Same exact Java version (i.e. whatever is displayed when running java

Re: Can anyone please explain the mapping in Elasticsearch ?

Analysis is done on a per-field basis. I do not believe you can conditionally change the analyzer for a single field depending on the nature of your data. However, you may be interested in the multi_field type which allows you define multiple analyzers for a single field. http://www.elasticsear

Re: Can anyone please explain the mapping in Elasticsearch ?

Coder, In general, full-text searches match tokens. If you're doing phrase or span queries, it will also look at positions. If you're doing range queries, like date-range, or numeric range, then it will also look at the actual types. On Wednesday, January 29, 2014 3:18:18 PM UTC-5, coder wrote

Problem with shingles as an autocomplete solution

Hi All, Still fairly new to Elasticsearch, but very impressed so far. Right now I'm working on a place finder service that will access a repository of place names. I'm attempting to build in some autocomplete functionality, and while I've made significant progress, it's not perfect. My curre

Performance issue for search query is taking taking around 5 to 6 secs

Hi All, we had created the index of 130 million data with 5 shards and four nodes.we had written the query with query string passing around 100 search values with space separate and passing the results to another query.total it is taking around 5 to 6 secs to complete get the results for t

Re: ProcessClusterEventTimeoutException in Elasticsearch. Is this timeout value configurable? If yes how?

Yes, I feel its little inconvenient not to have that default value configurable via the config file or an API. I hope ES team do something about it. I didn't want to touch my code rather I wanted to have some global configurable setting to make my life easy On Wed, Jan 29, 2014 at 12:07 PM,

Re: Can anyone please explain the mapping in Elasticsearch ?

Also, are there any conditional token filtering can be done in Elasticsearch. Let's say I want some of my documents to be tokenized with a certain set of token filters whereas others to be indexed using some other set of token filters. Thanks On Thu, Jan 30, 2014 at 1:48 AM, Mukul Gupta wrote:

Re: Can anyone please explain the mapping in Elasticsearch ?

Binh, I have one more doubt regarding matching of index time tokens with query time tokens. Can you please tell me how elasticsearch does this type of matching ? Is it like it only matches the token or it also matches other information like start_offset, end_offset, type and position. ? Thanks

Best way to search/index the data - with and without whitespace

Hello - I am having a problem indexing and searching for words that may or may not contain whitespace...Below is an example Here is how the index is created: curl -s -XPUT 'localhost:9200/test/name/1' -d '{ "street": "Lakeshore Dr" }' curl -s -XPUT 'localhost:9200/test/name/2' -d '{ "street": "S

Re: script filter slow query execution

Vinoth, When you access _source.X in a script, the _source field is loaded per doc, parsed, and then provided to the script. Depending on how many documents you are hitting, it can be slow if there are a lot. The document _boost should already be factored into the score, but if you need to ext

Re: script filter slow query execution

I tried removing _source.boost the query performance is significantly improved 30x Is there a reason why _source.boost would hit on the performance ? During index time we compute the document boost and store it in the boost field itself. Do we need to also store it in a separate field ? Thanks

Re: Can anyone please explain the mapping in Elasticsearch ?

Coder, 1) For the shingle token filter, there is a property called output_unigrams which is true by default. What it does is it will output the single tokens: "hotels", "in", etc in addition to the shingle tokens. If you set it to false, it will remove them. 2) Correct, it will output "ho" man

Re: filter on object non existing subfield

You should be fine. I tested on 0.90.7 as well as on 1.0 RC1. When the field does not exist, the prefix filter will not match the document, and the query succeeds with no errors. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe

Re: Can anyone please explain the mapping in Elasticsearch ?

Binh, I have a doubt in the above explanation. You mentioned that after suggestions_shingle: "hotels in hosur" becomes "hotels", "in", "hosur", "hotels in", "hotels in hosur", "in hosur" but shouldn't be "hotels" , "in", "hosur" should be removed since my min_shingle_size is 2 or is it like the or

Re: Transitioning percolation from 0.9 to 1.0...

That's how i understood it, too. But I was hoping to be able to keep both the _percolator and new .peroclator up, index to both after back-filling the new .percolator portion of the index, then switch over my _percolate calls after both indexes were in sync. The way things currently stand, unl

Re: Can anyone please explain the mapping in Elasticsearch ?

Coder, The best way to understand what an analyzer is doing is by using the _analyze api. For example if you do something like this: curl -XGET 'http://localhost:9200/auto_index/_analyze?analyzer=str_search_analyzer&pretty&text=hotels%20in%20hosur' It will tell you how that text is analyzed. I

Re: There were no results because no indices were found that match your selected time span

I found out some of my nodes were not upgraded, and I also did not have the marvel plugin installed on all nodes... data is now appearing in the marvel dashboard. Thanks, Mohamed. On Wednesday, January 29, 2014 12:35:00 PM UTC-5, Mohamed Lrhazi wrote: > > I have the same issue... I upgraded to

Re: Transitioning percolation from 0.9 to 1.0...

Hi Adam, the important part is that the queries can now be indexed under any index (thus they can scale out!), and they get stored under the .percolator type. Therefore instead of having your _percolator index you'll have to pick a name for your index, and migrate the queries, which will go to t

Re: ProcessClusterEventTimeoutException in Elasticsearch. Is this timeout value configurable? If yes how?

The master node timeout is configurable on each call using the master_timeout parameter. There's no global default exposed as settings though. Cheers Luca On Wednesday, January 29, 2014 6:32:07 PM UTC+1, Ivan Brusic wrote: > > It appears that this value is not configurable. IMHO, it should be,

Re: For those wanting to try Marvel

I shouldn't comment about the viability of various revenue generating opportunities since I have no experience in such matters. To me, Marvel seems like a value-add for customers choosing the support subscription. Here is an exclusive software package for signing up. I would be curious to see if ma

Re: Facet performance going bad on large indexes

Oreno, Just to clarify, I'm talking about doing a query like below (note the filter part inside the query-level). Does it not meet your requirement, or maybe I'm still confused? { "query": { "filtered" : { "query" : { whatever }, "filter" : { "nested" : { "pa

Can anyone please explain the mapping in Elasticsearch ?

Hi, I have an existing mapping with me but I'm not able to understand it fully. curl -XPUT 'http://localhost:9200/auto_index/' -d '{ "settings" : { "index" : { "number_of_shards" : 1, "number_of_replicas" : 1, "ana

Re: There were no results because no indices were found that match your selected time span

I have the same issue... I upgraded to 0.9.10, still seems to find, and display, no data. I have plugin "head" installed and it finds the indexes fine, but marvel does not. There is no configuration step, right? where do I check for the plugin's logs? Thanks, Mohamed. On Tuesday, January 28,

Re: ProcessClusterEventTimeoutException in Elasticsearch. Is this timeout value configurable? If yes how?

It appears that this value is not configurable. IMHO, it should be, so perhaps you should open an issue on Github and see if the elasticsearch team agrees. Better yet, submit a pull request. :) -- Ivan On Tue, Jan 28, 2014 at 1:08 PM, Ahaduzzaman Munna < ahaduzzaman.mu...@gmail.com> wrote: > G

Re: Facet performance going bad on large indexes

1.What I meant is that even in these 10 docs that came out of the main query, I only want to count products that have a date in the range specified. like "count how many users bought PRODUCT001 between range x and y" but I want the same query to execute on every product. What I don't want is for pr

Re: Missing Indexes After upgrade

Ryan, Please check and and verify that every single node in your cluster has: 1) Same ES version 2) Same exact Java version (i.e. whatever is displayed when running java -version) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscri

Re: Transitioning percolation from 0.9 to 1.0...

And I forgot to mention, index names cannot start with "_" anymore so definitely you will not be able to transition the "_percolator" index name. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving

Re: Need some help with Custom Score Query

Just to clarify, can you post the scores of the 2 documents in question: 1) script = "_score" 2) script = "doc['po'].value" Just curious to see the actual values. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group

Re: Facet performance going bad on large indexes

Oreno, 1) I'm not sure what you mean. The facet section in the query will only execute against the results narrowed down from your query section. So if your query section narrowed the results down to say 10 documents, the facet section will only facet against those 10 documents. I have found th

Re: Transitioning percolation from 0.9 to 1.0...

Adam, You'll likely need to bring it down and transition all the percolator queries as well as your code before you can bring in back up in 1.0. The best way to do this is to start with the transition using the latest 1.0 RC releases. Once you adjust everything to the new .percolator index and

filter on object non existing subfield

Hi, we're using elasticsearch 0.90.7 version. What is behaviour of this version regarding of filter on non existing object field? It seems it is working fine with dynamic mappings. For example, field 'person': person : { first_name: "john", age: 25} Using prefix filter on person.last_name - f

Re: Facet performance going bad on large indexes

Hi Binh, thanks for the input. 1.I have to put the filters on the facet level because I want the facets to count only products that appear in specific dates. If I run this on the query first, then I will also be counting products that are not in the range, just because they exist in the documents

Re: Need some help with Custom Score Query

just an update: I'm using default similarity only. Ignore that customsimilarity line. On Wed, Jan 29, 2014 at 9:47 PM, Mukul Gupta wrote: > But why am I getting exactly same score for two documents even though > their po field hase different values. If I'm multiplying my score by this > value t

Re: Need some help with Custom Score Query

But why am I getting exactly same score for two documents even though their po field hase different values. If I'm multiplying my score by this value the resultant value should be different for two, why am I getting same scores then ? Also, here is my mapping looks like: curl -XPUT 'http://localh

Re: Need some help with Custom Score Query

Coder, Your query is probably working properly. The explain does not show your custom score script logic but it should still be working as expected. You can verify it easily by changing your script logic. For example: Try: String script = "doc['po'].empty ? 1000 : doc['po'].value"; And you'll

Transitioning percolation from 0.9 to 1.0...

Is there a formal plan for how to transition between the old percolator mechanism and the new one? More specifically, does the old percolator mechanism still work in 1.0, in addition to the new distributed one? Or is just the latter? I ask, because I'm wondering how we're going to upgrade our t

Re: Marvel plugin incompatible with action.auto_create_index: false and index.mapper.dynamic: false configuration settings

Ah, I see the index auto creation setting is mentioned in the docs: http://www.elasticsearch.org/guide/en/marvel/current/#_other_relevant_elasticsearch_settings The dynamic type mapping setting should probably be added there as well, but I still feel like the plugin should be more explicit/help

Missing Indexes After upgrade

We upgraded our ES version to RC1.0 from 0.90.3. After restarting ES only half of our indexes were showing up. The indexes were still on the filesystem but were not being recognized by ES. Any ideas to what went wrong? There was one reoccurring error in the log: Message not fully read (r

Re: Marvel plugin incompatible with action.auto_create_index: false and index.mapper.dynamic: false configuration settings

Ah, I set the index auto create setting is mentioned in the docs for marvel: http://www.elasticsearch.org/guide/en/marvel/current/#_other_relevant_elasticsearch_settings The dynamic type mapping setting should also probably be mentioned there, but I still feel like the plugin could/should be mo

Marvel plugin incompatible with action.auto_create_index: false and index.mapper.dynamic: false configuration settings

Environment: I'm running ES 0.90.10 on a Windows 7 Pro box (my dev machine). *Steps:* 1. Add the following to your [es directory]/config/elasticsearch.yml *before* installing marvel: action.auto_create_index: false index.mapper.dynamic: false 2. Install marvel plugin 3. Restart your ES cluster

Re: NullPointerException in Elasticsearch 0.90.10 when searching with a nested filter in the parent document

Ok thanks for letting me know. I'll try it out as soon as 0.90.11 comes out. On Wednesday, January 29, 2014 4:49:55 AM UTC-5, Martijn v Groningen wrote: > > Hi, > > This particular issue has been fixed via another issue: > https://github.com/elasticsearch/elasticsearch/issues/4703 > > The 4703 iss

Marvel and basic_auth

Perhaps a silly question, and yes I've RTFM'd the online documenation - I'd like to know how to tell Marvel to use a username and password (and basic_auth) to talk to the ES servers? We have jetty configured to deny write access without user/pass and of course Marvel needs to know what those cr

Re: Sort multi_match query by exact matches first

That works, thank you!!! -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Sort-multi-match-query-by-exact-matches-first-tp4048342p4048356.html Sent from the ElasticSearch Users mailing list archive at Nabble.com. -- You received this message because you are su

Need some help with Custom Score Query

Hi, I'm using a custom score query for fetching my results from elasticsearch. I used a popularity "po" field for modifying _script parameter but somehow it's not working. I used explain parameter to see how the score is getting caluclated but my problem is it doesn't seem to take po into consi

Re: Add a custom lucene search filter to elasticsearch

Thank you, i found an easy way to incoporate my filters to ES in a plugin. https://github.com/elasticsearch/elasticsearch/issues/208 Guilhem Legal Le mardi 28 janvier 2014 19:39:23 UTC+1, Ivan Brusic a écrit : > > Jörg, what is the purpose of tweaking the IndexQueryParserService? I have > a f

Re: Sort multi_match query by exact matches first

Try something like this and see if it helps: { "query": { "bool": { "should": [ { "multi_match": { "query": "Marco", "fields": [ "f", "m", "l" ] } }, { "m

Adding arrays of objects using java update API and scripting.

How do I use updateRequestBuilder.addScriptParam to add List/Arrays. I want to replace "ctx._source.communities" with the list of maps called "new_communities" but it always ends up as an empty array. List> communityMapList = new ArrayList>(); for(CommunityRelationship commun

Re: Suggest API - Can't search numbers

I've solved it, simple really, but maybe a bug? Instead of: PUT restaurants/restaurant/_mapping { "location": { "index_analyzer": "whitespace", "search_analyzer": "whitespace", "properties": { "name_suggest": { "type": "completion",

Trouble understanding mlockall behaviour

Hi, Following the advice on different blog post on production setup for Elasticsearch, I'm using -Xms256m -Xmx256 and set bootstrap.mlockall to True. My guess was that the after restart, my Elasticsearch process would use about 256m of ram, and that it's memory usage would remain quasi-constan

Re: Query Time Analysis: Are field value also analyzed?

Karan, If you set person_name's analyzer to your custom one, analysis will generally be done at both query and index time. You also have the ability to set a different analyzer between index time and search time in which case they will behave differently when you search and when you index. See

filtered autocomplete (suggester)

Hi, I am trying to combine it with the query or a filter with suggester, so that suggestions are returned only for the hits. Is it possible to constraint autcomplete (or any suggester for that matter) to work on a subset of data. E.g. I've got some users with associated documents and I want

Suggest API - Can't search numbers

Hi, I'm using the Suggest API to create an autocomplete for restaurant names, but I've run into a small problem. Some restaurant names start with numbers, for example: 68 - 86 Bar & Restaurant I want to be able to type 68 and get the restaurant back. I've tried using the whitespace analyzer,

Re: Logstash RPM on centos5.x

Hi Kevin, The Packages are signed on a CentOS 6.x machine indeed which uses the V4 sigs. CentOS 5.x uses the V3 sigs but we are unable to create those sigs from a CentOS 6.x machine. I'll get back to you if/when we setup a special repo for CentOS 5.x. In the meantime you could use the non sign

[ANN] Google Compute Engine cloud plugin 1.2.0

 Heya, We have just released Google Compute Engine cloud 1.2.0:  https://github.com/elasticsearch/elasticsearch-cloud-gce There was a breaking change in GCE API which was not allowing us anymore to get a list of running instances, causing a can not get list of nodes. Disabling GCE discovery. er

[ANN][Book] ElasticSearch Cookbook

Hello All, Pleased to announce to the community the release of ElasticSearch Cookbook (http://www.packtpub.com/elasticsearch-cookbook/book) from Packtpub written by me and reviewed from Jettro Coenradie, Henrik Lindstrom, Richard Louapre, and Christian Pietsch. The main target is from novice t

Re: [RUBY] : elasticsearch-ruby : Special characters not escaped by the library

The `elasticsearch-ruby` library is what we call a "low level" client, closely matching the semantics and notation of the REST API. So, you pass in exactly the same query as you would into Curl. The only exception here are e.g. names of document types, which are part of the URL, and are correct

Sort multi_match query by exact matches first

Hi there, I am executing the following query: { "from":0, "size":5, "explain": "true", "query":{ "bool":{ "must":[ { "multi_match":{ "query":"Marco", "fields":["FirstName","MiddleName","LastName"], "fuzziness":0.

Re: NullPointerException in Elasticsearch 0.90.10 when searching with a nested filter in the parent document

Hi, This particular issue has been fixed via another issue: https://github.com/elasticsearch/elasticsearch/issues/4703 The 4703 issue fix will be included 0.90.11 and 1.0-rc2. Martijn On 29 January 2014 05:32, seallison wrote: > I'm getting a NullPointerException in Elasticsearch 0.90.10 w

Re: How to configure elasticsearch to sort the scored documents on a field after score for documents is calculated ?

Hi Ivan, I tried doing the above thing in my code but it seems there is no scoreSort() method. Are you the last sorting needs to be done this way : addSort(scoreSort(). order(SortOrder.DESC)); Thanks On Thu, Jan 23, 2014 at 12:51 AM, Ivan Brusic wrote: > Some pseudo-ish code using the Java AP