When searching for 'Boss' with fuzziness, get higher score for 'Bose' than 'Boss'. ???? How Comes !?!?

Any ideas? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit

Re: Just initialize shards when problems but no rebalance

Great, thank you. We are creating another cluster with more disk space to avoid this situations. By any chance do you have the link to the issue? 2015-01-15 13:26 GMT-03:00 Kimbro Staken ksta...@kstaken.com: I've experienced what you're describing. I called it a shard relocation storm and it's

Help creating a near real time streaming plugin to perform replication between clusters

Hey all, I would like to create a plugin, and I need a hand. Below are the requirements I have. - Our documents are immutable. They are only ever created or deleted, updates do not apply. - We want mirrors of our ES cluster in multiple AWS regions. This way if the WAN between

Re: Filter index to last 24h (REST)

A range filter on a date field with something like from now/d-1 to now/d+1 might work I think. If you don’t have a date field (could be a _timestamp field if you activated it), I’m afraid you can’t do that. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet

Filter index to last 24h (REST)

Hi all, Is there any way to only load the last 24 hours of indices? I am trying to apply a query to only show the number of documents created over the last 24 hours (over the REST API), but I have not had too much luck. Thanks! -- You received this message because you are subscribed to the

Re: Aggregation - Blank and date aggregation

Then it means that you want to use a date_histogram aggregation with interval=day. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html On Thu, Jan 15, 2015 at 4:43 PM, buddarapu nagaraju budda08n...@gmail.com wrote:

Re: How to find all docs where field_a === val1 and field_b === val2?

Thanks! I was thinking a bool query was something specific to fields with boolean values. Which is why I didn't understand the bool query example in the docs. Your posts helped me get what I wanted. :) On Wednesday, January 14, 2015 at 3:34:05 PM UTC-8, Brian wrote: By the way, David, the

How can I add additional parameters in aggregation?

I have documents with id and name and title. I am making aggregation according name, but how can I get in the results also the name and title? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving

Re: Is ElasticSearch truly scalable for analytics?

Regarding the accuracy of top-k lists This is perhaps an over-simplification - we deal with far more complex scenarios than a simple, single top-K list - we have whole aggregation trees with multiple layers of aggs: geo, time, nested, parent/child, percentiles, cardinalities etc etc

Re: Questions about scaling elasticsearch with regard to the number of documents indexed per second

Awesome! Great to know that. So as a conclusion the steps will be: 1) Stream tweets from twitter 2) Use the bulk API to make batches of 1000 (or more) tweets 3) Once the batch size is reached, spawn a new thread which will index the data into ES, meanwhile my original thread will continue

Re: Is ElasticSearch truly scalable for analytics?

I would be also very interested in node level shard results reduction but not for scalability but precision reasons. I would like to have an option for a node to do complete aggregations on its shards so the results are exact rather than approximate. There are many use cases when corpus of data

Re: Questions about scaling elasticsearch with regard to the number of documents indexed per second

Sounds good. If you are using Java, you could also look at the river code. Note that you should use BulkProcessor class which is super handy. BTW I said 1/s but not for tweets. I have less fields (20) than Twitter (100). With more fields, I guess it would take more time. Though with better

Re: need help for search

No worries for your english. Sorry. I missed your gist. Based on your examples, it sounds like you are french. Are you aware of the french mailing list? https://groups.google.com/forum/?hl=frfromgroups#!forum/elasticsearch-fr

Re: need help for search

Could you reproduce this with a full test case so we understand exactly What you are doing? May be simplify your test. See elasticsearch.org/help -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 15 janv. 2015 à 16:01, Thibaut Owczarz thib...@1001pharmacies.com a écrit

Re: Aggregation - Blank and date aggregation

Hey Adrien ,Thank you.I have one more question on aggregating on dates . We actually stored date time in a field called createdDateTime but I need only aggregates on date part of date time . Any ideas ? Or sample code can help us ? Regards Nagaraju 908 517 6981 On Wed, Jan 14, 2015 at 6:10

Re: When searching for 'Boss' with fuzziness, get higher score for 'Bose' than 'Boss'. ???? How Comes !?!?

This is because the score takes two factors into account: the document frequency and the edit distance. Quite likely in your case, even though Boss is closer than Bose, Bose has a much lower document frequency which helped it eventually get a better score. I guess we should have another rewrite

elasticsearch-hadoop for spark, index documents from a RDD in different index by day: myindex-2014-01-01 for example

Hi, I work on a complex workflow using Spark (Parsing, Cleaning, Machine Learning). At the end of the workflow I want to send aggregated results to elasticsearch so my portal could query data. There will be two types of processing: streaming and the possibility to relaunch workflow on all

Re: need help for search

hi, in my structure send in my gist, my question is just that: i have a search field. no say what i type in this field. but i need 1 request like this. { query : { bool: { must: [ ], must_not: [ ], should: [ {

Re: elasticsearch-hadoop for spark, index documents from a RDD in different index by day: myindex-2014-01-01 for example

My previous idea doesn't seem to work. Cannot send documents directly to _bulk only to index/type pattern On Thursday, January 15, 2015 at 4:17:57 PM UTC+1, Julien Naour wrote: Hi, I work on a complex workflow using Spark (Parsing, Cleaning, Machine Learning). At the end of the

Re: Just initialize shards when problems but no rebalance

I've experienced what you're describing. I called it a shard relocation storm and it's really tough to get under control. I opened a ticket on the issue and a fix was supposedly included in 1.4.2. What version are you running? If you want to truly manually manage this situation you could set

Re: need help for search

I guess it's most likely because you added all your filters in should clause instead of must? -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 15 janv. 2015 à 15:36, Thibaut Owczarz thib...@1001pharmacies.com a écrit : i found my first error, no need user. because i

Re: elasticsearch-hadoop for spark, index documents from a RDD in different index by day: myindex-2014-01-01 for example

I think I have a solution: Build JSON files so I could send it directly to _bulk saveJsonToEs(_bulk) Not sure if it will be optimized or even worked, I'll try. On Thursday, January 15, 2015 at 4:17:57 PM UTC+1, Julien Naour wrote: Hi, I work on a complex workflow using Spark (Parsing,

Re: need help for search

i'm ok, but my data search no say if is sku or code_internal or other field. if i do that, it's ok { query: { bool: { must: [ { term: { sku: 01b3ae496c0142f993cf131c607fe003 } } ], must_not: [], should: [ {

Re: need help for search

Thanks for elastisearch-fr mailing list tomorrow I do a little game simple data and I give the request that I want to do and the result i need Thanks Le jeudi 15 janvier 2015 17:31:28 UTC+1, David Pilato a écrit : No worries for your english. Sorry. I missed your gist. Based on your

Re: Just initialize shards when problems but no rebalance

So is this still happening with 1.4.2? Here's the ticket. Looks like the fix was supposed to be in 1.4.1 https://github.com/elasticsearch/elasticsearch/issues/8538 On Thu, Jan 15, 2015 at 10:55 AM, Matías Waisgold mwaisg...@gmail.com wrote: Great, thank you. We are creating another cluster

Re: real time match analysis

I was able to identify which field matched via explain, but couldn't see any information on which token filter was the reason for the match. I've tried specifying the analyzer name that the field uses as well as not specifying. If the explain is supposed to provide this data, I will give it

Re: Seeking a Director of Data Engineering in Austin TX

Hi Traci, This is a community based technical list. We'd greatly appreciate it if you didn't post job ads. On 16 January 2015 at 03:38, Traci Martin traci@gmail.com wrote: Hello All! I am a recruiter in Austin, TX trying to fill a Director of Data Engineering for my client, also in

Re: ElasticSearch: access document nested value in groovy script

I found this. I had to use _source.medals to access the nested documents which are stored in disk and not in memory. Thanks On Wednesday, January 14, 2015 at 10:55:15 AM UTC-8, Anil Kumar wrote: I have a document stored in ElasticSearch as below. _source: { firstname: John, lastname:

Re: Using icu_collation plugin in Unit Tests

Thanks David! Sorry for being a new one in the ES world. But where would i download the JAR file from and what calss should i be using for the icu_collation? Thank you very much, Kumar Subramanian, On Thursday, January 15, 2015 at 12:52:12 PM UTC-8, David Pilato wrote: You most likely just

Excluding Terms Using a Minus Sign

Is there a way to exclude a term if the user precedes it with a minus sign; the way google does. For example, if I want to search for the word lovre, but I don't want the museum in France, I can search for: *louve -museum* as my search terms. Does ES support this? I am not finding anything

Using icu_collation plugin in Unit Tests

Hi, I am new to ES. I am using NodeBuilder in my unit test to run a local instance of ES. I would like to use the icu_collation plugin. How can i install and run the plugin form within this local instance. Is there API that i should use? if not, what are the different ways i can do this? Thank

Re: Just initialize shards when problems but no rebalance

I'm on 1.4.1 and still seeing the same behavior. There should be a better practice than remove all shards at the same time and try to move a few. We are going to apply the same solution you mentioned, add more disk. Thank's for your help. 2015-01-15 16:09 GMT-03:00 Kimbro Staken

filtering/querying on script field

Is it possible to filter or query on script_fields. If so, can you provide any example.. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to

Re: Using icu_collation plugin in Unit Tests

You most likely just need to add it as a dependency. Which is easy if you are using maven. David Le 15 janv. 2015 à 21:03, Kumar S krskumar...@gmail.com a écrit : Hi, I am new to ES. I am using NodeBuilder in my unit test to run a local instance of ES. I would like to use the

Re: Help creating a near real time streaming plugin to perform replication between clusters

While it seems quite easy to attach listeners to an ES node to capture operations in translog-style and push out index/delete operations on shard level somehow, there will be more to consider for a reliable solution. The Couchbase developers have added a data replication protocol to their product

Kibana and nested documents -- include_in_parent

Hello, I am new to ElasticSearch and I have a very specific question. We have implemented our ElasticSearch cluster with a nested document structure. Each document is made of one ID, a key element and one field including several nested records that are inserted by the script api and the bulk

Re: How to remove a cluster setting?

This is a known issue​, see https://github.com/elasticsearch/elasticsearch/issues/6732​ On 15 January 2015 at 22:01, Gary Gao garygaow...@gmail.com wrote: why this didn't work on my es : GET /_cluster/settings { persistent: { discovery: { zen: {

Best pratices for index , search and updates

Am new to the elastic search ... Can some body throw me ideas about the best practices one should follow to get good performance for index ,search and updates -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and

Re: filtering/querying on script field

Hi Samatha, I don’t think so because script field is created from fields of hit document, results of query/filter. You can use script filter instead  http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html#query-dsl-script-filter. Masaru On January

Re: Slow Commands with 1.2.4 to 1.4.2 Upgrade

Just added 2 more nodes with the same specs, and still seeing the same slowness. These commands no longer return anything, because it's taking too long to return. On Tuesday, December 30, 2014 at 3:54:34 PM UTC-8, Mark Walkom wrote: How slow? Is the load on your system high? On 31

questions regarding elasticsearch-spark

Hi all, I'm quite familiar with ElasticSearch but new to spark, and elasticsearch-spark. My idea at this moment is that by using spark together with elasticsearch, it might be able to increase search performance when the time interval is fixed. question is, is hadoop need to be set up first to

Re: questions regarding elasticsearch-spark

Hi Lee, No. Hadoop isn't required . You can use the spark Standalone mode ( https://spark.apache.org/docs/1.2.0/spark-standalone.html) when running ElasticSearch on spark. Regards Ravi On Thu, Jan 15, 2015 at 10:15 PM, Seungjin Lee sweetest0...@gmail.com wrote: Hi all, I'm quite

Changing the Axis (X - Y) Label | Naming Legend

Hi, 1. Is there any way we can change the Label of X and Y axis 2. Is Kibana3, it was possible to name the legends, any way we can do this in Kiabana4 -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop

Perl client: Cannot combine params and body?

I have a remote node that I am attempting to connect to that requires an api key as a URL parameter in addition to the body in order to get it to work. The code is as follows: #!/usr/bin/perl use v5.14; use warnings; use Search::Elasticsearch; use Data::Dumper; my $API_KEY='API_KEY'; my $ES

Re: Sorting on nested object collections

I've run the query with the smallest possible subset and the query is returning the results in the expected order so it appears to be correct. The biggest question that I have is does the second sort condition know to run on the *first* projected valuation that had the max date from the first

Re: Complex search

Take a look at highlighting http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html for highlighting the relevant parts of matches and at multifield

How can we achieve an equivalent of this SQL a query in Elasticsearch?

What will be equivalent of the following query in the Elasticsearch world.. select myDate, col1, col2 from myTable where myDate = (select max(myDate) from myTable) -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group

Re: How can we achieve an equivalent of this SQL a query in Elasticsearch?

I think you need to run two queries for now. One is an aggregation (max). The other one use the result of this aggregation to search for documents. My 2 cents -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet https://twitter.com/dadoonet | @elasticsearchfr

Re: How to remove a cluster setting?

why this didn't work on my es : GET /_cluster/settings { persistent: { discovery: { zen: { minimum_master_nodes: 2 } } }, transient: { indices: { recovery: { translog_size: 1024kb, concurrent_streams: 3,

Re: How can I sort results by _id?

Making it index:not_analyzed should work, what is the issue with the results? Note that loading the _id in fielddata is typically very costly since the _id field is typically unique per document. On Thu, Jan 15, 2015 at 10:35 AM, Jason Zhang moc...@gmail.com wrote: I use a query dsl like: {

Re: how to combine aggregations

I believe you could run a terms aggregation on the city field, and under this terms aggregation put two sum aggregations, one for clicks and one for displays. And finally you could derive the click rate from the sum of clicks and displays on client side? If you are starting playing with

Out of memory on start with 38GB index

Hi, I am doing all my tests on a 38GB production index copy, with ES 1.4.2. I tried several memory settings and virtual machine sizes, but ES fails to start on a linux system with 48GB memory and 32GB for ES heap. Searching for similar issues, I encountered

Grandchild is not getting fetched by parent id

I am experiencing an issue while trying to retrieve a grandchild record by its parent ID. (child-grandchild relationship) The amount of hits in result is always zero. Also the same request is working fine for parent-child relationship. My records are getting organized kinda like this: Account

Re: Logstash output to Elastic search is not working

What do you mean by Can't see anything from the following command output: #curl http://localhost:9200/_search?pretty; from your first post? On Wednesday, January 14, 2015 at 3:27:57 AM UTC+1, zal...@gmail.com wrote: Hi Marc, I didn't find any .sincedb file from the file system.The problem

Re: How can I sort results by _id?

This is because the _id is a string field, so comparison is based on the lexicographical order, not numeric. On Thu, Jan 15, 2015 at 11:04 AM, Jason Zhang moc...@gmail.com wrote: What I'm confused is the 'sorted' results are still partly unordered. Also, if I query: { range: { _id: {

Re: How can I sort results by _id?

No, an ID has to be a string -- Itamar Syn-Hershko http://code972.com | @synhershko https://twitter.com/synhershko Freelance Developer Consultant Author of RavenDB in Action http://manning.com/synhershko/ On Thu, Jan 15, 2015 at 12:12 PM, Jason Zhang moc...@gmail.com wrote: Can I specify its

How can I sort results by _id?

I use a query dsl like: { filter: { exists: { field: info } }, sort: { _id: desc } } And the _id here is an integer like '123'. But the result is like: { took: 50, ... hits: { ... hits: [ { ... sort: [ null ] }] } } Also, I've tried to

Re: How can I sort results by _id?

What I'm confused is the 'sorted' results are still partly unordered. Also, if I query: { range: { _id: { gt: 1, lt: 1}}} the results contain _id: 199989. On Thursday, January 15, 2015 at 5:48:48 PM UTC+8, Adrien Grand wrote: Making it index:not_analyzed should work,

Re: How can I sort results by _id?

Can I specify its type as integer in _mapping? Because the _id I use is rewritten. On Thursday, January 15, 2015 at 6:07:22 PM UTC+8, Adrien Grand wrote: This is because the _id is a string field, so comparison is based on the lexicographical order, not numeric. On Thu, Jan 15, 2015 at

Re: Questions about scaling elasticsearch with regard to the number of documents indexed per second

No, so the whole point was that, will elasticsearch be able to index say 10,000 documents per second? If yes, I can simply hook up my twitter code to es. If not, I would need to think of how to make that happen. Typically I've seen es indexes just around 30 docs per second which is pretty low.

Re: Just initialize shards when problems but no rebalance

Yes, I've seen that but the problem is that when the threshold is reached it removes all shards from the server instead of just removing 1 and balance. And when that happens the cluster starts to move shards over everywhere and it never stops. Another problem we are having is that in the file

Re: How can we achieve an equivalent of this SQL a query in Elasticsearch?

Thanks.. Any other creative solutions? On Thursday, January 15, 2015 at 1:54:10 PM UTC+5:30, David Pilato wrote: I think you need to run two queries for now. One is an aggregation (max). The other one use the result of this aggregation to search for documents. My 2 cents -- *David

Deploy Elasticsearch in live

Hi all, I use ElasticSearch locally on my PC as a search engine in a content website developed with the Django framework. I would like your opinion on the choice of a host offers production, ideally a scalable offering. I consulted the offers of DigitalOcean, Amazon EC2, OVH (OVH VPC,

Re: Http Cors Setting

In my case I faced the same issue cause my web tier is hosted on a different domain. My configuration is working quite well, I can see the pre-flight (OPTIONS) call returning 200 and then subsequent POST or GET being succesfull. I have used the following configuration: http.cors.enabled: true

Re: How can we achieve an equivalent of this SQL a query in Elasticsearch?

Sorted query? GET /myIndex/_search { query:{match_all: {}}, fields:[myDate,col1], sort: [ { myDate: { order: desc } } ] } On Thursday, January 15, 2015 at 1:05:22 PM UTC, Lokesh Gupta wrote: Thanks.. Any other creative solutions?

Re: How can we achieve an equivalent of this SQL a query in Elasticsearch?

Thanks for the suggestion. Sorted query would work if I am okay with getting data for dates other than the max(date). But in the use case I have I need to restrict the results to be only for max(date). Is there a way to chain the output of a query as an input to another query? On Thursday,

Suggestion for an ElasticSearch plugin that forward documents at indexng time

Hi all, i would like to know if someone have play around an ElasticSearch plugin that can forward documents at indexing time to an external source, i dont want to do it throught logstash but only whene doc is indexed my goal is to take that plugin as an example of my custom one, i would like

need help for search

Hello, I start learning Elasticsearch, and i have a problem for understand how search. anyone could help me? My gist for all my structure and my data is here https://gist.github.com/thibaut1001/7a3000c3ff371be3a52d My problem is just in 4part To search in multi field by data like this ## We

Re: Excluding Terms Using a Minus Sign

Yes simple query string query supports this. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#query-dsl-simple-query-string-query David Le 15 janv. 2015 à 20:37, Cindy Conway cindyanncon...@gmail.com a écrit : Is there a way

Complex search

Hi, I'm looking to create a search behaviour like Amazon does. I have an index with 3 Fields : Title, Description and Category. I want to search in the fields title and descriptions for the word *car* and I would like to get scored result like this : car -- score : 1 in category

Re: Is ElasticSearch truly scalable for analytics?

Adding a 'node reduce phase' to aggregations is something I'm very interested in, and also investigating for the project I'm currently working on. If you introduce an extra reduction phase (for multiple shards on the same node) you introduce further potential for inaccuracies in the final

Re: Questions about scaling elasticsearch with regard to the number of documents indexed per second

I can index on my laptop 1-12000 docs per second. SSD drives of course. -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 15 janv. 2015 à 13:43, Chinch Pokli cpo...@gmail.com a écrit : No, so the whole point was that, will elasticsearch be able to index say 10,000

Re: need help for search

i found my first error, no need user. because i search already in user. but why when i search a defined sku, no found only one ? curl -XPOST 'http://localhost:9200/test_fr/user/_search' -d '{ query : { bool: { must: [ ], must_not: [ ], should: [

Seeking a Director of Data Engineering in Austin TX

Hello All! I am a recruiter in Austin, TX trying to fill a Director of Data Engineering for my client, also in Austin. They are ELK stack evangelists and would prefer some with, at least knowledge of Lucene or Hadoop. This is really a great company to work for and probably the nicest client I

Need help for Custom Score On Array Fields

Hi All, Currently I have an array type, and I need to calculate score base on num matched terms filters. For example: Here is my mappings : { tweet : { properties : { tags : {type : string, index_name : tag}, } } } My data will be indexed like that : {

Re: Grandchild is not getting fetched by parent id

Hi Iv, You’d need to specify both parent and routing when you index grand children. See  http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/grandparents.html Masaru On January 15, 2015 at 20:44:43, Iv Igi (sayon...@gmail.com) wrote: I am experiencing an issue while trying to