Re: Re: Is es really load balance?

Yes, i mean primary shards. And i use 10 threads for bulk request to the es.There are 5 threads get connet to the node0, and another 5 threads connet to the node1. So, i guess if the bulk write first going to the primary shards, and when there are enough data , then write to the replica shards(t

A problem about the Numerical range query in Kibana

HI Guys I met a problem like this,I wanna search the log in kibana by using "used_time:[100 to 800]" But the result show some record that the used_time is 20 or 30 . the used_time in elasticsearch is int ,not a string . Why the record shows like searching a string item? How can i get the record

Re: bulk indexing - optimal refresh_interval

I'd say because if you are inserting a lot of data, you will have a massive hit at the end when you need to index, as opposed to smaller ones along the way. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 29 July 201

bulk indexing - optimal refresh_interval

The 1.3.0 release notes state: >- Increase the refresh_interval > > > if >you are doing heavy bulk indexing, or you are happy with your search >results being refreshed less f

I meet several questions when using elasticsearch as below

hello: I meet several questions when using elasticsearch as below: 1,I display it by Kinana, in the beginning, it is simultaneous, but the delay becomes more and more serious when time past. I also tried to optimize the data, but it didn't work. 2, Data abnormal, when operation, I found the data

Re: Rolling Upgrading from 1.2.1 to 1.3.0 – java.lang.IllegalArgumentException: No enum constant org.apache.lucene.util.Version.4.3.1

Do you happen to know if optimize will create a segment larger than 5 gigs? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegrou

Re: Sorting on a custom script field in Java

Hi, I am facing the same issue here. Any idea how we could achieve this? Thanks! On Thursday, July 24, 2014 6:29:42 AM UTC-7, M_20 wrote: > > How can I use ES score in the script? > The following script works and I am able to sort the results based on this > script: > > String script = "doc['st

Re: Access _score from Sort script

Hi Vineeth, I want to use the score for some other purpose. (retrieve the number of children with a has_child query) Hence I can't modify it's calculation in script score. On Monday, July 28, 2014 7:53:01 PM UTC-7, vineeth mohan wrote: > > Hi , > > Its doable from here - > http://www.elasticse

Re: Sorting on a custom script field in Java

Same here, I am unable to access _score from a sorting script. Any idea if it's possible? On Thursday, July 24, 2014 6:29:42 AM UTC-7, M_20 wrote: > > How can I use ES score in the script? > The following script works and I am able to sort the results based on this > script: > > String script = "

Re: Access _score from Sort script

Hi , Its doable from here - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#_script_score Thanks Vineeth On Tue, Jul 29, 2014 at 7:53 AM, Maxime Nay wrote: > Hi, > > I am trying to sort using a script sort. > My script would

Re: [RFC] idea for a near duplicate filter

Hello Valentin , Thanks for this suggestion. We might find this useful in some of our projects. Thanks Vineeth On Tue, Jul 29, 2014 at 2:42 AM, Valentin wrote: > I have an idea for a filter with a technique I used on another project. I > thought I should share because this might be u

Access _score from Sort script

Hi, I am trying to sort using a script sort. My script would look like: _score * doc['someField'].value When I submit the query I am getting: unresolvable property or identifier: _score Is it not possible to use the score in a script sort? Thanks! -- You received this message because you are

Get children count using has_child query

Hi, Is it possible to easily get the children count when performing a has_child query? I am using elasticsearch 1.2.1 In elasticsearch 1.3 I noticed min/max children. Does this mean that elasticsearch 1.3 would expose the child count? Thanks! -- You received this message because you are subsc

Re: Rolling Upgrading from 1.2.1 to 1.3.0 – java.lang.IllegalArgumentException: No enum constant org.apache.lucene.util.Version.4.3.1

There was a bug in Lucene which caused problems with Elasticsearch 1.3.0. You might already know this, but 1.3.1 was released today to fix this issue: http://www.elasticsearch.org/blog/elasticsearch-1-3-1-released/ The issue should only affect older versions. Your version is newer, but the error

bulk API parameters

I'm uploading documents using syntax like the following. curl -XPOST 'http://localhost:9200/test/type1/_bulk' -d ' { "index" : { "_id" : "i1", "version": 3, "version_type": "external", "replication": "async", "timeout": "5m" } } { "fields": "values etc." } { "index" : { "_id" : "i2", "version": 1

[RFC] idea for a near duplicate filter

I have an idea for a filter with a technique I used on another project. I thought I should share because this might be useful to someone. Finding exact matches is an easy task but finding documents with small differences isnt. Google is using a technique which is very easy to implement and only

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

I guess I'll ask about this once more for now. This happened again today. I set allocation to new_primaries, restarted a node, set allocation back to all, and the cluster is assigning across all nodes the shards that were on the restarted node, and when it's done, which will probably take a day,

Logging of percolator reverse queries

Looks like the percolator queries are not getting recorded in any of existing slow query logs. Is it something that I am missing in configuration or logging for percolator is not available? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To

Correct place to report bug with Sense (included in Marvel plugin)?

I wasn't able to find a repo on github, how are bugs reported? Thanks, e. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroup

Re: Search by cardinality of a field

In case anyone else needs the answer, I was able to make it work with: { "filtered": { "filter": { "script": { "script": "doc['currentPatchSet.parents'].values.size() == 1" } } } } Hope that h

How to get more than 10 terms bucket with Java API?

I have an ES where I have indexed a bunch of files. Each file is tagged with a category field. I want to write Java code to get a list of all the categories. I am trying to do this using the terms aggregation: QueryBuilder qb = QueryBuilders.matchAllQuery(); SearchResponse sr = esClient.prepar

Issues during upsert for delta data indexing

Hi Team, I am looking for a solution to index delta data for an existing index using Nest (.Net). While on searching, I found the below code snippet for upsert.. client.Update(s => s .Index("myindex") .Id(id) .Document(objins) .Upsert(objins) ); My Id field is not int.. it is a string. My qu

Re: Specifying cluster name from pig

Can anyone help me on this? How do I specify the settings in elasticsearch-hadoop when ES node and Hadoop node are not co-located? On Thursday, July 24, 2014 10:34:43 AM UTC-7, ashish jain wrote: > > Hi, > > I'm trying to insert data into elasticsearch from pig. I can insert into > ES on the sa

Re: Recovery failed : IllegalArgumentException[No enum constant org.apache.lucene.util.Version.4.3.1]

This will really help us lot. Thank you. Pounraj Manikandan -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Recovery-failed-IllegalArgumentException-No-enum-constant-org-apache-lucene-util-Version-4-3-1-tp4060560p4060743.html Sent from the ElasticSearch Users

MapperParsingException - need help with dynamic property types

I writing an application to index application logs into elastic search. I'm having a problem when the type of a property changes in my data. The following works fine $ curl -XPOST http://localhost:9200/idx1/typ1 -d ' > {"data": [ > "0d7e6233e5f48b4f55c5376bf00b1be5", > "0d7e6233e5f48b4f55c53

[Connection refused] in elasticsearch-hadoop

I am trying to run a simple Map/reduce task. This is the config part: conf.set("es.nodes", "localhost:9200"); conf.set("es.resource", "twitter/tweet"); but I am getting this error: (Btw, I am able to run ES on that machine) 14/07/28 11:58:23 INFO jvm.JvmMetrics: Initializing JVM Metrics with pr

Re: Elasticsearch- Aggregation pagination

I would like to know the answer to this question as well. Adrien says in this thread that aggregation pagination is not supported at the moment , but they seem to have come close to a solution. On Mo

Re: Search by cardinality of a field

I found what I think should work in a script filter, but I get an access exception trying to use it. Adding the following filter: { "filtered": { "filter": { "script": { "lang": "mvel", "script": "doc['currentPatchSet.pa

Re: Use java Api to set a document's field as _id

The behavior is applied at the creation of the index within the mapping, not during the prepareIndex call. The example you provided is part of the mapping and not part of the document that gets indexed. If you want to override which field will be used as the _id field, you would need to change your

Re: If I have ELK stack running on EC2. How can I make the ES as a cluster?

There are some new options in the latest builds of ElasticSearch as I understand it that replace the old S3 Gateway. However, neither the S3 Gateway nor those others are requirements for setting up ElasticSearch on EC2. They are only disaster recovery options that will help you to get back u

Re: Recovery failed : IllegalArgumentException[No enum constant org.apache.lucene.util.Version.4.3.1]

FYI 1.3.1 has been release http://t.co/rCL9rAofta go ahead unless you wanna run on 1.3.1-SNAPSHOT thanks for raising this again! simon On Monday, July 28, 2014 5:17:46 PM UTC+2, Manikandan Pounraj wrote: > > Thanks Simon. > > We tested with latest build (1.3.1 SNAPSHOT). *All our old data (0.9

Re: Recovery failed : IllegalArgumentException[No enum constant org.apache.lucene.util.Version.4.3.1]

Thanks Simon. We tested with latest build (1.3.1 SNAPSHOT). *All our old data (0.90.2) got recovered without any issues in 1.3.1.*. Thanks Pounraj Manikandan -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Recovery-failed-IllegalArgumentException-No-enum-const

Re: sum-aggregation script doesn't allow negative values?

Hi Colin, now it gets really strange. First my alias curl 'http://localhost:9200/_alias?pretty' { "live-2014-07-27" : { "aliases" : { "aggtest" : { } } }, "live-2014-07-26" : { "aliases" : { "aggtest" : { } } } } I tried two different quer

Re: function score query filter cache does not work

No one? Ok opening a ticket ;) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web vi

Re: Clustering/Sharding impact on query performance

I think you confuse things and I disagree. If you see high CPU load, it is because your system runs tight of resources and tries harder to tackle these challenge automatically, e.g. by higher GC. It is not ES generating higher load just because of the index document count. There is no relation bet

Re: [Ann] JDBC River Plugin for ElasticSearch

Thanks a lot for sharing On Mon, Jul 28, 2014 at 6:16 PM, joergpra...@gmail.com < joergpra...@gmail.com> wrote: > From the docs > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients > > I conclude that Hive2 is not a JDBC Type 4 driver. Only JDBC Type 4 > drivers are suppo

Re: Unstable cluster - "suspect illegal state: trying to move shard from primary mode to replica mode"

Thanks Robin. For me the change that seemed to have worked, I keep my fingers crossed, is to add node.master=False to all nodes except one... black magic! Thanks, Mohamed. On Sun, Jul 27, 2014 at 7:35 AM, Robin Clarke wrote: > I adjusted the required master nodes to N-1, where N was the total

Re: sum-aggregation script doesn't allow negative values?

How are you searching over the multiple indexes? are you using aliases? It would be helpful if you could post your alias configuration (see [1]) and an cURL example of a search request that fails [1] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#alia

Re: Need mapping (or query) tips

I answer myself to my question: slop is the max number of "permutation" we have to do in the results to get a match. (See http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/slop.html) Le lundi 28 juillet 2014 14:41:39 UTC+2, Loïc Wenkin a écrit : > > Hi Alexander, > > Thanks a lo

ElasticsSearch 'Script Filter'

I have a ES search question and I think it can be solved using ES scripting, but I was not able to solve it and there may be a better way. The index has 3 document types, 'province', 'city' and 'neighborhood' Here is how the index is created: curl -s -XPUT 'localhost:9200/test/province/1' -d '{

Elasticsearch- Aggregation pagination

I am working on elastic search. I have to show report data but I need to group by them. Since I have large amount of data so I can not paginate client side. Can anyone explain how we can do pagination while using aggregation. -- You received this message because you are subscribed to the Googl

Re: sum-aggregation script doesn't allow negative values?

Hi Colin, thanks for checking. I could successfully reproduce your example and I even splitted it into 2 indeces and it worked (Elasticsearch 1.3.0). But as soon as I try it with my data it doesnt work. I ran some additional tests and it works if I only use the current index (day) and split it

Re: [Ann] JDBC River Plugin for ElasticSearch

>From the docs https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients I conclude that Hive2 is not a JDBC Type 4 driver. Only JDBC Type 4 drivers are supported by JDBC plugin. JDBC Type 4 does no longer need Class.forName. Jörg On Mon, Jul 28, 2014 at 1:27 PM, Santosh B wrote:

Re: Need mapping (or query) tips

Hi Alexander, Thanks a lot for your reply! Can you explain a little bit what is this "slop" parameter? Loïc Le lundi 28 juillet 2014 14:14:56 UTC+2, Alexander Reelsen a écrit : > > Hey, > > you might want to use the phrase match query and check out its slop > parameter for your second requirem

Re: Efficiency of search vs get

Hey, it is sufficient to set the routing via setRouting in the Java API... in case of doubts, you can always check the RestActions in the source and see how they do it... --Alex On Wed, Jul 16, 2014 at 7:28 PM, thale jacobs wrote: > From the example: > client.prepareSearch(indexName).setRout

Re: Need mapping (or query) tips

Hey, you might want to use the phrase match query and check out its slop parameter for your second requirement. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-match-query.html#_phrase --Alex On Wed, Jul 16, 2014 at 10:49 AM, Loïc Wenkin wrote: > Hello eve

Re: [Ann] JDBC River Plugin for ElasticSearch

Hi, Its a very good feature. I was trying to use JDBC driver to import from hive/Impala but it never works whereas mysql connector works perfectly fine. Is it something it was specifically designed to work for mysql,MSSQl...and few of them or any other databases which supports JDBC. Thanks, Sant

Re: Recovery failed : IllegalArgumentException[No enum constant org.apache.lucene.util.Version.4.3.1]

Hey, thanks for raising this. The exception is extremely helpful. This boils down to a bug / inconsistency in Lucene where some of the bugfix releases ie. 3.6.1, 4.3.1, 4.5.1 did specify the minor version in it's version constant where is shouldn't. I just opened [1] that fixes the issue. It's

Re: 1.1.1 to 1.3 upgrade possible?

It was just mentioned on IRC there is a bug that may cause problems with mixed version clusters, specifically with 1.3.0, that can cause shards to go unallocated. It's been recommended to wait until 1.3.1 which will contain a fix and is not far off. Regards, Mark Walkom Infrastructure Engineer Ca

Re: Rescore and track_scores

Did you get answer for your question? I found alternatives that I add to my post but we need to use sort especially with rescore.. Is there a way to make it happen? Also, one very important thing – rescoring cannot be combined with sorting, because sorting is done after changes introduced by

Re: Aggregations-only use case - performance tuning via config possible?

Hello Jörg, thank you. I wasn't aware that the correct key is called "NodeClient". It works. Kind regards Ben On Monday, July 28, 2014 11:57:44 AM UTC+2, Jörg Prante wrote: > > The logger name for bulk indexing in JDBC plugin is "NodeClient" > > So if you want to mute the log messages, use som

Re: 1.1.1 to 1.3 upgrade possible?

Great, thanks 4 your reply Mark On Monday, 28 July 2014 11:11:57 UTC+3, Thomas wrote: > > Hi, > > I maintain a working cluster which is in version 1.1.1 and I'm planning to > upgrade to version 1.3.0 which is released the previous week. I wanted to > ask whether it is compatible to upgrade or w

Re: Aggregations-only use case - performance tuning via config possible?

The logger name for bulk indexing in JDBC plugin is "NodeClient" So if you want to mute the log messages, use something like in config/logging.yml logger: NodeClient: OFF maybe with "log4j", I don't know: logger: log4j: NodeClient: OFF or if you use log4j, in log4j.properties: logge

Rolling Upgrading from 1.2.1 to 1.3.0 – java.lang.IllegalArgumentException: No enum constant org.apache.lucene.util.Version.4.3.1

Hi, We're upgrading our staging cluster from 1.2.1 to 1.3.0 one box at a time. We have stopped Elasticsearch on the first box, removed the Groovy plugin we were using with 1.2.1 and deployed using chef. The new box reports as 1.3.0 but when it rejoins the cluster (of two boxes, the other box is

function score query filter cache does not work

Hi!! this works { "query" : { "function_score": { "score_mode" : "first", "boost_mode" : "replace", "query" : { "match_all" : {} }, "functions": [ { "filter": { "query" : { "terms" : { "user" : ["kimchy"], "minimum_should_match" : 1 } } }, "script_score": { "script" : "100" } } ] }

Aggregations-only use case - performance tuning via config possible?

Hi there, I am using ES for calculating aggregations on a dataset of sales data (about 50,000,000 docs or 10GB of data). As an example, I am using the date histogram aggregation with term / sum sub-aggregations to get the sales sum per day and product. There is a product_id, a date field, and a

How I sort these jsons using java api?

"sort" : [ { "pt_float_props.value" : { "order" : "desc", "nested_filter" : { "term" : { "pt_float_props.name": "consume_vol_tot" } } } }, { "pt_str_props.value" : { "order" : "desc",

Re: 1.1.1 to 1.3 upgrade possible?

Sure is, check out http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html and whatever breaking changes and release notes as applicable. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com

1.1.1 to 1.3 upgrade possible?

Hi, I maintain a working cluster which is in version 1.1.1 and I'm planning to upgrade to version 1.3.0 which is released the previous week. I wanted to ask whether it is compatible to upgrade or whether I will have any known issues/problems, what to expect in general. Thank you very much Thom

Re: sum-aggregation script doesn't allow negative values?

Hi, I ran the commands in the following gist, on master, without error. Would you be able to post the error you get and a similar reproducible example to help diagnose the issue you are running into? Also, which version of Elasticsearch are you running? https://gist.github.com/colings86/46fbb

Re: Missing value key in value count aggregation? (ES 1.3.0)

This does seem to be a bug. I have opened #7050 for this On Sunday, 27 July 2014 16:10:05 UTC+1, Valentin wrote: > > Hi, > > this looks like an error to me? I tried to order a terms aggregation by > the value count sub aggregation an

Re: Unstable cluster - "suspect illegal state: trying to move shard from primary mode to replica mode"

I adjusted the required master nodes to N-1, where N was the total number of master nodes I have. On 23 July 2014 15:36, Mohamed Lrhazi wrote: > I think am running into this same issue, even after upgrading to 1.2.2. > > Did you stabilize your cluster? > > Thanks, > Mohamed. > > > On Saturday,

Re: Java API for multiple sub-aggregations

Hi, A possible solution is : TermsBuilder termsBuilder = null; TermsBuilder root = null; if (!aggregs.isEmpty()) { for (Aggreg aggreg : aggregs) { if (aggreg.aggregOk()) { if (aggreg.isTerms()) { if (termsBuilder != n

Cosine Similarity ElasticSearch

Hi, I'm using elasticsearch to index documents and then, with an other document, I score similarity using the "more_like_this" query. Just two questions: Does the "more_like_this" query use cosine similarity to score documents (I've read the documentation, but I'm still not sure)? There is a way

How to debug slow query problem

Hi all - We have elastic search cluster wherein each node has the following characteristics: 1. Holds 11-12 shards 2. 12 GB memory 3. No memory or I/O or any such resource constraint observed For queries, we see that elastic search takes from around 5 seconds to 20 seconds to respond.