Re: No failover if number_of_replicas exceeds number of nodes?

I've logged this as https://github.com/elasticsearch/elasticsearch/issues/9213. Will mark this thread as complete in favour of the github issue. On Friday, January 9, 2015 at 12:41:26 PM UTC+13, Mark Walkom wrote: It seems highly unusual that this is occurring. I'd recommend that you open

Re: counting items in a list [array] returns (what we think) are incorrect counts via groovy

On Thu, Jan 8, 2015 at 9:09 PM, Jeff Steinmetz jeffrey.steinm...@gmail.com wrote: Is there a better way to do this? Please see this gist (or even better yet, run the script locally see the issue). https://gist.github.com/jeffsteinmetz/2ea8329c667386c80fae You must have scripting enabled in

inspect contents of filter cache

I am trying to inspect the contents of the filter cache to debug some query performance issue. Is there any way to look at the contents of filter cache in elasticsearch ? Thanks Srini -- You received this message because you are subscribed to the Google Groups elasticsearch group. To

corruption when indexing large number of documents (4 billion+)

Hi, We have a 98 node cluster of ES with each node 32GB RAM. 16GB is reserved for ES via config file. The index has 98 shards with 2 replicas. On this cluster we are loading a large number of documents (when done it would be about 10 billion). About 40million documents are generated per hour

Re: counting items in a list [array] returns (what we think) are incorrect counts via groovy

Source is going to be pretty sloe, yeah. If its a one off then its probably fine but if you do it a lot probably best to index the count. On Jan 9, 2015 12:04 AM, Jeff Steinmetz jeffrey.steinm...@gmail.com wrote: Thank you, that worked. I was curious about the speed, is running a script using

Re: counting items in a list [array] returns (what we think) are incorrect counts via groovy

Now that I am into the real wold scenario, it gets a bit tricker - I have nested objects (keys). I have to test the existence of the key in the Groovy script to avoid parsing errors on insert. How do you access a nested object in groovy? and test for the existence of a nested object key? such

Re: How can I store 2 different data types in same field of 2 different document?

Hi Paresh, You're welcome. I'm this http://stuffmysisterswilllike.files.wordpress.com/2012/07/victory-kid.jpg glad I nailed it! Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Fri, Jan 9, 2015 at 9:25 AM,

Re: counting items in a list [array] returns (what we think) are incorrect counts via groovy

Transform worked well. Nice. Curious how to get it to save to source? Tried this below, no go. (I can however do range queries agains title_count, so transform was indexed and works well) transform : { script : ctx._source['\'title_count\''] = ctx._source['\'titles\''].size(),

Re: counting items in a list [array] returns (what we think) are incorrect counts via groovy

Thank you, that worked. I was curious about the speed, is running a script using _source slower that doc[] ? Totally understand a dynamic script is slower regardless of _source vs doc[]. Makes sense that having a count transformed up front during index to create a materialized value would

How to get only the required results from aggregation

Hi, I have an input JSON of the format *{* *shopName: Shop01,* *address: xyz,* *rackId: ac015,* *rackProductList: [* *{* *name: book,* *price: 111,* *weight: 123* *},* *{* *name: notebook,* *price:

Re: counting items in a list [array] returns (what we think) are incorrect counts via groovy

Transform never saves to source. You have to transform on the application side for that. It was designed for times when you wanted to index something like this that would just take up extra space in the source document. I imagine you could use a script field on the query if you need the result to

Re: How can I store 2 different data types in same field of 2 different document?

Thank you so much Rodu...solution worked for me... Regards, Paresh B. On Thursday, 8 January 2015 21:11:47 UTC+5:30, Radu Gheorghe wrote: Thanks, David! I had no idea it works until... about one hour ago :) -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch

counting items in a list [array] returns (what we think) are incorrect counts via groovy

Is there a better way to do this? Please see this gist (or even better yet, run the script locally see the issue). https://gist.github.com/jeffsteinmetz/2ea8329c667386c80fae You must have scripting enabled in your elasticsearch config for this to work. This was originally based on some

concurrent search request to elasticsearch

What is the maximum limit on the concurrent search requests with default Elastic search server settings. I am able to perform only 5 parallel search requests in my application with default settings. how can we improve the scalability of ES server search requests apart from increasing number of

Kibana geo polygon support

Hello, Is Kibana support geo polygon filter (as the following example)? Can I make filter/query for this? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html Thanks, Hilla -- You received this message because you are subscribed to the

Re: concurrent search request to elasticsearch

Hello, The search threadpool size (that is, how many requests can be actually worked on at once) defaults to 3 times the number of processors. This might be reduced in future, though, see: https://github.com/elasticsearch/elasticsearch/pull/9165 The queue size (how many requests ES can accept

Re: Is there any solution to do the “NOT IN” functionality in Elasticsearch?

Not sure this really helps you but it might be easier and more reliable of a search to do this as two separate queries the first would just be an agg listing all distinct users and the second an agg listing users who have an action of signup? and then just subtracting that list from the first.

Re: Regarding node architecture initial setup

Hello Phani, Usually the dedicated masters are much smaller than the data nodes, because they have much less work to do. If the 4 nodes you're talking about are equal, it might be inefficient to add a 5th so you can have 2 data and 3 master nodes. Maybe for the same budget of adding the 5th you

Re: concurrent search request to elasticsearch

Thanks a lot for your detailed response. We have got all default settings only.Single node and 5 shards. But there are lot of indices with huge number of records. search settings: threads : 12, queuesize : 1000, My query is very simple. which runs on a single index

Re: How can I input data from java to ES in real time?

Ok! thank you, and referent to cluster, because is so many data an everyday this increment and I can't have all data in only one machine, how many cluster I have to use? Thanks for all. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To

Re: attachment handler type not found

hello all, I am also getting the same error...please guide.. On Friday, August 17, 2012 8:42:22 PM UTC+5:30, msya wrote: Hello, I downloaded the plugin mapper-attachments and and tried to set a mapping where the type is attachment. I did restart after I installed the mapper-attachment.

performance getting even worse after optimization

Hi, I just did an _optimize operation on a cluster (10 data nodes, roughly 350,000,000 docs in total). This a cluster only has one index. However, the performance gets even worse: the response time doubled or even tripled. Any hint on this? Thanks! -- You received this message because

Regarding node architecture initial setup

Hi All I have chosen to establish 4 nodes in my cluster. I read concept of dedicated master nodes and only data holding nodes in elastic search.please explain me briefly how can i establish cluster by using the above four nodes. suppose if i have chosen N/2+1 for 4 nodes the minimum

PHP SDK - size problem

Hello, I want query to return me 15 results. This is my code: $params = array(); $params['body'][size] = 15; $params['body']['query']['match'][country] = 53; $results = $client-search($params); and response: *Error 500 transfer closed with 33476 bytes remaining to read* I spent a lot of

Won't import any data end up with error

Hi guys, i m trying to get up running Elasticsearch version 1.4.2 on my mac os x 10.8.5. my list of plugins : - plugins: [ - { - name: jdbc-1.4.0.8-b1a51d0, - version: 1.4.0.8, - description: JDBC plugin, - jvm: true, - site:

Re: Bucket query results | top hits performance

Micheal Dustin, what should reduce the query time a lot is if you set `collect_mode` to `breadth_first` on the `top-fingerprints` agg. Like this: GET /_search?search_type=count { aggs: { top-fingerprints: { terms: { field: fingerprint, size: 50, collect_mode:

Re: attachment handler type not found

I have Elasticsearch version:1.4.2 and mapper-attachment version 2.4.1 On Thursday, January 8, 2015 3:03:39 PM UTC+5:30, Shashi wrote: hello all, I am also getting the same error...please guide.. On Friday, August 17, 2012 8:42:22 PM UTC+5:30, msya wrote: Hello, I downloaded the

whether the heap size of http node can be promoted to more than 32 GB?

Hi All, The http node is independent, and without any data. As the guide said, 50% of RAM should be assigned to Lucene. Http node does not hold any data file, Does it mean that we can give more heap size than 32 GB, if the node have 128 GB RAM. The reason to do this is

Re: Bucket query results | top hits performance

Micheal: I'd would expect that setting the `size` option on the terms agg to a smaller value would have a positive impact on the total query time. Feels like I'm missing something, can you run hot threads api (

Re: concurrent search request to elasticsearch

You're welcome. So you're saying you're running 5 searches on a single index with 5 shards (25 per-shard queries in total) and you're getting an error? I assume that error doesn't say the queue is full because the queue is 1000. Can you post the full error and also a gist where you reproduce the

Re: How can I store 2 different data types in same field of 2 different document?

Hi Paresh, If you want to sort on the field, I think it has to be the same type. So if you make everything a double, it should work for all numeric fields. To do that, you can use dynamic templates

Re: concurrent search request to elasticsearch

Sorry , I was wrong with number of shards. actual number of shards is 320 for the index which i am querying. We are using rolling indices on a daily basis. max queue size is 1000 for search thread pool. We overcome the issue None of the configured nodes are available by keeping tcp connection

Re: concurrent search request to elasticsearch

OK, now it makes sense. 5 requests with 320 shards might saturate your queue. But 320 shards sounds like a lot for one index. I assume you don't need to scale that very index to 320 nodes (+ replicas). If you can get the number of shards down (say, to the default of 5) things will surely look

Re: concurrent search request to elasticsearch

Thanks for your prompt response. Surely will reduce the number of shards with nodes/replicas addition for the better performance of the search. -- View this message in context:

Re: How can I store 2 different data types in same field of 2 different document?

Very nice Radu. I love this trick. :) -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs Le 8 janv. 2015 à 14:43, Radu Gheorghe

Re: Ignore a field in the scoring

Thank you very much 2015-01-08 4:35 GMT-02:00 Masaru Hasegawa haniomas...@gmail.com: Hi, I believe it's intended according to https://lucene.apache.org/core/4_10_2/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html . It says: -- Note that CollectionStatistics.maxDoc() is

Re: Seeing Frequent NodeNotConnectedException errors

The issue I am seeing seems similar to what was reported at https://groups.google.com/forum/#!searchin/elasticsearch/sporadic$20/elasticsearch/jUsoUV3_mbo/nM1OtJ9tmW0J I enabled more logging on the transport layer and I see the following exceptions when the disconnect/reconnect happens.

Concurrency problem when automatically creating an index

Hi, we'd been using ES for a while now. Specifically version 0.90.3. A couple of months ago we decided to migrate to the latest version which was finally frozen to be 1.4.1. No data migration was necessary because we have a redundant MongoDB, but yesterday we enabled data writing to the new ES

Re: performance getting even worse after optimization

The index is of 149G, 19 shards with 1 replica. The es version is 1.4.1, and the java version is 1.7.0_71. I have specific routing strategy and the query used in testing only goes to one shard: heap.percentram.percent load 5158

Re: Upgrade from ES 1.2.x to ES 1.4 or 1.3?

Did anyone noticed this? On Wednesday, January 7, 2015 4:29:58 PM UTC-8, Bhumir Jhaveri wrote: Here is what I did - I had some data - I bumped up the ES version and then restarted the ES - It started giving following warnings - [2015-01-07 16:26:43,881][WARN ][cluster.action.shard ]

Re: How to log input user data (json)

Dears, any comments... -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit

field not analyzed

hi, how can i configure with java api a filed as : not analyzed. thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Thanks for the reply! That is a good idea to try, but another problem with this is that when i uninstall the plugins and keep at version 1.3.2 the node will not join cluster either. We upgraded to java 8 right before starting this upgrade and thinking it may have something to do with that -

Query Help: Grouping and then counting resulting buckets

I'm having an issue getting a query to provide the count that I want. It looks like something that may only be supported by reducers which are a coming feature; however, any insight would be appreciated even if it is just confirmation that this is not yet possible. What I want to do is take a

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Hello Ben, Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart? I would understand if you said no way I do 3

Marvel issue with Elasticsearch 1.4.2 version setup using Oracle Java 1.8.0.25

I am trying to set up monitoring cluster of marvel in test setup using new elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i elasticsearch/marvel/latest command) . without marvel ES starts on marvel nodes without any error . When ES nodes try to connect to monitoring nodes

Building Our Own Security for Inter-Node Communication in ES Cluster??

Hi, Where and what should I look at and for if I want to build and integrate security for inter-node communication in an ES cluster? The security best practices and security plugins, except for Shield, seem to address only client access to ES. What should I do if I want to use SSL/TLS for

Re: Concurrency problem when automatically creating an index

How many nodes do you have in the cluster? Jörg On Thu, Jan 8, 2015 at 6:57 PM, Tom tarossi.despe...@gmail.com wrote: Hi, we'd been using ES for a while now. Specifically version 0.90.3. A couple of months ago we decided to migrate to the latest version which was finally frozen to be 1.4.1.

Re: Concurrency problem when automatically creating an index

4 El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: How many nodes do you have in the cluster? Jörg On Thu, Jan 8, 2015 at 6:57 PM, Tom tarossi@gmail.com javascript: wrote: Hi, we'd been using ES for a while now. Specifically version 0.90.3. A couple of months

Re: Marvel issue with Elasticsearch 1.4.2 version setup using Oracle Java 1.8.0.25

Update : If Marvel deployed on data cluster itself i.e. data not shipped to monitoring nodes, it works fine. Thanks Ajay On Thursday, January 8, 2015 at 2:08:10 PM UTC-5, ajay@gmail.com wrote: I am trying to set up monitoring cluster of marvel in test setup using new elasticsearch

Re: Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Ended up finding out that the jdbc plugin requires a full cluster shutdown for restart - https://github.com/jprante/elasticsearch-river-jdbc/issues/433 Going to be doing that in maintenance window tomorrow and will verify that worked. -- You received this message because you are subscribed to