Restricting Bucket Aggregation to Certain Values

I want to know if it is possible to filter the contents of an aggregation rather than just the documents returned by the query. For instance, I have a set of documents that have an array property called related concepts. I want to count the number of documents for each related concept that meet

Re: Startup issues with ES 1.3.5

We disabled all monitoring before requesting help to ensure there was no load on ES from anything other than what it does internally. My understanding on using optimize was it shouldn't be done on indices that are regularly updated and to just let the background merge process handle it. The

Re: What is elastic search bounded by? Is it cpu, memory etc

I updated the post with more details. I added the gist urls. And I was playing with settings to see the behavior in general. I removed the field data cache and queue size. Please let me know, if you want more details. jstack thread dump https://gist.github.com/anonymous/562614d4faa75f3f269c

Re: Restricting Bucket Aggregation to Certain Values

Hi, Take a look at the includes/excludes feature on the terms aggregation. I think it should help with what you are trying to achieve. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_filtering_values Colin On Monday, 8

Re: Restricting Bucket Aggregation to Certain Values

Wow, this looks like exactly what I'm looking for. But I swear I spent most of last month scouring documentation. Is this a relatively new feature? Or am I just thick? Thanks, either way! On Mon, Dec 8, 2014 at 3:20 AM, Colin Goodheart-Smithe colin.goodheart-smi...@elasticsearch.com wrote: Hi,

partial update elaticsearch-perl

Dear Colleagues, I have the following instructions that I running in marvel/sense with a right result: POST /myindex/mytype/mykey/_update { doc : { link : http://www.kicoso.com;, sortierung : 4 } } but when I try to run the same instruction in perl I get the following mistake: Could

ElasticSearch + Eclipse + AWS = very strange behaviour

Hi, I'm trying to find why I have very strange behaviour when I'm using ES version 1.4.1 (java client) and AWS Java SDK. By using AWS SDK I'm working with SimpleDB and before I create a ES node in my code I can connect to SimpleDB without any problem. Once I have the node, running this code :

Re: delete by query api - ClusterBlockException[blocked by: [FORBIDDEN/8/index write (api)];] status: 403

Does anyone have any experience of this problem ? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this

Re: elasticsearch keeps crashing

I am using jetty plugin to have some layer of security. is there any recommendation regarding the security (for me to be able to decline jetty as you recommended) -- View this message in context:

ElasticSearch Error: MapperParsingException failed to parse

I am trying to integrate MYSQL in ElasticSearch On windows 7 from this link https://github.com/jprante/elasticsearch-river-jdbc I have completed the following steps: 1: Download Unzip in C Directory https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.4.zip 2:

Point in time backups with snapshot and restore

Hi All, I was hoping to that multiple incremental snapshots created using this mechanism would allow me to go back in time to revert few documents that got indexed. However what I observed was, if I have 3 incrementally built snapshots, I can only restore the most recent 3rd snapshot. When I

delimited_payload_filter to affect scoring of documents

I was recently working on Solr and i was able to give individual term boost using approach from article http://java.dzone.com/articles/payloads-are-neat-wheres. what I basically want is that, if i have 2 documents doc 1 { content : life is good|3.5 } doc 2 { content : good|4.5 things happen

Re: elasticsearch keeps crashing

Was just trying to help you to find the cause. Not suggesting that you should remove this plugin definitely. :) -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 8 déc. 2014 à 11:00, abdo abdo.a2...@gmail.com a écrit : I am using jetty plugin to have some layer of

Search result on Dashboard in Kibana 4 BETA2

Hi, In Kibana 3 I was able to show a simple table with the result of a query on the dashboard. In version 4 (BETA2) its only seems possible to create a table showing some aggregations on the data... Is this correct, or have I misunderstood something here? Cheers, Stefan -- You received

Re: delimited_payload_filter to affect scoring of documents

Using ES 1.4.0 On Monday, 8 December 2014 16:20:15 UTC+5:30, Saurabh Gupta wrote: I was recently working on Solr and i was able to give individual term boost using approach from article http://java.dzone.com/articles/payloads-are-neat-wheres. what I basically want is that, if i have 2

Re: elasticsearch keeps crashing

You say, it keeps crashing Is this exactly reproducible, or does the hs_err log file show crashes at different problematic frames? Jörg On Mon, Dec 8, 2014 at 8:18 AM, abdo abdo.a2...@gmail.com wrote: Hi all I am using elasticsearch 1.2.1 with jetty plugin on ubuntu linux and oracle jdk

Filtering documents by score via scripting

Hello, I have a specific scenario where I have an entity A that has collection of nested entities of B. While searching on A index, I need to compute B's that match a certain criteria and then use scripting to calculate a specific value with specific params. This value is obtained from some

Deleting old ES logs /var/log/elasticsearch

Hi guys, Help me configure log-retention for ES. I've struggling to delete old logfiles created by my ES clusters. I've tried several settings in logging.yml, and none of them (log4j) seems to work. The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups an others, no

terms aggregation order by percentile

Hi, does anyone know how to order a terms aggregation by a percentile sub-aggregation value. I only want to order by one value eg 50 Thanks Valentin -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop

Elastic Audit Index Repair

Dear ElasticSerach, Can you tell me what setup to the /etc/elasticsearch/logging.yml that I want to see the IP address in ElasticSearch logs. I setting Trace in severity level the IP is appeared, but the logs flood my system. I see the shield coming soon, but I can't wait. The second

Re: Deleting old ES logs /var/log/elasticsearch

The dailyRollingFIle appender parameter for retention is 'maxBackupIndex' E.g. file: type: dailyRollingFile file: ${path.logs}/${cluster.name}.log datePattern: '.'-MM-dd maxBackupIndex: 7 layout: type: pattern conversionPattern: [%d{ISO8601}][%-5p][%-25c] %m%n

Re: Securing apis

Front it with a reverse proxy, limit access to the DELETE method On Monday, December 8, 2014 6:04:52 AM UTC, Chetan Dev wrote: Hi, Is there any way i can secure the Apis ? I want to restrict the rights so that nobody other than administrator is able to delete or update indexes Thanks

Query Millions of records in Elasticsearch

Hi, I have an index with 6 Crores of records. My usecase is to read the entire index, check each record, whether it is present in new index or not.If not I have to index into new index. I used scan and scroll operation to read the index using JAVA Api. But this process is taking lot of time

Re: Query Millions of records in Elasticsearch

On Mon, Dec 8, 2014 at 9:11 AM, Sushmitha Chakka sushmi...@sigmoidanalytics.com Hi, I have an index with 6 Crores of records. My usecase is to read the entire index, check each record, whether it is present in new index or not.If not I have to index into new index. I used scan and scroll

Newbie question about Spark and Elasticsearch

am trying to understand how spark and ES work... could someone please help me answer this question.. val conf = new Configuration() conf.set(es.resource, radio/artists) conf.set(es.query, ?q=me*) val esRDD = sc.newHadoopRDD(conf, classOf[EsInputFormat[Text, MapWritable]],

Re: Newbie question about Spark and Elasticsearch

Hi, First off I recommend using the native integration (aka the Java/Scala APIs) instead of MapReduce. The latter works but the former is better performing and more flexible. ES works in a similar fashion to the HDFS store - the data doesn't go through the master rather, each task has its own

How to force unallocated shards to get allocated?

Hi :) I have a 3 node cluster that hit a disk watermark. I fixed the disk space issue, but now I have about 30 shards (both primaries and replicas) that are unassigned, and just sitting there. How can I get ES to wake up and allocate them? Thank you! Chris -- You received this message

Re: Writing a custom_score or custom_filters_score query based on field value frequency

Hi Jim, I came across your post while searching for any requirements for diversity in results. I'm working on an approach that allows you to limit results from any one choice of field (in your case type). Using this approach, all of the results are still selected on their individual merits

Transport client TTL, seconds or milliseconds.

I'm performing writes with the transport client. I can't find documentation for whether the ttl is in seconds or milliseconds. I know if you send documents via json the TTL is a strong so you can say 1s but the setTTL method in IndexRequestBuilder just takes a long. I THINK it's seconds

Re: Transport client TTL, seconds or milliseconds.

I agree. We should have both a Javadoc and some explanations in the java API documentation. Wanna contribute and send a PR for the javadoc or for both? -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet https://twitter.com/dadoonet | @elasticsearchfr

Determine Large Query Search

Hi, We are currently using Time Series Indexing (per day indexing for some types and per week indexing for some types). Our searches are also targeted searches meaning we calculate which indices should be searched on based on the date range in the query. We have fielddata enabled on fields we

Re: I have some problem with combined aggreagation avg with a given candation

Thank you David, I try to explain better. http://localhost/ctweets/tweet/_search Query: Define average age of users that have name ''Fisnik. I tried query that i write above but it does not calculated average age of users with name Fisnik. Shows me all users with name :Fisnik, but in

Re: I have some problem with combined aggreagation avg with a given candation

I understood what you want. I did not understand what you did. From what I read, your query sounds good to me. As it does not work for you, I think you should provide a full example which reproduces your issue: http://www.elasticsearch.org/help/ http://www.elasticsearch.org/help/ -- David

Re: Transport client TTL, seconds or milliseconds.

From my tests, it’s ms: IndexRequest indexRequest = new IndexRequest(ttl, doc, 1) .source(jsonBuilder().startObject().field(foo, bar).endObject()) .ttl(1000L); Means 1s delay -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet https://twitter.com/dadoonet |

Re: elasticsearch is taking longer time while doing update and search simultaneously

Hi, In this code I'm trying to update the field view_mode value as read. On Friday, August 22, 2014 11:25:45 PM UTC+5:30, vineeth mohan wrote: Hello Subhadip* , * What exactly are you trying to achieve using this code. updateResponse = client.prepareUpdate(index, type, id)

Snapshot restore process is not finished.

Hi , We have recently upgraded our Elasticsearch cluster to version 1.4.1 , i did a snapshot to one of the indexes (Index 2014_11,size:1.44 Ti with 1 Replica)and it was successful. When i am trying to restore the snapshot from s3 to a new cluster of Elasticsearch i have created (Also Es

How to set CORS (Cross Origin Resource Sharing) to speciic domain?

Trying to enter the correct setting in elasticsearch.yml. Although the first block below added to elasticsearch.yml does what it wants, it disables CORS security altogether. Am trying to enable only localhost. The ES documentation is not clear how to define enabled domain(s) only As follows,

Upsert and Script on large index cause the cluster to timeout.

Hello, We have a small cluster with 3 nodes running 1.3.6. I have an index setup with only two fields. { index: index_name, body: { settings: { number_of_shards: 3, store: { type: :mmapfs

Re: Upsert and Script on large index cause the cluster to timeout.

I'm not sure what is up but remember that post_ids in the script is a list not a set. You might be growing it without bounds. On Dec 8, 2014 2:49 PM, Christophe Verbinnen djp...@gmail.com wrote: Hello, We have a small cluster with 3 nodes running 1.3.6. I have an index setup with only two

Re: Restarting a 2-node elasticsearch cluster with zero downtime

We've gathered more information here and updated the question below but the question still stands: Shouldn't there be a way to avoid a no master block during a cluster restart when the master is explicitly shutdown? On Sunday, December 7, 2014 12:38:23 PM UTC-5, Nariman Haghighi wrote:

Re: Upsert and Script on large index cause the cluster to timeout.

I see what you mean but the way my records are it cannot happen unless I reindex it. Le lundi 8 décembre 2014 12:05:13 UTC-8, Nikolas Everett a écrit : I'm not sure what is up but remember that post_ids in the script is a list not a set. You might be growing it without bounds. On Dec 8,

Kibana 4 - visualizing derivative?

As I recall in in Kibana 3, there was a way to plot the derivative of values. Is there way to so the same or equivalent in Kibana 4 (as of Beta 2)? I have accumulating data values in log entries going through logstash into elasticsearch (that is, values that are increasing). It is the delta

newbie indexing question

Sorry for the rather novice question. I am trying to index a large set of data in which I have a field that is sometimes nested and other times not. I have tried a variety of things with the mapping file but for some reason, I always get an error when elasticsearch tries to index one of the

php client for elastic search

Hello, I have a question about the PHP client for Elasticsearch. I have installed a local instance of elastic search v1.4.1 and when I tried to install the PHP API I got an error that said that it was only compatible with versions 1.0 through 1.3. I wanted to use the PHP API for elastic search

include_global_state in snapshot/restore

Could someone detail exactly what is re(stored) when you set this value to true? Some subset of values returned by /_cluster/state? Why would you ever want to set this to true? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe

Re: performance issues

you could try to specify multiple fields in multifield mapping with string type (or type date) and different formats. Not sure if it is going to work though I typically do this kind of staff in actual data. maybe something like: timestamp: { type: date, format: date, fields: { year:

Re: Newbie question about Spark and Elasticsearch

Great Thanks a lot Costin. Are people supposed to deploy the Spark workers on the same ES cluster? I guess it would make sense for data to remain local and avoid network transfers altogether? Thanks a lot, Mohamed. On Monday, December 8, 2014 10:19:12 AM UTC-5, Costin Leau wrote: Hi,

Re: Problems upgrading an existing field to a multi-field

*bump* Does anyone know if upgrading an existing field with a specified analyzer to a multi-field has been fixed in the recent versions of ElasticSearch? On Monday, July 7, 2014 5:15:06 PM UTC-6, Ryan Tanner wrote: I'm having trouble upgrading an existing field to a multi-field. I've done

Re: Elasticsearch snapshots throttle problems

Those settings look correct to me. You can set using kb,mb,gb, etc. On Wednesday, December 3, 2014 4:25:38 AM UTC-5, Johan Öhr wrote: Hi, I have 12 elasticsearch nodes, with 10gb eth Ive been having alot of problem with the performance of snapshots, its throttles to 20 mb/s even tho i

Validating Mapping

Hi all, I've been trying to clean out the corner cases out of my code in elasticsearch, but I'm having a problem with mappings. Is there a function within the API which checks if a given Json is a valid mapping without calling the REST client? I find that I get

marvel.agent.exporter getting 500 back from logging cluster

Hello all We are running Marvel 1.2.1 on ES 1.3.4/Java 1.7.55 with a separate logging cluster that marvel.agent.exporter.es.hosts points to. The logging cluster is running the same bits as our production cluster with marvel.agent.enabled set to false. About a week ago, we stopped seeing