Re: ES JsonParseException

On Thursday, September 18, 2014 at 12:40 CEST, Foobar Geez wrote: > Thanks. I provided a bad example as I guess I over-simplified it and > also edited it to remove proprietary data (thus, missed }). > The following example exhibits the same issue as described in my > original post. > curl -

Re: Index design for very dynamic Form/Fields

Thanks David. Based on the system behavior, having all type as string is fine for queries. But for the aggregation level it might be trouble. For example a type of address is a complex JSON object: { field_1: { country: "US", province: "CA", city: "New York", address: "Street Address"} } If we tr

Re: Any zen discovery difference between master eligible and non eligible nodes?

It was immediate on my laptop. --  David Pilato | Technical Advocate | elasticsearch.com david.pil...@elasticsearch.com @dadoonet | @elasticsearchfr | @scrutmydocs Le 18 septembre 2014 à 17:32:18, Jinyuan Zhou (zhou.jiny...@gmail.com) a écrit: David, Thanks for taking time to look at my issue.

Re: Can i store my indexes on Azure Blob

No and you should not do that if it was possible. It would be dramatically slow. You can use Blob storage for snapshots (backup). But why do you want to use Blob storage and not attached disks? --  David Pilato | Technical Advocate | elasticsearch.com david.pil...@elasticsearch.com @dadoonet | @

Re: Index design for very dynamic Form/Fields

You could have one type per form although the cluster state will be very big. But you should test that option. Or if you don't really search for numbers as numbers (I mean with Range queries/filters), you could force each field to be a String and do the transformation at a client level. My 2 ce

Can i store my indexes on Azure Blob

Hi, Is it possible to use Azure blob storage to store the indexes. Please note i cannot use a mounted drive. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to e

Index design for very dynamic Form/Fields

Hi, We have a system very much like Google Forms, which allow users to design their own forms with various fields (single line text, paragraph, number, address etc, you can imagine that.) Without any doubt, it's running on top of MongoDB. Now it has 120K forms with nearly 10 million entries.

Re: Inter connection between clusters

Thanks Mark. On Thursday, 18 September 2014 20:13:49 UTC+5:30, Sree wrote: > > Hi all, > > I have a use case like different clusters set up in different locations ( > assume the case , if its in same network or diff network) . > Is it possible to combine different indices across the cluster und

Search, get bottom N search results, instead of top N, when size=N

Assume, Api = _search size=2 sort = based on timestamp in desc order Assume total result count is 10, since size is 2, my query will return top 2 results, but I want to get bottom 2, that is 9th and 10th results how do I write my query to get result that way? Thanks, Ziaul -- You received thi

Request updates to ES documentation page "limiting memory usage"

After I completed the analysis of the fielddata cache in my situation, I sent a longer analysis within Microsoft. A colleague pointed me to the page http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html . There is lots of great info there and I'm really

Re: Snapshot to AWS S3 failing ES 1.2.2 Cloud AWS 2.2.0

I don't think that's it... but s3cmd just stopped working too so something weird is going on with my AWS access, I think it might have to do with my buckets and not with elasticsearch at this point. Thanks for your help thought! On Thursday, September 18, 2014 4:28:55 PM UTC-5, David Pilato wro

Re: Highly variable query performance with ES 1.3.2 (filter + aggregations)

Problem solved. "indices.fielddata.cache.size:2GB" doesn't mean use 2GB for caching it means use up to 134MB (2GB / 16) for each of the 16 segments within the cache. Due to the particular combination of queries, shard / node and the eviction pattern, I can explain all of the perf results I hav

Re: Script in custom_filters_score/function_score

I guess at least for function_score an alternative is to specify boost_mode to "replace" instead of the default "multiply". But I don't think there is an equivalent alternative for custom_filters_score. In any case, I think the documentation should be updated or made clearer. Thanks, Hieu On T

Script in custom_filters_score/function_score

The documentation for custom_filters_score does not provide an example of how "script" can be used. However, it refers to function_score in later versions of ES search, and the doc of function_score has the following example: "custom_filters_score": { "filters": [ { "boo

Re: Inter connection between clusters

Check out Tribe Nodes - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-tribe.html#modules-tribe Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 19 September 2014 00:43, Sree wrote: > H

Re: Indexing is being throttled

You'd get a much greater benefit from RAID than you will by using all disks as individuals. You can however use multiple mountpoints to store ES data it's just an array in path.data. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignm

Re: Snapshot to AWS S3 failing ES 1.2.2 Cloud AWS 2.2.0

May be one of your node can not access to AWS API or don't have required credentials??? --  David Pilato Sent with Airmail Le 18 septembre 2014 à 23:22:50, rhea ghosh (rhea.gh...@gmail.com) a écrit: Ok that was totally it for that first error, I thought I had restarted the service but there we

Re: Snapshot to AWS S3 failing ES 1.2.2 Cloud AWS 2.2.0

Ok that was totally it for that first error, I thought I had restarted the service but there were a few nodes taht hadn't been restarted so weren't registering the plugin. Now I'm getting a different error that I'm still pretty lost on: curl -XPUT localhost:9200/_snapshot/my_s3_repository/snap

Re: Snapshot to AWS S3 failing ES 1.2.2 Cloud AWS 2.2.0

Not sure my theory is good but my first guess is that you have more than one node running and one of this nodes does not have the plugin installed or has not been restarted. Can you run: curl 'localhost:9200/_cat/plugins?v' And post results here -- David Pilato | Technical Advocate | Elasticse

Re: Snapshot to AWS S3 failing ES 1.2.2 Cloud AWS 2.2.0

As a follow up with more information this is the listing for the cloud-aws plugin: [rhea@server cloud-aws]$ ll total 14640 -rw-r--r-- 1 root root 12418940 Sep 16 10:43 aws-java-sdk-1.7.13.jar -rw-r--r-- 1 root root58160 Sep 16 10:43 commons-codec-1.4.jar -rw-r--r-- 1 root root60686 Sep 16

Find the number of Shards and Replicas on a Rest call for each node???

Hello, I have three different nodes with the default settings and was trying to find a rest call that would show me the number of shards and replicas that are being used on each node. I have done a cluster health to get the overall amount of shards and replicas but looking for a way to see it o

[ANN] Elasticsearch RSS River Plugin 1.3.0 released

Heya, I'm pleased to announce the release of the Elasticsearch RSS River Plugin, version 1.3.0. RSS River Plugin offers a simple way to index RSS feeds into elasticsearch.. https://github.com/dadoonet/rssriver/ Release Notes - rssriver - Version 1.3.0 Update: * [40] - Update to Rome 1.5.0

Snapshot to AWS S3 failing ES 1.2.2 Cloud AWS 2.2.0

Hi I'm trying to get snapshots up to an AWS S3 bucket from an elasticsearch cluster. I've successfully managed to install cloud-aws 2.2.0 and I believe this should be the right version that I need. So when I create the repository it seems like everything is fine: curl -XPOST localhost:9200/_s

Uneven search load on data nodes

We have built elastic search cluster for our application autosuggest and we are facing one strange issue, need immediate help. Below is the setup of ES cluster: We built the azure elastic search cluster with: Master only Nodes: 3 medium VMs with 7 GB RAM Non-master non-data nodes: 3 medium VMs

Re: Any zen discovery difference between master eligible and non eligible nodes?

David, Thanks for taking time to look at my issue. My issue is probably too slow to find a master node to join. I was wondering if non master eligible nodes takes longer to find a master node. May be some time out happened. I was looking for some parameters to allow more time to find a master node.

Re: Snapshots don't get compressed

thank you again, I'll keep an eye on this issue. Domonkos 2014.09.18. 16:38 keltezéssel, David Pilato írta: I don't know. I think this could happen in the future but unsure though. May be Igor could answer this? Here is a related doc PR: https://github.com/elasticsearch/elasticsearch/pull/765

Inter connection between clusters

Hi all, I have a use case like different clusters set up in different locations ( assume the case , if its in same network or diff network) . Is it possible to combine different indices across the cluster under an alias ? If thats possible , when i query the data , in which system the collecti

Re: query match

Yes, but that wouldn't serve the purpose I was looking for. Using "ÄND" operator, wouldn't give me any results at all since there are no documents which has all the words in the User Query. On Thursday, September 18, 2014 7:22:22 PM UTC+5:30, Ramy wrote: > > Have you tried like this? > > "query"

Re: Snapshots don't get compressed

I don't know. I think this could happen in the future but unsure though. May be Igor could answer this? Here is a related doc PR:  https://github.com/elasticsearch/elasticsearch/pull/7654 --  David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 18 septembre 201

Re: Snapshots don't get compressed

Thank you for the answer, but may I know why? What is the reason behind this? thanks, Domonkos 2014.09.18. 16:15 keltezéssel, David Pilato írta: Only metadata are compressed. -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos"

Re: Snapshots don't get compressed

Only metadata are compressed. -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos" a écrit > : > > Hi Everyone, > > I just started experimenting with the cool snapshot feature of ES (using ES > 1.3.2 on Ubuntu 14.04) using curat

Re: Indexing is being throttled

Unfortunately that is too hard/complicated. I have now enabled all 12 disks per machine, so going forward I will get some "sharing" across all disks. Not sure how it will allocate new data across the disks? If I move a shard from one node to another with the new 12-disk paths, will the receiv

Re: Snapshots don't get compressed

I tried using tar on the indices folder, it resulted in a 1.5G tarball (compared to the 2.8G size of the folder), so I really think something is wrong here. 2014.09.18. 15:36 keltezéssel, "Tomcsányi, Domonkos" wrote: Hi Everyone, I just started experimenting with the cool snapshot feature of

Re: query match

Have you tried like this? "query": { "match": { "test": { "query": "routers and switches for home", *"operator": "and"* } } } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails

Snapshots don't get compressed

Hi Everyone, I just started experimenting with the cool snapshot feature of ES (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository on a mounted NFS storage, using only the default options (compression turned on). I checked it using curl: user@myserver:~# curl -XGET 'h

Re: elasticsearch heapdump using jmap

It clearly says, you have to use -F On Friday, July 11, 2014 6:16:36 PM UTC+2, NC wrote: > > Over last few days noticed more GC in my es cluster and was trying to take > a look at heap .. I'm runnning Es over wrapper, running on EC2 , with > Amazon linux AMI > > Always get this message with jma

Re: Getting ElasticsearchIntegrationTest teardown failures :: "Delete Index failed - not acked"

No suggestions at all? On Friday, 5 September 2014 17:09:15 UTC+1, mooky wrote: > > > I am getting the following intermittent failure on random different tests > (I presume during the teardown) when the build is running on TeamCity. > I cant seem reproduce it locally. I get a failure about 1 in 1

query match

I am trying to match a query against the value in the "test" field Following are the documents indexed - curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{ "test" : "routers switches" }' curl -XPUT 'http://localhost:9200/twitter/tweet/2' -d '{ "test" : "routers" }' curl -XPUT 'ht

Re: Pushing data from Hive to Elastic Search

It's quite easy - the es.resource format is incorrect. It should be index/type as in radio/artists If you want/need to specify an ID or other criteria, you should do so in the query (through es.query). On 9/18/14 12:10 PM, Fabien Chung wrote: Hi, I would be glad to help you unfortunatly I st

Re: ES JsonParseException

Thanks. I provided a bad example as I guess I over-simplified it and also edited it to remove proprietary data (thus, missed }). The following example exhibits the same issue as described in my original post. curl -XPUT 'http://localhost:9200/test/test/test' -d ' { "rules": [

Sorting results on multiple fields

Hi, I'm having trouble sorting my results which has multiple types with different fields (where each field doesn't exist on the other type). This is my request: { "from": 0, "size": 12, "sort": { "learnArticlePostedDate": "desc", "publicationDate": "desc" } } The results that a

Re: Indexing is being throttled

Does your server have hardware RAID capabilities? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 18 September 2014 19:30, wrote: > Good point on heap, so I will bring that back down to 30GB > > Versions: > ES 1.3.2

Re: Indexing is being throttled

Good point on heap, so I will bring that back down to 30GB Versions: ES 1.3.2-1 java 1.7.0_67 I definitely want to start using all 12 disks, rather than the 1 at the moment! If I add paths for the other 11 disks and restart, will ES do any 'rebalancing'? If it won't then is there any way to mov

Can i work log results under Kibana/elastic search? our should i parse it before send the data to then?

Hello everyone, I'm not sure if i can do this kind of data filtering under kibana , but after importing some logs under elastic search with success. How could i turn display a pie or bar chart this information based on the call count? For example, Using this data how could i show this data unde

Re: Pushing data from Hive to Elastic Search

Hi, I would be glad to help you unfortunatly I stop for 1 year using ES and I can't remember how I solved this issue. Sorry, Fabien 2014-09-18 11:05 GMT+02:00 Mohit Kumar Yadav : > Hi Fabien, > I also getting the same error message. can you please tell me what is the > solution for it if you

Re: Pushing data from Hive to Elastic Search

Hi Fabien, I also getting the same error message. can you please tell me what is the solution for it if you have get rid of this error.? thanks in advance regards Mohit Kumar Yadav On Wednesday, June 26, 2013 2:59:54 PM UTC+5:30, Fabien Chung wrote: > > Hi, > > thanks for your answer. Sorry I di

User specific results

Hey, I have been trying to make a search which is user specific. I have two types of data. One contains names of different types of restaurants and other is user data which also contains the user history i.e. list of restaurants from which the user has ordered earlier. Now i want to make search

Re: Indexing is being throttled

Also given you're over 32GB heap your java pointers aren't going to be compressed, which means GC will suffer. You haven't mentioned what ES and java versions you are using, which would be useful. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web:

Newbie needs help: Logstash always creates index with a default name "logstash-xxxx.xx.xx"

Hi, Havn't been using ELK too long and I just can't undestand, why always logstash creates new index with the default name "logstash-.xx.xx" and ignores totally my config. I have tried to tell: index => "api-%{+.MM.dd}" ... or just "api" or anything, but it always creates the index wit

Re: Indexing is being throttled

Try disabling merge IO throttling, especially if your index is on SSD/s. (It's on by default at a paltry 20 MB/sec). Merge IO throttling causes merges to run slowly which eventually causes them to back up enough to the point where indexing must be throttled... Also see the recent post about tuni

Indexing is being throttled

Setup: 4 nodes Replication= 0 ES_HEAP_SIZE = 75GB Number of Indices = 59 (using logstash one index per month) Total shards = 234 (each index is 4 hards, one per node) Total docs = 7.4 billion Total size = 4.7TB When I add a new file, which I do usi

Re: Limiting pagination?

No, but since you should never expose your clusters to end users directly you could always impose this limit on the software facade that connects between your application and the cluster -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer &

Re: OLAP analytics in Elasticsearch

Any help to start of or its not possible at all? On Tuesday, September 16, 2014 10:38:29 PM UTC+5, Maaz wrote: > > I am working with Analytic of events, I use hadoop to process the logs and > store some results in Mysql. This did not work now due to scalability > issues as logs are keep coming d

Limiting pagination?

Hello, I wonder if it is possible to put a limit on the from parameter in pagination requests. For example refuse any paginating searches where the from is above X? This would be good to protect clusters which otherwise are easy to bring down. -- You received this message because you are subs

Re: HELP! how to disable _source and only allow a couple of fields to be stored....

Shay, I retried the same setup mutliple times and then restarted the elastic search. However, the mapping file is not picked up. This is what i did. 1) created a mapping file "datastore.json" with the following configuration {"datastore":{"_source":{"enabled":false},"properties":{"data1":{"type

Java API and aggregations result problem

I have a problem with Java API and aggregations result Serializer produce this error: No serializer found for class org.elasticsearch.common.text.StringText and no properties discovered to create BeanSerializer for this query : http://pastebin.com/Ph8unhsA with this response : http://pastebin.com

Java API and aggregations result problem

I have a problem with Java API and aggregations result Serializer produce this error: No serializer found for class org.elasticsearch.common.text.StringText and no properties discovered to create BeanSerializer for this query : http://pastebin.com/Ph8unhsA with this response : http://pastebin.c

Knowing which one of the boolean queries matched the result document

Hi, If I create a Bool Query with 3 OR clauses and I get back a bunch of results. Is there a way of knowing/associating a hit with the specific Bool Query? Thanks, Thanks, Sandeep -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubsc

Elastisearch query performance improvement

Hi all, I have 4 nodes, memory size respectively has 64GB, 64GB, 64GB and 128 GB for handing a huge data (8,577,271,728 documents) with 68 TB I release ES API for kibana and a webUI query, but some issues happen to slow-query or out of memory. Here is my briefly configuration: shards = 176 re

Re: Elasticsearch performance improvment

What version of ES and java are you running? What is your heap size? What sort of queries are you running? Replicas and extra nodes may/will help but a bit more info would clear things up :) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.c

Custom Collector using a plugin

Hello, i am just looking for an entry point for a custom (lucene) collector. is it possible to use a custom collector at all? what classes do i have to implement to run the collector in a distributed way? thanks a lot, tim -- You received this message because you are subscribed to the Google

Elasticsearch performance improvment

Hi all, I have 4 nodes, memory size respe respectively 64GB, 64GB, 64GB and 128 GB for handing 8,577,271,728 Documents and 68 TB data I open that API for kibana and a webUI query, but some issues happen to slow-query or out of memory. Here is my briefly configuration: shards = 176 replica = 0