Re: Delete Elasticsearch

rm -r data should help I guess -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 11 juin 2014 à 08:31, John a écrit : > > I tried upgrading elasticsearch to 1.2 yesterday and it broke most of my > scripts, even after trying adjustments listed in the documentation. I think

Delete Elasticsearch

I tried upgrading elasticsearch to 1.2 yesterday and it broke most of my scripts, even after trying adjustments listed in the documentation. I think there were problems with the update and I would like to completely remove it from my system and then do a fresh install. How can I do this in order

Creating a browse interface from ES

I am building a system that uses Elasticsearch to store and retrieve library catalogue data. One thing I've been asked for is a browse interface. Here's a definition of what this is: - The user does a search, for example "Author starts with" and they supply "Smith" - The system puts th

how can i make thai search service? by analyzer and external dictionary.

using elasticsearch wants to create thai search service. follow is the requirements. 1. we must be consist of effective index by stemming. 2. pre-registered terms of foreign loan words are must be included to the index. 3. pre-registered terms of stopword dictionary must be excluded from the in

How to map a dynamic map of key values?

Hey Guys, How can I map an arbitrary map of key/values in ES? My JSON looks like the following, where "name" and "age" are static but "attributes" is dynamic: { "name": "john", "age": 25, "attributes" : { "key1": value1, "key2": value2, "key3": value3, ... } } Things to

Re: Shard count and plugin questions

You could look at using a queuing system, like rabbitmq, where your application drops the data into, then have a logstash instance in each DC that pulls off the queue and pushes into ES. That way you can easily handle the replication of the data to multiple endpoints within rabbitmq. Regards, Mark

Re: Shard count and plugin questions

Hey Mark, Thanks for this. It seems like our best bet will be to manage indexes the same across all regions, since they're really mirrors. Since our documents are immutable, we'll just queue them up for each region, which will insert or delete them into their index in the region. It's the only

Re: Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

SSDs help, though there is likely some other issue here so it's probably not worth looking at, at this time. Have you checked hot threads or the slow query log? Can you provide more specs on your hardware? What java version are you running? Regards, Mark Walkom Infrastructure Engineer Campaign

Re: Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

Hi Mark, With Java 7, are pointers compressed by default. Other JVM Settings - -XX:UseCompressedOops - Compressed oops is supported and enabled by default in Java SE 6u23 and later. In Java SE 7, use of compressed oops is the default for 64-bit JVM proce

Re: node failures

Are you using a monitoring plugin such as marvel or elastichq? If not then installing those will give you a better insight into your cluster. You can also check the hot threads end point to check each node - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-thr

Re: Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

The Heap Size is being reduced to 30GB to ensure that's not the bottleneck. The servers currently run SAS Drives. Though SSDs are usually preferred for Elasticsearch, can this cause such disparities in performance? ElasticHQ reports very high Refresh Rates, Search-Fetch and Search-Query rates.

Re: Exposing elastic search query APIs at a public endpoint

Hi Pradeep, We are in the middle of doing the same thing, designing a system for reporting. And I want to create a middle API layer for the reasons you suggest and other reasons. I would like to exchange notes with you in a private message, if you want. You have to create some middle later, rig

Re: Shard count and plugin questions

There are a few people in the IRC channel that have done it, however, generally, cross-WAN clusters are not recommended as ES is sensitive to latency. You may be better off using the snapshot/restore process, or another export/import method. Regards, Mark Walkom Infrastructure Engineer Campaign

Re: 1.2.0 routing issue and explicit routing

Probably because it contains the release notes etc. You can't download any of the files from the links, though a note about it being removed would be handy I guess. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 11

Re: Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

You will likely see an increase by distributing it to one shard per machine, but that's hard to quantify without actually doing it. Also, you may be doing yourself a disservice with such a large heap size as Nik mentioned. Over 32GB, Java pointers are not compressed and you do lose a bit of perfor

Re: Inter-document Queries

I simplified the actual problem in order to avoid explaining the domain specific details. Allow me to add back more detail. We want to be able to search for multiple points of user action, towards a conversion funnel, and condition on multiple fields. Let's add another field (response) to the a

Re: 1.2.0 routing issue and explicit routing

As an aside, I am also wondering why this link is still active and available when it was supposed to be pulled. Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this gr

Kibana3 and Node.js and index Not Found

I am just trying to get Kibana3 up and running, and the shortest and easiest evaluation path seemed to be using Node.js. The ES Downloads page's Kibana archive does not contain a server.js file, and the following site contained the only method that did, and also referenced the ES github reposit

Re: Problem when indexing data

thanks a lot. It finally works. 在 2014年6月10日星期二UTC+2下午11时13分41秒，Jörg Prante写道： > > Just install Cygwin https://www.cygwin.com/ and leave Windows crappy > console behind. > > Jörg > > > On Tue, Jun 10, 2014 at 9:31 PM, Aaliyah > > wrote: > >> >> >> I've already read a lot about installing and set

Re: Problem when indexing data

thanks a lot. Finally it works. 在 2014年6月10日星期二UTC+2下午11时13分41秒，Jörg Prante写道： > > Just install Cygwin https://www.cygwin.com/ and leave Windows crappy > console behind. > > Jörg > > > On Tue, Jun 10, 2014 at 9:31 PM, Aaliyah > > wrote: > >> >> >> I've already read a lot about installing and set

1.2.0 routing issue and explicit routing

Hi, I use explicit routing in some of my logging indexes, and I have a *lot* of data. The tool provided to fix the routing issue that showed up in 1.2.0 takes forever on my indexes, but while it has been running, I'm wondering if I even have the problem sketched by the (very informative) blog i

Re: Is ES es.index.store.type=memory equivalent to Lucene's RAMDirectory?

Hi Jörg, Does ES allow switching from file-based store to memory-based store without re-indexing? We used to use file-based store to run a batch index, then switch to read-only memory-based index when using Lucene directly. We've found that read-only RAMDirectory is 20% faster than MmapDirecotr

Re: Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

Thanks for the clarification. The servers aren't under any (read) load yet. There is constant update of data in the background - Roughly about 60 Index Writes per second. The refresh interval is set to 60s. Can this be a performance bottleneck? We can add in more nodes to bring it up to 10 Node

Re: Problem when indexing data

Just install Cygwin https://www.cygwin.com/ and leave Windows crappy console behind. Jörg On Tue, Jun 10, 2014 at 9:31 PM, Aaliyah wrote: > > > I've already read a lot about installing and setting of elasticsearch. > Most of them are for non-windosw OS. However, it seems the principle is > sor

Re: Cannot Increase Write TPS in Elasticsearch by adding more nodes

On bare metal I can process sustained 10-12 MB/sec on a single node. Maybe you can measure throughput in bytes per second, this is easier to compare. Jörg On Tue, Jun 10, 2014 at 6:19 PM, pranav amin wrote: > Thanks Mark. > > We are using Java version - 1.7.0_25 > > What is your document size

Re: Is ES es.index.store.type=memory equivalent to Lucene's RAMDirectory?

Yes, it is equivalent. MMapDirectory is already using as much memory as possible, for reading data. RamDirectory store is when you want to push all data onto the heap, typically for volatile unit tests. For large index, it puts only burden on the heap and your performance will suffer from GC. Jö

Match a large number of terms

Hi, I wish to match a large number of terms (100's of strings) from a large set of logs (1,000,000s of entries). I what to ask if this is something that elasticsearch can do and secondly how best to go about it? This is something that will only be run a few times a year so performance is not

Re: elasticsearch Java API for function_score query

Try this import org.elasticsearch.action.search.SearchRequest; import org.elasticsearch.index.query.functionscore.FunctionScoreQueryBuilder; import java.util.Arrays; import static org.elasticsearch.client.Requests.searchRequest; import static org.elasticsearch.index.query.FilterBuilders.termsFil

Kibana as a site plugin

Referencing the post at https://groups.google.com/d/msg/elasticsearch/L9WtITL63Lo/kGi1rTWbSbIJ I am curious: To install Kibana as a site plugin, it says to "Try to install it under /plugins/kibana3/_site" Does this mean I should have an installation as follows: .../*plugins/kibana3/_site*/kib

Problem when indexing data

I've already read a lot about installing and setting of elasticsearch. Most of them are for non-windosw OS. However, it seems the principle is sort of the same thing. So I followed one site http://www.elasticsearchtutorial.com/elasticsearch-in-5-minutes.html#Indexing My OS is windows 7. I've

Re: Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

Short answer: yes. Long answer: 500ms is a long time for the empty query. I see 2ms from elasticsearch and 23ms from time in development. In production I see maybe 54ms from elasticsearch and 70 from time across far far more shards and more data. When I do the same query across thousands of shar

Re: Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

I am currently running only 1 index with 5 shards. So the both of those queries yield the same response time. My main question is to understand if scaling out is an Option given the current replication scheme.

Re: Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

I imagine that depends on lots of stuff. Are you doing elasticsearch:9200/_search or elasticsearch:9200/index/_search ? The former can take quite a while if you have lots index and lots of shards. If you can get away with not doing it, I would. The latter will only take a long time if you have t

Scaling out Elasticsearch Cluster to improve slow Empty Search query (520ms)

We currently run our Elasticsearch (*v1.0.2*) cluster on *3 Nodes* with *5 Shards and 1 Replication* Scheme. The total index size is about 70GB (~140GB with replication). The Empty Search (/_search) query takes 500-600 ms to respond. Will adding in more Nodes help in this case? The Servers are

Is ES es.index.store.type=memory equivalent to Lucene's RAMDirectory?

I would like to use MMapDirectory at the data indexing phase (in a batch). And then switch to index to in-memory and read only at time of serving real user queries to optimize the search latency. I used to achieve that when directly deal with Lucene by using RAMDirectory and read-only Searcher.

Re: Upgrade cluster from 0.90.11 to 1.1.1

Hi Thomas, I am following the same steps as you listed. I had one query I would be thankful if you can help me out. After copying the data and config folder to the latest elasticsearch version when I restart the elasticsearch 1.1.1 server, I see two nodes in the same cluster(instead of one). Ca

Re: Shard count and plugin questions

Hey guys, One last question. Does anyone do multi region replication with ES? My current understanding is that with a multi region cluster, documents will be routed to the Region with a node that "owns" the shard the document is being written to. In our use cases, our cluster must survive a

Re: Cannot Increase Write TPS in Elasticsearch by adding more nodes

Thanks Mark. We are using Java version - 1.7.0_25 What is your document size? I'm wondering if our document size i.e. 144 KB is causing the low TPS. Thanks Pranav. On Monday, June 9, 2014 6:29:19 PM UTC-4, Mark Walkom wrote: > > One thing you never mentioned was what version of Java you are on

Re: ElasticSearch and RabbitMQ as data source

Hi All, I am using the rabbitmq-river plugin for elasticsearch. My configuration for the river is as follows: curl -XPUT 'localhost:9200/_river/rabbit_river/_meta' -d '{ "type" : "rabbitmq", "rabbitmq" : { "host" : "lbha1.ir.clemson.edu", "port" : 5672, "user" :

Accessing Search Templates via Rest

Hello, maybe someone can help me. Is there a way to get the available search templates via rest api? havent found a way yet, hope you can help me. Best regards Sebastian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from thi

How to use elasticsearch graphical reports in asp.net

Please help us. We are trying to build few reports using your tool through ASP.NET Web application. We don't know what is the process. Please request help us and provide few sample applications to build reports through asp.net web. -- You received this message because you are subscribed to th

node failures

On our 4 node test cluster (1.1.2), seemingly out of the blue we had one node experience very high cpu usage and become unresponsive and then after about 8 hours another node experienced the same issue. The processes themselves stayed alive, gc activity was normal, they didn't experience an Out

Upgrade to 1.2.1 routing fix tool gives stack trace.

I am getting a stack trace when I run the routing fix tool against my index. Is this a known issue? Kind regards, Luke java -jar elasticsearch-fix-routing-1.0.jar 10.10.9.14 9300 global count Jun 10, 2014 1:03:16 PM org.elasticsearch.plugins INFO: [Choice] loaded [], sites [] Index: global, nu

I want help on searching

i am searching with the below keyword "no fat" within double quotes but search will returns the fat results also. Please advise how to avoid the fat result i want only "no fat". -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe f

[ANN] Elasticsearch Groovy language plugin 2.2.0 released

Heya, We are pleased to announce the release of the Elasticsearch Groovy language plugin, version 2.2.0 The Groovy language plugin allows to have groovy as the language of scripts to execute.. Release Notes - Version 2.2.0 Update [27] - Update to elasticsearch 1.2.0 Doc [24] - Add documentat

[ANN] Elasticsearch Python language plugin 2.2.0 released

Heya, We are pleased to announce the release of the Elasticsearch Python language plugin, version 2.2.0 The Python language plugin allows to have python as the language of scripts to execute.. Release Notes - Version 2.2.0 Update [11] - Update to elasticsearch 1.2.0 Issues, Pull requests, Fea

Re: Impossible to implement real custom boost query when the weight is in the child document?

Thanks Ivan for the tip, but I think the boost_mode is just fine in my queries. The problem is that I only can access the field of the child document, if I have an additional bool part query with the has_child query inside. This causes the sum. The custom score is multiplied with the has_child

Re: Kibana as a tool for performance tests analysis

Something like this would be super useful :) If no one else can provide an answer and you're willing, you could always code it up and submit a pull request, or alternatively, raise a feature request. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com we

Using Jetty plugin

Hi, Is it possible to restrict Kibana users from accessing particular indices/sub indices folders using the Jetty plugin? Regards, Gaurav -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving email

[ANN] Elasticsearch JavaScript language plugin 2.2.0 released

Heya, We are pleased to announce the release of the Elasticsearch JavaScript language plugin, version 2.2.0 The JavaScript language plugin allows to have javascript as the language of scripts to execute.. Release Notes - Version 2.2.0 Update [20] - Update to elasticsearch 1.2.0 Issues, Pull r

Re: Master node keeps crashing

The max file descriptors are all set to 64k. This is the output from one of the slave nodes - http://pastebin.com/RdmZsJbH On Tue, Jun 10, 2014 at 2:50 PM, sirkubax wrote: > curl -XGET 'http://localhost:9200/_nodes/_all/process?pretty=true' |less > > Have you been considering "max_file_descr

Kibana as a tool for performance tests analysis

Hello, This is my first post to this groups, so welcome everybody. I am evaluating Logstash+Elasticsearch+Kibana combo as a tool set for collecting data from performance tests which I could later analyse visually using Kibana. I found two main features missing and I am not sure if this is just

Sub-aggregations: percentiles of sum

Hi, I have a question about subaggregations. The case is that I have small documents (single opratations) with duration time of this operation which I want to aggregate and finally get the percentile of this sum. Example: | operation | time | | A | 10 | | A | 20 | | B

Re: Master node keeps crashing

curl -XGET 'http://localhost:9200/_nodes/_all/process?pretty=true' |less Have you been considering "max_file_descriptors"? W dniu wtorek, 10 czerwca 2014 09:36:35 UTC+2 użytkownik Gaurav Arora napisał: > > I am using the latest openjdk version 7 installed from ubuntu repos. > > ubuntu@es1:~$ ja

elasticsearch Java API for function_score query

I need to implement the below function_score query using Java APIs. I couldn't find any official documentation for function_score query in the Java API section of elasticsearch "function_score": { "functions": [ { "boost_factor": "3", "filter": {

Delete doesn't delete all indexes...

I am trying to delete a month of logstash indexes, and fire off e.g. this command: curl -XDELETE "http://localhost:9200/logstash-2014.05*?pretty"; which returns within a few seconds (less than a minute - the default timeout afaik) with: { "acknowledged" : true } But when I look at the indexe

Split child query to query and filter?

Hi, I need to perform a query + filter on child documents. For the query, I'm using TopChildren. Now I wonder what would be more efficient regarding query on *date/numeric (no score needed) fields* of this child - should I query on these fields using a HasChild filter in a bool query wit

Re: elasticsearch: simple cluster. Can't change nodes. Please, help.

*Try add ""data" : "false"* You can do it in you elasticserh.yml # You can exploit these settings to design advanced cluster topologies. # # 1. You want this node to never become a master node, only to hold data. #This will be the "workhorse" of your cluster. # # node.master: false # node.da

Re: elasticsearch: simple cluster. Can't change nodes. Please, help.

Try add " "data" : "false" to the node that you would like to remove data (you are trying to move from node4 -> node3 right?). I'm not sure, but when operation succeed, data will be copied to node3, *but* it *as far as I know* dhe data will stay on node, To be sure compare folder size, and if c

Re: Master node keeps crashing

I am using the latest openjdk version 7 installed from ubuntu repos. ubuntu@es1:~$ java -version java version "1.7.0_51" OpenJDK Runtime Environment (IcedTea 2.4.6) (7u51-2.4.6-1ubuntu4) OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode) ES is set to run with -Xms14075m -Xmx14075m with bootst

Re: Exposing elastic search query APIs at a public endpoint

It depend on your requirements and your product strategy - both is possible with pros and cons: - are your users proficient in a report language? Do they already write report specs in a "standard" report language? Do you want to support this report language standard? Do you like to share report st

Forcing sync of replicas

I had a problem with corrupted shards so I restarted my cluster with "index.shard.check_on_startup: fix" and the corrupted shards were fixed (i.e. deleted). Unfortunately the replicas and primaries then had differing numbers of documents despite them all being green. Fortunately the primaries a

Re: Master node keeps crashing

How much RAM per node, what java flavour and version, what ES version? Are the logs showing any OOM? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 10 June 2014 17:16, Gaurav Arora wrote: > I'm having a very odd

Master node keeps crashing

I'm having a very odd problem with one of my elasticsearch clusters. The master node on the cluster crashes randomly. The cluster is running on 3 different ec2 instances. My cluster configuration is: 3 nodes (1 master, all data nodes) 600 GB of data (3k IOPS EBS volumes) 700 million documents T

Exposing elastic search query APIs at a public endpoint

Hi - We are designing a system for reporting and are planning to use Elastic search as a backend. We want to expose reporting in such a way that users can build custom reports on top of their data without us coming in their way. One way to do this is to expose elastic search query APIs through