Re: Alerting in ELK stack?

We have and use SPM for all our metrics (ES, Kafka, Apache, MySQL, Hadoop, everything) and we feed our logs to Logsene (it has a Kibana UI and a "native" UI). SPM has alerting and anomaly detection, so we use that to get out of bed early

Re: What does it take to make a custom stemmer for ES?

Hi Nandiya, Have a look at Lucene and its source-code for token filters. You'd implement a custom stemmer at Lucene level, and then just use that in ES. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Monday, July 7, 20

Re: Kibana Elasticsearch Shards and replication

Once you have a cluster, all data on any node is accessible. It does this by passing the query to the master node which then collects the data as required from the other nodes. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.

Kibana Elasticsearch Shards and replication

Hi everyone, Sorry if this has been covered but a few pages of searching through the group hasn't sprung an answer for this. If I decided to have 3 elasticsearch nodes, with 3 shards, and 0 replicas, would kibana be able to retrieve all the data in my ES cluster or just the data from the elas

Re: Time range filter

Hello Tom , At this point , i can think of 2 approaches - 1. Store an additioanl field with just the time and not the date information. Do a normal range query here. 2. Create script filters - In the filter , take the time out and check the range. http://www.elasticsearch.org/gui

Re: How can I do intersection or union operation with two facets filter？

It doesn't work, the result is: {"term":"bbb","count":2}, {"term":"aaa","count":1 }, {"term":"ccc","count":1 } 于 2014年07月08日 02:07, Harish Ved 写道: Did you try the following query? "query":{ "filtered" : { "filter" : { "bool" : { "should" : [

Time range filter

All of the examples I can find on the web relate to date-range filtering. What I need is a time-range filter: i,e 19:00 - 23:30. So, in this example, I want all hits between 7PM and 11:30, regardless of the day... I'd do this in SQL by doing "Where TIME(column) BETWEEN x and y". Is this possib

What does it take to make a custom stemmer for ES?

I am interested in using elasticsearch for our website suttacentral.net, I've tried ES and found it pleasant to use with obvious power, the only challenge is that on suttacentral we host many buddhist texts in ancient languages, particularly the pali language, suffix to say there are no existin

Re: Elasticsearch Not Working

Hey, I tried executing the above commands specified by you. But still not getting the elastic-search working. Still giving the same status message, "elasticsearch dead but subsys locked" Thanks, Shriyansh On Monday, July 7, 2014 5:29:00 PM UTC-7, arshpreet singh wrote: > > On Tue, Jul 8, 2014 at

Re: Elasticsearch Not Working

On Tue, Jul 8, 2014 at 5:53 AM, shriyansh jain wrote: > Hey Arshpreet, > > I am not getting anything in output. Please avoid top posting and RTF while replying in mailing lists. IMHO your service is blocked and you need to kill the demon. sudo /etc/init.d/elasticsearch restart or sudo /etc/init.

Re: Elasticsearch Not Working

Hey, I am running the following command from terminal to verify the status *sudo /etc/init.d/elasticsearch status*Thanks, Shriyansh On Monday, July 7, 2014 5:07:04 PM UTC-7, Mark Walkom wrote: > > You need to provide more details for people to be able to effectively help. > > How are you verif

Re: Elasticsearch Not Working

What command? Please be explicit, provide what you are running and the output. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 8 July 2014 10:25, shriyansh jain wrote: > I am just running a command from terminal. >

Re: Elasticsearch Not Working

I am just running a command from terminal. Thanks, Shriyansh On Monday, July 7, 2014 5:07:04 PM UTC-7, Mark Walkom wrote: > > You need to provide more details for people to be able to effectively help. > > How are you verifying this, what method are you using? > > Regards, > Mark Walkom > > Infra

Re: Elasticsearch Not Working

Hey Arshpreet, I am not getting anything in output. Thank you, Shriyansh On Monday, July 7, 2014 5:12:05 PM UTC-7, arshpreet singh wrote: > > On Tue, Jul 8, 2014 at 5:33 AM, shriyansh jain > wrote: > > When I am verifying the elastic-search status, its giving me the > following > > error

Re: Elasticsearch Not Working

On Tue, Jul 8, 2014 at 5:33 AM, shriyansh jain wrote: > When I am verifying the elastic-search status, its giving me the following > error message. > > elasticsearch dead but subsys locked ipcs -s | grep elasticsearch Can you post output for the above command? -- Thanks Arshpreet singh http:/

Re: Elasticsearch Not Working

You need to provide more details for people to be able to effectively help. How are you verifying this, what method are you using? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 8 July 2014 10:03, shriyansh jain w

Elasticsearch Not Working

When I am verifying the elastic-search status, its giving me the following error message. *elasticsearch dead but subsys locked* Please help me out solving this. Thank you. Shriyansh Jain -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To

Problems upgrading an existing field to a multi-field

I'm having trouble upgrading an existing field to a multi-field. I've done this before with no issues on other fields. I think the issue here is that the original mapping specifically defines an analyzer: "mappings" : { "person" : { "properties" : { "domain_titles" : {

Re: excessive merging/small segment sizes

Could you pull all hot threads next time the problem happens? Mike McCandless http://blog.mikemccandless.com On Mon, Jul 7, 2014 at 3:47 PM, Kireet Reddy wrote: > All that seems correct (except I think this is for node 6, not node 5). We > don't delete documents, but we do some updates. The v

Re: Elasticsearch twitter river filtered stream question

It uses the filter functionality provided by Twitter API. --  David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 7 juillet 2014 à 21:54:02, Josh Harrison (hij...@gmail.com) a écrit: Quick question about the ES twitter river at https://github.com/elasticsearch

Re: Best practice to backup index daily?

The Elasticsearch curator now supports snapshots: https://github.com/elasticsearch/curator http://www.elasticsearch.org/blog/elasticsearch-curator-version-1-1-0-released/ You would still need to use cron to schedule tasks, but it would be a curator task instead of a direct curl request. Cheers,

Re: How to limit the fields of response when I search a keyword?

I responded differently to your other similar question, but you can also limit the fields, but explicitly asking for the set that you want: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html Cheers, Ivan On Sat, Jul 5, 2014 at 2:32 AM, 纪路 wrote:

Re: How to limit fields of response doc when I search certain keyword?

If I understand you correctly, you want to view the distribution of gender based on the results of a query? In that case, you want to look into aggregations, which work on top of the result set that is returned. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregati

issues with using repository-hdfs plug in for snapshot/restore operation

I am using elasticsearch 1.2.1 and CDH 4.6. quick start vm. My ES server is installed on the same vm. I have one successful senario: I used light version and add the result and command `hadoop classpath` to ES_CLASSPATH But I encoutered errros with the default version and hadoop2 version. Here

Re: Search thread pools not released

Yeah, already traced it back myself. Been using Elasticsearch for years and I have been only setting query timeouts. Need to re-architect a way to incorporate client-based timeouts. Had two different elasticsearch meltdowns this weekend, after a long period of stability. Both of them different and

Re: Search thread pools not released

Yes, actionGet() can be traced down to AbstractQueueSynchronizer's acquireSharedInterruptibly(-1) call http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/AbstractQueuedSynchronizer.html#acquireSharedInterruptibly(int) in org.elasticsearch.common.util.concurrent.BaseFuture which "

Saved Kibana Dashboards & Tribe

I have a Tribe node and a Kibana instance for the Tribe node. When I try to save a dashboard on the Kibana instance for the Tribe node, I get the following errors: PUT http://{tribe}:9200/kibana-int_{tribe}/dashboard/Logs%20Search

Re: Custom Plugin for specifying custom filter attributes at query time

In Elasticsearch, you can extend the existing queries and filters, by a plugin, with the help of addQuery/addFilter at IndexQueryParserModule Each query or filter comes in a pair of classes, a builder and a parser. A filter builder manages the syntax, the content serialization with the help of XC

Best practice to backup index daily?

I am able to take a snapshot of the index and back it up to AWS S3. What is the best way to automate this approach and have it done daily say every days at 12 midnight? I am aware that I can probably do it with crontab, but curious if other are doing it differently? -- You received this messa

Elasticsearch twitter river filtered stream question

Quick question about the ES twitter river at https://github.com/elasticsearch/elasticsearch-river-twitter The twitter streaming API allows you to filter, and you apparently get up to 1% of the stream total, with our search queries. So, if I were filtering for "coffee", I'd get "coffee" tweets th

Setting id of document with elasticsearch-hadoop that is not in source document

I am trying to update an elasticsearch index using elasticsearch-hadoop. I am aware of the *es.mapping.id* configuration where you can specify that field in the document to use as an id, but in my case the source document does not have the id (I used elasticsearch's autogenerated id when indexi

Re: excessive merging/small segment sizes

All that seems correct (except I think this is for node 6, not node 5). We don't delete documents, but we do some updates. The vast majority of documents get indexed into the large shards, but the smaller ones take some writes as well. We aren't using virtualized hardware and elasticsearch is t

Re: Opening TransportClient connection per Index

You can enlarge thread pools in TransportClient, also Netty worker threads. User session states should be managed in the front-end service (reverse proxy or Java middleware e.g.) so it is still ok to use a singleton TransportClient since it is stateless. It handles requests and the corresponding re

Re: How can I do intersection or union operation with two facets filter？

Did you try the following query? "query":{ "filtered" : { "filter" : { "bool" : { "should" : [ {"term" : { "field_B" : "" }}, {"term": {"field_B": "bbb"}} ] } } } }, "fa

Re: Search thread pools not released

Still analyzing all the logs and dumps that I have accumulated so far, but it looks like the blocking socket appender might be the issue. After that node exhausts all of its search threads, the TransportClient will still issue requests to it, although other nodes do not have issues. After a while,

Re: java.lang.NoSuchFieldError: ALLOW_UNQUOTED_FIELD_NAMES when trying to query elasticsearch using spark

Thanks for the analysis. It looks like Hadoop 1.0.4 POM has an invalid pom - though it uses Jackson 1.8.8 (see the distro) the pom declares version 1.0.1 for some reason. Hadoop version 1.2 (the latest stable) and higher has this fixed. We don't mark the jackson version within our POM since it's a

Re: Memory issues on ES client node

I think this is not a concurrency problem but the cluster wanted to deliver a huge portion of data (just guessing about such query responses because I do not know anything about the queries on your system). Client timeout of receiving data is around 30 secs IIRC. It only means that it could be pos

Re: New Logstash setup issue with iptables

Thank you, Linus! That did the trick! Peace and Joy, Lois On Saturday, July 5, 2014 8:51:19 AM UTC-4, Linus Askengren wrote: > > Hi Lois, > > I had the exact same problem, the discovery is running on udp 54328 by > default >

Re: ingest performance degrades sharply along with the documents having more fileds

Thanks Shay for updating us with perf improvements. Apart from using the default parameters, should we follow the guideline listed in http://elasticsearch-users.115913.n3.nabble.com/Is-ES-es-index-store-type-memory-equivalent-to-Lucene-s-RAMDirectory-td4057417.html Lucene supports MMapDirecto

Warmer queries - many small or one large

Hello The subject pretty much says it all...is there an advantage one way or the other to having several (or many) small (single term) warmer queries rather than a single large query that searches all desired fields? -- You received this message because you are subscribed to the Google Groups

Opening TransportClient connection per Index

I have a use case in which i want to create per index per account (assume an account represents an user), all data belong to that user will be kept in that index. My question is - what if we create connection per index and keep it alive during the user session. So this means for 100 active users

Re: java.lang.NoSuchFieldError: ALLOW_UNQUOTED_FIELD_NAMES when trying to query elasticsearch using spark

Here is the gradle build I was using originally: apply plugin: 'java' apply plugin: 'eclipse' sourceCompatibility = 1.7 version = '0.0.1' group = 'com.spark.testing' repositories { mavenCentral() } dependencies { compile 'org.apache.spark:spark-core_2.10:1.0.0' compile 'edu.stanford.nlp:stanfor

Re: ES Hadoop--Index only new documents without killing job from exceptions?

Thanks, Costin. That makes sense; I've also commented on the issue you mentioned on github. Having more control over the when to fail a job or choose to ignore certain errors would definitely be a great feature from my perspective. I've encountered a few different areas where I think extra contro

Re: excessive merging/small segment sizes

Indeed there are no big merges during that time ... I can see on node5, ~14:45 suddenly merges are taking a long time, refresh is taking much longer (4-5 seconds instead of < .4 sec), commit time goes up from < 0.5 sec to ~1-2 sec, etc., but other metrics are fine e.g. total merging GB, number of

Re: Terms Aggregation and scope

Awesome, just ran the curl command below, works fine! curl -XGET 'http://localhost:9200/_search?pretty=true' -d '{ "size": 0, "aggs": { "datum_uitspraak_ymd": { "terms": { "field": "datum_uitspraak_ymd", "include":"2014.*", "size":1 } } } }' Op Jul 7, 2014, om 3:10 PM heeft Colin Goodhear

Re: Latitude -> Lat, Longitude -> lon

Thanks for your reply. I know I tried that, but it's in an array, so Iw ould need to iterate or something, because trying to map the path leads to an error if there is an array: Here, items is an array: ctx.doc.items.location.longitude That's why i'm looking for an alternative solution. On Mon

Clustering/Sharding impact on query performance

Hi, *SCENARIO* Our Elasticsearch database has ~2.5 million entries. Each entry has the three analyzed fields "match", "sec_match" and "thi_match" (all contains 3-20 words) that will be used in this query: https://gist.github.com/anonymous/a8d1142512e5625e4e91 ES runs on two *types of server

Re: Custom Plugin for specifying custom filter attributes at query time

Hi, A little clarification: Assume sample data set of 50M documents. The documents need to be filtered by a field, Field1. However, at indexing time, this field is NOT written to the document in Lucene through ES. Field1 is a frequently changing field and hence, we will like to maintain it outsid

Re: Use arrays as update parameters with elasticsearch-hadoop-mr

Costin-- Great news, thanks for the update! James On Sun, Jul 6, 2014 at 4:41 PM, Costin Leau wrote: > Hi James, > > Fwiw, I plan to address this bug shortly - as you pointed out, the JSON > array needs to be handled separately before passing its content in. > > > On Thu, Jul 3, 2014 at 8:58

Re: Terms Aggregation and scope

Glad it worked. Yes, there are options for includes and excludes patterns. Take a look at the following link for information on how to use them. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_filtering_values Colin On Mo

Re: Terms Aggregation and scope

Hi Colin, Thank you, that works perfectly! Is there any way to limit the key-value pairs by a certain parameter, in the example below: to limit the aggregation to "datum_uitspraak_ymd" keys that start with "2014"? Or does that require the combination of a filter and an aggregation? Many than

Re: ES Hadoop--Index only new documents without killing job from exceptions?

I would recommend indexing the document since it's a 'cheap' operation per document and it covers the potential differences between the docs. Also from a performance POV you are not going to lose much since you are anyway sending the doc to ES, which does hashing and returns the error to the user.

How to limit the fields of response when I search a keyword?

Dear all: I have a reasonable need, but I can't find how to deal with it on es official docs and books, is anyone know, please teach it to me! thank you! I have a large set of docs, which hold a lot of fields, such as: uid2 = { "id": 1404999597, "idstr": "1404999597", "class": 1, "screen_name":

How to limit fields of response doc when I search certain keyword?

Dear all: There is a reasonable need, but I don't find a solve in official doc or book, can you help me? I have a large set of docs, which contains a lot of fields, such as: { "id": 1404999597, "idstr": "1404999597", "class": 1, "screen_name": "主播梦桐", "name": "主播梦桐", "province": "11", "city": "1

Re: Inter-document Queries

Hi, only saw this now I wouldn't worry too much about high space complexity - storage comes cheap nowadays, and the general practice in many systems is to store raw data and do processing on demand (most commonly known approach is event sourcing). I can understand an argument about high space com

Re: Terms Aggregation and scope

Diederik, To increase the number of terms returned by the terms aggregation you will need to add the 'size' parameter to your aggregation. The below curl command will return you the top 200 terms (ordered by decending doc_count). curl -XGET 'http://localhost:9200/_search?pretty=true' -d '{ "siz

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

On Mon, Jul 7, 2014 at 4:16 AM, Grégoire Seux wrote: > Andrew, > > Have you found a solution (or explaination) to your issue ? > We are using elasticsearch 1.1.1, what about you ? Hi, I haven't learned anything new. To be clear about my problem, I am aware that I must re-enable routing after hav

Terms Aggregation and scope

Dear list, I need to create an aggregation by a specific field, named "datum_uitspraak_ymd'. I am using the below curl command and it works fine in a sense that it returns the aggregation listed below. While this result seems OK enough, it seems that the keys listed in the aggregation are limi

Re: have we a way to use highlight and fuzzy together ?

I want to combine like this : GET my_index/my_type/_search?pretty=true {"size": 50, "query": { "multi_match": { "query": "my words", "fields": ["title_doc"] },*"fuzzy": 0.2* }, *"highlight" : {* *"fields" : {

How can I do intersection or union operation with two facets filter？

Dear All! I have some docs: {"field_A":"aaa","field_B":"bbb"} {"field_A":"aaa","field_B":"ccc"} {"field_A":"bbb","field_B":"bbb"} {"field_A":"bbb","field_B":"bbb"} {"field_A":"bbb","field_B":"eee"} {"field_A":"aaa","field_B":""} {"field_A":"ccc","field_B":""} first step: { "query":{ "filter

Re: Elasticsearch with azure cloud plugin

This doesn't sound like it's Azure-specific. For one, I'd try to use ES 1.2.1 as there has been a lot of work in that area (of GC and threads). I'd also try to avoid using the Azure plugin as long as possible and use Unicast instead - I've just blogged about exactly that, see http://code972.com/bl

What is exactly flush in node stats?

Hi, When i see the node stat of my cluster is show something like this: "flush" : { "total" : 30, "total_time_in_millis" : 6964 } My translog flush interval is 180 minutes or 1200 mb , so i think flush should be done based on these values. But if i keep on checking

Re: have we a way to use highlight and fuzzy together ?

I want to combine like this : GET my_index/my_type/_search?pretty=true {"size": 50, "query": { *"fuzzy": 0.2,* "multi_match": { "query": "my words", "fields": ["title_doc"] } }, *"highlight" : {* *"order" : "date_doc"

Query DSL problem

Dear All！ Can I do intersection or union operation with facets filter result？ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegr

How to Know Progress of Backup And Restore

Hi, I need Progress information of Back up and Restore in Elastic Search. How can i find how much data is processed. Please help me in this. -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/How-to-Know-Progress-of-Backup-And-Restore-tp4059347.html Sent from the

Re: Elasticsearch, how to make view with it?

Merci David! Le lundi 7 juillet 2014 10:20:19 UTC+2, David Pilato a écrit : > > Just answered on the french ML as well: > > You can use aliases on top of your indices and add a NGnix layer for > example to filter URLS per user/group. > > > > -- > *David Pilato* | *Technical Advocate* | *Elastics

Re: Elasticsearch with azure cloud plugin

Yes this is the case. I will post On Mon, Jul 7, 2014 at 11:24 AM, dadoonet [via ElasticSearch Users] < ml-node+s115913n405934...@n3.nabble.com> wrote: > So you are saying that when a node suddenly disappear for whatever reason > (network, GC…), he can't rejoin again the cluster automatically so

Re: Elasticsearch with azure cloud plugin

So you are saying that when a node suddenly disappear for whatever reason (network, GC…), he can't rejoin again the cluster automatically so you have to restart it? If so, could you open an issue in cloud-azure plugin repo and if possible attach logs from the both nodes? Thanks --  David Pila

Re: Elasticsearch, how to make view with it?

Just answered on the french ML as well: You can use aliases on top of your indices and add a NGnix layer for example to filter URLS per user/group. --  David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 7 juillet 2014 à 09:41:11, Villiers Tientcheu Ngandjeu

RE: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

Andrew, Have you found a solution (or explaination) to your issue ? We are using elasticsearch 1.1.1, what about you ? -- Grégoire -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from

Elasticsearch, how to make view with it?

Hello dear! I get three kind of logs to index with Elasticsearch, let say X, Y and Z for three different teams in the business! With SQL, it was possible to make view for customers. How can I make view with Elasticsearch for the teams. Or how can restrict access to data between them:Team X just

Elasticsearch with azure cloud plugin

Hello, We are using ES with the azure cloud plugin for node communication. Our current set-up is : 2 data node (hosted on azure ubuntu 14.04 VM) 3.5 giga RAM, 2 Cores (AKA medium VM) ES version 1.0.0.0. 5 shards and 2 replicas. The storing and query are done by Kibana and logstash which are hosted