Distributed Transaction Tracing using Elastic Search

Sometime back i did a basic Proof of Concept of Distributed Tracing, based on Google Dapper(http://research.google.com/pubs/pub36356.html ), using Elastic Search. Her

Re: Graylog2 recent is not working. cluster state is red

Your cluster is in a red state, which means you have unassigned primary shards. Install an ES plugin like elastichq or kopf to give you an idea on what is happening. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 1

Re: network.publish_host does not seem to do what it says it does.

That doesn't make a lot of sense then. Why would you be able to set the publish host to something different if they both have to bind to the same interface? I'm not understanding what the purpose of these configuration bits is. I was also told in the IRC room that this was how you separated

Graylog2 recent is not working. cluster state is red

Hi Experts, >From past few days there is a problem in our graylog2. *Need your help to resolve the issue* *Problem: while we are able to see the all messages, the recent is not working.* - On the home page we see this message oError! Syntax error in search query or empty inde

Re: Nested Document sorting and filtering

>From what i read here http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_nested_sorting_example I understand that sorting is supported within a nested field. In case of nested_filter i would like to filter out the nested documents that doesnt match

Re: Is it possible to add a customized merging strategy to alleviate split-brain impact?

I have no idea, but here is a recent comment: https://github.com/elasticsearch/elasticsearch/issues/2488#issuecomment-40301769 -- Ivan On Wed, Apr 16, 2014 at 4:41 PM, Jing Liu wrote: > Thanks Ivan for your response. > Is it possible to know when the new solution will come out? ES 1.2? > > Th

Re: Is it possible to add a customized merging strategy to alleviate split-brain impact?

Thanks Ivan for your response. Is it possible to know when the new solution will come out? ES 1.2? Thanks, Jing On Wednesday, April 16, 2014 4:30:15 PM UTC-7, Ivan Brusic wrote: > > I believe that the Elasticsearch team is more focused on eliminating > split-brain than the after effects of a s

Re: Memory Issues with our cluser set up

Document TTL can be a resource hog, you might want to look at logstash's curator instead to manage expiration. It might also be beneficial to increase your shard count so you have one shard per node, alternatively look at routing.allocation.awareness.attributes. ES will distribute things as evenly

Re: Is it possible to add a customized merging strategy to alleviate split-brain impact?

I believe that the Elasticsearch team is more focused on eliminating split-brain than the after effects of a split brain. Recent comments indicate that they are actively working on a solution. The new consensus algorithm (Paxos/RAFT?) will undoubtedly affect how conflicts are reconciled. Cheers,

Re: network.publish_host does not seem to do what it says it does.

As far as I am aware you can't split the two, they both need to be on the same interface/IP. I don't know if this is a bug or if the docs are incorrect. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 17 April 2014 0

Re: Running two independent clusters

Take a look at logstash as the conduit, it should be more than suitable. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 16 April 2014 21:45, Steven Williamson wrote: > Hi, > > I'm looking at running two ES cluster

network.publish_host does not seem to do what it says it does.

I'm trying to bind the API to 127.0.0.1 and use the apache reverse proxy settings with kibana to talk to ES. This works as I expected. However, I need to bind the inter-cluster communication to a real IP address, supposedly with network.publish_host, but this does not work. Both bind to 127.

Re: How to install kibana-authentication-proxy ??

Its just a node application, and the github page has installation instructions https://github.com/fangli/kibana-authentication-proxy Kevin On Wednesday, April 16, 2014 5:48:00 AM UTC-7, Andrea Martines wrote: > > Hello, I'm now looking for security in my elasticsearch cluster, and I > would lik

Re: Another odd ES freak out...

Hello - we are seeing the same issues, were you able to resolve this? thanks! On Sunday, 5 February 2012 16:48:25 UTC-6, Grant wrote: > > So we seem to be having recurring incidences of ES nodes getting into > a very odd state. In this particular case, one node because > unresponsive to test po

Re: Ranking result using key-words

You could do something like *"+\"Gisele Bundchen\"^5 "legal settlement"^1 jail^2 lawsuit^2"* which would mean results must have Gisele Bundchen while others are optional and help in ranking results. On Wednesday, April 16, 2014 4:57:41 PM UTC-4, Willian Gigliotti wrote: > > Hi, > > I

Ranking result using key-words

Hi, I have read only access to an ElasticSearch 0.90 version with a huge collection of Brazilian Official Gazette pages. We are using that to search for people names like "Gisele Bundchen". Our search is something like this: *{* * "from": 0,* * "size": 2,* *

Re: Existing multi-index cluster; now I need to search across it

Thanks Binh. That looks very promising. I was able to make a multi-index multi_match querying my index_named fields. Here's my Sense session: https://gist.github.com/AndrewO/10930544 I noticed a slight but consistent increase in size (~200b for 1 document) for an index with an index_name'd mapp

Re: Analyzing URLs for regexp queries

Hi Matt, that is curious, could you please try to enably trace logging for elasticsearch-py and look what exactly is being sent? My guess is that there is something that needs to be escaped in python though what that might be alludes me for the time being. to tenable the logging just do: import

Re: Indices total indexing time metric given by marvel

Hi, By the title, I assume you refer to the chart show in the Node Stats dashboard. This one is tricky. It shows you the total spent on indexing per second. This is best explained by an example - say you have 2 shards on a node. Say that each indexes 100% of the time (continuously indexing) and

ES v1.1 continuous young gc pauses old gc, stops the world when old gc happens and splits cluster

We have a 4 node (2 client only, 2 data/master nodes with 25G memory allocated to ES and 12 cores each) ES cluster, storing an index with 16 shards, ~200GB and 1 replica. Recently running scan/scroll requests to dump data and other faceting requests, the nodes disconnected from each and we had

Re: Calculating ArcDistance of a geopoint field within an object field

In the linked example I have only one record, and it contains data for all fields. On 16 Apr 2014, at 18:57, "Binh Ly" mailto:binhly...@yahoo.com>> wrote: Strange. The only thing I can think of is you have a document that does not have a value for the field: doc.all_places.location_point. Can

Re: Sense on github abandoned?

Here are a few tools that support sending REST queries to Elasticsearch http://mobz.github.io/elasticsearch-head/ https://github.com/lmenezes/elasticsearch-kopf https://github.com/polyfractal/elasticsearch-inquisitor -- Ivan On Wed, Apr 16, 2014 at 11:27 AM, jrizzi1 wrote: > Its not about ab

Re: Sense on github abandoned?

Its not about abandoning, it applies more to getting started, which is what we just did last month. There is literally no way I would have been able to get a working Elasticsearch solution with advanced mappings and multiple plugins without using sense, and did so within a short timeframe, but my

Re: Is it possible to add a customized merging strategy to alleviate split-brain impact?

Hi Brain, Thanks for your inputs. Yes, the above two cases are found during our tests. Case 1 will be handled automatically. Hopefully could get attention from ES team for the case 2 solution. Jing On Wednesday, April 16, 2014 6:43:14 AM UTC-7, InquiringMind wrote: > > Jing, > > I don't have

Re: Calculating ArcDistance of a geopoint field within an object field

Strange. The only thing I can think of is you have a document that does not have a value for the field: doc.all_places.location_point. Can you narrow down your search to only documents that contain that field and check? Thanks! -- You received this message because you are subscribed to the Goo

Re: Local testing not working

Just setRefresh(true) when you index a doc, and it should be there immediately when you search it afterwards. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to e

Re: Cluster configuration

Thank you Ivan. I am using NEST c# client, it has sniff functionality, I will use that. On Wednesday, April 16, 2014 12:14:12 PM UTC-5, Ivan Brusic wrote: > > It depends on which client you use. If you use the Java TransportClient > with "sniff" enabled, it detect all nodes upon client startup a

Re: how to merge the result after multi search, then I can sort them like sorting in queryDSL?

Unfortunately not at the moment, each result is independent and you'll need to "merge" them yourself if necessary. However, I don't know it this helps, but you definitely can search multiple indexes/types in 1 search call, if it helps. -- You received this message because you are subscribed to

Re: Synchronization problems

There is a near-realtime aspect to ES. What this means is by default, documents are searchable within 1 second (refresh interval) after they are indexed. You may index a document and explicitly refresh it immediately in which case, you can search for it immediately. For example: client.prepareI

Re: Sense on github abandoned?

Just playing the devil's advocate, but Lucidworks, the main commercial entity behind Solr, also has numerous paid closed-source offerings: http://www.lucidworks.com/lucidworks-search/ As great as Sense is, it would not be a deal breaker if it was not available publicly. I used Elasticsearch for ye

Re: Cluster configuration

It depends on which client you use. If you use the Java TransportClient with "sniff" enabled, it detect all nodes upon client startup and use that list for its request. All other clients (REST) round robin between whatever servers were set in the properties. -- Ivan On Wed, Apr 16, 2014 at 10:0

Re: Cluster configuration

Thank you. That worked. Now when i bring down any master, cluster keeps going with other 3 master nodes. I took down a master node that is bound to localhost:9200 on one machine1. I can still query using localhost:9201. Does it mean, do we need to change the URL on client from 9200 to 9201 if

Re: Unable to delete indexes - "failed to process cluster event (acquire index lock) within 30s"

Similar error [2014-04-16 16:57:08,464][DEBUG][action.admin.indices.create] [Jerry Jaxon] [myindex-2014.04.16] failed to create org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (acquire index lock) within 30s at org.elasticsearch.cl

Re: Manual copy of indices

The safest way is to not have the index at all in the destination node in the first place. Then you shutdown the the destination node, copy everything from the named index folder from the source node and put it under the indices folder in the destination node, and then start it back up again.

Re: Cluster configuration

I believe if you have two or more nodes on a single machine, you need to specify each in the unicast host property: discovery.zen.ping.unicast.hosts: ["machine1:port1", "machine2:port1", "machine1:port2", "machine2:port2"] Where port1 and port2 are probably 9300 and 9301. -- Ivan On Tue, Apr

Re: Sense on github abandoned?

Well, this is perhaps a too strong of a statement. AFAIK SOLR does not have any comparable front-end and not many would abandon elastic because marvel is a commercial product. I respect Elastic team trying to monetize its product(s) and do it in a rather nice way. So I would be perfectly happy

Indexing dates as text with synonyms for month and maybe 2/4 digit year

Hi, Is there a filter I can use when indexing a date (ISO date format without time) as text field? by default it is split on "-" and I would like to keep it and add month name as synonym for month number. on the other note, is there any way to make it not to be split on separate tokens when

Re: array of strings vs string

Hello Aleh , Both should be good for your purpose. But then if you want to match against "abc def" ( that is with the space) tomorrow , the array type will alone help. You can disable the analyzer for the field and achieve the functionality which you cant if its a normal string. Thanks

Re: Sense on github abandoned?

What this essentially does is limit a developer's options I went to my boss, and laid out the plans for implementing ES, and told them there was no cost, open-source Now i have to go back and explain we need licensing on our production VM if we need to use sense on that VM, we don't need marvel

Re: Completion Suggester on existing index and fields

Unfortunately not for the completion suggester. The reason is because the completion suggester uses an optimized data structure that is populated at index-time. So you have to reindex your data (with the completion type) to build that data structure. -- You received this message because you ar

Memory Issues with our cluser set up

Hi all We've noticed that a couple of our data nodes have been having memory issues eventually leading to them dropping out of the cluster. Other nodes in the cluster are nowhere near their limits in terms of memory usage. Looking at the gc logs of those problem boxes it seems like it reaches a

Re: Parametric Search with aggregations

Or you can make 2 separate aggregations, one for your number crunching, and another for your global parametrics. Something like this conceptually: { "aggs": { "metrics": { #all your standard metrics aggs here, all filtered by your query }, "parametrics": { "global": {} #

Re: Existing multi-index cluster; now I need to search across it

Not sure if this would help, but in your mapping, on a field, you can specify "index_name", in which case you can refer to that name in your queries. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop rece

Re: Is it possible to add a customized merging strategy to alleviate split-brain impact?

Jing, I don't have much experience with ES in a production cluster environment; all my experience has been with the Java API for mapping, bulk load, and query logic, and with huge databases and things like that. But my 3-node test ES cluster has gathered some dust over the past few months as ot

Re: Question on search on "_source"

1) You likely want a term filter: { "query": { "filtered": { "filter": { "term": { "searchItems.ifpresent": 1 } } } } } 2) If your document is structured like above, you cannot extract only a partial (searchItems element) that matches your criter

Re: Lingo3g Attribute to retrieve labels name with suppressing multiple keyword

Hello All, Did any one encountered the same issue if yes then please let me know the solution. -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Lingo3g-Attribute-to-retrieve-labels-name-with-suppressing-multiple-keyword-tp4054193p4054264.html Sent from the Ela

array of strings vs string

Hi i'd like to have a field in document to which i'm going to append strings and use span_near queries to search for a specific sequence of strings in it i wonder if that kind of usage makes sense, and if using array of strings instead of one long string would be better? i.e. which document

How to install kibana-authentication-proxy ??

Hello, I'm now looking for security in my elasticsearch cluster, and I would like to ask the user to login, and then to give him the right index. For example, the user adminSec would have access to the SECURITY-* index. So I found the kibana-authentication-proxy plugin but they don't explain ho

Re: Cross Fields w/ Fuzziness

I have the same issue, I would like to be able to do a fuzzy search for some of my fields in a multi_match query of type cross_fields. Right now I have to do two queries but gives me a disjunct set rather than a conjunctive result: { "query": { "bool": { "should": [ {

Re: 1 large index vs several smaller indexes

I would separate the performance issue from the logical structure of your domain. You really need to thing in terms of numbers of documents and shards (and not indices). You may want to look into using index aliases, which can take a filter. That way you can have one index and several branch a

out of memory error elastic search due to desearilzation

org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [fetch], [reduce] ; shardFailures {[whK0hSszSxqPMfRBHGHj9g][bw-2014-04-15-03][2]: RemoteTransportException[Failed to deserialize response of type [org.elasticsearch.search.fetch.FetchSearchResult]]; nested: Tra

Re: Does es-amazon-s3-river work with elasticsearch 1.0.2 ?

You should open an issue in https://github.com/lbroudoux/es-amazon-s3-river And ask for an upgrade. Or you need to fix it yourself and build a new version. --  David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 16 avril 2014 à 11:59:57, Kuo Hugo (tonyt...@gmai

Running two independent clusters

Hi, I'm looking at running two ES clusters across two sites. I.e avoid clustering across a WAN link, and to provide an environment to test upgrades etc. I'm looking at using RabbitMQ and the RabbitMQ river plugin to stream updates to both clusters to keep them in sync. Is anybody already doin

Does es-amazon-s3-river work with elasticsearch 1.0.2 ?

Version : elasticsearch 1.0.2 (installed from ubuntu repo) es-amazon-s3-river : https://github.com/lbroudoux/es-amazon-s3-river Hi all, There's a problem regarding to s3-river plugin. I followed the instruction to install the plugin in my elasticsearch box here. A Warning meg shows up in th

Indices total indexing time metric given by marvel

Hi, I'm new to elastic search and marvel . I'm currently investigating marvel and I want to know if the "Indices total indexing time" shown by marvel is "sum of the times taken by each shard" or "time between the arrival of request and completion of indexing". For example if there are 5 shard

Synchronization problems

Hi So I start a local cluster and place and index in it. The settings are ImmutableSettings.Builder builder = ImmutableSettings.settingsBuilder() .put("node.name", "node-test-" + System.currentTimeMillis()) .put("node.data", true) .put("cluster.nam

Re: ELK stack needs tuning

Well once you go over 31-32GB of heap you lose pointer compression which can actually slow you down. You might be better off reducing that and running multiple instances per physical. >0.90.4 or so compression is on by default, so no need to specify that. You might also want to change shards to a

ELK stack needs tuning

Hi all, At bol.com we use ELK for a logsearch platform, using 3 machines. We need fast indexing (to not loose events) and want fast & near realtime search. The search is currently not fast enough. Simple "give me the last 50 events from the last 15 minutes, from any type, from todays indices,

Re: ES cluster unable to assing new shards

Yesterday I set the replica count to 0 with curl -XPUT $(hostname -f):9200/_settings -d'{'index': { 'number_of_replicas':0}}' and today the ES cluster assigned the new shards as it should. I have now set the replica count back to 1 and will see if that's the problem tomorrow. On Tuesday, Apr