processing [index-aliases]: execute - whats that?

Hey guys, for some testing reasons, i activated the DEBUG log in elasticsearch and saw that the following messages appears every 5-10 seconds: [2014-10-20 08:18:09,507][DEBUG][cluster.service ] [NODE] processing [index-aliases]: execute [2014-10-20 08:18:09,509][DEBUG][cluster.service

Joda problem!!!!!!

Hi! I've been struggling with two date formats. I'll give you a sample and the corresponding date format I've used: 1. Oct 20, 2014 8:42:41 AM : MMM d hh:mm:ss aa 2. Mon Oct 20 06:42:41 + 2014 : EEE MMM d HH:mm:ss Z I've used the following mapping: { settings : {

Re: Multiple Timezones in Elasticsearch/Kibana

On Thursday, October 16, 2014 at 18:57 CEST, Kellan Strong vaid.kel...@gmail.com wrote: I am having a problem with different timezones sending their information to elasticsearch/kibana. One of the logs that is sending is at UTC time however the elasticsearch box is at local time zone.

Re: Joda problem!!!!!!

You have a comma in Oct 20, 2014 8:42:41 AM Oct 20 2014 8:42:41 AM will work. Jörg On Mon, Oct 20, 2014 at 9:06 AM, Peter Litsegård peter.litsega...@gmail.com wrote: Hi! I've been struggling with two date formats. I'll give you a sample and the corresponding date format I've used: 1.

Re: Joda problem!!!!!!

Hi Jörg! Thanks for your response. However, IMHO I don't think this is the problem. I changed the Joda specifikation in (1) to MMM d',' hh:mm:ss aa and when I used this simple test code DateTime dt = new DateTime(); DateTimeFormatter fmt =

Re: Joda problem!!!!!!

Maybe the old mapping is still active? This works here: import org.elasticsearch.common.joda.FormatDateTimeFormatter; import org.elasticsearch.common.joda.Joda; import org.elasticsearch.common.joda.time.DateTime; import org.testng.annotations.Test; public class JodaTest { @Test public

How to filter low term-frequency documents in elasticsearch?

Hey guys: We are using elasticsearch to push newly added documents to users. Thus the search results are ranked according publish time. In this case, users often receive low relevance documents, such as documents in which query words only appear once. How can we filter such documents in

Re: Joda problem!!!!!!

Just to give you a sample. Partial doc JSON: { id: 524107608429699072, createdAt: Oct 20, 2014 9:59:25 AM, source: { created_at: Mon Oct 20 07:59:25 + 2014, source: a href=\http://ElwynRoad.com\; rel=\nofollow\Elwyn Road/a, retweet_count: 0, retweeted: false,

Re: Joda problem!!!!!!

I decided to drop this for now and provide the timestamp field (createdAt) in milliseconds... Thanks for your help though... /Peter On Monday, October 20, 2014 9:06:44 AM UTC+2, Peter Litsegård wrote: Hi! I've been struggling with two date formats. I'll give you a sample and the

Re: Joda problem!!!!!!

I works here PUT /myindex PUT /myindex/test/_mapping { test: { properties: { created: { type : date, format : MMM d',' KK:mm:ss aa, store : true, index : no }

Re: Joda problem!!!!!!

Strange I decided to drop this for now due to a presentation tomorrow. Will pick it up afterwards...:) Cheers On Monday, October 20, 2014 9:06:44 AM UTC+2, Peter Litsegård wrote: Hi! I've been struggling with two date formats. I'll give you a sample and the corresponding date format

nested filter query issue

Hi All, I am working on nested filter query.But this query is searching only on nested object properties of type int ,double,date.But not searching when the properties of string type. My document structure is given below- sectionResults: [ { SectionName: Project Details,

Re: another EC2 publish_host public ip problem

Hi Martin, Did you check your firewall settings? Did you open 9200 ports so they can be accessible from your local machine? BTW, those settings are not used. You can comment/remove them: discovery.zen.ping.multicast.enabled: false network.publish_host: 54.31.403.195 --  David Pilato | 

update by query and refresh

Hello, We use _update_by_query plugin to bulk update the documents. In the tests we've hit an issue where not all the documents are updated because the index is may not be refreshed before we do _update_by_query. We have refresh interval set to 1 sec and this issue won't happen very often in

How to find the thread pool size of an ElasticSearch cluster?

I'm writing the concurrent code to send bulk index to ElasticSearch, and sending this query to get the thread pool size: GET /_cat/thread_pool?vh=host,bulk.active,bulk.queueSize The response is hostbulk.active bulk.queueSize 1D4HPY1 0 50 1D4HPY2 0

Re: How to find the thread pool size of an ElasticSearch cluster?

bulk.queueSize is the maximum size before requests are rejected. Jörg On Mon, Oct 20, 2014 at 12:09 PM, truong ha truonghat...@gmail.com wrote: I'm writing the concurrent code to send bulk index to ElasticSearch, and sending this query to get the thread pool size: GET

Term query on long type failing

Hi , I am extracting tweets from twitter and i found the following issue. On doing a terms facet on field retweet.id , i received some user ID's. Now on doing a term query on one of the value obtained , I am not getting any result. The facet is as following - { facets: { terms: {

Re: ES error while creating index on Solaris

Hi Abhinav It would be good to know exactly where this problem is coming from. Is it the way that Logstash adds the template, or is it in the Elasticsearch layer. Please could you try something: * Delete the existing template and index in Elasticsearch * Take the Logstash template and create it

Re: Question: shard awareness allocation and shard allocation filtering

Hi Grégoire A couple of comments: 2. at some point (disk on ssds is above 65%), one copy is moved to larger boxes (1 copy is still on ssd to help search, 1 copy on large box) Allocation awareness causes elasticsearch to spread the shards copies across the different values of the attribute.

Re: Term query on long type failing

Hi, The term query aims at querying documents based on the raw bytes of a term. It is not aware of your field mappings while numeric terms are not encoded verbatim: they use a special binary encoding that allows the binary representation of the numbers to be sortable lexicographically. Switching

Re: Term query on long type failing

Hello Adrien , Thanks for your reply. But match query is also not working for me - { query: { match: { retweet.id: 524120494964535300 } } } Gives 0 results. Thanks Vineeth On Mon, Oct 20, 2014 at 5:50 PM, Adrien Grand adrien.gr...@elasticsearch.com wrote: Hi,

Re: update by query and refresh

Hi Igor, It really depends on your indexing rate. If you plan on performing no more than one refresh per second, things will be fine (this is what elasticsearch does by default). However, running refresh much more often could cause a lot more flush/merge activity, and this will hurt not only your

Re: Search Plugin to intercept search response

Hi Jörg, thanks for response! I use default 'elasticsearch' cluster name and 'ordinary' match_all (as below) Client client = new TransportClient().addTransportAddress(new InetSocketTransportAddress(localhost, 9300)); SearchResponse response = client.prepareSearch().execute().actionGet(); works

JRE 1.7.0_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak?

### JRE 1.7.0_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak?

error: elasticsearch java : org.elasticsearch.ElasticsearchIllegalArgumentException: unknown property [content]

hello, i need help please. i'm trying to index an xml document after convert it to json but i have this error. do you have any idea please? Thanks in advance. --- 0[main] INFO org.elasticsearch.plugins - [Base] loaded [], sites [] Exception in thread main

Indexed documents not showing up in search results

Good afternoon. The subject might be a little misleading as to the true nature of my problem - which I'll try to explain here in as much detail as possible. First of all , I am rather new to Elasticsearch. Secondly , this problem has happened more than once (after dumping all indexes and

Re: How to find the thread pool size of an ElasticSearch cluster?

So in my case, which is the maximum requests I can send: *200 or 50*? On Monday, 20 October 2014 18:09:28 UTC+7, Jörg Prante wrote: bulk.queueSize is the maximum size before requests are rejected. Jörg On Mon, Oct 20, 2014 at 12:09 PM, truong ha truong...@gmail.com javascript: wrote:

Re: Indexed documents not showing up in search results

The elasticsearch log file was not showing any activity during the time when this was happening , but when I restarted the elasticsearch service - I got some warnings (see attached log file) , but then seem to proceed normally and has not complained any more . Note: My process is still

Re: error: elasticsearch java : org.elasticsearch.ElasticsearchIllegalArgumentException: unknown property [content]

Did you set any mapping, any setting? Actually, it would be better to give a full recreation so we can see immediately what is happening. Have a look at http://www.elasticsearch.org/help/ --  David Pilato | Technical Advocate | elasticsearch.com david.pil...@elasticsearch.com @dadoonet | 

Using different mtu value in elasticsearch

Hi all, I am setting up an ElasticSearch 1.3.4 cluster with multihomed servers (a corporate network x.x.x.x and a server-only network at 10Gbs on z.z.z.z, each server having two nic). I ran into some problems as I tried to use the server-only network to do all the cluster traffic and the

how to filter range on cardinality aggregations ES v1.3.4 please ?

Hello, First, thx you for very good ELK ! Anyone help me for how to filter range on cardinality aggregations please ? ok please look my query working example: curl -XGET 'http://localhost:9200/logstash-2014.10.20/_search?search_type=countpretty=true' -d '{ size:999, aggs: {

Re: How does the date_histogram aggregation choose its buckets? Is this tunable?

Hi Adrien, Thanks! The fact that the buckets start calculating from the UNIX epoch is what I didn't understand. The fact that it always landed on October 7th -- which seems like an arbitrary date -- confused me. I did some quick calculations and you're right; midnight on October 7th, 2014, is 545

Kibana histogram doesn't show data on some daily indices

This is a weird one, but hopefully it will make more sense to someone else. I have the following setup: Happy 4-node ES cluster. Data streaming via logstash, several daily rolling indices. In particular I have one index for CPU/mem/disk data called hardware-.MM.DD. I have a Kibana

Re: ES error while creating index on Solaris

When trying to reproduce, I have no luck running ES 1.3.4 on Solaris Sparc. Just like ES 1.2.2, it crashes with SIGBUS, but this time in lzf compression codec. So I will open an issue. Jörg # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0xa) at

Re: ES error while creating index on Solaris

I have added a comment to https://github.com/elasticsearch/elasticsearch/issues/6962 Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to

Re: copy index

Jorg, Not sure what you mean. There is a flag: createIndex=false which means : if the index already exists do not try to create it ie it is pre-created. Import will handle this. Will _push also ? I have another question which affects me: I was hoping that _push would write to the index

Re: copy index

There is no more parameter createIndex, the documentation is outdated - sorry for the confusion. The _push action does not use files. There is no need to do that, this would be very strange, Jörg On Mon, Oct 20, 2014 at 5:12 PM, euneve...@gmail.com wrote: Jorg, Not sure what you mean.

Re: Search Plugin to intercept search response

If you use TransportClient, implementing custom actions like SimpleAction gets a bit tricky: you must install the plugin both on the cluster on all nodes *and* on the TransportClient classpath. The reason is, the TransportClient initiates a proxy action to ask the cluster to execute the action

Re: How to find the thread pool size of an ElasticSearch cluster?

This is not the maximum number of requests you can send. It means when bulk indexing on a node gets too busy and must be queued, the maximum number of actions that are allowed to queue up before a client is notified of rejections is 50. Jörg On Mon, Oct 20, 2014 at 3:57 PM, truong ha

Re: copy index

So just to explain what I want: - I want to be able to push an existing index to another index which has new mappings Is this possible? Preferably it wouldn't go through an intermediate file-system file: that would be expensive and might not be enough disk available. Thanks. On

Write-Read question

I observe, and expect a slight delay between writing an object to ES and being able to fetch it back. In ordinary circumstances, this is not an issue. But, in the context of many agents processing information resources, creating indexes for them, but also needing to refer to those indexes right

Upper limits on indexes/shards in a cluster

Hi, We've been using elasticsearch on AWS for our application for two purposes: as a search engine for user-created documents, and as a cache for activity feeds in our application. We made a decision early-on to treat every customer's content as a distinct index, for full logical separation of

Re: copy index

The recipe is something like this 1. install knapsack 2. create new index. Example curl -XPUT 'localhost:9200/newindex' 3. create new mappings curl -XPUT 'localhost:9200/newindex/newmapping/_mapping' -d '{ ... }' 4. copy data curl -XPOST

Re: ES error while creating index on Solaris

I think this is fixed in v1.3.5 with https://github.com/elasticsearch/elasticsearch/pull/7468 On Monday, 20 October 2014 17:08:55 UTC+2, Jörg Prante wrote: I have added a comment to https://github.com/elasticsearch/elasticsearch/issues/6962 Jörg -- You received this message because you

Re: Upper limits on indexes/shards in a cluster

I might also note: the size of these indexes varies wildly, some being just a few documents, some being thousands, more or less following the power law. On Monday, October 20, 2014 11:34:36 AM UTC-4, David Ashby wrote: Hi, We've been using elasticsearch on AWS for our application for two

Re: Upper limits on indexes/shards in a cluster

How many nodes do you have in your cluster? Have you checked if your nodes run out of file descriptors or heap memory? Jörg On Mon, Oct 20, 2014 at 5:52 PM, David Ashby delta.mu.al...@gmail.com wrote: I might also note: the size of these indexes varies wildly, some being just a few

Re: Write-Read question

Hello Jack , What do you mean by fetching it back , is it a GET or a search. GET is realtime and Search is near realtime. You can use the refresh flag while indexing to make sure search is also real time , but its expensive. Thanks Vineeth On Mon, Oct 20, 2014 at 8:59 PM, Jack Park

Re: Upper limits on indexes/shards in a cluster

The unhealthy clusters were between four and five nodes. We switched to two two-node clusters and those have been stable. Bigdesk reports file descriptors, memory, and CPU all have plentiful headroom in all cases. On Monday, October 20, 2014 11:54:21 AM UTC-4, Jörg Prante wrote: How many

Shard data And Node name relationship

Hello, Suppose I have a cluster of 3 nodes. Each node when it goes down and then comes back up, returns to the cluster with the data it had before shutdown. However, the node does not necessarily join the cluster with the same node name. From the perspective of reinitializing shards and

Re: copy index

Okay when I try that I get this error. It's always at byte 48 Thanks in advance Caused by: java.lang.IndexOutOfBoundsException: Readable byte limit exceeded: 48 at org.elasticsearch.common.netty.buffer.AbstractChannelBuffer.readByte(AbstractChannelBuffer.java:236) at

Re: Search Plugin to intercept search response

Hi Jörg, I understand and what you described is exactly what I did: deployed plugin to both elasticsearch node/server (running single node cluster atm) and to the client code classpath which uses TransportClient - both log INFO 'simple action plugin loaded' on start, yet I can't execute simple

Re: Write-Read question

Hi Veneeth, In previous experiments, where I pre-load the index with a kind of typology -- the index is a topic map --, I set the refresh flag, and still had problems. In doing this work, there are both GET and search queries going in. The actual use case in question relates to a collection of

Re: copy index

By the way Es version 1.3.4 Knapsack version built with 1.3.4 Regards. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to

Re: word delimiter

any thoughts on how I am constructing my search query? I have tried escaping the special characters as well as passing the unescaped special chars. I was hoping to stick with a match query although i tried query string query, and match phrase and term query and had found no solution there.

Error with Elasticsearch

Hi all, I have a setup of ELK stack, and I am experiencing following error with elasticsearch frequently, please suggest me how can I debug this. http://pastebin.com/9j4q1UVd http://pastebin.com/Wj1nXuTt Please let me know if you need any other information. Thanks, Shriyansh -- You

Re: Upper limits on indexes/shards in a cluster

Two nodes are not stable with regard to split brains. All I can guess is that two nodes have a small volume of network traffic and that you may have had network problems. Without exact diagnostic messages it's hard to understand why nodes disconnected. There are plenty of reasons. Networking is

Re: copy index

I admit there is something overcautious in the knapsack release to prevent overwriting existing data. I will add a fix that will allow writing into an empty index. https://github.com/jprante/elasticsearch-knapsack/issues/57 Jörg On Mon, Oct 20, 2014 at 6:47 PM, euneve...@gmail.com wrote: By

Re: Upper limits on indexes/shards in a cluster

example log line: [DEBUG][action.admin.indices.status] [Red Ronin] [*index*][1], node[t60FJtJ-Qk-dQNrxyg8faA], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@36239161] org.elasticsearch.transport.NodeDisconnectedException:

Best practise: Searching complex docs with ElasticSearch

I'm wanting to search a series of documents which have a nested object nature. For instance a Github issue. I'm needing to ideally use the ES search lite http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/search-lite.html syntax I have some top level data (assignee,

Re: JRE 1.7.0_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak?

Hi Gavin, You might be hit by the following Guava bug: https://github.com/elasticsearch/elasticsearch/issues/6268. It was fixed in Elasticsearch 1.1.3/1.2.1/1.3.0 On Mon, Oct 20, 2014 at 3:27 PM, Gavin Seng seng.ga...@gmail.com wrote: ### JRE 1.7.0_11 / ES 1.0.1 - GC not collecting old gen

Re: ES error while creating index on Solaris

Hi Clinton, Thanks for your reply. Can you please help in performing the below steps? I don't completely understand the concepts of ES yet. Abhinav -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop

Re: ES error while creating index on Solaris

Hi Jorg, Is the LZF compression issue related to the original Jackson issue? Abhinav -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to

Kibana - missing settings gear/cog

I'm running kibana v3.0.1. The Settings gear/cog is no longer displayed for us. I recently updated the indexes to look for from the default to [ logstash-31days-].MM.DD,[logstash-forever].MM.DD. Refreshing the browser, trying other browsers, etc. doesn't help. There are no errors and

logstash: how to include/modify facility/priority

Hi Folks I m trying to insert/modify via logstash priority and facility. Using debian wheezy + rsyslog + logstash 1.4.2 and elasticsearch 1.1.1. part of my rsyslog to undersand that I wanna: /etc/rsyslog.conf: . # auth auth.=emerg -/var/log/auth/auth_emerg.log auth.=alert

Re: Best practise: Searching complex docs with ElasticSearch

Hey Neil, Sounds interesting. For these questions, I think it's helpful to consider the interface you're building for the user. What's the fundamental thing being shown in a list of search results? Nested documents can be convenient, but generally I think the modeling for this kind of scenario

Re: Best practise: Searching complex docs with ElasticSearch

The parallel of a github issue is a good one. There are top level elements (title, body) etc, and comments nested under that. Comment requests I see are to find all issues with a certain string in a comment, ordered by recency, but the item we want to show in the results is a link to the issue.

Access Elasticsearch term information through ES-Hadoop API

Hi, Is it possible to get access to terms information that is indexed in ES and run some kind of map-reduce on them using ES-Hadoop? For example, let's say I have a CSV data set that was indexed into ES, with one particular field that was setup to be analyzed using a snowball analyzer. So, can

Re: word delimiter

I resolved my own issue. #!/bin/sh curl -XPUT 'http://localhost:9200/specialchars' -d '{ settings : { index : { number_of_shards : 1, number_of_replicas : 1 }, analysis : { filter : { special_character_splitter : {

How to do a rolling downgrade of servers

Hi, I have 3 ES servers that are m1.large. I need to change the machines types and reduce to 1 node. The indexing and the shards were left at there default values and I am using unicast. So if I want to go to one node can I just stop 2 ES severs? And on the last reaming node, that is

Re: Kibana - missing settings gear/cog

I should also mention, more importantly, I'm unable to browse ANYTHING. The only items displayed in Kibana are it's logo on the left and the 4 remaining icons on the right. Any help is greatly appreciated. On Monday, October 20, 2014 3:15:16 PM UTC-5, Clif Smith wrote: I'm running kibana

Re: JRE 1.7.0_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak? (reposted with better formatting)

Thanks Adrien, my cache is exactly 32GB so I'm cautiously optimistic ... will try it out and report back! From Adrien Grand: You might be hit by the following Guava bug: https://github.com/elasticsearch/elasticsearch/issues/6268. It was fixed in Elasticsearch 1.1.3/1.2.1/1.3.0 On Monday,

Children aggregation (1.4.0.Beta1) Round-Robin result

Dear ES group, we've been using ES in production for a while and test eagerly all new-coming features such as cardinality and others. We try data modeling with parent-child relations (ES version 1.4.0.Beta1, 8 nodes, EC2 r3.xlarge, ssd, lot ram etc.) With data model of: *Parent* { key: value

Re: JRE 1.7.0_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak?

Please post all updates here (has pictures and better formatting): https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/VxkosQuKzaA Thanks Adrien, I've cross-posted your reply in the other post. On Monday, October 20, 2014 3:57:56 PM UTC-4, Adrien Grand wrote: Hi Gavin, You

Re: JRE 1.7.0_11 / ES 1.0.1 - GC not collecting old gen / Memory Leak? (reposted with better formatting)

Actually now that I read the bug a little more carefully, I'm not so optimistic. * The cache here (https://github.com/elasticsearch/elasticsearch/issues/6268) is the filter cache and mine was only set at 8 gb. * Maybe fielddata is a guava cache ... but I did set it to 30% for a run with 96gb

Re: How to do a rolling downgrade of servers

This should work as described. Make sure that you have enough disk space to hold all your primary shards. David. Le 21 oct. 2014 à 02:14, David Montgomery davidmontgom...@gmail.com a écrit : Hi, I have 3 ES servers that are m1.large. I need to change the machines types and reduce to