Cross-DC clusters - specific dangers

I am aware that cross data-center clusters are not recommended, since they violate one of the core assumptions of ES, namely that all nodes are equal. But what *specifically* (apart from obvious problems associated with network failure) can this lead too: is it just high or irregular latency

Elasticsearch on hive join query not working!

I have some question to using elastic search on hive I use a hive to join elasticsearch datas , but result value is null!(but each table has join key value) here is my queries, 1. Create table CREATE EXTERNAL TABLE join_1 ( fileownermailguid STRING, filecontent STRING ) STORED BY

Kibana 3 with nginx ldap authentication - doesn't accept OPTIONS method?

I have 2 virtual hosts on the same nginx server, that both use the nginx-auth-ldap https://github.com/kvspb/nginx-auth-ldap module. the first virtual host is the Kibana that is authenticating with ldap using nginx-auth-ldap, then it forwards the request to the second virtual host which is a

Re: Cross-DC clusters - specific dangers

Split-Brain risk is not related to latency, it can happen on any network which is dynamic. The main issue is latency, yes. This is a killer. If latency is too high, real-time systems can be seen as unusable from a user perspective. Second issue is network bandwith. LAN traffic is a magnitude

Re: Help mapping in elasticsearch for searching content

Hi Doug, These above settings just apart of my searching. It's make my index searchable. And I want to optimize this system as I said. Thanks for your reply On Tuesday, April 14, 2015 at 9:17:08 AM UTC+7, Doug Turnbull wrote: Hello Jin I don't see here how you're searching, but there's

Trouble with Timestamp format 'dd-MM-yyyy HH:mm:ss

Hi, I have trouble with the timestamp format required for elasticsearch. I have data in my hadoop that I want to load into ES. My date field was first as this : EEE, dd MMM HH:mm:ss Z I've transformed that with talend like this :

Re: Trouble with Timestamp format 'dd-MM-yyyy HH:mm:ss

It's 'es.mapping.names' = 'sendingTime:@timestamp'); not 'es.mapping.names' = 'date:@timestamp'); On Tuesday, April 14, 2015 at 11:19:13 AM UTC+2, BEN SALEM Omar wrote: Hi, I have trouble with the timestamp format required for elasticsearch. I have data in my hadoop that I want to load

Re: which is the best way to create types in terms of performance in elasticsearch

Could anybody let me know the suggestion :) On Tuesday, April 14, 2015 at 9:49:20 AM UTC+5:30, binu wrote: i have a RDBMS tables having multiple columns and its hetrogenous and need to create an index in elasticsearch from these tables. So which is the best practise intems of creation of

Re: refresh_interval:10s is better than refresh_interval:-1?

Possibly it is IO bound but I don't seem too many io wait on Cpu or write activity on iostat.By the way,uses ssd and xfs as file system and default Directory ( I think it becomes MMapDirectory). each single bulk request to one index is done concurrently 5X so you only need enough concurrent bulk

Compression in Elasticsearch documents

I would like to know if Elasticsearch documents/indices are stored in compressed format on disk . If yes, what type of compression options are available and it's performance overheads. and if these compression options are configurable. Thanks Ajay -- You received this message because you are

Re: Kibana 4 - ability to see source data from Dashboard

A colleague just pointed out that you can add a search to the dashboard. Seems to work :) On Tuesday, 14 April 2015 14:57:43 UTC+1, Thomas Bratt wrote: Hi, I can't seem to get access to the original data by drilling down on the visualizations on the dashboard. Am I missing something?

Re: regex is not working

Hi Vineeth, Thank you for your reply. Actually not_anaylzed is worked fro me but I have problem. let's get back to my example : urlFiled.raw: /lang1/page1/ OR urlFiled.raw:/lang2/page1/ OR urlFiled.raw: /lang3/page1/ works. Now I'm trying to write regex to match the above urls

Re: ScrollId doesn't advance with 2 indexes on a read alias 1.4.4

Are you sure that calling the same scroll_id won't return the next results? AFAIK, the scroll_id can be the same and still return new records 2015-04-14 14:26 GMT-03:00 Todd Nine tn...@apigee.com: Hey guys, I have 2 indexes. I have a read alias on both of the indexes (A and B), and a

Kibana 4 - ability to select a date range on dashboard that is reflected in other visualizations

Hi, I am using Kibana 4 with a Date Histogram. I can select a time range with the mouse but the other visualizations on the dashboard do not seem to update. I only have data from today which might be affecting things. Would appreciate it if someone could tell me how to get this to work :)

Re: Compression in Elasticsearch documents

Hi, Data are both duplicated to suit different access patterns and compressed. There are so many compression algorithms in-place that it would be hard to be exhaustive, but we have for instance Frame-Of-Reference compression for postings lists, LZ4 for the document store, bit packing for numeric

Re: Compression in Elasticsearch documents

Hi Adrian Thanks for quick response. When I loaded nearly 45m documents of test data with 3 replicas (each document approx 2K+ bytes in size), I got following info on storage: *health status index pri rep docs.count docs.deleted store.size pri.store.size* *green open test_insert

Re: Inconsistent results when aggregate by field from array

Hi, You want to look at nested objects: http://www.elastic.co/guide/en/elasticsearch/guide/master/nested-objects.html -- Nils Tip: try formatting your post, it was hard to read. On Tuesday, March 31, 2015 at 4:37:40 AM UTC+2, Iana Bondarskaia wrote: Hi All, I have array of objects in each

ScrollId doesn't advance with 2 indexes on a read alias 1.4.4

Hey guys, I have 2 indexes. I have a read alias on both of the indexes (A and B), and a write alias on 1 (B). I then insert 10 documents to the write alias which inserts them into index B. I perform the following query. { from : 0, size : 1, post_filter : { bool : { must :

ELK for app logs

Hi Experts, i'm trying to see how i can use ELK for my application logs,MVC Web App running on windows. From my understanding of documentation, only way looks like have the MVC App write logs to local file and then use logstash to forward the logs to ES. is this the recommended approach? pls

how to create different name to the Kibana web site

Hi Guys, I have one requirement from the user . The kibana web site which I have provided to the user is like http://system IP :5601. User wants it like http://dashboard:5601;. Is it possible that I can change system IP to Dashboard. Please suggest if it is possible then how I can achieve

Re: Trouble with Timestamp format 'dd-MM-yyyy HH:mm:ss

Hi, I'm not sure if it will fix your problem but you could try the type TIMESTAMP for your sendingTime field. TIMESTAMP type is equivalent to the date elasticsearch type according to http://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html Le mardi 14 avril 2015 11:26:49 UTC+2,

Re: Trouble with Timestamp format 'dd-MM-yyyy HH:mm:ss

I have transformed my field with talend to be like this -MM-ddTHH:mm:ssZ and that went good ! Doesn't ES accept another format? other than the one above? On Tue, Apr 14, 2015 at 5:05 PM, jean.freg...@gmail.com wrote: Hi, I'm not sure if it will fix your problem but you could try the type

[hadoop] Reading severals index with Hive

Hi, I have got severals index in my ElasticSearch starting with log-access. I would like to read all of them thanks to Hive. I have tried 'es.resource' = 'log-access*/access' which doesn't seem to be correct. Then i have tried 'es.resource' = 'log-access{@timestamp:.MM.dd}/access' and

Please suggest.

Hi Experts, Need your valuable suggestion here . I have following setup and it is working fine. 1) 2-ES nodes (in Cluster handled by AWS load balancer) 2) 1-LS node (push data to any ES) 3) 1-Kibana Node (will get index from any of the ES ) 4) One load balancer for ES

Re: Scoring based on the number of matches in the field

Hi Doug, Yes. it does make sense. I'll try to rewrite it and get back to you. Thank you again for your help, Andre -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email

BM25 for query itself

Hi, I'm wondering is there away to calculate BM25 score for the query itself (query-against-query)? Adding it to index seems to be invalid solution as it will influent the tf-idf of the index and make it corrupted. Thank you, Bohdan -- You received this message because you are subscribed

Re: Stop words and re indexing

Hey Vineeth, Thank you Is there a way I can specify the stop words only during searches . For example the word AND I want to index but I want my searches consider this Regards Nagaraju 908 517 6981 On Tue, Apr 14, 2015 at 10:59 PM, vineeth mohan vm.vineethmo...@gmail.com wrote: Hello , I

Re: Stop words and re indexing

Hello , You can specify the analyzer in match query http://www.elastic.co/guide/en/elasticsearch/reference/1.3/query-dsl-match-query.html . Creating the analyzer you want and mentioning the stopwords in it and referring it from the match query might actually work. Thanks Vineeth

Re: Stop words and re indexing

Hello , I need to double check if we can change the analyzer defenition. But then even if you change it , you need to re index the whole index to make it working. Else your index would be looking bad with a set of documents analyzed differently with different set of stop words and another set

Re: Threadpool sizes

I do observe latency spikes, and massive cluster outages during bulk indexing. I have moved to giving only 1 thread each to bulk and index pools. It is now a 4 core machine so I guess I have 2 cores free for search. What are the other thread pools that can misbehave? Refresh, merge? On Tue, Apr

Re: Searches slow down significantly for several seconds every minute with transport client

Have you checked the logs for GC events or similar? What about the web logs for events coming in? On 15 April 2015 at 09:03, Daryl Robbins darylrobb...@gmail.com wrote: I am seeing a consistent bottleneck in requests (taking about 2+ seconds) at the same second every minute across all four of

Getting error after upgrading to elasticsearch 1.5

HI Friends, I have upgraded my elasticsearch from 1.0 to 1.5. However now I am getting below error in logs and the app wont start. Please help me on this. This is Urgent. Please refer below details: - at

Re: Please suggest.

And what do you want help on? On 15 April 2015 at 00:38, vikas gopal vikas.ha...@gmail.com wrote: Hi Experts, Need your valuable suggestion here . I have following setup and it is working fine. 1) 2-ES nodes (in Cluster handled by AWS load balancer) 2) 1-LS node (push data to

Re: Getting error after upgrading to elasticsearch 1.5

Sounds like you are mixing versions or have incompatible plugins? David Le 15 avr. 2015 à 07:17, Prasad Lele prasad.lel...@gmail.com a écrit : HI Friends, I have upgraded my elasticsearch from 1.0 to 1.5. However now I am getting below error in logs and the app wont start. Please

Re: Kibana 4 - ability to select a date range on dashboard that is reflected in other visualizations

The other visualisations should definitely update. I'd suggest you might need more data to show. On 15 April 2015 at 00:36, Thomas Bratt thomas.br...@gmail.com wrote: Hi, I am using Kibana 4 with a Date Histogram. I can select a time range with the mouse but the other visualizations on the

Re: Please suggest.

Sorry if I was not clear , I want 2 things 1) How I can make Kibana highly available , I mean I want same setup as ES has if one down other will take over so my confusion is if I install kibana on 2 servers and use one load balancer on top of these 2 instances how 2 kibana instances will be in

Re: Difficult to tame cluster

Yes, replicas only receive data (and searches), they do no indexing. On 14 April 2015 at 02:03, Debayan Banerjee debayan.baner...@paytm.com wrote: Hi, We are having a hard time trying to keep our 4 node cluster (ES 1.4.4) up with high indexing load. I have noticed that one of the nodes,

Re: Please suggest.

Kibana is just a javascript web application (it stores some stuff in a shard called .kibana in the ES cluster as the data) , so, to make it HA, you just have to run a nginx with reverse proxy with any number of kibana apps. just treat is a normal static website. On Wed, Apr 15, 2015 at 1:31 PM,

I did download the newest version Kibana and got following error as starting

Hello, Folks I did download the newest version Kibana and got following error as starting: Kibana: This version of Kibana requires Elasticsearch 1.4.4 or higher on all nodes. I found the following incompatible nodes in your cluster: Elasticsearch-rtf v1.4.0 …... Which vesion Kibana

Re: Incorrect Aggregations returned from ES

Hi, To me this sounds a lot like an issue that was happening to me a week before the release of 1.0.0. This issue was related to internal memory reuse within Elasticsearch before the result was read out. The issue is documented here: https://github.com/elastic/elasticsearch/issues/5021 What I

Kibana 4 - ability to see source data from Dashboard

Hi, I can't seem to get access to the original data by drilling down on the visualizations on the dashboard. Am I missing something? Many thanks, Thomas -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop

Re: Incorrect Aggregations returned from ES

Nils: It looks different from your issue since document counts are correct here? MC: I think it is due to https://github.com/elastic/elasticsearch/issues/8688. Has your `max_val` field been dynamically mapped? The only way to prevent this issue is to map fields explicitely instead of relying on

Re: Threadpool sizes

Unfortunately it would not be enough since there are also merging threads that take CPI and I/O. I would recommend to stick with the defaults and only try to change settings if you observe latency spikes at search time which are due to indexing. On Tue, Apr 14, 2015 at 3:57 AM, Debayan Banerjee

Re: export indices from prod cluster to beta cluster

Hi Colin, Thank you for the reply it got worked. Mapped two cluster to same snapshot path and restored indices to new cluster. Thanks phani On Monday, April 13, 2015 at 8:05:40 PM UTC+5:30, Colin Goodheart-Smithe wrote: Phani, For this you should use the snapshot/restore APIs. You

jdbc feeder

Hi All, when i used jdbc river some record metrics showed previously currently rivers are deprecated for 1.5 so i am using jdbc feeder when i execute it is pulling records but no metrics displaying on console.i am getting following warning. log4j:WARN No appenders could be found for

Stop words and re indexing

Hi , am new to elastic stop words and I want to understand more can we configure the stop words at time of index creation and later update the stop words with _settings API . Does it requires to re index documents to reflect the new list of stop words ? I want to know if there is a way to

Re: Querystring search: Tokens are out of order

Thanks, though unless I am misunderstanding it, the docs imply otherwise: For example, from: http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html The query string is parsed into a series of *terms* and *operators*. A term can be a single word — quick

Re: Querystring search: Tokens are out of order

To perhaps answer my own question, I think I understand the difference. details:foo bar Would search for the tokens in the same order (implied by the docs I referenced). But details:foo-bar Would not honor the order. The quotes have more meaning than to enclose the phrase... if that is

Re: ELK for app logs

You can do it this way, I have also seen logging via TCP to Logstash. Lots of people also use nxlog on Windows, which then sends to LS for further processing. I'd also suggest you have a look at the https://groups.google.com/forum/?hl=en-GB#!forum/logstash-users as there are a number of old

Re: Courier Fetch error, maybe due to lack of @timestamp?

This is probably due to Kibana trying to load an empty index and therefor missing 5 shards (ES creates 5 shards for an index by default). I stumbled upon the same issue as Logstash created empty indices in Elasticsearch under certain circumstances. That was exactly it! I found my empty index,

Elasticsearch trouble when client node is disconnected

Im using elasticsearch 1.4.0 I got a 9 node cluster setup with 4 data node 3 master node 2 clients I have an application which sends data concurrently to the cluster through client nodes. - Running high availability test One client node is disconnected and suddenly appear more

Re: refresh_interval:10s is better than refresh_interval:-1?

May I ask, when you seek for better indexing performance, what your current performance is? How many nodes ( = hardware machines) do you have? Jörg On Tue, Apr 14, 2015 at 1:36 PM, Hajime placeofnomemor...@gmail.com wrote: Possibly it is IO bound but I don't seem too many io wait on Cpu or

Function Score Query based on dictionary

Hi all, Is it possible to apply a script on sort without using a numeric field, but one based on a string? I want to to re-order the results based on a dictionary of values that exist in a field. I have never tried scripts before, but if it is possible to write a cascaded statement, then I

Re: true embedded mode

I have use cases involving many requests with many fields in response. seril/deserl consume a lot of resource. it would help a lot if there is a true embedded client that would directly interact with shard (single node, single shard) by passing/returning java object directly. On Monday,

Searches slow down significantly for several seconds every minute with transport client

I am seeing a consistent bottleneck in requests (taking about 2+ seconds) at the same second every minute across all four of my client nodes who are connecting using the transport client from Java. These nodes are completely independent aside from their reliance on the ElasticSearch cluster