Re: What configuration is available to control MemoryMapDirectory

There may not be much you can do here beyond giving the system more RAM and moving to SSD. How bad is the problem? On 13 March 2015 at 17:03, Lindsey Poole wrote: > Hey guys, > > We're running into some problems under heavy write, nominal read volume > when the Lucene memory mapped files have ex

Re: Bug in Blog

Thanks for pointing this out, I'll pass it onto our team! On 13 March 2015 at 20:06, Zennet Wheatcroft wrote: > There was a useful blog post about D3 and aggregations > > http://webcache.googleusercontent.com/search?q=cache:QhzvfcKiM70J:www.elasticsearch.org/blog/data-visualization-elasticsearch

Re: ElasticSearch Website down for quite some days?

I've let our team know about this, anymore details that you can provide would be appreciated :) On 13 March 2015 at 21:58, chenlin rao wrote: > There are some .js being blocked in China... > > 2015-03-14 6:46 GMT+08:00 Jay Swan : > >> Try http://elastic.co >> >> Elasticsearch (the company) chang

Re: ElasticSearch Website down for quite some days?

There are some .js being blocked in China... 2015-03-14 6:46 GMT+08:00 Jay Swan : > Try http://elastic.co > > Elasticsearch (the company) changed their name to Elastic this week. The > site redirects for me, but maybe there's something amiss in your browser or > proxy. > > On Friday, March 13, 20

Bug in Blog

There was a useful blog post about D3 and aggregations http://webcache.googleusercontent.com/search?q=cache:QhzvfcKiM70J:www.elasticsearch.org/blog/data-visualization-elasticsearch-aggregations/ Now I get this page which is mostly broken https://www.elastic.co/blog/data-visualization-elasticsearc

Re: How to enable Garbage Collector logging in an elastic search nodes..

Anil, you can try this: # /etc/sysconfig/elasticsearch export ES_USE_GC_LOGGING=1 ES_JAVA_OPTS="-verbose:gc -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xloggc:" Gavin On Wednesday, March 4, 2015 at 9:22:50 AM UTC-5, Anil Karaka wrote: > >

Re: Dealing with spam in this forum

Mark, thanks for checking. My posts did show up in the end ... I got an email from google about 18 hours later!! ... and all my re-postings all showed up (had to go delete all the duplicates). Gavin On Friday, March 13, 2015 at 5:32:40 PM UTC-4, aa...@definemg.com wrote: > > I saw a similar t

Re: Are long queues in management threadpool a problem?

Thanks Mark. I guess I'm not sure what's considered too high ... 300 does seem kind of high ... in my mind I'm thinking that due to this the Master nodes might not have enough time to process cluster events (such as mapping updates, alias creations). Gavin On Friday, March 13, 2015 at 3:08:0

Are long queues in management threadpool a problem?

Graph: http://imgur.com/FCpqeLR.png Has anyone experienced this issue? More importantly, is it a problem? We haven't seen anything bad ... but just in case this is an indicator. * We are running 3 dedicated masters (Machine:4GB JVM:2GB CPU:2) * Cluster * 3 ma

Re: How to confirm that a node has successfully joined the cluster

Cool, I'll check that out. Thanks. On Monday, March 9, 2015 at 11:55:59 AM UTC-7, Mark Walkom wrote: > > I'd use the _cat API and then just grep for the node name. > > On 9 March 2015 at 12:35, Lindsey Poole > > wrote: > >> Hey guys, >> >> Regarding the rolling restart instructions: >> http://w

What configuration is available to control MemoryMapDirectory

Hey guys, We're running into some problems under heavy write, nominal read volume when the Lucene memory mapped files have exhausted available physical memory, and segments from disk must be paged into memory. Are there any configs available to control how much physical memory is available to

Re: ElasticSearch Website down for quite some days?

Try http://elastic.co Elasticsearch (the company) changed their name to Elastic this week. The site redirects for me, but maybe there's something amiss in your browser or proxy. On Friday, March 13, 2015 at 4:16:36 PM UTC-6, liu wei wrote: > > Starting from beginning of this week, i have troubl

ElasticSearch Website down for quite some days?

Starting from beginning of this week, i have trouble access that. Anyone have similar issues? http://www.elasticsearch.org/ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send

Re: Dealing with spam in this forum

I saw a similar thing happen yesterday, and the post then appeared today. It was not a new thread in this case but instead a reply. I blame Google. On Friday, March 13, 2015 at 1:31:53 PM UTC-6, Mark Walkom wrote: > > I have had a look through the backend and cannot see these threads at all. >

Re: Analyzers and JSON

Well.. I think I may see your issue. I decoded this string: L2hvbWUvYWhhcm1vbi90ZXN0L0EgUGx1cyAtIE1lZGlhIFBsYW4gU3VtbWFyeS54bHM= It is: /home/aharmon/test/A Plus - Media Plan Summary.xls Another is: /home/aharmon/test/A Plus - Summary by Venue.pdf I think you misunderstand the purpose or how

Re: Analyzers and JSON

Hello, I'm running an instance of elasticsearch 1.3.2 on ubuntu server 14.04 on a imac. I have the mapper-attachments plugin installed and elasticsearch gui which I'm using for my front end. It's possible that I am missing something here are all the things I've tried so far: I got the mapper

Field names with the same name across types having different index/type in Elasticsearch

I have detailed my question on stackoverflow here: http://stackoverflow.com/questions/29041509/field-names-with-the-same-name-across-types-having-different-index-type-in-elast Briefing it here as well : I have been reading a lot on mappings in Elasticsearch and here's something interesting tha

How to enable storing of payloads without also storing term vectors

I need to store payloads along with each toke, but I don't want to store term vectors. Is this possible, and if so, how would I enable it? Thanks a lot! -devarajaswami -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this g

Embedded fields not searchable when addressed by name and type of same name exists?

I ran into the following today in Elasticsearch 1.4.4 and am trying to determine if this is a bug in Elasticsearch or a bug in my understanding of Elasticsearch. I'm more than willing to believe it is the latter. It should be very reproducible with the commands I've pasted. Let's say I have two

Re: mapper-attachment and base64 encoding

Definitely better to do that externally instead of moving a large BASE64 encoded document over the wire. So I won’t do it as you plan to do it. It will consume a lot of memory and bandwidth. We have some plans in the future to add this to logstash but it’s not here yet. Now, answering some othe

Re: Analyzers and JSON

Thanks. I missed the post. Will answer there. -- David Pilato - Developer | Evangelist Elasticsearch.com @dadoonet | @elasticsearchfr | @scrutmydocs > Le 13 mars 2015 à 12:41, Aaron Meffo

Re: Analyzers and JSON

He posted limited details in a separate thread. "mapper-attachment and base64 encoding" I was not asserting that it does not work, just that it may not be the best way to handle "large number of documents". I suspect there is an issue with encoding or submitting the document. On Fri, Mar 13,

Re: Analyzers and JSON

I’m a bit concerned about your « it does not work » statement. We have only today 4 opened issues on it: https://github.com/elastic/elasticsearch-mapper-attachments/issues 1 bug and 3 feature requests. Could you explain a bit m

Re: Dealing with spam in this forum

I have had a look through the backend and cannot see these threads at all. On 12 March 2015 at 13:44, Gavin Seng wrote: > > Hi, > > What is the current policy on this? > > I just tried creating 2 new posts ... they showed up for awhile ... and > then disappeared. I thought that it could be becau

Re: Very slow index speeds with dynamic mapping and large volume of documents with new fields

Mapping changes do need to go through the master, so check how it is performing. On 11 March 2015 at 08:37, Todd Nine wrote: > Hey all, > We're bumping up against a production problem I could use a hand with. > We're experiencing steadily decreasing index speeds. We have 12 c3.4xl > data node

Re: snapshot and zone

The core issue is latency. The second one is that if your link drops then chances are you will end up with a split brain, unless you have set min masters correctly, or a completely inaccessible cluster. On 13 March 2015 at 04:57, Foobar Geez wrote: > Thanks. I tried using a local mount point a

Re: Are long queues in management threadpool a problem?

As long as they don't stay too high then it's not a critical problem. On 12 March 2015 at 13:23, Gavin Seng wrote: > > > Hi, > > Has anyone experienced this issue? More importantly, is it a problem? We > haven't seen anything bad ... but just in case this is an i

Counting the frequency of a term

I am almost new to ElasticSearch, and need to do a simple task. I want to count the frequency of a word in a single document (not the number of documents containing a term). Assume that my index consists of a single document. Thanks -- You received this message because you are subscribed to

Re: How do i enable logging for elasticsearch java client

So i found out one way of enabling logs by hardcoding following in my application import org.elasticsearch.common.logging.ESLoggerFactory; ESLoggerFactory.getRootLogger().setLevel("TRACE"); Now i am able to see some log being generated. Sunil On Friday, March 13, 2015 at 7:27:49 AM UTC-7, Su

Re: Analyzers and JSON

Have you looked at the StandAloneRunner included with that plugin? I would practice with that, seeing first if it can extract the content, then seeing if it can extract the content from your base64 encoded version of the document. When that is working, I suspect you should at that point be able t

Re: Oops! SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]

The following as suggested was able to fix all my previous indexes to make 0 replication and essentially removing the 5 unassigned shards we had per Indice. curl -XPUT 'localhost:9200/*/_settings' -d ' { "index" : { "number_of_replicas" : 0 } } ' Thank you. -- You received this message bec

Re: Analyzers and JSON

There is a plugin called mapper attachments: https://github.com/elastic/elasticsearch-mapper-attachments This plugin is supposed to use Tika to index the content of documents but it doesn't seem to be working correctly. I base64 encode the documents but it comes back as null when I decode it.

Re: multi-core support for elasticsearch

I saw this kind of behavior with some older versions of Java, but I mean old, not something I would expect to run ES, but then again your on an old build of ES so I thought I would mention it. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" g

Geohash Filter in kibana and Custom layers in maps

Hello Everyone, I'm trying to find a way to implement a geohash filter in kibana ? Also how would one go about adding custom layers to the maps ? Would really appreciate any resources you could point me to. Thanks -- You received this message because you are subscribed to the Google Groups "

Re: best practice for rebuilding an index using aliases

Weird that was the post I made yesterday morning that just now hit the list after vanishing. On Thu, Mar 12, 2015 at 10:21 AM, wrote: > I switched to using aliases about a year ago and I love it. I am able to > rebuild in the background and make a clean cutover once the process > completes. > >

What is a reasonable number of evictions for filter_cache and fielddata?

Hi, graph: http://i.imgur.com/HjlGX4w.png * Wanted to share our cache evictions/size for filter_cache and fielddata and see what others are getting just for sanity checks. * The above is for one of our hot nodes. * Machine:128GB JVM:96GB * indices.fielddata.cache.size: 31GB * indices.cach

Are long queues in management threadpool a problem?

Hi, Has anyone experienced this issue? More importantly, is it a problem? We haven't seen anything bad ... but just in case this is an indicator. * We are running 3 dedicated masters (Machine:4GB JVM:2GB CPU:2) * Cluster * 3 masters * 4 data nodes * 2 tr

What is a reasonable number of evictions for filter_cache and fielddata?

Hi, * Wanted to share our cache evictions/size for filter_cache and fielddata and see what others are getting just for sanity checks. * The above is for one of our hot nodes. * Machine:128GB JVM:96GB * indices.fielddata.cache.size: 31GB * indices.cache.fi

correctly analyzed field not found by query_string search

I am having an issue searching a field which was analyzed with a pattern analyzer. I have verified that the analyzer works correctly and the index contains the correct tokens. However when I search using query_string, it does not return any results for the field. The field contains "pharmacy_do

What is a reasonable number of evictions for filter_cache and fielddata?

Hi, * Wanted to share our cache evictions/size for filter_cache and fielddata and see what others are getting just for sanity checks. * The above is for one of our hot nodes. * Machine:128GB JVM:96GB * indices.fielddata.cache.size: 31GB * indices.cache.fi

What is a reasonable number of evictions for filter_cache and fielddata?

Hi, Graphs: http://i.imgur.com/HjlGX4w.png * Wanted to share our cache evictions/size for filter_cache and fielddata and see what others are getting just for sanity checks. * The above is for one of our hot nodes. * Machine:128GB JVM:96GB * indices.fielddata.cache.size: 31GB * indices.cac

Using ELK to analyze log warnings and exceptions - and mark them as "solved"

We run several servers running our code. Of course there are bugs which cause exceptions and warnings since something unusual occurs. I want to analyze our logs to find unhandled warnings. I am unsure if ELK can help us. There need to be some way to aggregate warnings to a warning of type X (

Re: best practice for rebuilding an index using aliases

I switched to using aliases about a year ago and I love it. I am able to rebuild in the background and make a clean cutover once the process completes. Here are a couple of thoughts for your situation. First create a second index that has the same format as your original. When you are read

Search with atmost n terms match

Hi: Like the condition on match Query that sets a minimum terms that Should match, is there a way to specify Most terms that should match. for example : matchQuery(field,value).minShouldMatch(n).maxShouldMatch(m) I have a Signature property of a document that represents the top significant

Re: Snapshot Scaling Problems

Yes it was m1.smalls that I first noticed the EBS throttling on. Things work well in bursts, but sustained EBS does not work well. It will work substantially better in an m3.medium and if you are using the new EBS SSD volumes. On Thu, Mar 12, 2015 at 10:30 PM, Andy Nemzek wrote: > Thank you gu

Re: Kibana with Hadoop directly?

To second what Costin said, it is trivial to load the data into ElasticSearch, then you have the data ready to use the full power of Kibana. ElasticSearch is very quick to setup and scales well. Trying to go the route of rolling your own setup to generate graphs with D3 will certainly be harder,

Re: Analyzers and JSON

Not certain what you are referring to so I expect not. I have used the elasticsearch mappings, but I cant see how those would directly integrate with Tika. On Fri, Mar 13, 2015 at 10:35 AM, Austin Harmon wrote: > Thank you for the information. This going to be very difficult I can tell. > Do you

Re: Analyzers and JSON

On 13/03/2015 15:42, Austin Harmon wrote: Thank you for the information. I've been trying to use the mapper attachment which has Apache Tika built into it. I am just surprised and confused that so many companies use elasticsearch but yet it is so difficult to index the contents of a document. If

Re: Analyzers and JSON

Thank you for the information. This going to be very difficult I can tell. Do you have experience with the mapper attachment? On Friday, March 13, 2015 at 11:15:18 AM UTC-5, Aaron Mefford wrote: > > Your going to have the same issue with SOLR, putting the contents in to > XML which is even heavi

Re: "now throttling indexing"

On Fri, Mar 13, 2015 at 6:09 AM, Michael McCandless wrote: > That is the right setting to disable store throttling, but even without > throttling writes MB/sec for merges, the merges can still fall behind, > leading to index throttling. ES does this to protect the health of the index > because

Re: Analyzers and JSON

Your going to have the same issue with SOLR, putting the contents in to XML which is even heavier than JSON. I wish that I had some more experience using Tika, I do not. I am aware of its capabilities but have not had reason to myself. I see what you are saying about others not having the same i

mapper-attachment and base64 encoding

Hello, I am trying to index a large amount of data with the mapper attachment so I can get the contents of the documents. My plan of attack is to write a php script that encodes all my files to base64. Then input a doc to see if it works. My question relates to inputing these docs into elastics

Re: Analyzers and JSON

Thank you for the information. I've been trying to use the mapper attachment which has Apache Tika built into it. I am just surprised and confused that so many companies use elasticsearch but yet it is so difficult to index the contents of a document. If I need to index the contents of document

Re: Please help to understand these Exceptions

Fair enough. :) Thanks for all the replies! Chris On Thu, Mar 12, 2015 at 11:08 PM, Mark Walkom wrote: > The limit of a node is hard to definitively know as use cases vary so > much, but from what I have seen 3TB on 3 nodes is pretty dense. > > On 12 March 2015 at 08:09, Chris Neal wrote: > >>

Re: Hive to elasticsearch Parsing exception.

Ignoring both null values and "null strings" worked. Will open a issue about this. Thanks a lot Costin. On Fri, Mar 13, 2015 at 12:08 AM, Costin Leau wrote: > Likely the issue is caused by the fact that in your manual mapping, the > "NULL" value is not actually mapped to null but actually to a

How do i enable logging for elasticsearch java client

I built this standalone Java application that is connecting to elasticSearch using Transport client on my local machine. I am trying to figure out how to enable logging in it (This is to help me debug a wiered streamcorrupexception that i am getting in my production cluster) but not able to get

Re: Clear deleted docs

Hi Mike, As I have lots of updates only on recent docs, terms that have been recently indexed have their docFreq really impacted for a while. I'll keep thinking about a way to limit the impact on docFreqs and not worry too much about deleted docs. Thanks for your answer and your post, it made

Re: Should clause behaves like a must clause in filtered query

The behavior of "should" with no "must" or "must_not" clauses is the documented behavior, inherited from the underlying Lucene engine - it's not a bug. If you're desperate to return results which aren't likely to be respondent to the query (which is usually the case when you have only a single sea

Re: ElasticSearch across multiple data center architecture design options

Ok. Got it. Thank you! On Thu, Mar 12, 2015 at 8:56 PM, wrote: > Perhaps you are misunderstanding me. ElasticSearch does not provide a > load balancer for this purpose. You would use a typical HTTP load balancer > which could be anything as simple as Nginx, to something costly and > expensive

Re: multi-core support for elasticsearch

How do you observe there is "only one core"? Elastisearch uses many threads by default and as many cores as possible. Jörg On Fri, Mar 13, 2015 at 12:40 PM, Alexander Petrovsky wrote: > Hi! > > I have the same problem on my singe elasticsearch instance. > > # dpkg -l | grep elas > ii elastics

Re: "now throttling indexing"

That is the right setting to disable store throttling, but even without throttling writes MB/sec for merges, the merges can still fall behind, leading to index throttling. ES does this to protect the health of the index because too many segments will cause all sorts of trouble. What IO system is

Re: Strange exception in Elasticsearch 1.4.3

Probably a version mismatch as that analyzer seem to only support 1.3.8 -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Lucene.NET committer and PMC member On Fri, Mar 13, 2015 at 3:06 PM, Itamar Syn-Hershko wrote: > This

Re: Strange exception in Elasticsearch 1.4.3

This looks like a bug in elasticsearch-analysis-combo, I'd post it as an issue there -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Lucene.NET committer and PMC member On Fri, Mar 13, 2015 at 1:35 PM, Angel Cross wrote:

Re: Clear deleted docs

Note that only_expunge_deletes=true will only merge the segment away if it has > 10% delete docs by default, otherwise it leaves the segment as is. See http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-merge.html on how to change that 10% default. But it's almost always

Re: Clear deleted docs

Hi Vineeth, Thanks for your answer. So I understand that the only way to surely clear all deleted documents is to use max_num_documents=1 and indeed in my case, this takes too much time to be launched often. In fact my problem is that it seems df statistic is calculated among all docs, includ

Re: Clear deleted docs

Hi , Ideally the enitre set of deleted documents can would be only removed when you execute a full merge into 1 segment. But then this eats lots of time if you have a large index. $ curl -XPOST 'http://localhost:9200/twitter/_optimize?max_num_segments=1' On using only_expunge_deletes , you are

Re: snapshot and zone

Thanks. I tried using a local mount point and I have backup created on all 4 nodes because the primary shards are spread across all nodes. I am guessing that I need all 4 nodes online whenever I need to, for e.g., restore a snapshot. The latency between the DCs is <2ms, so for all practical r

Re: multi-core support for elasticsearch

Hi! I have the same problem on my singe elasticsearch instance. # dpkg -l | grep elas ii elasticsearch 1.4.2 all Open Source, Distributed, RESTful Search Engine # uname -a Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/L

Re: multi-core support for elasticsearch

Hi! I have the same problem on my singe elasticsearch instance. # dpkg -l | grep elas ii elasticsearch 1.4.2 all Open Source, Distributed, RESTful Search Engine # uname -a Linux s2322.serverel.net 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u

Strange exception in Elasticsearch 1.4.3

Hello. Recently in our test system we started to notice the following exception. Googling and investigation of setup itself didn't make it more clear. Still have no idea why is this happening. Maybe somebody already faced the issue and knows the reason? Or have any ideas? java.lang.IllegalArgu

Clear deleted docs

Hi ! I have many deleted documents in my index and I'd like to clear them so that df/ttf statistics aren't skewed by these deleted docs. I try the following command : curl -XPOST ' http://localhost:9200/myindex/_optimize?only_expunge_deletes=true' But the number of deleted documents is still the

Re: whitespace tokenizer not working as I'd expect

>From which source did you assume that %20 is a white space? The mapping char filter understands \u notation (which is not documented in ES). With curl, on bash, you have to escape the \u notation with double backslash like this ". => \\u0020" Here is a working example https://gist.github.

Re: Kibana with Hadoop directly?

Thanks Aaron, Costin for validating what I suspected. Agreed that ES is realt-time vs. Hadoop. I was looking at Kibana as a standalone data visualization tool for batch scenarios due to its superior visualization capabilities. As you said, perhaps D3 is the only option as of now. Thanks again f

Re: Is it possible to write my own filter ?

Yes, that's an idea : ) Pattern Tokenizer seems to give the results I want but I don't know if it's possible to define patterns based on a list of specific words. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group a

Re: Should clause behaves like a must clause in filtered query

I agree. But then the only way to use dynamically generated "should clause(s)" is to include a mandatory "must clause" with some identity function i.e. added overhead. I also tried "minimum_should_match" of 0, but it gave 0 documents in result. I was expecting every document in the index as re

Re: "now throttling indexing"

I have the same problem. On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote: > > I set `indices.store.throttle.type: none` in the elasticsearch.yml, and > yet this shows up in the logs: > > now throttling indexing: numMergesInFlight=5, maxNumMerges=4 > stop throttling indexing: nu

Re: "now throttling indexing"

Using ES 1.4.4 I should add. On Friday, March 13, 2015 at 9:00:26 AM UTC+1, Alexander Jiteg wrote: > > I have the same problem. > > On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote: >> >> I set `indices.store.throttle.type: none` in the elasticsearch.yml, and >> yet this shows up i