Re: Where does ElasticSearch store an index by default?

2015-06-18 Thread Jason Wee
hey there, you should be able to find the location specified by path.data in elasticsearch.yml. read more here https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-dir-layout.html you should consider ask further question at https://discuss.elastic.co/ hth jason On Thu, Jun 18,

Re: Java API Query for _all field in match:

2015-06-04 Thread Jason Wee
query = filteredQuery(QueryBuilders.queryString(2000/sun).defaultOperator(and), null); response = client.prepareSearch(yemme).setTypes(ymmetype).setQuery(query).setSize(20).execute().actionGet(); in query string, you might want to escape the slash as it

Re: Good values JAVA options?

2015-06-01 Thread Jason Wee
it should be automatically determined , e.g. https://github.com/elastic/elasticsearch/blob/master/bin/elasticsearch.in.sh . are you fine tuning existing es node jvm setting? jason On Mon, Jun 1, 2015 at 5:52 PM, Alexandre in...@opendoc.net wrote: Hello everyone, I look for good values to use

Re: Whats the best way to add mappings programmatically ?

2015-05-28 Thread Jason Wee
mapping is not something you add/change every second or hour? just curious, why would you want to do it programmatically? the discussion have been moved to https://discuss.elastic.co/ , you might get more response there. jason On Tue, May 26, 2015 at 10:07 PM, mooky nick.minute...@gmail.com

Re: Are there any comprehensive documents which describe the detailed process of document indexing and searching?

2015-05-25 Thread Jason Wee
If you have that basic knowledge, perhaps the next steps would be read into the code, for instance, you mentioned about translog, so find for the translog class file and start from there? jason On Fri, May 22, 2015 at 10:47 PM, Shih-Peng Lin shihpeng@gmail.com wrote: Although I have read a

Re: Are there any comprehensive documents which describe the detailed process of document indexing and searching?

2015-05-25 Thread Jason Wee
-in-elasticsearch On May 25, 2015 5:45 AM, Jason Wee peich...@gmail.com wrote: If you have that basic knowledge, perhaps the next steps would be read into the code, for instance, you mentioned about translog, so find for the translog class file and start from there? jason On Fri, May 22, 2015 at 10

Re: How to get Elasticsearch boolean match working for multiple fields

2015-05-07 Thread Jason Wee
what es version is that? On Fri, May 8, 2015 at 9:07 AM, Dominic Nicholas dominic.s.nicho...@gmail.com wrote: Hi, I need some expert guidance on trying to get a bool match working. I'd like the query to only return a successful search result if *both* 'message' matches 'Failed password

Re: Scaling problem

2015-05-04 Thread Jason Wee
it for me ?? On Monday, May 4, 2015 at 12:35:10 PM UTC+6, Jason Wee wrote: If you are using java transport client in elasticsearch, you just add one of the elasticsearch node with setting sniff to on. The the transport client should have all the information it needed. When the transport

Re: Troubleshooting ES Resharding. Nature of immediate tasks and other questions

2015-05-04 Thread Jason Wee
hello, just out of curiosity, what is the cluster index size and what is the rationale to have so many shards? jason On Tue, May 5, 2015 at 5:05 AM, tomas.n...@despegar.com wrote: Hi all, We've been facing some trouble with our Elastic Search installation (v 1.5.1), mostly trying to bring

Re: enabling log compression

2015-05-04 Thread Jason Wee
there are example here in elasticsearch 1.5? https://github.com/elastic/elasticsearch/blob/v1.5.0/config/logging.yml#L42-L51 jason On Tue, May 5, 2015 at 8:13 AM, gleeco gle...@gmail.com wrote: i'm wondering how / if there is support for automatic compression of elasticsearch.log via

Re: term and string

2015-05-03 Thread Jason Wee
There are some terminology explain at this link. http://www.elastic.co/guide/en/elasticsearch/reference/0.90/glossary.html On Fri, Apr 24, 2015 at 10:09 AM, Jason Wee peich...@gmail.com wrote: Yeap, that help, thanks Doug! :-) On Thu, Apr 23, 2015 at 10:56 PM, Doug Turnbull dturnb

elasticsearch 0.90 backup/snapshot

2015-04-29 Thread Jason Wee
Hello, I was reading through how exactly to do backup/snapshot for elasticsearch 0.90.7. Unfortunately, from the documentation, it appear that to snapshot , elasticsearch that give that feature is 1.x and onward. Question, for elasticsearch 0.90, how do you do snapshot/backup, just in case

Re: elasticsearch 0.90 backup/snapshot

2015-04-29 Thread Jason Wee
* - Developer | Evangelist *elastic.co http://elastic.co* @dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs Le 29 avr. 2015 à 10:29, Jason Wee peich...@gmail.com a écrit : Hello, I was reading through how

Re: upgrade java for elasticsearch node

2015-04-28 Thread Jason Wee
by one.. No such thing as rolling upgrading. Jason On Wed, Apr 22, 2015 at 6:13 PM, Jason Wee peich...@gmail.com wrote: Thank Jörg, fully aware of Java 7 eol. Index will remain as is as in, after java upgraded to 7 for all nodes, client can query/index without any problem. If internally

Re: Bulk indexing creates a lot of disk read OPS

2015-04-24 Thread Jason Wee
merging graph you shared, looks normal to me. we had es with 10 shards too, and i monitor the segment using segmentspy, the segment graph in your attachment shown pretty same with ours. jason On Fri, Apr 24, 2015 at 4:45 PM, Eran era...@gmail.com wrote: Hey David, I suspect it indeed might

term and string

2015-04-23 Thread Jason Wee
Can anybody explain what is the different between term and string in elasticsearch context? When we index using default mapping (http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html), the default type is string. But when we query, we use the word term

Re: Issue when MatchPhasePrefix and Sort

2015-04-23 Thread Jason Wee
Hmmm why would a field that large? at 594mb? Jason On Fri, Apr 24, 2015 at 2:11 AM, TB txind...@gmail.com wrote: When executing a search with MatchPhasePrefix on a Propety which is a large string. The search fails with error. Data too large, data for [Field] would be larger than limit of

Re: Field Data Cache Size and Eviction

2015-04-23 Thread Jason Wee
A bit late from the OP posted this, not sure if it is still relevant but anyway... Under what circumstances will an ES node evict entries from it's field data cache? We're also deleting documents from the index, can this have an impact? What other things should I be looking it to find a

Re: term and string

2015-04-23 Thread Jason Wee
, etc based on the associated analyzer. Other types like numeric types have other low-level magic that helps convert them to the inverted index data structure. Hope that helps, -Doug On Thu, Apr 23, 2015 at 10:42 AM, Jason Wee peich...@gmail.com wrote: Can anybody explain what

upgrade java for elasticsearch node

2015-04-22 Thread Jason Wee
Hello, We are using java6 for our elasticsearch node of version 0.90 and googled, nothing discuss specifically on how to or procedure to upgrade java deployed in elasticsearch cluster. The one came close is this,

Re: upgrade java for elasticsearch node

2015-04-22 Thread Jason Wee
codes for hash maps are computed differently in Java 8) but this exposes only in rare cases. I am not sure what you mean by index will remain as is. In any case, I would always backup the data before upgrade. Jörg On Wed, Apr 22, 2015 at 11:51 AM, Jason Wee peich...@gmail.com wrote: Thanks

Re: upgrade java for elasticsearch node

2015-04-22 Thread Jason Wee
à 09:14, Jason Wee peich...@gmail.com a écrit : Hello, We are using java6 for our elasticsearch node of version 0.90 and googled, nothing discuss specifically on how to or procedure to upgrade java deployed in elasticsearch cluster. The one came close is this, https://svn.apache.org/repos

transport client threadpool and down node

2015-04-02 Thread Jason Wee
Hello, elasticsearch java transport client , for index, is the thread spawn bounded? if not, is there a way to set it to a fix value? When one of a node in cluster is down, java transport client will remove this downed node from its list, right? But when the downed node has joined back into the

Re: transport client threadpool and down node

2015-04-02 Thread Jason Wee
to true, then nodes are detected automatically http://www.elastic.co/guide/en/elasticsearch/client/java-api/current/client.html#transport-client Jörg On Thu, Apr 2, 2015 at 12:25 PM, Jason Wee peich...@gmail.com wrote: Hello, elasticsearch java transport client , for index, is the thread

Re: encountered monitor.jvm warning

2015-03-24 Thread Jason Wee
what is the new gen setting? how much is the system memory in total? how many cpu cores in the node? what query do you use? jason On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain huss...@novacom.mygbiz.com wrote: Hi all, we today got (for the first time) warning messages which seem to indicate a

Re: encountered monitor.jvm warning

2015-03-24 Thread Jason Wee
schrieb Jason Wee: what is the new gen setting? how much is the system memory in total? how many cpu cores in the node? what query do you use? jason On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain hus...@novacom.mygbiz.com wrote: Hi all, we today got (for the first time) warning messages

Re: Elasticsearch crashing after a while

2015-03-22 Thread Jason Wee
two heap settings? -Xmx200m -Xms200m -Xms256m -Xmx1g did you by any chance to reinstall jvm? jason On Mon, Mar 23, 2015 at 8:18 AM, Le Livre connie.cara...@gmail.com wrote: Sorry about that log in here. Here is my elasticsearch.yml file: http://pastebin.com/wAEHirvr I am running the newest

Re: Difference between index

2015-03-21 Thread Jason Wee
If it is not in the documentation or asked in mailing list, then you will have to read into the code. Jason On Sat, Mar 21, 2015 at 9:37 AM, Sagar Shah sagarshah1...@gmail.com wrote: Hello everyone, I am looking at stats of my index created for an application. Index has been created with 5

Re: elasticsearch heavy garbage collection

2015-02-26 Thread Jason Wee
when gc duration more than a second, client application will notice the cluster slow down, perhaps find out what are the hot thread? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html and took a heap dump in elasticsearch node

Re: function score filter function doesn't seem to be cached

2015-02-12 Thread Jason Wee
set explain to true and read the explain output if cache is use? Jason On Fri, Feb 13, 2015 at 3:47 AM, ffors...@homeaway.com wrote: I am using a function score with a function filter to score a subset of results higher. I specify to cache the function filter, but its performance is slower

manually trigger gc in elasticsearch node

2015-01-27 Thread Jason Wee
Hello, Is there a way to manually trigger garbage collector in elasticsearch node? Read that jmx connection has been removed from es since version 0.90. Jason -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and

Re: elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

2015-01-11 Thread Jason Wee
Hi Ed, soft restart as in client.close() ? I did that as well, does not seem to work, hence I resolve to tomcat restart instead... Jason On Sun, Jan 11, 2015 at 3:03 PM, Ed Kim edki...@gmail.com wrote: Other members can correct me if I'm wrong, but I notice that when you lose connection

Re: SearchContextMissingException Failed to execute fetch phase

2014-12-15 Thread Jason Wee
Anyone has any idea? On Wednesday, December 10, 2014 2:46:15 PM UTC+8, Jason Wee wrote: Hi elasticsearch, The following exception thrown in cluster 0.90.7 and I'm not sure what is this exception and why the id is not available? Based on the following stack trace, my guess

Re: Cluster went to red stage due to SearchContextMissingException

2014-12-11 Thread Jason Wee
Hmm... few days ago, I asked a similar question, https://groups.google.com/forum/#!topic/elasticsearch/1TlJDwuKXiA But we dont' get timedout of oom error.. Jason On Thu, Dec 11, 2014 at 10:44 PM, Bala krishnan balakrishnan.te...@gmail.com wrote: Hi, We are using elasticsearch 1.3.1

SearchContextMissingException Failed to execute fetch phase

2014-12-09 Thread Jason Wee
Hi elasticsearch, The following exception thrown in cluster 0.90.7 and I'm not sure what is this exception and why the id is not available? Based on the following stack trace, my guess is that the problem could be probably due to during execute fetch of a search, the id is not found in the

Re: Releasing elasticsearch connection

2014-11-30 Thread Jason Wee
Have you try using client.close() ? Jason On Sun, Nov 30, 2014 at 8:23 PM, prachi...@gmail.com wrote: Hi, I am deploying a webapp, which internally queries elasticsearch. I am creating only one connection, and I want to close it when the webapp shuts down. I am using singleton pattern for

Re: increasing number of open files descriptors in elastic search

2014-11-26 Thread Jason Wee
When the elasticsearch instance restarted, how do you check? Did you check using this command cat /proc/elasticsearch instance pid/limits ? Jason On Wed, Nov 26, 2014 at 6:09 PM, Vijay Tiwary vijaykr.tiw...@gmail.com wrote: I am increasing the number of open files descriptors for the root

Re: upgrading from 0.90.7 to 1.4. Gotchas?

2014-11-20 Thread Jason Wee
I would be interested too, we are using the same 0.90.7 version. Jason On Thu, Nov 20, 2014 at 2:03 PM, Yves Dorfsman y...@zioup.com wrote: Are there any precautions to take before upgrading from 0.9 to 1.4? Different data types? Different API calls? etc... And, what is the best way to

Re: ES bulk insert time out

2014-11-19 Thread Jason Wee
hmm.. java 6 allocated more than 8GB of heap... you saying you have 64GB for the es instance?! java 6, iirc still using cms as the gc, when it exceed 12GB, I started to see strange phenomenon. Also, what's your segment size? Check your log if you have the exception pool rejection something, if

Re: ElasticSearch server 1.4.0 throws ClosedChannelException and IOException when connected using elasticsearch-head

2014-11-17 Thread Jason Wee
could it be broken pipe like network break or timedout? the logger level is DEBUG, it should not be a concern right? On Tue, Nov 18, 2014 at 6:30 AM, andyrobin1 via elasticsearch elasticsearch@googlegroups.com wrote: Hi, I installed ElasticSearch server on CentOS and launched. I downloaded

Re: EsRejectedExecutionException: rejected execution (queue capacity 1000)

2014-11-02 Thread Jason Wee
unbounded queue for threadpool, isn't changes such as unbounded will exhaust system resources? It could either exhaust system resources due to each thread use specific amount of system resources, thus render the system not stable or if the system has file descriptor set, probably via ulimit,

Re: Elasticsearch queries are slow.

2014-10-30 Thread Jason Wee
Will adding node improve the query time assuming the rest remain the same? Jason On Thu, Oct 30, 2014 at 8:54 PM, Alberto Paro alberto.p...@gmail.com wrote: How many shards? if your shards are too small in number, their size it's too big. Typical shards bigger than 10gb gives you bad

Re: OutOfMemory

2014-10-23 Thread Jason Wee
It could tell more if you have the memory error. We are using version 0.90 as well and recently our traffic surge and our problem was due to the file descriptor reach, based on the error in the log, so we increase the file descriptor for the user, max number of processes per user. This does not

TransportClientNodesService RetryListener.onFailure stackoverflow error

2014-09-26 Thread Jason Wee
Hi Elasticsearch, https://github.com/elasticsearch/elasticsearch/blob/v0.90.7/src/main/java/org/elasticsearch/client/transport/TransportClientNodesService.java#L262 Will that line cause java.lang.StackOverflowError? If yes, is there a way to mitigate the problem? Thank you. Jason -- You

Re: index size load from what file?

2014-09-12 Thread Jason Wee
anyone? On Thursday, September 4, 2014 9:44:29 PM UTC+8, Jason Wee wrote: Hello ES, With curl showing the index statistics as below: $ curl 'http://localhost:9200/_cat/indices?v' health index pri rep docs.count docs.deleted store.size pri.store.size green twitter 1 0 0

index size load from what file?

2014-09-04 Thread Jason Wee
Hello ES, With curl showing the index statistics as below: $ curl 'http://localhost:9200/_cat/indices?v' health index pri rep docs.count docs.deleted store.size pri.store.size green twitter 1 0 00 123b 123b 123b is the index size? and where does

which class file trigger writing of segments.gen / segments_1

2014-08-29 Thread Jason Wee
Hello people, Anybody know which class/component in elastic search trigger writing of segments.gen and segments_1? I'm currently using elastic search version 1.2.1. It would be great if you can provide link pin point which line in the class does that. Thank you. /Jason -- You received this

Re: which class file trigger writing of segments.gen / segments_1

2014-08-29 Thread Jason Wee
indexing starts. Look at the SegmentsInfo class https://lucene.apache.org/core/4_9_0/core/org/apache/lucene/index/SegmentInfos.html Jörg On Fri, Aug 29, 2014 at 2:38 PM, Jason Wee peich...@gmail.com wrote: Hello people, Anybody know which class/component in elastic search trigger writing

Re: Elasticsearch on java7u55 ?

2014-04-18 Thread Jason Wee
will these two links help? https://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/SYSTEM_REQUIREMENTS.txt http://people.apache.org/~mikemccand/lucenebench/indexing.html lucene performance test is using java 1.70 u40. that's the same version i'm using for lucene 4.6.0. jason On Fri, Apr 18,

Re: How's the encoding handling power of ES?

2014-01-09 Thread Jason Wee
There is example in index and query in this SO http://stackoverflow.com/questions/8734888/how-to-search-for-utf-8-special-characters-in-elasticsearch hth Jason On Thu, Jan 9, 2014 at 5:13 PM, HongXuan Ji hxua...@gmail.com wrote: Hi all, I am wondering how the ElasticSearch deal with

Re: Exception cause unwrapping ran for 10 levels

2014-01-07 Thread Jason Wee
Jörg, Done, https://github.com/elasticsearch/elasticsearch/issues/4639 Today when I investigated this issue, and just do a query to the time stamp when the exceptions is happening, data were indexed though. The reason I query is that, we worry if there is no data index during that period

Re: Transport Client hangs in my web application during search.

2014-01-07 Thread Jason Wee
Does it show anything in the log? Perhaps try catch block on your code and set a query timeout. HTH /Jason On Wed, Jan 8, 2014 at 4:41 AM, Search User feedwo...@gmail.com wrote: I have a web application in which I create a Transport Client using Spring (singleton) and inject it into my

Re: Exception cause unwrapping ran for 10 levels

2014-01-06 Thread Jason Wee
Hi Jörg, Sure, my bad. There is no OOM and here is the second gist, https://gist.github.com/jasonwee/8294514 . The exceptions are repetitive, so that explains large number of lines in the log file. Does the second gist help to determine the problem? I can get the log if it does not tell where

Re: Bulk throughput issues

2013-12-23 Thread Jason Wee
Hi, this is interesting discussion, have a few extra questions if it is okay. // 7k per sec depend also on doc size. The larger the docs, the slower. Is there a way to quickly determine what is the size of a doc? // Note that by default, 50% heap memory of a node is dedicated for search, but 10%