Re: Bad performance for terms facet on no hits result

2014-05-27 Thread Ivan Brusic
Is this a repeated test? There might be some cache loading going on during the first request. Values must be loaded into the cache before they can be filtered on. Try a repeated test. -- Ivan On Mon, May 26, 2014 at 11:12 PM, Hui dannyhui1...@gmail.com wrote: Hi All, My elasticsearch

Re: Elasticsearch Search Down - Server running

2014-05-27 Thread Ivan Brusic
Did you upgrade to 1.2? Dynamic scripts are now disabled by default. https://github.com/elasticsearch/elasticsearch/pull/5943 -- Ivan On Mon, May 26, 2014 at 11:16 PM, Pratik Poddar pratik.ph...@gmail.comwrote: My server was running fine until I start getting this error for search

Re: Where is SearchOperationThreading in ES 1.2

2014-05-27 Thread Ivan Brusic
Removed: https://github.com/elasticsearch/elasticsearch/pull/6042 -- Ivan On Tue, May 27, 2014 at 12:11 PM, InquiringMind brian.from...@gmail.comwrote: For all my previous ES versions (up to and including 1.1.1), my Java code passed the

Re: Filtering *before* a query

2014-05-27 Thread Ivan Brusic
I have never used the geo features, so I could be wrong, but I believe that geo filters are expensive are should be used as post filters: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-post-filter.html One of the reasons is that geo filters are not cached by

Re: Returning only unique results

2014-05-27 Thread Ivan Brusic
Perhaps the top hits aggregation coming in 1.3 could help: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-top-hits-aggregation.html -- Ivan On Tue, May 27, 2014 at 5:36 AM, ES USER es.user.2...@gmail.com wrote: This is similar to the

Re: Sequence Numbers for Replica Recovery

2014-05-27 Thread Ivan Brusic
My understanding is that Lucene now provides checksums, but that sequence numbers is a functionality built into Elasticsearch. I do not think that functionality has been released with 1.2, but I could be wrong. The item below it, zen discovery, is definitely not part of 1.2 -- Ivan On Tue,

Re: Filtering *before* a query

2014-05-28 Thread Ivan Brusic
27, 2014 4:32:08 PM UTC-5, Ivan Brusic wrote: I have never used the geo features, so I could be wrong, but I believe that geo filters are expensive are should be used as post filters: http://www.elasticsearch.org/guide/en/elasticsearch/reference/ current/search-request-post-filter.html One

Re: Filtering *before* a query

2014-05-28 Thread Ivan Brusic
the terms query so that I reduce the number of documents that I have to query over. On Wednesday, May 28, 2014 4:22:05 PM UTC-5, Ivan Brusic wrote: Why do you think that geo_bounding_box should be fast? Since the filter is not cached, it needs to run on every document in the index. -- Ivan

Re: Search issue with snowball stemmer

2014-05-29 Thread Ivan Brusic
You should use the Analyze API to ensure that the tokens you are producing are correct: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-analyze.html -- Ivan On Thu, May 29, 2014 at 7:13 AM, Александр Шаманов al3xsha...@gmail.comwrote: Hello everyone, I have

Re: Implicit Custom Filter?

2014-05-29 Thread Ivan Brusic
Two options come to mind: 1) Filtered aliases: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#filtered 2) Search template and Template queries: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-template.html

Re: Hide some system fields

2014-05-29 Thread Ivan Brusic
I do not think you can exclude the search metadata fields. -- Ivan On Thu, May 29, 2014 at 5:29 AM, Florentin Zorca goo...@zorca.de wrote: Hi, try using fields to specify only the fields you are interested in:

Re: Filtering *before* a query

2014-05-30 Thread Ivan Brusic
For some reason, when I viewed that page at work, I was not seeing the strategy section either, but I was able to at home. Try refreshing the page, worked for me. Perhaps I should play around with those settings the next time I fine tune my queries. I use a combination of both pre and post

Re: EsRejectedExecutionException[rejected execution (queue capacity 50)

2014-05-30 Thread Ivan Brusic
The queue sizes are defined in the threadpool section of the configuration. IIRC, they cannot be updated dynamically: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html -- Ivan On Fri, May 30, 2014 at 2:56 PM, srikanth ramineni ri.srika...@gmail.com

Re: ES 1.1.1 - Plugins _site not found

2014-05-31 Thread Ivan Brusic
I ran into the same issue when I upgraded to 1.x. The code in the plugin service will scan the plugins every minute. You can either disable the lookup or reduce the logging level (which is what I did). -- Ivan On May 31, 2014 7:23 AM, Dan Spinner dan.spin...@sonian.net wrote: Has anyone else

Re: recovery took long

2014-06-03 Thread Ivan Brusic
Which version of 0.90 were you using? Starting with 0.90.1, recovery is throttled to 20 mb per sec, which is very low IMHO. Is this setting set in your new cluster? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html -- Ivan On Tue, Jun 3, 2014 at

Re: Setting up alerts

2014-06-03 Thread Ivan Brusic
With logstash, you can use either the email or pagerduty outputs. You can setup a conditional for your specific log entries. http://logstash.net/docs/1.4.1/outputs/email http://logstash.net/docs/1.4.1/outputs/pagerduty -- Ivan On Sat, May 31, 2014 at 10:11 AM, Joshua Hall

Re: [ANN] Elasticsearch Simple Action Plugin

2014-06-04 Thread Ivan Brusic
Jörg, thanks for the plugin to help as a starting point for plugin development. Although I have built a few plugins during the years, they were river or analysis plugins, which are fairly easy. Writing a custom action required a lot more digging, especially since there are very few to learn from.

Re: elasticsearch QueryBuilder with dynamic value in term query

2014-06-04 Thread Ivan Brusic
Off the top of my head, you can either using a nested bool query for the IP address or use a terms query with the minimum match set to the size of the list. *Option 1:* QueryBuilder ipQuery = QueryBuilders.boolQuery(); for (String ip: ipList) { ipQuery.must(must(QueryBuilders.termQuery(address,

Re: Span first queries

2014-06-04 Thread Ivan Brusic
The limitation of only able to use term queries comes from Lucene. I never looked into why there is such a limitation in Lucene, but since they have a lot of smart people working on the code, I assume there must be a good reason. :) Phrase queries do not have such a limitation. I use span queries

Re: Identify word as dominant word in search

2014-06-04 Thread Ivan Brusic
I agree with Itamar. It sounds like you do have a list of colors and brands (tagging), so you can add a boost value as a payload to the relevant terms. You can use these payloads with a function score script or a custom similarity. Not an easy solution. If you can maintain a mapping of values in

Re: [ANN] Elasticsearch Simple Action Plugin

2014-06-04 Thread Ivan Brusic
Don't forget your slides. :) http://code972.com/blog/2014/05/72-the-ultimate-guide-for-elasticsearch-plugins-video-slides On Wed, Jun 4, 2014 at 2:30 PM, Itamar Syn-Hershko ita...@code972.com wrote: You should have released this before my talk last week, I could have mentioned it :\

Re: Elasticsearch/Lucene Delete space reuse? recovery?

2014-06-04 Thread Ivan Brusic
Lucene will hold onto deleted documents until a merged is performed. An update in Lucene is basically an atomic delete/insert. An optimize will help reclaim the space used by deleted documents. Did you change your merge settings? Deleted documents should eventually be removed whenever new

Re: Queries, filters and match_all

2014-06-04 Thread Ivan Brusic
There is no label, but the change was made last December: https://github.com/elasticsearch/elasticsearch/pull/4461 It appears that the REST API still supports the old notation, but the change did break Java backwards compatibility

Re: A plugin to change the result set before sending it back to the http client

2014-06-05 Thread Ivan Brusic
If you are only modifying the REST API calls and not the Java API, such a plugin should be easy. You are not creating a new type of action, merely using the current search one, but changing the output format. Here are two tutorials on simple REST plugins:

Re: A plugin to change the result set before sending it back to the http client

2014-06-05 Thread Ivan Brusic
There is no way to eliminate returning the search metadata. It has been requested often. -- Ivan On Thu, Jun 5, 2014 at 12:40 PM, Mario Mueller ma...@xenji.com wrote: Hey Joerg, I just need the whole content of the _source field like so: [ { HotelName: Plaka,

Re: A plugin to change the result set before sending it back to the http client

2014-06-05 Thread Ivan Brusic
at 9:49 PM, Ivan Brusic i...@brusic.com wrote: There is no way to eliminate returning the search metadata. It has been requested often. -- Ivan On Thu, Jun 5, 2014 at 12:40 PM, Mario Mueller ma...@xenji.com wrote: Hey Joerg, I just need the whole content of the _source field like so

Re: A plugin to change the result set before sending it back to the http client

2014-06-05 Thread Ivan Brusic
, Jun 5, 2014 at 10:28 PM, Ivan Brusic i...@brusic.com wrote: I just looked it up and it should be as easy as creating your own RestResponseListener that takes a SearchResponse and creates a simplified version with no metadata. Should be an interesting quick plugin, but it looks like Jorg is going

Re: A plugin to change the result set before sending it back to the http client

2014-06-06 Thread Ivan Brusic
Plugins are essential to ES's success and are not going away any time soon. The river plugins, aka cluster singletons, are the ones which are discouraged from use. Good ahead and create more plugins! -- Ivan On Fri, Jun 6, 2014 at 7:23 AM, Brian brian.from...@gmail.com wrote: I also have

Re: [ANN] Elasticsearch Simple Action Plugin

2014-06-06 Thread Ivan Brusic
Function score code is perhaps the easiest to write for Elasticsearch. You do not need to create a plugin for it, simply write the code with the appropriate classes and deploy at as a jar to Elasticsearch's lib directory. Done. That said, I still prefer to write native scripts as plugins because

Re: Impossible to implement real custom boost query when the weight is in the child document?

2014-06-06 Thread Ivan Brusic
Did you change the boost_mode of your function score script? The default should be multiply, which is the behavior you want, not sum, which is what you are experiencing. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html I have never used it

Re: Help with cluster.routing.allocation.enable Property

2014-06-06 Thread Ivan Brusic
The cluster state will be yellow while shards are being rebalanced, so you can just execute a cluster healt request and wait for green status: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html#request-params You can also poll the same API and wait until

Re: Queries, filters and match_all

2014-06-09 Thread Ivan Brusic
:03 UTC+7 пользователь Ivan Brusic написал: There is no label, but the change was made last December: https://github.com/elasticsearch/elasticsearch/pull/4461 It appears that the REST API still supports the old notation, but the change did break Java backwards compatibility https

Re: Queries, filters and match_all

2014-06-09 Thread Ivan Brusic
this to remove obsolete code from my library. понедельник, 9 июня 2014 г., 14:28:07 UTC+7 пользователь Ivan Brusic написал: Does it matter? From what I can tell, some code was added in 0.90 https://github.com/elasticsearch/elasticsearch/commit/ 6687ecb038b55416d4bb37d29746e86f2624f06b https://github.com

Re: Elasticsearch 1.2 Delete and Reinstall

2014-06-11 Thread Ivan Brusic
Regarding your problem, are you perhaps running into the fact that dynamic scripts are now disabled by default since 1.2? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_enabling_dynamic_scripting In terms of deleting the existing version, it all

Re: index template updating problem

2014-06-13 Thread Ivan Brusic
The index template will only be applied when a new index is created. -- Ivan On Thu, Jun 12, 2014 at 5:54 AM, sri 1.fr@gmail.com wrote: Hello all, If i update the mapping in an existing index template the change is not reflected automatically, i have to manually delete the old mapping

Re: exclude some documents (and category filter combination) for some queries

2014-06-13 Thread Ivan Brusic
Currently not possible. Elasticsearch will return all the nested documents as long as one of the nested documents satisfies the query. https://github.com/elasticsearch/elasticsearch/issues/3022 The issue is my personal #1 feature requested. Frustrating considering there has been a working

Re: elasticsearch curator — version 1.1.0 released

2014-06-14 Thread Ivan Brusic
. -- Ivan On Fri, Jun 13, 2014 at 8:54 PM, Mark Walkom ma...@campaignmonitor.com wrote: It has a prefix setting, but not a suffix. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 14 June 2014 13:35, Ivan Brusic i

Re: Boost field does not work..

2014-06-17 Thread Ivan Brusic
How do you know that the search is not working? Can you post an example query and perhaps an example explanation? If you are searching against the all field, you can set include_in_all to false for that field. You are better off not searching a field instead of trying to set a boost. Cheers,

Re: No handler found for uri when creating a mapping

2014-06-17 Thread Ivan Brusic
An index can be comprised of multiple types, so the type is not needed in the URL. Try simply 192.168.1.103:9200/nxtxnlogs Cheers, Ivan On Tue, Jun 17, 2014 at 1:14 AM, Abhishek Mukherjee 4271...@gmail.com wrote: Hi, I am following the ES Definitive guide. I am trying to create a mapping

Re: exclude some documents (and category filter combination) for some queries

2014-06-17 Thread Ivan Brusic
wrote: Hi Ivan Thanks for your reply. Yeah, I do understand that currently elasticsearch returns the whole nested doc. Can you help me how can i get the negative query with multiple categories working ? Thanks Srini On Fri, Jun 13, 2014 at 10:58 AM, Ivan Brusic i...@brusic.com wrote

Re: updating a document using Java API

2014-06-18 Thread Ivan Brusic
Are you storing the document source? If so, you can use the update API. If not, you would need to reindex from the original source. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html -- Ivan On Tue, Jun 17, 2014 at 11:23 AM, ESUser neerav...@gmail.com

Re: Get X word before and after search word

2014-06-19 Thread Ivan Brusic
Span queries are another option, but the main drawback is that they use non-analyzed term queries. -- Ivan On Thu, Jun 19, 2014 at 2:11 AM, Alexander Reelsen a...@spinscale.de wrote: Hey, you potentially could use the termvectors API for this, see

Re: guarding from double-start

2014-06-20 Thread Ivan Brusic
You can either use the startup scripts that come with the package when you install via apt/yum [1] or use the service wrapper [2]. [1] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html [2] https://github.com/elasticsearch/elasticsearch-servicewrapper

Re: deleting documents that are missing fields

2014-06-20 Thread Ivan Brusic
I do not use delete by query, but have you tried using a fully formed query and not just a filter? Perhaps an implicit match_all query is not being set. Try using a filtered query with a match_all query and your filter.

Re: issues with file input from logstash to elastic - please read

2014-06-21 Thread Ivan Brusic
The path shows an windows file name, so I am not sure if using tail would work. On cygwin, there is no -F option, at least on the version I use. On Linux, the file input works great, especially with rotated file. I am not a Logstash expert, but I use the file input with the sincedb option

Re: deleting documents that are missing fields

2014-06-21 Thread Ivan Brusic
Yes, that was what I meant by fully formed query. I leave nothing to chance. -- Ivan On Jun 21, 2014 8:04 AM, Jeff Dupont jeff.dup...@gmail.com wrote: I finally got it!! Thanks to this thread https://groups.google.com/forum/#!topic/elasticsearch/Eb5ERjHXp4Y. The syntax has changed where

Re: Issue concerning mapping and source

2014-06-23 Thread Ivan Brusic
What exactly is the issue? Are you getting back results, just with no data? By default, a query will only return the _source field. If you want to return other stored fields, then you would need to explicit name them:

Re: No node available

2014-06-23 Thread Ivan Brusic
Are you using the same version of the Elasticsearch jar with respect to the server? Which version are you using? Do you have any added jar dependencies to your project such as netty or Lucene? -- Ivan On Mon, Jun 23, 2014 at 7:13 AM, Aaliyah zhangqi1...@gmail.com wrote: The log file has this

Re: Splunk vs. Elastic search performance?

2014-06-23 Thread Ivan Brusic
I agree. I thought elasticsearch_http was actually the recommended route. Also, I have seen no reported issues with different client/server versions since 1.0. My current logstash setup (which is not production level, simply a dev logging tool) uses Elasticsearch 1.2.1 with Logstash 1.4.1 using

Re: Wait for yellow status

2014-06-23 Thread Ivan Brusic
checks that the cluster goes yellow after a data node leaves. We are using polling to validate this, but I would much rather have a blocking wait call. -T On Thursday, May 22, 2014 9:20:48 AM UTC-7, Ivan Brusic wrote: While doing some tests, I thought I uncovered a bug in the cluster

Secondary sort on aggregation buckets

2014-06-24 Thread Ivan Brusic
I started investigating switching from facets to aggregations in order to have access to some of the new features aggregations offer. One of them is the ability to sort on a sub-aggregation metric, which is working well, but the buckets that are tied come back in a random order. Since it is not

Re: Jespen article reaction

2014-06-24 Thread Ivan Brusic
Considering that most of the talking points come directly from the community, I do not think there will be much of a reaction here. The main issue referenced in the article has a few of the answers to your questions. Cheers, Ivan On Tue, Jun 24, 2014 at 9:04 AM, John Smith

Re: Searching by nested fields

2014-06-25 Thread Ivan Brusic
That feature is currently not supported by Elasticsearch. You would need to change to parent/child documents in order to get back only the relevant documents. -- Ivan On Tue, Jun 24, 2014 at 10:03 AM, Danylo Vivchar vivchar.dan...@gmail.com wrote: I have a trouble searching by nested field.

Re: Query on Id field of nested documents fails.

2014-06-27 Thread Ivan Brusic
It is not related to that issue. In that issue, your query would work, but all the nested documents are returned, not just the relevant. It seems like the query fails on fields named id. If you rename that field, the query works, so it has nothing to do with your mapping. I would report it as a

Re: What does open_source token means?

2014-06-29 Thread Ivan Brusic
open_source is just an example value in the tags array. The example has two documents, both with just a single field called tags. The first one has one value search and the second one has two values (both in the same field, therefore it is an array) search and open_source Document 1 { tags :

Re: Realtime search + fast indexing

2014-07-01 Thread Ivan Brusic
GET requests use both the Lucene index and the transaction log to retrieve documents. Search requests will use only Lucene since the inverted index is not updated until the transaction log is flushed. I haven't paid too much attention to the distributed aspects of the code in a while, but this

Re: Realtime search + fast indexing

2014-07-01 Thread Ivan Brusic
, Ivan Brusic i...@brusic.com wrote: GET requests use both the Lucene index and the transaction log to retrieve documents. Search requests will use only Lucene since the inverted index is not updated until the transaction log is flushed. I haven't paid too much attention to the distributed

Re: [ANN] Denormalization plugin for Elasticsearch analysis

2014-07-01 Thread Ivan Brusic
Extremely interesting. Should be ideal for a fixed set of terms. I am still hoping for Matt Weber's pull request to be merged ( https://github.com/elasticsearch/elasticsearch/pull/3278) since I would like to denormalize against a dynamic set of terms, but your approach would be more efficient if

Re: Custom Query variables ?

2014-07-02 Thread Ivan Brusic
If you enable explanations, you can see why Lucene the rational behind the scoring: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-explain.html You are probably correct in that the array length is influencing the scoring. By default, Lucene will rate higher

Re: Wrong Scoring using match query on Sense

2014-07-02 Thread Ivan Brusic
If you enable explanations, you would see that length normalization is scoring the document with the shorter field higher than the document with a term frequency of 2. The fieldNorm is incredibly lossy since it uses only 1 byte, so there must be some inconsistencies between the example and your

Re: Tutorial on Java interface to ElasticSearch?

2014-07-03 Thread Ivan Brusic
Currently the best way to learn the Java API is to view the Elasticsearch search code. All the REST examples use the Java API underneath, so you can see how the Java API is used by viewing a REST action such as RestSearchAction, RestIndexAction, etc...

Re: Problem with token delimiter and regular expression

2014-07-03 Thread Ivan Brusic
Did you apply your mapping with the new analyzer before indexing documents? First, you should verify that your mapping is correct by using the mapping API. Do not just look at your templates, use the API in case there is a problem in the templates:

Re: How to update a document using the Elasticsearch Update API when _source is disabled?

2014-07-03 Thread Ivan Brusic
Source is required to update documents. If not, you would need access to the original information and re-index. -- Ivan On Thu, Jul 3, 2014 at 2:06 AM, Veerapuram Varadhan v.varad...@gmail.com wrote: Hi, In my setup, ES is used for only indexing and searching and thus have disabled

Search thread pools not released

2014-07-06 Thread Ivan Brusic
Having an issue on one of my clusters running version 1.1.1 with 8 master/data nodes, unicast, connecting via the Java TransportClient. A few REST queries are executed via monitoring services. Currently there is almost no traffic on this cluster. The few queries that are currently running are

Re: Search thread pools not released

2014-07-06 Thread Ivan Brusic
this connection. -- Ivan On Sun, Jul 6, 2014 at 1:55 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Can be anything seen in a thread dump what looks like stray queries? Maybe some facet queries hanged while resources went low and never returned? Jörg On Sun, Jul 6, 2014 at 9:59 PM, Ivan

Re: Search thread pools not released

2014-07-07 Thread Ivan Brusic
On Sun, Jul 6, 2014 at 2:36 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Yes, socket appender blocks. Maybe the async appender of log4j can do better ... http://ricardozuasti.com/2009/asynchronous-logging-with-log4j/ Jörg On Sun, Jul 6, 2014 at 11:22 PM, Ivan Brusic i

Re: Search thread pools not released

2014-07-07 Thread Ivan Brusic
/AbstractQueuedSynchronizer.html#acquireSharedInterruptibly(int) in org.elasticsearch.common.util.concurrent.BaseFuture which waits forever until interrupted. But there are twin methods, like actionGet(long millis), that time out. Jörg On Mon, Jul 7, 2014 at 7:53 PM, Ivan Brusic i...@brusic.com wrote

Re: How to limit fields of response doc when I search certain keyword?

2014-07-07 Thread Ivan Brusic
If I understand you correctly, you want to view the distribution of gender based on the results of a query? In that case, you want to look into aggregations, which work on top of the result set that is returned.

Re: How to limit the fields of response when I search a keyword?

2014-07-07 Thread Ivan Brusic
I responded differently to your other similar question, but you can also limit the fields, but explicitly asking for the set that you want: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html Cheers, Ivan On Sat, Jul 5, 2014 at 2:32 AM, 纪路

Re: Best practice to backup index daily?

2014-07-07 Thread Ivan Brusic
The Elasticsearch curator now supports snapshots: https://github.com/elasticsearch/curator http://www.elasticsearch.org/blog/elasticsearch-curator-version-1-1-0-released/ You would still need to use cron to schedule tasks, but it would be a curator task instead of a direct curl request. Cheers,

Re: CLA sign required even for tiny typo fix?

2014-07-08 Thread Ivan Brusic
Elasticsearch as a company is relatively new, so I hope it will adjust its practices, not just the CLA, as time goes on. The codebase has been evolving so rapidly, that I would assume they are working on the code and the revenue stream and not its licensing model. -- Ivan On Tue, Jul 8, 2014

Re: How to index documents without location field ?

2014-07-08 Thread Ivan Brusic
In terms of the parsing exception, can you simply index the document with the field entirely? As far as sorting goes, it makes sense to push the location-less documents to the top or bottom. You lost me on the part regarding the rescorer. Do you need the location-less documents to be returned in

Re: CLA sign required even for tiny typo fix?

2014-07-08 Thread Ivan Brusic
for inspiration. Regards, Lukas On Tue, Jul 8, 2014 at 6:12 PM, Ivan Brusic i...@brusic.com wrote: Elasticsearch as a company is relatively new, so I hope it will adjust its practices, not just the CLA, as time goes on. The codebase has been evolving so rapidly, that I would assume

Re: New Errors when upgraded from V1.0 to V1.1.0

2014-07-08 Thread Ivan Brusic
How did you upgrade? Are you using repos or tarballs? It could be that you are missing the Lucene jar files or you have different versions of Lucene. Also, are you using the same version of Java across nodes? Java broke network serialization backward compatibility early in 1.7. Probably not the

Re: How to index documents without location field ?

2014-07-09 Thread Ivan Brusic
July 2014 23:29:49 UTC+5:30, Ivan Brusic wrote: I made a very important mistake in my first response. What I meant to say is Can you simply index the document WITHOUT the field entirely? What it appears that you are doing is indexing a field with no value. One solution would be to simply omit

Re: Aggregations and special characters

2014-07-09 Thread Ivan Brusic
Aggregations work on the tokens for the specified field. These tokens are generated when a tokenizer is applied to a field. In your case, you do not want the field to be tokenized at all, so you would either need to define is as not_analyzed or use a keyword tokenizer, which does not separate

Re: Simple setup question

2014-07-09 Thread Ivan Brusic
Which versions of logstash and elasticsearch are you using? There was some class renaming back in 1.0 and it caused a few ClassNotFoundException: org.elasticsearch.ElasticsearchException for several plugins. https://github.com/elasticsearch/elasticsearch/issues/4634 Other than that, I am not

Re: Elastic search dynamic number of replicas from Java API

2014-07-10 Thread Ivan Brusic
Jörg, have you actually implemented your own ClusterStateListener? I never had much success. Tried using that interface or even PublishClusterStateAction.NewClusterStateListener, but either I could not configure successfully the module (the former) or received no events (the latter). Implemented

Re: Elastic search dynamic number of replicas from Java API

2014-07-10 Thread Ivan Brusic
transport request/response cycle, they must poll for new events ... Jörg On Thu, Jul 10, 2014 at 6:38 PM, Ivan Brusic i...@brusic.com wrote: Jörg, have you actually implemented your own ClusterStateListener? I never had much success. Tried using that interface or even

Re: Elastic search dynamic number of replicas from Java API

2014-07-10 Thread Ivan Brusic
/elasticsearch-transport-websocket It would be possible to attach a server-side service with ClusterStateListener to such a websocket channel for subscribing to cluster state events. Maybe I find time to implement this for demonstration. Jörg On Thu, Jul 10, 2014 at 7:03 PM, Ivan Brusic i

Re: Elastic search dynamic number of replicas from Java API

2014-07-10 Thread Ivan Brusic
Sticking to your use case, you might want to use the auto_expand_replicas setting to all [1]: Never used it, but it sounds what you are looking for. By default, the ongoing recovery is not cancelled when the missing node rejoins the cluster. You can change the gateway settings [2] to control when

Re: Elastic search dynamic number of replicas from Java API

2014-07-10 Thread Ivan Brusic
:57 PM, Ivan Brusic i...@brusic.com wrote: Sticking to your use case, you might want to use the auto_expand_replicas setting to all [1]: Never used it, but it sounds what you are looking for. By default, the ongoing recovery is not cancelled when the missing node rejoins the cluster. You can

Re: How to add several name fields to an unmach definition in a mapping definition?

2014-07-11 Thread Ivan Brusic
Besides stop works, you can use a bool query one clause is the match all, and the other clause is must not with the terms in question. Something like: { query: { bool: { must: [ { match_all: {} } ], must_not: [

Re: Indexing files from filesystem

2014-07-11 Thread Ivan Brusic
Never used FSRiver, but from what I read, it should be exactly what you want. The code is open-sourced, so I would just check out the project, update the Elasticsearch version to 1.2.1 and find whatever bugs come up. Then submit a pull request and contribute back to the project. :) Cheers, Ivan

Re: Indexing files from filesystem

2014-07-11 Thread Ivan Brusic
Hopefully you accept pull requests faster than the core team. :) -- Ivan On Fri, Jul 11, 2014 at 11:55 AM, David Pilato da...@pilato.fr wrote: I love your plan Ivan! :-) -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 11 juil. 2014 à 20:36, Ivan Brusic i

Re: Can I use the java client of newer version to connect to a old version server?

2014-07-11 Thread Ivan Brusic
The code is suspicious since it has an explicit check for versions prior to 1.2 https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/admin/cluster/state/ClusterStateRequest.java#L121-L124 Don't know much else about the code to comment further.

Re: What do these metrics mean?

2014-07-11 Thread Ivan Brusic
Your second paragraph is correct. The threads are the total number of search threads at your disposal, active is the number of ongoing threads and queue are the number of threads that cannot be run since your thread pool is exhausted, which should be when active == threads, but not always the

Re: spam

2014-07-12 Thread Ivan Brusic
Read the recent comments regarding with the recent spam: https://groups.google.com/d/msg/elasticsearch/byATcjKgdYE/_Neoiof4fKIJ This new spam account has been banned. -- Ivan On Sat, Jul 12, 2014 at 2:50 PM, Warner Onstine warn...@gmail.com wrote: Could we please turn on first post filters

Re: Optimizing a query that matches a large number of documents

2014-07-14 Thread Ivan Brusic
Since the script is executed against lots of matched documents, perhaps converting it into a native Java script (not Javascript) would provide a performance boost. Note that using fields in scripts will force their values to be loaded into the cache. -- Ivan On Sun, Jul 13, 2014 at 8:54 AM,

Re: Elasticsearch 1.2 list of settings

2014-07-14 Thread Ivan Brusic
There are a few settings where the full named is not specified in the code, but is relative to the module it is in. Does your grep code account for these settings? A repo with pull requests might be too much for the maintainer, but a wiki would work well. Great job, Ivan On Mon, Jul 14, 2014

Re: How to change similarity settings runtime?

2014-07-14 Thread Ivan Brusic
Jörg is correct. In general, it would be a bad idea to change the similarity during runtime, but there are cases were it would be acceptable and the system should allow for those cases: https://github.com/elasticsearch/elasticsearch/issues/4403 -- Ivan On Mon, Jul 14, 2014 at 12:00 AM, Jörg

Re: Disabling _all-Field but keep Netflow-Events searchable

2014-07-14 Thread Ivan Brusic
This technically sounds like a Kibana question, so you might have better luck with the Logstash mailing list. Can't you simply prepend the field name in the query instead of relying on the default field? You can also change field names in Logstash. Another option is the copy-to-field. Similar to

Re: Upgrade 0.26.6 - 1.2.2 any catches?

2014-07-14 Thread Ivan Brusic
First of all, there is no version 0.26. I am assuming you meant 0.20.6. Either way, any upgrade from prior of 1.0 to 1.x will require a full cluster restart. 1. No clue 2. Many settings like omit_norms were deprecated, but are still support. I think that omit_tf has been changed. 3. I would

Re: Upgrade 0.26.6 - 1.2.2 any catches?

2014-07-15 Thread Ivan Brusic
Read more about it here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html#store-throttling Previously it was unbounded, but now the default is 20mb, which I found to be extremely low. Also, prior to 1.2, there was a Lucene bug which made throttling

Re: Direct buffer memory problem on master Discovery

2014-07-15 Thread Ivan Brusic
Direct memory is off heap memory. Are elasticsearch and logstash the only processes on those servers? Did you set an explicit direct memory value? -- Ivan On Jul 15, 2014 3:46 PM, Mark Walkom ma...@campaignmonitor.com wrote: How much data do you have in ES, index count and total size of all

Re: Direct buffer memory problem on master Discovery

2014-07-16 Thread Ivan Brusic
Most users do not set the direct memory setting. mlockall is set, but does the server allow it? You would see an error on startup if it didn't. Did you change the vm swapiness on the server? -- Ivan On Wed, Jul 16, 2014 at 2:40 AM, Pedro Jerónimo pedropregue...@gmail.com wrote: *Java: *java

Re: term query no hits was Re: No parser for element [term]

2014-07-16 Thread Ivan Brusic
By default, string fields are analyzed using the standard analyzer, which will tokenize and lowercase the input (I believe stop words are now NOT removed). A term query does not analyze the query, so it only works on non analyzed fields (or fields that use a keyword tokenizer). A term query for

Re: term query no hits was Re: No parser for element [term]

2014-07-16 Thread Ivan Brusic
what the problem is for that query. On Wed, Jul 16, 2014 at 10:27 AM, Ivan Brusic i...@brusic.com wrote: By default, string fields are analyzed using the standard analyzer, which will tokenize and lowercase the input (I believe stop words are now NOT removed). A term query does not analyze

Re: term query no hits was Re: No parser for element [term]

2014-07-16 Thread Ivan Brusic
behaving as if it has been analyzed. On Wed, Jul 16, 2014 at 11:00 AM, Ivan Brusic i...@brusic.com wrote: I would verify that the field is in fact non_analyzed and that your data is indexed in the way you think it is. Use the analyze API to analyze the term. Make sure you use the last

Re: term query no hits was Re: No parser for element [term]

2014-07-16 Thread Ivan Brusic
As predicted, your actual mapping does not match your perceived mapping. Something is not matching up. Perhaps the mapping is for a different index or type. Best way is to share your mapping and perhaps how you created your index as indicated at http://www.elasticsearch.org/help -- Ivan On

  1   2   3   4   5   6   >