Re: Kibana response time is too slow, need help identifying why
Well, my slow logs are 0 bytes. My logging.yml looks okay but I don't think they are configured. I looked at the ES docs and saw that I should have these set somewhere. I'm thinking elastic search.yml configuration file? #index.search.slowlog.threshold.query.warn: 10s #index.search.slowlog.threshold.query.info: 5s #index.search.slowlog.threshold.query.debug: 2s #index.search.slowlog.threshold.query.trace: 500ms #index.search.slowlog.threshold.fetch.warn: 1s #index.search.slowlog.threshold.fetch.info: 800ms #index.search.slowlog.threshold.fetch.debug: 500ms #index.search.slowlog.threshold.fetch.trace: 200ms And querying for hot threads never returns a response. I have marvel installed as well. Is there something else I can look at? Thanks, Tony -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1354b750-e44a-4138-864a-153449f99df4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
how to use my customer lucene analyzer(tokenizer)?
I want to use my own Chinese analyzer and I can write lucene analyzer class myself. How can I integrate it to elasticsearch? I googled and found http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-analyzers.html. But it only combine existing tokenizers and filters. I can use tokenizer writing in java by myself. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: how to use my customer lucene analyzer(tokenizer)?
I think you should write your own plugin with your analyzer implementation, so you can register your analyzer into the ES and use it later on. You could have a look and see how it done maybe in this plugin: https://github.com/elasticsearch/elasticsearch-analysis-kuromoji 05 август 2014, вторник, 09:58:16 UTC+3, fanc...@gmail.com написа: I want to use my own Chinese analyzer and I can write lucene analyzer class myself. How can I integrate it to elasticsearch? I googled and found http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-analyzers.html. But it only combine existing tokenizers and filters. I can use tokenizer writing in java by myself. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fe02dc31-991c-49f9-9820-63ba96a37d33%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: how to use my customer lucene analyzer(tokenizer)?
Hi, I think this plugin will be helpful for you. https://github.com/elasticsearch/elasticsearch-analysis-kuromoji 2014/08/05 15:58 fancye...@gmail.com: I want to use my own Chinese analyzer and I can write lucene analyzer class myself. How can I integrate it to elasticsearch? I googled and found http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-analyzers.html. But it only combine existing tokenizers and filters. I can use tokenizer writing in java by myself. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPW8A5xqnsWa76TJuSAWwxPyrUA8F9cJ6tyTs2ZJxXFpb4vEMg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Inexplicable wrong results in automated tests
I also had timing problems in my automated tests, the solution was to
invoke index refresh and wait for it, then execute any search requests.
This way you can make sure that all test data is ready for searching.
03 август 2014, неделя, 18:22:50 UTC+3, John D. Ament написа:
Hi
So after running a few rounds of local automated tests, I've noticed that
sometimes I get the wrong results in my index. This seems to only be an
issue with my automated tests and not when running the application manually
(at least I haven't seen the wrong results after several executions).
My search looks like this:
SearchResponse searchResponse =
esClient.client().prepareSearch(indexName).setTypes(RECORD_TYPE)
.setFetchSource(true)
.setPostFilter(FilterBuilders.andFilter(
FilterBuilders.inFilter(typeId,types.toArray(new
Integer[]{})).cache(false),
FilterBuilders.inFilter(stateId,states.toArray(new
Integer[]{})).cache(false)
).cache(false))
.addSort(dateCreated.value, SortOrder.DESC)
.addSort(recordId,SortOrder.DESC)
.execute().actionGet();
The issue appears both with and without the cache flag passed in.
The way my tests work is that I execute a bunch of seeds, then run queries
against the seeds to verify I get the right results. I'll create 5 records
in my test, where the typeId's are always 1,2,3,4 and the stateIds are
anything between 1 and 14, except for 6.
5 is a special state in my case. I only want to include that state
sometimes. So I'll run one query with all the states except 5 and 6. I
expect that this will give me 4 records back (the 5th record is in state
5). Instead I'm getting back 5 results, as if ES is also including state 5
in the list even though I didn't want it.
In my test I run this query twice. The test fails sometimes on the first
execution, never on the second execution (I have an arquillian deployment,
and start up the app once, then seed data, run the first query in one test
method, run the second query in a second test method). I'm assuming that
these filters are acting like a pure AND - the record must match both
fields to be returned. So, any idea why I might be getting the wrong
results?
John
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/16211527-5442-4e0e-ae64-62a0f30253d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Using facets/aggretagions on parent document, queried by TopChildren
Hi, The TopChildren query works with an estimated hit size, and the TotalHits might be incorrect if there are more child docs matching the required hits. How does that affect facets or aggregations defined on the parent document? Would their count might be likewise be incorrent? Or would they cause the TopChildren to actually expand to look for all possible children? Thank you. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f2a87846-64ab-4786-ad91-305b197152c1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Context suggest display results for all the categories (mimic the completion suggester behavior)
Thank you Alex, I also came up with that idea, but I think it is not that useful when you have context, mapped to a field, which usually contains a single value, especially numeric fields (despite that https://github.com/elasticsearch/elasticsearch/issues/6512 is not fixed yet) -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6ca0e8df-b21a-4f3a-afa5-0b57c91c03c1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
How to validate user input?
I try to use
Elasticsearch\Endpoints\Indices\ValidateQuery
and i get some abstract error.
here is part of my code
$parm['hosts'] = array('localhost:80');
$test = new Elasticsearch\Endpoints\Indices\ValidateQuery($parm);
$test1 = new Elasticsearch\Endpoints\Indices\Validate\Query($parm);
$test-setBody($json);
$test-performRequest();
can someone tell give me example of the code that need to be in parm or some
working example of the _validateapi in php. Thansk in front.
--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/How-to-validate-user-input-tp4061223.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1407165605261-4061223.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch 1.3 Transform Scripts
I have a problem i need to add dynamic fields according to certain fields value , i use groovy in simple case and it worked but the logic become more complicated and i need to separate it in external groovy script file and pass context as parameters but it didn't work and this must in mapping stage . -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-1-3-Transform-Scripts-tp4060592p4061206.html Sent from the ElasticSearch Users mailing list archive at Nabble.com. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1407153137385-4061206.post%40n3.nabble.com. For more options, visit https://groups.google.com/d/optout.
Re: Context suggest display results for all the categories (mimic the completion suggester behavior)
Hey, maybe the new transform feature could help here to create a default value (just an idea, havent tested it). See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-transform.html#mapping-transform --Alex On Tue, Aug 5, 2014 at 10:13 AM, Tihomir Lichev shot...@gmail.com wrote: Thank you Alex, I also came up with that idea, but I think it is not that useful when you have context, mapped to a field, which usually contains a single value, especially numeric fields (despite that https://github.com/elasticsearch/elasticsearch/issues/6512 is not fixed yet) -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6ca0e8df-b21a-4f3a-afa5-0b57c91c03c1%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/6ca0e8df-b21a-4f3a-afa5-0b57c91c03c1%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9b7Nw-PhVV8WoVq2prx0-egCR94er2SMm%2B3xRM0PLVYw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
TTL value must be 0. Illegal value provided
I recently recieved some exceptions in my log while updating documents
into my index.
org.elasticsearch.ElasticSearchIllegalArgumentException: TTL value must be
0. Illegal value provided [-17810]
at
org.elasticsearch.action.index.IndexRequest.ttl(IndexRequest.java:291)
at
org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:153)
at
org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:60)
at
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:183)
at
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:179)
at
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:63)
at
org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.run(TransportInstanceSingleOperationAction.java:192)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
I insert/update a lot of documents into Elasticsearch daily at 3:00 am by
using an UPSERT command.
All of these documents (offers) have a TTL of 30 days (set using a template)
client.prepareUpdate(indexName, documentTypeName, offer.internal_id)
.setDoc(offer)
.setUpsert(newOffer)
I don't understand why I receive these errors. The expired documents
*should already be removed* by elasticsearch right? Any help or pointers
are much appreciated.
Questions:
1) I insert a document with a TTL of 30 days. I update this document daily.
Will the TTL be reset to 30 days on every update or will it just decrease
no matter what I do with the document.?
2) I found a similar topic
https://groups.google.com/forum/#!msg/elasticsearch/ifvWZJjQuvU/cZzkhfmg6YoJ
with Github issue
https://github.com/elasticsearch/elasticsearch/issues/3256. This case
describes a race condition while updating a document right after the TTL
expired. In my case however documents are updated daily so expired document
should already be removed right?
3) is there a way (e.g. a query) to tell what documents are about to expire
/ or are expired and need to be removed by elasticsearch.
4) When query elasticsearch for document which have a TTL 0 , it returns
no documents. How come? This is the query I execute:
{
fields: [_ttl],
query: {
match_all: {}
},
filter: {
range: {
_ttl: {
lte: 1
}
}
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5c671844-94ba-4be1-a94c-d6053f01d04a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
How to rebalance shard
Now my cluster is not rebalance the data. How can I rebalance it please see the summary below. https://lh6.googleusercontent.com/-bH-VCgOCVWk/U-Ck6ykiJgI/SlI/S54duefOraA/s1600/Screen+Shot+2557-08-05+at+4.32.44+PM.png -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/06aab8c2-2dc8-4019-8a8c-219deca6e634%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: TTL value must be 0. Illegal value provided
forgot to mention I'm using version 0.90.5
Op dinsdag 5 augustus 2014 11:08:37 UTC+2 schreef Dennis de Boer:
I recently recieved some exceptions in my log while updating documents
into my index.
org.elasticsearch.ElasticSearchIllegalArgumentException: TTL value must be
0. Illegal value provided [-17810]
at
org.elasticsearch.action.index.IndexRequest.ttl(IndexRequest.java:291)
at
org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:153)
at
org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:60)
at
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:183)
at
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:179)
at
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:63)
at
org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.run(TransportInstanceSingleOperationAction.java:192)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
I insert/update a lot of documents into Elasticsearch daily at 3:00 am by
using an UPSERT command.
All of these documents (offers) have a TTL of 30 days (set using a
template)
client.prepareUpdate(indexName, documentTypeName, offer.internal_id)
.setDoc(offer)
.setUpsert(newOffer)
I don't understand why I receive these errors. The expired documents
*should already be removed* by elasticsearch right? Any help or pointers
are much appreciated.
Questions:
1) I insert a document with a TTL of 30 days. I update this document
daily. Will the TTL be reset to 30 days on every update or will it just
decrease no matter what I do with the document.?
2) I found a similar topic
https://groups.google.com/forum/#!msg/elasticsearch/ifvWZJjQuvU/cZzkhfmg6YoJ
with Github issue
https://github.com/elasticsearch/elasticsearch/issues/3256. This case
describes a race condition while updating a document right after the TTL
expired. In my case however documents are updated daily so expired document
should already be removed right?
3) is there a way (e.g. a query) to tell what documents are about to
expire / or are expired and need to be removed by elasticsearch.
4) When query elasticsearch for document which have a TTL 0 , it returns
no documents. How come? This is the query I execute:
{
fields: [_ttl],
query: {
match_all: {}
},
filter: {
range: {
_ttl: {
lte: 1
}
}
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ed71b65f-f77d-4b39-b9b3-4cab06d5be1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Using existing field for mapping
I am using Java API, but have tried doing this using,
1) putting a mapping file in /config/mapping etc.
2) Using curl
On Monday, August 4, 2014 1:35:00 PM UTC+5:30, Tihomir Lichev wrote:
What API are you using to communicate with ES ?
03 август 2014, неделя, 11:14:59 UTC+3, Ayush написа:
I am new to elastic search, I have created an index cmn with a type
mention. I am trying to import data from my existing solr to
elasticsearch, so I want to map an existing field to the _id field.
I have created the following file under /config/mappings/cmn/,
{
mappings: {
mentions:{
_id : {
path : docKey
}
}
}
}
But this doesn't seem to be working, every time I index a record the
following _id is created,
_index: cmn,
_type: mentions,
_id: k4E0dJr6Re2Z39HAIjYMmg,
_score: 1
Also, the mapping is not reflects. I have also tried the following option,
{
mappings: {
_id : {
path : docKey
}
}
}
SAMPLE DOCUMENT: Basically a tweet.
{
usrCreatedDate: 2012-01-24 21:34:47,
sex: U,
listedCnt: 2,
follCnt: 432,
state: Southampton,
classified: 0,
favCnt: 468,
timeZone: Casablanca,
twitterId: 47038,
lang: en,
stnostem: #ootd #ootw #fashion #styling #photography
#white #pink #playsuit #prada #sunny #spring http://t.co/YbPFrXlpuh;,
sourceId: tw,
timestamp: 2014-04-09T22:58:00.396Z,
sentiment: 0,
updatedOnGMTDate: 2014-04-09T22:56:57.000Z,
userLocation: Southampton,
age: 0,
priorityScore: 57.4700012207031,
statusCnt: 14612,
name: YazzyK,
profilePicUrl:
http://pbs.twimg.com/profile_images/453578494556270594/orsA0pKi_normal.jpeg
,
mentions: ,
sourceStripped: Instagram,
collectionName: STREAMING,
tags: 557/161/193/197,
msgid: 1397084280396.33,
_version_: 1464949081784713200,
url2: {\urls\:[{\url\:\http://t.co/YbPFrXlpuh\
,\expandedURL\:\http://instagram.com/p/mliZbgxVZm/\
,\displayURL\:\instagram.com/p/mliZbgxVZm/\
http://instagram.com/p/mliZbgxVZm/%5C,\start\:88,\end\:110}]},
links: http://t.co/YbPFrXlpuh;,
retweetedStatus: ,
twtScreenName: YazKader,
postId: 454030232501358592,
country: Bermuda,
message: #ootd #ootw #fashion #styling #photography
#white #pink #playsuit #prada #sunny #spring http://t.co/YbPFrXlpuh;,
source: a href=\http://instagram.com\;
rel=\nofollow\Instagram/a,
parentStatusId: -1,
bio: Live and breathe Fashion. Persian and proud-
Instagram: @Yazkader,
createdOnGMTDate: 2014-04-09T22:56:57.000Z,
searchText: #ootd #ootw #fashion #styling #photography
#white #pink #playsuit #prada #sunny #spring http://t.co/YbPFrXlpuh;,
isFavorited: False,
frenCnt: 214,
docKey: tw_454030232501358592
}
Also, how can we create unique mapping for each TYPE and not just the
index.
Thanks
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aaa67013-17d7-4c25-a697-65b7cbc4240d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
How to forbid the analyzing for a certain data type (e.g. string)
Hi all: we gathering data(in json format) from user input. I want to know is there any way to forbid the analyzing process for certain data type(e.g. string), so that if we detect the value of some field is in string format, we will not analyze it. thanks -- 不学习，不知道 -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BJstLBM_%3D6EEChk9vB0K1gtJpBAiXgTcFn7PC5bHsTDBTG%2Brw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Inexplicable wrong results in automated tests
Hmm. So how do I invoke a reindex via the java API?
On Tue, Aug 5, 2014 at 3:40 AM, Tihomir Lichev shot...@gmail.com wrote:
I also had timing problems in my automated tests, the solution was to
invoke index refresh and wait for it, then execute any search requests.
This way you can make sure that all test data is ready for searching.
03 август 2014, неделя, 18:22:50 UTC+3, John D. Ament написа:
Hi
So after running a few rounds of local automated tests, I've noticed that
sometimes I get the wrong results in my index. This seems to only be an
issue with my automated tests and not when running the application manually
(at least I haven't seen the wrong results after several executions).
My search looks like this:
SearchResponse searchResponse = esClient.client().
prepareSearch(indexName).setTypes(RECORD_TYPE)
.setFetchSource(true)
.setPostFilter(FilterBuilders.andFilter(
FilterBuilders.inFilter(typeId,types.toArray(new
Integer[]{})).cache(false),
FilterBuilders.inFilter(stateId,states.toArray(new
Integer[]{})).cache(false)
).cache(false))
.addSort(dateCreated.value, SortOrder.DESC)
.addSort(recordId,SortOrder.DESC)
.execute().actionGet();
The issue appears both with and without the cache flag passed in.
The way my tests work is that I execute a bunch of seeds, then run
queries against the seeds to verify I get the right results. I'll create 5
records in my test, where the typeId's are always 1,2,3,4 and the stateIds
are anything between 1 and 14, except for 6.
5 is a special state in my case. I only want to include that state
sometimes. So I'll run one query with all the states except 5 and 6. I
expect that this will give me 4 records back (the 5th record is in state
5). Instead I'm getting back 5 results, as if ES is also including state 5
in the list even though I didn't want it.
In my test I run this query twice. The test fails sometimes on the first
execution, never on the second execution (I have an arquillian deployment,
and start up the app once, then seed data, run the first query in one test
method, run the second query in a second test method). I'm assuming that
these filters are acting like a pure AND - the record must match both
fields to be returned. So, any idea why I might be getting the wrong
results?
John
--
You received this message because you are subscribed to a topic in the
Google Groups elasticsearch group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/uy0YrsXXw44/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/16211527-5442-4e0e-ae64-62a0f30253d0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/16211527-5442-4e0e-ae64-62a0f30253d0%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAOqetn8Yg3--qYSCvy60B_iqro7abU_cAEG%2BC7-HuzmDpogOfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: wrong boosting in ES?
After using MatchQuery and also trying TermQuery and FunctionScoreQuery I can say that Elasticsearch always produces NumericRangeQuery for numeric fields with boosting (or even for all numeric queries ???). fieldName = oa , value = 1 , boost = 400 E.g. oa:1 -- oa:[1 to 1] As for MatchQuery and TermQuery the boosting is doubled. E.g. oa:1^400 -- ConstantScore(oa:[1 TO 1]^400.0)^400.0 The FunctionScoreQuery also produces a NumericRangeQuery but can handle the boost. E.g. oa:1^400 -- ConstantScore(oa:[1 TO 1]) with value : 400.0, static boost factor By the way, this is how a working boosted numeric query should be (like in Solr): oa:1^400 -- (MATCH) weight(oa:`\b\u\u\u\u0001^400.0 in ...) Regards Bernd Am Dienstag, 5. August 2014 09:19:18 UTC+2 schrieb Bernd Fehling: Currently I'm playing with Solr 4.6.1 and ES 1.0.2 which both use Lucene 4.6.1. The field content of oa has very low cardinality, actually only one of the values 0,1 or 2. Also, in Solr I have omitNorms=true because I don't want any index-time boost or anything else, and the precisionStep is zero. Belief me, it works like a charm for years now with Solr and all is 100 percent compliant to Lucene, the problem is Elasticsearch. I just wanted to implement boost-query to my ES interface as it is in Solr for years. For example the boost should be if oa=1. I don't know why I should deal with huge function score query if I just want an extra boost during the query (selectable by the user). It seams like ES is not 100 Percent Lucene conform because it is not using omitNorms=true on numeric fields :-( The issue you mentioned is years ago and also fixed. Nevertheless the boosting problem of ES is somewhere in the QueryParsers which transforms the result of QueryBuilders to a Lucene query. Am Montag, 4. August 2014 18:57:11 UTC+2 schrieb Jörg Prante: Because integer fields have no norms, it is quite uncommon to use them for boosting. More common is the use for interpreting integer values as input for scoring algorithm with function score. Which Solr version is this? Solr did not follow the Lucene default in previous versions regarding integer boosting https://issues.apache.org/jira/browse/SOLR-3140 Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c1ade4e7-2499-4a05-9c0b-0b7eba1400a2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Using existing field for mapping
1. Try go execute GET http://yourhost:9200/_mapping and check if the mapping mentions is properly created with correct _id path 2. When indexing the documents you can construct your index request this way: IndexRequestBuilder indexRequestBuilder = getClient().prepareIndex(index, type).setSource(data); where data is your Map or JSON you want to index I tested with curl (-XPUT for mapping and -XPOST for indexing ! ) and seems like it works :) I'm also using _id path and Java API in my own project and have no problems with it so far -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d8e367d3-e6f7-4b54-9f3a-30154eb66a69%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Inexplicable wrong results in automated tests
Sort of: TransportClient transportClient = *new* TransportClient(settings); // just an example of transport client instantiation, you can use your own transportClient.getClient().admin().indices().refresh(*Requests* .refreshRequest(indexName)).actionGet(); // actionGet() as you may know it to make the call synchronous and wait for it until done (otherwise it is async) -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/348578e9-4fd1-4209-8995-1637c963fe70%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
strange highlight result, can anyone explain it?
Hi all,
I am using highlight function in ES 1.0.1. I found a very strange situation
as follow:
I want to highlight the field group.* which is the inside a
dictionary, whose mapping is
{properties: {
group:
{ type: object,
dynamic: false,
include_in_all: true,
properties: {
data: {type: string, index: analyzed, analyzer:
name_analyzer, term_vector: with_positions_offsets,
fielddata:{format: disabled}},
data_2: {type: string, index: analyzed, analyzer:
nickname_analyzer, term_vector: with_positions_offsets,
fielddata:{format: disabled}},
(skip)
}
}
}
}
The analyzers of each field inside group are all different.
I query the word Adobe and I got the following result:
highlight: {
group.data: [
I want it all, and I want it now 106\n\nUsing the
emAdobe/em ActionScript 3 SDK for Facebook platform 106\nTime,
– obtaining data in pages 126\nTime for action –
emadding/em limit and offset to GraphRequest instances 128,
requesting data based on date 131\nTime for
action – emadding/em since and until to GraphRequest instances
133\nTime
]
}
by using the highlight command:
{'highlight': {'fields': {'group.*': {'fragment_size': 100,
'number_of_fragments':
3
As you saw, it highlight the adding word. I cannot understand why this
comes.
And I am pretty sure the analyzer of group.data field cannot normalize
Adobe and adding into same form.
Any explanations? Please help me to understand what happened.
Thanks.
Ivan
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a047e9e5-410a-471d-9b10-cd03b0669197%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[ANN] Elasticsearch AWS cloud plugin 2.3.0 released
Heya, We are pleased to announce the release of the Elasticsearch AWS cloud plugin, version 2.3.0. The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.. https://github.com/elasticsearch/elasticsearch-cloud-aws/ Release Notes - elasticsearch-cloud-aws - Version 2.3.0 Fix: * [86] - Wrong exception thrown when snapshot doesn't exist (https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/86) Update: * [109] - Switch to https communication for Amazon APIs by default (https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/109) * [107] - Tests: update to Lucene 4.9.0 (https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/107) * [89] - Update to elasticsearch 1.3.0 (https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/89) New: * [101] - Allow https communication per ec2 or s3 service (https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/101) Doc: * [106] - Simplify documentation (https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/106) Issues, Pull requests, Feature requests are warmly welcome on elasticsearch-cloud-aws project repository: https://github.com/elasticsearch/elasticsearch-cloud-aws/ For questions or comments around this plugin, feel free to use elasticsearch mailing list: https://groups.google.com/forum/#!forum/elasticsearch Enjoy, -The Elasticsearch team -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53e0c74e.cf4cb40a.7735.0d7dSMTPIN_ADDED_MISSING%40gmr-mx.google.com. For more options, visit https://groups.google.com/d/optout.
Re: strange highlight result, can anyone explain it?
The query command I used is as
{'multi_match': {'fields': ['_all', 'name', 'group.*'],
'operator': 'and',
'query': 'Adobe'}}
I doubt the problem might be because the various analyzers of the
group.*.
What is the analyzer to be used during the highlight?
Ivan
Ivan Ji於 2014年8月5日星期二UTC+8下午7時58分42秒寫道：
Hi all,
I am using highlight function in ES 1.0.1. I found a very strange
situation as follow:
I want to highlight the field group.* which is the inside a
dictionary, whose mapping is
{properties: {
group:
{ type: object,
dynamic: false,
include_in_all: true,
properties: {
data: {type: string, index: analyzed, analyzer:
name_analyzer, term_vector: with_positions_offsets,
fielddata:{format: disabled}},
data_2: {type: string, index: analyzed, analyzer:
nickname_analyzer, term_vector: with_positions_offsets,
fielddata:{format: disabled}},
(skip)
}
}
}
}
The analyzers of each field inside group are all different.
I query the word Adobe and I got the following result:
highlight: {
group.data: [
I want it all, and I want it now 106\n\nUsing
the emAdobe/em ActionScript 3 SDK for Facebook platform 106\nTime,
– obtaining data in pages 126\nTime for action –
emadding/em limit and offset to GraphRequest instances 128,
requesting data based on date 131\nTime for
action – emadding/em since and until to GraphRequest instances
133\nTime
]
}
by using the highlight command:
{'highlight': {'fields': {'group.*': {'fragment_size': 100,
'number_of_fragments':
3
As you saw, it highlight the adding word. I cannot understand why this
comes.
And I am pretty sure the analyzer of group.data field cannot normalize
Adobe and adding into same form.
Any explanations? Please help me to understand what happened.
Thanks.
Ivan
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/60fd5110-56bc-46f0-b767-e8cfc70bb7e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: How to forbid the analyzing for a certain data type (e.g. string)
Probably you should use dynamic templates for mappings: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-root-object-type.html#_dynamic_templates 05 август 2014, вторник, 14:08:29 UTC+3, 潘飞 написа: Hi all: we gathering data(in json format) from user input. I want to know is there any way to forbid the analyzing process for certain data type(e.g. string), so that if we detect the value of some field is in string format, we will not analyze it. thanks -- 不学习，不知道 -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f7b3f7b2-e9e4-42e6-812e-0edd33c7478f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Multiple master problem in elasticsearch 0.90.10
Hi All, I am using 2 master node in my elasticsearch cluster and 2 data node. I ma using 2 RHEL server ( 1 master + 1 data node on each ) When i setup the above it is working fine for some time around 15 - 20 hours. but after then my cluster health is yellow. and the no available shards are reduce to half. when i check the elasticsearch data and master node , all 4 are running fine. Please help me in resolving the above issue. The requirement was to implement elasticsearch cluster on two server, for high availability ( 2 master on different server and 2 node on different server ). Please let me know if someone implemented high availability different from the above Thanks Ankit Mittla -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02683f10-8b05-45b0-a608-f35029bf4e1e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
rss river for 1.3
Hi David, a probably easy to answer question: is the current stable release (rssriver-1.1.0) supposed to work with elasticsearch 1.3.1? Will there be a new release any time soon? Or a logstash input? ;) Thanks! Andrej https://github.com/dadoonet/rssriver/releases/tag/rssriver-1.1.0 -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5a0dac1b-83e2-438e-9407-d24f00978c2e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
java.lang.ClassNotFoundException: org.elasticsearch.transport.RemoteTransportException
Using elasticsearch embedded in tomcat 7 where we have custom classloader
that shares spring application beans with many webapps. The API's to these
implementations are in shared but the implementations are in a separate
classloader ComponentLoader. Our search implementation is loaded from
ComponentLoader where elasticsearch has been promoted as our default search
engine.
Everything works fine on a single node but when in a cluster seeing the
following ClassNotFoundException issue when shards are trying to update on
other nodes.
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.transport.RemoteTransportException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
I thought this might have been related
to https://github.com/elasticsearch/elasticsearch/issues/4634 but that
doesn't seem to be the case.
My next idea is to look at why
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass is
invoking WebappClassLoader instead of our custom ComponentLoader and coerce
it to use ours.
But I wanted to get some opinions on this strategy as I am new to
elasticsearch ;)
my thanks,
Earle
Full stack trace:
2014-08-05 01:25:26,685 WARN elasticsearch[app02][generic][T#1]
org.elasticsearch.indices.cluster - [app02] [sakai_index][0] failed to
start shard
org.elasticsearch.indices.recovery.RecoveryFailedException:
[sakai_index][0]: Recovery failed from
[app01][0YtJIFeHSuehUfjjfMgv6A][ip-10-93-162-196][inet[/10.93.162.196:9300]]{local=false}
into
[app02][yshRulD0QjaNyu40Z6EGWQ][ip-10-7-174-145][inet[/10.7.174.145:9300]]{local=false}
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:307)
at
org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65)
at
org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: Failed to
deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize exception response from stream
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
Set active index
Hi All, I have setup a EFK system just for testing at the moment. It's running in a VM with not much RAM and I am having problem with the elasticsearch process because of this. The VIRT = 12GB which is approximately the total size of the indexes. My indexes are split by date like so... logstash-2014.06.01 logstash-2014.06.02... and so on. I'm guessing elasticsearch is trying to hold all of this in RAM. Is there a way I can setup elasticsearch to only search a specific index (or number of indices)? Is it just a case of archiving the logs I don't want ES to deal with? Ideally I'd like to work with only the last day or two of indexes which will hopefully all fit into RAM. Cheers, Rhys -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4b2533dd-752b-442f-9ba3-a71de0cac6ff%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: java.lang.ClassNotFoundException: org.elasticsearch.transport.RemoteTransportException
The cluster wants to transport an exception to your web app container, and
the web app does not have access to elasticsearch jar.
You should have a look at the ES server logs, if there are any exceptions,
to find the real problem.
Then, after fixing the real problem, you should try to configure your web
app so the elasticsearch jar is in the classpath.
To your question about WebAppClassLoader: ES uses the thread context class
loader
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/Classes.java#L56
Also, do you use any ES plugins? If plugins throw exceptions, they must
also be available in the web app.
Jörg
On Tue, Aug 5, 2014 at 4:57 PM, Earle Nietzel earle.niet...@gmail.com
wrote:
Using elasticsearch embedded in tomcat 7 where we have custom classloader
that shares spring application beans with many webapps. The API's to these
implementations are in shared but the implementations are in a separate
classloader ComponentLoader. Our search implementation is loaded from
ComponentLoader where elasticsearch has been promoted as our default search
engine.
Everything works fine on a single node but when in a cluster seeing the
following ClassNotFoundException issue when shards are trying to update on
other nodes.
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.transport.RemoteTransportException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
I thought this might have been related to
https://github.com/elasticsearch/elasticsearch/issues/4634 but that
doesn't seem to be the case.
My next idea is to look at why
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass is
invoking WebappClassLoader instead of our custom ComponentLoader and coerce
it to use ours.
But I wanted to get some opinions on this strategy as I am new to
elasticsearch ;)
my thanks,
Earle
Full stack trace:
2014-08-05 01:25:26,685 WARN elasticsearch[app02][generic][T#1]
org.elasticsearch.indices.cluster - [app02] [sakai_index][0] failed to
start shard
org.elasticsearch.indices.recovery.RecoveryFailedException:
[sakai_index][0]: Recovery failed from
[app01][0YtJIFeHSuehUfjjfMgv6A][ip-10-93-162-196][inet[/10.93.162.196:9300]]{local=false}
into
[app02][yshRulD0QjaNyu40Z6EGWQ][ip-10-7-174-145][inet[/10.7.174.145:9300]]{local=false}
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:307)
at
org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65)
at
org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: Failed to
deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize exception response from stream
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
Re: Kibana response time is too slow, need help identifying why
Turns out you shouldn't use the head plugin when querying for hot threads. I was able to get them by querying the API directly. Thanks for the tip! On Monday, August 4, 2014 11:28:16 PM UTC-7, Tony Chong wrote: Well, my slow logs are 0 bytes. My logging.yml looks okay but I don't think they are configured. I looked at the ES docs and saw that I should have these set somewhere. I'm thinking elastic search.yml configuration file? #index.search.slowlog.threshold.query.warn: 10s #index.search.slowlog.threshold.query.info: 5s #index.search.slowlog.threshold.query.debug: 2s #index.search.slowlog.threshold.query.trace: 500ms #index.search.slowlog.threshold.fetch.warn: 1s #index.search.slowlog.threshold.fetch.info: 800ms #index.search.slowlog.threshold.fetch.debug: 500ms #index.search.slowlog.threshold.fetch.trace: 200ms And querying for hot threads never returns a response. I have marvel installed as well. Is there something else I can look at? Thanks, Tony -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5e9c8cc8-ef5b-422c-b15e-48264036028d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Some observations with Curator
Using the most recent release (1.2.2) of Curator, I noticed that the documentation says --logfile while curator itself rejects --logfile anywhere and requires -l in front of the other options to direct its log entries. No big deal; I just tested it until it worked before adding it to the cron job. And it is working superbly. We will be standing up several ELK instances in various QA data centers to analyze several independent product load tests. These ELK instances are also independent, as we do not wish to flood the logstash data across any of our inter-data-center VPN / router connections. And because they are independent, our operations folks are leery of manually keeping track of multiple instances of the ELK stack with which they have no familiarity. And so, Elasticsearch Curator is becoming an integral part of the automation of the ELK stack for us, as it helps to keep our hard-working operations folks from overload. We wish for ELK to be an asset and not an added drain on time and effort, and Curator is a vital part of that goal. To the point where I no longer think of it as simply the ELK stack, but rather the KELTIC stack: *Kibana, Elasticsearch, Logstash, Time-based Indices, Curator*. But whether ELK or KELTIC, the stack is awesome! Many thanks to all who contributed and who continue to drive it forward! Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/39d8300d-27fc-42da-b10b-3bb8280573d4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Some observations with Curator
I am still fully in the nothing but E stack! Is anyone else using Elasticsearch for ... search? :) -- Ivan On Tue, Aug 5, 2014 at 10:50 AM, Brian brian.from...@gmail.com wrote: Using the most recent release (1.2.2) of Curator, I noticed that the documentation says --logfile while curator itself rejects --logfile anywhere and requires -l in front of the other options to direct its log entries. No big deal; I just tested it until it worked before adding it to the cron job. And it is working superbly. We will be standing up several ELK instances in various QA data centers to analyze several independent product load tests. These ELK instances are also independent, as we do not wish to flood the logstash data across any of our inter-data-center VPN / router connections. And because they are independent, our operations folks are leery of manually keeping track of multiple instances of the ELK stack with which they have no familiarity. And so, Elasticsearch Curator is becoming an integral part of the automation of the ELK stack for us, as it helps to keep our hard-working operations folks from overload. We wish for ELK to be an asset and not an added drain on time and effort, and Curator is a vital part of that goal. To the point where I no longer think of it as simply the ELK stack, but rather the KELTIC stack: *Kibana, Elasticsearch, Logstash, Time-based Indices, Curator*. But whether ELK or KELTIC, the stack is awesome! Many thanks to all who contributed and who continue to drive it forward! Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/39d8300d-27fc-42da-b10b-3bb8280573d4%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/39d8300d-27fc-42da-b10b-3bb8280573d4%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAd_07%3DkKv2EndY_uyfm%2B7CVwLR3mBXgWgv5E%2B6GR-bSA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
How can I combine two elasticsearch responses into one?
I am currently stuck with a result response from a multi-search that returns an array of responses for each query I add, but I need the responses to be combined so that the relevancy to determine their order. Getting back two separated lists of hits isn't great for creating a unified search results page. My current code for searching is: SearchRequestBuilder srb1 = client.prepareSearch(category) .setSearchType(SearchType.QUERY_THEN_FETCH) .setQuery(QueryBuilders.queryString(query).field(body).field(title).field(author) .addHighlightedField(body).addHighlightedField(title).addHighlightedField(author) .setHighlighterPreTags(div class='highlight').setHighlighterPostTags(/div); SearchRequestBuilder srb2 = client.prepareSearch(category+_pdfs) .setSearchType(SearchType.QUERY_THEN_FETCH) .setQuery( QueryBuilders.queryString(query) .field(file) .field(title) .field(author) ) .addHighlightedField(file).addHighlightedField(title).addHighlightedField(author) .setHighlighterPreTags(div class='highlight').setHighlighterPostTags(/div); MultiSearchResponse sr = client.prepareMultiSearch() .add(srb1) .add(srb2) .execute().actionGet(); Thanks for any help you all can give! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a309df1d-77a7-4450-ae41-e286054e6bda%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Constant re-syncing of mapping, ES 1.3.1
Hi, on my master node I constantly see in the logs the following messages: [2014-08-05 12:06:27,763][WARN ][cluster.metadata ] [ark.com. 01.ark74] [ark-profiles-2014-07-09] re-syncing mappings with cluster state for types [[profiles_v1]] [2014-08-05 12:06:27,783][WARN ][cluster.metadata ] [ark.com. 01.ark74] [ark-profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]] [2014-08-05 12:06:28,097][WARN ][cluster.metadata ] [ark.com. 01.ark74] [ark-profiles-2014-07-09] re-syncing mappings with cluster state for types [[profiles_v1]] [2014-08-05 12:06:28,118][WARN ][cluster.metadata ] [ark.com. 01.ark74] [ark-profiles-2014-08-01] re-syncing mappings with cluster state for types [[profiles_v1]] Any idea why this is happening? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c04f5cfa-a385-4be1-972f-047239622637%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
has_parent slow in filter, fast in query?
We ran into a problem with a slow query. Turns out having a has_child (or
has_parent) query in the filter part of a filtered query is a lot
slower than having the same has_child filter in the query part of the
filtered query.
The faster of the two queries runs in under 100ms, while the slower one
takes 250-300ms. I don't understand why these would have different run
times, any ideas?
The index is around 2GB. The child with about 600,000 docs, the parent with
about 3,000,000 docs.
These are the two queries:
Slower:
query: {
filtered: {
filter: {
and: {
filters: [
{
range: {
price: {
gt: 0
}
}
},
{
has_parent: {
type: product,
query: {
match_all: {}
}
}
}
]
}
}
}
}
Faster:
query: {
filtered: {
query: {
has_parent: {
type: product,
query: {
match_all: {}
}
}
},
filter: {
and: {
filters: [
{
range: {
price: {
gt: 0
}
}
}
]
}
}
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d811bca1-9798-49c5-b1f6-79cb5857f547%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: ES 1.3.0 and 1.2.3 released
Bump for Bigdesk 3 :) On Thu, Jul 24, 2014 at 12:40 AM, Lukáš Vlček lukas.vl...@gmail.com wrote: Hi, Bigdesk (2.x) uses JSONP. Elasticsearch is dropping JSONP support starting with 1.3 release. Still you can enable it in Elasticsearch but I do not recommend it if Bigdesk would be the only reason. From my point of view JSONP is hack-ish anti pattern and should be avoided. There is this ticket for Bigdesk https://github.com/lukas-vlcek/bigdesk/issues/55 I am still stuck with older version of Elasticsearch which means I do not feel the heat of dropped JSONP but feel free to jump to this ticket and have your say in comment so that I can see how much urgent it is for you. On a general note, I consider Bigdesk 2.x series more in maintenance mode, something new (Bigdesk 3.x) should be out hopefully by the end of this year. This means new release of Bigdesk 2.x will come out only if community express need for it. :-) Regards, Lukas On Thu, Jul 24, 2014 at 5:49 AM, vjbangis jessviray0...@gmail.com wrote: is the bigdesk plugin still supported in the new release? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e88b62d6-9ef5-4306-b880-b99182baa551%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/e88b62d6-9ef5-4306-b880-b99182baa551%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9cvUb-Uf0d1MdvTbS727Z227Y-k%2BFDqXK-qq0NeM5YnFsh4g%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAO9cvUb-Uf0d1MdvTbS727Z227Y-k%2BFDqXK-qq0NeM5YnFsh4g%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAND3Dph%2Bg0LFCiUW9NNeYqMLyFcuO3p70AvFth2_CfScshB7BQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Group by field and then sum the groups
Hi all,
I have an elastic database of posts, each post has a *user_id* and has
*likes* field. My goal is to output for a query how many likes in total
each user has.
I wondered if any one had any advice/direction I could take to achieve this?
input:
{user_id: 10, likes: 20}
{user_id: 9, likes: 10}
{user_id: 10, likes: 25}
{user_id: 9, likes: 15}
output:
User: 10 likes: 45
User: 9 likes: 25
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d016c5d0-ad03-488e-9c7b-73cf85999e10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: How to rebalance shard
Are you applying a custom routing to your documents? -- Ivan On Tue, Aug 5, 2014 at 2:33 AM, Warat Wongmaneekit canopyb...@gmail.com wrote: Now my cluster is not rebalance the data. How can I rebalance it please see the summary below. https://lh6.googleusercontent.com/-bH-VCgOCVWk/U-Ck6ykiJgI/SlI/S54duefOraA/s1600/Screen+Shot+2557-08-05+at+4.32.44+PM.png -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/06aab8c2-2dc8-4019-8a8c-219deca6e634%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/06aab8c2-2dc8-4019-8a8c-219deca6e634%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAkZSJRNFn03bki_kOBJGrEQJ2Oa-rGHVtZrW8TYLV6-A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Restoring a snapshot from one machine to a different machine.
I would like to use the Snapshot functionality to move indices from an OLD
server to a NEW server. My approach to do this task was to take a snapshot
from OLD server, copy the snapshot over to the NEW server (I zipped this
snapshot because it contained multiple indices), and try to restore that
snapshot from NEW server. However, I am not able to restore my snapshot on
NEW server.
Here are the commands that I've typed.
OLD$ curl -X PUT 'http://localhost:9200/_snapshot/my_backup' -d '{
type : fs,
settings : {
location : /mnt1/elasticsearch_backup/my_backup,
compress : true
}
}'
OLD$ curl -XPUT
localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true
OLD$ tar cvfz elasticsearch_backup.tgz /mnt1/elasticsearch_backup/
OLD$ scp elasticsearch_backup.tgz user@NEW:/mnt1/dropbox (We are using
AmazonAWS server)
NEW$ tar -zxvf elasticsearch_backup.tgz
curl -XPUT 'http://localhost:9200/_snapshot/my_backup2' -d '{
type: fs,
settings: {
location: /mnt1/dropbox/mnt1/my_backup/,
compress: true
}
}'
^ (At that location, I see my indices directory that I've unzipped from tar
file that I've copied over from OLD server)
NEW$ curl -X POST
localhost:9200/_snapshot/my_backup2/snapshot_1/_restore?wait_for_completion=true
and the error message I get at this point is
{error:SnapshotMissingException[[my_backup2:snapshot_1] is missing];
nested:
FileNotFoundException[/mnt1/dropbox/mnt1/my_backup/snapshot-snapshot_1 (No
such file or directory)]; ,status:404}
It would be great if someone can point out what I can possibly do wrong here
Thanks.
Additionally, if I type
http://OLD:9200/_snapshot/my_backup/_all
I see
{
- snapshots:
[
-
{
- snapshot: snapshot_1,
- indices:
[
- user_events_43294971_1401773217,
- user_events_111630610_1404969230,
- user_events_85270411_1398889365,
- user_events_86845965_1399787308
],
- state: SUCCESS,
- start_time: 2014-08-05T01:42:15.655Z,
- start_time_in_millis: 1407202935655,
- end_time: 2014-08-05T02:16:39.919Z,
- end_time_in_millis: 140720419,
- duration_in_millis: 2064264,
- failures: [ ],
- shards:
{
- total: 32,
- failed: 0,
- successful: 32
}
}
]
}
However, if I type
http://NEW:9200/_snapshot/my_backup2/_all
All I see is
{
- snapshots: [ ]
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1360def-15d8-4eab-8cbd-1257b56adca7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Storing Elasticsearch configuraton and deploying new clusters
We've been using Solr for the past several months, but have been considering Elasticsearch recently. One think I can't quite wrap my head around is how to keep track of Elasticsearch cluster configuration. Solr is largely based on config files written in XML. These are very straight forward to handle. You store them in your repository, and when deploying a new system, you drop the config files in the right places and start up Solr. Elasticsearch configuration all seems to happen via the API. If I set up a cluster for QA, and get it configured the way I want, how would I then go about deploying a production environment with the same configuration? What processes do people here use for deploying new environments? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54240e8e-3e91-43c8-9275-343c49333b0f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: How to transform SQL columns using JDBC River?
Just released - stored procedures are available in JDBC plugin 1.3.0.4 https://github.com/jprante/elasticsearch-river-jdbc/ Jörg On Tue, Aug 5, 2014 at 12:24 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: I have started work on stored procedures https://github.com/jprante/elasticsearch-river-jdbc/commit/247a6f5c029da5725cb47e3b970698326c37a22e It works with MySQL, but assigning renamed field names for Elasticsearch is still an open issue. Jörg On Tue, Aug 5, 2014 at 12:49 AM, Christopher Ambler ch...@insiderhouse.com wrote: Well, I can surely help test it out as it becomes ready for consumption, given a little guidance on usage (being undocumented and all :-)). But yeah, mapping will be key. Specifically, I have a column coming out of the SP (the first column, called domain) that will need to be mapped to the _id field. Normally, I'd SELECT domain AS _id ... but with an SP, of course, life is more difficult. On Monday, August 4, 2014 3:26:20 PM UTC-7, Jörg Prante wrote: At the moment it is an undocumented (incomplete) feature that can register field names to callable statement result parameters. You hit the nail - how to map result set output to field names is not done yet. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eac8c53b-d7a0-4b30-bc62-173a517d9f42%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/eac8c53b-d7a0-4b30-bc62-173a517d9f42%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFN-pYEgGxKrevO%2BwAmYKCv73pVDPYrq%2Bn%2Bm7sYXu5Ybg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Elasticsearch cluster with AWS Spot Instances
Hey guys, I've been thinking about the idea of running a elasticsearch cluster in AWS/EC2 mixing normal (on-demand or reserved) instances with spot instances. The motivation is simple: spot instances run just as fast as a normal instance but with a much lower price/hour. With the only caveat being that the instance can die at any moment (if the floating price goes higher than your bid). The idea is to have a 50/50 or 30/70 split between normal and spot instances. For example, 4 normal instances and 8 spot instances. My questions/doubts are: - Has any one tried this kind of setup? - One safety issue is that master/primary shards can be allocated in spot instances which are prone to die. Anyway around this? Maybe allocate all primaries in the normal instances? - The number of spot instances can't be much higher than the number of normal instances. That would mean that, maybe, a shard woulnd't have a replica in at least one of the normal instances, which could be risky. Anyway around this? Any thoughts? Thanks Felipe Hummel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20674cb7-5359-477f-b5c0-b2ac9b08cfce%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Searching email
Hi, We're using ES to index email, specifically mailing list messages. We'd like search to work similar to Gmail in that we'd like to match on either the subject or body of the email, and if it matches on the subject, we only want to display one result for that match (say the first message in that thread). In our naive implementation, we have an ES index for subjects and another for message bodies. But that gets us two sets of results, not combined. Is there a better way to structure the data, or a query that we're missing so that we get one set of combined results? Thanks, Mark -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d44932e2-dc6a-4574-a458-c02edbe9a13e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Storing Elasticsearch configuraton and deploying new clusters
There are some settings you can set via the API, but not all. The docs are pretty indepth so check out http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 6 August 2014 07:43, Hayden Muhl haydenm...@gmail.com wrote: We've been using Solr for the past several months, but have been considering Elasticsearch recently. One think I can't quite wrap my head around is how to keep track of Elasticsearch cluster configuration. Solr is largely based on config files written in XML. These are very straight forward to handle. You store them in your repository, and when deploying a new system, you drop the config files in the right places and start up Solr. Elasticsearch configuration all seems to happen via the API. If I set up a cluster for QA, and get it configured the way I want, how would I then go about deploying a production environment with the same configuration? What processes do people here use for deploying new environments? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54240e8e-3e91-43c8-9275-343c49333b0f%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/54240e8e-3e91-43c8-9275-343c49333b0f%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zj%3DdgB%3DKBL5D6wpwxzzUGHHHDdmEOzoo3WeenQY1jngw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Storing Elasticsearch configuraton and deploying new clusters
I was thinking more in terms of things like mappings. The only way of setting up mappings that I can find is via the API. If I've set up and tested a particular way of indexing and querying my documents on a QA system, I want to set my production system up the same way. What is your process for doing this? On Tue, Aug 5, 2014 at 4:45 PM, Mark Walkom ma...@campaignmonitor.com wrote: There are some settings you can set via the API, but not all. The docs are pretty indepth so check out http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 6 August 2014 07:43, Hayden Muhl haydenm...@gmail.com wrote: We've been using Solr for the past several months, but have been considering Elasticsearch recently. One think I can't quite wrap my head around is how to keep track of Elasticsearch cluster configuration. Solr is largely based on config files written in XML. These are very straight forward to handle. You store them in your repository, and when deploying a new system, you drop the config files in the right places and start up Solr. Elasticsearch configuration all seems to happen via the API. If I set up a cluster for QA, and get it configured the way I want, how would I then go about deploying a production environment with the same configuration? What processes do people here use for deploying new environments? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54240e8e-3e91-43c8-9275-343c49333b0f%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/54240e8e-3e91-43c8-9275-343c49333b0f%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/RDbwZSPSII0/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zj%3DdgB%3DKBL5D6wpwxzzUGHHHDdmEOzoo3WeenQY1jngw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEM624Zj%3DdgB%3DKBL5D6wpwxzzUGHHHDdmEOzoo3WeenQY1jngw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGqgWec_r_o%3De9RQq3Hjk6sUcdv2%2BA_NoM%2BDPr5jTbJMg1tmSw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Storing Elasticsearch configuraton and deploying new clusters
You can store mappings on the filesystem - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-conf-mappings.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 6 August 2014 10:03, Hayden Muhl haydenm...@gmail.com wrote: I was thinking more in terms of things like mappings. The only way of setting up mappings that I can find is via the API. If I've set up and tested a particular way of indexing and querying my documents on a QA system, I want to set my production system up the same way. What is your process for doing this? On Tue, Aug 5, 2014 at 4:45 PM, Mark Walkom ma...@campaignmonitor.com wrote: There are some settings you can set via the API, but not all. The docs are pretty indepth so check out http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 6 August 2014 07:43, Hayden Muhl haydenm...@gmail.com wrote: We've been using Solr for the past several months, but have been considering Elasticsearch recently. One think I can't quite wrap my head around is how to keep track of Elasticsearch cluster configuration. Solr is largely based on config files written in XML. These are very straight forward to handle. You store them in your repository, and when deploying a new system, you drop the config files in the right places and start up Solr. Elasticsearch configuration all seems to happen via the API. If I set up a cluster for QA, and get it configured the way I want, how would I then go about deploying a production environment with the same configuration? What processes do people here use for deploying new environments? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54240e8e-3e91-43c8-9275-343c49333b0f%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/54240e8e-3e91-43c8-9275-343c49333b0f%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/RDbwZSPSII0/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zj%3DdgB%3DKBL5D6wpwxzzUGHHHDdmEOzoo3WeenQY1jngw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEM624Zj%3DdgB%3DKBL5D6wpwxzzUGHHHDdmEOzoo3WeenQY1jngw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGqgWec_r_o%3De9RQq3Hjk6sUcdv2%2BA_NoM%2BDPr5jTbJMg1tmSw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAGqgWec_r_o%3De9RQq3Hjk6sUcdv2%2BA_NoM%2BDPr5jTbJMg1tmSw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624avLLZSqNp6U22q4h-FyrCuc9s%2Bamx1N-8do71jvGhqUA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Some observations with Curator
Hmm. What version of python are you using? I am able to use --logfile or -l interchangeably. I'm glad you like Curator, and I like KELTIC :) Nice acronym. On Tuesday, August 5, 2014 11:50:06 AM UTC-6, Brian wrote: Using the most recent release (1.2.2) of Curator, I noticed that the documentation says --logfile while curator itself rejects --logfile anywhere and requires -l in front of the other options to direct its log entries. No big deal; I just tested it until it worked before adding it to the cron job. And it is working superbly. We will be standing up several ELK instances in various QA data centers to analyze several independent product load tests. These ELK instances are also independent, as we do not wish to flood the logstash data across any of our inter-data-center VPN / router connections. And because they are independent, our operations folks are leery of manually keeping track of multiple instances of the ELK stack with which they have no familiarity. And so, Elasticsearch Curator is becoming an integral part of the automation of the ELK stack for us, as it helps to keep our hard-working operations folks from overload. We wish for ELK to be an asset and not an added drain on time and effort, and Curator is a vital part of that goal. To the point where I no longer think of it as simply the ELK stack, but rather the KELTIC stack: *Kibana, Elasticsearch, Logstash, Time-based Indices, Curator*. But whether ELK or KELTIC, the stack is awesome! Many thanks to all who contributed and who continue to drive it forward! Brian -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c806e490-b533-4b6d-a629-da2a1d031d7f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
System Requirements for ElasticSearch stack
Hello all, I'm new to the ELK stack. I will be logging Windows Events, Syslogs from firewalls, routers etc into my elasticsearch. I am expecting daily data of around 2GB to be logged into my elasticsearch server. I will be creating indices on daily or weekly basis. And my logs are going to be stored for atleast a year online and offline after that. I have been looking around and also searched this forum, but I was not able to find a definitive guide that explained how to design the architecture - RAM, # of CPU cores, # of Elastcisearch nodes and shards / node. The system will be mainly used for logging purposes only. So there won't be that many concurrent users. Appreciate any pointers on best practices in setting up the Elasticsearch deployment. Thanks, Gopinath -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/23818203-6fe3-49ae-996d-443c2250ea34%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Multiple master problem in elasticsearch 0.90.10
There should be no need to run a master and a data node on each machine. Only two masters is not enough to reliably form a consensus and you are only taking away processing power from the data node. -- Ivan On Tue, Aug 5, 2014 at 5:30 AM, Ankit Mittal ankit.lnc...@gmail.com wrote: Hi All, I am using 2 master node in my elasticsearch cluster and 2 data node. I ma using 2 RHEL server ( 1 master + 1 data node on each ) When i setup the above it is working fine for some time around 15 - 20 hours. but after then my cluster health is yellow. and the no available shards are reduce to half. when i check the elasticsearch data and master node , all 4 are running fine. Please help me in resolving the above issue. The requirement was to implement elasticsearch cluster on two server, for high availability ( 2 master on different server and 2 node on different server ). Please let me know if someone implemented high availability different from the above Thanks Ankit Mittla -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02683f10-8b05-45b0-a608-f35029bf4e1e%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/02683f10-8b05-45b0-a608-f35029bf4e1e%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQALFjMcaR7VT9aV4OdOEMeiXkGUOFKW-dFoXZcqrn-1Ww%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Searching email
Isn't better to create single document for each mail with fields subject and body (and whatever else you need from the mail) ? This way you can search by any or all of the fields, also you can define boosting for each field. For instance when your search matches the subject the mail will be scored higher in the result than if it matches the body, and you will get single set of results. 06 август 2014, сряда, 02:12:52 UTC+3, Mark Fletcher написа: Hi, We're using ES to index email, specifically mailing list messages. We'd like search to work similar to Gmail in that we'd like to match on either the subject or body of the email, and if it matches on the subject, we only want to display one result for that match (say the first message in that thread). In our naive implementation, we have an ES index for subjects and another for message bodies. But that gets us two sets of results, not combined. Is there a better way to structure the data, or a query that we're missing so that we get one set of combined results? Thanks, Mark -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bc4cf98-fc1c-408f-8c29-7291afcd3cdf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Group by field and then sum the groups
You can use aggregations:
{
aggs: {
user_likes: {
terms: {
field: user_id
},
aggs: {
likes_sum: {
terms: {
field: likes
}
}
}
}
}
}
05 август 2014, вторник, 23:11:59 UTC+3, Cameron Barker написа:
Hi all,
I have an elastic database of posts, each post has a *user_id* and has
*likes* field. My goal is to output for a query how many likes in total
each user has.
I wondered if any one had any advice/direction I could take to achieve
this?
input:
{user_id: 10, likes: 20}
{user_id: 9, likes: 10}
{user_id: 10, likes: 25}
{user_id: 9, likes: 15}
output:
User: 10 likes: 45
User: 9 likes: 25
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7bec5f9a-16cf-4723-87a6-7e95de45d0ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
