RE: Heavy indexing cause severe delay for searching
ES will send one query request to each shard when query on this index. So, if the number of shard is too big, the number of query request will also be too big to use up all query threads. From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Hajime Sent: Friday, January 23, 2015 3:19 PM To: elasticsearch@googlegroups.com Subject: Re: Heavy indexing cause severe delay for searching I still don't get why having many shards in one index matter.Since the index is just a logical grouping of shards or lucene threads,perhaps total num of shards per the cluster should be more significant?For Elasticsearch to grouping the shards cost a lot? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com mailto:elasticsearch+unsubscr...@googlegroups.com . To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsqDSevYiO7B_TE7UVV7uZRhm_oEB9%2B7R4futaufL5_BRw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsqDSevYiO7B_TE7UVV7uZRhm_oEB9%2B7R4futaufL5_BRw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/003501d036ec%24e0868b40%24a193a1c0%24%40gmail.com. For more options, visit https://groups.google.com/d/optout.
elk cluster plan with 7000EPS an 100/s search
Hi folks, I am building an elk cluster to index and search lots of http access log, about more than 7000Event per second and also there will be more than 100 cocurrent searchs. I have 2 machines. One of them has 24 cpu cores, 64G memory and 2T sata disk(no raid). The other one is much powerful, which has 24 core cpu, 384G memory and 300G sas disk*8. My plan is to build a 3-node elasticsearch, one running on small server, the other two running on the big one. Can I route all index request to one node while all search request to the other two nodes? Is this a good idea to do like this? Any comments? Thank you guys and happy holiday! Alan -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/000801d019ac%2400659130%240130b390%24%40gmail.com. For more options, visit https://groups.google.com/d/optout.
RE: term filter failed on very long fields
Hi all, is there any comment on this issues, is this a bug of elasticsearch.
This is really a big problem for me cause we have to query on this big long
field!
thank you!
Wang
From: Wang Yong [mailto:cnwangy...@gmail.com]
Sent: Monday, November 10, 2014 5:33 PM
To: elasticsearch@googlegroups.com
Subject: RE: term filter failed on very long fields
Thank you Vineeth, I changed the mapping to :
PUT test/_mapping/t
{
t : {
properties : {
message : {type : string, index: not_analyzed, store :
true }
}
}
}
And the result is the same.
Wang
From: mailto:elasticsearch@googlegroups.com elasticsearch@googlegroups.com [
mailto:elasticsearch@googlegroups.com mailto:elasticsearch@googlegroups.com]
On Behalf Of vineeth mohan
Sent: Monday, November 10, 2014 4:54 PM
To: mailto:elasticsearch@googlegroups.com elasticsearch@googlegroups.com
Subject: Re: term filter failed on very long fields
Hello Wang ,
Can you disable analyzer and try again .
Thanks
Vineeth
On Mon, Nov 10, 2014 at 2:07 PM, Wang Yong cnwangy...@gmail.com
mailto:cnwangy...@gmail.com wrote:
Hi folks,
I was trying to do a term filter on a very long string field, maybe more then
500 bytes, but I got 0 hit. So, I am wondering if there is a limitation on the
length of field while using term filter. The elasticsearch is 1.3.0 with the
map like this :
curl -XPUT 'http://localhost:9200/test/_mapping/t' -d '
{
t : {
properties : {
message : {type : string, store : true }
}
}
}
'
For the test, I put a doc into this map first by :
POST /test/t/
{
message:
12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
}
And then, I tried to search by :
GET test/_search
{
from : 0,
size : 20,
query : {
filtered : {
query : {
match_all : { }
},
filter : {
and : {
filters : [{
term : {
message :
12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
}
} ]
}
}
}
}
}
I got the result:
{
took: 0,
timed_out: false,
_shards: {
total: 1,
successful: 1,
failed: 0
},
hits: {
total: 0,
max_score: null,
hits: []
}
}
Any comment will be appreciated, thanks a lot!
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com
mailto:elasticsearch+unsubscr...@googlegroups.com .
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/002b01cffcc1%2484429f00%248cc7dd00%24%40gmail.com
https://groups.google.com/d/msgid/elasticsearch/002b01cffcc1%2484429f00%248cc7dd00%24%40gmail.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com
mailto:elasticsearch+unsubscr...@googlegroups.com .
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nC0O%3DD91YMO0mV0YD7k5QKY%2BAZddyVysUapNpGABMMYw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nC0O%3DD91YMO0mV0YD7k5QKY%2BAZddyVysUapNpGABMMYw%40mail.gmail.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/00a501cffee6%24ae4f00a0%240aed01e0%24%40gmail.com.
For more options, visit https
term filter failed on very long fields
Hi folks,
I was trying to do a term filter on a very long string field, maybe more
then 500 bytes, but I got 0 hit. So, I am wondering if there is a limitation
on the length of field while using term filter. The elasticsearch is 1.3.0
with the map like this :
curl -XPUT 'http://localhost:9200/test/_mapping/t' -d '
{
t : {
properties : {
message : {type : string, store : true }
}
}
}
'
For the test, I put a doc into this map first by :
POST /test/t/
{
message:
123456789012345678901234567890123456789012345678901234567890123456789012345
6789012345678901234567890123456789012345678901234567890123456789012345678901
2345678901234567890123456789012345678901234567890123456789012345678901234567
8901234567890123456789012345678901234567890123456789012345678901234567890123
4567890123456789012345678901234567890123456789012345678901234567890123456789
0123456789012345678901234567890123456789012345678901234567890123456789012345
678901234567890123456789012345678901234567890
}
And then, I tried to search by :
GET test/_search
{
from : 0,
size : 20,
query : {
filtered : {
query : {
match_all : { }
},
filter : {
and : {
filters : [{
term : {
message :
123456789012345678901234567890123456789012345678901234567890123456789012345
6789012345678901234567890123456789012345678901234567890123456789012345678901
2345678901234567890123456789012345678901234567890123456789012345678901234567
8901234567890123456789012345678901234567890123456789012345678901234567890123
4567890123456789012345678901234567890123456789012345678901234567890123456789
0123456789012345678901234567890123456789012345678901234567890123456789012345
678901234567890123456789012345678901234567890
}
} ]
}
}
}
}
}
I got the result:
{
took: 0,
timed_out: false,
_shards: {
total: 1,
successful: 1,
failed: 0
},
hits: {
total: 0,
max_score: null,
hits: []
}
}
Any comment will be appreciated, thanks a lot!
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/002b01cffcc1%2484429f00%248cc7dd00%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
RE: term filter failed on very long fields
Thank you Vineeth, I changed the mapping to :
PUT test/_mapping/t
{
t : {
properties : {
message : {type : string, index: not_analyzed, store :
true }
}
}
}
And the result is the same.
Wang
From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On
Behalf Of vineeth mohan
Sent: Monday, November 10, 2014 4:54 PM
To: elasticsearch@googlegroups.com
Subject: Re: term filter failed on very long fields
Hello Wang ,
Can you disable analyzer and try again .
Thanks
Vineeth
On Mon, Nov 10, 2014 at 2:07 PM, Wang Yong cnwangy...@gmail.com
mailto:cnwangy...@gmail.com wrote:
Hi folks,
I was trying to do a term filter on a very long string field, maybe more then
500 bytes, but I got 0 hit. So, I am wondering if there is a limitation on the
length of field while using term filter. The elasticsearch is 1.3.0 with the
map like this :
curl -XPUT 'http://localhost:9200/test/_mapping/t' -d '
{
t : {
properties : {
message : {type : string, store : true }
}
}
}
'
For the test, I put a doc into this map first by :
POST /test/t/
{
message:
12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
}
And then, I tried to search by :
GET test/_search
{
from : 0,
size : 20,
query : {
filtered : {
query : {
match_all : { }
},
filter : {
and : {
filters : [{
term : {
message :
12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
}
} ]
}
}
}
}
}
I got the result:
{
took: 0,
timed_out: false,
_shards: {
total: 1,
successful: 1,
failed: 0
},
hits: {
total: 0,
max_score: null,
hits: []
}
}
Any comment will be appreciated, thanks a lot!
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com
mailto:elasticsearch+unsubscr...@googlegroups.com .
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/002b01cffcc1%2484429f00%248cc7dd00%24%40gmail.com
https://groups.google.com/d/msgid/elasticsearch/002b01cffcc1%2484429f00%248cc7dd00%24%40gmail.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com
mailto:elasticsearch+unsubscr...@googlegroups.com .
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nC0O%3DD91YMO0mV0YD7k5QKY%2BAZddyVysUapNpGABMMYw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nC0O%3DD91YMO0mV0YD7k5QKY%2BAZddyVysUapNpGABMMYw%40mail.gmail.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/004401cffcc9%2462ce7f80%24286b7e80%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
index size impact on search performance?
Hi folks, I have an index storing lots of time serial data. The data are put
into index by :
curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '
{
val: 23,
timestamp: 1406822400
}'
And the only thing I search in this index is histogram facet in a very short
time range, like recent 5 min. I found that the performance was pretty
good at first. But when the index get bigger, the performance dropped to
unacceptable. I found the IO maybe the bottleneck by checking the result of
iostat.
My question is, even I only facet in a very short time range, why the size
of index has so big impact on the performance of such query? Do I have to
use daily index, just like logstash?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: index size impact on search performance?
Thank you david
most of my query looks like:
{
filtered: {
query: {
match_all: {}
},
filter: {
range: {
timestamp: {
from: 1403567280,
to: 1403567340,
include_lower: true,
include_upper: false
}
}
}
},
facets : {
val: {
statistical: {
field: val
}
}
}
}
Sent from Surface
From: David Pilato
Sent: Friday, August 1, 2014 10:43 AM
To: elasticsearch@googlegroups.com
Well. I guess it depends on your query. What does it look like?
--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 1 août 2014 à 04:14, Wang Yong cnwangy...@gmail.com a écrit :
Hi folks, I have an index storing lots of time serial data. The data are put
into index by :
curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '
{
val: 23,
timestamp: 1406822400
}'
And the only thing I search in this index is histogram facet in a very short
time range, like “recent 5 min”. I found that the performance was pretty good
at first. But when the index get bigger, the performance dropped to
unacceptable. I found the IO maybe the bottleneck by checking the result of
iostat.
My question is, even I only facet in a very short time range, why the size of
index has so big impact on the performance of such query? Do I have to use
daily index, just like logstash?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53db1703.a567440a.7a19.36b5%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.
Re: index size impact on search performance?
Thank you Mark,
in your word “time based indexes”, is that means create one index every day? if
I index my data in this way, i have to specify which index to search when
create query in my java client, based on the “from” and “to”.
Sent from Surface
From: Mark Walkom
Sent: Friday, August 1, 2014 11:02 AM
To: elasticsearch@googlegroups.com
If you're using time series data then it makes sense to use time based indexes.
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com
On 1 August 2014 12:43, David Pilato da...@pilato.fr wrote:
Well. I guess it depends on your query. What does it look like?
--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 1 août 2014 à 04:14, Wang Yong cnwangy...@gmail.com a écrit :
Hi folks, I have an index storing lots of time serial data. The data are put
into index by :
curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '
{
val: 23,
timestamp: 1406822400
}'
And the only thing I search in this index is histogram facet in a very short
time range, like “recent 5 min”. I found that the performance was pretty good
at first. But when the index get bigger, the performance dropped to
unacceptable. I found the IO maybe the bottleneck by checking the result of
iostat.
My question is, even I only facet in a very short time range, why the size of
index has so big impact on the performance of such query? Do I have to use
daily index, just like logstash?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YEg4XVNGAN%2BL8P6QHKjFn_9JTqkJE7TtMHh8fPqZhFxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53db1a2f.8d5b460a.30e5.41d5%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.
RE: best practice wanted for huge number of index time serial data
Thank you Mark, if I use daily index, I have to specify multiple indexes based on the time range. That will make my service a little more complicate. So I am wondering, even if I put all data in one huge index, as long as I limit the time range in my query, it looks like es will locate the data as quickly as I do it in a much smaller daily index, cause es will not need to search through the whole index, just need to locate the data first by the time range specified in the query. Is that true? Alan From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Mark Walkom Sent: 2014年7月14日 10:03 To: elasticsearch@googlegroups.com Subject: Re: best practice wanted for huge number of index time serial data This is pretty standard for logstash type data. Use daily indexes, don't use TTL. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com mailto:ma...@campaignmonitor.com web: www.campaignmonitor.com http://www.campaignmonitor.com On 14 July 2014 11:40, LiMac cnwangy...@gmail.com mailto:cnwangy...@gmail.com wrote: Hi folks, I am trying to index a huge number of time serial data. The total number will be 5k docs for one second which will continue for several months. I also need to search these data, but only inside a very small time rage, maybe one hour. Is there any best practice for this kind of use case? Thanks! Alan -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com mailto:elasticsearch+unsubscr...@googlegroups.com . To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7b659b5f-7f50-483d-a2d5-de9c2e4b650c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/7b659b5f-7f50-483d-a2d5-de9c2e4b650c%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com mailto:elasticsearch+unsubscr...@googlegroups.com . To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Y4GSxMc39SJf%3DCk5MwANMt7PRpnXt_oY3ZSyqNZoBpzw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEM624Y4GSxMc39SJf%3DCk5MwANMt7PRpnXt_oY3ZSyqNZoBpzw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/010601cfa155%24e55579b0%24b0006d10%24%40gmail.com. For more options, visit https://groups.google.com/d/optout.
