Re: How to diagnose slow queries every 10 minutes exactly?
Ok, I've identified the problem, and it has nothing to do with ES :) It's something funky with the network card or its driver. I isolated the problem by testing each of my two nodes independently, identified it was only a problem on one of the two. Then spun up a simple node http listener and was able to reproduce the problem with that, thus eliminating ES as even being involved. Still, this thread was very helpful, as I had a few other issues that reducing my shard count seems to have resolved. Thank you so much for your help. ES rocks. On Tuesday, April 21, 2015 at 2:20:22 PM UTC-7, Dave Reed wrote: > > Thanks for the info, but there's no load balancer involved here. No VMs > either.. nothing fancy. > > On Tuesday, April 21, 2015 at 1:55:18 PM UTC-7, AlexR wrote: >> >> it could be entirely unrelated but if I recall someone reported similar >> regular interval slowness. it proved to be the load balancer they used if I >> remember correctly. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/047ae734-f07d-4e4d-b0f1-f1417ddee044%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: How to diagnose slow queries every 10 minutes exactly?
Thanks for the info, but there's no load balancer involved here. No VMs either.. nothing fancy. On Tuesday, April 21, 2015 at 1:55:18 PM UTC-7, AlexR wrote: > > it could be entirely unrelated but if I recall someone reported similar > regular interval slowness. it proved to be the load balancer they used if I > remember correctly. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fade4dea-91c9-4459-82ef-56f780dcb704%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: How to diagnose slow queries every 10 minutes exactly?
Ok, I have deleted everything and restarted the ES service, on both nodes, and rebuilt the indexes. I have changed my shard count to 1 -- thank you for explaining that, I didn't realize the overhead was as large as it was. I do have some indexes that are much larger than others. The largest one has around 400k issues. Search performance seems fine with the 1 shard though, so I'll leave it at that. I can introduce a per-project shard count setting if needed later on. Results: My search count issue appears to be resolved, it was probably related to my large shard count. However, the slow queries once every 10 minutes remains. It's even still on the same schedule -- always at wall clock time with 9 minutes, e.g. 12:29pm. Here is a hot_threads call I ran while the slow searches were running (I replaced machine names and IPs with tokens for obscurity): ::: [][TbKES0BpQ---dAP1J9kKKQ][][inet[/:9300]] Hot threads at 2015-04-21T19:29:27.841Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: ::: [][aPof4TmzQlm7Ck7frfGeuA][][inet[/:9300]] Hot threads at 2015-04-21T19:29:26.250Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 0.0% (0s out of 500ms) cpu usage by thread 'Attach Listener' unique snapshot unique snapshot unique snapshot unique snapshot unique snapshot unique snapshot unique snapshot unique snapshot unique snapshot unique snapshot Here is the slow search log for the same timeframe (first # is the time to get the response, second # is the .took field from the ES response json) 4/21/2015, 12:29:27 PM 8586 8361 4/21/2015, 12:29:27 PM 9681 9454 4/21/2015, 12:29:27 PM 7685 7457 There's nothing in ES logs after the indexes were initially created, and it's just info type things like this: [2015-04-21 10:48:05,029][INFO ][cluster.metadata ] [] updating number_of_replicas to [1] for indices [] It's so interesting that it's every 10 minutes and always at x9 minutes, regardless of the fact that I've restarted ES. There still could be an external reason for this, but like I said, there's no CPU usage or Disk I/O going on during these periods, so it's very strange. I have no leads other than these slow search times. Thanks so much for your help. Any other troubleshooting ideas would be greatly appreciated. -Dave On Tuesday, April 21, 2015 at 12:12:38 AM UTC-7, David Pilato wrote: > > Some notes. You are using defaults. So you have 5 shards per index. 1000 > primary shards. > With replicas, it means 1000 shards per Node. Which means 1000 Lucene > instances. > > First thing to do is IMO to use only one shard per index unless you need > more for "big" indices. > > Then, have 3 nodes and set minimum_master_nodes to 2. You probably ran > into a split brain issue which could explain the difference you are seeing. > I would probably set replica to 0 and then to 1 again. But if you follow > my first advice, you have to reindex, so reindex with 1 shard, 0 replica, > then set replica to 1. > > > My 2 cents > > David > > Le 21 avr. 2015 à 08:31, Dave Reed > a > écrit : > > The logs are basically empty. There's activity from when I am creating the > indexes, but that's about it. Is there a logging level that could be > increased? I will run hot_thread as soon as I can in the morning and post > results when I can. > > I have each index set to 1 replica. If it matters, I first import them > with 0 replicas then set it to 1 when they are done. Shard count I will > have to check and get back to you, but it's whatever the default would be, > we haven't tweaked that. > > I have 200+ indexes because I have 200+ different "projects" represented, > and each one has it's own set of mappings. Mappings which could collide on > name. I originally tried having a single index with each project > represented by a Type instead, but a conversation I had on this forum about > that led me away from it due to the fact two different projects may > sometimes have the same field names but with different types. Most searches > (99.9%) are done on a per-project basis, and changes to projects can create > a need to reindex the project, so having the segregation is nice, lest I > have to reimport the entire thing. > > In case this is related, I also found that a document was changed and > reindexed, but searches would sometimes include it and sometimes not. I > could literally just refresh the search over and over again, and it would > appear in the results roughly 50% of the time. 0 results, then 0, 0, then 1 > result, 1, 1, then 0 again, etc.. I was running the search against one of > the nodes via _head. The cluster health was green during all
Re: How to diagnose slow queries every 10 minutes exactly?
The logs are basically empty. There's activity from when I am creating the indexes, but that's about it. Is there a logging level that could be increased? I will run hot_thread as soon as I can in the morning and post results when I can. I have each index set to 1 replica. If it matters, I first import them with 0 replicas then set it to 1 when they are done. Shard count I will have to check and get back to you, but it's whatever the default would be, we haven't tweaked that. I have 200+ indexes because I have 200+ different "projects" represented, and each one has it's own set of mappings. Mappings which could collide on name. I originally tried having a single index with each project represented by a Type instead, but a conversation I had on this forum about that led me away from it due to the fact two different projects may sometimes have the same field names but with different types. Most searches (99.9%) are done on a per-project basis, and changes to projects can create a need to reindex the project, so having the segregation is nice, lest I have to reimport the entire thing. In case this is related, I also found that a document was changed and reindexed, but searches would sometimes include it and sometimes not. I could literally just refresh the search over and over again, and it would appear in the results roughly 50% of the time. 0 results, then 0, 0, then 1 result, 1, 1, then 0 again, etc.. I was running the search against one of the nodes via _head. The cluster health was green during all of this. That was surprising and something I was going to investigate more on my own, but perhaps these two problems are related. I'm really hitting the limit of what I know how to troubleshoot with ES, hence I am really hoping for help here :) On Monday, April 20, 2015 at 10:23:46 PM UTC-7, David Pilato wrote: > > Could you run a hot_threads API call when this happens? > Anything in logs about GC? > > BTW 200 indices is a lot for 2 nodes. And how many shards/replicas do you > have? > Why do you need so many indices for 2m docs? > > > David > > Le 21 avr. 2015 à 01:16, Dave Reed > a > écrit : > > I have a 2-node cluster running on some beefy machines. 12g and 16g of > heap space. About 2.1 million documents, each relatively small in size, > spread across 200 or so indexes. The refresh interval is 0.5s (while I > don't need realtime I do need relatively quick refreshes). Documents are > continuously modified by the app, so reindex requests trickle in > constantly. By trickle I mean maybe a dozen a minute. All index requests > are made with _bulk, although a lot of the time there's only 1 in the list. > > Searches are very fast -- normally taking 50ms or less. > > But oddly, exactly every 10 minutes, searches slow down for a moment. The > exact same query that normally takes <50ms takes 9000ms, for example. Any > other queries regardless of what they are also take multiple seconds to > complete. Once this moment passes, search queries return to normal. > > I have a tester I wrote that continuously posts the same query and logs > the results, which is how I discovered this pattern. > > Here's an excerpt. Notice that query time is great at 3:49:10, then at :11 > things stop for 10 seconds. At :21 the queued up searches finally come > through. The numbers reported are the "took" field from the ES search > response. Then things resume as normal. This is true no matter which node I > run the search against. > > This pattern repeats like this every 10 minutes, to the second, for days > now. > > 3:49:09, 47 > 3:49:09, 63 > 3:49:10, 31 > 3:49:10, 47 > 3:49:11, 47 > 3:49:11, 62 > 3:49:21, 8456 > 3:49:21, 5460 > 3:49:21, 7457 > 3:49:21, 4509 > 3:49:21, 3510 > 3:49:21, 515 > 3:49:21, 1498 > 3:49:21, 2496 > 3:49:22, 2465 > 3:49:22, 2636 > 3:49:22, 6506 > 3:49:22, 7504 > 3:49:22, 9501 > 3:49:22, 4509 > 3:49:22, 1638 > 3:49:22, 6646 > 3:49:22, 9641 > 3:49:22, 655 > 3:49:22, 3667 > 3:49:22, 31 > 3:49:22, 78 > 3:49:22, 47 > 3:49:23, 47 > 3:49:23, 47 > 3:49:24, 93 > > I've ruled out any obvious periodical process running on either node in > the cluster. It's pretty clean. Disk I/O, CPU, RAM, etc stays pretty > consistent and flat even during one of these blips. These are beefy > machines like I said, so there's plenty of CPU and RAM available. > > Any advice on how I can figure out what ES is waiting for would be > helpful. Is there any process ES runs every 10 minutes by default? > > Thanks! > -Dave > > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. &
How to diagnose slow queries every 10 minutes exactly?
I have a 2-node cluster running on some beefy machines. 12g and 16g of heap space. About 2.1 million documents, each relatively small in size, spread across 200 or so indexes. The refresh interval is 0.5s (while I don't need realtime I do need relatively quick refreshes). Documents are continuously modified by the app, so reindex requests trickle in constantly. By trickle I mean maybe a dozen a minute. All index requests are made with _bulk, although a lot of the time there's only 1 in the list. Searches are very fast -- normally taking 50ms or less. But oddly, exactly every 10 minutes, searches slow down for a moment. The exact same query that normally takes <50ms takes 9000ms, for example. Any other queries regardless of what they are also take multiple seconds to complete. Once this moment passes, search queries return to normal. I have a tester I wrote that continuously posts the same query and logs the results, which is how I discovered this pattern. Here's an excerpt. Notice that query time is great at 3:49:10, then at :11 things stop for 10 seconds. At :21 the queued up searches finally come through. The numbers reported are the "took" field from the ES search response. Then things resume as normal. This is true no matter which node I run the search against. This pattern repeats like this every 10 minutes, to the second, for days now. 3:49:09, 47 3:49:09, 63 3:49:10, 31 3:49:10, 47 3:49:11, 47 3:49:11, 62 3:49:21, 8456 3:49:21, 5460 3:49:21, 7457 3:49:21, 4509 3:49:21, 3510 3:49:21, 515 3:49:21, 1498 3:49:21, 2496 3:49:22, 2465 3:49:22, 2636 3:49:22, 6506 3:49:22, 7504 3:49:22, 9501 3:49:22, 4509 3:49:22, 1638 3:49:22, 6646 3:49:22, 9641 3:49:22, 655 3:49:22, 3667 3:49:22, 31 3:49:22, 78 3:49:22, 47 3:49:23, 47 3:49:23, 47 3:49:24, 93 I've ruled out any obvious periodical process running on either node in the cluster. It's pretty clean. Disk I/O, CPU, RAM, etc stays pretty consistent and flat even during one of these blips. These are beefy machines like I said, so there's plenty of CPU and RAM available. Any advice on how I can figure out what ES is waiting for would be helpful. Is there any process ES runs every 10 minutes by default? Thanks! -Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0c715e1e-f45b-46fb-ae39-52318e126203%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Querystring search: Tokens are out of order
To perhaps answer my own question, I think I understand the difference. details:"foo bar" Would search for the tokens in the same order (implied by the docs I referenced). But details:foo-bar Would not honor the order. The quotes have more meaning than to enclose the phrase... if that is true then these two queries are not the same, which is different than I thought: details:foo\ bar != details:"foo bar" Or am I barking up the wrong tree... On Tuesday, April 14, 2015 at 1:34:28 PM UTC-7, Dave Reed wrote: > > Thanks, though unless I am misunderstanding it, the docs imply otherwise: > > For example, from: > > http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html > > The query string is parsed into a series of *terms* and *operators*. A >> term can be a single word — quick or brown — or a phrase, surrounded by >> double quotes — "quick brown" — which searches for all the words in the >> phrase, in the same order. > > > So what gives? :) > > >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b28591e3-3818-4b12-8a22-cac466c9ec7c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Querystring search: Tokens are out of order
Thanks, though unless I am misunderstanding it, the docs imply otherwise:
For example, from:
http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
The query string is parsed into a series of *terms* and *operators*. A term
> can be a single word — quick or brown — or a phrase, surrounded by double
> quotes — "quick brown" — which searches for all the words in the phrase,
> in the same order.
So what gives? :)
On Tuesday, April 14, 2015 at 1:15:24 PM UTC-7, James Macdonald wrote:
>
> You analysis of what is going on sounds correct. However, Elasticsearch's
> results are also correct. When it analyzes the search string, your query
> becomes a match query on "foo" AND "bar", which matches any document
> containing both of those terms. Most queries against analyzed fields do not
> respect the original ordering of the terms.
>
> One thing you could try is looking into the match_phrase query (
> http://www.elastic.co/guide/en/elasticsearch/guide/master/phrase-matching.html)
>
> which is aware of the ordering of the terms. Using the base match_phrase
> query for "foo bar" will not match either "foo xyz bar" or "bar xyz foo".
> If you still need to match things like "foo xyz bar" you may be able to do
> that using the slop parameter, depending on what exactly the use case is.
>
> James
>
> On Tue, Apr 14, 2015 at 2:03 PM, Dave Reed > wrote:
>
>> I have the following search:
>>
>> {
>> "query": {
>> "filtered": {
>> "query": {
>> "query_string": {
>> "default_operator": "AND",
>> "query": "details:foo\\-bar"
>> }
>> },
>> "filter": {
>> "term": {
>> "deleted": false
>> }
>> }
>> }
>> }
>> }
>>
>>
>>
>> The details field is analyzed using pattern tokenizer, as so:
>>
>> settings: {
>> index.analysis.analyzer.letterordigit.pattern: "[^\\p{L}\\p{N}]+",
>> index.analysis.analyzer.letterordigit.type: "pattern"
>> }
>>
>>
>> This breaks the field into tokens separated by any non-letter or
>> non-numeric character.
>>
>> But the user is searching for "foo-bar" which contains a non alphanumeric
>> character. I assume, but correct me if I'm wrong, that ES will apply the
>> same analyzer to that string. So it is broken into two tokens: ["foo",
>> "bar"], and then the default_operator kicks in and essentially turns the
>> query into "details:foo AND detail:bar".
>>
>> My problem is that it will match documents containing "foo xyz bar" and
>> "bar xyz foo" -- in the latter case, the tokens are in the reverse order
>> from the user's search. I'm fine with it matching the former, but it's a
>> stretch to convince the user that the latter is intended.
>>
>> The search string is provided by the user, so I can't really build a
>> complex query with different query types, hence the basic querystring
>> search.
>>
>> Any advice or corrections to my assumptions is appreciated!
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7a355b94-358f-4c5a-ac16-31ac7a0c0abe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Querystring search: Tokens are out of order
I have the following search:
{
"query": {
"filtered": {
"query": {
"query_string": {
"default_operator": "AND",
"query": "details:foo\\-bar"
}
},
"filter": {
"term": {
"deleted": false
}
}
}
}
}
The details field is analyzed using pattern tokenizer, as so:
settings: {
index.analysis.analyzer.letterordigit.pattern: "[^\\p{L}\\p{N}]+",
index.analysis.analyzer.letterordigit.type: "pattern"
}
This breaks the field into tokens separated by any non-letter or
non-numeric character.
But the user is searching for "foo-bar" which contains a non alphanumeric
character. I assume, but correct me if I'm wrong, that ES will apply the
same analyzer to that string. So it is broken into two tokens: ["foo",
"bar"], and then the default_operator kicks in and essentially turns the
query into "details:foo AND detail:bar".
My problem is that it will match documents containing "foo xyz bar" and
"bar xyz foo" -- in the latter case, the tokens are in the reverse order
from the user's search. I'm fine with it matching the former, but it's a
stretch to convince the user that the latter is intended.
The search string is provided by the user, so I can't really build a
complex query with different query types, hence the basic querystring
search.
Any advice or corrections to my assumptions is appreciated!
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: How to handle searching across all indexes while reindex procedure is running
Ah ha, yes indeed :) I knew an index could have multiple aliases, but didn't connect the dots. That will work perfectly :) Thanks! On Tuesday, February 17, 2015 at 7:39:03 PM UTC-8, David Pilato wrote: > > Why not creating another alias "index_all" on all indices? > And update it within step 3? > > -- > David ;-) > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 18 févr. 2015 à 01:23, Dave Reed > a > écrit : > > I have a setup that will cause there to be a few hundred indexes, each of > which may need to be rebuilt at any given time. To accomplish this, I'm > creating an alias for each index, which point to an incrementing index > number. For example, the state of the indexes may look like this: > > Index -> Alias > indexA -> indexA_v1 > indexB -> indexB_v2 > indexC -> indexC_v11 > indexD -> indexD_v3 > > An index may need to be rebuilt because the mappings need to change. I > detect when this happens and go through a reindex procedure: > > 1. Create a new index (e.g. IndexA_v2) with the new mappings. > 2. Start indexing all the documents into it (could take an hour or two). > 3. Swap the "IndexA" alias to point to IndexA_v2. > 4. Delete IndexA_v1. > > I need these to be in separate indexes, not just separate types, because > the mappings can vary widely and potentially include the same field names > but with different mappings. I've learned that mappings should have unique > names within an index, even though types can have separate mappings. > > For searches, there are index-specific searches that will just search on > the alias name for the index required (e.g. IndexA). No problem there. > > But there is type of search in the app that needs to search across all of > these indexes. But since there could be hundreds of them, adding them to > the search URL seems wrong -- I would think I'd hit some kind of URL limit > at some point: > > http://server:9200/indexA,indexB,indexC.indexZZZ/documenttype/_search > > I could use "_all" instead of listing them individually, but with a > specific type that I know only those indexes will have: > > http://server:9200/_all/documenttype/_search > > When there is no reindex in progress, that works great. But... the "_all" > will include results from any indexes that are in the process of being > rebuilt, meaning that I could get duplicate results from those indexes > during the reindex procedure. > > I hope that makes sense. I'm open to a different layout if this isn't the > ideal way to handle it! > > Thanks for any help > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearc...@googlegroups.com . > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/7ad24fda-a7ca-4ed8-82ff-70797da1a736%40googlegroups.com > > <https://groups.google.com/d/msgid/elasticsearch/7ad24fda-a7ca-4ed8-82ff-70797da1a736%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/968e8714-e7dd-4c3a-8747-a022a147ca16%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
How to handle searching across all indexes while reindex procedure is running
I have a setup that will cause there to be a few hundred indexes, each of which may need to be rebuilt at any given time. To accomplish this, I'm creating an alias for each index, which point to an incrementing index number. For example, the state of the indexes may look like this: Index -> Alias indexA -> indexA_v1 indexB -> indexB_v2 indexC -> indexC_v11 indexD -> indexD_v3 An index may need to be rebuilt because the mappings need to change. I detect when this happens and go through a reindex procedure: 1. Create a new index (e.g. IndexA_v2) with the new mappings. 2. Start indexing all the documents into it (could take an hour or two). 3. Swap the "IndexA" alias to point to IndexA_v2. 4. Delete IndexA_v1. I need these to be in separate indexes, not just separate types, because the mappings can vary widely and potentially include the same field names but with different mappings. I've learned that mappings should have unique names within an index, even though types can have separate mappings. For searches, there are index-specific searches that will just search on the alias name for the index required (e.g. IndexA). No problem there. But there is type of search in the app that needs to search across all of these indexes. But since there could be hundreds of them, adding them to the search URL seems wrong -- I would think I'd hit some kind of URL limit at some point: http://server:9200/indexA,indexB,indexC.indexZZZ/documenttype/_search I could use "_all" instead of listing them individually, but with a specific type that I know only those indexes will have: http://server:9200/_all/documenttype/_search When there is no reindex in progress, that works great. But... the "_all" will include results from any indexes that are in the process of being rebuilt, meaning that I could get duplicate results from those indexes during the reindex procedure. I hope that makes sense. I'm open to a different layout if this isn't the ideal way to handle it! Thanks for any help -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7ad24fda-a7ca-4ed8-82ff-70797da1a736%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Search sort using a field with an "index_name" results in "No mapping found"
Just in the interest of having a two-way link, I opened a github issue about it here: https://github.com/elasticsearch/elasticsearch/issues/8980 On Monday, December 15, 2014 9:41:01 AM UTC-8, Dave Reed wrote: > > I see you tried adding index_name to the inner field as well. Nope, I'm > afraid that did not work. > > Anyone have any thoughts here? This definitely seems like a bug :) > >> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/648de326-80eb-402e-91a0-32ccc57ba7ce%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Search sort using a field with an "index_name" results in "No mapping found"
I see you tried adding index_name to the inner field as well. Nope, I'm
afraid that did not work.
Anyone have any thoughts here? This definitely seems like a bug :)
On Thursday, December 11, 2014 10:12:02 PM UTC-8, David Pilato wrote:
>
> What happen if you try:
>
> "mappings": {
> "people": {
> "properties": {
>"work_email": {
> "type": "string",
> "index_name": "email",
> "fields": {
> "raw": {
> "type": "string",
> "index_name": "email.raw",
> "index": "not_analyzed"
> }
> }
>}
> }
> }
> }
> }'
>
> Does this work?
>
> David
>
> Le 12 déc. 2014 à 01:12, Dave Reed > a
> écrit :
>
> I ran into this problem, and discovered this StackOverflow post (with no
> answer):
>
> http://stackoverflow.com/questions/23348172/elasticsearch-index-name-with-multi-field
>
> That original poster included a gist that reproduces it quite nicely
> (confirmed on 1.4.1):
> https://gist.github.com/pmishev/11375297
>
> Partially replicated here for convenience (see the link for full curl
> steps to reproduce):
>
> # Create mapping
> curl -XPOST localhost:9200/index1/ -d '{
> "mappings": {
> "people": {
> "properties": {
>"work_email": {
> "type": "string",
> "index_name": "email",
> "fields": {
> "raw": {
> "type": "string",
> "index": "not_analyzed"
> }
> }
>}
> }
> }
> }
> }'
>
>
>
> Basically, I have an analyzed string field and a non-analyzed version of
> it so that I can sort by it. But instead of sorting by "field.sort" or
> "field.raw", I need to use "field_sort", basically for back compat reasons.
> To accomplish that I tried giving the non-analyzed sortable version of the
> string field an index_name. But trying to sort using the given index_name
> results in an error: "No mapping found for [field_sort] in order to sort
> on".
>
> The gist referenced above is slightly different than I am explaining,
> because it puts the "index_name" on the parent field instead of the sorted
> part, but the result is the same.
>
> To put it simply, it seems that you cannot sort by a field by its
> index_name, you have to use the full path to it, which is a problem for me.
> Is this expected? Why should I be able to search for a field by its
> index_name but not sort by it -- that seems unintended. Am I doing it
> wrong? Is there a better way to accomplish the same thing?
>
> Thanks for the help!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/af25ab54-407b-4e58-95ea-94c12fb3fb65%40googlegroups.com
>
> <https://groups.google.com/d/msgid/elasticsearch/af25ab54-407b-4e58-95ea-94c12fb3fb65%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/71f2a195-b6b1-4666-93df-517603d568ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Search sort using a field with an "index_name" results in "No mapping found"
I ran into this problem, and discovered this StackOverflow post (with no
answer):
http://stackoverflow.com/questions/23348172/elasticsearch-index-name-with-multi-field
That original poster included a gist that reproduces it quite nicely
(confirmed on 1.4.1):
https://gist.github.com/pmishev/11375297
Partially replicated here for convenience (see the link for full curl steps
to reproduce):
# Create mapping
curl -XPOST localhost:9200/index1/ -d '{
"mappings": {
"people": {
"properties": {
"work_email": {
"type": "string",
"index_name": "email",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}'
Basically, I have an analyzed string field and a non-analyzed version of it
so that I can sort by it. But instead of sorting by "field.sort" or
"field.raw", I need to use "field_sort", basically for back compat reasons.
To accomplish that I tried giving the non-analyzed sortable version of the
string field an index_name. But trying to sort using the given index_name
results in an error: "No mapping found for [field_sort] in order to sort
on".
The gist referenced above is slightly different than I am explaining,
because it puts the "index_name" on the parent field instead of the sorted
part, but the result is the same.
To put it simply, it seems that you cannot sort by a field by its
index_name, you have to use the full path to it, which is a problem for me.
Is this expected? Why should I be able to search for a field by its
index_name but not sort by it -- that seems unintended. Am I doing it
wrong? Is there a better way to accomplish the same thing?
Thanks for the help!
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/af25ab54-407b-4e58-95ea-94c12fb3fb65%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Filtered alias with a _default_ mapping?
I have an index that will end up containing a few hundred types. I'm
dynamically creating these types based on some app logic. Each one can have
a different mapping, but they have some similarities.
One of these similarities is that all the documents across all the types
can be soft-deleted. Soft deleted documents are still indexed for some
administration purposes, but I don't want them to appear in search results.
So I tried creating a filtered alias.
First the mapping sets up the deleted field mapping, using _default_:
"mappings": {
"_default_": {
"properties": {
"deleted": {
"type": "boolean"
}
}
}
}
Then, I create the alias right after creating the index, like so:
POST /_aliases
{
"actions": [
{
"add": {
"index": "myindex_v1",
"alias": "myindex",
"filter": { "term": { "deleted": false } }
}
}
]
}
But, when searching by this alias, no documents are ever matched. It's not
a problem with the search, because changing it to use the actual index name
works fine. I only get 0 results when using the index alias name. If I
recreate the alias with a remove/add alias post after I've indexed
documents into it, it starts working.
I suspected the _default_ mapping wasn't working with the alias filter. I
though maybe it works if I recreate it after the documents are indexed
because the real type mapping was dynamically generated by that time. So
just to try it out, I hard coded my type mapping to a specific type,
instead of _default_, and sure enough, that works.
So, am I correct that a _default_ mapping is not sufficient to make a
filtered alias work? Is there a better way to achieve the requirement of
always filtering out soft-deleted documents from search results without
having to tag on filter criteria to every search?
Thanks!
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aeb464cf-17a3-4e23-9539-c759e30c6b3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Query_string search containing a dash has unexpected results
Yes, and this was the key, thank you so much. But see my reply above about
the docs on that param being confusing. That was really the source of the
problem for me.
On Monday, November 10, 2014 4:15:05 PM UTC-8, Amish Asthana wrote:
>
> No I am not saying that . I am saying this :
> GET my_index_v1/mytype/_search
> {
> "query": {
> "query_string": {
> "default_field": "name",
> "query": "welcome-doesnotmatchanything",
> "default_operator": "AND"
> }
> }
> }
>
> Here I will not get a match as expected. If I do not specify then OR is
> the deafult operator and it will match.
> amish
>
>
> On Monday, November 10, 2014 4:01:14 PM UTC-8, Dave Reed wrote:
>>
>> My default operator doesn't matter if I understand it correctly, because
>> I'm specifying the operate explicitly. Also, I can reproduce this behavior
>> using a single search term, so there's no operator to speak of. Unless
>> you're saying that the default operator applies to a single term query if
>> it is broken into tokens?
>>
>>
>>> Note that using the welcome-doesnotmatchanything analzyzer will break
>>> into two tokens with OR and your document will match unless you use AND
>>
>>
>> This concerns me... my search looks like:
>>
>> message:welcome-doesnotmatchanything
>>
>> I cannot break that into an AND. The entire thing is a value provided by
>> the end user. You're saying I should on the app side break the string they
>> entered into tokens and join them with ANDs? That doesn't seem viable...
>>
>> Let me back up and say what I'm expecting the user to be able to do.
>> There's a single text box where they can enter a search query, with the
>> following rules:
>> 1. The user may use a trailing wildcard, e.g. foo*
>> 2. The user may enter multiple terms separated by a space. Only documents
>> containing all of the terms will match.
>> 3. The user might enter special characters, such as in "battle-axe",
>> simply because that is what they think they should search for, which should
>> match documents containing "battle" and "axe" (the same as a search for
>> "battle axe").
>>
>> To that end, I am taking their search string and forming a search like
>> this:
>>
>> message: AND...
>>
>> Where the string is split on spaces and joined with the AND clauses. For
>> each individual part of the search phrase, I take care of escaping special
>> characters (except "*" since I am allowing them to use wildcards). For
>> example, if they entered "foo bar!", I would generate this query:
>>
>> message:foo AND message:bar\!
>>
>> The problem is they are entering "battle-axe", causing me to generate
>> this:
>>
>> message:battle\-axe
>>
>> But that ends up being the same as:
>>
>> (message:battle OR message:axe)
>>
>> I guess that is what I was not expecting. Because of this behavior, I
>> have to know from my app point of view what tokens I should be splitting
>> the original string on, so that I can join them back together with ANDs.
>> But that means basically reimplementing the tokenizer on my end, does it
>> not? There must be a better way? Like specifying I want those terms to be
>> joined with ANDs instead?
>>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4d64842d-6374-465d-b261-452d845a3985%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Query_string search containing a dash has unexpected results
Ok... specifying default_operator: AND worked In that case, I'd like to say that the docs on that option are incomplete or confusing. It says: The default operator used if no explicit operator is specified. For example, with a default operator of OR, the query capital of Hungary is translated to capital OR of OR Hungary, and with default operator of AND, the same query is translated to capital AND of AND Hungary. The default value is OR. That's all well and good, but my query does not have multiple terms like that. I have a single term for a single field. The default operator is applying to the resulting tokens of that, after they are generated by the analyzer. I assumed that the default operator applied at the level of the query being parsed and that had nothing at all to do with the analyzer. Making that clearer could have saved me a lot of time :) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c1a058ca-b179-495a-8b82-e65fece4f99f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Query_string search containing a dash has unexpected results
My default operator doesn't matter if I understand it correctly, because I'm specifying the operate explicitly. Also, I can reproduce this behavior using a single search term, so there's no operator to speak of. Unless you're saying that the default operator applies to a single term query if it is broken into tokens? > Note that using the welcome-doesnotmatchanything analzyzer will break > into two tokens with OR and your document will match unless you use AND This concerns me... my search looks like: message:welcome-doesnotmatchanything I cannot break that into an AND. The entire thing is a value provided by the end user. You're saying I should on the app side break the string they entered into tokens and join them with ANDs? That doesn't seem viable... Let me back up and say what I'm expecting the user to be able to do. There's a single text box where they can enter a search query, with the following rules: 1. The user may use a trailing wildcard, e.g. foo* 2. The user may enter multiple terms separated by a space. Only documents containing all of the terms will match. 3. The user might enter special characters, such as in "battle-axe", simply because that is what they think they should search for, which should match documents containing "battle" and "axe" (the same as a search for "battle axe"). To that end, I am taking their search string and forming a search like this: message: AND... Where the string is split on spaces and joined with the AND clauses. For each individual part of the search phrase, I take care of escaping special characters (except "*" since I am allowing them to use wildcards). For example, if they entered "foo bar!", I would generate this query: message:foo AND message:bar\! The problem is they are entering "battle-axe", causing me to generate this: message:battle\-axe But that ends up being the same as: (message:battle OR message:axe) I guess that is what I was not expecting. Because of this behavior, I have to know from my app point of view what tokens I should be splitting the original string on, so that I can join them back together with ANDs. But that means basically reimplementing the tokenizer on my end, does it not? There must be a better way? Like specifying I want those terms to be joined with ANDs instead? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/924a04d5-4163-41b5-a7e7-e3ca2982d078%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Query_string search containing a dash has unexpected results
Also interesting... if I run the query with explain=true, I see information
in the details about the "welcome" token, but there's no mention at all
about the "doesnotmatch" token. I guess it wouldn't mention it though,
since if it did, the document shouldn't match in the first place.
On Monday, November 10, 2014 2:45:05 PM UTC-8, Dave Reed wrote:
>
> Yes of course :) Here we go:
>
> {
>
>- valid: true
>- _shards: {
> - total: 1
> - successful: 1
> - failed: 0
>}
>- explanations: [
> - {
> - index: index_v1
> - valid: true
> - explanation: message:welcome message:doesnotmatch
> }
>]
>
> }
>
> It pasted a little weird but that's it.
>
>
>
> On Monday, November 10, 2014 2:25:33 PM UTC-8, Amish Asthana wrote:
>>
>> Can you run the validate query output. That will be helpful.
>> amish
>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/632d1e74-31a0-42f2-ad09-40e3030449d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Query_string search containing a dash has unexpected results
Yes of course :) Here we go:
{
- valid: true
- _shards: {
- total: 1
- successful: 1
- failed: 0
}
- explanations: [
- {
- index: index_v1
- valid: true
- explanation: message:welcome message:doesnotmatch
}
]
}
It pasted a little weird but that's it.
On Monday, November 10, 2014 2:25:33 PM UTC-8, Amish Asthana wrote:
>
> Can you run the validate query output. That will be helpful.
> amish
>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/83422fed-2e1c-4e27-825e-5bd9f334f85a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Query_string search containing a dash has unexpected results
I'm not using the standard analyzer, I'm using a pattern that will break
the text on all non-word characters, like this:
"analyzer": {
"letterordigit": {
"type": "pattern",
"pattern": "[^\\p{L}\\p{N}]+"
}
}
I have verified that the message field is being broke up into the tokens I
expect (example in my first post).
So when I run a search for message:welcome-doesnotmatch, I'm expecting that
string to be broken into tokens like so:
welcome
doesnotmatch
And for the search to therefore find 0 documents. But it doesn't -- it
finds 1 document, the document that contains my sample message, which does
not include the token "doesnotmatch".
So why on Earth would this search match that document? It is behaving as if
everything after the "-" is completely ignored. It does not matter what I
put there, it will still match the document.
This is coming up because an end user is searching for a hyphenated word,
like "battle-axe", and it's matching a document that does not contain the
word "axe" at all.
On Friday, November 7, 2014 12:24:30 AM UTC-8, Jun Ohtani wrote:
>
> Hi Dave,
>
> I think the reason is your "message" field using "standard analyzer".
> Standard analyzer divide text by "-".
> If you change analyzer to whitespace analyzer, it matches 0 documents.
>
> _validate API is useful for checking exact query.
> Example request:
>
> curl -XGET "/YOUR_INDEX/_validate/query?explain" -d'
> {
> "query": {
> "query_string": {
> "query": "id:3955974 AND message:welcome-doesnotmatchanything"
> }
> }
> }'
>
> You can get the following response. In this example, "message" field is
> "index": "not_analyzed".
> {
>"valid": true,
>"_shards": {
> "total": 1,
> "successful": 1,
> "failed": 0
>},
>"explanations": [
> {
> "index": "YOUR_INDEX,
> "valid": true,
> "explanation": "+id:3955974 +message:welcome-doesnotmatchanything"
> }
>]
> }
>
>
> See:
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-validate.html#search-validate
>
> I hope that those help you out.
>
> Regards,
> Jun
>
>
> 2014-11-07 9:47 GMT+09:00 Dave Reed >:
>
>> I have a document with a field "message", that contains the following
>> text (truncated):
>>
>> Welcome to test.com!
>>
>> The assertion field is mapped to have an analyzer that breaks that string
>> into the following tokens:
>>
>> welcome
>> to
>> test
>> com
>>
>> But, when I search with a query like this:
>>
>> {
>> "query": {
>>
>> "query_string": {
>> "query": "id:3955974 AND message:welcome-doesnotmatchanything"
>> }
>> }
>> }
>>
>>
>>
>> To my surprise, it finds the document (3955974 is the document id). The
>> dash and everything after it seems to be ignored, because it does not
>> matter what I put there, it will still match the document.
>>
>> I've tried escaping it:
>>
>> {
>> "query": {
>> "query_string": {
>> "query": "id:3955974 AND message:welcome\\-doesnotmatchanything"
>> }
>> }
>> }
>> (note the double escape since it has to be escaped for the JSON too)
>>
>> But that makes no difference. I still get 1 matching document. If I put
>> it in quotes it works:
>>
>> {
>> "query": {
>> "query_string": {
>> "query": "id:3955974 AND message:\"welcome-doesnotmatchanything\""
>> }
>> }
>> }
>>
>> It works, meaning it matches 0 documents, since that document does not
>> contain the "doesnotmatchanything" token. That's great, but I don't
>> understand why the unquoted version does not work. This query is being
>> generated so I can't easily just decide to start quoting it, and I can't
>> always do that anyway since the user is sometimes going to use wildcards,
>> which can't be quoted if I want them to function. I was under the
>> assumption that an EscapedUnquotedString is the same as a quoted unespaced
>> string (in other words, foo:a\b\c
Query_string search containing a dash has unexpected results
I have a document with a field "message", that contains the following text
(truncated):
Welcome to test.com!
The assertion field is mapped to have an analyzer that breaks that string
into the following tokens:
welcome
to
test
com
But, when I search with a query like this:
{
"query": {
"query_string": {
"query": "id:3955974 AND message:welcome-doesnotmatchanything"
}
}
}
To my surprise, it finds the document (3955974 is the document id). The
dash and everything after it seems to be ignored, because it does not
matter what I put there, it will still match the document.
I've tried escaping it:
{
"query": {
"query_string": {
"query": "id:3955974 AND message:welcome\\-doesnotmatchanything"
}
}
}
(note the double escape since it has to be escaped for the JSON too)
But that makes no difference. I still get 1 matching document. If I put it
in quotes it works:
{
"query": {
"query_string": {
"query": "id:3955974 AND message:\"welcome-doesnotmatchanything\""
}
}
}
It works, meaning it matches 0 documents, since that document does not
contain the "doesnotmatchanything" token. That's great, but I don't
understand why the unquoted version does not work. This query is being
generated so I can't easily just decide to start quoting it, and I can't
always do that anyway since the user is sometimes going to use wildcards,
which can't be quoted if I want them to function. I was under the
assumption that an EscapedUnquotedString is the same as a quoted unespaced
string (in other words, foo:a\b\c === foo:"abc", assuming all special
characters are escaped in the unquoted version).
I'm only on ES 1.01, but I don't see anything new or changes that would
have impacted this behavior in later versions.
Any insights would be helpful! :)
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1dbfa1d5-7301-460b-ae9c-3665cfa79c96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
