Re: How do people typically handle shard failures in their results?
I see. Thanks. I wasnt expecting to have to check for shard failures. Would it make sense to have a config or request setting whereby any shard failures are reported as exceptions (java api). In the space where we have been using elastic, partial results are bad. We would end up having to always write code to turn shard failures into exceptions -Nick On Friday, 20 June 2014 13:26:35 UTC+1, kimchy wrote: Ahh, I see. If its related to searches, then yea, the search response includes details about the total shards that the search was executed on, the successful shards, and failed shards. They are important to check to understand if one gets partial results. In the REST API, if there is a total “failure”, then it will return the “worst” status code out of all the shards in the response. In the Java API, the search response will be returned (with no exception), so the content of the search has to be checked (which is a good practice anyhow). It might make sense to raise an exception in the Java API if all shards failed, I am on the fence on this one, since anyhow a check needs to be performed on the result. On Jun 20, 2014, at 13:22, Nikolas Everett nik...@gmail.com javascript: wrote: On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon kim...@gmail.com javascript: wrote: If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on “all” shards would help here? I think its more about searches and they can fail on one shard but not other for all sorts of reasons. Queue full, unfortunate script, bug, only one shard had results and the query asked for something weird like to use the postings highlighter when postings aren't stored. Lots of reasons. I log the event and move on. I toyed with outputting a warning to the user but didn't have time to implement it. We're pretty diligent with our logs so we'd notice the log and run it down. If the failure is caused by the queue being full only on one node, we'd likely notice that real quick as ganglia would lose it. This happened to me recently when we put a node without an ssd into a cluster with ssds. It couldn't keep up and dropped a ton of searches. In our defense, we didn't know the rest of the cluster had ssds so we were double surprised. Nik -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/63f3b5a8-02b2-4a2f-bb47-b42b46ff4d54%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: How do people typically handle shard failures in their results?
If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on all shards would help here? On Jun 18, 2014, at 11:45, mooky nick.minute...@gmail.com wrote: If I understand correctly, we can get an OK response from elastic (ie no error) but if there are shard failures in the response, it potentially means that results are incomplete/incorrect. From my observation, we can get failures on all shards - and elastic still returns OK (which was a bit surprising to me) What kinds of approaches to people typically use to deal with shard failures? For my application, if there are shard failures, essentially my results are inaccurate/incorrect - so I need to return an error to the client. Returning bad results is worse than returning an error. I am inclined to turn any shard failure into an exception. Is this quite common? Does it make sense to add a feature to the elastic api ? (ie request.setTreatShardFailuresAsErrors(true) -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/FFDC7083-24CB-484D-B337-65582596D555%40gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: How do people typically handle shard failures in their results?
On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon kim...@gmail.com wrote: If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on “all” shards would help here? I think its more about searches and they can fail on one shard but not other for all sorts of reasons. Queue full, unfortunate script, bug, only one shard had results and the query asked for something weird like to use the postings highlighter when postings aren't stored. Lots of reasons. I log the event and move on. I toyed with outputting a warning to the user but didn't have time to implement it. We're pretty diligent with our logs so we'd notice the log and run it down. If the failure is caused by the queue being full only on one node, we'd likely notice that real quick as ganglia would lose it. This happened to me recently when we put a node without an ssd into a cluster with ssds. It couldn't keep up and dropped a ton of searches. In our defense, we didn't know the rest of the cluster had ssds so we were double surprised. Nik -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: How do people typically handle shard failures in their results?
Ahh, I see. If its related to searches, then yea, the search response includes details about the total shards that the search was executed on, the successful shards, and failed shards. They are important to check to understand if one gets partial results. In the REST API, if there is a total failure, then it will return the worst status code out of all the shards in the response. In the Java API, the search response will be returned (with no exception), so the content of the search has to be checked (which is a good practice anyhow). It might make sense to raise an exception in the Java API if all shards failed, I am on the fence on this one, since anyhow a check needs to be performed on the result. On Jun 20, 2014, at 13:22, Nikolas Everett nik9...@gmail.com wrote: On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon kim...@gmail.com wrote: If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on all shards would help here? I think its more about searches and they can fail on one shard but not other for all sorts of reasons. Queue full, unfortunate script, bug, only one shard had results and the query asked for something weird like to use the postings highlighter when postings aren't stored. Lots of reasons. I log the event and move on. I toyed with outputting a warning to the user but didn't have time to implement it. We're pretty diligent with our logs so we'd notice the log and run it down. If the failure is caused by the queue being full only on one node, we'd likely notice that real quick as ganglia would lose it. This happened to me recently when we put a node without an ssd into a cluster with ssds. It couldn't keep up and dropped a ton of searches. In our defense, we didn't know the rest of the cluster had ssds so we were double surprised. Nik -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9A9246D5-338B-4B93-B2FD-4D3B93F621F2%40gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: How do people typically handle shard failures in their results?
Every man and his own business requirements and SLAs. Many scenarios I've seen would consider two-thirds of shards responding as an OK scenario and not even show a notice about results being partial. But of course it makes sense to throw an exception if your business requires / expects that. FWIW, this is usually why you use replicas, so even if a shard goes down there's a back-up shard (ideally more than one) that you can fallback to. -- Itamar Syn-Hershko http://code972.com | @synhershko https://twitter.com/synhershko Freelance Developer Consultant Author of RavenDB in Action http://manning.com/synhershko/ On Wed, Jun 18, 2014 at 12:45 PM, mooky nick.minute...@gmail.com wrote: If I understand correctly, we can get an OK response from elastic (ie no error) but if there are shard failures in the response, it potentially means that results are incomplete/incorrect. From my observation, we can get failures on all shards - and elastic still returns OK (which was a bit surprising to me) What kinds of approaches to people typically use to deal with shard failures? For my application, if there are shard failures, essentially my results are inaccurate/incorrect - so I need to return an error to the client. Returning bad results is worse than returning an error. I am inclined to turn any shard failure into an exception. Is this quite common? Does it make sense to add a feature to the elastic api ? (ie request.setTreatShardFailuresAsErrors(true) -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu7ECzf7bA_YG0XoGaVrPr1CUyKzhq1VRn4XD47CH%2B0MQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
