Yes, it did when we restarted the node while trying to reproduce this problem. We also were able to access the data using the Scan search api after restarting the node.
However we have seen quite a few of the bulk update errors in our 20-node production cluster and have suffered data loss on other aliases (The alias filter being the user-id) as well. We think the data loss is because of this bulk update error. Is there a chance of losing data on shards when enough of these bulk updates happen concurrently on multiple aliases (users)? Thanks On Sun, Jun 22, 2014 at 1:10 PM, Boaz Leskes <b.les...@gmail.com> wrote: > If you restart the node it's on, it doesn't come back? > > > On Sun, Jun 22, 2014 at 10:01 PM, Rohit Jaiswal <rohit.jais...@gmail.com> > wrote: > >> Hi Boaz, >> Thanks for replying. After we get this error, the cluster >> health changes to Yellow with a replica shard in Unassigned state. Is there >> a specific way to recover that shard? We dont want to lose other data on >> that shard. >> >> Thanks, >> Rohit >> >> >> On Sun, Jun 22, 2014 at 12:50 PM, Boaz Leskes <b.les...@gmail.com> wrote: >> >>> Hi Rohit, >>> >>> This issue means update fails anyway, but it breaks the entire request. >>> You should indeed set the retry_on_conflict option to make the update >>> request succeed. PS - you should really upgrade - a lot has happened and >>> was fixed since 0.90.2 ... >>> >>> Cheers, >>> Boaz >>> >>> >>> On Monday, June 16, 2014 10:26:06 PM UTC+2, Rohit Jaiswal wrote: >>>> >>>> Hi Boaz, >>>> We are using 0.90.2 and run into this issue. As i >>>> understand, one option is to upgrade to 0.90.3. If we continue using 0.90.2 >>>> and use (increase) retry_on_conflict, we will not see the problem? Please >>>> clarify. >>>> >>>> Thanks, >>>> Rohit >>>> On Wednesday, August 7, 2013 9:39:56 AM UTC-7, Boaz Leskes wrote: >>>> >>>>> HI Eric, >>>>> >>>>> OK. Based on the gist you sent, i tracked down a problem at fixed it: >>>>> https://github.com/elasticsearch/elasticsearch/issues/3448 . Thanks!! >>>>> The fix is part of 0.90.3, so I'd recommend upgrading. This is a secondary >>>>> problem which occurs when two requests try to update the same document at >>>>> exactly the same time. One of them succeeds and the other fails with a >>>>> version conflict (that error was masked by the error you were seeing). You >>>>> can use (or increase) the retry_on_conflict parameter to make the failing >>>>> request try again. >>>>> >>>>> I'm still curious about your reporting of loosing replicas. Can you >>>>> elaborate more about what happens? Do you see anything in the logs? >>>>> >>>>> Cheers, >>>>> Boaz >>>>> >>>>> On Tuesday, August 6, 2013 5:09:26 AM UTC+2, Eric Sites wrote: >>>>>> >>>>>> Boaz, >>>>>> >>>>>> Sorry but I no longer have those logs, I upgraded to 0.90.2 from >>>>>> 0.90.0 and wiped the logs when I did. >>>>>> I did the upgrade to use the _bulk api for my update. >>>>>> >>>>>> Basically the "lang", "js" was not the issue. >>>>>> >>>>>> I was using different scripts with the same set of params and an >>>>>> upcert. The fix was to use a different param name for different scripts, >>>>>> about 10 unique scripts in total. >>>>>> >>>>>> I was losing replicated shards about every 10,000 to 30,000 updates, >>>>>> never the primary shard. >>>>>> >>>>>> I have 185 million + large json documents, with 100 shards in 1 index >>>>>> with 1 replication, so 200 shards total over 6 servers. Each shard is >>>>>> about >>>>>> 10.4 GB in size. >>>>>> About 2 TB of data, 1 TB primary, 1 TB replicated. >>>>>> >>>>>> Cheers, >>>>>> Eric Sites >>>>>> >>>>>> From: Boaz Leskes <b.le...@gmail.com> >>>>>> Reply-To: <elasti...@googlegroups.com> >>>>>> Date: Monday, August 5, 2013 5:38 PM >>>>>> To: <elasti...@googlegroups.com> >>>>>> Subject: Re: 0.90.2 _update or _bulk update causing >>>>>> NullPointerException in logs and I start losing shards >>>>>> >>>>>> Hi Eric, >>>>>> >>>>>> Glad to hear you solved it. It would be great if you can share the >>>>>> failed logs from the _update (non bulk call). A failed script shouldn't >>>>>> cause shards to drop so I would like to research it some more. >>>>>> >>>>>> Cheers, >>>>>> Boaz >>>>>> >>>>>> >>>>>> On Mon, Aug 5, 2013 at 6:40 PM, Eric Sites <eric_...@mac.com> wrote: >>>>>> >>>>>>> Boaz, >>>>>>> >>>>>>> I found and fixed the problem. >>>>>>> >>>>>>> I added the "lang", "js" to the update json, that was not needed >>>>>>> before in es 0.90.0. >>>>>>> I also changed the name of new_tracking to match the name of the >>>>>>> action in the params section. >>>>>>> So for example the script now looks like this: >>>>>>> >>>>>>> if (ctx._source['tracking'] != null) { >>>>>>> if (ctx._source.tracking['some_action'] != null) { >>>>>>> ctx._source.tracking.some_action += param1; >>>>>>> } else { >>>>>>> ctx._source.tracking['some_action'] = 1; >>>>>>> } >>>>>>> } else { >>>>>>> ctx._source.tracking = new_some_action; >>>>>>> } >>>>>>> >>>>>>> "params" : { "param1" : 1, "new_some_action" : { "some_action" : 1 } >>>>>>> } >>>>>>> >>>>>>> Cheers, >>>>>>> Eric Sites >>>>>>> >>>>>>> From: Boaz Leskes <b.le...@gmail.com> >>>>>>> Reply-To: <elasti...@googlegroups.com> >>>>>>> Date: Monday, August 5, 2013 10:35 AM >>>>>>> To: <elasti...@googlegroups.com> >>>>>>> Subject: Re: 0.90.2 _update or _bulk update causing >>>>>>> NullPointerException in logs and I start losing shards >>>>>>> >>>>>>> Hi Eric, >>>>>>> >>>>>>> This is interesting. The log stack trace from the gist comes from >>>>>>> the bulk calls. Can you also post one from a failed _update? Cross >>>>>>> checking >>>>>>> them might help pin pointing the issue. >>>>>>> >>>>>>> Cheers, >>>>>>> Boaz >>>>>>> >>>>>>> On Monday, August 5, 2013 1:34:16 AM UTC+2, eric_...@mac.com wrote: >>>>>>>> >>>>>>>> I am getting java.lang.NullPointerException exception in my >>>>>>>> ElasticSearch cluster logs when I am doing a _bulk update or just an >>>>>>>> _update. >>>>>>>> I am sending a lot of data to my clusters. After I get this error I >>>>>>>> lose a shard and it has to be recreated. >>>>>>>> >>>>>>>> version 0.90.2 >>>>>>>> >>>>>>>> gist: https://gist.github.com/EricSites/6152468 >>>>>>>> >>>>>>>> I get this using the _bulk api or just normal _update api. >>>>>>>> >>>>>>>> My update script is a little complicated. >>>>>>>> I am adding a tracking object to my document if it does not exists. >>>>>>>> There should only be one of these and it should not be an array of >>>>>>>> these. >>>>>>>> If the object does exists, I am trying to add a new field to the >>>>>>>> tracking object to keep track on counts. >>>>>>>> So if the field does not exists I create it, else just += to it. >>>>>>>> >>>>>>>> if (ctx._source['tracking'] != null) { >>>>>>>> if (ctx._source.tracking['some_action'] != null) { >>>>>>>> ctx._source.tracking.some_action += param1; >>>>>>>> } else { >>>>>>>> ctx._source.tracking['some_action'] = 1; >>>>>>>> } >>>>>>>> } else { >>>>>>>> ctx._source.tracking = new_tracking; >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> Here is my mapping for this: >>>>>>>> { >>>>>>>> "sample" : { >>>>>>>> "index_options" : "docs", >>>>>>>> "properties" : { >>>>>>>> "tracking" : { >>>>>>>> "type" : "object", >>>>>>>> "dynamic" : true >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "elasticsearch" group. >>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>>>> topic/elasticsearch/yk7HvjqCgOg/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> elasticsearc...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>>> >>>>>> >>>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/a22ffbaa-af7e-4d15-ac5a-e1dcd5b76976%40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/a22ffbaa-af7e-4d15-ac5a-e1dcd5b76976%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAP_rV8GKheAXK%3Dq%2BG2vdyfgRBURuk4_udO8XFLNCTmDV3EnWiA%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAP_rV8GKheAXK%3Dq%2BG2vdyfgRBURuk4_udO8XFLNCTmDV3EnWiA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAKzwz0oDUnYonpURtCVis-9UxS0FRiRMvLW1wZZybo2gOZboTA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAKzwz0oDUnYonpURtCVis-9UxS0FRiRMvLW1wZZybo2gOZboTA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAP_rV8GexzoN8Nrf3GBaCrrXrVdKjUzzrkqw%3DYLwTW9YwEst5A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
