Would either of you be able to write up the steps to reproduce this and to open an issue about it?
thanks On 17 March 2014 17:58, Mac Jouz <mac.j...@gmail.com> wrote: > > Same symptom for me, neither OOM, nor full disk space, only an ES > restart... > > Le lundi 17 mars 2014 14:00:11 UTC+1, bizzorama a écrit : >> >> No, when things are running everything is ok, indexes break during >> restart/powerdown >> 17-03-2014 13:11, "Clinton Gormley" <cl...@traveljury.com> napisał(a): >> >>> Are you sure you didn't run out of disk space or file handles at some >>> stage, or have an OOM exception? >>> >>> >>> On 16 March 2014 16:37, bizzorama <bizz...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> it turned out that it was not a problem of ES version (we tested on >>>> both 0.90.10 and 0.90.9) but just a ES bug ... >>>> after restarting pc or even just the service indices got broken ... we >>>> found out that this was the case of missing mappings. >>>> We observed that broken indices had their mappings corrupted (only some >>>> default fields were observed). >>>> You can check this by calling: http:\\es_address:9200\ >>>> indexName\_mapping >>>> >>>> Our mappings were dynamic (not set manually - just figured out by ES >>>> when the records were incoming). >>>> >>>> The solution was to add a static mapping file like the one described >>>> here: >>>> http://www.elasticsearch.org/guide/en/elasticsearch/ >>>> reference/current/mapping-conf-mappings.html (we added the default >>>> one). >>>> >>>> I just copied mappings from a healty index, made some changes, turned >>>> it to a mapping file and copied to the ES server. >>>> >>>> Now everything works just fine. >>>> >>>> Regards, >>>> Karol >>>> >>>> >>>> W dniu niedziela, 16 marca 2014 14:54:00 UTC+1 użytkownik Mac Jouz >>>> napisał: >>>> >>>>> >>>>> Hi Bizzorama, >>>>> >>>>> I had a similar problem with the same configuration than you gave. >>>>> ES ran since the 11th of February and was fed every day at 6:00 AM by >>>>> 2 LS. >>>>> Everything worked well (kibana reports were correct and no data loss) >>>>> until >>>>> I restarted yesterday ES :-( >>>>> Among 30 index (1 per day), 4 were unusable and data within kibana >>>>> report >>>>> for the related period were unavailable (same org.elasticsearch.search. >>>>> facet.FacetPhaseExecutionException: Facet[0]: (key) field >>>>> [@timestamp] not found) >>>>> >>>>> Do you confirm when you downgraded ES to 0.90.9 that you retrieved >>>>> your data >>>>> (i.e you was able to show your data in kibana reports) ? >>>>> >>>>> I will try to downgrade ES version as you suggested and will let you >>>>> know >>>>> more >>>>> >>>>> Thanks for your answer >>>>> >>>>> >>>>> >>>>> Sorry for the delay. >>>>> >>>>> Looks like you were right, after downgrading ES to 0.90.9 i couldn't >>>>> reproduce the issue in such manner. >>>>> >>>>> Unfortunately, I found some other problems, and one looks like a >>>>> blocker .... >>>>> >>>>> After whole ES cluster powerdown, ES just started replaying 'no >>>>> mapping for ... <name of field>' for each request. >>>>> >>>>> W dniu czwartek, 20 lutego 2014 16:42:20 UTC+1 użytkownik Binh Ly >>>>> napisał: >>>>>> >>>>>> Your error logs seem to indicate some kind of version mismatch. Is it >>>>>> possible for you to test LS 1.3.2 against ES 0.90.9 and take a sample of >>>>>> raw logs from those 3 days and test them through to see if those 3 days >>>>>> work in Kibana? The reason I ask is because LS 1.3.2 (specifically the >>>>>> elasticsearch output) was built using the binaries from ES 0.90.9. >>>>>> >>>>>> Thanks. >>>>>> >>>>> >>>>> Le mardi 11 février 2014 13:18:01 UTC+1, bizzorama a écrit : >>>>>> >>>>>> Hi, >>>>>> >>>>>> I've noticed a very disturbing ElasticSearch behaviour ... >>>>>> my environment is: >>>>>> >>>>>> 1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch >>>>>> (0.90.10) + kibana >>>>>> >>>>>> which process about 7 000 000 records per day, >>>>>> everything worked fine on our test environment, untill we run some >>>>>> tests for a longer period (about 15 days). >>>>>> >>>>>> After that time, kibana was unable to show any data. >>>>>> I did some investigation and it looks like some of the indexes (for 3 >>>>>> days to be exact) seem to be corrupted. >>>>>> Now every query from kibana, using those corrupted indexes - failes. >>>>>> >>>>>> Errors read from elasticsearch logs: >>>>>> >>>>>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: >>>>>> Facet[terms]: failed to find mapping for Name* ... a couple of other >>>>>> columns* >>>>>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet >>>>>> [0]: (key) field [@timestamp] not found >>>>>> >>>>>> ... generaly all queries end with those errors >>>>>> >>>>>> When elasticsearch is started we find something like this: >>>>>> >>>>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>>>> Message not fully read (request) for [243445] and action >>>>>> [cluster/nodeIndexCreated], resetting >>>>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>>>> Message not fully read (request) for [249943] and action >>>>>> [cluster/nodeIndexCreated], resetting >>>>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>>>> Message not fully read (request) for [246740] and action >>>>>> [cluster/nodeIndexCreated], resetting >>>>>> >>>>>> And a little observations: >>>>>> >>>>>> 1. When using elasticsearch-head plugin, when querying records >>>>>> 'manually', i can see only elasticsearch columns (_index, _type, _id, >>>>>> _score). >>>>>> But when I 'randomly' select columns and overview their raw json >>>>>> they look ok. >>>>>> >>>>>> 2, When I tried to process same data again - everything is ok >>>>>> >>>>>> Is it possible that some corrupted data found its way to >>>>>> elasticsearch and now whole index is broken ? >>>>>> Can this be fixed ? reindexed or sth ? >>>>>> This data is very importand and can't be lost ... >>>>>> >>>>>> Best Regards, >>>>>> Karol >>>>>> >>>>>> >>>>> Le mardi 11 février 2014 13:18:01 UTC+1, bizzorama a écrit : >>>>>> >>>>>> Hi, >>>>>> >>>>>> I've noticed a very disturbing ElasticSearch behaviour ... >>>>>> my environment is: >>>>>> >>>>>> 1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch >>>>>> (0.90.10) + kibana >>>>>> >>>>>> which process about 7 000 000 records per day, >>>>>> everything worked fine on our test environment, untill we run some >>>>>> tests for a longer period (about 15 days). >>>>>> >>>>>> After that time, kibana was unable to show any data. >>>>>> I did some investigation and it looks like some of the indexes (for 3 >>>>>> days to be exact) seem to be corrupted. >>>>>> Now every query from kibana, using those corrupted indexes - failes. >>>>>> >>>>>> Errors read from elasticsearch logs: >>>>>> >>>>>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: >>>>>> Facet[terms]: failed to find mapping for Name* ... a couple of other >>>>>> columns* >>>>>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet >>>>>> [0]: (key) field [@timestamp] not found >>>>>> >>>>>> ... generaly all queries end with those errors >>>>>> >>>>>> When elasticsearch is started we find something like this: >>>>>> >>>>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>>>> Message not fully read (request) for [243445] and action >>>>>> [cluster/nodeIndexCreated], resetting >>>>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>>>> Message not fully read (request) for [249943] and action >>>>>> [cluster/nodeIndexCreated], resetting >>>>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>>>> Message not fully read (request) for [246740] and action >>>>>> [cluster/nodeIndexCreated], resetting >>>>>> >>>>>> And a little observations: >>>>>> >>>>>> 1. When using elasticsearch-head plugin, when querying records >>>>>> 'manually', i can see only elasticsearch columns (_index, _type, _id, >>>>>> _score). >>>>>> But when I 'randomly' select columns and overview their raw json >>>>>> they look ok. >>>>>> >>>>>> 2, When I tried to process same data again - everything is ok >>>>>> >>>>>> Is it possible that some corrupted data found its way to >>>>>> elasticsearch and now whole index is broken ? >>>>>> Can this be fixed ? reindexed or sth ? >>>>>> This data is very importand and can't be lost ... >>>>>> >>>>>> Best Regards, >>>>>> Karol >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearc...@googlegroups.com. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/2f25e8f8-7c62-4474-a7a4-ee64a433eeca% >>>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/2f25e8f8-7c62-4474-a7a4-ee64a433eeca%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>> topic/elasticsearch/7ZwB6SNFkDc/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> elasticsearc...@googlegroups.com. >>> >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/CAPt3XKSE6%3DX-QjwJYzs0Aq7FXOHbE6Vu8Am- >>> 5BGvznmAsbFikw%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAPt3XKSE6%3DX-QjwJYzs0Aq7FXOHbE6Vu8Am-5BGvznmAsbFikw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/22a9d9a3-d7cf-4b87-8244-b027387005f4%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/22a9d9a3-d7cf-4b87-8244-b027387005f4%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKSCXwhUkXJt-XhiYvCkT53kWSO8x%2BPc75HK9wi9CYTpSw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
