[ https://issues.apache.org/jira/browse/METRON-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Allen updated METRON-1832: ------------------------------- Attachment: error-message.json > Recurrent Large Indexing Error Messages > --------------------------------------- > > Key: METRON-1832 > URL: https://issues.apache.org/jira/browse/METRON-1832 > Project: Metron > Issue Type: Bug > Reporter: Nick Allen > Priority: Major > Attachments: error-message.json > > > If any index destination like HDFS, Elasticsearch, or Solr goes down while > the Indexing topology is running, an error message is created and sent back > to the user-defined error topic. By default, this is defined to also be the > 'indexing' topic. > The Indexing topology then consumes this error message and attempts to write > it again. If the index destination is still down, another error occurs and > another error message is created that encapsulates the original error > message. That message is then sent to the 'indexing' topic, which is later > consumed, yet again, by the Indexing topology. > These error messages will continue to be recycled and grow larger and larger > as each new error message encapsulates all previous error messages in the > "raw_message" field. > Once the index destination recovers, one giant error message will finally be > written that contains massively duplicated, useless information which can > further negatively impact performance of the index destination. > See attachment for example of once of these error messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)