We already have a capability to capture bolt errors and validation errors and pipe them into a Kafka topic. I want to propose that we attach a writer topology to the error and validation failed kafka topics so that we can (a) create a new ES index for these errors and (b) create a new Kibana dashboard to visualize them. The benefit would be that errors and validation failures would be easier to see and analyze.
I am seeking feedback on the following: - How granular would we want this feature to be? Think we would want one index/dashboard per source? Or would it be better to collapse everything into the same index? - Do we care about storing these errors in HDFS as well? Or is indexing them enough? - What types of errors should we record? I am proposing: For error reporting: --Message failed to parse --Enrichment failed to enrich --Threat intel feed failures --Generic catch-all for all other errors For validation reporting: --What part of message failed validation --What stellar validator caused the failure ------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
