Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/453
In response to "Is there any reason we didn't just use the normal indexing
topology". Here are the issues I see with doing that. First, I think we
should be careful about putting additional load on the indexing topology. For
parser errors I think it's a wash, an error gets indexed with the original
message instead of the parsed message. But for enrichment errors, new error
messages are generated and passed along. If we send all errors and messages to
the same indexing topology a burst of errors could affect the latency of other
messages.
Here is another scenario that could cause problems. Imagine a message
fails to index and an error is thrown in the indexing topology. If we send all
errors and messages to the same topology, that error could be replayed in the
indexing topology over and over. We could add some logic to keep that from
happening but now writer bolts are more complex and specific to the indexing
topology. You will notice the error indexing topology is different in that it
does not write back out to Kafka.
I feel like by mixing error messages and legitimate messages we are asking
for trouble. Having them separate allows us to tune the topologies separately
and avoid the problems I mention above. Tuning the indexing topology as it
stands now is challenging enough. I can understand not wanting to allocate
additional worker slots though. Maybe I am being too conservative.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
