Github user arunmahadevan commented on the issue:
https://github.com/apache/metron/pull/940
Managing threadpools within a bolt isn't fundamentally wrong, we have see
some use cases where this is done. However, we have been putting efforts to
reduce the overall number of threads created internally within storm since the
thread context switches were causing performance bottlenecks. I assume the
threadpool threads are mostly IO/network bound so it should not cause too much
harm.
Do you need multiple threads since the enrichments involve external DB look
ups and are time consuming ? Maybe you could compare the performance of
maintaining a thread pool v/s increasing the bolt's parallelism to achieve a
similar effect.
Another option might be to prefetch the enrichment data and load it into
each bolt so that you might not need separate threads to do the enrichment.
If you are able to manage without threads, that would be preferable. Even
otherwise its not that bad as long as you don't create too many threads and
they are cleaned up properly. (we have had some cases were the internal threads
were causing workers to hang).
---
