[ https://issues.apache.org/jira/browse/METRON-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386861#comment-16386861 ]
ASF GitHub Bot commented on METRON-1460: ---------------------------------------- Github user ben-manes commented on the issue: https://github.com/apache/metron/pull/940 Guava defaults to a `concurrencyLevel` of 4, given its age and a desire to not abuse memory in low concurrent situations. You probably want to increase it to 64 in a heavy workload, which has a ~4x throughput gain on reads. It won't scale much higher, since it has internal bottlenecks and I could never get patches reviewed to fix those. I've only noticed overall throughput be based on threads, and never realized there was a capacity constraint to its performance. One should expect some due to the older hash table design resulting in more collisions, whereas CHMv8 does much better there. Still, I would have expected it to even out enough unless have a bad hash code? > Create a complementary non-split-join enrichment topology > --------------------------------------------------------- > > Key: METRON-1460 > URL: https://issues.apache.org/jira/browse/METRON-1460 > Project: Metron > Issue Type: New Feature > Reporter: Casey Stella > Priority: Major > > There are some deficiencies to the split/join topology. > * It's hard to reason about > * Understanding the latency of enriching a message requires looking at > multiple bolts that each give summary statistics > * The join bolt's cache is really hard to reason about when performance > tuning > * During spikes in traffic, you can overload the join bolt's cache and drop > messages if you aren't careful > * In general, it's hard to associate a cache size and a duration kept in > cache with throughput and latency > * There are a lot of network hops per message > * Right now we are stuck at 2 stages of transformations being done > (enrichment and threat intel). It's very possible that you might want > stellar enrichments to depend on the output of other stellar enrichments. In > order to implement this in split/join you'd have to create a cycle in the > storm topology > > I propose that we move to a model where we do enrichments in a single bolt in > parallel using a static threadpool (e.g. multiple workers in the same process > would share the threadpool). IN all other ways, this would be backwards > compatible. A transparent drop-in for the existing enrichment topology. > There are some pros/cons about this too: > * Pro > * Easier to reason about from an individual message perspective > * Architecturally decoupled from Storm > * This sets us up if we want to consider other streaming technologies > * Fewer bolts > * spout -> enrichment bolt -> threatintel bolt -> output bolt > * Way fewer network hops per message > * currently 2n+1 where n is the number of enrichments used (if using stellar > subgroups, each subgroup is a hop) > * Easier to reason about from a performance perspective > * We trade cache size and eviction timeout for threadpool size > * We set ourselves up to have stellar subgroups with dependencies > * i.e. stellar subgroups that depend on the output of other subgroups > * If we do this, we can shrink the topology to just spout -> > enrichment/threat intel -> output > * Con > * We can no longer tune stellar enrichments independent from HBase > enrichments > * To be fair, with enrichments moving to stellar, this is the case in the > split/join approach too > * No idea about performance > What I propose is to submit a PR that will deliver an alternative, completely > backwards compatible topology for enrichment that you can use by adjusting > the start_enrichment_topology.sh script to use remote-unified.yaml instead of > remote.yaml. If we live with it for a while and have some good experiences > with it, maybe we can consider retiring the old enrichment topology. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)