Are you sure it is a cache miss problem? It could be that the HBASE enrichment component could just be optimized. It would be nice if our enrichment cache mechanism reported hit/miss stats or something.
On June 19, 2017 at 09:58:25, Ali Nazemian (alinazem...@gmail.com) wrote: I have already increased the cache value. However, clearly, the HBase enrichment is a bottleneck for the entire enrichment topology. The average latency for that part is about 30-40 ms, but the latency for other parts in total is less than 0.5 ms. On Mon, Jun 19, 2017 at 11:45 PM, Ryan Merriman <merrim...@gmail.com> wrote: > We already front HBase enrichments with a cache (actually all enrichments > have a cache in front, see the GenericEnrichmentBolt class). You probably > just need to increase the cache size. This setting is in the enrichment > flux file at $METRON_HOME/flux/enrichment/remote.yaml. Look for the > "withMaxCacheSize" setting under the appropriate bolt. > > Ryan > > On Mon, Jun 19, 2017 at 8:31 AM, Casey Stella <ceste...@gmail.com> wrote: > >> The stellar enrichments do cache results. It caches at the bolt level, >> so it will associate an input message with computed output. >> >> On Mon, Jun 19, 2017 at 6:28 AM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> Surely the caching should make this effectively an in memory lookup. >>> Does the stellar enrichment function not use the same clientside caching as >>> the Hbase bolt? >>> >>> Simon >>> >>> On 19 Jun 2017, at 06:21, Casey Stella <ceste...@gmail.com> wrote: >>> >>> In order to do that, the easiest thing to do is to create a stellar >>> function to load and do in-memory lookups. >>> >>> On Sun, Jun 18, 2017 at 11:48 PM, Ali Nazemian <alinazem...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> We are using Metron HBase enrichment for a few use cases, but we have >>>> noticed the achievable throughput is not very great. I was wondering >>>> whether there is a way to load the external enrichment data in-memory and >>>> use it with normal Stellar enrichments. In our use cases, the number of >>>> rows in the external enrichments that we are dealing with is less than a >>>> 100k and it is a static list, so it is feasible to load them in-memory and >>>> use that for the enrichment. However, I am not sure how that would be >>>> achievable from the Metron capabilities. >>>> >>>> Regards, >>>> Ali >>>> >>> >>> >> > -- A.Nazemian
