I need to check it in a long run. Perhaps my test scenario was not long enough to hit the cache frequently.
On Tue, Jun 20, 2017 at 12:21 AM, Ryan Merriman <merrim...@gmail.com> wrote: > "It would be nice if our enrichment cache mechanism reported hit/miss > stats or something." > > +1 to that. We do this in the JoinBolt already so it's an easy thing to > add. > > On Mon, Jun 19, 2017 at 9:04 AM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> Are you sure it is a cache miss problem? It could be that the HBASE >> enrichment component could just be optimized. >> It would be nice if our enrichment cache mechanism reported hit/miss >> stats or something. >> >> >> On June 19, 2017 at 09:58:25, Ali Nazemian (alinazem...@gmail.com) wrote: >> >> I have already increased the cache value. However, clearly, the HBase >> enrichment is a bottleneck for the entire enrichment topology. The average >> latency for that part is about 30-40 ms, but the latency for other parts in >> total is less than 0.5 ms. >> >> On Mon, Jun 19, 2017 at 11:45 PM, Ryan Merriman <merrim...@gmail.com> >> wrote: >> >>> We already front HBase enrichments with a cache (actually all >>> enrichments have a cache in front, see the GenericEnrichmentBolt class). >>> You probably just need to increase the cache size. This setting is in the >>> enrichment flux file at $METRON_HOME/flux/enrichment/remote.yaml. Look >>> for the "withMaxCacheSize" setting under the appropriate bolt. >>> >>> Ryan >>> >>> On Mon, Jun 19, 2017 at 8:31 AM, Casey Stella <ceste...@gmail.com> >>> wrote: >>> >>>> The stellar enrichments do cache results. It caches at the bolt level, >>>> so it will associate an input message with computed output. >>>> >>>> On Mon, Jun 19, 2017 at 6:28 AM, Simon Elliston Ball < >>>> si...@simonellistonball.com> wrote: >>>> >>>>> Surely the caching should make this effectively an in memory lookup. >>>>> Does the stellar enrichment function not use the same clientside caching >>>>> as >>>>> the Hbase bolt? >>>>> >>>>> Simon >>>>> >>>>> On 19 Jun 2017, at 06:21, Casey Stella <ceste...@gmail.com> wrote: >>>>> >>>>> In order to do that, the easiest thing to do is to create a stellar >>>>> function to load and do in-memory lookups. >>>>> >>>>> On Sun, Jun 18, 2017 at 11:48 PM, Ali Nazemian <alinazem...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We are using Metron HBase enrichment for a few use cases, but we have >>>>>> noticed the achievable throughput is not very great. I was wondering >>>>>> whether there is a way to load the external enrichment data in-memory and >>>>>> use it with normal Stellar enrichments. In our use cases, the number of >>>>>> rows in the external enrichments that we are dealing with is less than a >>>>>> 100k and it is a static list, so it is feasible to load them in-memory >>>>>> and >>>>>> use that for the enrichment. However, I am not sure how that would be >>>>>> achievable from the Metron capabilities. >>>>>> >>>>>> Regards, >>>>>> Ali >>>>>> >>>>> >>>>> >>>> >>> >> >> >> -- >> A.Nazemian >> >> > -- A.Nazemian
