Hello Nadav, Is the database updated by some other data flow other that this topology? How are the database changes detected and the “cache” updated? The size of the cached data and update volume will also influence the design. Without knowing some crucial details of data, size, update frequency, natural partitioning, network speed, etc it is hard to give a general answer. But assuming that you are looking at a storm topology to itself serve as a cache provider and cache is small, one “possible” way to do this would be to have the updates hit all executors, but queries hit one of the “many” executors. Sent from Yahoo Mail for iPhone
On Sunday, August 7, 2022, 8:25 AM, Nadav Glickman <nadav.glickman@orchestra.group> wrote: Hi all, We're looking for a caching solution to cache data reads from the DB and have it available to the entire topology.We need the data to be updated and the same for all the bolts, so we can't have the same cache split among different executors.Ideally we can have some in-memory solution within Storm. We tried an enum, and singletons, but they aren't shared between executors. I know distributed caching DBs like memcached and Redis are viable options, but I'd really like to find a solution that won't require another machine and another piece of technology in our stack. Looking forward to your ideas! Thanks,Nadav
