Hi, As far as I know there are no plans to support other state backends with BroadcastState. I don't know about any particular technical limitation, it probably just hasn't been done. Also I don't know how much effort that would be. Probably it wouldn't be easy.
Timo, can you chip in how for example Table API/SQL is solving this problem? I'm pretty sure Tablie API is using broadcast joins after all? Best, Piotrek czw., 17 cze 2021 o 02:53 Rion Williams <rionmons...@gmail.com> napisał(a): > Hey Flink folks, > > I was discussing the use of the Broadcast Pattern with some colleagues > today for a potential enrichment use-case and noticed that it wasn’t > currently backed by RocksDB. This seems to indicate that it would be solely > limited to the memory allocated, which might not support a large enrichment > data set that our use case might run into (thousands of tenants with users > and various other entities to enrich by). > > Are there any plans to eventually add support for BroadcastState to be > backed by a non-memory source? Or perhaps some technical limitations that > might not make that possible? If the latter is true, is there a preferred > pattern for handling enrichment/lookups for a very large set of data that > may not be memory-bound? > > Any advice or thoughts would be welcome! > > Rion
