Hey Flink folks, I was discussing the use of the Broadcast Pattern with some colleagues today for a potential enrichment use-case and noticed that it wasn’t currently backed by RocksDB. This seems to indicate that it would be solely limited to the memory allocated, which might not support a large enrichment data set that our use case might run into (thousands of tenants with users and various other entities to enrich by).
Are there any plans to eventually add support for BroadcastState to be backed by a non-memory source? Or perhaps some technical limitations that might not make that possible? If the latter is true, is there a preferred pattern for handling enrichment/lookups for a very large set of data that may not be memory-bound? Any advice or thoughts would be welcome! Rion
