Thanks Zakelly and Junrui.
I was actually exploring RocksDB as a state backend and I thought maybe Redis
could offer more features as a state backend. For e.g. maybe state sharing
between operators, geo-red of state, partitioning etc. I understand these are
not native use cases for Flink, but maybe something that can be considered in
future. Maybe even as an off the shelf state backend framework which allows
embedding any other cache as a state backend.
The links you shared are useful and will really help me. Really appreciate it.
Thanks
On Tuesday, 30 January, 2024 at 01:43:14 pm IST, Zakelly Lan
<[email protected]> wrote:
And I found some previous discussion, FYI:1.
https://issues.apache.org/jira/browse/FLINK-3035
2. https://www.mail-archive.com/[email protected]/msg10666.html
Hope this helps.
Best,Zakelly
On Tue, Jan 30, 2024 at 4:08 PM Zakelly Lan <[email protected]> wrote:
Hi Chirag
That's an interesting idea. IIUC, storing key-values can be simply implemented
for Redis, but supporting checkpoint and recovery is relatively challenging.
Flink's checkpoint should be consistent among all stateful operators at the
same time. For an embedded and file-based key value store like RocksDB, it is
easier to implement by uploading files of specific time asynchronously.
Moreover if you want to store your state basically in memory, then why not
using the HashMapStateBackend. It saves the overhead of serialization and
deserialization and may achieve better performance compared with Redis I guess.
Best,Zakelly
On Tue, Jan 30, 2024 at 2:15 PM Chirag Dewan via user <[email protected]>
wrote:
Hi,
I was looking at the FLIP-254: Redis Streams Connector and I was wondering if
Flink ever considered Redis as a state backend? And if yes, why was it
discarded compared to RocksDB?
If someone can point me towards any deep dives on why RocksDB is a better fit
as a state backend, it would be helpful.
Thanks,Chirag
