Re: redis sink from flink

2021-08-17 Thread Jin Yi
great, thanks for the pointers everyone. i'm going to pursue rolling my own built around lettuce since it seems more feature-full wrt async semantics. On Mon, Aug 16, 2021 at 7:21 PM Yik San Chan wrote: > By the way, this post in Chinese showed how we do it exactly with code. > > https://yiksan

Re: redis sink from flink

2021-08-16 Thread Yik San Chan
By the way, this post in Chinese showed how we do it exactly with code. https://yiksanchan.com/posts/flink-bulk-insert-redis And yes it had buffered writes support by leveraging Flink operator state, and Redis Pipelining. Feel free to let you know if you have any questions. On Tue, Aug 17, 2021

Re: redis sink from flink

2021-08-16 Thread Yik San Chan
Hi Jin, I was in the same shoes. I tried bahir redis connector at first, then I felt it was very limited, so I rolled out my own. It was actually quite straightforward. All you need to do is to extend RichSinkFunction, then put your logic inside. Regarding Redis clients, Jedis (https://github.com

Re: redis sink from flink

2021-08-16 Thread Yangze Guo
Hi, Jin IIUC, the DataStream connector `RedisSink` can still be used. However, the Table API connector `RedisTableSink` might not work (at least in the future) because it is implemented based on the deprecated Table connector abstraction. You can still give it a try, though. Best, Yangze Guo On

redis sink from flink

2021-08-16 Thread Jin Yi
is apache bahir still a thing? it hasn't been touched for months (since redis 2.8.5). as such, looking at the current flink connector docs, it's no longer pointing to anything from the bahir project. looking around in either the flink or bahir newsgroups doesn't turn up anything regarding bahir'