Re: redis sink from flink
great, thanks for the pointers everyone. i'm going to pursue rolling my own built around lettuce since it seems more feature-full wrt async semantics. On Mon, Aug 16, 2021 at 7:21 PM Yik San Chan wrote: > By the way, this post in Chinese showed how we do it exactly with code. > > https://yiksanchan.com/posts/flink-bulk-insert-redis > > And yes it had buffered writes support by leveraging Flink operator state, > and Redis Pipelining. Feel free to let you know if you have any questions. > > > On Tue, Aug 17, 2021 at 10:15 AM Yik San Chan > wrote: > >> Hi Jin, >> >> I was in the same shoes. I tried bahir redis connector at first, then I >> felt it was very limited, so I rolled out my own. It was actually quite >> straightforward. >> >> All you need to do is to extend RichSinkFunction, then put your logic >> inside. Regarding Redis clients, Jedis (https://github.com/redis/jedis) >> is quite popular and simple to get started. >> >> Let me know if you love to learn more details about our implementation. >> >> Best, >> Yik San >> >> On Tue, Aug 17, 2021 at 9:15 AM Jin Yi wrote: >> >>> is apache bahir still a thing? it hasn't been touched for months (since >>> redis 2.8.5). >>> >>> as such, looking at the current flink connector docs, it's no longer >>> pointing to anything from the bahir project. looking around in either the >>> flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL. >>> >>> is the best bet for a flink to redis sink something i roll on my own >>> (inclined to go this route w/ buffered writes)? or should i try going >>> through via kafka and using confluent's kafka redis connector (flink => >>> kafka => redis)? >>> >>
Re: redis sink from flink
By the way, this post in Chinese showed how we do it exactly with code. https://yiksanchan.com/posts/flink-bulk-insert-redis And yes it had buffered writes support by leveraging Flink operator state, and Redis Pipelining. Feel free to let you know if you have any questions. On Tue, Aug 17, 2021 at 10:15 AM Yik San Chan wrote: > Hi Jin, > > I was in the same shoes. I tried bahir redis connector at first, then I > felt it was very limited, so I rolled out my own. It was actually quite > straightforward. > > All you need to do is to extend RichSinkFunction, then put your logic > inside. Regarding Redis clients, Jedis (https://github.com/redis/jedis) > is quite popular and simple to get started. > > Let me know if you love to learn more details about our implementation. > > Best, > Yik San > > On Tue, Aug 17, 2021 at 9:15 AM Jin Yi wrote: > >> is apache bahir still a thing? it hasn't been touched for months (since >> redis 2.8.5). >> >> as such, looking at the current flink connector docs, it's no longer >> pointing to anything from the bahir project. looking around in either the >> flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL. >> >> is the best bet for a flink to redis sink something i roll on my own >> (inclined to go this route w/ buffered writes)? or should i try going >> through via kafka and using confluent's kafka redis connector (flink => >> kafka => redis)? >> >
Re: redis sink from flink
Hi Jin, I was in the same shoes. I tried bahir redis connector at first, then I felt it was very limited, so I rolled out my own. It was actually quite straightforward. All you need to do is to extend RichSinkFunction, then put your logic inside. Regarding Redis clients, Jedis (https://github.com/redis/jedis) is quite popular and simple to get started. Let me know if you love to learn more details about our implementation. Best, Yik San On Tue, Aug 17, 2021 at 9:15 AM Jin Yi wrote: > is apache bahir still a thing? it hasn't been touched for months (since > redis 2.8.5). > > as such, looking at the current flink connector docs, it's no longer > pointing to anything from the bahir project. looking around in either the > flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL. > > is the best bet for a flink to redis sink something i roll on my own > (inclined to go this route w/ buffered writes)? or should i try going > through via kafka and using confluent's kafka redis connector (flink => > kafka => redis)? >
Re: redis sink from flink
Hi, Jin IIUC, the DataStream connector `RedisSink` can still be used. However, the Table API connector `RedisTableSink` might not work (at least in the future) because it is implemented based on the deprecated Table connector abstraction. You can still give it a try, though. Best, Yangze Guo On Tue, Aug 17, 2021 at 9:15 AM Jin Yi wrote: > > is apache bahir still a thing? it hasn't been touched for months (since > redis 2.8.5). > > as such, looking at the current flink connector docs, it's no longer pointing > to anything from the bahir project. looking around in either the flink or > bahir newsgroups doesn't turn up anything regarding bahir's EOL. > > is the best bet for a flink to redis sink something i roll on my own > (inclined to go this route w/ buffered writes)? or should i try going > through via kafka and using confluent's kafka redis connector (flink => kafka > => redis)?
redis sink from flink
is apache bahir still a thing? it hasn't been touched for months (since redis 2.8.5). as such, looking at the current flink connector docs, it's no longer pointing to anything from the bahir project. looking around in either the flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL. is the best bet for a flink to redis sink something i roll on my own (inclined to go this route w/ buffered writes)? or should i try going through via kafka and using confluent's kafka redis connector (flink => kafka => redis)?
