Hi Vincent, Thanks for your contribution! I'm happy to work with you on this when you contribute the code into Beam.
Another thing is that it would be preferable to use Splittable DoFn <https://beam.apache.org/documentation/programming-guide/#splittable-dofns> instead of using UnboundedSource to write a new IO. On Thu, Nov 19, 2020 at 10:18 AM Vincent Marquez <vincent.marq...@gmail.com> wrote: > Currently, Redis offers a streaming queue functionality similar to > Kafka/Kinesis/Pubsub, but Beam does not currently have a connector for it. > > I've written an UnboundedSource connector that makes use of Redis Streams > as a POC and it seems to work well. > > If someone is willing to work with me, I could write up a JIRA and/or open > up a WIP pull request if there is interest in getting this as an official > connector. I would mostly need guidance on naming/testing aspects. > > https://redis.io/topics/streams-intro > > *~Vincent* >
