[Question] Using RedisIO with Redis Cluster

2021-10-15 Thread Gaurav Nakum
Hello! I am trying to use the RedisIO connector with Redis cluster but it looks like the Jedis client that RedisIO uses only works on a standalone Redis server, not on a cluster. I get this error when trying to read from Redis: Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineEx

Output from Window not getting materialized

2020-09-10 Thread Gaurav Nakum
Hi everyone! We are developing a new IO connector using the SDF API, and testing it with the following simple counting pipeline: p.apply(MyIO.read()     .withStream(inputStream)     .withStreamPartitions(Arrays.asList(0))     .withConsumerConfig(config)     ) // gets a PCollection>

Re: KafkaIO committing semantics

2020-09-08 Thread Gaurav Nakum
Thank you very much for your explanation! commitOffsetsInFinalize() -> although checkpointing depends on the runner is it not configurable in a connector implementation? Basically, I want to understand how this can be done with a new IO connector implementation, esp. with the new SDF API. If I