Re: writing a single record to Kafka ...

Thanks, Lukasz. Appreciate your advice. *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com * On Tue, Sep 11, 2018 at 6:19 PM Lukasz Cwik wrote: > A PCollection is a bag of elements. PCollections can be empty, have only > one element or have many. It is up to you

Re: writing a single record to Kafka ...

A PCollection is a bag of elements. PCollections can be empty, have only one element or have many. It is up to you to choose how many elements are emitted into the PCollection by the upstream transforms. If you can limit the number of elements to the PCollection that you applied KafkaIO to to only

writing a single record to Kafka ...

Hello - I'd like to write a single record to kafka topic through beam. However, I only see examples that work with PCollection. Any thoughts about how I can approach to this? Thank you. Regards, Mahesh *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com *

Re: Problem with KafkaIO

Specifically, I am interested if you have any thread running 'consumerPollLoop()' [1]. There should always be one (if a worker is assigned one of the partitions). It is possible that KafkaClient itself is hasn't recovered from the group coordinator error (though unlikely). https://github.com/apach

Re: EOS with KafkaIO Writes on Flink

On Wed, Aug 22, 2018 at 9:37 AM Abdul Qadeer wrote: > Hi! > > I came across this comment for KafkaIO.Write.withEOS in Beam 2.6.0: > > "Flink runner is > * one of the runners whose checkpoint semantics are not compatible with > current > * implementation (hope to provide a solution in near future

Re: Problem with KafkaIO

Hi Eduardo, In case of any error, the pipeline should keep on trying to fetch. I don't know about this particular error. Do you see any others afterwards in the log? Couple of things you could try if the logs are not useful : - login to one of the VMs and get stacktrace of java worker (look for a

Problem with KafkaIO

Hi, We have a Apache Beam pipeline running in Google Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our other workers from other pipelines continued to get Kafka messages. At the moment it stopped we got these messages: I [Consumer clientId=consumer-1, grou

Re: Acknowledging Pubsub messages in Flink Runner

Hey Encho, The Flink Runner acknowledges messages through PubSubIO's `CheckpointMark#finalizeCheckpoint()` method. The Flink Runner wraps the PubSubIO source via the UnboundedSourceWrapper. When Flink takes a checkpoint of the running Beam streaming job, the wrapper will retrieve the Checkpo