Re: Problem with KafkaIO

Filed BEAM-5375 . I will fix it later this week. On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi wrote: > > > On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi wrote: > >> Thanks for the job id, I looked at the worker logs (following usual >> support

Re: Problem with KafkaIO

On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi wrote: > Thanks for the job id, I looked at the worker logs (following usual > support oncall access protocol that provides temporary access to things > like logs in GCP): > > Root issue looks like consumerPollLoop() mentioned earlier needs to handle

Re: Problem with KafkaIO

Thanks for the job id, I looked at the worker logs (following usual support oncall access protocol that provides temporary access to things like logs in GCP): Root issue looks like consumerPollLoop() mentioned earlier needs to handle unchecked exception. In your case it is clear that poll thread

Re: Lateness for Spark

event time triggers (via watermarks) should be supported as well. On Sun, Sep 9, 2018 at 11:57 PM Vishwas Bm wrote: > Hi, > > Thanks for the reply. As per the beam capability matrix only > Processing-time triggers is supported by spark runner. > As this page is not updated, what other triggers

Re: Problem with KafkaIO

Hi Raghu! The job_id of our dev job is 2018-09-12_06_11_48-5600553605191377866. Thanks! Em qua, 12 de set de 2018 às 14:18, Raghu Angadi escreveu: > Thanks for debugging. > Can you provide the job_id of your dev job? The stacktrace shows that > there is no thread running 'consumerPollLoop()'

Re: Problem with KafkaIO

Thanks for debugging. Can you provide the job_id of your dev job? The stacktrace shows that there is no thread running 'consumerPollLoop()' which can explain stuck reader. You will likely find a logs at line 594 & 587 [1]. Dataflow caches its readers and DirectRunner may not. That can explain

Scio 0.6.1 released

Hi all, We just released Scio 0.6.1. This is mainly a bug fix release. Also just a heads up, we'll be releasing alpha/beta of 0.7.0 soon which will include some major breaking changes. Keep an eye on this section of

Re: writing a single record to Kafka ...

Your on the right track. Now you want to apply the KafkaIO PTransform and then run the pipeline. Pipeline pipeline = ... List li = // ArrayList of single element PCollection p = Create.of(li); p.apply(KafkaIO.write() .withBootstrapServers("broker_1:9092,broker_2:9092")

Re: Problem with KafkaIO

Hi Raghu, thanks for your help. Just answering your previous question, the following logs were the same as before the error, as if the pipeline were still getting the messages, for example: (...) Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 10. Resetting offset for

Re: writing a single record to Kafka ...

Hi Lukasz - I am trying to send a kafka message one per element in process_element function. As per your suggestion, I am trying to create a single element PCollection. However, how can I have a PCollection by self? I was hoping this would work ... List li = // ArrayList of single element

Re: Acknowledging Pubsub messages in Flink Runner

Hello Max, Thanks for the answer. My guess was that they are acknowledged at completion of Flink's checkpoints, but wanted to make sure since that doesn't explain my problem. Whenever a subscription is nearly empty the job gets slower overall and the Flink's checkpoints start taking much more

Minimum time between checkpoints in Flink runner

Hello, I am using Flink runner with Apache Beam 2.6.0. An important configuration of Flink is the 'minimum time between checkpoints' parameter in addition to just the checkpointing interval. This is important since if checkpoints are heavy, the job might end up doing mostly checkpointing and not

Re: Apache Beam Summit in Europe

Any left or returned tickets for the conference (advanced)? On Thu, 5 Jul 2018 at 09:14, Etienne Chauchot wrote: > Hi, > > Just a comment, I'm not sure 28-29/09 is very practical because some of > the Beam community will be at the apachecon in Montreal ending Sept 27th. > > Etienne > > Le