Re: read messages from kakfa: 2 different message types in kafka topic

2022-08-09 Thread Sigalit Eliazov
Thanks for your response we have different messages with separate schemas. I'll review the suggested solution. BR Sigalit On Tue, Aug 9, 2022 at 3:40 PM Moritz Mack wrote: > Hi Sigalit, > > > > Could you explain a bit more in detail what you mean by 2 different types > of messages? > > Do they

Re: Dataflow SQL streaming extensions

2022-08-09 Thread Andrew Pilloud via user
Hi Marcin, I'm having a little trouble understanding this. I think this is a summary of your problem statement: You have a pipeline that windows data on event time. Your event generator has an artificial 30 second delay. The pipeline appears to be experiencing a 10-20 second delay instead of the e

Re: read messages from kakfa: 2 different message types in kafka topic

2022-08-09 Thread Moritz Mack
Hi Sigalit, Could you explain a bit more in detail what you mean by 2 different types of messages? Do they share the same schema, e.g. using a union / one of type? Or are you in fact talking about different messages with separate schemas (e.g. discriminated using a message header)? The recomme

read messages from kakfa: 2 different message types in kafka topic

2022-08-09 Thread Sigalit Eliazov
Hi all we have a single kafka topic which is used to receive 2 different types of messages. These 2 messages are Avro. So when reading messages from kafka i used the GenericRecord KafkaIO.read() .withBootstrapServers(bootstrapServers) .withTopic(topic) .withConsumerConfigUp