Re: Stream to KTable internals

Yes, a StreamThread has one consumer. The number of StreamThreads per instance is configurable via `num.stream.threads`. Partitions are assigned to threads similar to consumer is a plain consumer group. It seems you run with the default of one thread per instance. As you spin up 12 instances,

Re: gRPC with Kafka

Hi Matthew, If you can give more details about the course objectives that would help me understand where your questions are coming from but I think they have to do with the mixup between message formats and communication mechanisms. I think you are mixing up the communication mechanisms with the

Re: Stream to KTable internals

Yes, this helped. I have some additional questions. Does StreamThread have one consumer? (Looks like it, but just want to confirm) Is there a separate StreamThread for each topic including the KTable? If a KTable is a StreamThread and there is a StreamTask for that KTable, could my buffer be gett

gRPC with Kafka

I am involved with a course whose project has introduced REST, Kafka and gRPC on its topic of message passing. However, in the project phase of this section, we are tasked with passing gRPC messages through Kafka to handle large amounts of location data. Has anyone used these two items togethe

Re: Stream to KTable internals

Thank you for the information. This is actually not my issue. In my scenario the records were written to the KTable topic before the record was written to the KStream topic. We only get a handful of these issues per day, and they seem to happen when many transactions are being run through the syst

Re: Stream to KTable internals

Hi Chad, > I'm wondering if someone can point me to the Kafka streams internal code that reads records for the join? --> You can check StreamThread#pollPhase, where stream thread (main consumer) periodically poll records. And then, it'll process each topology node with these polled records in strea

Re: Stream to KTable internals

Hi Chad, this talk around 24:00 clearly explains what you’re seeing https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/