That might work if my consumers were in the same process but the db consumer is a python job running under Airflow and the realtime consumer wold be running as a backend service on another server.
Also how would I seed the realtime consumer at startup if the consumer isn't running which would could be possible if it hit the end stream, The db consumer is designed to read until no new message is delivered then exit till it's next spawned. Ronald Fenner Network Architect Game Circus LLC. rfen...@gamecircus.com > On May 28, 2021, at 12:05 AM, Ran Lupovich <ranlupov...@gmail.com> wrote: > > https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seek(org.apache.kafka.common.TopicPartition,%20long) > > בתאריך יום ו׳, 28 במאי 2021, 08:04, מאת Ran Lupovich <ranlupov...@gmail.com >> : > >> While your DB consumer is running you get the access to the partition >> ${partition} @ offset ${offset} >> >> https://github.com/confluentinc/examples/blob/6.1.1-post/clients/cloud/nodejs/consumer.jswhen >> setting your second consumers for real time just set them tostart from that >> point >> >> >> בתאריך יום ו׳, 28 במאי 2021, 01:51, מאת Ronald Fenner < >> rfen...@gamecircus.com>: >> >>> I'm trying to figure out how to pragmatically read a consumer groups >>> offset for a topic. >>> What I'm trying to do is read the offsets of our DB consumers that run >>> once an hour and batch lad all new messages. I then would have another >>> consumer that monitors the offsets that have been consumed and consume the >>> message not yet loaded storing them in memory to be able to send them to a >>> viewer. As messages get consumed they then get pruned from the in memory >>> cache. >>> >>> Basically I'm wanting to create window on the messages that haven't been >>> loaded into the db. >>> >>> I've seen ways of getting it from the command line but I'd like to from >>> with in code. >>> >>> Currently I'm using node-rdkafka. >>> >>> I guess as a last resort I could shell the command line for the offsets >>> then parse it and get it that way. >>> >>> >>> Ronald Fenner >>> Network Architect >>> Game Circus LLC. >>> >>> rfen...@gamecircus.com >>> >>>
