Re: Copying data from one topic into another
I like it i was just kidding I’ll subscribe there too for sure!! I was just adoringly teasing you all 😛 On Wed, Nov 30, 2022 at 12:04 PM nagpavan chilakam < nagpavan.chila...@gmail.com> wrote: > Maybe you can even explore `kafka-console-consumer` to get the messages > from a certain offset > > On Wed, Nov 30, 2022 at 8:11 PM Tecno Brain > wrote: > > > By the way, we use Avro for the messages. > > > > > > > > On Wed, Nov 30, 2022 at 9:35 AM Tecno Brain < > cerebrotecnolog...@gmail.com> > > wrote: > > > > > Hi, > > > I'm using Kafka 2.8 > > > I have a topic with an original retention of 1 day with a large > amount > > > of messages. > > > > > > We had a problem with some messages being corrupted and we decided to > > > skip about 3.5 hours of messages; but not all of them are corrupted. > So, > > we > > > would like to process this period of time again with a modified version > > of > > > our consumer application. > > > > > > In order to avoid the messages being deleted, we extended the > retention > > > to 3 days. > > > But we are now again close to have the messages being deleted and we > > > cannot extend the retention anymore (we would go over the limit of the > > disk > > > space). > > > > > > So, I would like to extract the messages between the offsets that > cover > > > the 3.5h > > > > > > Is there a quick way to extract these messages and copy them into > > > another topic ? > > > (After that, I want to set the retention back to the original 1 day) > > > > > > Would kcat help? > > > > > > > > > > > > > > > > > >
Re: Copying data from one topic into another
Maybe you can even explore `kafka-console-consumer` to get the messages from a certain offset On Wed, Nov 30, 2022 at 8:11 PM Tecno Brain wrote: > By the way, we use Avro for the messages. > > > > On Wed, Nov 30, 2022 at 9:35 AM Tecno Brain > wrote: > > > Hi, > > I'm using Kafka 2.8 > > I have a topic with an original retention of 1 day with a large amount > > of messages. > > > > We had a problem with some messages being corrupted and we decided to > > skip about 3.5 hours of messages; but not all of them are corrupted. So, > we > > would like to process this period of time again with a modified version > of > > our consumer application. > > > > In order to avoid the messages being deleted, we extended the retention > > to 3 days. > > But we are now again close to have the messages being deleted and we > > cannot extend the retention anymore (we would go over the limit of the > disk > > space). > > > > So, I would like to extract the messages between the offsets that cover > > the 3.5h > > > > Is there a quick way to extract these messages and copy them into > > another topic ? > > (After that, I want to set the retention back to the original 1 day) > > > > Would kcat help? > > > > > > > > > > >
[ANNOUNCE] Call for papers: Kafka Summit London 2023
Hi everyone, The call for papers (https://sessionize.com/kafka-summit-london-2023/) is now open for Kafka Summit London 2023, and you are all welcome to submit a talk. We are looking for the most interesting, most informative, most advanced, and most generally applicable talks on Apache Kafka® and the tools, technologies, and techniques in the Kafka ecosystem. People from all industries, backgrounds, and experience levels are encouraged to submit! If you have any questions about submitting, reach out to Danica Fine, the program chair, at df...@confluent.io. The call for papers closes on Monday, January 9 2022 at 23:59 GMT. Thanks, Chris
Re: Copying data from one topic into another
This is kind of boring I’m sorry I subscribed but I’ll keep them because I might learn from you On Wed, Nov 30, 2022 at 9:41 AM Tecno Brain wrote: > By the way, we use Avro for the messages. > > > > On Wed, Nov 30, 2022 at 9:35 AM Tecno Brain > wrote: > > > Hi, > > I'm using Kafka 2.8 > > I have a topic with an original retention of 1 day with a large amount > > of messages. > > > > We had a problem with some messages being corrupted and we decided to > > skip about 3.5 hours of messages; but not all of them are corrupted. So, > we > > would like to process this period of time again with a modified version > of > > our consumer application. > > > > In order to avoid the messages being deleted, we extended the retention > > to 3 days. > > But we are now again close to have the messages being deleted and we > > cannot extend the retention anymore (we would go over the limit of the > disk > > space). > > > > So, I would like to extract the messages between the offsets that cover > > the 3.5h > > > > Is there a quick way to extract these messages and copy them into > > another topic ? > > (After that, I want to set the retention back to the original 1 day) > > > > Would kcat help? > > > > > > > > > > >
Re: Copying data from one topic into another
By the way, we use Avro for the messages. On Wed, Nov 30, 2022 at 9:35 AM Tecno Brain wrote: > Hi, > I'm using Kafka 2.8 > I have a topic with an original retention of 1 day with a large amount > of messages. > > We had a problem with some messages being corrupted and we decided to > skip about 3.5 hours of messages; but not all of them are corrupted. So, we > would like to process this period of time again with a modified version of > our consumer application. > > In order to avoid the messages being deleted, we extended the retention > to 3 days. > But we are now again close to have the messages being deleted and we > cannot extend the retention anymore (we would go over the limit of the disk > space). > > So, I would like to extract the messages between the offsets that cover > the 3.5h > > Is there a quick way to extract these messages and copy them into > another topic ? > (After that, I want to set the retention back to the original 1 day) > > Would kcat help? > > > > >
Copying data from one topic into another
Hi, I'm using Kafka 2.8 I have a topic with an original retention of 1 day with a large amount of messages. We had a problem with some messages being corrupted and we decided to skip about 3.5 hours of messages; but not all of them are corrupted. So, we would like to process this period of time again with a modified version of our consumer application. In order to avoid the messages being deleted, we extended the retention to 3 days. But we are now again close to have the messages being deleted and we cannot extend the retention anymore (we would go over the limit of the disk space). So, I would like to extract the messages between the offsets that cover the 3.5h Is there a quick way to extract these messages and copy them into another topic ? (After that, I want to set the retention back to the original 1 day) Would kcat help?
