Re: Potential Kafka Connector FLIP: Large Message Handling

gt; > From: Kevin Lam > Date: Wednesday, 10 July 2024 at 19:20 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: Potential Kafka Connector FLIP: Large Message > Handling > Hey David, Yes my proposal was originally to do this at the connector > level, as you mentioned it does

RE: Potential Kafka Connector FLIP: Large Message Handling

Hi Kevin, That sounds good, unfortunately the image did not come through in the email for me,kind regards, David From: Kevin Lam Date: Wednesday, 10 July 2024 at 19:20 To: dev@flink.apache.org Subject: [EXTERNAL] Re: Potential Kafka Connector FLIP: Large Message Handling Hey David, Yes my

Re: Potential Kafka Connector FLIP: Large Message Handling

the external > storage lives and authentication. Limitations around stack and heap sizes > would be worth considering. > > Am I understanding your intent correctly? > Kind regards, David. > > > From: Kevin Lam > Date: Wednesday, 10 July 2024 at 14:35 > T

RE: Potential Kafka Connector FLIP: Large Message Handling

correctly? Kind regards, David. From: Kevin Lam Date: Wednesday, 10 July 2024 at 14:35 To: dev@flink.apache.org Subject: [EXTERNAL] Re: Potential Kafka Connector FLIP: Large Message Handling Hey all, just a follow-up here. I was able to insert our Large Message handling by overriding

Re: Potential Kafka Connector FLIP: Large Message Handling

Hey all, just a follow-up here. I was able to insert our Large Message handling by overriding value.serializer and value.deserializer in the consu

Re: Potential Kafka Connector FLIP: Large Message Handling

Hi Fabian, Awesome, this project looks great! Thanks for sharing. It would work well with KafkaSource and the DataStream API as you've mentioned. We have something similar internally, but where we are encountering difficulty is integrating it with the Flink SQL Kafka DynamicTable Source and Sinks.

Re: Potential Kafka Connector FLIP: Large Message Handling

Hi Kevin, I worked on a project [1] in the past that had a similar purpose. You should be able to use a similar approach with the existing KafkaSource by implementing your own KafkaRecordDeserializationSchema that hides the logic of pulling the records from blob storage from the connector. You can

Re: Potential Kafka Connector FLIP: Large Message Handling

Hi all, Thanks for the responses. Grace those are indeed both challenges, thanks for flagging them. Regarding expiry, we could consider having a Mark and Sweep garbage collection system. A service can consume the topics with large messages, and track references. When there are no references left

Re: Potential Kafka Connector FLIP: Large Message Handling

Hi Kevin, I just want to double check, were you planning to send this proposal to the Kafka Dev mailing list? Because I don't see directly how this affects Flink :) Best regards, Martijn On Mon, Jul 8, 2024 at 8:05 AM Grace Grimwood wrote: > Hi Kevin, > > Thanks for starting this thread. > >

Re: Potential Kafka Connector FLIP: Large Message Handling

Hi Kevin, Thanks for starting this thread. This idea is something that was discussed in Kroxylicious (an open source Kafka proxy, I'm a maintainer there). In that discussion [1] we came to the conclusion that there are a couple of issues with implementing this: 1. Doesn't scale - very large messa