Re: [DISCUSS] KIP idea: Support of multipart messages

2020-09-11 Thread Alexander Sibiryakov
to be a problem, but maybe we didn't try with newer GC > > and > > > simple tuning will solve it? > > > 2. We have head-of-line blocking issue on the queue. There are > approaches > > > to solve that too. > > > > > > I'd love to see more ex

Re: [DISCUSS] KIP idea: Support of multipart messages

2020-09-11 Thread Alexander Sibiryakov
anaged Kafka services like Confluent Cloud. These are forced to use either referenced-based or chunking. For those who are deploying Kafka themselves there are options to increase message size, at the price of slowing down the broker. A. > > > > On Thu, Sep 10, 2020 at 12:21 AM Alexa

Re: [DISCUSS] KIP idea: Support of multipart messages

2020-09-10 Thread Alexander Sibiryakov
had something like this. Becket did a talk on it a few years ago. > It would be interesting to know what became of it and if there were lessons > learned. > https://www.youtube.com/watch?v=ZrbaXDYUZY8 > > On Fri, 4 Sep 2020 at 08:17, Alexander Sibiryakov < > sibirya...@scrapinghub

[DISCUSS] KIP idea: Support of multipart messages

2020-09-04 Thread Alexander Sibiryakov
Hello, I would like to get your opinions on this KIP idea. In short it will allow to transfer messages of bigger size than allowed by the broker. https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2QDQg/edit?usp=sharing If all that makes sense, I'll create a full fledged

[jira] [Created] (KAFKA-10335) Blocking of producer IO thread when calling send() from callback

2020-07-31 Thread Alexander Sibiryakov (Jira)
Alexander Sibiryakov created KAFKA-10335: Summary: Blocking of producer IO thread when calling send() from callback Key: KAFKA-10335 URL: https://issues.apache.org/jira/browse/KAFKA-10335

Kafka broker architecture overview

2020-06-18 Thread Alexander Sibiryakov
Hello, could anyone please point me to a document or blog post describing Kafka broker internal architecture: threads, data flows, startup/shutdown routine? Unfortunately, I wasn't able to find anything quickly in Google. Thank you. A.

default auto.offset.reset value is reset to none

2020-05-06 Thread Alexander Sibiryakov
Hello, I'm facing an issue in one of our Kafka Streams applications using GlobalKTable. The idea is to have a GlobalKTable over compacted topic and be able to re-read it on startup. We had a consumer group and topic sometime ago, recently I've recreated a topic, requiring consumer offsets to be