Hi James, A JIRA would be helpful. It looks like something we should fix.
Ismael On Mon, Mar 30, 2020 at 8:17 PM James Olsen <ja...@inaseq.com> wrote: > Christophe, > > See "Problems when Consuming from multiple Partitions” in the list > archive. I’ll forward you the full conversation privately. It includes > debug logs that demonstrate fetches being discarded and refetched, but only > after one or more full expirations of the fetch.max.wait.ms even though > messages are available. It appears to be due to the re-fetch only querying > one of the Partitions allocated to the Consumer, so if that Partition is > empty you get the full delay even though messages are available (and in > fact already fetched) from another Partition. > > The exact behaviour differs depending on the server and client versions. > I tried 2.2.1 and 2.3.1 servers with a 2.4.0 client. Both introduced > delays and/or some partitions not being processed at all. > > The problems were only observed where a Consumer subscribed to multiple > Partitions of the same Topic. > > I haven’t raised an issue for it as I would have no expectation of it > being fixed. I just made my client and server versions match and > implemented a workaround for the issue I wanted the 2.4.0 client to resolve. > > Regards, James. > > On 31/03/2020, at 12:16, JALADY Christophe <christophe.jal...@laposte.fr > <mailto:christophe.jal...@laposte.fr>> wrote: > > There are serious latency issues when mixing different client and server > version > > Could you be more specific ? Link to any issue ? > > Thanks by advance ! > > Christophe > ________________________________ > De : James Olsen <ja...@inaseq.com<mailto:ja...@inaseq.com>> > Envoyé : vendredi 27 mars 2020 01:48 > À : users@kafka.apache.org<mailto:users@kafka.apache.org> < > users@kafka.apache.org<mailto:users@kafka.apache.org>> > Objet : Re: Reg : Slowness in Kafka > > Also check your Kafka Client and Server versions. There are serious > latency issues when mixing different client and server versions IF your > consumers handle multiple partitions. > > On 27/03/2020, at 12:59, Chris Larsen <clar...@confluent.io<mailto: > clar...@confluent.io>> wrote: > > Hi Vidhya, > > How many tasks are you running against the topic? How many partitions are > on the topic? Can you post the connector config anonymized? > > Best, > Chris > > On Thu, Mar 26, 2020 at 17:58 Vidhya Sakar <sakar.bl...@gmail.com<mailto: > sakar.bl...@gmail.com>> wrote: > > Hi Team, > > The Kafka consumer is reading only 8 records per second.We have implemented > apache Kafka and confluent connect S3. The confluent connect S3 collects > the records and pushes it to S3 bucket. > In this process, we are seeing some slowness like on an average only 8 > records is being processed for a second. am planning to have a higher > throughout results, so that where there is a higher data load, it should > process more number of records. > > Thanks in advance. > > -- > > *Chris Larsen* > > Systems Engineer | Confluent > > 847.274.3735 > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > <https://www.confluent.io/blog/> > > > Post-scriptum La Poste > > Ce message est confidentiel. Sous reserve de tout accord conclu par > ecrit entre vous et La Poste, son contenu ne represente en aucun cas un > engagement de la part de La Poste. Toute publication, utilisation ou > diffusion, meme partielle, doit etre autorisee prealablement. Si vous > n'etes pas destinataire de ce message, merci d'en avertir immediatement > l'expediteur. > >
