RE: replica.fetch.max.bytes split message or not ?
Thank Hans ! I will read the KIP Adrien De : Hans Jespersen Envoyé : dimanche 25 février 2018 17:30:12 À : users@kafka.apache.org Objet : Re: replica.fetch.max.bytes split message or not ? This is the KIP-74 write up if you want to learn more about the motivation and implementation of the fetch.max.bytes feature. For example, the 5th message in your example can now be fetched even if it is larger than fetch.max.bytes which was a great improvement in large message handling. Previous the consumer could get stuck and not make progress. https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes -hans > On Feb 25, 2018, at 8:04 AM, adrien ruffie wrote: > > Hi Waleed, > > thank for you reply, that I thought too ! > but it was just to check if someone thought like me. > > Best regards, > Adrien > > De : Waleed Fateem > Envoyé : dimanche 25 février 2018 16:36:28 > À : users@kafka.apache.org > Objet : Re: replica.fetch.max.bytes split message or not ? > > I would say you will get that 5th message in the next request. > > I don't believe under any circumstance a Kafka broker will send or receive > a partial message. > > On Feb 24, 2018 10:52 AM, "adrien ruffie" wrote: > >> Hello, >> >> >> I have found this description in only documentation (without taking into >> account spelling errors 'byes of messages'), a question stays in my mind ... >> >> what happens if the size does not fall correctly on a message number? >> >> Example if in one parition I have 10(messages) of 1024 bytes --> I have >> 10240 bytes. >> >> But If I set the replica.fetch.max.bytes option to 5000, >> >> the 5th message will be splitted of of 120 bytes, and send after ? Or just >> 4 messages will be sent and the 5th will be sent in the next fetch request ? >> >> >> Thank and best regards, >> >> Adrien >> >> replica.fetch.max.bytes 1024 * 1024 The number of byes of messages to >> attempt to fetch for each partition in the fetch requests the replicas send >> to the leader. >> >>
Re: replica.fetch.max.bytes split message or not ?
This is the KIP-74 write up if you want to learn more about the motivation and implementation of the fetch.max.bytes feature. For example, the 5th message in your example can now be fetched even if it is larger than fetch.max.bytes which was a great improvement in large message handling. Previous the consumer could get stuck and not make progress. https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes -hans > On Feb 25, 2018, at 8:04 AM, adrien ruffie wrote: > > Hi Waleed, > > thank for you reply, that I thought too ! > but it was just to check if someone thought like me. > > Best regards, > Adrien > > De : Waleed Fateem > Envoyé : dimanche 25 février 2018 16:36:28 > À : users@kafka.apache.org > Objet : Re: replica.fetch.max.bytes split message or not ? > > I would say you will get that 5th message in the next request. > > I don't believe under any circumstance a Kafka broker will send or receive > a partial message. > > On Feb 24, 2018 10:52 AM, "adrien ruffie" wrote: > >> Hello, >> >> >> I have found this description in only documentation (without taking into >> account spelling errors 'byes of messages'), a question stays in my mind ... >> >> what happens if the size does not fall correctly on a message number? >> >> Example if in one parition I have 10(messages) of 1024 bytes --> I have >> 10240 bytes. >> >> But If I set the replica.fetch.max.bytes option to 5000, >> >> the 5th message will be splitted of of 120 bytes, and send after ? Or just >> 4 messages will be sent and the 5th will be sent in the next fetch request ? >> >> >> Thank and best regards, >> >> Adrien >> >> replica.fetch.max.bytes 1024 * 1024 The number of byes of messages to >> attempt to fetch for each partition in the fetch requests the replicas send >> to the leader. >> >>
RE: replica.fetch.max.bytes split message or not ?
Hi Waleed, thank for you reply, that I thought too ! but it was just to check if someone thought like me. Best regards, Adrien De : Waleed Fateem Envoyé : dimanche 25 février 2018 16:36:28 À : users@kafka.apache.org Objet : Re: replica.fetch.max.bytes split message or not ? I would say you will get that 5th message in the next request. I don't believe under any circumstance a Kafka broker will send or receive a partial message. On Feb 24, 2018 10:52 AM, "adrien ruffie" wrote: > Hello, > > > I have found this description in only documentation (without taking into > account spelling errors 'byes of messages'), a question stays in my mind ... > > what happens if the size does not fall correctly on a message number? > > Example if in one parition I have 10(messages) of 1024 bytes --> I have > 10240 bytes. > > But If I set the replica.fetch.max.bytes option to 5000, > > the 5th message will be splitted of of 120 bytes, and send after ? Or just > 4 messages will be sent and the 5th will be sent in the next fetch request ? > > > Thank and best regards, > > Adrien > > replica.fetch.max.bytes 1024 * 1024 The number of byes of messages to > attempt to fetch for each partition in the fetch requests the replicas send > to the leader. > >
Re: Kafka Zookeeper Connection
You also have to update the property `zookeeper.session.timeout.ms On Sat, Feb 24, 2018 at 11:25 PM, Ted Yu wrote: > Please take a look at maxSessionTimeout under: > http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin. > html#sc_advancedConfiguration > > On Sat, Feb 24, 2018 at 9:46 AM, Soheil Pourbafrani > > wrote: > > > Thanks, Manna, Can you say which property (in Kafka or Zookeeper) should > I > > increase? > > > > On Sat, Feb 24, 2018 at 7:49 PM, M. Manna wrote: > > > > > Have you tried increasing the timeouts for zookeeper nodes and Kafka > > > brokers to see if they make a difference? > > > > > > > > > > > > On Sat, 24 Feb 2018 at 14:55, Soheil Pourbafrani < > soheil.i...@gmail.com> > > > wrote: > > > > > > > Hi and Thanks, > > > > Excuse me, The Kafka version is 0.11 and Zookeeper version is 3.4.10 > > > > > > > > I've check Zookeeper logs and sessions are expiring and renewing > > > > continuously there. I use the same Zookeeper cluster for Hadoop HA > and > > it > > > > works well. > > > > > > > > I Answer the same question with more details in the following link: > > > > > > > > Here > > > > < > > > > https://stackoverflow.com/questions/48885469/kafka- > > > zookeeper-connection-drop-continuously > > > > > > > > > > > > > > >
Re: replica.fetch.max.bytes split message or not ?
I would say you will get that 5th message in the next request. I don't believe under any circumstance a Kafka broker will send or receive a partial message. On Feb 24, 2018 10:52 AM, "adrien ruffie" wrote: > Hello, > > > I have found this description in only documentation (without taking into > account spelling errors 'byes of messages'), a question stays in my mind ... > > what happens if the size does not fall correctly on a message number? > > Example if in one parition I have 10(messages) of 1024 bytes --> I have > 10240 bytes. > > But If I set the replica.fetch.max.bytes option to 5000, > > the 5th message will be splitted of of 120 bytes, and send after ? Or just > 4 messages will be sent and the 5th will be sent in the next fetch request ? > > > Thank and best regards, > > Adrien > > replica.fetch.max.bytes 1024 * 1024 The number of byes of messages to > attempt to fetch for each partition in the fetch requests the replicas send > to the leader. > >
