I've just updated the wiki with most of your comments hoping to make it
more clear. I stole much of Joel's comment so all the credit goes to him if
it works! The line between documenting the code and describing it is always
thin. My intention was not to make anything official at first either, hence
Marc, thanks much for documenting the guts!
There is one correction for Fetch Request handling:
When is it satisfied?
The fetch size requested is reached - ie. the amount of data the consumer
wishes to receive in one response
Consumer configuration: *fetch.message.max.bytes*
As per the code:
Marc - thanks again for doing this. Couple of suggestions:
- I would suggest removing the disclaimer and email quotes since this
can become a stand-alone clean document on what the purgatory is and
how it works.
- A diagram would be helpful - it could say, show the watcher map and
the expir
gt; > Libo
> >
> >
> > -Original Message-
> > From: Joel Koshy [mailto:jjkosh...@gmail.com]
> > Sent: Friday, November 08, 2013 2:01 AM
> > To: users@kafka.apache.org
> > Subject: Re: Purgatory
> >
> > Excellent - thanks for putting t
add a summary
> at the beginning about what it is and how it may impact a user.
>
> Regards,
>
> Libo
>
>
> -Original Message-
> From: Joel Koshy [mailto:jjkosh...@gmail.com]
> Sent: Friday, November 08, 2013 2:01 AM
> To: users@kafka.apache.org
> Subject:
Subject: Re: Purgatory
Excellent - thanks for putting that together! Will review it more carefully
tomorrow and suggest some minor edits if required.
On Thu, Nov 07, 2013 at 10:45:40PM -0500, Marc Labbe wrote:
> I've just added a page for purgatory, feel free to comment/modify at will.
&g
Excellent - thanks for putting that together! Will review it more
carefully tomorrow and suggest some minor edits if required.
On Thu, Nov 07, 2013 at 10:45:40PM -0500, Marc Labbe wrote:
> I've just added a page for purgatory, feel free to comment/modify at will.
> I hope I didn't misinterpret too
I've just added a page for purgatory, feel free to comment/modify at will.
I hope I didn't misinterpret too much of the code.
https://cwiki.apache.org/confluence/display/KAFKA/Request+Purgatory+(0.8)
I added a few questions of my own.
On Fri, Nov 1, 2013 at 9:43 PM, Joe Stein wrote:
> To edit
To edit the Wiki you need to send an ICLA
http://www.apache.org/licenses/#clas to Apache and then once that is done
an email to priv...@kafka.apache.org (or to me and I will copy private)
with your Wiki username and that you sent the ICLA to Apache.
Then, I can add you to edit the Wiki.
/
Hi Joel,
I used to have edit to the wiki, I made a few additions to it a while ago
but it's seem I don't have it anymore. It might have been lost in the
confluence update. I would be glad to add what I have written if I get it
back. Otherwise, feel free to paste my words in one of the pages, I don
Marc, thanks for writing that up. I think it is worth adding some
details on the request-purgatory on a wiki (Jay had started a wiki
page for kafka internals [1] a while ago, but we have not had time to
add much to it since.) Your write-up could be reviewed and added
there. Do you have edit permiss
Guozhang,
I have to agree with Priya the doc isn't very clear. Although the
configuration is documented, it is simply rewording the name of the config,
which isn't particularly useful if you want more information about what the
purgatory is. I searched the whole wiki and doc and could not find any
Priya, if you want you can look at RequestPurgatory.scala for some more
details.
The config is the size of the atomic requestCounter.
Basically the purge in the purgatory is a way to check if the request has
been satisfied and delayed and can get removed. It is a background scan
when the size re
Guozhang,
The documentation is not very clear.
Marc's response for producer purgatory makes sense.
I am not entirely clear on fetch purgatory.
How does broker use purgatory? Is it a temporary holding area? What happens
to the messages if purge interval is exceeded in case of either/both
producer an
Hello Priya,
You can find the definitions of these two configs here:
http://kafka.apache.org/documentation.html#brokerconfigs
Guozhang
On Thu, Oct 31, 2013 at 11:20 AM, Marc Labbe wrote:
> Hi Priya
>
> my understanding is producer requests will be delayed (and put in request
> purgatory) onl
Hi Priya
my understanding is producer requests will be delayed (and put in request
purgatory) only if your producer uses ack=-1. It will be in the purgatory
(delayed) until all brokers have acknowledged the messages to be
replicated. The documentation suggests to monitor the
ProducerRequestPurgato
16 matches
Mail list logo