Thanks Andrew.
I'll give linger.ms a try.
I was testing worse case scenarios so linger.ms was set to 0. also the
producer was doing ack=all. which definitely adds all the producer requests
to the purgatory waiting to be acknowledged.
thanks.
On Sat, Mar 4, 2023 at 2:57 PM Andrew Grant
leader ends
up having to wait longer to hear back from the followers before sending the
produce response, which in turn could explain why the producer purgatory is
bigger. See the section "Commit time: Replicating the record from leader to
followers" in
https://www.confluent.io/blog/confi
ter. messages per second matches exactly the
> number of requests.
> but on the ceph cluster messages do not match total produce requests per
> second.
>
> and the only thing I can find is that the Producer purgatory in ceph kafka
> cluster has more request queued up than th
nd is that the Producer purgatory in ceph kafka
cluster has more request queued up than the local disk.
Also RemoteTime-ms for producers is high, which could explain why there are
more requests on the purgatory.
To me , I think this means that the Producer is waiting to hear from all
the acks. whic
understand that purgatory has been
rewritten (again) in 0.8.3, so might it be worth trying a trunk build? Is
there an ETA for a beta release of 0.8.3?
Thanks,
Evan
On Tue, Apr 14, 2015 at 8:40 PM, Evan Huus wrote:
> On Tue, Apr 14, 2015 at 8:31 PM, Jiangjie Qin
> wrote:
>
>> Hey
ly. The setup is A -> Mirrormaker -> B so mirrormaker is
consuming from A and producing to B.
Cluster A is always fine. Cluster B is fine when mirrormaker is stopped.
Cluster B has the weird purgatory issue when mirrormaker is running.
Today I rolled out a change to reduce the
`fetch.purgatory.purg
;s still occurring...
>
>Is there a separate mailing list or project for mirrormaker that I could
>ask?
>
>Thanks,
>Evan
>
>On Thu, Apr 9, 2015 at 4:36 PM, Evan Huus wrote:
>
>> Hey Folks, we're running into an odd issue with mirrormaker and the
>>fetch
Any ideas on this? It's still occurring...
Is there a separate mailing list or project for mirrormaker that I could
ask?
Thanks,
Evan
On Thu, Apr 9, 2015 at 4:36 PM, Evan Huus wrote:
> Hey Folks, we're running into an odd issue with mirrormaker and the fetch
> request purgator
Hey Folks, we're running into an odd issue with mirrormaker and the fetch
request purgatory on the brokers. Our setup consists of two six-node
clusters (all running 0.8.2.1 on identical hw with the same config). All
"normal" producing and consuming happens on cluster A. Mirrormaker
Looking for ideas from those who have been using kafka for some time.
Should I be concerned about the fetch purgatory size increasing to high numbers
and consistently remaining there while the producing data rates vary between
210k to 270k per sec (60M to 82M per sec)?
There are 5 brokers
Bytes
> }
>
>
>
> On Fri, Nov 8, 2013 at 1:01 PM, Joel Koshy wrote:
>
> > Marc - thanks again for doing this. Couple of suggestions:
> >
> > - I would suggest removing the disclaimer and email quotes since this
> > can become a stand-alone clean documen
ne clean document on what the purgatory is and
> how it works.
> - A diagram would be helpful - it could say, show the watcher map and
> the expiration queue, and it will be especially useful if it can
> show the flow of producer/fetch requests through the purgatory. That
> wo
Marc - thanks again for doing this. Couple of suggestions:
- I would suggest removing the disclaimer and email quotes since this
can become a stand-alone clean document on what the purgatory is and
how it works.
- A diagram would be helpful - it could say, show the watcher map and
the
gt; > Libo
> >
> >
> > -Original Message-
> > From: Joel Koshy [mailto:jjkosh...@gmail.com]
> > Sent: Friday, November 08, 2013 2:01 AM
> > To: users@kafka.apache.org
> > Subject: Re: Purgatory
> >
> > Excellent - thanks for putting t
add a summary
> at the beginning about what it is and how it may impact a user.
>
> Regards,
>
> Libo
>
>
> -Original Message-
> From: Joel Koshy [mailto:jjkosh...@gmail.com]
> Sent: Friday, November 08, 2013 2:01 AM
> To: users@kafka.apache.org
> Subject:
Subject: Re: Purgatory
Excellent - thanks for putting that together! Will review it more carefully
tomorrow and suggest some minor edits if required.
On Thu, Nov 07, 2013 at 10:45:40PM -0500, Marc Labbe wrote:
> I've just added a page for purgatory, feel free to comment/modify at will.
&g
Excellent - thanks for putting that together! Will review it more
carefully tomorrow and suggest some minor edits if required.
On Thu, Nov 07, 2013 at 10:45:40PM -0500, Marc Labbe wrote:
> I've just added a page for purgatory, feel free to comment/modify at will.
> I hope I didn
I've just added a page for purgatory, feel free to comment/modify at will.
I hope I didn't misinterpret too much of the code.
https://cwiki.apache.org/confluence/display/KAFKA/Request+Purgatory+(0.8)
I added a few questions of my own.
On Fri, Nov 1, 2013 at 9:43 PM, Joe Stein wr
l free to paste my words in one of the pages, I don't
> intend on asking for copyrights for this :).
>
> marc
>
>
> On Fri, Nov 1, 2013 at 4:32 PM, Joel Koshy wrote:
>
> > Marc, thanks for writing that up. I think it is worth adding some
> > details on th
e pages, I don't
intend on asking for copyrights for this :).
marc
On Fri, Nov 1, 2013 at 4:32 PM, Joel Koshy wrote:
> Marc, thanks for writing that up. I think it is worth adding some
> details on the request-purgatory on a wiki (Jay had started a wiki
> page for kafka internals
Marc, thanks for writing that up. I think it is worth adding some
details on the request-purgatory on a wiki (Jay had started a wiki
page for kafka internals [1] a while ago, but we have not had time to
add much to it since.) Your write-up could be reviewed and added
there. Do you have edit
Guozhang,
I have to agree with Priya the doc isn't very clear. Although the
configuration is documented, it is simply rewording the name of the config,
which isn't particularly useful if you want more information about what the
purgatory is. I searched the whole wiki and doc and coul
Priya, if you want you can look at RequestPurgatory.scala for some more
details.
The config is the size of the atomic requestCounter.
Basically the purge in the purgatory is a way to check if the request has
been satisfied and delayed and can get removed. It is a background scan
when the size
Guozhang,
The documentation is not very clear.
Marc's response for producer purgatory makes sense.
I am not entirely clear on fetch purgatory.
How does broker use purgatory? Is it a temporary holding area? What happens
to the messages if purge interval is exceeded in case of either/both
pro
Hello Priya,
You can find the definitions of these two configs here:
http://kafka.apache.org/documentation.html#brokerconfigs
Guozhang
On Thu, Oct 31, 2013 at 11:20 AM, Marc Labbe wrote:
> Hi Priya
>
> my understanding is producer requests will be delayed (and put in request
&g
Hi Priya
my understanding is producer requests will be delayed (and put in request
purgatory) only if your producer uses ack=-1. It will be in the purgatory
(delayed) until all brokers have acknowledged the messages to be
replicated. The documentation suggests to monitor the
Hello,
What is purgatory? I believe the following two properties relate to
consumer and producer respectively.
Could someone please explain the significance of these?
fetch.purgatory.purge.interval.requests=100
producer.purgatory.purge.interval.requests=100
Thanks,
Priya
27 matches
Mail list logo
