+1 (non binding)
Thanks for the KIP
On Wed, Jan 16, 2019 at 5:47 PM vcgat...@gmail.com wrote:
>
>
>
> On 2018/12/11 10:48:02, Rajini Sivaram wrote:
> > Hi all,
> >
> > I have submitted a KIP to improve fairness in channel processing in
> > SocketServer to protect brokers from connection storms:
On 2018/12/11 10:48:02, Rajini Sivaram wrote:
> Hi all,
>
> I have submitted a KIP to improve fairness in channel processing in
> SocketServer to protect brokers from connection storms:
>
>-
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-402%3A+Improve+fairness+in+SocketSe
https://drive.google.com/open?id=14W1-hEu6rFHhZGFxY_vUGD2HAR-cRYEj
On 2018/12/11 10:48:02, Rajini Sivaram wrote:
> Hi all,
>
> I have submitted a KIP to improve fairness in channel processing in
> SocketServer to protect brokers from connection storms:
>
>-
>
> https://cwiki.apache.org
On 15.01.2019 13:27, Rajini Sivaram wrote:
> Hi Jan,
>
> If the queue of one Processor is full, we move to the next Processor
> immediately without blocking. So as long as the queue of any Processor is
> not full, we accept the connection immediately. If the queue of all
> Processors are full, we a
Thanks Colin. I have updated the KIP.
On Tue, Jan 15, 2019 at 9:20 PM Colin McCabe wrote:
> On Tue, Jan 15, 2019, at 12:59, Rajini Sivaram wrote:
> > Hi Colin,
> >
> > `AcceptorIdlePercent` indicates the total amount of time the acceptor is
> > inactive and not accepting any connections because
On Tue, Jan 15, 2019, at 12:59, Rajini Sivaram wrote:
> Hi Colin,
>
> `AcceptorIdlePercent` indicates the total amount of time the acceptor is
> inactive and not accepting any connections because it is blocked on
> Processors. But I agree the name could be improved. There is back pressure
> at the
Hi Colin,
`AcceptorIdlePercent` indicates the total amount of time the acceptor is
inactive and not accepting any connections because it is blocked on
Processors. But I agree the name could be improved. There is back pressure
at the Java level (which we can't monitor) and back pressure we apply wi
I think the point is that we distribute the time more fairly between
connection handling and other operations where before we could block on the
TLS handshake for a long time given a large number of connections.
Ismael
On Tue, Jan 15, 2019 at 11:39 AM Colin McCabe wrote:
> Hi Rajini,
>
> Thanks
Hi Rajini,
Thanks for this. The KIP looks really useful.
>
> A new metric will be added to track the amount of time Acceptor is blocked
> from accepting connections due to backpressure. This will be a yammer
> Meter, consistent with other SocketServer metrics.
>
> kafka.network:type=Accepto
Hi Jan,
If the queue of one Processor is full, we move to the next Processor
immediately without blocking. So as long as the queue of any Processor is
not full, we accept the connection immediately. If the queue of all
Processors are full, we assign a Processor and block until the connection
can b
> The connection queue for Processors will be changed to ArrayBlockingQueue
> with a fixed size of 20. Acceptor will use round-robin allocation to allocate
> each new connection to the next available Processor to which the connection
> can be added without blocking. If a Processor's queue is f
of client connections on a
> > >> different listener.
> > >>
> > >> do you think it would make sense to also allow the
> `num.network.threads`
> > >> to have an optional per-listener prefix ?
> > >>
> > >> ciao,
>
o also allow the `num.network.threads`
> >> to have an optional per-listener prefix ?
> >>
> >> ciao,
> >> Edo
> >> ----------
> >>
> >> Edoardo Comar
> >>
> >> IBM Event Stre
ional per-listener prefix ?
>>
>> ciao,
>> Edo
>> --
>>
>> Edoardo Comar
>>
>> IBM Event Streams
>> IBM UK Ltd, Hursley Park, SO21 2JN
>>
>>
>> Rajini Sivaram wrote on 11/12/2018 18:22
---
>
> Edoardo Comar
>
> IBM Event Streams
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
> Rajini Sivaram wrote on 11/12/2018 18:22:03:
>
> > From: Rajini Sivaram
> > To: dev
> > Date: 11/12/2018 18:22
> > Subject: Re: [DISCUSS]
21 2JN
Rajini Sivaram wrote on 11/12/2018 18:22:03:
> From: Rajini Sivaram
> To: dev
> Date: 11/12/2018 18:22
> Subject: Re: [DISCUSS] KIP-402: Improve fairness in SocketServer
processors
>
> Hi Harsha,
>
> Thanks for reviewing the KIP.
>
> 1) Yes, agree
Thanks for the details Rajini. It would be great if you can add a few details
to the KIP, on how many connections you are able to handle in your cluster with
number 20 to give some context.
Thanks,
Harsha
On Tue, Dec 11, 2018, at 10:22 AM, Rajini Sivaram wrote:
> Hi Harsha,
>
> Thanks for rev
Hi Harsha,
Thanks for reviewing the KIP.
1) Yes, agree that we also need a max.connections configuration per-broker.
I was thinking of doing that in a separate KIP, but I could add that here
as well.
2) The number of connections processed in each iteration doesn't feel like
an externalizable conf
Hi Rajini,
Overall KIP looks good to me. Is it possible to use
max.connections config that we already have, althought its per IP.
But broker level max.connections would also be good have to guard against
DOS'ing a broker.
Eitherway having constant like 20 without a configurable
Hi all,
I have submitted a KIP to improve fairness in channel processing in
SocketServer to protect brokers from connection storms:
-
https://cwiki.apache.org/confluence/display/KAFKA/KIP-402%3A+Improve+fairness+in+SocketServer+processors
Feedback and suggestions welcome.
Thank you,
Raj
20 matches
Mail list logo
