Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Ismael, I'll start a vote thread. Jon On Wed, Apr 25, 2018 at 2:28 PM, Dong Lin wrote: > + Jon > > On Tue, Apr 24, 2018 at 10:04 PM, Ismael Juma wrote: > >> Hi Jon, >> >> Not sure about this approach. It's worth mentioning this in the vote >> thread >> as well so that the people who voted

Re: [DISCUSS] KIP-219 - Improve Quota Communication

+ Jon On Tue, Apr 24, 2018 at 10:04 PM, Ismael Juma wrote: > Hi Jon, > > Not sure about this approach. It's worth mentioning this in the vote thread > as well so that the people who voted originally have a chance to comment. > Also, we should really get input from developers of Kafka clients > (

Re: [DISCUSS] KIP-219 - Improve Quota Communication

I have to develop Real-time recommendation System. I am trying to use Apache Kafka for Website activity tracking Bu am not able to understand how can I track website activity. Please tell me the way how can I track website activity. Right now I am using Apache Kafka, apache spark, and python. pleas

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Jon, Not sure about this approach. It's worth mentioning this in the vote thread as well so that the people who voted originally have a chance to comment. Also, we should really get input from developers of Kafka clients (librdkafka, kafka-python, etc.) for this KIP. Ismael On Thu, Apr 5, 201

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi, I have been implementing KIP-219. I discussed the interface changes with Becket Qin and Dong Lin, and we decided to bump up the protocol version of ApiVersionsRequest and ApiVersionsResponse only, instead of bumping up all requests/responses that may be throttled, to indicate clients whether o

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Thanks Rajini, I updated the KIP wiki to clarify the scope of the KIP. To summarize, the current quota has a few caveats: 1. The brokers are only throttling the NEXT request even if the current request is already violating quota. So the broker will always process at least one request from the clie

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Becket, The current user quota doesn't solve the problem. But I was thinking that if we could ensure we don't read more from the network than the quota allows, we may be able to fix the issue in a different way (throttling all connections, each for a limited time prior to reading large requests

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Rajini, We are using SSL so we could use user quota. But I am not sure if that would solve the problem. The key issue in our case is that each broker can only handle ~300 MB/s of incoming bytes, but the MapReduce job is trying to push 1-2 GB/s, unless we can throttle the clients to 300 MB/s, th

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Becket, For the specific scenario that you described, would it be possible to use user quotas rather than client-id quotas? With user quotas, perhaps we can throttle more easily before reading requests as well (as you mentioned, the difficulty with client-id quota is that we have to read partia

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Since we will bump up the wire request version, another option is for clients that are sending old request versions the broker can just keep the current behavior. For clients sending the new request versions, the broker can respond then mute the channel as described in the KIP wiki. In this case, m

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Jun, Hmm, even if a connection is closed by the client when the channel is muted. After the channel is unmuted, it seems Selector.select() will detect this and close the socket. It is true that before the channel is unmuted the socket will be in a CLOSE_WAIT state though. So having an arbitrari

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Rajini, Thanks for the detail explanation. Please see the reply below: 2. Limiting the throttle time to connection.max.idle.ms on the broker side is probably fine. However, clients may have a different configuration of connection.max.idle.ms and still reconnect before the throttle time (which

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi, Jiangjie, 3. If a client closes a socket connection, typically the server only finds this out by reading from the channel and getting a negative size during the read. So, if a channel is muted by the server, the server won't be able to detect the closing of the connection by the client after t

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Becket, 2. Perhaps it is fine, though it may be better to limit throttle time to *connections.max.idle.ms* , which is typically a rather large number. 3. The maximum CLOSE_WAIT time is currently bounded by *connections.max.idle.ms

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Thanks Rajini. 1. Good point. We do need to bump up the protocol version so that the new clients do not wait for another throttle time when they are talking to old brokers. I'll update the KIP. 2. That is true. But the client was not supposed to send request to the broker during that period anywa

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi Becket, Thank you for the KIP. A few comments: 1.KIP says: "*No public interface changes are needed. We only propose behavior change on the broker side.*" But from the proposed changes, it sounds like clients will be updated to wait for throttle-time before sending next response, and also no

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Thanks for the comment, Jun, 1. Yes, you are right. This could also happen with the current quota mechanism because we are essentially muting the socket during throttle time. There might be two ways to solve this. A) use another socket to send heartbeat. B) let the GroupCoordinator know that the c

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi, Jiangjie, Thanks for bringing this up. A couple of quick thoughts. 1. If the throttle time is large, what can happen is that a consumer won't be able to heart beat to the group coordinator frequent enough. In that case, even with this KIP, it seems there could be frequent consumer group rebal

[DISCUSS] KIP-219 - Improve Quota Communication

Hi, We would like to start the discussion on KIP-219. The KIP tries to improve quota throttling time communication between brokers and clients to avoid clients timeout in case of long throttling time. The KIP link is following: https://cwiki.apache.org/confluence/display/KAFKA/KIP-219+-+Improve+