+1 On Wed, Nov 2, 2016 at 10:34 AM, radai <radai.rosenbl...@gmail.com> wrote:
> In my opinion a lot of kafka configuration options were added using the > "minimal diff" approach, which results in very nuanced and complicated > configs required to indirectly achieve some goal. case in point - timeouts. > > The goal here is to control the memory requirement. the 1st config was max > size of a single request, now the proposal is to control the number of > those in flight - which is inaccurate (you dont know the actual size and > must over-estimate), would have an impact on throughput in case of > over-estimation, and also fails to completely achieve the goal (what about > decompression?) > > I think a memory pool in combination with Jay's proposal to only pick up > from socket conditionally when memory is available is the correct approach > - it deals with the problem directly and would result in a simler and more > understandable configuration (a single property for max memory > consumption). > > in the future the accuracy of the limit can be improved by, for example, > declaring both the compressed _AND UNCOMPRESSED_ sizes up front, so that we > can pick up from socket when we have enough memory to decompress as well - > this would obviously be a wire format change and outside the scope here, > but my point is that it could be done without adding any new configs) > > On Mon, Oct 31, 2016 at 10:25 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > Agreed with this approach. > > One detail to be wary of is that since we multiplex various other > requests > > (e.g., heartbeats, offset commits, metadata, etc.) over the client that > > connects to the coordinator this could delay some of these critical > > requests. Realistically I don't think it will be an issue except in > extreme > > scenarios where someone sets the memory limit to be unreasonably low. > > > > Thanks, > > > > Joel > > > > On Sun, Oct 30, 2016 at 12:32 PM, Jun Rao <j...@confluent.io> wrote: > > > > > Hi, Mickael, > > > > > > I agree with others that it's better to be able to control the bytes > the > > > consumer can read from sockets, instead of limiting the fetch requests. > > > KIP-72 has a proposal to bound the memory size at the socket selector > > > level. Perhaps that can be leveraged in this KIP too. > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 72%3A+Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests > > > > > > Thanks, > > > > > > Jun > > > > > > On Thu, Oct 27, 2016 at 3:23 PM, Jay Kreps <j...@confluent.io> wrote: > > > > > > > This is a good observation on limiting total memory usage. If I > > > understand > > > > the proposal I think it is that the consumer client would stop > sending > > > > fetch requests once a certain number of in-flight fetch requests is > > met. > > > I > > > > think a better approach would be to always issue one fetch request to > > > each > > > > broker immediately, allow the server to process that request, and > send > > > data > > > > back to the local machine where it would be stored in the socket > buffer > > > (up > > > > to that buffer size). Instead of throttling the requests sent, the > > > consumer > > > > should ideally throttle the responses read from the socket buffer at > > any > > > > given time. That is, in a single poll call, rather than reading from > > > every > > > > single socket it should just read until it has a given amount of > memory > > > > used then bail out early. It can come back and read more from the > other > > > > sockets after those messages are processed. > > > > > > > > The advantage of this approach is that you don't incur the additional > > > > latency. > > > > > > > > -Jay > > > > > > > > On Mon, Oct 10, 2016 at 6:41 AM, Mickael Maison < > > > mickael.mai...@gmail.com> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I would like to discuss the following KIP proposal: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > 81%3A+Max+in-flight+fetches > > > > > > > > > > > > > > > Feedback and comments are welcome. > > > > > Thanks ! > > > > > > > > > > Mickael > > > > > > > > > > > > > > > -- *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>
