selectively reading from sockets achieves memory control (up to and not including talk of (de)compression)
this is exactly what i (also, even mostly) did for kip-72 - which i hope in itself should be a reason to think about both KIPs at the same time because the changes will be similar (at least in intent) and might result in duplicated effort. a pool API is a way to "scale" all the way from just maintaining a variable holding amount of available memory (which is what my current kip-72 code does and what this kip proposes IIUC) all the way up to actually re-using buffers without any changes to the code using the pool - just drop in a different pool impl. for "edge nodes" (producer/consumer) the performance gain in actually pooling large buffers may be arguable, but i suspect for brokers regularly operating on 1MB-sized requests (which is the norm at linkedin) the resulting memory fragmentation is an actual bottleneck (i have initial micro-benchmark results to back this up but have not had the time to do a full profiling run). so basically I'm saying we may be doing (very) similar things in mostly the same areas of code. On Wed, Nov 2, 2016 at 11:35 AM, Mickael Maison <mickael.mai...@gmail.com> wrote: > electively reading from the socket should enable to > control the memory usage without impacting performance. I've had look > at that today and I can see how that would work. > I'll update the KIP accordingly tomorrow. >
