Looks like the specific issue I was referring to was addressed in Netty 4.0.29 [1]. The comment for the commit is a little concerning:
<quote> Result: ThreadPoolCache is now also usable and so gives performance improvements when allocation and deallocation thread are different. Performance when using same thread for allocation and deallocation is noticable worse then before. </quote> We might want to do a performance run to make sure things are no worse than before. [1] https://github.com/netty/netty/commit/f765053ae740e300a6b696840d7dfe5de32afeb3 On Mon, May 15, 2017 at 5:46 PM, Parth Chandra <[email protected]> wrote: > The per thread allocation cache in Netty causes unbounded memory growth in > Drill because we pass the ownership of a buffer from one thread to another. > The version we use has a fix for the Drill use case where Netty will no > longer add a buffer to its per thread cache if the buffer was allocated by > a thread which is different from the thread freeing the buffer. > This fix was reversed in a subsequent release and the latest version has > the same issue. > There might have been a fix in Netty for this in some other place which I > am not aware of (perhaps they removed it altogether as Paul seems to have > seen). > AFAIK, we do not have a direct reference to that code in Drill's > allocator. If you try to upgrade and hit an issue, post it here. > If you are able to upgrade the Netty version, then run a longevity test to > make sure there is no 'leaking' of memory from one thread to another. > > > > On Mon, May 15, 2017 at 4:08 PM, Paul Rogers <[email protected]> wrote: > >> As it turns out, Drill makes clever use of the internal details of the >> Netty memory allocator. But, that code changed significantly in the last >> couple of years. When I attempted to upgrade, I found that the private >> features of Netty that the Drill allocator uses no longer exist in the >> latest Netty. >> >> So, someone will need to understand what that part of the Drill allocator >> does and design an alternative integration. >> >> The particular issue seems to be that Netty had a per-thread allocation >> cache which seems to not exist in the latest version. >> >> - Paul >> >> > On May 15, 2017, at 3:58 PM, Sudheesh Katkam <[email protected]> wrote: >> > >> > Hi all, >> > >> > As part of working on DRILL-5431 [1], I found a bug in Netty [2], which >> is due to be fixed in 4.0.48 [3]. Drill is currently using 4.0.27 [4]. Does >> anyone foresee issues with upgrading to the latest version of Netty? I >> noticed Apache Arrow upgraded to 4.0.41 [5]. >> > >> > Thank you, >> > Sudheesh >> > >> > [1] https://issues.apache.org/jira/browse/DRILL-5431 >> > [2] https://github.com/netty/netty/issues/6709 >> > [3] https://github.com/netty/netty/pull/6713 >> > [4] https://github.com/apache/drill/blob/master/pom.xml#L550 >> > [5] https://github.com/apache/arrow/commit/3487c2f0cdc2297a80ba3 >> 525c192745313b3da48 >> >> >
