Hi Haeyoon, I like the overall design. I have minor questions about some details.
In (5. a. iv.), what do you mean by freeing the MemoryChunks to be garbage-collected? My understanding was that a MemoryChunk is outside of the JVM heap, and thus is never accessed by the JVM garbage collector. Also, why does the garbage collection happen "when the executor shuts down"? I like the interfaces in (6. b.), and am curious to know whether they can be also used in performance-critical places other than DirectByteBufferOutputstream/SerializedMemoryStore. For example, can we use the interfaces to sort data off-heap as in Apache Crail and Apache Spark [1.2]? [1] Sorting on a 100Gbit/s Cluster using Spark/Crail https://crail.incubator.apache.org/blog/2017/01/sorting.html [2] Project Tungsten: Bringing Apache Spark Closer to Bare Metal https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html Thanks, John On Thu, May 23, 2019 at 11:33 AM Gyewon Lee <strayyy...@gmail.com> wrote: > Hi, Haeyoon. > > Thanks a lot for your working. The initial design seems reasonable. > > Folks, could you give some feedbacks on the design document, so that > Haeyoon can quickly move onto implementation? She will welcome any > feedbacks, as she told us :) > > Best, > Gyewon > > 2019년 5월 22일 (수) 오후 7:45, Haeyoon Cho <chohy0...@gmail.com>님이 작성: > > > Hi all, > > > > Just as I mentioned in the previous email, we are about to implement > > components for off-heap memory management. Before that, we wrote an > initial > > design document and reported a JIRA issue for this(NEMO-388 > > <https://issues.apache.org/jira/browse/NEMO-388>). We welcome any > > feedbacks > > regarding the design. > > > > > > Best, > > Haeyoon > > >
