Hi John, Garbage-collecting the MemoryChunks in (5. a. iv.) means that the objects that refer to the off-heap buffers to be garbage-collected, as the MemoryChunks itself is a JVM object that points to the native memory. Now that I think of it, it wouldn't be necessary to explicitly do this since shutting down the executor means the java process to terminate, which implies that garbage collections are not needed in the first place. I guess it is better to delete (5. a. iv) since it is too obvious. Thanks :)
For the off-heap sorting, it is expected to be feasible with the current design plan just like you mentioned. The MemoryChunk is intended to be a representation of off-heap buffers (i.e. java.nio.ByteBuffer) and also a specialized class for operations on off-heap data at the same time (swap, compare, etc.). I will propose another design plan on this issue later for further discussion. Thank you for the review! It was a really big help. Best, Haeyoon 2019년 5월 24일 (금) 오전 10:03, John Yang <johnya...@gmail.com>님이 작성: > Hi Haeyoon, > > I like the overall design. I have minor questions about some details. > > In (5. a. iv.), what do you mean by freeing the MemoryChunks to be > garbage-collected? My understanding was that a MemoryChunk is outside of > the JVM heap, and thus is never accessed by the JVM garbage collector. > Also, why does the garbage collection happen "when the executor shuts > down"? > > I like the interfaces in (6. b.), and am curious to know whether they can > be also used in performance-critical places other than > DirectByteBufferOutputstream/SerializedMemoryStore. For example, can we use > the interfaces to sort data off-heap as in Apache Crail and Apache Spark > [1.2]? > > [1] Sorting on a 100Gbit/s Cluster using Spark/Crail > https://crail.incubator.apache.org/blog/2017/01/sorting.html > [2] Project Tungsten: Bringing Apache Spark Closer to Bare Metal > > https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html > > Thanks, > John > > > On Thu, May 23, 2019 at 11:33 AM Gyewon Lee <strayyy...@gmail.com> wrote: > > > Hi, Haeyoon. > > > > Thanks a lot for your working. The initial design seems reasonable. > > > > Folks, could you give some feedbacks on the design document, so that > > Haeyoon can quickly move onto implementation? She will welcome any > > feedbacks, as she told us :) > > > > Best, > > Gyewon > > > > 2019년 5월 22일 (수) 오후 7:45, Haeyoon Cho <chohy0...@gmail.com>님이 작성: > > > > > Hi all, > > > > > > Just as I mentioned in the previous email, we are about to implement > > > components for off-heap memory management. Before that, we wrote an > > initial > > > design document and reported a JIRA issue for this(NEMO-388 > > > <https://issues.apache.org/jira/browse/NEMO-388>). We welcome any > > > feedbacks > > > regarding the design. > > > > > > > > > Best, > > > Haeyoon > > > > > >
