Awesome. That makes sense, thanks! John
On Fri, May 24, 2019 at 11:11 AM Haeyoon Cho <chohy0...@gmail.com> wrote: > Hi John, > > Garbage-collecting the MemoryChunks in (5. a. iv.) means that the objects > that refer to the off-heap buffers to be garbage-collected, as the > MemoryChunks itself is a JVM object that points to the native memory. Now > that I think of it, it wouldn't be necessary to explicitly do this since > shutting down the executor means the java process to terminate, which > implies that garbage collections are not needed in the first place. I guess > it is better to delete (5. a. iv) since it is too obvious. Thanks :) > > For the off-heap sorting, it is expected to be feasible with the current > design plan just like you mentioned. The MemoryChunk is intended to be a > representation of off-heap buffers (i.e. java.nio.ByteBuffer) and also a > specialized class for operations on off-heap data at the same time (swap, > compare, etc.). I will propose another design plan on this issue later for > further discussion. > > Thank you for the review! It was a really big help. > > Best, > Haeyoon > > > > 2019년 5월 24일 (금) 오전 10:03, John Yang <johnya...@gmail.com>님이 작성: > > > Hi Haeyoon, > > > > I like the overall design. I have minor questions about some details. > > > > In (5. a. iv.), what do you mean by freeing the MemoryChunks to be > > garbage-collected? My understanding was that a MemoryChunk is outside of > > the JVM heap, and thus is never accessed by the JVM garbage collector. > > Also, why does the garbage collection happen "when the executor shuts > > down"? > > > > I like the interfaces in (6. b.), and am curious to know whether they can > > be also used in performance-critical places other than > > DirectByteBufferOutputstream/SerializedMemoryStore. For example, can we > use > > the interfaces to sort data off-heap as in Apache Crail and Apache Spark > > [1.2]? > > > > [1] Sorting on a 100Gbit/s Cluster using Spark/Crail > > https://crail.incubator.apache.org/blog/2017/01/sorting.html > > [2] Project Tungsten: Bringing Apache Spark Closer to Bare Metal > > > > > https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html > > > > Thanks, > > John > > > > > > On Thu, May 23, 2019 at 11:33 AM Gyewon Lee <strayyy...@gmail.com> > wrote: > > > > > Hi, Haeyoon. > > > > > > Thanks a lot for your working. The initial design seems reasonable. > > > > > > Folks, could you give some feedbacks on the design document, so that > > > Haeyoon can quickly move onto implementation? She will welcome any > > > feedbacks, as she told us :) > > > > > > Best, > > > Gyewon > > > > > > 2019년 5월 22일 (수) 오후 7:45, Haeyoon Cho <chohy0...@gmail.com>님이 작성: > > > > > > > Hi all, > > > > > > > > Just as I mentioned in the previous email, we are about to implement > > > > components for off-heap memory management. Before that, we wrote an > > > initial > > > > design document and reported a JIRA issue for this(NEMO-388 > > > > <https://issues.apache.org/jira/browse/NEMO-388>). We welcome any > > > > feedbacks > > > > regarding the design. > > > > > > > > > > > > Best, > > > > Haeyoon > > > > > > > > > >
