Hi Haeyoon, Thanks for sharing the analysis. It is very nice. AFAIK, we have only used jstat to monitor the memory usage of Nemo executors. It is great to see how the YourKit profiler can be used to analyze and visualize memory usage to gain deeper insights. I think analysis like this will be quite useful for understanding 'why' job completion times differ. Once the PR is created, I'll also run some experiments using the TPC-H benchmark and the LargeShufflePass to see how much the performance improves for such types of workloads and optimizations.
Cheers, John On Wed, May 22, 2019 at 11:40 AM Gyewon Lee <gw...@apache.org> wrote: > Hello, Haeyoon. > > Nice work! The initial result seems quite promising. I will look forward to > your PR :) > > Best, > Gyewon > > 2019년 5월 22일 (수) 오전 11:33, Haeyoon Cho <chohy0...@gmail.com>님이 작성: > > > Hi, > > The dev mail system seems to have had temporal error sending the attached > > file. > > I am attaching the file once again, also a google drive link just in > case: > > Link > > < > https://drive.google.com/file/d/1zKzNjIF9Wl2wvbNzXMCjLOqs-f429r8G/view?usp=sharing > > > > > > Best, > > Haeyoon > > > > 2019년 5월 22일 (수) 오전 10:22, Haeyoon Cho <chohy0...@gmail.com>님이 작성: > > > >> Hi all, > >> > >> We implemented off-heap SerializedMemoryStore and did some experiments > to > >> see the difference it brought. The good news is that JCT was decreased > >> because of reduced the JVM garbage collection time. I have attached a > >> detailed report, including the graphs regarding the experiment. > >> > >> PR will be available after implementing MemoryManager(alias) for the > >> management of the native memory region, since the current implementation > >> does not control the off-heap memory usage. > >> > >> Best, > >> Haeyoon > >> > >> >
