Re: spark binary map

2021-10-16 Thread Reynold Xin
Read up on Unsafe here: https://mechanical-sympathy.blogspot.com/ On Sat, Oct 16, 2021 at 12:41 AM, Rohan Bajaj < rohanbaja...@gmail.com > wrote: > > In 2015 Reynold Xin made improvements to Spark and it was basically moving > some structures that were on the java heap and moving them off heap.

spark binary map

2021-10-16 Thread Rohan Bajaj
In 2015 Reynold Xin made improvements to Spark and it was basically moving some structures that were on the java heap and moving them off heap. In particular it seemed like the memory did not require any serialization/deserialization. How was the performed? Was the data memory mapped? If it was,