Hi Alan

I'm a middleware engineer (transaction engine, message queues, etc) and I evolved the current API design whilst making some of Red Hat's Jakarta EE stack work with persistent memory. It's a good fit for our needs because it pretty much matches they way we currently do off-heap and persistent storage, so porting existing code is a breeze. For anything that is a 'make this bunch of bytes persistent' use case there isn't really a complex API. We're not trying to pass data structures to and fro as we would when calling a richer C library. The serialization layer takes care of flattening all the structures to an opaque byte[] or ByteBuffer already. We just need to be able to reason about the persistence guarantees the same way we can with the existing sync() call. We already take care of the threading, since existing storage solutions wouldn't work without those safeguards anyhow. So, there are certainly some use cases for which the current API is a good fit, because those are the ones I designed it for, based on code that already uses and copes with the limitations of MappedByteBuffer.

However... There are cases where we may want to get further optimizations by eliding the serialization to byte[]/ByteBuffer and instead be able to access persistent memory *as objects*. That's a harder problem and may involve language integration rather than just API changes, for example being able to allocate an object whose state (primitive fields, perhaps also object pointers) is backed by an (optionally explicitly specified area) of pmem. It's definitely a more powerful model, but also a much bigger problem to chew on.

Some halfway solution in which we can use Java objects to point into specific areas of memory in a typesafe way (e.g. 'that pmem address should be considered an int') would seem to be something that Panama could overlap with, but it's a convenience layer that could also be modelled by putting higher level abstractions over the proposed low level API. Over time we may have e.g. PersistentLong in the same way that today we have AtomicLong, but it's something that could be tested out in a 3rd party library initially and then migrated into the standard library if it's shown to be useful.

Is the proposed API sufficient for all use cases? Probably not. But it's useful for some and, so far as I can tell, non-harmful to others. Under the new release model what we have now is useful in its own right and should ship sooner rather than later, with additional functionality following later in a modular, agile fashion? I don't really see sufficient advantage in holding this pending e.g. investigation of integration with Panama, though that's definitely an interesting avenue for future work.

Regards

Jonathan

On 10/09/2018 19:05, Alan Bateman wrote:
...
I realize we've been through several iterations on this but I'm now wondering if the MappedByteBuffer is the right API. As you've shown, it's straight forward to map a region of NVM and use the existing API, I'm just not sure if it's the right API. I think I'd like to see a few examples of how the API might be used. ByteBuffers aren't intended for use by concurrent threads and I just wonder if the examples might need that. I also wonder if there is a possible connection with work in Project Panama and whether it's worth exploring if its scopes and pointers could be used to backed by NVM. The Risks and Assumption section mentions the 2GB limit which is another reminder that the MBB API may not be the right API.


--
Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

Reply via email to