Hi Alan
I'm a middleware engineer (transaction engine, message queues, etc) and
I evolved the current API design whilst making some of Red Hat's Jakarta
EE stack work with persistent memory. It's a good fit for our needs
because it pretty much matches they way we currently do off-heap and
persistent storage, so porting existing code is a breeze. For anything
that is a 'make this bunch of bytes persistent' use case there isn't
really a complex API. We're not trying to pass data structures to and
fro as we would when calling a richer C library. The serialization layer
takes care of flattening all the structures to an opaque byte[] or
ByteBuffer already. We just need to be able to reason about the
persistence guarantees the same way we can with the existing sync()
call. We already take care of the threading, since existing storage
solutions wouldn't work without those safeguards anyhow. So, there are
certainly some use cases for which the current API is a good fit,
because those are the ones I designed it for, based on code that already
uses and copes with the limitations of MappedByteBuffer.
However... There are cases where we may want to get further
optimizations by eliding the serialization to byte[]/ByteBuffer and
instead be able to access persistent memory *as objects*. That's a
harder problem and may involve language integration rather than just API
changes, for example being able to allocate an object whose state
(primitive fields, perhaps also object pointers) is backed by an
(optionally explicitly specified area) of pmem. It's definitely a more
powerful model, but also a much bigger problem to chew on.
Some halfway solution in which we can use Java objects to point into
specific areas of memory in a typesafe way (e.g. 'that pmem address
should be considered an int') would seem to be something that Panama
could overlap with, but it's a convenience layer that could also be
modelled by putting higher level abstractions over the proposed low
level API. Over time we may have e.g. PersistentLong in the same way
that today we have AtomicLong, but it's something that could be tested
out in a 3rd party library initially and then migrated into the standard
library if it's shown to be useful.
Is the proposed API sufficient for all use cases? Probably not. But it's
useful for some and, so far as I can tell, non-harmful to others. Under
the new release model what we have now is useful in its own right and
should ship sooner rather than later, with additional functionality
following later in a modular, agile fashion? I don't really see
sufficient advantage in holding this pending e.g. investigation of
integration with Panama, though that's definitely an interesting avenue
for future work.
Regards
Jonathan
On 10/09/2018 19:05, Alan Bateman wrote:
...
I realize we've been through several iterations on this but I'm now
wondering if the MappedByteBuffer is the right API. As you've shown,
it's straight forward to map a region of NVM and use the existing API,
I'm just not sure if it's the right API. I think I'd like to see a few
examples of how the API might be used. ByteBuffers aren't intended for
use by concurrent threads and I just wonder if the examples might need
that. I also wonder if there is a possible connection with work in
Project Panama and whether it's worth exploring if its scopes and
pointers could be used to backed by NVM. The Risks and Assumption
section mentions the 2GB limit which is another reminder that the MBB
API may not be the right API.
--
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
