On 21/09/2018 16:44, Andrew Dinn wrote:
Hi Alan,
Thanks for the response and apologies for failing to notice you had
posted it some days ago (doh!).
Jonathan Halliday has already explained how Red Hat might want to use
this API. Well, what he said, essentially! In particular, this model
provides a way of ensuring that raw byte data is able to be persisted
coherently from Java with the minimal possible overhead. It would be up
to client code above this layer to implement structuring mechanisms for
how those raw bytes get populated with data and to manage any associated
issues regarding atomicity, consistency and isolation (i.e. to provide
the A, C and I of ACID to this API's D).
The main point of the JEP is to ensure that this such a performant base
capability is available for known important cases where that is needed
such as, for example, a transaction manager or a distributed cache. If
equivalent middleware written in C can use persistent memory to bring
the persistent storage tier nearer to the CPU and, hence, lower data
durability overheads then we really need an equivalently performant
option in Java or risk Java dropping out as a player in those middleware
markets.
I am glad to hear that other alternatives might be available and would
be happy to consider them. However, I'm not sure that this means this
option is not still desirable, especially if it is orthogonal to those
other alternatives. Most importantly, this one has the advantage that we
know it is ready to use and will provide benefits (we have already
implemented a journaled transaction log over it with promising results
and someone from our messaging team has already been looking into using
it to persist log messages). Indeed, we also know we can use it to
provide a base for supporting all the use cases addressed by Intel's
libpmem and available to C programmers, e.g. a block store, simply by
implementing Java client libraries that provide managed access to the
persistent buffer along the same lines as the Intel C libraries.
I'm afraid I am not familiar with Panama 'scopes' and 'pointers' so I
can't really compare options here. Can you point me at any info that
explains what those terms mean and how it might be possible to use them
to access off-heap, persistent data.
I'm not questioning the need to support NVM, instead I'm trying to see
whether MappedByteBuffer is the right way to expose this in the standard
API. Buffers were designed in JSR-51 with specific use-cases in mind but
they are problematic for many off-heap cases as they aren't thread safe,
are limited to 2GB, lack confinement, only support homogeneous data (no
layout support). At the same time, Project Panama (foreign branch in
panama/dev) has the potential to provide the right API to work with
memory. I see Jonathan's mail where he seems to be using object
serialization so the solution on the table works for his use-case but it
may not be the right solution for more general multi-threaded access to
NVM. There is some interest in seeing whether this part of Project
Panama could be advanced to address many of the cases where developers
are resorting to using Unsafe today. There would of course need to be
some integration with buffers too. There's no concrete proposal/JEP at
this time, I'm just pointing out that many of the complaints about
buffers that are really cases where it's the wrong API and the real need
is something more fundamental.
So where does this leave us? If support for persistent memory is added
to FileChannel.map as we've been discussing then it may not be too bad
as the API surface is small. The API surface is just new map modes and a
MappedByteBuffer::isPersistent method. The force method that specify a
range is something useful to add to MBB anyway. If (and I hope when)
there is support for memory regions or pointers then I could imagine
re-visiting this so that there are alternative ways to get a memory
region or pointer that is backed by NVM. If the timing were different
then I think we'd skip the new map modes and we would be having a
different discussion here. An alternative is course to create the mapped
buffer via a JDK-specific API as that would be easier to deprecate and
remove in the future if needed.
I'm interested to see if there is other input on this topic before it
gets locked into extending the standard API.
-Alan.
