On Tue, Jan 10, 2017 at 7:08 PM, Robert Varga <n...@hq.sk> wrote: > On 01/10/2017 09:20 PM, Robert Varga wrote: > > On 01/10/2017 08:16 PM, Tom Pantelis wrote: > >> Since the length field of an array is an int, it's constrained to > >> Integer.MAX_VALUE (~2G). To handle snapshots larger than 2G we'll have > >> to chuck it into multiple byte[]. > > That could get funky on the reassembly side of things. If we go that > > route, we may as well switch to something more friendly, like > > java.nio.ByteBuffer(), too. > > Thinking about it a bit more, we really need some way of streaming the > snapshot. > > Tom, is there something in akka persistence which would allow us to feed > the data directly to an OutputStream? > > If not, then would it be okay to use something else for snapshots? > > We could get rid of the intermediate buffer on both snapshot and recovery. > > yeah I was thinking about that too. Instead of serializing and storing the byte[] in Snapshot we should be able to store the root NormalizedNode and stream it directly to the ObjectOutputStream when akka serializes to persistence. I hope the ObjectOutputStream is backed by a FileOutputStream and not a ByteArrayOutputStream :)
However we will run into the same issue when installing a snapshot on a follower. We do chunk that over the wire (actually via an InstallSnapshot message), as per raft, but we use the fully serialized byte[] to determine the # of chunks and drive it. We'd have to think of some other way. Perhaps we can use a temporary RandomAccessFile to stage it. > Bye, > Robert > >
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
