NSAmelchev commented on code in PR #13710:
URL: https://github.com/apache/ignite/pull/13710#discussion_r4445537670
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotRestoreProcess.java:
##########
@@ -1797,6 +1786,52 @@ private void copyLocalAsync(
});
}
+ /**
+ * Copies a partition of a compressed cache group page by page writing
only the compressed bytes of each page, so the
+ * rest of the page is left unallocated by the file system and the copy is
sparse without punching holes in it.
+ */
+ private void copyCompressed(SnapshotRestoreContext opCtx, int grpId, File
from, File to) {
+ FileIOFactory ioFactory =
ctx.cache().context().snapshotMgr().ioFactory();
+ FilePageStoreManager storeMgr =
(FilePageStoreManager)ctx.cache().context().pageStore();
+ int pageSize = ctx.cache().context().database().pageSize();
+
+ try (FileIO src = ioFactory.create(from, READ); FileIO dest =
ioFactory.create(to, CREATE, WRITE)) {
+ ByteBuffer buf = locBuff.get();
+
+ buf.clear();
+ src.readFully(buf, 0);
+ buf.flip();
+
+ buf.getLong(); // Signature.
+
+ int hdrSize = storeMgr.getPageStoreFactory(grpId,
null).headerSize(buf.getInt());
+
+ buf.rewind();
+ buf.limit(hdrSize);
+ dest.writeFully(buf, 0);
Review Comment:
Yes, this block only finds the header size and copies the header as is. The
size depends on the page store version stored in the file (V1 or V2).
FileVersionCheckingFactory#createPageStore reads the version too, but it opens
the file with CREATE, READ and WRITE, which fails on a read-only snapshot. So
the version is read here and passed to headerSize()
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotRestoreProcess.java:
##########
@@ -1797,6 +1786,52 @@ private void copyLocalAsync(
});
}
+ /**
+ * Copies a partition of a compressed cache group page by page writing
only the compressed bytes of each page, so the
+ * rest of the page is left unallocated by the file system and the copy is
sparse without punching holes in it.
+ */
+ private void copyCompressed(SnapshotRestoreContext opCtx, int grpId, File
from, File to) {
+ FileIOFactory ioFactory =
ctx.cache().context().snapshotMgr().ioFactory();
+ FilePageStoreManager storeMgr =
(FilePageStoreManager)ctx.cache().context().pageStore();
+ int pageSize = ctx.cache().context().database().pageSize();
+
+ try (FileIO src = ioFactory.create(from, READ); FileIO dest =
ioFactory.create(to, CREATE, WRITE)) {
+ ByteBuffer buf = locBuff.get();
+
+ buf.clear();
+ src.readFully(buf, 0);
+ buf.flip();
+
+ buf.getLong(); // Signature.
+
+ int hdrSize = storeMgr.getPageStoreFactory(grpId,
null).headerSize(buf.getInt());
+
+ buf.rewind();
+ buf.limit(hdrSize);
+ dest.writeFully(buf, 0);
+
+ for (long off = hdrSize, len = src.size(); off < len; off +=
pageSize) {
+ if (opCtx.stopChecker.getAsBoolean())
+ throw new IgniteInterruptedException("The operation has
been stopped on copy file: " + from.getAbsolutePath());
+
+ if (Thread.interrupted())
+ throw new IgniteInterruptedException("Thread has been
interrupted: " + Thread.currentThread().getName());
+
+ buf.clear();
+ src.readFully(buf, off);
+ buf.flip();
+
+ if (PageIO.getCompressionType(buf) !=
CompressionProcessor.UNCOMPRESSED_PAGE)
+ buf.limit(PageIO.getCompressedSize(buf));
Review Comment:
You’re right, skipping bytes doesn’t guarantee holes. On XFS, extending
writes trigger speculative preallocation, and the blocks between pages stay
allocated: a 100 MiB partition with 50 MiB of compressed data took 100 MiB,
even after a remount (see “speculative preallocation” in
https://xfs.org/index.php/XFS_FAQ). ext4 isn’t affected. The file now gets its
full size before the pages are written, so the writes no longer extend it, and
the same file takes 50 MiB on XFS. Calling punchHole() per page gives the same
size, but brings back one fallocate per page. Added
SnapshotCompressionRestoreSizeTest with partitions large enough to trigger the
preallocation: it fails on XFS without the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]