Creating compressed backing_store as swapfile
Hi, I have one requirement: I wanted to have a swapfile (64MB to 256MB) on my system. But I wanted the data to be compressed and stored on the disk in my swapfile. [Similar to zram, but compressed data should be moved to disk, instead of RAM]. Note: I wanted to optimize RAM space, so performance is not important right now for our requirement. So, what are the options available, to perform this in 4.x kernel version. My Kernel: 4.9.x Board: any - (arm64 mostly). As I know, following are the choices: 1) ZRAM: But it compresses and store data in RAM itself 2) frontswap + zswap : Didn't explore much on this, not sure if this is helpful for our case. 3) Manually creating swapfile: but how to compress it ? 4) Any other options ? Thanks, Pintu ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Creating compressed backing_store as swapfile
On 11/5/2018 10:58 AM, Adam Borowski wrote: On Mon, Nov 05, 2018 at 08:31:46PM +0530, Pintu Agarwal wrote: Hi, I have one requirement: I wanted to have a swapfile (64MB to 256MB) on my system. But I wanted the data to be compressed and stored on the disk in my swapfile. [Similar to zram, but compressed data should be moved to disk, instead of RAM]. Note: I wanted to optimize RAM space, so performance is not important right now for our requirement. So, what are the options available, to perform this in 4.x kernel version. My Kernel: 4.9.x Board: any - (arm64 mostly). As I know, following are the choices: 1) ZRAM: But it compresses and store data in RAM itself 2) frontswap + zswap : Didn't explore much on this, not sure if this is helpful for our case. 3) Manually creating swapfile: but how to compress it ? 4) Any other options ? Loop device on any filesystem that can compress (such as btrfs)? The performance would suck, though -- besides the indirection of loop, btrfs compresses in blocks of 128KB while swap wants 4KB writes. Other similar option is qemu-nbd -- it can use compressed disk images and expose them to a (local) nbd client. Swap on any type of a networked storage device (NBD, iSCSI, ATAoE, etc) served from the local system is _really_ risky. The moment the local server process for the storage device gets forced out to swap, you deadlock. Performance isn't _too_ bad for the BTRFS case though (I've actually tested this before), just make sure you disable direct I/O mode on the loop device, otherwise you run the risk of data corruption. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Creating compressed backing_store as swapfile
On Mon, 05 Nov 2018 20:31:46 +0530, Pintu Agarwal said: > I wanted to have a swapfile (64MB to 256MB) on my system. > But I wanted the data to be compressed and stored on the disk in my swapfile. > [Similar to zram, but compressed data should be moved to disk, instead of > RAM]. What platform are you on that you're both storage constrained enough to need swap, and also so short on disk space that compressing it makes sense? Understanding the hardware constraints here would help in advising you. > Note: I wanted to optimize RAM space, so performance is not important > right now for our requirement. > > So, what are the options available, to perform this in 4.x kernel version. > My Kernel: 4.9.x Given that this is a greenfield development, why are you picking a kernel that's 2 years out of date? You *do* realize that 4.9.135 does *not* contain all the bugfixes since then, only that relatively small subset that qualify for 'stable' (see Documentation/process/stable-kernel-rules.rst for the gory details). One possible total hack would be to simply use a file-based swap area, but put the file on a filesystem that supports automatic inline compression. Note that this will probably *totally* suck on performance, because there's no good way to find where 4K block 11,493 starts inside the compressed file, so it would have to read/decompress from the file beginning. Also, if you write data to a previously unused location (or even a previously used spot that compressed the 4K page to a different length), you have a bad time inserting it. (Note that zram can avoid most of this because it can (a) keep a table of pointers to where each page starts and (b) it isn't constrained to writing to 4K blocks on disk, so if the current compression takes a 4K page down to 1,283 bytes, it doesn't have to care *too* much if it stores that someplace that crosses a page boundary. Another thing that you will need to worry about is what happens in low-memory situations - the time you *most* need to do a swap operation, you may not have enough memory to do the I/O. zram basically makes sure it *has* the memory needed beforehand, and swap directly to pre-allocated disk doesn't need much additional memory. pgp9ikzGEsGfC.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Creating compressed backing_store as swapfile
On Mon, 05 Nov 2018 11:07:12 -0500, "Austin S. Hemmelgarn" said: > Performance isn't _too_ bad for the BTRFS case though (I've actually > tested this before), just make sure you disable direct I/O mode on the > loop device, otherwise you run the risk of data corruption. Did you test that for random-access. or just sequential read/write? (Also, see the note in my other mail regarding doing a random-access write to the middle of the file...) pgp9db3HLgH6J.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Creating compressed backing_store as swapfile
On 11/5/2018 11:14 AM, valdis.kletni...@vt.edu wrote: On Mon, 05 Nov 2018 11:07:12 -0500, "Austin S. Hemmelgarn" said: Performance isn't _too_ bad for the BTRFS case though (I've actually tested this before), just make sure you disable direct I/O mode on the loop device, otherwise you run the risk of data corruption. Did you test that for random-access. or just sequential read/write? (Also, see the note in my other mail regarding doing a random-access write to the middle of the file...) Actual swap usage. About 16 months ago, I had been running a couple of Intel NUC5PPYH boxes (Pentium N3700 CPU's, 4GB of DDR3-1333 RAM) for some network prototyping. On both, I had swap set up to use a file on BTRFS via a loop device, and I made a point to test both with LZ4 inline compression and without any compression, and saw negligible performance differences (less than 1% in most cases). It was, of course, significantly worse than running on ext4, but on a system that's so resource constrained that both storage and memory are at a premium to this degree, the performance hit is probably going to be worth it. Also, it's probably worth noting that BTRFS doesn't need to decompress the entire file to read or write blocks in the middle, it splits the file into 128k blocks and compresses each of those independent of the others, so it can just decompress the 128k block that holds the actual block that's needed. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Creating compressed backing_store as swapfile
On Mon, 05 Nov 2018 11:28:49 -0500, "Austin S. Hemmelgarn" said: > Also, it's probably worth noting that BTRFS doesn't need to decompress > the entire file to read or write blocks in the middle, it splits the > file into 128k blocks and compresses each of those independent of the > others, so it can just decompress the 128k block that holds the actual > block that's needed. Presumably it does something sane with block allocation for the now-compressed 128K that's presumably much smaller. Also, that limits the damage from writing to the middle of a compression unit That *does* however increase the memory requirement - you can OOM or deadlock if your read/write from the swap needs an additional 128K for the compression buffer at an inconvenient time... pgpLJYMAk5PeA.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Creating compressed backing_store as swapfile
On 11/5/2018 11:53 AM, valdis.kletni...@vt.edu wrote: On Mon, 05 Nov 2018 11:28:49 -0500, "Austin S. Hemmelgarn" said: Also, it's probably worth noting that BTRFS doesn't need to decompress the entire file to read or write blocks in the middle, it splits the file into 128k blocks and compresses each of those independent of the others, so it can just decompress the 128k block that holds the actual block that's needed. Presumably it does something sane with block allocation for the now-compressed 128K that's presumably much smaller. Also, that limits the damage from writing to the middle of a compression unit That *does* however increase the memory requirement - you can OOM or deadlock if your read/write from the swap needs an additional 128K for the compression buffer at an inconvenient time... Indeed, and I can't really comment on how it might behave under those circumstances (the systems I did the testing on never saw memory pressure quite _that_ bad, and I had them set up to swap things out pretty early and really aggressively). ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: error: /boot/vmlinuz-4.18.16-1-default has invalid signature
if I had the uefi enabled, then I deactivated it but I set the error of the grub, I turned it on again and it is no longer possible to enter D: neither going back to re-configure the system El jue., 1 nov. 2018 a las 0:52, Mulyadi Santosa () escribió: > > > On Tue, Oct 30, 2018 at 5:24 AM Manuel Quintero Fonseca > wrote: > >> I Compiled the stable kernel obtained from kernel.org, when starting the >> grub >> the error came out: >> >> Loadin Linux 4.18.16-1-default .. >> error: /boot/vmlinuz-4.18.16-1-default has invalid signature. >> loading initial ramdisk ... >> error: the kernel must be loaded first >> >> Just a hunch, are you using UEFI/secure boot? > > >> >> The steps for the process were: >> >> $ git clone git: // >> git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git >> cd linux-stable >> >> $ git checkout -b stable v4.18.16 >> >> $ cp / boot / config-`uname -r` * .config >> >> $ make >> #make modules_install install >> >> #reboot >> >> with that it is supposed to be compiled and installed in the system and I >> configure the grub >> the source: https://kernelnewbies.org/KernelBuild >> >> openSUSE TW >> >> NOTE: I tested it on a virtual machine in the same way, and did not frame >> the error >> >> regards >> ___ >> Kernelnewbies mailing list >> Kernelnewbies@kernelnewbies.org >> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies >> > > > -- > regards, > > Mulyadi Santosa > Freelance Linux trainer and consultant > > blog: the-hydra.blogspot.com > training: mulyaditraining.blogspot.com > ___ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
