Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage) with check/repair/sync
Hello again... So I spent all weekend doing further tests, since this issue is really bugging me for obvious reasons. I thought it would be beneficial if I created a bug report that summarized and centralized everything in one place rather than having everything spread across several lists and posts. Here the bug report I created: https://bugzilla.kernel.org/show_bug.cgi?id=135481 If anyone has any suggestions, ideas or wants me to do further tests, please just let me know. There is not much more I can do at this point without further help/guidance. Thanks, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage) with check/repair/sync
Hello... I am rather persistent (stubborn?) when it comes to tracking down bugs, if somehow possible... and it seems it paid off... somewhat. ;-) So I did quite a lot more further tests and came up with something very interesting: As long as the RAID is in sync (as-in: sync_action=idle), I can not for the life of me trigger this issue -- the used memory still explodes to most of the RAM but it oscillates back and forth. I did very stupid things to stress the machine while dd was running as usual on the dm-crypt device. I opened a second dd instance with the same parameters on the dm-crypt device. I wrote a simple program that allocated random amounts of memory (up to 10 GiB), memset them and after a random amount of time released it again -- in a continuous loop. I put heavy network stress on the machine... whatever I could think of. No matter what, the issue did not trigger. And I repeated said tests quite a few times over extended time periods (usually an hour or so). Everything worked beautifully with nice speeds and no noticeable system slow-downs/lag. As soon as I issued a "check" to sync_action of the RAID device, it was just a matter of a second until the OOM killer kicked in and all hell broke loose again. And basically all of my tests where done while the RAID device was syncing -- due to a very unfortunate series of events. I tried to repeat that same test with an external (USB3) connected disk with a Linux s/w RAID10 over two partitions... but unfortunately that behaves rather differently. I assume it is because it is connected through USB and not SATA. While doing those tests on my RAID10 with the 4 internal SATA3 disks, you can see w/ free that the "used memory" does explode to most of the RAM and then oscillates back and forth. With the same test on the external disk through, that does not happen at all. The used memory stays pretty much constant and only the buffers vary... but most of the memory is still free in that case. I hope my persistence on the matter is not annoying and finally leads us somewhere where the real issue hides. Any suggestions, opinions and ideas are greatly appreciated as I have pretty much exhausted mine at this time. Last but not least: I switched my testing to a OpenSuSE Tumbleweed Live system (x86_64 w/ kernel 4.6.3) as Rawhide w/ 4.7.0rcX behaves rather strangely and unstable at times. Thanks, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
Hello Ondrej... On 2016-07-13 18:24, Ondrej Kozina wrote: One step after another. Sorry, it was not meant to be rude or anything... more frustration since I cannot be of more help and I really would like to jump in head-first and help fixing it... but lack the necessary insight into the kernel internals. But who knows, I started reading Robert Love's book... so, in a good decade or so. ;-)) https://marc.info/?l=linux-mm&m=146825178722612&w=2 Thanks for that link. I have to read those more closely tomorrow, since there are some nice insights into dm-crypt there. :) Still, you have to admit, it is also rather frustrating/scary if such a crucial subsystem can have bugs over several major versions that do result in complete hangs (and can thus cause corruption) and are quite easily triggerable. It does not instill too much confidence that said subsystem is so intensively used/tested after all. That's at least how I feel about it... So long, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
Hello... On 2016-07-13 15:47, Michal Hocko wrote: This is getting out of my area of expertise so I am not sure I can help you much more, I am afraid. That's okay. Thank you so much for investing the time. For what it is worth, I did some further tests and here is what I came up with: If I create the plain dm-crypt device with --perf-submit_from_crypt_cpus, I can run the tests for as long as I want but the memory problem never occurs, meaning buffer/cache increase accordingly and thus free memory decreases but used mem stays pretty constant low. Yet the problem here is, the system becomes sluggish and throughput is severely impacted. ksoftirqd is hovering at 100% the whole time. Somehow my guess is that normally dm-crypt simply takes every request, encrypts it and queues it internally by itself. And that queue is then slowly emptied to the underlying device kernel queue. That is why I am seeing the exploding increase in used memory (rather than in buffer/cache) which in the end causes a OOM situation. But that is just my guess. And IMHO that is not the right thing to do (tm), as can be seen in this case. No matter what, I have no clue how to further diagnose this issue. And given that I already had unsolvable issues with dm-crypt a couple of months ago with my old machine where the system simply hang itself or went OOM when the swap was encrypted and just a few kilobytes needed to be swapped out, I am not so sure anymore I can trust dm-crypt with a full disk encryption to the point where I feel "safe"... as-in, nothing bad will happen or the system won't suddenly hang itself due to it. Or if a bug is introduced, that it will actually be possible to diagnose it and help fix it or that it will even be eventually fixed. Which is really a pity, since I would really have liked to help solve this. With the swap issue, I did git bisects, tests, narrowed it down to kernel versions when said bug was introduced... but in the end, the bug is still present as far as I know. :( I will probably look again into ext4 fs encryption. My whole point is just that in case any of disks go faulty and needs to be replaced or sent in for warranty, I don't have to worry about mails, personal or business data still being left on the device (e.g. if it is no longer accessible or has reallocated sectors or whatever) in a readable form. Oh well. Pity, really. Thanks again, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
Hello Michal, many thanks for all your time and help on this issue. It is very much appreciated and I hope we can track this down somehow. On 2016-07-13 14:18, Michal Hocko wrote: So it seems we are accumulating bios and 256B objects. Buffer heads as well but so much. Having over 4G worth of bios sounds really suspicious. Note that they pin pages to be written so this might be consuming the rest of the unaccounted memory! So the main question is why those bios do not get dispatched or finished. Ok. It is the Block IOs that do not get completed. I do get it right that those bio-3 are already the encrypted data that should be written out but do not for some reason? I tried to figure this out myself but couldn't find anything -- what does the number "-3" state? It is the position in some chain or has it a different meaning? Do you think a trace like you mentioned would help shed some more light on this? Or would you recommend something else? I have also cc' Mike Snitzer who commented on this issue before, maybe he can see some pattern here as well. Pity that Neil Brown is no longer available as I think this is also somehow related to it being a Intel Rapid Storage RAID10... since it is the only way I can reproduce it. :( Thanks, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
Hello Michal... On 2016-07-12 16:07, Michal Hocko wrote: /proc/slabinfo could at least point on who is eating that memory. Thanks. I have made another test (and thus again put the RAID10 out of sync for the 100th time, sigh) and made regular snapshots of slabinfo which I have attached to this mail. Direct IO doesn't get throttled like buffered IO. Is buffered i/o not used in both cases if I don't explicitly request direct i/o? dd if=/dev/zero /dev/md126p5 bs=512K and dd if=/dev/zero /dev/mapper/test-device bs=512K Given that the test-device is dm-crypt on md125p5. Aren't both using buffered i/o? the number of pages under writeback was more or less same throughout the time but there are some local fluctuations when some pages do get completed. The pages under writeback are those directly destined for the disk, so after dm-crypt had done its encryption? If not you can enable allocator trace point for a particular object size (or range of sizes) and see who is requesting them. If that support is baked into the Fedora provided kernel that is. If you could give me a few hints or pointers, how to properly do a allocator trace point and get some decent data out of it, that would be nice. Thanks, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration slabinfo.txt.gz Description: GNU Zip compressed data -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
Hello Michal... On 2016-07-12 13:49, Michal Hocko wrote: I am not a storage expert (not even mention dm-crypt). But what those counters say is that the IO completion doesn't trigger so the PageWriteback flag is still set. Such a page is not reclaimable obviously. So I would check the IO delivery path and focus on the potential dm-crypt involvement if you suspect this is a contributing factor. Sounds reasonable... except that I have no clue how to trace that with the limited means I have at my disposal right now and with the limited knowledge I have of the kernel internals. ;-) Who is consuming those objects? Where is the rest 70% of memory hiding? Is there any way to get a more detailed listing of where the memory is spent while dd is running? Something I could pipe every 500ms or so for later analysis or so? Writer will get throttled but the concurrent memory consumer will not normally. So you can end up in this situation. Hm, okay. I am still confused though: If I, for example, let dd do the exact same thing on a raw partition on the RAID10, nothing like that happens. Wouldn't we have the same race and problem then too...? It is only with dm-crypt in-between that all of this shows itself. But I do somehow suspect the RAID10 Intel Rapid Storage to be the cause or at least partially. Like I said, if you have any pointers how I could further trace this or figure out who is exactly consuming what memory, that would be very helpful... Thanks. So long, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
Hello Michal... On 2016-07-12 11:50, Michal Hocko wrote: This smells like file pages are stuck in the writeback somewhere and the anon memory is not reclaimable because you do not have any swap device. Not having a swap device shouldn't be a problem -- and in this case, it would cause even more trouble as in disk i/o. What could cause the file pages to get stuck or stopped from being written to the disk? And more importantly, what is so unique/special about the Intel Rapid Storage that it happens (seemingly) exclusively with that and not the the normal Linux s/w raid support? Also, if the pages are not written to disk, shouldn't something error out or slow dd down? Obviously dd is capable of copying zeros a lot faster than they could ever be written to disk -- and still, it works just fine without dm-crypt in-between. It is only when dm-crypt /is/ involved, that the memory gets filled up and things get out of control. Thanks, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
[dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
Hello, I posted this issue already on linux-mm, linux-kernel and dm-devel a few days ago and after further investigation it seems like that this issue is somehow related to the fact that I am using an Intel Rapid Storage RAID10, so I am summarizing everything again in this mail and include linux-raid in my post. Sorry for the noise... :( I am currently setting up a new machine (since my old one broke down) and I ran into a lot of " Unable to allocate memory on node -1" warnings while using dm-crypt. I have attached as much of the full log as I could recover. The encrypted device is sitting on a RAID10 (software raid, Intel Rapid Storage). I am currently limited to testing via Linux live images since the machine is not yet properly setup but I did my tests across several of those. Steps to reproduce are: 1) cryptsetup -s 512 -d /dev/urandom -c aes-xts-plain64 open --type plain /dev/md126p5 test-device 2) dd if=/dev/zero of=/dev/mapper/test-device status=progress bs=512K While running and monitoring the memory usage with free, it can be seen that the used memory increases rapidly and after just a few seconds, the system is out of memory and page allocation failures start to be issued as well as the OOM killer gets involved. I have also seen this behavior with mkfs.ext4 being used on the same device -- at least with 1.43.1. Using direct i/o will work fine and not cause any issue. Also if dm-crypt is out of the picture, the problem does also not occur. I did further tests: 1) dd block size has no influence on the issue whatsoever 2) using dm-crypt on an image located on an ext2 on the RAID10 works fine 3) using an external (connected through USB3) hd with two partitions and using either a RAID1 or RAID10 on it via Linux s/w RAID with dm-crypt on-top, does also work fine But as soon as I use dm-crypt on the Intel Rapid Storage RAID10, the issue is 100% reproducible. I tested all of this on a Fedora Rawhide Live Image as I currently still am in the process of setting the new machine up. Those images are available here to download: download.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/x86_64/iso/ The machine itself has 32 GiB of RAM (plenty), no swap (live image) and is a 6700k on a Z170 chipset. The kernel is the default provided with the live image... right now that is a very recent git after 4.7.0rc6 but before rc7. But the issue also shows on 4.4.8 and 4.5.5. The stripe size of the RAID10 is 64k, if that matters. I am now pretty much out of ideas what else to test and where the problem could stem from. Suffice to say that this has impacted my trust in this particular setup. I hope I can help to find the cause of this. If there is anything I can do to help, please let me know. Also, since I am not subscribed to the lists right now (I have to make due with a crappy WebMail interface until everything is setup), please cc' me accordingly. Thanks a lot. With Kind Regards from Germany, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administrationPersonalities : [raid10] md126 : active raid10 sda[3] sdb[2] sdc[1] sdd[0] 3907023872 blocks super external:/md127/0 64K chunks 2 near-copies [4/4] [] md127 : inactive sdc[3](S) sdb[2](S) sda[1](S) sdd[0](S) 10064 blocks super external:imsm unused devices: nr_free_pages 7943696 nr_alloc_batch 5873 nr_inactive_anon 296 nr_active_anon 105347 nr_inactive_file 29921 nr_active_file 56980 nr_unevictable 13140 nr_mlock 2716 nr_anon_pages 107204 nr_mapped 27670 nr_file_pages 98500 nr_dirty 14 nr_writeback 0 nr_slab_reclaimable 8887 nr_slab_unreclaimable 16975 nr_page_table_pages 7137 nr_kernel_stack 490 nr_unstable 0 nr_bounce 0 nr_vmscan_write 9828326 nr_vmscan_immediate_reclaim 67360474 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 593 nr_dirtied 506466654 nr_written 506466595 nr_pages_scanned 0 numa_hit 5258670960 numa_miss 0 numa_foreign 0 numa_interleave 38217 numa_local 5258670960 numa_other 0 workingset_refault 336993 workingset_activate 61553 workingset_nodereclaim 7919435 nr_anon_transparent_hugepages 0 nr_free_cma 0 nr_dirty_threshold 1592267 nr_dirty_background_threshold 795161 pgpgin 10489537 pgpgout 2025884337 pswpin 0 pswpout 0 pgalloc_dma 684558 pgalloc_dma32 328009673 pgalloc_normal 5767713958 pgalloc_movable 0 pgfree 6106436104 pgactivate 813221 pgdeactivate 1284082 pgfault 1653795 pgmajfault 46351 pglazyfreed 0 pgrefill_dma 0 pgrefill_dma32 66114 pgrefill_normal 1407169 pgrefill_movable 0 pgsteal_kswapd_dma 0 pgsteal_kswapd_dma32 22181873 pgsteal_kswapd_normal 425875886 pgsteal_kswapd_movable 0 pgsteal_direct_dma 0 pgsteal_direct_dma32 10723905 pgsteal_direct_normal 45330060 pgsteal_direct_movable 0 pgscan_kswapd_dma 0 pgscan_kswapd_dma32 32470709 pgscan_kswapd_normal 758168190 pgscan_kswapd_movable 0 pgscan_direct_dma 0 pgscan_direct
Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt
Hello Mike... It (at least) seems like that the fact that this is a Intel Rapid Storage RAID10 does play a role after all. I did several tests this morning, after I finally managed to get another disk hooked up via USB3 (the best I could do for now). I tried dm-crypt w/ a single partition and with a Linux s/w RAID1 and RAID10. Unfortunately the RAID tests were done with several partitions that were on the same disk since that is all I have for testing. No matter what I did, I could not get to the point where all memory got exhausted... not even close. So with all tests I have done over the last couple of days, the only one were I could always hit this issue was with my original RAID10 that is a Intel Rapid Storage RAID -- and hooked up to the internal SATA3 ports instead of the USB3 port (which I hope makes no huge difference for this issue). I will post to the linux raid list in a few moments and cc' this list as well. Thanks again for the help thus far, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt
Hello Mike... On 2016-07-11 15:30, Mike Snitzer wrote: But that is expected given you're doing an unbounded buffered write to the device. What isn't expected, to me anyway, is that the mm subsystem (or the default knobs for buffered writeback) would be so aggressive about delaying writeback. Ok. But, and please correct me if I am wrong, I was under the impression that only the file caches/buffers were affected, iow, if I use free to monitor the memory usage, the used memory increases to the point where it consumes all memory, not the buffers/file caches... that is what I am seeing here. Also, if I use dd directly on the device w/o dm-crypt in-between, there is no problem. Sure, buffers increase hugely also... but only those. Why are you doing this test anyway? Such a large buffered write doesn't seem to accurately model any application I'm aware of (but obviously it should still "work"). It is not a test per se. I simply wanted to fill the partition with noise. And doing it this way is faster than using urandom or anything. ;-) That is why I stumbled over this issue in the first place. Now that is weird. Are you (or the distro you're using) setting any mm subsystem tunables to really broken values? You can see those in my initial mail. I attached the kernel warnings, all sysctl tunables and more. Maybe that helps. What is your raid10's full stripesize? 4 disks in RAID10, with a stripe size of 64k. Is your dd IO size of 512K somehow triggering excess R-M-W cycles which is exacerbating the problem? The partitions are properly aligned. And as you can see, with that stripe size, there is no issue. In the meantime I did some further tests: I created an ext2 on the partition as well as a 60GiB container image on it. I used that image with dm-crypt, same parameters as before. No matter what I do here, I cannot trigger the same behavior. Maybe it is an interaction issue between dm-crypt and the s/w RAID. But at this point, I have no idea how to further diagnose/test it. If you can point me in any direction that would be great... With Kind Regards from Germany Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt
Hello Mike... On 2016-07-11 15:18, Mike Snitzer wrote: Something must explain the execessive nature of your leak but it isn't a known issue. Since I am currently setting up the new machine, all tests were performed w/ various live cd images (Fedora Rawhide, Gentoo, ...) and I saw the exact same behavior everywhere. Have you tried running with kmemleak enabled? I would have to check if that is enabled on the live images but even if it is, how would that work? The default interval is 10min. If I fire up a dd, the memory is full within two seconds or so... and after that, the OOM killer kicks in and all hell breaks loose unfortunately. I don't think this is a particular unique issue on my side. You could, if I am right, easily try a Fedora Rawhide image and reproduce it there yourself. The only unique point here is my RAID10 which is a Intel Rapid Storage s/w RAID. I have no clue if this could indeed cause such a "bug" and how. Thanks, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt
Hello, I made a few more tests and here my observations: - kernels 4.4.8 and 4.5.5 show the same behavior - the moment dd starts, memory usage spikes rapidly and within a just a few seconds has filled up all 32 GiB of RAM - dd w/ direct i/o works just fine - mkfs.ext4 unfortunately shows the same behavior as dd w/o direct i/o and such makes creation of an ext4 fs on dm-crypt a game of luck (much more exposed so with e2fsprogs 1.43.1) I am kind of puzzled that this bug has seemingly gone so long unnoticed since it is rather severe and makes dm-crypt unusable to a certain degree for fs encryption (or at least the initial creation of the fs). Am I missing something here or doing something terribly stupid? With Kind Regards from Germany Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
[dm-devel] [4.7.0rc6] Page Allocation Failures with dm-crypt
Hello, the following went to the linux-mm and linux-kernel lists as well yesterday but for whatever reason, it never showed on dm-devel, so I am re-sending it for the third time (and last time) and hoping for the best that this time it works. I am currently setting up a new machine (since my old one broke down) and I ran into a lot of " Unable to allocate memory on node -1" warnings while using dm-crypt. I have attached as much of the full log as I could recover. The encrypted device is sitting on a RAID10 (software raid, Intel Rapid Storage). Kernel is 4.7.0rc6 as released. Steps to reproduce are: 1) cryptsetup -s 512 -d /dev/urandom -c aes-xts-plain64 open --type plain /dev/md126p5 test-device 2) dd if=/dev/zero of=/dev/mapper/test-device status=progress bs=512K While running, there are several allocation failure warnings issued to the log. Doing the same on the device w/o dm-crypt in-between does not cause any such warnings. The machine did not crash or hang but the OOM killer got involved. With that much of RAM and that very simple task, that should never have happened in the first place, imho. I tested all of this on a Fedora Rawhide Live Image as I currently still am in the process of setting the new machine up. Those images are available here to download: download.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/x86_64/iso/ The machine itself has 32 GiB of RAM (plenty), no swap (live image) and is a 6700k on a Z170 chipset. The kernel, like I said, is the default provided with the live image -- and in the version I tested, it was the released 4.7.0rc6. No other tasks were running, except for the usual. If there is anything I can do to help, please let me know. Also, since I am not subscribed to the lists right now (I have to make due with a crappy WebMail interface until everything is setup), please cc' me accordingly. Thanks a lot. With Kind Regards from Germany, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administrationPersonalities : [raid10] md126 : active raid10 sda[3] sdb[2] sdc[1] sdd[0] 3907023872 blocks super external:/md127/0 64K chunks 2 near-copies [4/4] [] md127 : inactive sdc[3](S) sdb[2](S) sda[1](S) sdd[0](S) 10064 blocks super external:imsm unused devices: nr_free_pages 7943696 nr_alloc_batch 5873 nr_inactive_anon 296 nr_active_anon 105347 nr_inactive_file 29921 nr_active_file 56980 nr_unevictable 13140 nr_mlock 2716 nr_anon_pages 107204 nr_mapped 27670 nr_file_pages 98500 nr_dirty 14 nr_writeback 0 nr_slab_reclaimable 8887 nr_slab_unreclaimable 16975 nr_page_table_pages 7137 nr_kernel_stack 490 nr_unstable 0 nr_bounce 0 nr_vmscan_write 9828326 nr_vmscan_immediate_reclaim 67360474 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 593 nr_dirtied 506466654 nr_written 506466595 nr_pages_scanned 0 numa_hit 5258670960 numa_miss 0 numa_foreign 0 numa_interleave 38217 numa_local 5258670960 numa_other 0 workingset_refault 336993 workingset_activate 61553 workingset_nodereclaim 7919435 nr_anon_transparent_hugepages 0 nr_free_cma 0 nr_dirty_threshold 1592267 nr_dirty_background_threshold 795161 pgpgin 10489537 pgpgout 2025884337 pswpin 0 pswpout 0 pgalloc_dma 684558 pgalloc_dma32 328009673 pgalloc_normal 5767713958 pgalloc_movable 0 pgfree 6106436104 pgactivate 813221 pgdeactivate 1284082 pgfault 1653795 pgmajfault 46351 pglazyfreed 0 pgrefill_dma 0 pgrefill_dma32 66114 pgrefill_normal 1407169 pgrefill_movable 0 pgsteal_kswapd_dma 0 pgsteal_kswapd_dma32 22181873 pgsteal_kswapd_normal 425875886 pgsteal_kswapd_movable 0 pgsteal_direct_dma 0 pgsteal_direct_dma32 10723905 pgsteal_direct_normal 45330060 pgsteal_direct_movable 0 pgscan_kswapd_dma 0 pgscan_kswapd_dma32 32470709 pgscan_kswapd_normal 758168190 pgscan_kswapd_movable 0 pgscan_direct_dma 0 pgscan_direct_dma32 55064390 pgscan_direct_normal 449388285 pgscan_direct_movable 0 pgscan_direct_throttle 16 zone_reclaim_failed 0 pginodesteal 329 slabs_scanned 75784518 kswapd_inodesteal 3324 kswapd_low_wmark_hit_quickly 18086579 kswapd_high_wmark_hit_quickly 562 pageoutrun 18100603 allocstall 739928 pgrotated 357590082 drop_pagecache 0 drop_slab 0 numa_pte_updates 0 numa_huge_pte_updates 0 numa_hint_faults 0 numa_hint_faults_local 0 numa_pages_migrated 0 pgmigrate_success 562476 pgmigrate_fail 34076511 compact_migrate_scanned 390290706 compact_free_scanned 17609026156 compact_isolated 37387419 compact_stall 17 compact_fail 10 compact_success 7 compact_daemon_wake 3013752 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 69728 unevictable_pgs_scanned 0 unevictable_pgs_rescued 57566 unevictable_pgs_mlocked 62928 unevictable_pgs_munlocked 59182 unevictable_pgs_cleared 18 unevictable_pgs_stranded 18 thp_fault_alloc 0 thp_fault_fallback 0 thp_collapse_alloc 0 thp_collapse_alloc_failed 0 thp_split_page 0 thp_spli
Re: [dm-devel] [4.4, 4.5, 4.6] Regression: encrypted swap (dm-crypt) freezes system while under memory pressure and swapping
Hello Ondrej...
Sorry, I somehow managed to sent you the wrong version of the little C
program. I wanted to clean it up before sending it and you got a nice
in-between version between the old and new one. :-)
Double-checked now, that's the right one. Sorry.
Have a nice day,
Matthias
--
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
services: custom software [desktop, mobile, web], server administration
#include
#include
#include
#include
int main(int argc, char** argv)
{
unsigned long chunkSize = 1 << 20; // 1 MiB per \ total mem alloc
unsigned long maxRuns = 8 * (1 << 10); // 2^10 runs / of 8 GiB
void* buffer = NULL;
if(argc == 3)
{
chunkSize = strtoul(argv[1], NULL, 10);
maxRuns = strtoul(argv[2], NULL, 10);
}
unsigned long maxMemAllocInBytes = chunkSize * maxRuns;
unsigned long maxMemAllocInMB= maxMemAllocInBytes / (1 << 20);
unsigned long curMemAllocInBytes = 0;
while(curMemAllocInBytes < maxMemAllocInBytes)
{
buffer = malloc(chunkSize);
memset(buffer, 0, chunkSize);
curMemAllocInBytes += chunkSize;
printf("%lu MB / %lu MiB allocated (%lu bytes).\n", (curMemAllocInBytes / (1 << 20)), maxMemAllocInMB, curMemAllocInBytes);
usleep(500);
}
return 0;
}
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [4.4, 4.5, 4.6] Regression: encrypted swap (dm-crypt) freezes system while under memory pressure and swapping
Hello Ondrej...
Thanks for taking the time.
On 27/04/16 17:13, Ondrej Kozina wrote:
> I observe that putting swap on top of dm-crypt (tested in current 4.6-rc
> kernel) have some unexpected side effects. Most notably it seems to me
> that oom killer joins the party quite prematurely even when there
> remains plenty of free swap space (almost all). But I didn't experience
> any permanent system freeze or crash. Just, let us dig deeper for a
> while...
Yes, 4.6-rc* behaves different from 4.4.x and >= 4.5-rc3 but it still
does not work correctly at all. You may want to try the little test
program I created and attached. I used it quite a lot to stress the system.
With that, you should have no problem seeing the described behavior on
the affected kernels. You may have to adjust the sleep a bit, depending
on your machine, though.
If there is anything else I can do to help, please let me know.
Have a nice day,
Matthias
--
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
services: custom software [desktop, mobile, web], server administration
#include
#include
#include
#include
int main(int argc, char** argv)
{
unsigned long chunkSize = 1 << 20; // 1 MiB per \ total mem alloc
unsigned long maxRuns = 8 * (1 << 10); // 2^10 runs / of 8 GiB
void* buffer = NULL;
if(argc == 3)
{
chunkSize = atol(argv[1]);
maxRuns = atol(argv[2]);
}
unsigned long maxMemAllocInMB= (chunkSize * maxRuns) / (1 << 20);
unsigned long curMemAllocInBytes = 0;
for(unsigned int i = 0; i < maxRuns; ++i)
{
buffer = malloc(chunkSize);
memset(buffer, 0, chunkSize);
curMemAllocInBytes += chunkSize;
printf("%lu MB / %lu MiB allocated (%lu bytes).\n", (curMemAllocInBytes / (1 << 20)), maxMemAllocInMB, curMemAllocInBytes);
usleep(500);
}
return 0;
}
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [4.4, 4.5, 4.6] Regression: encrypted swap (dm-crypt) freezes system while under memory pressure and swapping
Hello @all, since I haven't seen any replies to this so far, is there anything else I can do to help track this bug down? I have posted the original message to both the lkml as well as dm-devel lists, by the way, without much luck. ;-) Thanks in advance -- and sorry for kind of re-posting the issue. So long, Matthias -- Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu services: custom software [desktop, mobile, web], server administration -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
