[Bug 1764246] Re: kdump kernel panics on Bionic
I had two up-to-date Azure instances (16.04 and 18.04) where Linux- crashdump was installed. Both failed, with the following being shown at the end of the serial console: [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 18.407975] Out of memory: Kill process 496 (cloud-init) score 127 or sacrifice child [ 18.417791] Killed process 496 (cloud-init) total-vm:66840kB, anon-rss:12720kB, file-rss:0kB, shmem-rss:0kB [FAILED] Failed to start Initial cloud-init job (pre-networking). See 'systemctl status cloud-init-local.service' for details. [ OK ] Reached target Network (Pre). Starting Network Service... [ OK ] Started Network Service. [ OK ] Reached target Network. Starting Wait for Network to be Configured... [ OK ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. [ 37.269354] Out of memory: Kill process 657 (snap) score 40 or sacrifice child [ 37.282042] Killed process 657 (snap) total-vm:95160kB, anon-rss:4008kB, file-rss:0kB, shmem-rss:0kB [* ] (2 of 3) A start job is running for… to be Configured (32s / no limit)[ 47.413761] Out of memory: Kill process 656 (snap) score 41 or sacrifice child [ 47.422837] Killed process 656 (snap) total-vm:242624kB, anon-rss:4072kB, file-rss:0kB, shmem-rss:0kB [ OK ] Found device Virtual_Disk 1. Starting File System Check on /dev/disk/cloud/azure_resource-part1... [ *** ] (3 of 3) A start job is running for…re_resource-part1 (44s / no limit)[ 56.870810] Out of memory: Kill process 662 (snap) score 41 or sacrifice child [ 56.887294] Killed process 662 (snap) total-vm:242880kB, anon-rss:4116kB, file-rss:0kB, shmem-rss:0kB [ OK ] Started File System Check Daemon to report status. [ 71.643108] Out of memory: Kill process 660 (snap) score 42 or sacrifice child [ 71.652187] Killed process 660 (snap) total-vm:242880kB, anon-rss:4220kB, file-rss:0kB, shmem-rss:0kB [ OK ] Started File System Check on /dev/disk/cloud/azure_resource-part1. [ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device. [DEPEND] Dependency failed for /boot/efi. [DEPEND] Dependency failed for Local File Systems. [ OK ] Started Emergency Shell. [ OK ] Reached target Emergency Mode. [ 114.872426] Out of memory: Kill process 655 (snap) score 49 or sacrifice child [ 114.881579] Killed process 655 (snap) total-vm:243936kB, anon-rss:4844kB, file-rss:0kB, shmem-rss:0kB Starting Tell Plymouth To Write Out Runtime Data... Starting Create Volatile Files and Directories... Starting AppArmor initialization... You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode. Press Enter for maintenance (or press Control-D to continue): So, no more kernel panics, but (1) still no kdump saved; (2) the servers end waiting for user intervention (which is quite bad for a cloud instance). Both servers had a cmdline with "crashkernel=512M-:192M". I edited /etc/default/grub.d/kdump-tools.cfg, changed crashkernel to 256M, and ran update-grub. Re-tested, and it worked. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
OK, so it seems it's really the minimum of reserved memory kernel needs in Bionic that is affecting the success of kdump here. Thanks for the report Gaëtan -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
By increasing the size reserved to 256MB it's working. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
I got the same behavior on Xenial with kernel 4.15.0-43-generic. [4.598980] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [4.601946] Kernel panic - not syncing: Out of memory and no killable processes... [4.601946] [4.605054] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0-43-generic #46~16.04.1-Ubuntu [4.605943] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 10/17/2018 [4.605943] Call Trace: [4.605943] dump_stack+0x63/0x8b [4.605943] panic+0xe4/0x244 [4.60header+0x18f/0x285 [4.605943] out_of_memory+0x364/0x4c0 [4.605943] __alloc_pages_slowpath+0xda2/0xe90 [4.605943] ? radix_tree_extend+0xda/0x160 [4.605943] __alloc_pages_nodemask+0x265/0x280 [4.605943] alloc_page_interleave+0x17/0x80 [4.605943] alloc_pages_current+0xc8/0xe0 [4.605943] __page_cache_alloc+0x86/0x90 [4.605943] pagecache_get_page+0x88/0x2c0 [4.605943] grab_cache_page_write_begin+0x23/0x40 [4.605943] simple_write_begin+0x29/0x1b0 [4.605943] generic_perform_write+0xba/0x1b0 [4.605943] __generic_file_write_iter+0x1a6/0x1f0 [4.605943] ? md_run_setup+0x99/0x99 [4.605943] generic_file_write_iter+0xf1/0x1d0 [4.605943] new_sync_write+0xe5/0x140 [4.605943] __vfs_write+0x29/0x40 [4.605943] vfs_write+0xb8/0x1b0 [4.605943] ? zlib_updatewindow+0x2d/0xd0 [4.605943] ? md_run_setup+0x99/0x99 [4.605943] SyS_write+0x55/0xc0 [4.605943] xwrite+0x2e/0x61 [4.605943] do_copy+0x8f/0xbd [4.605943] write_buffer+0x2b/0x3c [4.605943] flush_buffer+0x30/0x8a [4.605943] __gunzip+0x27e/0x322 [4.605943] ? bunzip2+0x3a05943] ? write_buffer+0x3c/0x3c [4.605943] ? __gunzip+0x322/0x322 [4.605943] gunzip+0x11/0x13 [4.605943] ? md_run_setup+0x99/0x99 [4.605943] unpack_to_rootfs+0x168/0x2bb [4.605943] ? md_run_setup+0x99/0x99 [4.605943] ? unpack_to_rootfs+0x2bb/0x2bb [4.605943] populate_rootfs+0x61/0x10f [4.605943] do_one_initcall+0x55/0x1ac [4.605943] kernel_init_freeable+0x19e/0x238 [4.605943] ? rest_init+0xb0/0xb0 [4.605943] kernel_init+0xe/0x110 [4.605943] ret_from_fork+0x35/0x40 [4.605943] ---[ end Kernel panic - not syncing: Out of memory and no killable processes... [4.605943] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
The OOM seems to be happening during initrd decompression. That has been observed before and also in Cosmic. I want to reduce the initrd footprint in case framebuffer drivers are included, which might happen when cryptsetup or plymouth are installed. On the other hand, just incrementing the reserved memory for crashkernel will easily fix this. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
I tried here with a Bionic VM (kernel 4.15.0-34) and kdump worked fine, using the default config (128M reserved for the crash kernel). I'll try to mimic the HW of Daniel's guest and perhaps the kernel version, although it's super old and seems not available anymore. @Paul, perhaps worth for you to append "earlyprintk=ttyS0" to your kdump cmdline (assuming ttyS0 is your serial console output) in order to dump some messages while it boots. Cheers, Guilherme -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
A console screen when I triggered. ** Attachment added: "trigger.png" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+attachment/5193476/+files/trigger.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
kdump is not still working on 18.04. Kernel : 4.15.0-34-generic linux-crashdump : 4.15.0.34.36 ubuntu@u-VM-07:~$ cat /var/crash/kexec_cmd /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-4.15.0-34-generic root=UUID=51adaf62-9023-4c45-bf94-889906e3d2d2 ro net.ifnames=0 biosdevname=0 nr_cpus=1 systemd.unit=kdump-tools.service irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz If I trigger, the machine gets hold and no response at all. $echo c > /proc/sysrq-trigger -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
i also see the kdump cannot work on 18.04 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1764246] Re: kdump kernel panics on Bionic
** Changed in: linux (Ubuntu) Importance: Undecided => High ** Also affects: linux (Ubuntu Bionic) Importance: High Status: Confirmed ** Tags added: kernel-da-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1764246 Title: kdump kernel panics on Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1764246/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
