So I did a little bisecting today on mainline x86_64 kernels with CONFIG_SLUB:
v4.4 Startup finished in 2.718s (kernel) + 11.831s (userspace) = 14.549s v4.5 Startup finished in 2.707s (kernel) + 12.777s (userspace) = 15.484s v4.6-rc4-1018-g5a5f0792e98b Startup finished in 2.970s (kernel) + 13.641s (userspace) = 16.611s v4.6-3615-g1eccc6e1529e Startup finished in 3.005s (kernel) + 14.063s (userspace) = 17.069s v4.6-5396-g07b75260ebc2 Startup finished in 3.044s (kernel) + 14.932s (userspace) = 17.977s v4.6-5931-gf4c80d5a16eb Startup finished in 3.057s (kernel) + 13.429s (userspace) = 16.487s v4.6-6084-g78975f23cba0 Startup finished in 3.050s (kernel) + 13.903s (userspace) = 16.954s v4.6-5050-g18726ca8b34b Startup finished in 3.015s (kernel) + 13.510s (userspace) = 16.525s v4.6-5055-g6052b7880a95 Startup finished in 3.029s (kernel) + 13.519s (userspace) = 16.549s v4.6-5058-g76b342bdc71b Startup finished in 2.998s (kernel) + 14.179s (userspace) = 17.177s v4.6-5060-g801faf0db894 Startup finished in 3.064s (kernel) + 14.667s (userspace) = 17.732s v4.6-5061-g81ae6d03952c Startup finished in 3.057s (kernel) + 18.357s (userspace) = 21.414s v4.6-5073-g9cc3387fa29f Startup finished in 3.073s (kernel) + 17.976s (userspace) = 21.049s v4.6-5119-g682a3385e773 Startup finished in 3.089s (kernel) + 17.987s (userspace) = 21.077s v4.6-6266-ga05a70db34ba Startup finished in 3.037s (kernel) + 18.550s (userspace) = 21.588s v4.6-7135-g0eff4589c36e Startup finished in 3.008s (kernel) + 18.074s (userspace) = 21.082s v4.7-rc5-1395-g877fa9a42ddc Startup finished in 3.179s (kernel) + 19.147s (userspace) = 22.327s v4.8 Startup finished in 3.197s (kernel) + 20.836s (userspace) = 24.034s v4.9.7 Startup finished in 3.674s (kernel) + 20.034s (userspace) = 23.708s v4.9.7 + the two patches Startup finished in 3.518s (kernel) + 13.027s (userspace) = 16.546s What we can observe from this: The big bump came in https://github.com/torvalds/linux/commit/81ae6d03952c1bfb96e1a716809bd65e7cd14360 (or alternatively in https://github.com/torvalds/linux/commit/801faf0db8947e01877920e848a4d338dd7a99e7 for CONFIG_SLAB, if bz.k.o is trustworthy). This is likely fixed by https://github.com/torvalds/linux/commit/86d9f48534e800e4d62cdc1b5aaf539f4c1d47d6, https://github.com/torvalds/linux/commit/13583c3d3224508582ec03d881d0b68dd3ee8e10 and https://github.com/torvalds/linux/commit/89e364db71fb5e7fc8d93228152abfa67daf35fa. The last two are not in stable, though. Speaking of https://github.com/torvalds/linux/commit/86d9f48534e800e4d62cdc1b5aaf539f4c1d47d6, this went into v4.8.7 as https://github.com/torvalds/linux/commit/68adb469dd3b28513c5b548df3437f2bc1a83e43, so that may be the fix you were looking for in comment #65. There are also lots of small performance regressions here and there that may be difficult to bisect. A second in userspace between v4.4 and v4.5 and another second between v4.5 and v4.6. Some 300 ms in kernel between v4.5 and v4.6, 200 ms between v4.6 and v4.7 and another 300 ms between v4.8 and v4.9. That's here, on my .config and my hardware. Dimitrij has 10 seconds in kernel between v4.7 and v4.8-rc1, but I can't see anything like that here. (In this case you might want to look into dmesg if it's a general slowdown or there's one thing that blocks for 10 secs. I suspect there might be.) I think that these other slowdowns that I and Dimitrij and possibly others are experiencing are likely unrelated to the main issue here (https://bugzilla.kernel.org/show_bug.cgi?id=172981, https://bugzilla.kernel.org/show_bug.cgi?id=172991) and we'll need to bisect further ourselves. :-) ** Bug watch added: Linux Kernel Bug Tracker #172981 http://bugzilla.kernel.org/show_bug.cgi?id=172981 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1626436 Title: [4.8 regression] boot has become very slow Status in linux package in Ubuntu: In Progress Status in linux source package in Yakkety: In Progress Bug description: With yakkety's recent update from linux 4.4 to 4.8 booting has become a lot slower. It's not one service in particular, but without "quiet" and "splash" you can now easily read every single line instead of that whole wall of text zipping by. It now takes over 20s instead of ~10 seconds to boot. This is even more dramatic when factoring out the recent boot hang of NetworkManager (bug 1622893) and disabling lightdm: sudo systemctl mask NetworkManager NetworkManager-wait-online lightdm then booting with 4.4 takes 1.5s and with 4.8 19.5s (!). Some excerps from systemd-analyze blame: 4.4: 474ms postfix@-.service 395ms lxd-containers.service 305ms networking.service 4.8: 4.578s postfix@-.service 7.300s lxd-containers.service 6.285s networking.service I attach the full outputs of critical-chain and analyze for 4.4 and 4.8 for reference. This is much less noticeable in the running system. There is no immediate feeling of sluggishness (although my system is by and large idle). I compared the time of sbuilding colord under similar circumstances (-j4, building on tmpfs, thus no hard disk delays; running with fully pre-loaded apt-cacher-ng thus no random network delays), and with 4.4 it takes 6.5 minutes and with 4.8 it takes 7.5. So that got a bit slower, but much less dramatically than during boot, so this is either happening when a lot of processes run in parallel, or is perhaps related to setting up cgroups. One thing I noticed that during sbuild in 4.8 "top" shows ridiculous loads (~ 250) under 4.8, while it's around 4 or 5 under 4.4. But that doesn't reflect in actual sluggishness, so this might be just an unrelated bug. ProblemType: Bug DistroRelease: Ubuntu 16.10 Package: linux-image-4.8.0-11-generic 4.8.0-11.12 ProcVersionSignature: Ubuntu 4.8.0-11.12-generic 4.8.0-rc6 Uname: Linux 4.8.0-11-generic x86_64 ApportVersion: 2.20.3-0ubuntu7 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/pcmC0D0c: martin 3049 F...m pulseaudio /dev/snd/pcmC0D0p: martin 3049 F...m pulseaudio /dev/snd/controlC0: martin 3049 F.... pulseaudio Date: Thu Sep 22 09:42:56 2016 EcryptfsInUse: Yes MachineType: LENOVO 2324CTO ProcEnviron: TERM=linux PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=de_DE.UTF-8 SHELL=/bin/bash ProcFB: 0 inteldrmfb ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-4.8.0-11-generic.efi.signed root=UUID=f86539b0-3a1b-4372-83b0-acdd029ade68 ro rootflags=subvol=@ systemd.debug-shell RelatedPackageVersions: linux-restricted-modules-4.8.0-11-generic N/A linux-backports-modules-4.8.0-11-generic N/A linux-firmware 1.161 SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 07/09/2013 dmi.bios.vendor: LENOVO dmi.bios.version: G2ET95WW (2.55 ) dmi.board.asset.tag: Not Available dmi.board.name: 2324CTO dmi.board.vendor: LENOVO dmi.board.version: 0B98401 Pro dmi.chassis.asset.tag: No Asset Information dmi.chassis.type: 10 dmi.chassis.vendor: LENOVO dmi.chassis.version: Not Available dmi.modalias: dmi:bvnLENOVO:bvrG2ET95WW(2.55):bd07/09/2013:svnLENOVO:pn2324CTO:pvrThinkPadX230:rvnLENOVO:rn2324CTO:rvr0B98401Pro:cvnLENOVO:ct10:cvrNotAvailable: dmi.product.name: 2324CTO dmi.product.version: ThinkPad X230 dmi.sys.vendor: LENOVO To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1626436/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
