Re: [systemd-devel] Improve boot-time of systemd-based device, revisited
2015-06-19 15:34 GMT+08:00 Chaiken, Alison ali...@she-devel.com: cee1 fykc...@gmail.com writes: 3.1 consider disabling readahead collection in the shipped devices, but leave readahead replay enabled. ceel, are you aware that readahead is deprecated in systemd and has not been included since about release 216? Some of us in automotive are still working on it. I have some patches here https://github.com/chaiken/systemd-hacks/tree/packfilelist against 215 that add various features. We may soon be forward-porting these, along with readahead itself, to the latest version. Glad to hear that :) The readahead doesn't work very well on my experiment, I spent considerable time performing boot experiments on production hardware, including trying different I/O schedulers.My conclusion was that readahead provides benefits in boot-time only when large, monolithic binaries start. If these gigantic applications were rearchitected to be more modular and could load libraries dynamically when needed instead of all at once, I suspect that the speedup associated with readahead would vanish. Nonetheless, under the right conditions, readahead may speed up boot on real hardware in product-relevant conditions. The problem is actually quite complex in the case of eMMC boot devices, which have their own sophisticated embedded controllers. To properly optimize the whole system, we need to know the behavior of that controller and model what happens at boot in the full system using different Linux I/O schedulers and readahead strategies. Unfortunately we don't have all that information. My suspicion is that we might actually boot faster from raw NAND flash, but then of course we have to perform our own wear-levelling and block sparing. BTW, I wonder whether the F2FS helps, which seems very friendly to flash storage. The replaying sequence: A, B, C The actual requesting sequence: C, B, A If we can figure out the requesting sequence, it can achieve real read ahead[1]. I have verified in detail that readahead worked as intended: the degree to which the system was I/O-bound did decrease, even in cases where there was no net speedup. Any idea why? 4. Get rid of systemd-cgroups-agent. This requires introduction of a new kernel interface to get notifications for cgroups running empty, for example via fanotify() on cgroupfs. Is there any related work in processing? Are you aware of JoinControllers? You appear to have old versions of software, which doesn't garner much sympathy from developers. So this option can reduce the times of invoking systemd-cgroups-agent? Note the points list in my previous mail come from http://freedesktop.org/wiki/Software/systemd/Optimizations/ and https://wiki.tizen.org/wiki/Automotive_Fast_Boot_Optimization, they seems interesting to me. These makes it hard to use systemd in a customized system. The Linux services business benefits from churn in userspace code . . . Kernel scheduler of an analogy - there's no kernel scheduler specific for embedded device, nor a kernel scheduler specific for linux server, but a scheduler for all the cases. So it should do with systemd, right? What I call for is to make the cold boot logic declarative, something like: main.c: log_to_A mount_X mount_Y Good news: you are free to choose SysVInit. What I mean is the initialization stage of systemd, that's e.g. mounting the API filesystem, etc. I expect a declarative expression of that, which will help to customization and debugging(without going deep to the code) I wonder whether a property system also makes sense in systemd's world? systemd unit files are already declarative lists of properties, right? The property system is something likes a system preference system(i.e. similar to a system dconf), IIRC, os x has a similar thing. My question is do we need a similar thing in systemd world, since systemd seems aiming to provide the basic infrastructure of a linux distribution? -- Regards, - cee1 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd-based device, revisited
cee1 fykc...@gmail.com writes: 2. Add a kernel sockopt for AF_UNIX to increase the maximum datagram queue length for SOCK_DGRAM sockets. ceel, are you aware of the (hopefully) pending full merge of kdbus in kernel 4.2? And that it is essentially a bottoms-up redesign of IPC that supports the existing D-Bus API? 3. Since boot-up tends to be IO bound, some IO optimization: That's not true in many systems, of course, notably the ones that need network to come up, as discussed previously. 3.1 consider disabling readahead collection in the shipped devices, but leave readahead replay enabled. ceel, are you aware that readahead is deprecated in systemd and has not been included since about release 216? Some of us in automotive are still working on it. I have some patches here https://github.com/chaiken/systemd-hacks/tree/packfilelist against 215 that add various features. We may soon be forward-porting these, along with readahead itself, to the latest version. The readahead doesn't work very well on my experiment, I spent considerable time performing boot experiments on production hardware, including trying different I/O schedulers.My conclusion was that readahead provides benefits in boot-time only when large, monolithic binaries start. If these gigantic applications were rearchitected to be more modular and could load libraries dynamically when needed instead of all at once, I suspect that the speedup associated with readahead would vanish. Nonetheless, under the right conditions, readahead may speed up boot on real hardware in product-relevant conditions. The problem is actually quite complex in the case of eMMC boot devices, which have their own sophisticated embedded controllers. To properly optimize the whole system, we need to know the behavior of that controller and model what happens at boot in the full system using different Linux I/O schedulers and readahead strategies. Unfortunately we don't have all that information. My suspicion is that we might actually boot faster from raw NAND flash, but then of course we have to perform our own wear-levelling and block sparing. The replaying sequence: A, B, C The actual requesting sequence: C, B, A If we can figure out the requesting sequence, it can achieve real read ahead[1]. I have verified in detail that readahead worked as intended: the degree to which the system was I/O-bound did decrease, even in cases where there was no net speedup. 4. Get rid of systemd-cgroups-agent. This requires introduction of a new kernel interface to get notifications for cgroups running empty, for example via fanotify() on cgroupfs. Is there any related work in processing? Are you aware of JoinControllers? You appear to have old versions of software, which doesn't garner much sympathy from developers. These makes it hard to use systemd in a customized system. The Linux services business benefits from churn in userspace code . . . What I call for is to make the cold boot logic declarative, something like: main.c: log_to_A mount_X mount_Y Good news: you are free to choose SysVInit. I wonder whether a property system also makes sense in systemd's world? systemd unit files are already declarative lists of properties, right? Best wishes, Alison --- Tunable parameter values found on the Internet . . . [are] akin to raiding someone else's medicine cabinet . . . -- Brendan Gregg, _Systems Performance_, p.23 http://brendangregg.com/linuxperf.html ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd-based device, revisited
On 06/19/2015 09:34 AM, Chaiken, Alison wrote: ceel, are you aware that readahead is deprecated in systemd and has not been included since about release 216? Some of us in automotive are still working on it. I have some patches here https://github.com/chaiken/systemd-hacks/tree/packfilelist against 215 that add various features. We may soon be forward-porting these, along with readahead itself, to the latest version. FWIW, an approach that will probably save you a lot of rebasing work in the future would be to rip out the readahead bits from that tree and put it into a repository of it own. It's a standalone tool that just happened to live in the systemd repo in the past. Some of the helper functions would need to be copied over as well, but that should be manageable. That way, you can even accept patches from other people and roll your own releases etc. HTH, Daniel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd-based device, revisited
On Mon, Jun 15, 2015 at 5:44 AM, Harald Hoyer harald.ho...@gmail.com wrote: On 14.06.2015 15:17, cee1 wrote: BTW, systemd-bootchart has a option to chart entropy, how is the entropy involved in boot up procedure? Well, if daemons need bytes from /dev/random (think sshd key generation), I guess they will have to wait for enough entropy, and so does the boot process in the end. I added the option since we discovered that the entropy pool can take a very long time to fill, and while this generally doesn't affect startup (sshd etc. are usually started on demand) it may be problematic for services like kerberos, an SSL enabled web server. It's largely irrelevant for the basic OS components, though. Cheers, Auke ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd-based device, revisited
On 14.06.2015 15:17, cee1 wrote: Hi all, I've recently got another chance to improve the boot-time of a systemd-based device. I'd like to share the experience here, and some thoughts and questions. The first time I tried to improve the boot-time of systemd: http://lists.freedesktop.org/archives/systemd-devel/2011-March/001707.html, after that, we have systemd-bootchart and systemd-analyze, which help a lot. It seems the biggest challenge of reducing boot-time of the ARM board at hand is taking are of the poor I/O performance: * A single fgets() call may randomly cause 200-300ms * A (big)service may spend 2-3s to complete its so loading - only ~100ms spent on CPU. I tried to first delay services which are less important, to save the I/O bandwidth in the early stage, and raise the priority of important services to SCHED_RR/IOPRIO_CLASS_RT: 1. I need to find the top I/O hunger processes (and then delay them if not important), but it's not straightforward to figure it out in bootchart, so adding *** iotop feature *** in bootchart seems very useful. 2. I think raising CPU scheduling priority works because it reduces chances of issuing I/O requests from other processes. Some thoughts: * The priority feature of I/O scheduler(CFQ) seems not work very well - IDLE I/O can still slow down Normal/RT I/O [1] * I don't know the detail of CFQ, but I wonder whether a rate limit helps - may reduce the latency between issuing I/O command and full filling the command? Last, I tried some readahead(ureadahead), but not do the magic, I guess it's because I/O is busy in the early stage, there's simply no ahead chance. What readahead helps, IMHO, is a snapshot of accessed disk blocks during boot up, in the order of they're requested. Thus a linear readahead against the snapshot will always read ahead of actual requesting blocks. BTW, systemd-bootchart has a option to chart entropy, how is the entropy involved in boot up procedure? Well, if daemons need bytes from /dev/random (think sshd key generation), I guess they will have to wait for enough entropy, and so does the boot process in the end. --- 1. http://linux-kernel.vger.kernel.narkive.com/0FC8rduf/ioprio-set-idle-class-doesn-t-work-as-its-name-suggests Regards, - cee1 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Wed, Apr 20, 2011 at 02:43:36AM +0200, Lennart Poettering wrote: On Fri, 18.03.11 13:40, Gustavo Sverzut Barbieri (barbi...@profusion.mobi) wrote: On Fri, Mar 18, 2011 at 10:52 AM, Andrey Borzenkov arvidj...@mail.ru wrote: On Fri, Mar 18, 2011 at 12:35 PM, fykc...@gmail.com fykc...@gmail.com wrote: Hi all, We did a series boot-time tests of systemd, and found some interesting things: (Note the tests were performed on a laptop with a 4-core mips cpu, AMD RS780 chipset, 2GB Memory, rotate harddisk with ext4 filesystem, debian squeeze, Linux 2.6.36 with fanotify enabled, systemd-v20, only boot to console.) 1. What can readahead affect boot-time? Sadly observed negative affect -- boot-time increases at least 1s. From subjective feelings (no real measurements) I confirm it. I noticed it as well, even with my Macbook Pro i7 with 8Gb RAM and 128SSD using btrfs. Note that btrfs is actually not particularly fast when mounting. We do serialization of fsck on rotating media already. This might be something that fixes itself as soon as the btrfs mounting isn't as slow anymore as it currently is. You can speed up btrfs mounting with free space caching. Right now it is enable with -o space_cache mount option. But beware: there was some problem identified when free space reporting. I am not sure if fix went to .38 and .39-rc already. -- Tomasz Torcz God, root, what's the difference? xmpp: zdzich...@chrome.pl God is more forgiving. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Fri, 18.03.11 09:35, fykc...@gmail.com (fykc...@gmail.com) wrote: Hi all, We did a series boot-time tests of systemd, and found some interesting things: (Note the tests were performed on a laptop with a 4-core mips cpu, AMD RS780 chipset, 2GB Memory, rotate harddisk with ext4 filesystem, debian squeeze, Linux 2.6.36 with fanotify enabled, systemd-v20, only boot to console.) 1. What can readahead affect boot-time? Sadly observed negative affect -- boot-time increases at least 1s. With bootchart, I find more I/O at boot compared with no readahead, see attachment noreadahead-vs-readahead.png. Thoughts: Maybe we need only monitor files with read access, not all opend files? P.S. inotify seems enough for the job (with one more step to open each file). In general if what you boot is minimal the effect of readahead will be minimal too, but the cost of spawning yet another service is what you pay for. 2. udev-settle.service serializes the boot process, see attachment udev-settle.png. I tried to create a hotplug.target(which is activated after default.target), and made udev-settle reside at it, this rendered a unbootable system. systemd depends on udev at early time. udev-settle is unnecessary, unless you use LVM and very few other broken services. As soon as they are fixed we can remove this for good. I don't use this service on my machine anymore. Also see my more recent blog story about this: http://0pointer.de/blog/projects/blame-game BTW, bootchart seems not very intuitive(no service, only processes. Also some processes may be missed if they live between two ps aux call of bootchart), is it possible to add similar feature to systemd? We have that now with systemd-analyze plot. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Fri, 18.03.11 13:52, Andrey Borzenkov (arvidj...@mail.ru) wrote: I tried to create a hotplug.target(which is activated after default.target), and made udev-settle reside at it, this rendered a unbootable system. systemd depends on udev at early time. Thoughts: devtmpfs is mounted, so all cold-plug jobs can be done without udev involved. IMHO, fast boot doesn't mean get all services ready in a short time, but means popup an UI as soon as possible. Windows seems do hotplug jobs after user log in. Mandriva uses so called speedboot with sysvint - where GUI is started as soon as possible. It is handcrafted so that only several device classes are coldplugged and then DM is launched effectively from rc.sysinit already. We want this to become the default actually. Currently the semantics of rc-local still block us from doing that. But ideally gdm would popup a login dialog on every screen as it shows up with no delay in any way, and without waiting for any other services. Users did mention that boot under systemd actually feels slower then using sysvinit. We can spawn the full set of userspace services (reasonably complete GNOME session) now in less than 1s. I doubt anybody else has been capable of doing anything like that in this time so far. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Wed, 30.03.11 02:28, fykc...@gmail.com (fykc...@gmail.com) wrote: 2011/3/29 Lennart Poettering lenn...@poettering.net: On Tue, 29.03.11 03:36, fykc...@gmail.com (fykc...@gmail.com) wrote: To make boot fast, it seems udev's coldplug do too much jobs -- what I expect is only coldplug local block devices and tty in that stage. This can save ~2s in my boot test. Is it possible to support .device unit files? Hmm? Note sure I understand the question, but for a .device unit to show up in systemd it must be tagged systemd in udev, which only can happen when the device was triggered after udev is started. I've already known that currently systemd can only add .device units from udev. My question was Is it suitable for systemd to add supports of loading .device units from .device unit files in /lib/systemd/system ? systemd reads .device units just fine from disk. In fact you don't have to do any kind of configuration for them. If you do systemctl start foobar.device this call will wait for a device of that name to show up. You can make up any name you want with this. If such a device never shows up then it might eventually time out though. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Sat, 19.03.11 06:05, Chen Jie (ch...@lemote.com) wrote: 2011/3/18 Kay Sievers kay.siev...@vrfy.org: On Fri, Mar 18, 2011 at 14:40, Gustavo Sverzut Barbieri barbi...@profusion.mobi wrote: On Fri, Mar 18, 2011 at 10:52 AM, Andrey Borzenkov arvidj...@mail.ru wrote: On Fri, Mar 18, 2011 at 12:35 PM, fykc...@gmail.com fykc...@gmail.com wrote: 1. What can readahead affect boot-time? Sadly observed negative affect -- boot-time increases at least 1s. From subjective feelings (no real measurements) I confirm it. I noticed it as well, even with my Macbook Pro i7 with 8Gb RAM and 128SSD using btrfs. It's ~0.5 sec faster here with readahead on a SSD. Each time runs readahead-replay may cause readahead-collect to record more blocks to read ahead -- size of /.readahead never reduces. Also I found /.readahead recorded files like .xession-errors which is written only, thus should not be read ahead. Well, fanotify curently doesn't inform us whether open() was called for O_RDONLY or O_RDWR/O_WRONLY. If it did we could ignore the latter. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Sun, 20.03.11 05:28, fykc...@gmail.com (fykc...@gmail.com) wrote: 2011/3/19 Chen Jie ch...@lemote.com: 2011/3/18 Kay Sievers kay.siev...@vrfy.org: It's ~0.5 sec faster here with readahead on a SSD. Each time runs readahead-replay may cause readahead-collect to record more blocks to read ahead -- size of /.readahead never reduces. Also I found /.readahead recorded files like .xession-errors which is written only, thus should not be read ahead. Maybe adjust readahead-collect-done.timer to 5s will help. Current readahead implementation has some problems: 1. It can't separate *real* block read requests from all read requests(which includes more blocks read by the kernel's readahead logic) Shouldn't make a big difference, since on replay we turn off additional kernel-side readahead. However, it is true that the file will only ever increase, never decrease in size. 2. It just gives advices for how to do kernel's readahead, causes the first read of a fille to spend more time. Hmm? I revisited Booting Linux in five seconds article[1], AIUI, they did readahead in a different way: 1. They determine which blocks need to read ahead by a patch against kernel. Well, the meego readahead implementation uses a kernel patch to store in each inode struct when it was first read, and then iterates through the FS hierarchy and reads that value. That is a workable solution if you plan to run the collector only once at distro build-time and on a limited size FS, but for a generic distro we need to run it on every boot basically, because you end up reiterating through your FS tree at each boot, and that can be a massive amount of time. Note that the meego implementation relies on mincore() to determine what block to readahead, which is precisely what we do. The only difference is how the list of files to use mincore() on is generated. We use fanotify (which requires no kernel patch), and they use the inode timestamp plus FS iteration. 2. They do read ahead(aka replay) by reading each block with the idle I/O scheduler. We do that too. We use idle on SSD, and realtime on HDD. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Thu, 24.03.11 10:20, Gustavo Sverzut Barbieri (barbi...@profusion.mobi) wrote: On Thu, Mar 24, 2011 at 9:35 AM, Andrey Borzenkov arvidj...@mail.ru wrote: On Thu, Mar 24, 2011 at 12:19 PM, fykc...@gmail.com fykc...@gmail.com wrote: Hi, 2011/3/18 Gustavo Sverzut Barbieri barbi...@profusion.mobi: 2. should we do (or have a way to) force a manual ordering to help Xorg/DM/WM by avoiding spawn of concurrent services? We know these have the higher priority, but it's a higher priority only during startup, later on they should all have the same priority... otherwise we could just do it by means of systemd's service settings. I prefer to start UI first, and then UI notifies systemd to start more services. The original boot sequence seems server-oriented, and I need desktop-oriented boot sequence -- let most services start AFTER a usable basic UI popup. KDM/GDM get really confused when host name changes after they are started. And I have seen complaints that displayed host name is wrong. So it probably should depend at least on network being available. That is stupid, as the hostname may change due lots of reasons, maybe you wifi changed and now you got another home domain from dhcp? What would they do? X does hostname based auth. It's seriously broken. In fact, almost everything is broken in this context: A) dhcp should not modify the local hostname. B) X shouldn't be so retarded to use literal host name strings for authentication purposes. C) KDM shouldn't set up xauth that way. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
2011/3/28 Lennart Poettering lenn...@poettering.net: On Sun, 20.03.11 05:28, fykc...@gmail.com (fykc...@gmail.com) wrote: Current readahead implementation has some problems: 1. It can't separate *real* block read requests from all read requests(which includes more blocks read by the kernel's readahead logic) Shouldn't make a big difference, since on replay we turn off additional kernel-side readahead. However, it is true that the file will only ever increase, never decrease in size. For collect, it can't filter out: 1. Kernel-side readahead, whether the readahead is initiated by kernel(when no /.readhead data), or the replay process. 2. Written blocks of files(opened as r+, w+, a). The written blocks resides at memory when boot time. IMHO, the kernel lacks some APIs to notify each *real* read requests. e.g, It can be done by tracking each read syscall (mmap seems not easy to handle, though). 2. It just gives advices for how to do kernel's readahead, causes the first read of a fille to spend more time. Hmm? posix_fadvise(...) may make each read do more readahead(more than the kernel guess way), thus spend more time. e.g. * When no replay, someone reads A part of file X -- do some job -- reads B part of file X. * When replay, both A and B parts of file X are read in one time, thus more I/O usage. Other services may spend more time waiting for I/O.(This can be observed from bootchart diagram) BTW, does posix_fadvise apply globally or just for the process which calls it? We do that too. We use idle on SSD, and realtime on HDD. Why realtime on HDD? BTW, According to test, the idle is not really *idle*, see the attachment. That means 'replay' will always impact other one's I/O. For 'replay' in idle I/O class on HDD, other one's I/O performance will reduce by half, according to the test. -- Regards, - cee1 attachment: effect_of_IO_IDLE.png___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Thu, Mar 24, 2011 at 12:19 PM, fykc...@gmail.com fykc...@gmail.com wrote: Hi, 2011/3/18 Gustavo Sverzut Barbieri barbi...@profusion.mobi: 2. should we do (or have a way to) force a manual ordering to help Xorg/DM/WM by avoiding spawn of concurrent services? We know these have the higher priority, but it's a higher priority only during startup, later on they should all have the same priority... otherwise we could just do it by means of systemd's service settings. I prefer to start UI first, and then UI notifies systemd to start more services. The original boot sequence seems server-oriented, and I need desktop-oriented boot sequence -- let most services start AFTER a usable basic UI popup. KDM/GDM get really confused when host name changes after they are started. And I have seen complaints that displayed host name is wrong. So it probably should depend at least on network being available. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Thu, Mar 24, 2011 at 9:35 AM, Andrey Borzenkov arvidj...@mail.ru wrote: On Thu, Mar 24, 2011 at 12:19 PM, fykc...@gmail.com fykc...@gmail.com wrote: Hi, 2011/3/18 Gustavo Sverzut Barbieri barbi...@profusion.mobi: 2. should we do (or have a way to) force a manual ordering to help Xorg/DM/WM by avoiding spawn of concurrent services? We know these have the higher priority, but it's a higher priority only during startup, later on they should all have the same priority... otherwise we could just do it by means of systemd's service settings. I prefer to start UI first, and then UI notifies systemd to start more services. The original boot sequence seems server-oriented, and I need desktop-oriented boot sequence -- let most services start AFTER a usable basic UI popup. KDM/GDM get really confused when host name changes after they are started. And I have seen complaints that displayed host name is wrong. So it probably should depend at least on network being available. That is stupid, as the hostname may change due lots of reasons, maybe you wifi changed and now you got another home domain from dhcp? What would they do? -- Gustavo Sverzut Barbieri http://profusion.mobi embedded systems -- MSN: barbi...@gmail.com Skype: gsbarbieri Mobile: +55 (19) 9225-2202 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Thu, Mar 24, 2011 at 10:20:38AM +, Gustavo Sverzut Barbieri wrote: On Thu, Mar 24, 2011 at 9:35 AM, Andrey Borzenkov arvidj...@mail.ru wrote: On Thu, Mar 24, 2011 at 12:19 PM, fykc...@gmail.com fykc...@gmail.com wrote: Hi, 2011/3/18 Gustavo Sverzut Barbieri barbi...@profusion.mobi: 2. should we do (or have a way to) force a manual ordering to help Xorg/DM/WM by avoiding spawn of concurrent services? We know these have the higher priority, but it's a higher priority only during startup, later on they should all have the same priority... otherwise we could just do it by means of systemd's service settings. I prefer to start UI first, and then UI notifies systemd to start more services. The original boot sequence seems server-oriented, and I need desktop-oriented boot sequence -- let most services start AFTER a usable basic UI popup. KDM/GDM get really confused when host name changes after they are started. And I have seen complaints that displayed host name is wrong. So it probably should depend at least on network being available. That is stupid, as the hostname may change due lots of reasons, maybe you wifi changed and now you got another home domain from dhcp? What would they do? They show the old hostname :( ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
Any comments about these 2 points: On Fri, Mar 18, 2011 at 1:40 PM, Gustavo Sverzut Barbieri barbi...@profusion.mobi wrote: 2. udev-settle.service serializes the boot process, see attachment udev-settle.png. I have feeling that increased parallelism during boot (like starting fsck/mount as soon as device becomes available) actually has negative effect on consumer grade devices. My HDD in notebook simply is not prepared for it ... ACK. Maybe we should add some intelligence to systemd's automatic unit-mount creation and serialize partition mounts of the same device? For traditional systems it's easy, just make all /dev/sda* depend on each other, but world is bit harder and multiple-device FS like btrfs or even DM will screw with that. Ideas? Maybe we could do it just based on /etc/fstab, sorting dependencies based on /dev/sda* and respective mountpoints. I tried to create a hotplug.target(which is activated after default.target), and made udev-settle reside at it, this rendered a unbootable system. systemd depends on udev at early time. Thoughts: devtmpfs is mounted, so all cold-plug jobs can be done without udev involved. IMHO, fast boot doesn't mean get all services ready in a short time, but means popup an UI as soon as possible. Windows seems do hotplug jobs after user log in. Mandriva uses so called speedboot with sysvint - where GUI is started as soon as possible. It is handcrafted so that only several device classes are coldplugged and then DM is launched effectively from rc.sysinit already. Users did mention that boot under systemd actually feels slower then using sysvinit. Well, I never tried other distro other than Gentoo on this Macbook and here it's kinda fast at 7s to be 100% ready with E17 (I have an autostart .desktop that writes to /dev/kmsg to measure it), Startup finished in 2s 360ms 651us (kernel) + 1s 753ms 783us (userspace) = 4s 114ms 434us. But it could be improved yes. As you all said, maybe we should handle udev hotplug in a more throttled way by postponing non-critical devices and having everything else to be hotplug aware? AFAIK Xorg will handle nicely new input devices. ConnMan/NetworkManager will handle nicely network devices. Same for bluez. We could even just activate these services based on the presence of the devices, at least E17 will handle nicely daemons appearing later by means of DBus NameOwnerChanged. Ideas: 1. should we change ConnMan and NetworkManager to work as BlueZ an be able to be activated/shutdown by udev hotplug actions (but cooperative with systemd, bluetoothd is not AFAIR); 2. should we do (or have a way to) force a manual ordering to help Xorg/DM/WM by avoiding spawn of concurrent services? We know these have the higher priority, but it's a higher priority only during startup, later on they should all have the same priority... otherwise we could just do it by means of systemd's service settings. A hackish(?) solution would be to have a BootPriority=(True|False), set to False by default and True for services we care most. Lower priority services would be set to background priority in IO, CPU and others, then being reset to actual values when systemd is notified. Problem is we need to notify Systemd of that, as it's not a matter of just starting gdm, but actually gdm being in a usable state (defined by gdm itself) or desktop being ready if users use autologin (like I do). This could also be stated as system is idle for X seconds, which would be monitored by systemd itself and then no manual notification is required. -- Gustavo Sverzut Barbieri http://profusion.mobi embedded systems -- MSN: barbi...@gmail.com Skype: gsbarbieri Mobile: +55 (19) 9225-2202 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
2011/3/18 Kay Sievers kay.siev...@vrfy.org: On Fri, Mar 18, 2011 at 14:40, Gustavo Sverzut Barbieri barbi...@profusion.mobi wrote: On Fri, Mar 18, 2011 at 10:52 AM, Andrey Borzenkov arvidj...@mail.ru wrote: On Fri, Mar 18, 2011 at 12:35 PM, fykc...@gmail.com fykc...@gmail.com wrote: 1. What can readahead affect boot-time? Sadly observed negative affect -- boot-time increases at least 1s. From subjective feelings (no real measurements) I confirm it. I noticed it as well, even with my Macbook Pro i7 with 8Gb RAM and 128SSD using btrfs. It's ~0.5 sec faster here with readahead on a SSD. Each time runs readahead-replay may cause readahead-collect to record more blocks to read ahead -- size of /.readahead never reduces. Also I found /.readahead recorded files like .xession-errors which is written only, thus should not be read ahead. Maybe adjust readahead-collect-done.timer to 5s will help. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
2011/3/19 Chen Jie ch...@lemote.com: 2011/3/18 Kay Sievers kay.siev...@vrfy.org: It's ~0.5 sec faster here with readahead on a SSD. Each time runs readahead-replay may cause readahead-collect to record more blocks to read ahead -- size of /.readahead never reduces. Also I found /.readahead recorded files like .xession-errors which is written only, thus should not be read ahead. Maybe adjust readahead-collect-done.timer to 5s will help. Current readahead implementation has some problems: 1. It can't separate *real* block read requests from all read requests(which includes more blocks read by the kernel's readahead logic) 2. It just gives advices for how to do kernel's readahead, causes the first read of a fille to spend more time. I revisited Booting Linux in five seconds article[1], AIUI, they did readahead in a different way: 1. They determine which blocks need to read ahead by a patch against kernel. 2. They do read ahead(aka replay) by reading each block with the idle I/O scheduler. Regards, cee1 --- [1] http://lwn.net/Articles/299483/ Intel made five seconds boot: 1s for kernel, 1s for early boot, 1s for X and 2s for desktop environment. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Improve boot-time of systemd
On Fri, Mar 18, 2011 at 10:52 AM, Andrey Borzenkov arvidj...@mail.ru wrote: On Fri, Mar 18, 2011 at 12:35 PM, fykc...@gmail.com fykc...@gmail.com wrote: Hi all, We did a series boot-time tests of systemd, and found some interesting things: (Note the tests were performed on a laptop with a 4-core mips cpu, AMD RS780 chipset, 2GB Memory, rotate harddisk with ext4 filesystem, debian squeeze, Linux 2.6.36 with fanotify enabled, systemd-v20, only boot to console.) 1. What can readahead affect boot-time? Sadly observed negative affect -- boot-time increases at least 1s. From subjective feelings (no real measurements) I confirm it. I noticed it as well, even with my Macbook Pro i7 with 8Gb RAM and 128SSD using btrfs. With bootchart, I find more I/O at boot compared with no readahead, see attachment noreadahead-vs-readahead.png. Thoughts: Maybe we need only monitor files with read access, not all opend files? P.S. inotify seems enough for the job (with one more step to open each file). 2. udev-settle.service serializes the boot process, see attachment udev-settle.png. I have feeling that increased parallelism during boot (like starting fsck/mount as soon as device becomes available) actually has negative effect on consumer grade devices. My HDD in notebook simply is not prepared for it ... ACK. Maybe we should add some intelligence to systemd's automatic unit-mount creation and serialize partition mounts of the same device? For traditional systems it's easy, just make all /dev/sda* depend on each other, but world is bit harder and multiple-device FS like btrfs or even DM will screw with that. Ideas? Maybe we could do it just based on /etc/fstab, sorting dependencies based on /dev/sda* and respective mountpoints. I tried to create a hotplug.target(which is activated after default.target), and made udev-settle reside at it, this rendered a unbootable system. systemd depends on udev at early time. Thoughts: devtmpfs is mounted, so all cold-plug jobs can be done without udev involved. IMHO, fast boot doesn't mean get all services ready in a short time, but means popup an UI as soon as possible. Windows seems do hotplug jobs after user log in. Mandriva uses so called speedboot with sysvint - where GUI is started as soon as possible. It is handcrafted so that only several device classes are coldplugged and then DM is launched effectively from rc.sysinit already. Users did mention that boot under systemd actually feels slower then using sysvinit. Well, I never tried other distro other than Gentoo on this Macbook and here it's kinda fast at 7s to be 100% ready with E17 (I have an autostart .desktop that writes to /dev/kmsg to measure it), Startup finished in 2s 360ms 651us (kernel) + 1s 753ms 783us (userspace) = 4s 114ms 434us. But it could be improved yes. As you all said, maybe we should handle udev hotplug in a more throttled way by postponing non-critical devices and having everything else to be hotplug aware? AFAIK Xorg will handle nicely new input devices. ConnMan/NetworkManager will handle nicely network devices. Same for bluez. We could even just activate these services based on the presence of the devices, at least E17 will handle nicely daemons appearing later by means of DBus NameOwnerChanged. Ideas: 1. should we change ConnMan and NetworkManager to work as BlueZ an be able to be activated/shutdown by udev hotplug actions (but cooperative with systemd, bluetoothd is not AFAIR); 2. should we do (or have a way to) force a manual ordering to help Xorg/DM/WM by avoiding spawn of concurrent services? We know these have the higher priority, but it's a higher priority only during startup, later on they should all have the same priority... otherwise we could just do it by means of systemd's service settings. A hackish(?) solution would be to have a BootPriority=(True|False), set to False by default and True for services we care most. Lower priority services would be set to background priority in IO, CPU and others, then being reset to actual values when systemd is notified. Problem is we need to notify Systemd of that, as it's not a matter of just starting gdm, but actually gdm being in a usable state (defined by gdm itself) or desktop being ready if users use autologin (like I do). This could also be stated as system is idle for X seconds, which would be monitored by systemd itself and then no manual notification is required. As a last note, what piss me off is the fact that some things affect boot time but they should not, and it's not systemd's fault. For instance with both nvidia and intel on another PC I have around 1s slowdown when I'm using external monitors! Damn, 14% of my boot time because of that? -- Gustavo Sverzut Barbieri http://profusion.mobi embedded systems -- MSN: barbi...@gmail.com Skype: gsbarbieri Mobile: +55 (19) 9225-2202 ___ systemd-devel mailing list
Re: [systemd-devel] Improve boot-time of systemd
On Fri, Mar 18, 2011 at 14:40, Gustavo Sverzut Barbieri barbi...@profusion.mobi wrote: On Fri, Mar 18, 2011 at 10:52 AM, Andrey Borzenkov arvidj...@mail.ru wrote: On Fri, Mar 18, 2011 at 12:35 PM, fykc...@gmail.com fykc...@gmail.com wrote: 1. What can readahead affect boot-time? Sadly observed negative affect -- boot-time increases at least 1s. From subjective feelings (no real measurements) I confirm it. I noticed it as well, even with my Macbook Pro i7 with 8Gb RAM and 128SSD using btrfs. It's ~0.5 sec faster here with readahead on a SSD. But it could be improved yes. As you all said, maybe we should handle udev hotplug in a more throttled way by postponing non-critical devices and having everything else to be hotplug aware? That's not really possible, you can not really make such list, and you need to handle all parent devices from all 'interesting' devices anyway to expose them. The 'settle' service is only there for broken services. Originally it wasn't even pulled into the base target but was free-hanging with nobody getting blocked by it. Lennart pulled it in for a few broken things and selinux to work, and it ended up blocking the base target to be on the safe side for non-hotplug aware stuff. We might want to re-check if that's really what we want. Ideally, the entire 'settle' step would not even exist. I guess in your limited environments, you can just drop the entire thing and let the coldplug task run in the background. Services need to cope with uninitialized devices then, and filter them out when enumerating subsystems, and wait for the proper event from udev. That should all be pretty straight-forward if you don't need to support all the insanity in usual Linux distros. Kay ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
