[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Using the testing method from #17 for the package oxygen-icon-theme (5890 mostly small files), tested on a 5400 RPM laptop harddisk formatted with ext4. dpkg 1.15.5.6ubuntu4: unpack: 37s real: 53s dpkg 1.15.5.6ubuntu5~ppa1: unpack: 5s real: 22s Also tested on a RAID 0 array consisting of two 7200 RPM desktop drives, formatted with XFS: dpkg 1.15.5.6ubuntu4: unpack: 56s real: 53s dpkg 1.15.5.6ubuntu5~ppa1: unpack: 13s real: 21s So quite a noticable difference in both cases :-) -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
What benefit is provided by all the fsync action? Without it, a power-failed install may have zero-length or wrong-content files. With it, a power-failed install still has a broken package -- each individual file may be fully there or fully not-there, but there will be missing files. It doesn't help achieve a successful package install. In fact, it _increases_ vulnerability to power failures by making the vulnerable time window more than twice as long (all of the added time is vulnerable time, while some of the original time must be safe prep time). Either way, system powers back on with a broken package. Either way, the user or the dpkg system must deal with it. dpkg _should_ do a regular sync() after each package; I imagine (without checking source) that it already does. It already has notes on which packages were in transition. Make sure _those_ are fully sync'd, fsync'd if that's the right way to do it -- those tell dpkg where to pick up, which package to fix, after the power cycle. I think this code should be retracted, even for normal post-install package installs. Cleanup is going to be needed after a mid-install power failure either way; don't make users suffer through slow, noisy, HD-punishing package installs for no [or negative] benefit. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Ok, looked at Colin's backport: It's the right way to do it; and I suspect this is true for all Unixish OSes with all filesystems, whether or not they have synchronous sync(). -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
dpkg from Colin Watsons PPA solved the problem: I installed Blender and the unpacking phase of the 10Mb package took less than a second. And I experience no more rattling disks when installing packages. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
I wonder if my problem is related to this bug - I am observing excessive harddisk activity in the unpack phase of package installations on all three Lucid systems: https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/580537 -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
I would appreciate it if people affected by this bug could test the dpkg package from: https://launchpad.net/~cjwatson/+archive/ppa This is probably easier to test if you're seeing problems with dpkg performance after installation, although I do plan to hack it into an installation environment after I get back from UDS and do some comparisons. Please report before-and-after timings if you can, preferably with a cold cache. If this makes a significant improvement, I'll get it into Ubuntu 10.04.1. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Colin- Poke me when you have an install media ready to test. That's my primary use case, and I'll gladly help test that. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
I ran some tests on my Dell Latitude D430, which has ext4 and a ridiculously slo w hard disk. I tested with: $ echo 3 | sudo tee /proc/sys/vm/drop_caches; sudo dpkg -i /var/cache/apt/archives/openoffice.org-common_1%3a3.2.0-7ubuntu4_all.deb | ts old dpkg: openoffice.org-common: 56s unpack, 25s config/triggers ubuntu-docs: 25s unpack, 4s config Colin's PPA dpkg: openoffice.org-common: 23s unpack, 23s config/triggers ubuntu-docs: 23s unpack, 4s config So it makes quite a difference for the unpacking of openoffice.org- common (2860 files, huge), and not so much for ubuntu-docs (7454 files, smaller) -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Test on ext3 (Karmic upgraded to Lucid), 5400rpm notebook hd. Note that i think in addition to dropping the caches, we have to sync before running the test to get somewhat stable results. Commands used (as root): sync; echo 3 /proc/sys/vm/drop_caches; time dpkg -i /var/cache/apt/archives/openoffice.org-common_1%3a3.2.0-7ubuntu4_all.deb | ts sync; echo 3 /proc/sys/vm/drop_caches; time dpkg -i /var/cache/apt/archives/ubuntu-docs_10.04.3_all.deb | ts Results, old dpkg (1.15.5.6ubuntu4): openoffice.org-common: 26s unpack, 93s real ubuntu-docs: 24s unpack, 51s real Results, Colin's PPA dpkg (1.15.5.6ubuntu5~ppa1): openoffice.org-common: 31s unpack, 94s real ubuntu-docs: 23s unpack, 54s real So ext3 doesn't care at all - it's not regressing at least. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Am 05/05/10 15:53, schrieb Jean-Baptiste Lallement: ** Bug watch added: Linux Kernel Bug Tracker #15910 http://bugzilla.kernel.org/show_bug.cgi?id=15910 ** Also affects: linux via http://bugzilla.kernel.org/show_bug.cgi?id=15910 Importance: Unknown Status: Unknown I wonder if this says what you meant to say ( from the kernel.org bug ): --- auto_da_alloc doesn't detect the replace-via-rename (at least in the case of dpkg.) --- Is dpkg really ***replacing***-via-rename when installing new packages? Where do the old files that are overwritten come from? Thanks! -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Note that for the install case where corruption does not matter, a export LD_PRELOAD=./libeatmydata.so should bring performance back to normal (or even above) in a very simple way. See http://www.flamingspork.com/projects/libeatmydata/ -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
** Bug watch added: Linux Kernel Bug Tracker #15910 http://bugzilla.kernel.org/show_bug.cgi?id=15910 ** Also affects: linux via http://bugzilla.kernel.org/show_bug.cgi?id=15910 Importance: Unknown Status: Unknown -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Jean-Baptists Lallement tested this behaviour and found that ext4's guarantee seems to be ... not so much of a guarantee in reality. I asked him to file it on bugzilla.kernel.org. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Sorry, Jean-Baptiste Lallement -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
I second that, i love to see some background info! But it looks like either the release notes or the ext4 documenation needs updating: === Release notes === `ext4` does not guarantee atomic renames of new files over existing files in the event of a power failure shortly after the rename === ext4 docs ( http://www.mjmwired.net/kernel/Documentation/filesystems/ext4.txt#338 ) === the data blocks of the new file are forced to disk before the rename() operation is committed. This provides roughly the same level of guarantees as ext3, and avoids the zero-length problem that can happen when a system crashes before the delayed allocation blocks are forced to disk. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
The explanation in the release notes is confusing. First of all, as Jakob wrote above, ext4 starting with 2.6.30 provides similar behavior to ext3 with data=ordered. So, it is not clear why fsync() is necessary on ext4, but not on ext3. More importantly, the default mode for ext3 was changed to data=writeback in 2.6.30, which does not provide the same guarantee as ordered. Even if the default was changed to be ordered in Ubuntu kernel, users still may use an upstream kernel or change the mode to writeback on ext3, because the ordered mode has horrible latency (20 times or more than writeback). Does it mean that anyone running ext3 with data=writeback can face corruption of their package repository? -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
I've release-noted this as follows (feel free to tweak from here): == Default file system; package manager performance == The default file system for installations of Ubuntu 10.04 LTS is `ext4`, the latest version in the popular series of Linux extended file systems. `ext4` includes a number of performance tuning changes relative to previous versions such as `ext3`, the file system used by default up to Ubuntu 9.04. These generally produce improvements, but some particular workloads are known to be significantly slower when using `ext4` than when using `ext3`. If you have performance-sensitive applications, we recommend that you run benchmarks using multiple file systems in your environment and select the most appropriate. In particular, the `dpkg` package manager is known to run significantly slower on `ext4` (causing installations using the server or alternate install CD to take on the order of twice as long as before). `ext4` does not guarantee atomic renames of new files over existing files in the event of a power failure shortly after the rename, and so `dpkg` needs to force the contents of the new file out to disk before renaming it in order to avoid leaving corrupt zero-length files after power failures. This operation involves waiting for the disk significantly more than it strictly needs to, and so degrades performance. If fast package management operations are most important to you, then you should use `ext3` instead. (Bug:570805) The simplest way to select a different file system such as `ext3` at installation time is to add the `partman/default_filesystem=ext3` boot parameter when starting the installer. If you are deploying Ubuntu automatically using Kickstart or preseeding, then you can set a different file system in the partitioning recipe instead. ** Changed in: ubuntu-release-notes Status: New = Fix Released ** Changed in: ubuntu-release-notes Assignee: (unassigned) = Colin Watson (cjwatson) -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Looks great. Very informative, Colin. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Thierry- Regarding upgrades, I think I agree with the current design. Data integrity on a running system is of paramount importance. New installs, though, are a different beast, in my opinion, since a failure would necessitate a reinstall anyway. ** Summary changed: - [regression] dpkg fsync cause massive regression in Ubuntu Server installation times + [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
This appears to be a duplicate of bug #537241. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
Phillip- Thanks for the link. The cause, is in fact, the same (fsync's in dpkg). The solution, though, might be subtlety different. As I've said above, I can understand and agree with the change in behavior for updates/upgrades on a running system. This current bug, though, is about what's going on in the installer on fresh installs, when you have no critical data yet on the system. I'm hoping this bug will be fixed separately from bug #537241. -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 570805] Re: [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times
** Tags added: iso-testing -- [regression] dpkg fsync cause massive regression in Ubuntu Server and Alternate installation times https://bugs.launchpad.net/bugs/570805 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
