Re: slow untar during installation
On Mon, Aug 03, 2009 at 09:32:32AM +, Stuart Henderson wrote: > On 2009-08-03, Jan Stary wrote: > > On Jul 30 22:06:41, Stuart Henderson wrote: > >> On 2009-07-30, Jan Stary wrote: > >> > is 128M, that's why I use xserv45.tgz and not base45.tgz; > >> > and it should have been tar xzpf I guess - does it make > >> > a difference? And it shloud be done in single user - but > >> > the machine is almost idle, really.) > >> > >> xserv has a small number of large files, whereas base has a mixture, > >> but it has a lot of small files in it. xzpf won't matter for this test, > >> but if my theory is right, you will need to write lots of small files > >> to see a difference. > > > > I just repeated the same test ten times with xfont45.tgz (is that > > a lot of small files?) - with 'async' it consistently takes between > > 4:00 and 4:10. With 'softdep', it consistently takes between 4:40 > > and 4:50. > > interesting, I don't know why that is, then. > > > Which leaves the original question unanswered: why is the untaring > > during an install so slow when it's async, which I just found to > > perform better than softdep on a running system? > > > >> > For some reason (no reason), I have always thought that nothing can > >> > be faster than async. What is the rationale for mounting the target > >> > filesystems async mounts during an install, anyway? > >> > >> It's a lot faster than mounting them sync. (ramdisks don't have softdep, > >> and also softdep on OpenBSD doesn't free up space from pending deletes > >> quickly enough to be a good choice for untarring new OS file sets). > > > > Again I am confused: how much pending deletes are there during an install > > when untaring the OS sets? > > when you update an already-installed system, loads. > And keep in mind that the ramdisk kernels don't come with softdep. So on updates the filesystem is mounted sync whereas on installs it is async. -- :wq Claudio
Re: slow untar during installation
On 2009-08-03, Jan Stary wrote: > On Jul 30 22:06:41, Stuart Henderson wrote: >> On 2009-07-30, Jan Stary wrote: >> > is 128M, that's why I use xserv45.tgz and not base45.tgz; >> > and it should have been tar xzpf I guess - does it make >> > a difference? And it shloud be done in single user - but >> > the machine is almost idle, really.) >> >> xserv has a small number of large files, whereas base has a mixture, >> but it has a lot of small files in it. xzpf won't matter for this test, >> but if my theory is right, you will need to write lots of small files >> to see a difference. > > I just repeated the same test ten times with xfont45.tgz (is that > a lot of small files?) - with 'async' it consistently takes between > 4:00 and 4:10. With 'softdep', it consistently takes between 4:40 > and 4:50. interesting, I don't know why that is, then. > Which leaves the original question unanswered: why is the untaring > during an install so slow when it's async, which I just found to > perform better than softdep on a running system? > >> > For some reason (no reason), I have always thought that nothing can >> > be faster than async. What is the rationale for mounting the target >> > filesystems async mounts during an install, anyway? >> >> It's a lot faster than mounting them sync. (ramdisks don't have softdep, >> and also softdep on OpenBSD doesn't free up space from pending deletes >> quickly enough to be a good choice for untarring new OS file sets). > > Again I am confused: how much pending deletes are there during an install > when untaring the OS sets? when you update an already-installed system, loads.
Re: slow untar during installation
On Jul 30 22:06:41, Stuart Henderson wrote: > On 2009-07-30, Jan Stary wrote: > > is 128M, that's why I use xserv45.tgz and not base45.tgz; > > and it should have been tar xzpf I guess - does it make > > a difference? And it shloud be done in single user - but > > the machine is almost idle, really.) > > xserv has a small number of large files, whereas base has a mixture, > but it has a lot of small files in it. xzpf won't matter for this test, > but if my theory is right, you will need to write lots of small files > to see a difference. I just repeated the same test ten times with xfont45.tgz (is that a lot of small files?) - with 'async' it consistently takes between 4:00 and 4:10. With 'softdep', it consistently takes between 4:40 and 4:50. Which leaves the original question unanswered: why is the untaring during an install so slow when it's async, which I just found to perform better than softdep on a running system? > > For some reason (no reason), I have always thought that nothing can > > be faster than async. What is the rationale for mounting the target > > filesystems async mounts during an install, anyway? > > It's a lot faster than mounting them sync. (ramdisks don't have softdep, > and also softdep on OpenBSD doesn't free up space from pending deletes > quickly enough to be a good choice for untarring new OS file sets). Again I am confused: how much pending deletes are there during an install when untaring the OS sets? Jan
Re: slow untar during installation
On 2009-07-30, Jan Stary wrote: > is 128M, that's why I use xserv45.tgz and not base45.tgz; > and it should have been tar xzpf I guess - does it make > a difference? And it shloud be done in single user - but > the machine is almost idle, really.) xserv has a small number of large files, whereas base has a mixture, but it has a lot of small files in it. xzpf won't matter for this test, but if my theory is right, you will need to write lots of small files to see a difference. > For some reason (no reason), I have always thought that nothing can > be faster than async. What is the rationale for mounting the target > filesystems async mounts during an install, anyway? It's a lot faster than mounting them sync. (ramdisks don't have softdep, and also softdep on OpenBSD doesn't free up space from pending deletes quickly enough to be a good choice for untarring new OS file sets). >> I have a feeling softdep may be faster in this case, but it has > >> drawbacks too. > > Meaning, a slow IO? > What's specific about async vs. softdep in this case? Making many small updates to a single area of disk (like a directory entry) means a lot of read/write/erase cycles. You can't just update a few bytes in the middle with flash; the controller must read a whole flash block (not related to filesystem block sizes; much larger), make the change, write out the whole block to a new flash block, and erase the old one. Softdep and async do totally different things. Softdep collects metadata updates over a period of time and writes them out together. AIUI this means you skip a huge number of directory writes in this use case. >> The speed rating by itself isn't very useful. The random-access write >> speed has the biggest effect when used on a computer and since it is >> hardly relevant to use with a camera, doesn't usually get mentioned. > > This confuses me: isn't the write speed very relevant when > making shots in quick succession, for example? The key word is *random*, cameras don't do random-access writes, they buffer to RAM then write sequentially, mostly to blank areas of the card rather than making small changes to existing stored data.
Re: slow untar during installation
On Jul 30 11:25:23, Stuart Henderson wrote: > >> it was even async, which makes me curious about the install slowness. > >> Could the card or the exact mount options be the cause of that? > > Well, you already noticed the main difference; one is async, the > other uses softdep. That should suggest a simple test you can make; > untarring on a running system with async vs softdep... Here's what I just did: - downloaded xserv45.tgz onto /home - (re)mounted /tmp with 'async' or 'softdep' respectively - cd /tmp ; time tar xzf /home/hans/xserv45.tgz - rm -rf what came out of it ; umount /tmp - repeat several times, toggling the mount option (I am using /tmp because that's the only filesystem whose options I can fool around with now; and my /tmp is 128M, that's why I use xserv45.tgz and not base45.tgz; and it should have been tar xzpf I guess - does it make a difference? And it shloud be done in single user - but the machine is almost idle, really.) With async, the untar took anywhere between 1:25 and 1:40. With softdep, the untar took anywhere between 1:16 and 1:36. So they are not that much different, and neither of them shows the ultraslow behaviour I was experiencing during the install. What else could it be? For some reason (no reason), I have always thought that nothing can be faster than async. What is the rationale for mounting the target filesystems async mounts during an install, anyway? > I have a feeling softdep may be faster in this case, but it has > drawbacks too. Meaning, a slow IO? What's specific about async vs. softdep in this case? > > I suppose you use an CF-card with about x133 speed. I had the same problem. > > After using a card with at least x233 speed I didn't have that problem when > > untaring anymore. If buying a 'faster' card solves my problem, so be it. But I would like to know, anyway. > The speed rating by itself isn't very useful. The random-access write > speed has the biggest effect when used on a computer and since it is > hardly relevant to use with a camera, doesn't usually get mentioned. This confuses me: isn't the write speed very relevant when making shots in quick succession, for example? Or is it that the write speed is high enough for any camera, and therefore not relevant in a camera, as opposed to computer use, where data could be generated much more quickly? > In particular look for cards which do multisector transfers (i.e. > they /don't/ say "1-sector PIO" in the attach line in dmesg), e.g. > almost all recent sandisk, some innodisk cards, and not many others. Thanks for the tip. $ grep 'sector PIO' dmesg/* dmesg/alix1c:wd0: 1-sector PIO, LBA, 3847MB, 7880544 sectors dmesg/alix2c1:wd0: 1-sector PIO, LBA, 3871MB, 7928928 sectors dmesg/alz:wd0: 16-sector PIO, LBA, 19092MB, 39102336 sectors dmesg/armada:wd0: 16-sector PIO, LBA, 14403MB, 29498112 sectors dmesg/fjfi:wd0: 16-sector PIO, LBA48, 239371MB, 490232639 sectors dmesg/gw:wd0: 1-sector PIO, LBA, 3871MB, 7928928 sectors dmesg/vlada:wd0: 16-sector PIO, LBA, 38166MB, 78165360 sectors dmesg/vlada:wd1: 16-sector PIO, LBA, 78167MB, 160086528 sectors dmesg/wind:wd0: 1-sector PIO, LBA, 7647MB, 15662304 sectors dmesg/wind:wd1: 16-sector PIO, LBA48, 476940MB, 976773168 sectors dmesg/www:wd0: 1-sector PIO, LBA, 3847MB, 7880544 sectors Damn. Seems all of my CF cards are '1-sector PIO' (and all of my hard disks use '16-sector PIO'). > Also on the Alix systems note that there's a UDMA option you can > enable in tinyBIOS that defaults to Off. I don't remember whether I set this, but wd0 at pciide0 channel 0 drive 0: wd0: 1-sector PIO, LBA, 3847MB, 7880544 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 makes me think I have DMA enabled. Thanks for your time Jan
Re: slow untar during installation
On Jul 30 14:29:06, Tobias Walkowiak wrote: > On Thu, Jul 30, 2009 at 11:25:23AM +, Stuart Henderson wrote: > > > > > I suppose you use an CF-card with about x133 speed. I had the same > > > problem. > > > After using a card with at least x233 speed I didn't have that problem > > > when > > > untaring anymore. > > > > The speed rating by itself isn't very useful. The random-access write > > speed has the biggest effect when used on a computer and since it is > > hardly relevant to use with a camera, doesn't usually get mentioned. > > In particular look for cards which do multisector transfers (i.e. > > they /don't/ say "1-sector PIO" in the attach line in dmesg), e.g. > > almost all recent sandisk, some innodisk cards, and not many others. > > I'm using a 2 GB Transcend x300 which says > > # dmesg | grep wd0 > wd0 at pciide0 channel 0 drive 0: > wd0: 1-sector PIO, LBA, 1943MB, 3980592 sectors > wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 > > The speed is absolutely OK and sufficient -- no stalling and nothing that > slows anything down. Repeat: my system, once installed, is absolutely fast enough - it's just during_the_installation the untaring is slow.
Re: slow untar during installation
On Thu, Jul 30, 2009 at 11:25:23AM +, Stuart Henderson wrote: > > > I suppose you use an CF-card with about x133 speed. I had the same problem. > > After using a card with at least x233 speed I didn't have that problem when > > untaring anymore. > > The speed rating by itself isn't very useful. The random-access write > speed has the biggest effect when used on a computer and since it is > hardly relevant to use with a camera, doesn't usually get mentioned. > In particular look for cards which do multisector transfers (i.e. > they /don't/ say "1-sector PIO" in the attach line in dmesg), e.g. > almost all recent sandisk, some innodisk cards, and not many others. I'm using a 2 GB Transcend x300 which says # dmesg | grep wd0 wd0 at pciide0 channel 0 drive 0: wd0: 1-sector PIO, LBA, 1943MB, 3980592 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 The speed is absolutely OK and sufficient -- no stalling and nothing that slows anything down. > Also on the Alix systems note that there's a UDMA option you can > enable in tinyBIOS that defaults to Off. Didn't know that. Thanks for the hint :) -- t.walkow...@wallstreet-online.de
Re: slow untar during installation
>> it was even async, which makes me curious about the install slowness. >> Could the card or the exact mount options be the cause of that? Well, you already noticed the main difference; one is async, the other uses softdep. That should suggest a simple test you can make; untarring on a running system with async vs softdep... I have a feeling softdep may be faster in this case, but it has drawbacks too. > I suppose you use an CF-card with about x133 speed. I had the same problem. > After using a card with at least x233 speed I didn't have that problem when > untaring anymore. The speed rating by itself isn't very useful. The random-access write speed has the biggest effect when used on a computer and since it is hardly relevant to use with a camera, doesn't usually get mentioned. In particular look for cards which do multisector transfers (i.e. they /don't/ say "1-sector PIO" in the attach line in dmesg), e.g. almost all recent sandisk, some innodisk cards, and not many others. Also on the Alix systems note that there's a UDMA option you can enable in tinyBIOS that defaults to Off.
Re: slow untar during installation
On Wed, Jul 29, 2009 at 06:14:18PM +0200, Jan Stary wrote: > When installing 4.5 on an ALIX (see dmesg below) the ALIX is a good choice! > phase of actually untar'ing the tgz sets was very slow: > sets became -stalled- even if untared from local disk (pre-downloaded). I suppose you use an CF-card with about x133 speed. I had the same problem. After using a card with at least x233 speed I didn't have that problem when untaring anymore. > The installation (sans X) took about an hour because of that. And you will wait ages when untaring and compiling the kernel ... > it was even async, which makes me curious about the install slowness. > Could the card or the exact mount options be the cause of that? The card. > I am not trying to tweak the machine in any way to become 'faster'; > once installed, it runs just fine. I just don't want to spend an hour > waiting for the tgz's to untar on the next reinstall. The hardware in itself is fast enogh -- it's only the poor I/O that slows everything down. > wd0 at pciide0 channel 0 drive 0: Better spend your money for a fast 2 GB card than for a slow 4 GB. -- t.walkow...@wallstreet-online.de
