RE: EXT MAIL : Re: USB install is slow
Sorry was travelling Times are correct because the USB drive for netbsd 10 I used is usb3 while the netbsd 9 is on a old usb 2 device Anyways I have been able to check this further When I choose system drive during upgrade the upgrade takes long i.e 25 to 45 minutes However if I choose the actual drive then its done in less than 5 minutes.. so I will use the actual drive going further Thanks Everyone -Original Message- From: Martin Husemann Sent: Thursday, March 14, 2024 4:39 AM To: Derrick Lobo Cc: pms-...@outlook.com; netbsd-users@netbsd.org Subject: Re: EXT MAIL : Re: USB install is slow On Thu, Mar 14, 2024 at 02:02:44AM +, Derrick Lobo wrote: > Here are the details. Has anyone else experience this issue netbsd > 9.0 time cat /mnt/amd64/binary/sets/base.tarxz > /dev/null 5.65s real > 0.00s user 0.06s system > > netbsd 10rc5 > time cat /mnt/amd64/binary/sets/base.tarxz > /dev/null 1.59s real > 0.00s user 0.11s system Are those times swapped? Otherwise I would read them as 10.0_RC5 being about 3 times faster... Martin
Re: EXT MAIL : Re: USB install is slow
On Thu, Mar 28, 2024 at 12:56:29PM +, Derrick Lobo wrote: > When I choose system drive during upgrade the upgrade takes long i.e 25 to 45 > minutes > > However if I choose the actual drive then its done in less than 5 minutes.. > so I will use the actual drive going further Sorry, I fail to parse that - can you tell more about the two drives you used? Martin
Re: EXT MAIL : Re: USB install is slow
On Thu, Mar 14, 2024 at 02:02:44AM +, Derrick Lobo wrote: > Here are the details. Has anyone else experience this issue > netbsd 9.0 > time cat /mnt/amd64/binary/sets/base.tarxz > /dev/null > 5.65s real 0.00s user 0.06s system > > netbsd 10rc5 > time cat /mnt/amd64/binary/sets/base.tarxz > /dev/null > 1.59s real 0.00s user 0.11s system Are those times swapped? Otherwise I would read them as 10.0_RC5 being about 3 times faster... Martin
RE: EXT MAIL : Re: USB install is slow
Here are the details. Has anyone else experience this issue netbsd 9.0 time cat /mnt/amd64/binary/sets/base.tarxz > /dev/null 5.65s real 0.00s user 0.06s system netbsd 10rc5 time cat /mnt/amd64/binary/sets/base.tarxz > /dev/null 1.59s real 0.00s user 0.11s system -Original Message- From: Martin Husemann Sent: Monday, March 11, 2024 10:11 AM To: Derrick Lobo Cc: pms-...@outlook.com; netbsd-users@netbsd.org Subject: Re: EXT MAIL : Re: USB install is slow On Mon, Mar 11, 2024 at 01:31:26PM +, Derrick Lobo wrote: > I have reproduced this issue on multiple servers with the following > cpus using the amd64 USB image on all of them takes 20 to 45 minutes > which normally on netbsd9 took 3 minutes max to complete the USB > install. None of our servers have CD/DVD It would be really suprising if any of that CPUs would make a difference. Have you tried a different USB stick? It is still totally unclear where your machines spends all the time. Instead of installation you could benchmark reading the sets first, something like: time cat /mnt/.../base.tar.xz > /dev/null and compare that with 9.x numbers. Martin
RE: EXT MAIL : Re: USB install is slow
Yes tried with multiple usb even different brands I will try to benchmark the reads tomorrow when I have physical access to the devices -Original Message- From: Martin Husemann Sent: Monday, March 11, 2024 10:11 AM To: Derrick Lobo Cc: pms-...@outlook.com; netbsd-users@netbsd.org Subject: Re: EXT MAIL : Re: USB install is slow On Mon, Mar 11, 2024 at 01:31:26PM +, Derrick Lobo wrote: > I have reproduced this issue on multiple servers with the following > cpus using the amd64 USB image on all of them takes 20 to 45 minutes > which normally on netbsd9 took 3 minutes max to complete the USB > install. None of our servers have CD/DVD It would be really suprising if any of that CPUs would make a difference. Have you tried a different USB stick? It is still totally unclear where your machines spends all the time. Instead of installation you could benchmark reading the sets first, something like: time cat /mnt/.../base.tar.xz > /dev/null and compare that with 9.x numbers. Martin
Re: EXT MAIL : Re: USB install is slow
On Mon, Mar 11, 2024 at 01:31:26PM +, Derrick Lobo wrote: > I have reproduced this issue on multiple servers with the following > cpus using the amd64 USB image on all of them takes 20 to 45 minutes > which normally on netbsd9 took 3 minutes max to complete the USB > install. None of our servers have CD/DVD It would be really suprising if any of that CPUs would make a difference. Have you tried a different USB stick? It is still totally unclear where your machines spends all the time. Instead of installation you could benchmark reading the sets first, something like: time cat /mnt/.../base.tar.xz > /dev/null and compare that with 9.x numbers. Martin
RE: EXT MAIL : Re: USB install is slow
I have reproduced this issue on multiple servers with the following cpus using the amd64 USB image on all of them takes 20 to 45 minutes which normally on netbsd9 took 3 minutes max to complete the USB install. None of our servers have CD/DVD The 3 types of cpus are listed below [ 1.04] cpu0 at mainbus0 apid 0 [ 1.04] cpu0: Use lfence to serialize rdtsc [ 1.04] cpu0: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7 [ 1.04] cpu0: node 0, package 0, core 0, smt 0 [ 1.04] cpu1 at mainbus0 apid 2 - [ 1.04] cpu0 at mainbus0 apid 32 [ 1.04] cpu0: Use mfence to serialize rdtsc [ 1.04] cpu0: AMD Opteron(tm) Processor 6376 , id 0x600f20 [ 1.04] cpu0: node 0, package 0, core 0, smt 0 - [ 1.04] cpu79 at mainbus0 apid 121 [ 1.04] cpu79: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz, id 0x406f1 [ 1.04] cpu79: node 1, package 1, core 28, smt 1 -Original Message- From: netbsd-users-ow...@netbsd.org On Behalf Of pms-...@outlook.com Sent: Friday, March 8, 2024 4:10 PM To: netbsd-users@netbsd.org Subject: EXT MAIL : Re: USB install is slow Derrick Lobo wrote: > Hi All > I have noticed the USB install taking way too long anyone else noticed > it Based on regular instructions I used the usb installer rawrite32 to > create a usb install stick The installs for all netbsd10-rcx have been > taking roughly 25 minutes, when I do an upgrade from 9.. with version > 9.3 and below the upgrade via usb used to be until 5 minutes.. is this > a bug or am I doing something wrong.. and has anyone else noticed > this.. > Derrick What kind of CPU do you have? CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: USB install is slow
Derrick Lobo wrote: Hi All I have noticed the USB install taking way too long anyone else noticed it Based on regular instructions I used the usb installer rawrite32 to create a usb install stick The installs for all netbsd10-rcx have been taking roughly 25 minutes, when I do an upgrade from 9.. with version 9.3 and below the upgrade via usb used to be until 5 minutes.. is this a bug or am I doing something wrong.. and has anyone else noticed this.. Derrick What kind of CPU do you have?
Re: EXT MAIL : Re: USB install is slow
On Fri, Mar 08, 2024 at 08:18:14PM +, Derrick Lobo wrote: > Hi Thierry > > Manual untar seems fine no issues at all. This seems more like > copying the file between usb to the local drive as I see stalls when > the file is copied over But you said the time was spent in the download mostly - and I am totally confused what times you are comparing against which. The sets should be on the install usb medium, so no download is necessary. The download speed itself may vary widely and is totaly outside of our controll. When you say untarring the sets is fast, is that after copying them from the USB stick to hard disk or untarring from the USB source directly? Martin
RE: EXT MAIL : Re: USB install is slow
Hi Thierry Manual untar seems fine no issues at all. This seems more like copying the file between usb to the local drive as I see stalls when the file is copied over I am hoping someone can reproduce this issue I have reproduced this on 6 servers.. most of them supermicros 1 was a lanner device -Original Message- From: tlaro...@kergis.com Sent: Friday, March 8, 2024 2:43 PM To: Martin Husemann Cc: Derrick Lobo ; netbsd-users@netbsd.org Subject: Re: EXT MAIL : Re: USB install is slow On Fri, Mar 08, 2024 at 04:57:00PM +0100, Martin Husemann wrote: > On Fri, Mar 08, 2024 at 03:52:15PM +, Derrick Lobo wrote: > > I ended just untaring the sets and running etcupdate as that was > > much quicker > > Huh? If that was much quicker (you were running the same kernel, from > the install image?) - where did the upgrade spend all the additional time? > Extracting sets should be the major part, plus a bit of "postinstall" > where it hashes all the (new with -10) SSL certificates (and the > script doing so is suboptimal). Neither should take even close to 10 > minutes on amd64 hardware from the last 10 years or so. A shot in the dark: there is a problem report on pkgsrc mailing list linked with tar and extended attributes. In the 10.* series, extended attributes have been added to libc (man 3 acl). Furthermore, modifications have been made, concerning extended attributes, to UFS (with a problem of incompatibility). Possible culprit: tar and extended attributes (libc, kernel, filesystem), involving acrobatics costing a lot of time. -- Thierry Laronde http://www.kergis.com/ http://kertex.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: EXT MAIL : Re: USB install is slow
On Fri, Mar 08, 2024 at 04:57:00PM +0100, Martin Husemann wrote: > On Fri, Mar 08, 2024 at 03:52:15PM +, Derrick Lobo wrote: > > I ended just untaring the sets and running etcupdate as that was much > > quicker > > Huh? If that was much quicker (you were running the same kernel, from the > install image?) - where did the upgrade spend all the additional time? > Extracting sets should be the major part, plus a bit of "postinstall" where > it hashes all the (new with -10) SSL certificates (and the script doing so > is suboptimal). Neither should take even close to 10 minutes on amd64 hardware > from the last 10 years or so. A shot in the dark: there is a problem report on pkgsrc mailing list linked with tar and extended attributes. In the 10.* series, extended attributes have been added to libc (man 3 acl). Furthermore, modifications have been made, concerning extended attributes, to UFS (with a problem of incompatibility). Possible culprit: tar and extended attributes (libc, kernel, filesystem), involving acrobatics costing a lot of time. -- Thierry Laronde http://www.kergis.com/ http://kertex.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
RE: EXT MAIL : Re: USB install is slow
Most of the time was spent downloading the tarfile and extracting from the USB once the files are download the install completes quickly -Original Message- From: Martin Husemann Sent: Friday, March 8, 2024 10:57 AM To: Derrick Lobo Cc: netbsd-users@netbsd.org Subject: Re: EXT MAIL : Re: USB install is slow On Fri, Mar 08, 2024 at 03:52:15PM +, Derrick Lobo wrote: > I ended just untaring the sets and running etcupdate as that was much > quicker Huh? If that was much quicker (you were running the same kernel, from the install image?) - where did the upgrade spend all the additional time? Extracting sets should be the major part, plus a bit of "postinstall" where it hashes all the (new with -10) SSL certificates (and the script doing so is suboptimal). Neither should take even close to 10 minutes on amd64 hardware from the last 10 years or so. Martin
Re: EXT MAIL : Re: USB install is slow
On Fri, Mar 08, 2024 at 03:52:15PM +, Derrick Lobo wrote: > I ended just untaring the sets and running etcupdate as that was much > quicker Huh? If that was much quicker (you were running the same kernel, from the install image?) - where did the upgrade spend all the additional time? Extracting sets should be the major part, plus a bit of "postinstall" where it hashes all the (new with -10) SSL certificates (and the script doing so is suboptimal). Neither should take even close to 10 minutes on amd64 hardware from the last 10 years or so. Martin
Re: USB install is slow
On Thu, Mar 07, 2024 at 03:40:15PM +, Derrick Lobo wrote: > I have noticed the USB install taking way too long anyone else noticed it On what hardware? This may depend on (a) the target disk [e.g. new driver bugs making writes slow] or (b) the USB stick and what controller it is pluged in [i.e. slow reads] - or something else going wrong in the new kernel during the update [e.g. an interrupt storm due to some unrelated device/driver bug]. Please file a PR and include full dmesg output of the installer kernel. Martin
USB install is slow
Hi All I have noticed the USB install taking way too long anyone else noticed it Based on regular instructions I used the usb installer rawrite32 to create a usb install stick The installs for all netbsd10-rcx have been taking roughly 25 minutes, when I do an upgrade from 9.. with version 9.3 and below the upgrade via usb used to be until 5 minutes.. is this a bug or am I doing something wrong.. and has anyone else noticed this.. Derrick
