kern.cam…delay, suspend and ZFS pools on USB (was: rc.d/zpool runs before ada(4) attaches)

[Graham Perrin]

On 01/12/2020 20:23, tech-lists wrote:

On Tue, Dec 01, 2020 at 08:34:33AM -0700, Ian Lepore wrote:

On Tue, 2020-12-01 at 16:22 +0100, Ronald Klop wrote:


You can define these in /boot/loader.conf:
#kern.cam.boot_delay="1" # Delay (in ms) of root mount for CAM
bus
#kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI

Maybe that helps.

Ronald.



Those settings control waiting before mounting root.  Harry's problem
is that root is mounted quickly, before other drives are ready for zfs.

The zpool script waits for 'disks'.  It would be nice if the cam
subsystem had something like a sysctl it set to indicate when initial
probing for disks was done, then there could be an rc.d/camprobe script
with 'PROVIDE: disks' which waits for the probing to complete.

-- Ian


kern.cam.boot_delay should still fix it because what is required is a 
delay
while the devices (all of the disks, zfs or not) get ready. Because 
root has to happen before disks/zfs. 



Where a pool is on a USB 2.0 device: might kern.cam.boot_delay, or some 
other delay, negate the need to export the pool before suspending the 
computer?


Related: 



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: rc.d/zpool runs before ada(4) attaches

[Edward Tomasz Napierała]

On 1201T0834, Ian Lepore wrote:
> On Tue, 2020-12-01 at 16:22 +0100, Ronald Klop wrote:

[..]

> > You can define these in /boot/loader.conf:
> > #kern.cam.boot_delay="1" # Delay (in ms) of root mount for CAM
> > bus
> > #kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI
> > 
> > Maybe that helps.
> > 
> > Ronald.
> > 
> 
> Those settings control waiting before mounting root.  Harry's problem
> is that root is mounted quickly, before other drives are ready for zfs.
>  
> The zpool script waits for 'disks'.  It would be nice if the cam
> subsystem had something like a sysctl it set to indicate when initial
> probing for disks was done, then there could be an rc.d/camprobe script
> with 'PROVIDE: disks' which waits for the probing to complete.

I believe such a mechanism already exists, although it might be somewhat
misnamed: the vfs.root_mount_hold sysctl.  Some rc scripts use the
root_hold_wait() function, which uses the sysctl.  Perhaps the ZFS
scripts should do the same?

Note that this is somewhat more involved than just CAM - you also need
to wait for GEOM to settle.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: rc.d/zpool runs before ada(4) attaches

[Harry Schmalzbauer]

Am 01.12.2020 um 16:34 schrieb Ian Lepore::
:
:
:

You can define these in /boot/loader.conf:
#kern.cam.boot_delay="1" # Delay (in ms) of root mount for CAM
bus
#kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI

Maybe that helps.

Ronald.


Those settings control waiting before mounting root.  Harry's problem
is that root is mounted quickly, before other drives are ready for zfs.
  
The zpool script waits for 'disks'.  It would be nice if the cam

subsystem had something like a sysctl it set to indicate when initial
probing for disks was done, then there could be an rc.d/camprobe script
with 'PROVIDE: disks' which waits for the probing to complete.

-- Ian


Until something described above is available, or anybody is aware of any 
other trick,
here's a tested workaround: 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251610


It turned out that also swapon and dumpon suffer from early 
root_hold_wait() release.
For dumpon, cam(4) doesn't even start probing.  Luckily all target:LUNs 
are visible at that earliest stage in rc.d/dumpon.
So that's the point where I check if any real target is in state 
unattached (()) or probing ((aprobe)).
I don't know details of the involved vfs.root_mount_hold sysctl, but 
assume this is dead end currently...


Best regards,
-harry
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: rc.d/zpool runs before ada(4) attaches

[Oleg Sidorkin]

Tue, 1 Dec. 2020 г. в 23:24, tech-lists :
>
> On Tue, Dec 01, 2020 at 08:34:33AM -0700, Ian Lepore wrote:
> >On Tue, 2020-12-01 at 16:22 +0100, Ronald Klop wrote:
> >>
> >> You can define these in /boot/loader.conf:
> >> #kern.cam.boot_delay="1" # Delay (in ms) of root mount for CAM
> >> bus
> >> #kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI
> >>
> >> Maybe that helps.
> >>
> >> Ronald.
> >>
> >
> >Those settings control waiting before mounting root.  Harry's problem
> >is that root is mounted quickly, before other drives are ready for zfs.
> >
> >The zpool script waits for 'disks'.  It would be nice if the cam
> >subsystem had something like a sysctl it set to indicate when initial
> >probing for disks was done, then there could be an rc.d/camprobe script
> >with 'PROVIDE: disks' which waits for the probing to complete.
> >
> >-- Ian
>
> kern.cam.boot_delay should still fix it because what is required is a delay
> while the devices (all of the disks, zfs or not) get ready. Because root
> has to happen before disks/zfs.
> --
> J.

I've reported a similar problem here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242189 some time
ago.
I just added the patch that solves it for me by delaying startup until
CAM and USB scan is complete (that's overkill probably).


-- 
Oleg Sidorkin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: rc.d/zpool runs before ada(4) attaches

[tech-lists]

On Tue, Dec 01, 2020 at 08:34:33AM -0700, Ian Lepore wrote:

On Tue, 2020-12-01 at 16:22 +0100, Ronald Klop wrote:


You can define these in /boot/loader.conf:
#kern.cam.boot_delay="1" # Delay (in ms) of root mount for CAM
bus
#kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI

Maybe that helps.

Ronald.



Those settings control waiting before mounting root.  Harry's problem
is that root is mounted quickly, before other drives are ready for zfs.

The zpool script waits for 'disks'.  It would be nice if the cam
subsystem had something like a sysctl it set to indicate when initial
probing for disks was done, then there could be an rc.d/camprobe script
with 'PROVIDE: disks' which waits for the probing to complete.

-- Ian


kern.cam.boot_delay should still fix it because what is required is a delay
while the devices (all of the disks, zfs or not) get ready. Because root 
has to happen before disks/zfs. 
--

J.


signature.asc
Description: PGP signature

Re: rc.d/zpool runs before ada(4) attaches

[Ronald Klop]

Van: Ian Lepore 
Datum: dinsdag, 1 december 2020 16:34
Aan: Ronald Klop , FreeBSD Current 

Onderwerp: Re: rc.d/zpool runs before ada(4) attaches


On Tue, 2020-12-01 at 16:22 +0100, Ronald Klop wrote:
>  
> Van: Harry Schmalzbauer 

> Datum: dinsdag, 1 december 2020 12:51
> Aan: Ronald Klop , FreeBSD Current <
> freebsd-current@freebsd.org>
> Onderwerp: Re: rc.d/zpool runs before ada(4) attaches
> >
> > Am 01.12.2020 um 10:33 schrieb Ronald Klop:
> > :
> > :
> > :
> > > > One machine fails importing zpool because the correponding
> > > > vdevs >> (ada0-ada2)
> > > > are not available at the time rc.d/zpool runs.
> > > >
> > > >
> > > > Adhoc  I'm not aware of any rc(8) vs. driver awareness.
> > > > Is there any?
> > > >
> > > > Suggestions how to fix else than 'sleep 1'?
> > > >
> > > > Thanks,
> > > >
> > > > -harry
> > > >
> > > > P.S.: ahci(4) is compiled into kernel, machine is a HPE U48
> > > > (Gen 10 >> plus MicroServer), zfsloader loads root_MFS kernel
> > > > module
> > > >
> > >
> > >
> > > There have been some changes to etc/rc.d/zpool in September.
> > > Do you have the latest version? Compare with:
> > >
https://github.com/freebsd/freebsd/blob/master/libexec/rc/rc.d/zpool
> > > or
> > >
https://svnweb.freebsd.org/base/head/libexec/rc/rc.d/zpool?revision=365354&view=markup
> > >  >
> > >
> > > Otherwise it would be helpful for readers if you could post some
> > > logs > which indicate what is happening.
> > > /var/run/dmesg.boot or the output of "dmesg"
> > > Part of /var/log/messages
> > > Part of /var/log/console.log if it exists.
> > >
> >
> > Thanks, I'm on -current from view days ago.
> > The problem is that cam(4) is still probing devices, when
> > rc.d/zpool runs, since mount_root_from succeeded, because it is a
> > RAM disk, so succeeds independent of any real drive/controller
> > probing.
> > I can imagine of other seldom edgecases hitting the issue too.
> >
> > So my proposed patch, working for me, looks like this:
> > Index: libexec/rc/rc.d/zpool
> > ===
> > --- libexec/rc/rc.d/zpool   (revision 368202)
> > +++ libexec/rc/rc.d/zpool   (working copy)
> > @@ -18,8 +18,16 @@
> >
> >   zpool_start()
> >   {
> > -   local cachefile
> > +local cachefile n=0 camlist=`camcontrol devlist -v`
> >
> > +   # Wait for cam(4) devices attaching, 4 times at max by
> > increasing
> > +   # 1s each (10s max in total)
> > +while [ X"${camlist#*target*lun*probe}" != X"${camlist}"
> > ]; do
> > +   [ $n -lt 4 ] || break
> > +   sleep $((n+=1))
> > +   camlist=`camcontrol devlist -v`
> > +   done
> > +
> >  for cachefile in /etc/zfs/zpool.cache
> > /boot/zfs/zpool.cache; do
> >  if [ -r $cachefile ]; then
> >  zpool import -c $cachefile -a -N && break
> >
> > best,
> > -harry
> >
> >
> >
>
> You can define these in /boot/loader.conf:
> #kern.cam.boot_delay="1" # Delay (in ms) of root mount for CAM
> bus
> #kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI
>
> Maybe that helps.
>
> Ronald.
>

Those settings control waiting before mounting root.  Harry's problem
is that root is mounted quickly, before other drives are ready for zfs.
 
The zpool script waits for 'disks'.  It would be nice if the cam

subsystem had something like a sysctl it set to indicate when initial
probing for disks was done, then there could be an rc.d/camprobe script
with 'PROVIDE: disks' which waits for the probing to complete.

-- Ian

 






Or a devd event "ada-arrived" which calls rc.d/zpool.
It sounds a bit like we need systemd. :-)

Ronald.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: rc.d/zpool runs before ada(4) attaches

[Ian Lepore]

On Tue, 2020-12-01 at 16:22 +0100, Ronald Klop wrote:
>  
> Van: Harry Schmalzbauer 
> Datum: dinsdag, 1 december 2020 12:51
> Aan: Ronald Klop , FreeBSD Current <
> freebsd-current@freebsd.org>
> Onderwerp: Re: rc.d/zpool runs before ada(4) attaches
> > 
> > Am 01.12.2020 um 10:33 schrieb Ronald Klop:
> > :
> > :
> > :
> > > > One machine fails importing zpool because the correponding
> > > > vdevs >> (ada0-ada2)
> > > > are not available at the time rc.d/zpool runs.
> > > > 
> > > > 
> > > > Adhoc  I'm not aware of any rc(8) vs. driver awareness.
> > > > Is there any?
> > > > 
> > > > Suggestions how to fix else than 'sleep 1'?
> > > > 
> > > > Thanks,
> > > > 
> > > > -harry
> > > > 
> > > > P.S.: ahci(4) is compiled into kernel, machine is a HPE U48
> > > > (Gen 10 >> plus MicroServer), zfsloader loads root_MFS kernel
> > > > module
> > > > 
> > > 
> > > 
> > > There have been some changes to etc/rc.d/zpool in September.
> > > Do you have the latest version? Compare with:
> > > 
https://github.com/freebsd/freebsd/blob/master/libexec/rc/rc.d/zpool
> > > or
> > > 
https://svnweb.freebsd.org/base/head/libexec/rc/rc.d/zpool?revision=365354&view=markup
> > >  >
> > > 
> > > Otherwise it would be helpful for readers if you could post some
> > > logs > which indicate what is happening.
> > > /var/run/dmesg.boot or the output of "dmesg"
> > > Part of /var/log/messages
> > > Part of /var/log/console.log if it exists.
> > > 
> > 
> > Thanks, I'm on -current from view days ago.
> > The problem is that cam(4) is still probing devices, when
> > rc.d/zpool runs, since mount_root_from succeeded, because it is a
> > RAM disk, so succeeds independent of any real drive/controller
> > probing.
> > I can imagine of other seldom edgecases hitting the issue too.
> > 
> > So my proposed patch, working for me, looks like this:
> > Index: libexec/rc/rc.d/zpool
> > ===
> > --- libexec/rc/rc.d/zpool   (revision 368202)
> > +++ libexec/rc/rc.d/zpool   (working copy)
> > @@ -18,8 +18,16 @@
> > 
> >   zpool_start()
> >   {
> > -   local cachefile
> > +local cachefile n=0 camlist=`camcontrol devlist -v`
> > 
> > +   # Wait for cam(4) devices attaching, 4 times at max by
> > increasing
> > +   # 1s each (10s max in total)
> > +while [ X"${camlist#*target*lun*probe}" != X"${camlist}"
> > ]; do
> > +   [ $n -lt 4 ] || break
> > +   sleep $((n+=1))
> > +   camlist=`camcontrol devlist -v`
> > +   done
> > +
> >  for cachefile in /etc/zfs/zpool.cache
> > /boot/zfs/zpool.cache; do
> >  if [ -r $cachefile ]; then
> >  zpool import -c $cachefile -a -N && break
> > 
> > best,
> > -harry
> > 
> > 
> > 
> 
> You can define these in /boot/loader.conf:
> #kern.cam.boot_delay="1" # Delay (in ms) of root mount for CAM
> bus
> #kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI
> 
> Maybe that helps.
> 
> Ronald.
> 

Those settings control waiting before mounting root.  Harry's problem
is that root is mounted quickly, before other drives are ready for zfs.
 
The zpool script waits for 'disks'.  It would be nice if the cam
subsystem had something like a sysctl it set to indicate when initial
probing for disks was done, then there could be an rc.d/camprobe script
with 'PROVIDE: disks' which waits for the probing to complete.

-- Ian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: rc.d/zpool runs before ada(4) attaches

[Ronald Klop]

Van: Harry Schmalzbauer 
Datum: dinsdag, 1 december 2020 12:51
Aan: Ronald Klop , FreeBSD Current 

Onderwerp: Re: rc.d/zpool runs before ada(4) attaches


Am 01.12.2020 um 10:33 schrieb Ronald Klop:
:
:
:
>> One machine fails importing zpool because the correponding vdevs >> 
(ada0-ada2)
>> are not available at the time rc.d/zpool runs.
>>
>>
>> Adhoc  I'm not aware of any rc(8) vs. driver awareness.
>> Is there any?
>>
>> Suggestions how to fix else than 'sleep 1'?
>>
>> Thanks,
>>
>> -harry
>>
>> P.S.: ahci(4) is compiled into kernel, machine is a HPE U48 (Gen 10 >> plus 
MicroServer), zfsloader loads root_MFS kernel module
>>
>
>
> There have been some changes to etc/rc.d/zpool in September.
> Do you have the latest version? Compare with:
> https://github.com/freebsd/freebsd/blob/master/libexec/rc/rc.d/zpool
> or
> 
https://svnweb.freebsd.org/base/head/libexec/rc/rc.d/zpool?revision=365354&view=markup
 >
>
> Otherwise it would be helpful for readers if you could post some logs > which 
indicate what is happening.
> /var/run/dmesg.boot or the output of "dmesg"
> Part of /var/log/messages
> Part of /var/log/console.log if it exists.
>

Thanks, I'm on -current from view days ago.
The problem is that cam(4) is still probing devices, when rc.d/zpool runs, 
since mount_root_from succeeded, because it is a RAM disk, so succeeds 
independent of any real drive/controller probing.
I can imagine of other seldom edgecases hitting the issue too.

So my proposed patch, working for me, looks like this:
Index: libexec/rc/rc.d/zpool
===
--- libexec/rc/rc.d/zpool   (revision 368202)
+++ libexec/rc/rc.d/zpool   (working copy)
@@ -18,8 +18,16 @@

  zpool_start()
  {
-   local cachefile
+local cachefile n=0 camlist=`camcontrol devlist -v`

+   # Wait for cam(4) devices attaching, 4 times at max by increasing
+   # 1s each (10s max in total)
+while [ X"${camlist#*target*lun*probe}" != X"${camlist}" ]; do
+   [ $n -lt 4 ] || break
+   sleep $((n+=1))
+   camlist=`camcontrol devlist -v`
+   done
+
 for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do
 if [ -r $cachefile ]; then
 zpool import -c $cachefile -a -N && break

best,
-harry





You can define these in /boot/loader.conf:
#kern.cam.boot_delay="1" # Delay (in ms) of root mount for CAM bus
#kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI

Maybe that helps.

Ronald.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: rc.d/zpool runs before ada(4) attaches

[Harry Schmalzbauer]

Am 01.12.2020 um 10:33 schrieb Ronald Klop:
:
:
:
One machine fails importing zpool because the correponding vdevs 
(ada0-ada2)

are not available at the time rc.d/zpool runs.


Adhoc  I'm not aware of any rc(8) vs. driver awareness.
Is there any?

Suggestions how to fix else than 'sleep 1'?

Thanks,

-harry

P.S.: ahci(4) is compiled into kernel, machine is a HPE U48 (Gen 10 
plus MicroServer), zfsloader loads root_MFS kernel module





There have been some changes to etc/rc.d/zpool in September.
Do you have the latest version? Compare with:
https://github.com/freebsd/freebsd/blob/master/libexec/rc/rc.d/zpool
or
https://svnweb.freebsd.org/base/head/libexec/rc/rc.d/zpool?revision=365354&view=markup 



Otherwise it would be helpful for readers if you could post some logs 
which indicate what is happening.

/var/run/dmesg.boot or the output of "dmesg"
Part of /var/log/messages
Part of /var/log/console.log if it exists.



Thanks, I'm on -current from view days ago.
The problem is that cam(4) is still probing devices, when rc.d/zpool 
runs, since mount_root_from succeeded, because it is a RAM disk, so 
succeeds independent of any real drive/controller probing.

I can imagine of other seldom edgecases hitting the issue too.

So my proposed patch, working for me, looks like this:
Index: libexec/rc/rc.d/zpool
===
--- libexec/rc/rc.d/zpool   (revision 368202)
+++ libexec/rc/rc.d/zpool   (working copy)
@@ -18,8 +18,16 @@

 zpool_start()
 {
-   local cachefile
+    local cachefile n=0 camlist=`camcontrol devlist -v`

+   # Wait for cam(4) devices attaching, 4 times at max by increasing
+   # 1s each (10s max in total)
+    while [ X"${camlist#*target*lun*probe}" != X"${camlist}" ]; do
+   [ $n -lt 4 ] || break
+   sleep $((n+=1))
+   camlist=`camcontrol devlist -v`
+   done
+
    for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do
    if [ -r $cachefile ]; then
    zpool import -c $cachefile -a -N && break

best,
-harry
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: rc.d/zpool runs before ada(4) attaches

[Ronald Klop]

Van: Harry Schmalzbauer 
Datum: dinsdag, 1 december 2020 09:43
Aan: freebsd-current@freebsd.org
Onderwerp: rc.d/zpool runs before ada(4) attaches


Hello,  I'm playing with HEAD (post r364746, Merge OpenZFS support in to HEAD) 
on some of my non-out-of-box setups.


One machine fails importing zpool because the correponding vdevs (ada0-ada2)
are not available at the time rc.d/zpool runs.


Adhoc  I'm not aware of any rc(8) vs. driver awareness.
Is there any?

Suggestions how to fix else than 'sleep 1'?

Thanks,

-harry

P.S.: ahci(4) is compiled into kernel, machine is a HPE U48 (Gen 10 plus 
MicroServer), zfsloader loads root_MFS kernel module
 



There have been some changes to etc/rc.d/zpool in September.
Do you have the latest version? Compare with:
https://github.com/freebsd/freebsd/blob/master/libexec/rc/rc.d/zpool
or
https://svnweb.freebsd.org/base/head/libexec/rc/rc.d/zpool?revision=365354&view=markup

Otherwise it would be helpful for readers if you could post some logs which 
indicate what is happening.
/var/run/dmesg.boot or the output of "dmesg"
Part of /var/log/messages
Part of /var/log/console.log if it exists.

Regards,
Ronald.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

rc.d/zpool runs before ada(4) attaches

[Harry Schmalzbauer]

Hello,  I'm playing with HEAD (post r364746, Merge OpenZFS support in to 
HEAD) on some of my non-out-of-box setups.



One machine fails importing zpool because the correponding vdevs (ada0-ada2)
are not available at the time rc.d/zpool runs.


Adhoc  I'm not aware of any rc(8) vs. driver awareness.
Is there any?

Suggestions how to fix else than 'sleep 1'?

Thanks,

-harry

P.S.: ahci(4) is compiled into kernel, machine is a HPE U48 (Gen 10 plus 
MicroServer), zfsloader loads root_MFS kernel module


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"