A friendly weekly ping.
Thanks,
Ash
Ashton Fagg <[email protected]> writes:
> I have been working on fixing an issue (which was partially fixed by a
> diff I sent in earlier this year) with iscsid. With iscsi disks in
> /etc/fstab, sometimes the devices aren't fully up and ready before fsck
> tries to run - causing the machine to go into single user mode on boot.
>
> The diff that was merged earlier in the year added a poll routine which
> monitors for connection success before letting iscsictl return. This
> fixed the issue in some cases, however it's still only half the fix. The
> principled fix is to, additionally, wait until the scsi_probe calls are
> complete - at which point we can reasonably assume the disk device are
> ready for use. This requires some work to the vscsi driver to make this
> happen, as well as changes to both iscsid and iscsictl. The diffs
> attached here are a full implementation of this.
>
> I was still encountering this issue on one of my machines (much slower
> than my normal machine), where the connections would be up but the
> scsi_probe calls would not have completed - even with my earlier diff
> this would still cause the machine to go to single user mode. However,
> this indeed fixes that problem completely and I've been running it for a
> couple of weeks with no problems.
>
> The diffs are designed to be applied in the order they appear. In
> summary, the proposed changes are:
>
> (1) taskq.diff adds a taskq_empty function, which lets us check to see
> if a taskq is, well, empty. This is used in (2). Updates the man pages
> for taskq/task_add to reflect this.
>
> (2) vscsi.diff adds an ioctl to the vscsi driver which lets us monitor
> for device event queue completion. To aid in this it also separates
> calls to scsi_probe and scsi_detach to a dedicated taskq, rather than
> using systq. Updates the man pages for vscsi to reflect this.
>
> (3) iscsid.diff does the plumbing to actually call the new ioctl and
> provide that information back to iscsictl during status poll.
>
> (4) iscsictl.diff integrates the device queue information into the
> polling routine. Updates the man page for iscsictl to describe the new
> behavior.
>
> Based on commmit messages around vscsi it seems there was some plan to
> do this quite some years ago but it's hard for me to know what the
> proposed method was (though I assume what was envisaged might be similar
> to something like this).
>
> Feedback sought and greatly welcomed. I am basically certain there are
> ways this can be improved.
>
> Thanks,
>
> Ash
Index: sys/sys/task.h
===================================================================
RCS file: /cvs/src/sys/sys/task.h,v
retrieving revision 1.18
diff -u -p -u -p -r1.18 task.h
--- sys/sys/task.h 1 Aug 2020 08:40:20 -0000 1.18
+++ sys/sys/task.h 8 Jun 2021 23:42:00 -0000
@@ -51,6 +51,8 @@ void taskq_barrier(struct taskq *);
void taskq_del_barrier(struct taskq *, struct task *);
+int taskq_empty(struct taskq *);
+
void task_set(struct task *, void (*)(void *), void *);
int task_add(struct taskq *, struct task *);
int task_del(struct taskq *, struct task *);
Index: sys/kern/kern_task.c
===================================================================
RCS file: /cvs/src/sys/kern/kern_task.c,v
retrieving revision 1.31
diff -u -p -u -p -r1.31 kern_task.c
--- sys/kern/kern_task.c 1 Aug 2020 08:40:20 -0000 1.31
+++ sys/kern/kern_task.c 8 Jun 2021 23:42:16 -0000
@@ -394,6 +394,18 @@ task_del(struct taskq *tq, struct task *
}
int
+taskq_empty(struct taskq *tq)
+{
+ int rv;
+
+ mtx_enter(&tq->tq_mtx);
+ rv = TAILQ_EMPTY(&tq->tq_worklist);
+ mtx_leave(&tq->tq_mtx);
+
+ return (rv);
+}
+
+int
taskq_next_work(struct taskq *tq, struct task *work)
{
struct task *next;
Index: share/man/man9/task_add.9
===================================================================
RCS file: /cvs/src/share/man/man9/task_add.9,v
retrieving revision 1.22
diff -u -p -u -p -r1.22 task_add.9
--- share/man/man9/task_add.9 8 Jun 2020 00:29:51 -0000 1.22
+++ share/man/man9/task_add.9 8 Jun 2021 23:42:29 -0000
@@ -20,6 +20,7 @@
.Sh NAME
.Nm taskq_create ,
.Nm taskq_destroy ,
+.Nm taskq_empty ,
.Nm taskq_barrier ,
.Nm taskq_del_barrier ,
.Nm task_set ,
@@ -43,6 +44,8 @@
.Fn taskq_barrier "struct taskq *tq"
.Ft void
.Fn taskq_del_barrier "struct taskq *tq" "struct task *t"
+.Ft int
+.Fn taskq_empty "struct taskq *tq"
.Ft void
.Fn task_set "struct task *t" "void (*fn)(void *)" "void *arg"
.Ft int
@@ -86,6 +89,9 @@ argument:
The threads servicing the taskq will be run without the kernel big lock.
.El
.Pp
+.Fn taskq_empty
+indicates whether there are any queued tasks.
+.Pp
.Fn taskq_destroy
causes the resources associated with a previously created taskq to be freed.
It will wait till all the tasks in the work queue are completed before
@@ -220,6 +226,11 @@ or 0 if the task was not already on the
.Fn task_pending
will return non-zero if the task is queued to run, or 0 if the task
is not queued.
+.Pp
+.Fn taskq_empty
+will return 1 if there are no tasks queued, or 0 if there is at least
+one task queued.
+
.Sh SEE ALSO
.Xr autoconf 9 ,
.Xr spl 9
Index: sys/dev/vscsivar.h
===================================================================
RCS file: /cvs/src/sys/dev/vscsivar.h,v
retrieving revision 1.5
diff -u -p -u -p -r1.5 vscsivar.h
--- sys/dev/vscsivar.h 5 Apr 2011 15:28:49 -0000 1.5
+++ sys/dev/vscsivar.h 8 Jun 2021 23:07:53 -0000
@@ -44,7 +44,7 @@ struct vscsi_ioc_data {
size_t datalen;
};
-#define VSCSI_DATA_READ _IOW('I', 1, struct vscsi_ioc_data)
+#define VSCSI_DATA_READ _IOW('I', 1, struct vscsi_ioc_data)
#define VSCSI_DATA_WRITE _IOW('I', 2, struct vscsi_ioc_data)
struct vscsi_ioc_t2i {
@@ -65,7 +65,8 @@ struct vscsi_ioc_devevent {
int lun;
};
-#define VSCSI_REQPROBE _IOW('I', 4, struct vscsi_ioc_devevent)
-#define VSCSI_REQDETACH _IOW('I', 5, struct vscsi_ioc_devevent)
+#define VSCSI_REQPROBE _IOW('I', 4, struct vscsi_ioc_devevent)
+#define VSCSI_REQDETACH _IOW('I', 5, struct vscsi_ioc_devevent)
+#define VSCSI_DEVEVENT_POLL _IOR('I', 6, int)
#endif /* _SYS_DEV_VSCSIVAR_H */
Index: sys/dev/vscsi.c
===================================================================
RCS file: /cvs/src/sys/dev/vscsi.c,v
retrieving revision 1.58
diff -u -p -u -p -r1.58 vscsi.c
--- sys/dev/vscsi.c 25 Dec 2020 12:59:52 -0000 1.58
+++ sys/dev/vscsi.c 8 Jun 2021 23:08:04 -0000
@@ -73,6 +73,8 @@ struct vscsi_softc {
struct selinfo sc_sel;
struct mutex sc_sel_mtx;
+
+ struct taskq *sc_tq;
};
#define DEVNAME(_s) ((_s)->sc_dev.dv_xname)
@@ -104,6 +106,7 @@ int vscsi_t2i(struct vscsi_softc *, str
int vscsi_devevent(struct vscsi_softc *, u_long,
struct vscsi_ioc_devevent *);
void vscsi_devevent_task(void *);
+int vscsi_devevent_poll(struct vscsi_softc *, int *);
void vscsi_done(struct vscsi_softc *, struct vscsi_ccb *);
void * vscsi_ccb_get(void *);
@@ -156,6 +159,10 @@ vscsi_attach(struct device *parent, stru
sc->sc_scsibus = (struct scsibus_softc *)config_found(&sc->sc_dev,
&saa, scsiprint);
+
+ sc->sc_tq = taskq_create("vscsi", 1, IPL_BIO, 0);
+ if (sc->sc_tq == NULL)
+ panic("vscsi: Couldn't allocate task queue.");
}
void
@@ -317,7 +324,9 @@ vscsiioctl(dev_t dev, u_long cmd, caddr_
err = vscsi_devevent(sc, cmd,
(struct vscsi_ioc_devevent *)addr);
break;
-
+ case VSCSI_DEVEVENT_POLL:
+ err = vscsi_devevent_poll(sc, (int *) addr);
+ break;
default:
err = ENOTTY;
break;
@@ -487,7 +496,7 @@ vscsi_devevent(struct vscsi_softc *sc, u
dt->cmd = cmd;
device_ref(&sc->sc_dev);
- task_add(systq, &dt->t);
+ task_add(sc->sc_tq, &dt->t);
return (0);
}
@@ -525,6 +534,17 @@ gone:
device_unref(&sc->sc_dev);
free(dt, M_TEMP, sizeof(*dt));
+}
+
+int
+vscsi_devevent_poll(struct vscsi_softc *sc, int *tq_status)
+{
+ if (tq_status == NULL)
+ return (EINVAL);
+
+ *tq_status = taskq_empty(sc->sc_tq);
+
+ return (0);
}
int
Index: share/man/man4/vscsi.4
===================================================================
RCS file: /cvs/src/share/man/man4/vscsi.4,v
retrieving revision 1.14
diff -u -p -u -p -r1.14 vscsi.4
--- share/man/man4/vscsi.4 12 Sep 2017 15:06:20 -0000 1.14
+++ share/man/man4/vscsi.4 8 Jun 2021 23:08:31 -0000
@@ -108,6 +108,11 @@ struct vscsi_ioc_devevent {
u_int lun;
};
.Ed
+.Pp
+.It Dv VSCSI_DEVEVENT_POLL Fa "int"
+Returns a status flag indicating whether the device event queue (utilized by
+VSCSI_REQPROBE and VSCSI_REQDETACH operations) has outstanding events.
+.Ed
.El
.Sh FILES
.Bl -tag -width /dev/vscsi0
Index: usr.sbin/iscsid/iscsid.h
===================================================================
RCS file: /cvs/src/usr.sbin/iscsid/iscsid.h,v
retrieving revision 1.17
diff -u -p -u -p -r1.17 iscsid.h
--- usr.sbin/iscsid/iscsid.h 16 Apr 2021 14:37:06 -0000 1.17
+++ usr.sbin/iscsid/iscsid.h 8 Jun 2021 23:10:08 -0000
@@ -259,6 +259,7 @@ struct session_poll {
int conn_failed_count;
int conn_waiting_count;
int sess_conn_status;
+ int devev_queue_status;
};
struct connection {
@@ -399,6 +400,7 @@ void vscsi_dispatch(int, short, void *);
void vscsi_data(unsigned long, int, void *, size_t);
void vscsi_status(int, int, void *, size_t);
void vscsi_event(unsigned long, u_int, u_int);
+void vscsi_poll_devevent_queue(int *);
struct vscsi_stats *vscsi_stats(void);
/* Session polling */
Index: usr.sbin/iscsid/poll.c
===================================================================
RCS file: /cvs/src/usr.sbin/iscsid/poll.c,v
retrieving revision 1.1
diff -u -p -u -p -r1.1 poll.c
--- usr.sbin/iscsid/poll.c 16 Apr 2021 14:37:06 -0000 1.1
+++ usr.sbin/iscsid/poll.c 8 Jun 2021 23:10:08 -0000
@@ -82,13 +82,24 @@ poll_session(struct session_poll *p, str
}
/*
- * Set session_poll sess_conn_status based on the number of sessions
- * versus number of logged in or failed sessions. If the numbers are
- * equal iscsictl can stop polling since all sessions reached a stable state.
+ * Here, we finalize the polling by checking to see if
+ * the requested probe/detach ops have been completed
+ * by the vscsi driver. We only bother to do this once
+ * we know all the sessions have settled.
*/
void
poll_finalize(struct session_poll *p)
{
+
+ /*
+ * If we aren't awaiting anything else to connect, the combined total of
+ * logged in and failed connections should equal the number of
+ * configured sessions.
+ */
p->sess_conn_status = (p->session_count == p->conn_logged_in_count +
p->conn_failed_count);
+
+ /* All connections up, so check the dev event queue. */
+ if (p->sess_conn_status)
+ vscsi_poll_devevent_queue(&p->devev_queue_status);
}
Index: usr.sbin/iscsid/vscsi.c
===================================================================
RCS file: /cvs/src/usr.sbin/iscsid/vscsi.c,v
retrieving revision 1.17
diff -u -p -u -p -r1.17 vscsi.c
--- usr.sbin/iscsid/vscsi.c 16 Aug 2016 18:41:57 -0000 1.17
+++ usr.sbin/iscsid/vscsi.c 8 Jun 2021 23:10:08 -0000
@@ -338,3 +338,10 @@ vscsi_stats(void)
{
return &v.stats;
}
+
+void
+vscsi_poll_devevent_queue(int *arg)
+{
+ if (ioctl(v.fd, VSCSI_DEVEVENT_POLL, arg) != 0)
+ fatal("vscsi_devevent_poll");
+}
Index: usr.sbin/iscsictl/iscsictl.8
===================================================================
RCS file: /cvs/src/usr.sbin/iscsictl/iscsictl.8,v
retrieving revision 1.7
diff -u -p -u -p -r1.7 iscsictl.8
--- usr.sbin/iscsictl/iscsictl.8 5 May 2021 12:34:12 -0000 1.7
+++ usr.sbin/iscsictl/iscsictl.8 8 Jun 2021 23:10:23 -0000
@@ -47,8 +47,9 @@ to communicate with
The following commands are available:
.Bl -tag -width Ds
.It Cm reload
-Reload the configuration file and wait to return until iscsid reports all
-connections have completed (successfully or otherwise), or for up to 10
+Reload the configuration file and wait to return until iscsid reports
+all connections have completed (successfully or otherwise) and all
+scsi_probe and scsi_detach requests have completed, or for up to 10
seconds.
.It Cm show Op Cm summary
Show a list of all configured sessions.
Index: usr.sbin/iscsictl/iscsictl.c
===================================================================
RCS file: /cvs/src/usr.sbin/iscsictl/iscsictl.c,v
retrieving revision 1.12
diff -u -p -u -p -r1.12 iscsictl.c
--- usr.sbin/iscsictl/iscsictl.c 16 Apr 2021 14:39:33 -0000 1.12
+++ usr.sbin/iscsictl/iscsictl.c 8 Jun 2021 23:10:23 -0000
@@ -431,7 +431,7 @@ poll_and_wait(void)
poll_session_status();
/* Poll says we are good to go. */
- if (poll_result.sess_conn_status != 0) {
+ if (poll_result.sess_conn_status != 0 && poll_result.devev_queue_status != 0) {
printf("ok\n");
/* wait a bit longer so all is settled. */
sleep(POLL_DELAY_SEC);
