Hi Alexander,
Thank-you for getting back to me. I tried as you suggested going through
scsi instead. While that does work being able to mount /dev/da*
devices, an issue I have run into is that suspend/resume features do not
seem to work with virtio-scsi.
If I set it back to virtio-blk through original /dev/zvol/zroot/* devices
or through the scsi /dev/da* devices then suspend/resume works.
Let me elaborate, here is what I put together today based on your
suggestion:
router:/root # cat /etc/ctl.conf
portal-group pg0 {
discovery-auth-group no-authentication
listen 127.0.0.1:3260
}
target iqn.2005-02.com.sunsaturn:target0 {
auth-group no-authentication
portal-group pg0
#bhyve virti-iscsi disk - /dev/cam/ctl1.0
port ioctl/1
lun 0 {
path /dev/zvol/zroot/asterisk
serial 000c2937247001
device-id "iSCSI Disk 000c2937247001"
option vendor "FreeBSD"
option product "iSCSI Disk"
option revision "0123"
option insecure_tpc on
}
}
target iqn.2005-02.com.sunsaturn:target1 {
auth-group no-authentication
portal-group pg0
#bhyve virti-iscsi disk - /dev/cam/ctl2.0
port ioctl/2
lun 0 {
path /dev/zvol/zroot/asterisk2
serial 000c2937247002
device-id "iSCSI Disk 000c2937247002"
option vendor "FreeBSD"
option product "iSCSI Disk"
option revision "0123"
option insecure_tpc on
}
lun 1 {
path
/vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso
#byhve seems to just hang when I set it to an actual CDROM
so let it default to type 0
#device-type 5
serial 000c2937247003
device-id "iSCSI CDROM ISO 000c2937247003"
option vendor "FreeBSD CDROM"
option product "iSCSI CDROM"
option revision "0123"
option insecure_tpc on
}
}
router:/root # cat /etc/iscsi.conf
t0 {
TargetAddress = 127.0.0.1:3260
TargetName = iqn.2005-02.com.sunsaturn:target0
}
t1 {
TargetAddress = 127.0.0.1:3260
TargetName = iqn.2005-02.com.sunsaturn:target1
}
router:/root # tail -13 /etc/rc.conf
#ISCSI - service ctld start && service iscsid start
#server
ctld_enable="YES" #load /etc/ctl.conf
iscsid_enable="YES" #start iscsid process to connect to ctld
#client - service iscsictl start
iscsictl_enable="YES" #connect to all targets in /etc/iscsi.conf
iscsictl_flags="-Aa"
#to kill all sessions:
#iscsictl -Ra
#to restart everything:
#iscsictl -Ra
#service ctld stop && service iscsid stop
#service ctld start && service iscsid start && service iscsictl start
router:/root #
Now anytime I use:
-s 4:0,virtio-scsi,/dev/cam/ctl1.0
I get coredumps as follows when resuming the guest:
./asterisk.sh: line 101: 7621 Segmentation fault (core
dumped) bhyve -c $CPU -m $RAM -w -H -A -s 0:0,hostbridge -s
4:0,virtio-scsi,$STORAGE -s 5:0,virtio-net,$TAP -s
29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT -s 30,xhci,tablet -s 31,lpc -l
com1,$SERIAL -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -r
$DIR/default.ckp $VMNAME
The exit code was : 139
However if I used:
-s 4:0,virtio-blk,/dev/da1
or even:
-s 4:0,virtio-blk,/dev/zvol/zroot/asterisk
suspend/resume works properly. Is virtio-scsi supported through the ioctl
devices?
I'll include my test suspend/resume script if you want to have a look:
#####asterisk.sh
#!/bin/bash
#
# General script to test bhyve suspend/resume features
#
# Requirements: FreeBSD current
# git clone https://git.FreeBSD.org/src.git /usr/src
# cd /usr/src/sys/amd64/conf (edit MYKERNEL) cp GENERIC MYKERNEL; (add:
options BHYVE_SNAPSHOT)
# cd /usr/src
# (find amount of CPUs and adjust -j below - "dmesg|grep SMP")
# make -j12 buildworld -DWITH_BHYVE_SNAPSHOT -DWITH_MALLOC_PRODUCTION
# make -j12 buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL
# shutdown -r now
# cd /usr/src; make installworld
# shutdown -r now
# etcupdate -B
# pkg bootstrap -f #if new freebsd version
# pkg upgrade -f #if new freebsd version
#
# Report anomolies to d...@sunsaturn.com
##############EDIT ME#####################
HOST="127.0.0.1" # vncviewer 127.0.0.1:5900 - pkg
install tightvnc
PORT="5900"
WIDTH="800"
HEIGHT="600"
VMNAME="asterisk"
ISO="/vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso"
DIR="/vm/asterisk" # Used to hold files when guest
suspended
SERIAL="/dev/nmdm_asteriskA" # For "screen /dev/nmdm_asteriskB"
- pkg install screen
TAP="tap0"
CPU="8"
RAM="8G"
#For testing virtio-scsi
STORAGE="/dev/cam/ctl1.0" # port from /etc/ctl.conf(port
ioctl/1) - core dumping on resume
DEVICE="virtio-scsi"
#for testing virtio-blk # Comment out above 2 lines if
using these
#DEVICE="virtio-blk"
#STORAGE="/dev/zvol/zroot/asterisk" # Standard zvol
#STORAGE="/dev/da1" # Block device created from
iscsictl
#########################################
usage() {
echo "Usage: $1 start (Start the guest: $VMNAME)";
echo "Usage: $1 stop (Stop the guest: $VMNAME)";
echo "Usage: $1 resume (Resume the guest from last suspend:
$VMNAME)";
echo "Usage: $1 suspend (Suspend the guest: $VMNAME)";
echo "Usage: $1 install (Install new guest: $VMNAME)";
exit
}
if [ ! -d "$DIR" ]; then
mkdir -p $DIR
fi
#if [ -z "$2" ]; then
# usage
#else
# VMNAME=$2
#fi
if [ "$1" == "install" ]; then
#Kill it before starting it
echo "Execute: screen $SERIAL"
bhyvectl --destroy --vm=$VMNAME
bhyve -c $CPU -m $RAM -w -H -A \
-s 0:0,hostbridge \
-s 3:0,ahci-cd,$ISO \
-s 4:0,virtio-scsi,$STORAGE \
-s 5:0,virtio-net,$TAP \
-s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
$VMNAME
#kill it after
bhyvectl --destroy --vm=$VMNAME
elif [ "$1" == "start" ]; then
while true
do
echo "Starting $VMNAME -s
29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
#Kill it before starting it
bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
if [ -f "$DIR/default.ckp" ]; then
bhyve -c $CPU -m $RAM -w -H -A \
-s 0:0,hostbridge \
-s 4:0,virtio-scsi,$STORAGE \
-s 5:0,virtio-net,$TAP \
-s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
-s 30,xhci,tablet \
-s 31,lpc -l com1,$SERIAL \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
$VMNAME
fi
#DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
#if [ "$?" != 0 ];
#then
# echo "The exit code was not reboot code 0!: $?"
# exit
#fi
echo "The exit code was : $?"
exit
done
elif [ "$1" == "resume" ]; then
while true
do
echo "Starting $VMNAME -s
29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
#Kill it before starting it
bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
if [ -f "$DIR/default.ckp" ]; then
bhyve -c $CPU -m $RAM -w -H -A \
-s 0:0,hostbridge \
-s 4:0,virtio-scsi,$STORAGE \
-s 5:0,virtio-net,$TAP \
-s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
-s 30,xhci,tablet \
-s 31,lpc -l com1,$SERIAL \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-r $DIR/default.ckp \
$VMNAME
fi
#DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
#if [ "$?" != 0 ];
#then
# echo "The exit code was not reboot code 0!: $?"
# exit
#fi
echo "The exit code was : $?"
exit
done
elif [ "$1" == "suspend" ];
then
bhyvectl --suspend $DIR/default.ckp --vm=$VMNAME
elif [ "$1" == "stop" ]; then
bhyvectl --destroy --vm=$VMNAME
else
usage
fi
################
Dan.
--
Dan The Man
CEO & Founder
Websites, Domains and Everything else
http://www.SunSaturn.com/aboutus.php
Email: d...@sunsaturn.com
PGP Key: https://SunSaturn.com/pgp.txt
A1A7 6E84 FB0B 8994 C3B5 A1BA FF6F 4997 7311 C386
On Mon, 7 Nov 2022, Alexander Motin wrote:
On 07.11.2022 18:53, Dan The Man wrote:
router:~ # sysctl vfs.zfs.vol.recursive=1
vfs.zfs.vol.recursive: 0 -> 1
router:~ # zpool import
pool: testing
id: 8013833172609421701
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
testing ONLINE
zvol/zroot/asterisk2p3 ONLINE
router:~ # zpool import -fR /mnt testing
This hangs forever....
The only way to import that pool from the zvol that I know of.....
Mounting ZFS from ZVOLs is blocked for a reason. It causes deadlocks due to
lock recursion. I don't know what you are trying to achieve, but as
alternatives, the ZVOL can be passed inside VM, it can be shared via iSCSI
(even inside the host itself), etc.
--
Alexander Motin
