[PATCH v3 00/11] virtio: S4 support
Hi, These patches add support for S4 to virtio (pci) and all drivers. For each driver, all vqs are removed before hibernation, and then re-created after restore. Some driver-specific uninit and init work is also done in the freeze and restore functions. All the drivers in testing work fine: * virtio-blk is used for the only disk in the VM, IO works fine before and after. * virtio-console: port IO keeps working fine before and after. * If a port is waiting for data from the host (blocking read(2) call), this works fine in both the cases: host-side connection is available or unavailable after resume. In case the host-side connection isn't available, the blocking call is terminated. If it is available, the call continues to remain in blocked state till further data arrives. * virtio-net: ping remains active across S4. * virtio-balloon: Works fine before and after. Forgets the ballooned value across S4. If it's desirable to maintain the ballooned value, a new config option can be created to do this. All of these tests are run in parallel. I have some more tests lined up on similar lines above. I'll reply here if something breaks. Please review and apply if appropriate, v3: - Reset vqs before deleting them (Sasha Levin) - Flush block queue before freeze (Rusty) - Detach netdev before freeze (Michael S. Tsirkin) v2: - fix checkpatch errors/warnings Amit Shah (11): virtio: pci: switch to new PM API virtio: pci: add PM notification handlers for restore, freeze, thaw, poweroff virtio: console: Move out vq and vq buf removal into separate functions virtio: console: Add freeze and restore handlers to support S4 virtio: blk: Move out vq initialization to separate function virtio: blk: Add freeze, restore handlers to support S4 virtio: net: Move out vq initialization into separate function virtio: net: Move out vq and vq buf removal into separate function virtio: net: Add freeze, restore handlers to support S4 virtio: balloon: Move out vq initialization into separate function virtio: balloon: Add freeze, restore handlers to support S4 drivers/block/virtio_blk.c | 55 +++-- drivers/char/virtio_console.c | 126 ++- drivers/net/virtio_net.c| 99 ++ drivers/virtio/virtio_balloon.c | 68 +++-- drivers/virtio/virtio_pci.c | 66 +++- include/linux/virtio.h |4 + 6 files changed, 336 insertions(+), 82 deletions(-) -- 1.7.7.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 01/11] virtio: pci: switch to new PM API
The older PM API doesn't have a way to get notifications on hibernate
events. Switch to the newer one that gives us those notifications.
Signed-off-by: Amit Shah
---
drivers/virtio/virtio_pci.c | 16
1 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
index 3d1bf41..5d36bc0 100644
--- a/drivers/virtio/virtio_pci.c
+++ b/drivers/virtio/virtio_pci.c
@@ -690,19 +690,28 @@ static void __devexit virtio_pci_remove(struct pci_dev
*pci_dev)
}
#ifdef CONFIG_PM
-static int virtio_pci_suspend(struct pci_dev *pci_dev, pm_message_t state)
+static int virtio_pci_suspend(struct device *dev)
{
+ struct pci_dev *pci_dev = to_pci_dev(dev);
+
pci_save_state(pci_dev);
pci_set_power_state(pci_dev, PCI_D3hot);
return 0;
}
-static int virtio_pci_resume(struct pci_dev *pci_dev)
+static int virtio_pci_resume(struct device *dev)
{
+ struct pci_dev *pci_dev = to_pci_dev(dev);
+
pci_restore_state(pci_dev);
pci_set_power_state(pci_dev, PCI_D0);
return 0;
}
+
+static const struct dev_pm_ops virtio_pci_pm_ops = {
+ .suspend = virtio_pci_suspend,
+ .resume = virtio_pci_resume,
+};
#endif
static struct pci_driver virtio_pci_driver = {
@@ -711,8 +720,7 @@ static struct pci_driver virtio_pci_driver = {
.probe = virtio_pci_probe,
.remove = __devexit_p(virtio_pci_remove),
#ifdef CONFIG_PM
- .suspend= virtio_pci_suspend,
- .resume = virtio_pci_resume,
+ .driver.pm = &virtio_pci_pm_ops,
#endif
};
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 02/11] virtio: pci: add PM notification handlers for restore, freeze, thaw, poweroff
Handle restore and freeze notification from the PM core. Expose these
to individual virtio drivers that can quiesce and resume vq operations.
These functions also save device-specific data so that the device can be
put in pre-suspend state after resume, and disable and enable the PCI
device in the freeze and resume functions, respectively.
Signed-off-by: Amit Shah
---
drivers/virtio/virtio_pci.c | 50 +++
include/linux/virtio.h |4 +++
2 files changed, 54 insertions(+), 0 deletions(-)
diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
index 5d36bc0..0db6f98 100644
--- a/drivers/virtio/virtio_pci.c
+++ b/drivers/virtio/virtio_pci.c
@@ -55,6 +55,10 @@ struct virtio_pci_device
unsigned msix_vectors;
/* Vectors allocated, excluding per-vq vectors if any */
unsigned msix_used_vectors;
+
+ /* Status saved during hibernate/restore */
+ u8 saved_status;
+
/* Whether we have vector per vq */
bool per_vq_vectors;
};
@@ -708,9 +712,55 @@ static int virtio_pci_resume(struct device *dev)
return 0;
}
+static int virtio_pci_freeze(struct device *dev)
+{
+ struct pci_dev *pci_dev = to_pci_dev(dev);
+ struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev);
+ struct virtio_driver *drv;
+ int ret;
+
+ drv = container_of(vp_dev->vdev.dev.driver,
+ struct virtio_driver, driver);
+
+ ret = 0;
+ vp_dev->saved_status = vp_get_status(&vp_dev->vdev);
+ if (drv && drv->freeze)
+ ret = drv->freeze(&vp_dev->vdev);
+
+ if (!ret)
+ pci_disable_device(pci_dev);
+ return ret;
+}
+
+static int virtio_pci_restore(struct device *dev)
+{
+ struct pci_dev *pci_dev = to_pci_dev(dev);
+ struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev);
+ struct virtio_driver *drv;
+ int ret;
+
+ drv = container_of(vp_dev->vdev.dev.driver,
+ struct virtio_driver, driver);
+
+ ret = pci_enable_device(pci_dev);
+ if (ret)
+ return ret;
+ pci_set_master(pci_dev);
+ vp_set_status(&vp_dev->vdev, vp_dev->saved_status);
+ vp_finalize_features(&vp_dev->vdev);
+ if (drv && drv->restore)
+ ret = drv->restore(&vp_dev->vdev);
+
+ return ret;
+}
+
static const struct dev_pm_ops virtio_pci_pm_ops = {
.suspend = virtio_pci_suspend,
.resume = virtio_pci_resume,
+ .freeze = virtio_pci_freeze,
+ .thaw= virtio_pci_restore,
+ .restore = virtio_pci_restore,
+ .poweroff = virtio_pci_suspend,
};
#endif
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index 4c069d8..1c26416 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -146,6 +146,10 @@ struct virtio_driver {
int (*probe)(struct virtio_device *dev);
void (*remove)(struct virtio_device *dev);
void (*config_changed)(struct virtio_device *dev);
+#ifdef CONFIG_PM
+ int (*freeze)(struct virtio_device *dev);
+ int (*restore)(struct virtio_device *dev);
+#endif
};
int register_virtio_driver(struct virtio_driver *drv);
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 03/11] virtio: console: Move out vq and vq buf removal into separate functions
This common code will be shared with the PM freeze function.
Signed-off-by: Amit Shah
---
drivers/char/virtio_console.c | 68 -
1 files changed, 40 insertions(+), 28 deletions(-)
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 8e3c46d..e14f5aa 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1271,6 +1271,20 @@ static void remove_port(struct kref *kref)
kfree(port);
}
+static void remove_port_data(struct port *port)
+{
+ struct port_buffer *buf;
+
+ /* Remove unused data this port might have received. */
+ discard_port_data(port);
+
+ reclaim_consumed_buffers(port);
+
+ /* Remove buffers we queued up for the Host to send us data in. */
+ while ((buf = virtqueue_detach_unused_buf(port->in_vq)))
+ free_buf(buf);
+}
+
/*
* Port got unplugged. Remove port from portdev's list and drop the
* kref reference. If no userspace has this port opened, it will
@@ -1278,8 +1292,6 @@ static void remove_port(struct kref *kref)
*/
static void unplug_port(struct port *port)
{
- struct port_buffer *buf;
-
spin_lock_irq(&port->portdev->ports_lock);
list_del(&port->list);
spin_unlock_irq(&port->portdev->ports_lock);
@@ -1300,14 +1312,7 @@ static void unplug_port(struct port *port)
hvc_remove(port->cons.hvc);
}
- /* Remove unused data this port might have received. */
- discard_port_data(port);
-
- reclaim_consumed_buffers(port);
-
- /* Remove buffers we queued up for the Host to send us data in. */
- while ((buf = virtqueue_detach_unused_buf(port->in_vq)))
- free_buf(buf);
+ remove_port_data(port);
/*
* We should just assume the device itself has gone off --
@@ -1659,6 +1664,28 @@ static const struct file_operations portdev_fops = {
.owner = THIS_MODULE,
};
+static void remove_vqs(struct ports_device *portdev)
+{
+ portdev->vdev->config->del_vqs(portdev->vdev);
+ kfree(portdev->in_vqs);
+ kfree(portdev->out_vqs);
+}
+
+static void remove_controlq_data(struct ports_device *portdev)
+{
+ struct port_buffer *buf;
+ unsigned int len;
+
+ if (!use_multiport(portdev))
+ return;
+
+ while ((buf = virtqueue_get_buf(portdev->c_ivq, &len)))
+ free_buf(buf);
+
+ while ((buf = virtqueue_detach_unused_buf(portdev->c_ivq)))
+ free_buf(buf);
+}
+
/*
* Once we're further in boot, we get probed like any other virtio
* device.
@@ -1764,9 +1791,7 @@ free_vqs:
/* The host might want to notify mgmt sw about device add failure */
__send_control_msg(portdev, VIRTIO_CONSOLE_BAD_ID,
VIRTIO_CONSOLE_DEVICE_READY, 0);
- vdev->config->del_vqs(vdev);
- kfree(portdev->in_vqs);
- kfree(portdev->out_vqs);
+ remove_vqs(portdev);
free_chrdev:
unregister_chrdev(portdev->chr_major, "virtio-portsdev");
free:
@@ -1804,21 +1829,8 @@ static void virtcons_remove(struct virtio_device *vdev)
* have to just stop using the port, as the vqs are going
* away.
*/
- if (use_multiport(portdev)) {
- struct port_buffer *buf;
- unsigned int len;
-
- while ((buf = virtqueue_get_buf(portdev->c_ivq, &len)))
- free_buf(buf);
-
- while ((buf = virtqueue_detach_unused_buf(portdev->c_ivq)))
- free_buf(buf);
- }
-
- vdev->config->del_vqs(vdev);
- kfree(portdev->in_vqs);
- kfree(portdev->out_vqs);
-
+ remove_controlq_data(portdev);
+ remove_vqs(portdev);
kfree(portdev);
}
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 04/11] virtio: console: Add freeze and restore handlers to support S4
Remove all vqs and associated buffers in the freeze callback which
prepares us to go into hibernation state. On restore, re-create all the
vqs and populate the input vqs with buffers to get to the pre-hibernate
state.
Note: Any outstanding unconsumed buffers are discarded; which means
there's a possibility of data loss in case the host or the guest didn't
consume any data already present in the vqs. This can be addressed in a
later patch series, perhaps in virtio common code.
Signed-off-by: Amit Shah
---
drivers/char/virtio_console.c | 58 +
1 files changed, 58 insertions(+), 0 deletions(-)
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index e14f5aa..fd2fd6f 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1844,6 +1844,60 @@ static unsigned int features[] = {
VIRTIO_CONSOLE_F_MULTIPORT,
};
+#ifdef CONFIG_PM
+static int virtcons_freeze(struct virtio_device *vdev)
+{
+ struct ports_device *portdev;
+ struct port *port;
+
+ portdev = vdev->priv;
+
+ vdev->config->reset(vdev);
+
+ cancel_work_sync(&portdev->control_work);
+ remove_controlq_data(portdev);
+
+ list_for_each_entry(port, &portdev->ports, list) {
+ /*
+* We'll ask the host later if the new invocation has
+* the port opened or closed.
+*/
+ port->host_connected = false;
+ remove_port_data(port);
+ }
+ remove_vqs(portdev);
+
+ return 0;
+}
+
+static int virtcons_restore(struct virtio_device *vdev)
+{
+ struct ports_device *portdev;
+ struct port *port;
+ int ret;
+
+ portdev = vdev->priv;
+
+ ret = init_vqs(portdev);
+ if (ret)
+ return ret;
+
+ if (use_multiport(portdev))
+ fill_queue(portdev->c_ivq, &portdev->cvq_lock);
+
+ list_for_each_entry(port, &portdev->ports, list) {
+ port->in_vq = portdev->in_vqs[port->id];
+ port->out_vq = portdev->out_vqs[port->id];
+
+ fill_queue(port->in_vq, &port->inbuf_lock);
+
+ /* Get port open/close status on the host */
+ send_control_msg(port, VIRTIO_CONSOLE_PORT_READY, 1);
+ }
+ return 0;
+}
+#endif
+
static struct virtio_driver virtio_console = {
.feature_table = features,
.feature_table_size = ARRAY_SIZE(features),
@@ -1853,6 +1907,10 @@ static struct virtio_driver virtio_console = {
.probe =virtcons_probe,
.remove = virtcons_remove,
.config_changed = config_intr,
+#ifdef CONFIG_PM
+ .freeze = virtcons_freeze,
+ .restore = virtcons_restore,
+#endif
};
static int __init init(void)
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 05/11] virtio: blk: Move out vq initialization to separate function
The probe and PM restore functions will share this code.
Signed-off-by: Amit Shah
---
drivers/block/virtio_blk.c | 19 ++-
1 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 4d0b70a..467f218 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -349,6 +349,18 @@ static void virtblk_config_changed(struct virtio_device
*vdev)
queue_work(virtblk_wq, &vblk->config_work);
}
+static int init_vq(struct virtio_blk *vblk)
+{
+ int err = 0;
+
+ /* We expect one virtqueue, for output. */
+ vblk->vq = virtio_find_single_vq(vblk->vdev, blk_done, "requests");
+ if (IS_ERR(vblk->vq))
+ err = PTR_ERR(vblk->vq);
+
+ return err;
+}
+
static int __devinit virtblk_probe(struct virtio_device *vdev)
{
struct virtio_blk *vblk;
@@ -390,12 +402,9 @@ static int __devinit virtblk_probe(struct virtio_device
*vdev)
sg_init_table(vblk->sg, vblk->sg_elems);
INIT_WORK(&vblk->config_work, virtblk_config_changed_work);
- /* We expect one virtqueue, for output. */
- vblk->vq = virtio_find_single_vq(vdev, blk_done, "requests");
- if (IS_ERR(vblk->vq)) {
- err = PTR_ERR(vblk->vq);
+ err = init_vq(vblk);
+ if (err)
goto out_free_vblk;
- }
vblk->pool = mempool_create_kmalloc_pool(1,sizeof(struct virtblk_req));
if (!vblk->pool) {
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 06/11] virtio: blk: Add freeze, restore handlers to support S4
Delete the vq and flush any pending requests from the block queue on the
freeze callback to prepare for hibernation.
Re-create the vq in the restore callback to resume normal function.
Signed-off-by: Amit Shah
---
drivers/block/virtio_blk.c | 36
1 files changed, 36 insertions(+), 0 deletions(-)
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 467f218..f73fb2d 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -568,6 +568,38 @@ static void __devexit virtblk_remove(struct virtio_device
*vdev)
ida_simple_remove(&vd_index_ida, index);
}
+#ifdef CONFIG_PM
+static int virtblk_freeze(struct virtio_device *vdev)
+{
+ struct virtio_blk *vblk = vdev->priv;
+
+ flush_work(&vblk->config_work);
+ spin_lock_irq(vblk->disk->queue->queue_lock);
+ blk_stop_queue(vblk->disk->queue);
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+ blk_sync_queue(vblk->disk->queue);
+
+ /* Stop all the virtqueues. */
+ vdev->config->reset(vdev);
+ vdev->config->del_vqs(vdev);
+ return 0;
+}
+
+static int virtblk_restore(struct virtio_device *vdev)
+{
+ struct virtio_blk *vblk = vdev->priv;
+ int ret;
+
+ ret = init_vq(vdev->priv);
+ if (!ret) {
+ spin_lock_irq(vblk->disk->queue->queue_lock);
+ blk_start_queue(vblk->disk->queue);
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+ }
+ return ret;
+}
+#endif
+
static const struct virtio_device_id id_table[] = {
{ VIRTIO_ID_BLOCK, VIRTIO_DEV_ANY_ID },
{ 0 },
@@ -593,6 +625,10 @@ static struct virtio_driver __refdata virtio_blk = {
.probe = virtblk_probe,
.remove = __devexit_p(virtblk_remove),
.config_changed = virtblk_config_changed,
+#ifdef CONFIG_PM
+ .freeze = virtblk_freeze,
+ .restore= virtblk_restore,
+#endif
};
static int __init init(void)
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 07/11] virtio: net: Move out vq initialization into separate function
The probe and PM restore functions will share this code.
Signed-off-by: Amit Shah
---
drivers/net/virtio_net.c | 47 +++--
1 files changed, 28 insertions(+), 19 deletions(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 6ee8410..6baa563 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -954,15 +954,38 @@ static void virtnet_config_changed(struct virtio_device
*vdev)
virtnet_update_status(vi);
}
+static int init_vqs(struct virtnet_info *vi)
+{
+ struct virtqueue *vqs[3];
+ vq_callback_t *callbacks[] = { skb_recv_done, skb_xmit_done, NULL};
+ const char *names[] = { "input", "output", "control" };
+ int nvqs, err;
+
+ /* We expect two virtqueues, receive then send,
+* and optionally control. */
+ nvqs = virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ) ? 3 : 2;
+
+ err = vi->vdev->config->find_vqs(vi->vdev, nvqs, vqs, callbacks, names);
+ if (err)
+ return err;
+
+ vi->rvq = vqs[0];
+ vi->svq = vqs[1];
+
+ if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ)) {
+ vi->cvq = vqs[2];
+
+ if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VLAN))
+ vi->dev->features |= NETIF_F_HW_VLAN_FILTER;
+ }
+ return 0;
+}
+
static int virtnet_probe(struct virtio_device *vdev)
{
int err;
struct net_device *dev;
struct virtnet_info *vi;
- struct virtqueue *vqs[3];
- vq_callback_t *callbacks[] = { skb_recv_done, skb_xmit_done, NULL};
- const char *names[] = { "input", "output", "control" };
- int nvqs;
/* Allocate ourselves a network device with room for our info */
dev = alloc_etherdev(sizeof(struct virtnet_info));
@@ -1034,24 +1057,10 @@ static int virtnet_probe(struct virtio_device *vdev)
if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs = true;
- /* We expect two virtqueues, receive then send,
-* and optionally control. */
- nvqs = virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ) ? 3 : 2;
-
- err = vdev->config->find_vqs(vdev, nvqs, vqs, callbacks, names);
+ err = init_vqs(vi);
if (err)
goto free_stats;
- vi->rvq = vqs[0];
- vi->svq = vqs[1];
-
- if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ)) {
- vi->cvq = vqs[2];
-
- if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VLAN))
- dev->features |= NETIF_F_HW_VLAN_FILTER;
- }
-
err = register_netdev(dev);
if (err) {
pr_debug("virtio_net: registering device failed\n");
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 08/11] virtio: net: Move out vq and vq buf removal into separate function
The remove and PM freeze functions will share this code.
Signed-off-by: Amit Shah
---
drivers/net/virtio_net.c | 23 ++-
1 files changed, 14 insertions(+), 9 deletions(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 6baa563..fbff37a 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1123,24 +1123,29 @@ static void free_unused_bufs(struct virtnet_info *vi)
BUG_ON(vi->num != 0);
}
-static void __devexit virtnet_remove(struct virtio_device *vdev)
+static void remove_vq_common(struct virtnet_info *vi)
{
- struct virtnet_info *vi = vdev->priv;
-
- /* Stop all the virtqueues. */
- vdev->config->reset(vdev);
-
-
- unregister_netdev(vi->dev);
cancel_delayed_work_sync(&vi->refill);
/* Free unused buffers in both send and recv, if any. */
free_unused_bufs(vi);
- vdev->config->del_vqs(vi->vdev);
+ vi->vdev->config->del_vqs(vi->vdev);
while (vi->pages)
__free_pages(get_a_page(vi, GFP_KERNEL), 0);
+}
+
+static void __devexit virtnet_remove(struct virtio_device *vdev)
+{
+ struct virtnet_info *vi = vdev->priv;
+
+ /* Stop all the virtqueues. */
+ vdev->config->reset(vdev);
+
+ unregister_netdev(vi->dev);
+
+ remove_vq_common(vi);
free_percpu(vi->stats);
free_netdev(vi->dev);
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 09/11] virtio: net: Add freeze, restore handlers to support S4
Remove all the vqs and detach from the netdev on hibernation.
Re-create vqs after restoring from a hibernated image and re-attach the
netdev. This keeps networking working across hibernation.
Signed-off-by: Amit Shah
---
drivers/net/virtio_net.c | 37 ++---
1 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index fbff37a..167b555 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1127,6 +1127,9 @@ static void remove_vq_common(struct virtnet_info *vi)
{
cancel_delayed_work_sync(&vi->refill);
+ /* Stop all the virtqueues. */
+ vi->vdev->config->reset(vi->vdev);
+
/* Free unused buffers in both send and recv, if any. */
free_unused_bufs(vi);
@@ -1140,9 +1143,6 @@ static void __devexit virtnet_remove(struct virtio_device
*vdev)
{
struct virtnet_info *vi = vdev->priv;
- /* Stop all the virtqueues. */
- vdev->config->reset(vdev);
-
unregister_netdev(vi->dev);
remove_vq_common(vi);
@@ -1151,6 +1151,33 @@ static void __devexit virtnet_remove(struct
virtio_device *vdev)
free_netdev(vi->dev);
}
+#ifdef CONFIG_PM
+static int virtnet_freeze(struct virtio_device *vdev)
+{
+ struct virtnet_info *vi = vdev->priv;
+
+ netif_device_detach(vi->dev);
+ remove_vq_common(vi);
+
+ return 0;
+}
+
+static int virtnet_restore(struct virtio_device *vdev)
+{
+ struct virtnet_info *vi = vdev->priv;
+ int err;
+
+ err = init_vqs(vi);
+ if (err)
+ return err;
+
+ try_fill_recv(vi, GFP_KERNEL);
+
+ netif_device_attach(vi->dev);
+ return 0;
+}
+#endif
+
static struct virtio_device_id id_table[] = {
{ VIRTIO_ID_NET, VIRTIO_DEV_ANY_ID },
{ 0 },
@@ -1175,6 +1202,10 @@ static struct virtio_driver virtio_net_driver = {
.probe =virtnet_probe,
.remove = __devexit_p(virtnet_remove),
.config_changed = virtnet_config_changed,
+#ifdef CONFIG_PM
+ .freeze = virtnet_freeze,
+ .restore = virtnet_restore,
+#endif
};
static int __init init(void)
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 10/11] virtio: balloon: Move out vq initialization into separate function
The probe and PM restore functions will share this code.
Signed-off-by: Amit Shah
---
drivers/virtio/virtio_balloon.c | 48 --
1 files changed, 30 insertions(+), 18 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 94fd738..1ff3cf4 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -275,32 +275,21 @@ static int balloon(void *_vballoon)
return 0;
}
-static int virtballoon_probe(struct virtio_device *vdev)
+static int init_vqs(struct virtio_balloon *vb)
{
- struct virtio_balloon *vb;
struct virtqueue *vqs[3];
vq_callback_t *callbacks[] = { balloon_ack, balloon_ack, stats_request
};
const char *names[] = { "inflate", "deflate", "stats" };
int err, nvqs;
- vdev->priv = vb = kmalloc(sizeof(*vb), GFP_KERNEL);
- if (!vb) {
- err = -ENOMEM;
- goto out;
- }
-
- INIT_LIST_HEAD(&vb->pages);
- vb->num_pages = 0;
- init_waitqueue_head(&vb->config_change);
- vb->vdev = vdev;
- vb->need_stats_update = 0;
-
- /* We expect two virtqueues: inflate and deflate,
-* and optionally stat. */
+ /*
+* We expect two virtqueues: inflate and deflate, and
+* optionally stat.
+*/
nvqs = virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ) ? 3 : 2;
- err = vdev->config->find_vqs(vdev, nvqs, vqs, callbacks, names);
+ err = vb->vdev->config->find_vqs(vb->vdev, nvqs, vqs, callbacks, names);
if (err)
- goto out_free_vb;
+ return err;
vb->inflate_vq = vqs[0];
vb->deflate_vq = vqs[1];
@@ -317,6 +306,29 @@ static int virtballoon_probe(struct virtio_device *vdev)
BUG();
virtqueue_kick(vb->stats_vq);
}
+ return 0;
+}
+
+static int virtballoon_probe(struct virtio_device *vdev)
+{
+ struct virtio_balloon *vb;
+ int err;
+
+ vdev->priv = vb = kmalloc(sizeof(*vb), GFP_KERNEL);
+ if (!vb) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ INIT_LIST_HEAD(&vb->pages);
+ vb->num_pages = 0;
+ init_waitqueue_head(&vb->config_change);
+ vb->vdev = vdev;
+ vb->need_stats_update = 0;
+
+ err = init_vqs(vb);
+ if (err)
+ goto out_free_vb;
vb->thread = kthread_run(balloon, vb, "vballoon");
if (IS_ERR(vb->thread)) {
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v3 11/11] virtio: balloon: Add freeze, restore handlers to support S4
Delete the vqs on the freeze callback to prepare for hibernation.
Re-create the vqs in the restore callback to resume normal function.
Signed-off-by: Amit Shah
---
drivers/virtio/virtio_balloon.c | 20
1 files changed, 20 insertions(+), 0 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 1ff3cf4..90149d1 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -363,6 +363,22 @@ static void __devexit virtballoon_remove(struct
virtio_device *vdev)
kfree(vb);
}
+#ifdef CONFIG_PM
+static int virtballoon_freeze(struct virtio_device *vdev)
+{
+ /* Now we reset the device so we can clean up the queues. */
+ vdev->config->reset(vdev);
+
+ vdev->config->del_vqs(vdev);
+ return 0;
+}
+
+static int virtballoon_restore(struct virtio_device *vdev)
+{
+ return init_vqs(vdev->priv);
+}
+#endif
+
static unsigned int features[] = {
VIRTIO_BALLOON_F_MUST_TELL_HOST,
VIRTIO_BALLOON_F_STATS_VQ,
@@ -377,6 +393,10 @@ static struct virtio_driver virtio_balloon_driver = {
.probe =virtballoon_probe,
.remove = __devexit_p(virtballoon_remove),
.config_changed = virtballoon_changed,
+#ifdef CONFIG_PM
+ .freeze = virtballoon_freeze,
+ .restore = virtballoon_restore,
+#endif
};
static int __init init(void)
--
1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 09/11] virtio: net: Add freeze, restore handlers to support S4
On Thu, Nov 17, 2011 at 05:27:40PM +0530, Amit Shah wrote:
> Remove all the vqs and detach from the netdev on hibernation.
>
> Re-create vqs after restoring from a hibernated image and re-attach the
> netdev. This keeps networking working across hibernation.
>
> Signed-off-by: Amit Shah
> ---
> drivers/net/virtio_net.c | 37 ++---
> 1 files changed, 34 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index fbff37a..167b555 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1127,6 +1127,9 @@ static void remove_vq_common(struct virtnet_info *vi)
> {
> cancel_delayed_work_sync(&vi->refill);
>
> + /* Stop all the virtqueues. */
> + vi->vdev->config->reset(vi->vdev);
> +
> /* Free unused buffers in both send and recv, if any. */
> free_unused_bufs(vi);
>
> @@ -1140,9 +1143,6 @@ static void __devexit virtnet_remove(struct
> virtio_device *vdev)
> {
> struct virtnet_info *vi = vdev->priv;
>
> - /* Stop all the virtqueues. */
> - vdev->config->reset(vdev);
> -
> unregister_netdev(vi->dev);
>
> remove_vq_common(vi);
> @@ -1151,6 +1151,33 @@ static void __devexit virtnet_remove(struct
> virtio_device *vdev)
> free_netdev(vi->dev);
> }
>
> +#ifdef CONFIG_PM
> +static int virtnet_freeze(struct virtio_device *vdev)
> +{
> + struct virtnet_info *vi = vdev->priv;
> +
> + netif_device_detach(vi->dev);
> + remove_vq_common(vi);
This stops TX in progress, if any, but not RX
which might use the RX VQ. Then remove_vq_common
might delete this VQ while it's still in use.
So I think we need to call something like napi_disable.
However, the subtle twist is that we need to call that
*after interrupts have been disabled*.
Otherwise we might schedule another napi callback.
> +
> + return 0;
> +}
> +
> +static int virtnet_restore(struct virtio_device *vdev)
> +{
> + struct virtnet_info *vi = vdev->priv;
> + int err;
> +
> + err = init_vqs(vi);
> + if (err)
> + return err;
> +
> + try_fill_recv(vi, GFP_KERNEL);
> +
> + netif_device_attach(vi->dev);
> + return 0;
> +}
> +#endif
> +
> static struct virtio_device_id id_table[] = {
> { VIRTIO_ID_NET, VIRTIO_DEV_ANY_ID },
> { 0 },
> @@ -1175,6 +1202,10 @@ static struct virtio_driver virtio_net_driver = {
> .probe =virtnet_probe,
> .remove = __devexit_p(virtnet_remove),
> .config_changed = virtnet_config_changed,
> +#ifdef CONFIG_PM
> + .freeze = virtnet_freeze,
> + .restore = virtnet_restore,
> +#endif
> };
>
> static int __init init(void)
> --
> 1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 11/11] virtio: balloon: Add freeze, restore handlers to support S4
On Thu, Nov 17, 2011 at 05:27:42PM +0530, Amit Shah wrote:
> Delete the vqs on the freeze callback to prepare for hibernation.
> Re-create the vqs in the restore callback to resume normal function.
>
> Signed-off-by: Amit Shah
> ---
> drivers/virtio/virtio_balloon.c | 20
> 1 files changed, 20 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 1ff3cf4..90149d1 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -363,6 +363,22 @@ static void __devexit virtballoon_remove(struct
> virtio_device *vdev)
> kfree(vb);
> }
>
> +#ifdef CONFIG_PM
> +static int virtballoon_freeze(struct virtio_device *vdev)
> +{
> + /* Now we reset the device so we can clean up the queues. */
> + vdev->config->reset(vdev);
> +
This is a weird thing to do. We normally delete vqs then reset.
For example, I don't know whether we guarantee what happens
to MSI-X vectors assigned meanwhile if you reset.
IIRC qemu doesn't use MSI-X for the balloon, but nothing
prevents it.
> + vdev->config->del_vqs(vdev);
What prevents requests being submitted at this point?
I see all kind of handling of freezing in the balloon thread ...
maybe that gives us some guarantees, but if yes
it makes sense to add a comment to point this out.
> + return 0;
> +}
> +
> +static int virtballoon_restore(struct virtio_device *vdev)
> +{
> + return init_vqs(vdev->priv);
> +}
> +#endif
> +
> static unsigned int features[] = {
> VIRTIO_BALLOON_F_MUST_TELL_HOST,
> VIRTIO_BALLOON_F_STATS_VQ,
> @@ -377,6 +393,10 @@ static struct virtio_driver virtio_balloon_driver = {
> .probe =virtballoon_probe,
> .remove = __devexit_p(virtballoon_remove),
> .config_changed = virtballoon_changed,
> +#ifdef CONFIG_PM
> + .freeze = virtballoon_freeze,
> + .restore = virtballoon_restore,
> +#endif
> };
>
> static int __init init(void)
> --
> 1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 06/11] virtio: blk: Add freeze, restore handlers to support S4
On Thu, Nov 17, 2011 at 05:27:37PM +0530, Amit Shah wrote:
> Delete the vq and flush any pending requests from the block queue on the
> freeze callback to prepare for hibernation.
>
> Re-create the vq in the restore callback to resume normal function.
>
> Signed-off-by: Amit Shah
> ---
> drivers/block/virtio_blk.c | 36
> 1 files changed, 36 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 467f218..f73fb2d 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -568,6 +568,38 @@ static void __devexit virtblk_remove(struct
> virtio_device *vdev)
> ida_simple_remove(&vd_index_ida, index);
> }
>
> +#ifdef CONFIG_PM
> +static int virtblk_freeze(struct virtio_device *vdev)
> +{
> + struct virtio_blk *vblk = vdev->priv;
> +
> + flush_work(&vblk->config_work);
What if a new config_work is scheduled after this point?
The right thing to do seems to be to flush after disabling interrupts.
> + spin_lock_irq(vblk->disk->queue->queue_lock);
> + blk_stop_queue(vblk->disk->queue);
> + spin_unlock_irq(vblk->disk->queue->queue_lock);
> + blk_sync_queue(vblk->disk->queue);
> +
> + /* Stop all the virtqueues. */
> + vdev->config->reset(vdev);
> + vdev->config->del_vqs(vdev);
I think del_vqs should be followed by reset:
del_vqs assumes the device in a state find put it in.
> + return 0;
> +}
> +
> +static int virtblk_restore(struct virtio_device *vdev)
> +{
> + struct virtio_blk *vblk = vdev->priv;
> + int ret;
> +
> + ret = init_vq(vdev->priv);
> + if (!ret) {
> + spin_lock_irq(vblk->disk->queue->queue_lock);
> + blk_start_queue(vblk->disk->queue);
> + spin_unlock_irq(vblk->disk->queue->queue_lock);
> + }
> + return ret;
> +}
> +#endif
> +
> static const struct virtio_device_id id_table[] = {
> { VIRTIO_ID_BLOCK, VIRTIO_DEV_ANY_ID },
> { 0 },
> @@ -593,6 +625,10 @@ static struct virtio_driver __refdata virtio_blk = {
> .probe = virtblk_probe,
> .remove = __devexit_p(virtblk_remove),
> .config_changed = virtblk_config_changed,
> +#ifdef CONFIG_PM
> + .freeze = virtblk_freeze,
> + .restore= virtblk_restore,
> +#endif
> };
>
> static int __init init(void)
> --
> 1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 09/11] virtio: net: Add freeze, restore handlers to support S4
On (Thu) 17 Nov 2011 [14:19:09], Michael S. Tsirkin wrote:
> On Thu, Nov 17, 2011 at 05:27:40PM +0530, Amit Shah wrote:
> > +#ifdef CONFIG_PM
> > +static int virtnet_freeze(struct virtio_device *vdev)
> > +{
> > + struct virtnet_info *vi = vdev->priv;
> > +
> > + netif_device_detach(vi->dev);
> > + remove_vq_common(vi);
>
> This stops TX in progress, if any, but not RX
> which might use the RX VQ. Then remove_vq_common
> might delete this VQ while it's still in use.
>
> So I think we need to call something like napi_disable.
> However, the subtle twist is that we need to call that
> *after interrupts have been disabled*.
> Otherwise we might schedule another napi callback.
resetting the vqs will mean the host won't pass us any data in the
vqs. Plus we're removing the vqs altogether. Also, we're disabling
the pci device in virtio_pci.c, so all of these actions will take
care of that, isn't it?
In addition, once the vqs are taken off, there's no chance for any
other rx to happen, so napi_disable() after plugging off vqs doesn't
make sense.
Amit
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 04/11] virtio: console: Add freeze and restore handlers to support S4
On Thu, Nov 17, 2011 at 05:27:35PM +0530, Amit Shah wrote:
> Remove all vqs and associated buffers in the freeze callback which
> prepares us to go into hibernation state. On restore, re-create all the
> vqs and populate the input vqs with buffers to get to the pre-hibernate
> state.
>
> Note: Any outstanding unconsumed buffers are discarded; which means
> there's a possibility of data loss in case the host or the guest didn't
> consume any data already present in the vqs. This can be addressed in a
> later patch series, perhaps in virtio common code.
>
> Signed-off-by: Amit Shah
> ---
> drivers/char/virtio_console.c | 58
> +
> 1 files changed, 58 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> index e14f5aa..fd2fd6f 100644
> --- a/drivers/char/virtio_console.c
> +++ b/drivers/char/virtio_console.c
> @@ -1844,6 +1844,60 @@ static unsigned int features[] = {
> VIRTIO_CONSOLE_F_MULTIPORT,
> };
>
> +#ifdef CONFIG_PM
> +static int virtcons_freeze(struct virtio_device *vdev)
> +{
> + struct ports_device *portdev;
> + struct port *port;
> +
> + portdev = vdev->priv;
> +
> + vdev->config->reset(vdev);
This does a reset but that's not a guarantee that
interrupt is not running on another CPU.
> +
> + cancel_work_sync(&portdev->control_work);
And then work can get scheduled after this point.
> + remove_controlq_data(portdev);
And after this point this will lead to a use after free.
> +
> + list_for_each_entry(port, &portdev->ports, list) {
> + /*
> + * We'll ask the host later if the new invocation has
> + * the port opened or closed.
> + */
> + port->host_connected = false;
> + remove_port_data(port);
> + }
> + remove_vqs(portdev);
> +
> + return 0;
> +}
> +
> +static int virtcons_restore(struct virtio_device *vdev)
> +{
> + struct ports_device *portdev;
> + struct port *port;
> + int ret;
> +
> + portdev = vdev->priv;
> +
> + ret = init_vqs(portdev);
> + if (ret)
> + return ret;
> +
> + if (use_multiport(portdev))
> + fill_queue(portdev->c_ivq, &portdev->cvq_lock);
> +
> + list_for_each_entry(port, &portdev->ports, list) {
> + port->in_vq = portdev->in_vqs[port->id];
> + port->out_vq = portdev->out_vqs[port->id];
> +
> + fill_queue(port->in_vq, &port->inbuf_lock);
> +
> + /* Get port open/close status on the host */
> + send_control_msg(port, VIRTIO_CONSOLE_PORT_READY, 1);
> + }
> + return 0;
> +}
> +#endif
> +
> static struct virtio_driver virtio_console = {
> .feature_table = features,
> .feature_table_size = ARRAY_SIZE(features),
> @@ -1853,6 +1907,10 @@ static struct virtio_driver virtio_console = {
> .probe =virtcons_probe,
> .remove = virtcons_remove,
> .config_changed = config_intr,
> +#ifdef CONFIG_PM
> + .freeze = virtcons_freeze,
> + .restore = virtcons_restore,
> +#endif
> };
>
> static int __init init(void)
> --
> 1.7.7.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 11/11] virtio: balloon: Add freeze, restore handlers to support S4
On (Thu) 17 Nov 2011 [14:25:08], Michael S. Tsirkin wrote:
> On Thu, Nov 17, 2011 at 05:27:42PM +0530, Amit Shah wrote:
> > Delete the vqs on the freeze callback to prepare for hibernation.
> > Re-create the vqs in the restore callback to resume normal function.
> >
> > Signed-off-by: Amit Shah
> > ---
> > drivers/virtio/virtio_balloon.c | 20
> > 1 files changed, 20 insertions(+), 0 deletions(-)
> >
> > diff --git a/drivers/virtio/virtio_balloon.c
> > b/drivers/virtio/virtio_balloon.c
> > index 1ff3cf4..90149d1 100644
> > --- a/drivers/virtio/virtio_balloon.c
> > +++ b/drivers/virtio/virtio_balloon.c
> > @@ -363,6 +363,22 @@ static void __devexit virtballoon_remove(struct
> > virtio_device *vdev)
> > kfree(vb);
> > }
> >
> > +#ifdef CONFIG_PM
> > +static int virtballoon_freeze(struct virtio_device *vdev)
> > +{
> > + /* Now we reset the device so we can clean up the queues. */
> > + vdev->config->reset(vdev);
> > +
>
> This is a weird thing to do. We normally delete vqs then reset.
No, all the drivers first reset, then delete.
> For example, I don't know whether we guarantee what happens
> to MSI-X vectors assigned meanwhile if you reset.
> IIRC qemu doesn't use MSI-X for the balloon, but nothing
> prevents it.
>
> > + vdev->config->del_vqs(vdev);
>
> What prevents requests being submitted at this point?
> I see all kind of handling of freezing in the balloon thread ...
> maybe that gives us some guarantees, but if yes
> it makes sense to add a comment to point this out.
Once reset, the host won't write anything to the vqs anyway..
Amit
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 09/11] virtio: net: Add freeze, restore handlers to support S4
On Thu, Nov 17, 2011 at 05:57:06PM +0530, Amit Shah wrote:
> On (Thu) 17 Nov 2011 [14:19:09], Michael S. Tsirkin wrote:
> > On Thu, Nov 17, 2011 at 05:27:40PM +0530, Amit Shah wrote:
>
> > > +#ifdef CONFIG_PM
> > > +static int virtnet_freeze(struct virtio_device *vdev)
> > > +{
> > > + struct virtnet_info *vi = vdev->priv;
> > > +
> > > + netif_device_detach(vi->dev);
> > > + remove_vq_common(vi);
> >
> > This stops TX in progress, if any, but not RX
> > which might use the RX VQ. Then remove_vq_common
> > might delete this VQ while it's still in use.
> >
> > So I think we need to call something like napi_disable.
> > However, the subtle twist is that we need to call that
> > *after interrupts have been disabled*.
> > Otherwise we might schedule another napi callback.
>
> resetting the vqs will mean the host won't pass us any data in the
> vqs. Plus we're removing the vqs altogether. Also, we're disabling
> the pci device in virtio_pci.c, so all of these actions will take
> care of that, isn't it?
I don't think so.
> In addition, once the vqs are taken off, there's no chance for any
> other rx to happen, so napi_disable() after plugging off vqs doesn't
> make sense.
>
> Amit
Yes but napi might have been scheduled before remove_vq_common was
called, and run afterwards.
--
MST
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 11/11] virtio: balloon: Add freeze, restore handlers to support S4
On Thu, Nov 17, 2011 at 05:59:20PM +0530, Amit Shah wrote:
> On (Thu) 17 Nov 2011 [14:25:08], Michael S. Tsirkin wrote:
> > On Thu, Nov 17, 2011 at 05:27:42PM +0530, Amit Shah wrote:
> > > Delete the vqs on the freeze callback to prepare for hibernation.
> > > Re-create the vqs in the restore callback to resume normal function.
> > >
> > > Signed-off-by: Amit Shah
> > > ---
> > > drivers/virtio/virtio_balloon.c | 20
> > > 1 files changed, 20 insertions(+), 0 deletions(-)
> > >
> > > diff --git a/drivers/virtio/virtio_balloon.c
> > > b/drivers/virtio/virtio_balloon.c
> > > index 1ff3cf4..90149d1 100644
> > > --- a/drivers/virtio/virtio_balloon.c
> > > +++ b/drivers/virtio/virtio_balloon.c
> > > @@ -363,6 +363,22 @@ static void __devexit virtballoon_remove(struct
> > > virtio_device *vdev)
> > > kfree(vb);
> > > }
> > >
> > > +#ifdef CONFIG_PM
> > > +static int virtballoon_freeze(struct virtio_device *vdev)
> > > +{
> > > + /* Now we reset the device so we can clean up the queues. */
> > > + vdev->config->reset(vdev);
> > > +
> >
> > This is a weird thing to do. We normally delete vqs then reset.
>
> No, all the drivers first reset, then delete.
BTW, reset does not flush callbacks at the moment.
It probably should.
> > For example, I don't know whether we guarantee what happens
> > to MSI-X vectors assigned meanwhile if you reset.
> > IIRC qemu doesn't use MSI-X for the balloon, but nothing
> > prevents it.
> >
> > > + vdev->config->del_vqs(vdev);
> >
> > What prevents requests being submitted at this point?
> > I see all kind of handling of freezing in the balloon thread ...
> > maybe that gives us some guarantees, but if yes
> > it makes sense to add a comment to point this out.
>
> Once reset, the host won't write anything to the vqs anyway..
>
> Amit
But thread might have already been scheduled, handling
previous requests.
--
MST
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH v2] virtio-mmio: Devices parameter parsing
This patch adds an option to instantiate guest virtio-mmio devices
basing on a kernel command line (or module) parameter, for example:
virtio_mmio.devices=0x100@0x100b:48,1K@0x1001e000:74
Signed-off-by: Pawel Moll
---
drivers/virtio/Kconfig | 31 +++
drivers/virtio/virtio_mmio.c | 181 +-
2 files changed, 211 insertions(+), 1 deletions(-)
Hi Rusty,
This version adds the "first_id" parameter I mentioned yesterday,
but still using single charp parameter instead of the _cb version.
And unless you are really (_really_) against it, I'd rather see
this variant.
Cheers!
Paweł
diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index 816ed08..00f2c82 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -46,4 +46,35 @@ config VIRTIO_BALLOON
If unsure, say N.
+config VIRTIO_MMIO_CMDLINE_DEVICES
+ bool "Memory mapped virtio devices parameter parsing"
+ depends on VIRTIO_MMIO
+ ---help---
+Allow virtio-mmio devices instantiation via the kernel command line
+or module parameter. Be aware that using incorrect parameters (base
+address in particular) can crash your system - you have been warned.
+
+The format for the parameter is as follows:
+
+ [virtio_mmio.]devices=[]
+
+where:
+ := @:
+ := ',' or ';'
+:= size (can use standard suffixes like K or M)
+:= physical base address
+ := interrupt number (as passed to request_irq())
+
+Example kernel command line parameter:
+
+ virtio_mmio.devices=0x100@0x100b:48,1K@0x1001e000:74
+
+This will register platform devices virtio_mmio., where
+values are consecutive integer numbers starting from 0 by default.
+The first id value can be changed with "first_id" parameter:
+
+ [virtio_mmio.]first_id=
+
+If unsure, say 'N'.
+
endmenu
diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
index acc5e43..05b39c1 100644
--- a/drivers/virtio/virtio_mmio.c
+++ b/drivers/virtio/virtio_mmio.c
@@ -6,6 +6,55 @@
* This module allows virtio devices to be used over a virtual, memory mapped
* platform device.
*
+ * The guest device(s) may be instantiated in one of three equivalent ways:
+ *
+ * 1. Static platform device in board's code, eg.:
+ *
+ * static struct platform_device v2m_virtio_device = {
+ * .name = "virtio-mmio",
+ * .id = -1,
+ * .num_resources = 2,
+ * .resource = (struct resource []) {
+ * {
+ * .start = 0x1001e000,
+ * .end = 0x1001e0ff,
+ * .flags = IORESOURCE_MEM,
+ * }, {
+ * .start = 42 + 32,
+ * .end = 42 + 32,
+ * .flags = IORESOURCE_IRQ,
+ * },
+ * }
+ * };
+ *
+ * 2. Device Tree node, eg.:
+ *
+ * virtio_block@1e000 {
+ * compatible = "virtio,mmio";
+ * reg = <0x1e000 0x100>;
+ * interrupts = <42>;
+ * }
+ *
+ * 3. Kernel module (or command line) parameter:
+ *
+ * [virtio_mmio.]devices=[]
+ *where:
+ *:= @:
+ * := ',' or ';'
+ * := size (can use standard suffixes like K or M)
+ * := physical base address
+ * := interrupt number (as passed to request_irq())
+ *eg.:
+ * virtio_mmio.devices=0x100@0x100b:48,1K@0x1001e000:74
+ *
+ *This will register platform devices virtio_mmio., where
+ *values are consecutive integer numbers starting from 0 by default.
+ *The first id value can be changed with "first_id" parameter:
+ *
+ * [virtio_mmio.]first_id=
+ *
+ *
+ *
* Registers layout (all 32-bit wide):
*
* offset d. name description
@@ -42,6 +91,8 @@
* See the COPYING file in the top-level directory.
*/
+#define pr_fmt(fmt) "virtio-mmio: " fmt
+
#include
#include
#include
@@ -443,6 +494,130 @@ static int __devexit virtio_mmio_remove(struct
platform_device *pdev)
+/* Devices list parameter */
+
+#if defined(CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES)
+
+static char *virtio_mmio_cmdline_devices;
+module_param_named(devices, virtio_mmio_cmdline_devices, charp, 0);
+
+static int virtio_mmio_cmdline_id;
+module_param_named(first_id, virtio_mmio_cmdline_id, int, 0);
+
+static struct device virtio_mmio_cmdline_parent = {
+ .init_name = "virtio-mmio-cmdline",
+};
+
+static int virtio_mmio_register_cmdline_devices(void)
+{
+ int err;
+ char *device;
+ char *token = NULL;
+
+ err = device_register(&virtio_mmio_cmdline_parent);
+ if (err) {
+
Re: [PATCH v3 11/11] virtio: balloon: Add freeze, restore handlers to support S4
On Thu, Nov 17, 2011 at 05:59:20PM +0530, Amit Shah wrote:
> On (Thu) 17 Nov 2011 [14:25:08], Michael S. Tsirkin wrote:
> > On Thu, Nov 17, 2011 at 05:27:42PM +0530, Amit Shah wrote:
> > > Delete the vqs on the freeze callback to prepare for hibernation.
> > > Re-create the vqs in the restore callback to resume normal function.
> > >
> > > Signed-off-by: Amit Shah
> > > ---
> > > drivers/virtio/virtio_balloon.c | 20
> > > 1 files changed, 20 insertions(+), 0 deletions(-)
> > >
> > > diff --git a/drivers/virtio/virtio_balloon.c
> > > b/drivers/virtio/virtio_balloon.c
> > > index 1ff3cf4..90149d1 100644
> > > --- a/drivers/virtio/virtio_balloon.c
> > > +++ b/drivers/virtio/virtio_balloon.c
> > > @@ -363,6 +363,22 @@ static void __devexit virtballoon_remove(struct
> > > virtio_device *vdev)
> > > kfree(vb);
> > > }
> > >
> > > +#ifdef CONFIG_PM
> > > +static int virtballoon_freeze(struct virtio_device *vdev)
> > > +{
> > > + /* Now we reset the device so we can clean up the queues. */
> > > + vdev->config->reset(vdev);
> > > +
> >
> > This is a weird thing to do. We normally delete vqs then reset.
>
> No, all the drivers first reset, then delete.
Interesting. I note that for PCI, reset actually just did an I/O
write, which in PCI is really posted, that is it
can complete on CPU before the device has received it.
Further, interrupts might have been pending on the CPU,
so device might get used after reset.
Patch coming.
--
MST
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH] virtio-pci: make reset operation safer
virtio pci device reset actually just does an I/O
write, which in PCI is really posted, that is it
can complete on CPU before the device has received it.
Further, interrupts might have been pending on
another CPU, so device callback might get invoked after reset.
This conflicts with how drivers use reset, which is typically:
reset
unregister
a callback running after reset completed can race with
unregister, potentially leading to use after free bugs.
Fix by flushing out the write, and flushing pending interrupts.
This assumes that device is never reset from
its vq/config callbacks, or in parallel with being
added/removed, document this assumption.
Signed-off-by: Michael S. Tsirkin
---
Tested with virtio-net only.
Rusty, a bugfix, so 3.2 material?
drivers/virtio/virtio_pci.c | 18 ++
include/linux/virtio_config.h |2 ++
2 files changed, 20 insertions(+), 0 deletions(-)
diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c
index d242fcc..cb1090e 100644
--- a/drivers/virtio/virtio_pci.c
+++ b/drivers/virtio/virtio_pci.c
@@ -321,11 +321,29 @@ static void vp_set_status(struct virtio_device *vdev, u8
status)
iowrite8(status, vp_dev->ioaddr + VIRTIO_PCI_STATUS);
}
+/* wait for pending irq handlers */
+static void vp_synchronize_vectors(struct virtio_device *vdev)
+{
+ struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+ int i;
+
+ if (vp_dev->intx_enabled)
+ synchronize_irq(vp_dev->pci_dev->irq);
+
+ for (i = 0; i < vp_dev->msix_vectors; ++i)
+ synchronize_irq(vp_dev->msix_entries[i].vector);
+}
+
static void vp_reset(struct virtio_device *vdev)
{
struct virtio_pci_device *vp_dev = to_vp_device(vdev);
/* 0 status means a reset. */
iowrite8(0, vp_dev->ioaddr + VIRTIO_PCI_STATUS);
+ /* Flush out the status write, and flush in device writes,
+* including MSi-X interrupts, if any. */
+ ioread8(vp_dev->ioaddr + VIRTIO_PCI_STATUS);
+ /* Flush pending VQ/configuration callbacks. */
+ vp_synchronize_vectors(vdev);
}
/* the notify function used when creating a virt queue */
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index add4790..e9e72bd 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -85,6 +85,8 @@
* @reset: reset the device
* vdev: the virtio device
* After this, status and feature negotiation must be done again
+ * Device must not be reset from its vq/config callbacks, or in
+ * parallel with being added/removed.
* @find_vqs: find virtqueues and instantiate them.
* vdev: the virtio_device
* nvqs: the number of virtqueues to find
--
1.7.5.53.gc233e
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 1/3] virtio_console: Fix locking of vtermno.
On (Fri) 11 Nov 2011 [14:57:20], Rusty Russell wrote: > On Tue, 08 Nov 2011 13:44:58 -0800, Miche Baker-Harvey > wrote: > > Some modifications of vtermno were not done under the spinlock. > > > > Moved assignment from vtermno and increment of vtermno together, > > putting both under the spinlock. Revert vtermno on failure. > > > > Signed-off-by: Miche Baker-Harvey > > Does it matter? It's normal not to lock in a function called > "init_XXX", since it's not exposed yet. > > Or is it? Slight misnomer, I suppose. We do this init_console_port() as part of add_port() if the port is a console port. Should it be named 'mark_console_port()'? Dunno, doesn't sound like the right name. init fits closest. Amit ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC] [ver3 PATCH 3/6] virtio_net: virtio_net driver changes
On Fri, 2011-11-18 at 01:08 +, Ben Hutchings wrote:
> On Fri, 2011-11-11 at 18:34 +0530, Krishna Kumar wrote:
> > Changes for multiqueue virtio_net driver.
> [...]
> > @@ -677,25 +730,35 @@ static struct rtnl_link_stats64 *virtnet
> > {
> > struct virtnet_info *vi = netdev_priv(dev);
> > int cpu;
> > - unsigned int start;
> >
> > for_each_possible_cpu(cpu) {
> > - struct virtnet_stats __percpu *stats
> > - = per_cpu_ptr(vi->stats, cpu);
> > - u64 tpackets, tbytes, rpackets, rbytes;
> > -
> > - do {
> > - start = u64_stats_fetch_begin(&stats->syncp);
> > - tpackets = stats->tx_packets;
> > - tbytes = stats->tx_bytes;
> > - rpackets = stats->rx_packets;
> > - rbytes = stats->rx_bytes;
> > - } while (u64_stats_fetch_retry(&stats->syncp, start));
> > -
> > - tot->rx_packets += rpackets;
> > - tot->tx_packets += tpackets;
> > - tot->rx_bytes += rbytes;
> > - tot->tx_bytes += tbytes;
> > + int qpair;
> > +
> > + for (qpair = 0; qpair < vi->num_queue_pairs; qpair++) {
> > + struct virtnet_send_stats __percpu *tx_stat;
> > + struct virtnet_recv_stats __percpu *rx_stat;
>
> While you're at it, you can drop the per-CPU stats and make them only
> per-queue. There is unlikely to be any benefit in maintaining them
> per-CPU while receive and transmit processing is serialised per-queue.
It allows you to update stats without a lock.
Whats the benefit of having them per queue?
--
Sasha.
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
