[ovs-dev] [PATCH v9 00/15] Reconfigure netdev at runtime
Currently we treat set_multiq() calls specially in netdev and dpif-netdev: every pmd thread must be stopped and set_multiq() is allowed to destroy and recreate the device. I think we can improve this by: * Generalizing the mechanism to allow changing other parameters at runtime (such as MTU). * Involving less the above layer (dpif-netdev). The request for changes often comes from below (netdev_dpdk_set_config(), or the vhost new_device() callback). There's no need for dpif-netdev to remember the requested value, all that it needs to know is that a configuration change is requested. This series implements exactly this: a mechanism to allow a netdev provider to request configuration changes, to which dpif-netdev will respond by stopping rx/tx and calling a netdev function to appy the new configuration. The new mechanism is used in this series to replace the set_multiq() call, but the idea is to use it also at least for: * Changing the MTU at runtime * Automatically detecting the number of rx queues for a vhost-user device * Move a DPDK vhost device to the proper NUMA socket The first commits refactor some code in dpif-netdev and, most importantly avoid using RCU for ports. Each thread will have its local copy of all the ports in the datapath. The series is also available here: https://github.com/ddiproietto/ovs/tree/configchangesv9 v9: * Fix HMAP_FOR_EACH_POP: now it's O(n) in the number of buckets * Avoid using & in clang thread safety annotations * Fix for non pmd devices: dp_netdev_set_pmds_on_numa() and dp_netdev_set_nonpmd() now add the ports to the pmd local cache. * Merged patch "dpif-netdev: Remove duplicate code in dp_netdev_set_pmds_on_numa()." with "dpif-netdev: Add pmd thread local port cache for transmission." v8: * Update comment in rcu.h: ovs_mutex_cond_wait doesn't quiesce. * Change 'set_multiq' to 'set_tx_multiq'. * Added documentation in comments and commit messages explaining thread local port cache. * Fixed style issues reported by checkpatch.py. * Fixed race condition when deleting pmd thread. v7: * Dropped already applied patches. * Stop using RCU for ports. * Rebased against master. v6: * Rebased against master. * Check return value of netdev_rxq_open(). * Fix comment. v5: * Style fixes. * Fixed a bug in dp_netdev_free() in patch 6. v4: * Added another patch to uniform names of variables in netdev-dpdk (no functional change) * Update some netdev comments to document the relation between netdev_set_multiq() and netdev_reconfigure() * Clarify that when netdev_reconfigure() is called no call to netdev_send() or netdev_rxq_recv() must be issued. * Move check to skip reconfiguration in netdev_dpdk_reconfigure() before rte_eth_dev_stop(). v3: * Fixed another outdated comment about rx queue configuration, as pointed out by Mark * Removed unnecessary and buggy initialization of requested_n_rxq in reconfigure_pmd_threads(). * Removed unused 'err' variable in netdev_dpdk_set_multiq(). * Changed comparison in netdev_set_multiq() to use previous 'netdev->requested_n_txq' instead of 'netdev->up.n_txq' * Return immediately in netdev_dpdk_reconfigure() if configuration didn't change anything. v2: * Fixed do_add_port(): we have to call netdev_reconfigure() before opening the rxqs. This prevents memory leaks, and makes sure that the datapath polls the appropriate number of queues * Fixed netdev_dpdk_vhost_set_multiq(): it must call netdev_request_reconfigure(). Since it is now equal to netdev_dpdk_set_multiq(), the two function have been merged. * Fixed netdev_dpdk_set_config(): dev->requested_n_rxq is now accessed while holding the appropriate mutex. * Fixed some outdated comments about rx queue configuration. Daniele Di Proietto (15): dpif-netdev: Destroy 'port_mutex' in dp_netdev_free(). dpif-netdev: Remove unused 'index' in dp_netdev_pmd_thread. dpif-netdev: Factor out port_create() from do_add_port(). dpif-netdev: Add functions to modify rxq without reloading pmd threads. dpif-netdev: Fix race condition in pmd thread initialization. hmap: Add HMAP_FOR_EACH_POP. dpif-netdev: Add pmd thread local port cache for transmission. hmap: Use struct for hmap_at_position(). dpif-netdev: Use hmap for ports. ovs-thread: Do not quiesce in ovs_mutex_cond_wait(). ofproto-dpif: Call dpif_poll_threads_set() before dpif_run(). dpif-netdev: Change pmd thread configuration in dpif_netdev_run(). dpif-netdev: Handle errors in reconfigure_pmd_threads(). netdev: Add reconfigure request mechanism. netdev-dpdk: Use ->reconfigure() call to change rx/tx queues. lib/cfm.c| 5 +- lib/dpif-netdev.c| 756 --- lib/dpif-provider.h | 3 +- lib/hmap.c | 26 +- lib/hmap.h | 27 +- lib/id-pool.c| 5 +- lib/learning-switch.c| 5 +- lib/netdev-bsd.c | 3 +- lib/netdev-dpdk.c
Re: [ovs-dev] [PATCH v9 00/15] Reconfigure netdev at runtime
I think, this is ready for merging. Tested-by: Ilya Maximets Acked-by: Ilya Maximets On 23.04.2016 04:02, Daniele Di Proietto wrote: > Currently we treat set_multiq() calls specially in netdev and dpif-netdev: > every pmd thread must be stopped and set_multiq() is allowed to destroy and > recreate the device. > > I think we can improve this by: > * Generalizing the mechanism to allow changing other parameters at runtime > (such as MTU). > * Involving less the above layer (dpif-netdev). The request for changes > often comes from below (netdev_dpdk_set_config(), or the vhost new_device() > callback). There's no need for dpif-netdev to remember the requested value, > all that it needs to know is that a configuration change is requested. > > This series implements exactly this: a mechanism to allow a netdev provider > to request configuration changes, to which dpif-netdev will respond by > stopping rx/tx and calling a netdev function to appy the new configuration. > > The new mechanism is used in this series to replace the set_multiq() call, > but the idea is to use it also at least for: > > * Changing the MTU at runtime > * Automatically detecting the number of rx queues for a vhost-user device > * Move a DPDK vhost device to the proper NUMA socket > > The first commits refactor some code in dpif-netdev and, most importantly > avoid using RCU for ports. Each thread will have its local copy of all the > ports in the datapath. > > The series is also available here: > > https://github.com/ddiproietto/ovs/tree/configchangesv9 > > v9: > * Fix HMAP_FOR_EACH_POP: now it's O(n) in the number of buckets > * Avoid using & in clang thread safety annotations > * Fix for non pmd devices: dp_netdev_set_pmds_on_numa() and > dp_netdev_set_nonpmd() now add the ports to the pmd local cache. > * Merged patch "dpif-netdev: Remove duplicate code in > dp_netdev_set_pmds_on_numa()." with "dpif-netdev: Add pmd thread > local port cache for transmission." > > v8: > * Update comment in rcu.h: ovs_mutex_cond_wait doesn't quiesce. > * Change 'set_multiq' to 'set_tx_multiq'. > * Added documentation in comments and commit messages explaining thread local > port cache. > * Fixed style issues reported by checkpatch.py. > * Fixed race condition when deleting pmd thread. > > v7: > * Dropped already applied patches. > * Stop using RCU for ports. > * Rebased against master. > > v6: > * Rebased against master. > * Check return value of netdev_rxq_open(). > * Fix comment. > > v5: > * Style fixes. > * Fixed a bug in dp_netdev_free() in patch 6. > > v4: > * Added another patch to uniform names of variables in netdev-dpdk (no > functional change) > * Update some netdev comments to document the relation between > netdev_set_multiq() and netdev_reconfigure() > * Clarify that when netdev_reconfigure() is called no call to netdev_send() > or netdev_rxq_recv() must be issued. > * Move check to skip reconfiguration in netdev_dpdk_reconfigure() before > rte_eth_dev_stop(). > > v3: > * Fixed another outdated comment about rx queue configuration, as pointed out > by Mark > * Removed unnecessary and buggy initialization of requested_n_rxq in > reconfigure_pmd_threads(). > * Removed unused 'err' variable in netdev_dpdk_set_multiq(). > * Changed comparison in netdev_set_multiq() to use previous > 'netdev->requested_n_txq' instead of 'netdev->up.n_txq' > * Return immediately in netdev_dpdk_reconfigure() if configuration didn't > change anything. > > v2: > * Fixed do_add_port(): we have to call netdev_reconfigure() before opening > the rxqs. This prevents memory leaks, and makes sure that the datapath > polls the appropriate number of queues > * Fixed netdev_dpdk_vhost_set_multiq(): it must call > netdev_request_reconfigure(). Since it is now equal to > netdev_dpdk_set_multiq(), the two function have been merged. > * Fixed netdev_dpdk_set_config(): dev->requested_n_rxq is now accessed > while holding the appropriate mutex. > * Fixed some outdated comments about rx queue configuration. > > > Daniele Di Proietto (15): > dpif-netdev: Destroy 'port_mutex' in dp_netdev_free(). > dpif-netdev: Remove unused 'index' in dp_netdev_pmd_thread. > dpif-netdev: Factor out port_create() from do_add_port(). > dpif-netdev: Add functions to modify rxq without reloading pmd > threads. > dpif-netdev: Fix race condition in pmd thread initialization. > hmap: Add HMAP_FOR_EACH_POP. > dpif-netdev: Add pmd thread local port cache for transmission. > hmap: Use struct for hmap_at_position(). > dpif-netdev: Use hmap for ports. > ovs-thread: Do not quiesce in ovs_mutex_cond_wait(). > ofproto-dpif: Call dpif_poll_threads_set() before dpif_run(). > dpif-netdev: Change pmd thread configuration in dpif_netdev_run(). > dpif-netdev: Handle errors in reconfigure_pmd_threads(). > netdev: Add reconfigure request mechanism. > netdev-dpdk: Use ->reconfigure() call to change rx/tx queues. > > lib/cfm.c
Re: [ovs-dev] [PATCH v9 00/15] Reconfigure netdev at runtime
Thanks for all the suggestion and the testing! I've applied this to master with your acks, Daniele On 15/05/2016 23:27, "Ilya Maximets" wrote: >I think, this is ready for merging. > >Tested-by: Ilya Maximets >Acked-by: Ilya Maximets > >On 23.04.2016 04:02, Daniele Di Proietto wrote: >> Currently we treat set_multiq() calls specially in netdev and dpif-netdev: >> every pmd thread must be stopped and set_multiq() is allowed to destroy and >> recreate the device. >> >> I think we can improve this by: >> * Generalizing the mechanism to allow changing other parameters at runtime >> (such as MTU). >> * Involving less the above layer (dpif-netdev). The request for changes >> often comes from below (netdev_dpdk_set_config(), or the vhost new_device() >> callback). There's no need for dpif-netdev to remember the requested >> value, >> all that it needs to know is that a configuration change is requested. >> >> This series implements exactly this: a mechanism to allow a netdev provider >> to request configuration changes, to which dpif-netdev will respond by >> stopping rx/tx and calling a netdev function to appy the new configuration. >> >> The new mechanism is used in this series to replace the set_multiq() call, >> but the idea is to use it also at least for: >> >> * Changing the MTU at runtime >> * Automatically detecting the number of rx queues for a vhost-user device >> * Move a DPDK vhost device to the proper NUMA socket >> >> The first commits refactor some code in dpif-netdev and, most importantly >> avoid using RCU for ports. Each thread will have its local copy of all the >> ports in the datapath. >> >> The series is also available here: >> >> https://github.com/ddiproietto/ovs/tree/configchangesv9 >> >> v9: >> * Fix HMAP_FOR_EACH_POP: now it's O(n) in the number of buckets >> * Avoid using & in clang thread safety annotations >> * Fix for non pmd devices: dp_netdev_set_pmds_on_numa() and >> dp_netdev_set_nonpmd() now add the ports to the pmd local cache. >> * Merged patch "dpif-netdev: Remove duplicate code in >> dp_netdev_set_pmds_on_numa()." with "dpif-netdev: Add pmd thread >> local port cache for transmission." >> >> v8: >> * Update comment in rcu.h: ovs_mutex_cond_wait doesn't quiesce. >> * Change 'set_multiq' to 'set_tx_multiq'. >> * Added documentation in comments and commit messages explaining thread local >> port cache. >> * Fixed style issues reported by checkpatch.py. >> * Fixed race condition when deleting pmd thread. >> >> v7: >> * Dropped already applied patches. >> * Stop using RCU for ports. >> * Rebased against master. >> >> v6: >> * Rebased against master. >> * Check return value of netdev_rxq_open(). >> * Fix comment. >> >> v5: >> * Style fixes. >> * Fixed a bug in dp_netdev_free() in patch 6. >> >> v4: >> * Added another patch to uniform names of variables in netdev-dpdk (no >> functional change) >> * Update some netdev comments to document the relation between >> netdev_set_multiq() and netdev_reconfigure() >> * Clarify that when netdev_reconfigure() is called no call to netdev_send() >> or netdev_rxq_recv() must be issued. >> * Move check to skip reconfiguration in netdev_dpdk_reconfigure() before >> rte_eth_dev_stop(). >> >> v3: >> * Fixed another outdated comment about rx queue configuration, as pointed out >> by Mark >> * Removed unnecessary and buggy initialization of requested_n_rxq in >> reconfigure_pmd_threads(). >> * Removed unused 'err' variable in netdev_dpdk_set_multiq(). >> * Changed comparison in netdev_set_multiq() to use previous >> 'netdev->requested_n_txq' instead of 'netdev->up.n_txq' >> * Return immediately in netdev_dpdk_reconfigure() if configuration didn't >> change anything. >> >> v2: >> * Fixed do_add_port(): we have to call netdev_reconfigure() before opening >> the rxqs. This prevents memory leaks, and makes sure that the datapath >> polls the appropriate number of queues >> * Fixed netdev_dpdk_vhost_set_multiq(): it must call >> netdev_request_reconfigure(). Since it is now equal to >> netdev_dpdk_set_multiq(), the two function have been merged. >> * Fixed netdev_dpdk_set_config(): dev->requested_n_rxq is now accessed >> while holding the appropriate mutex. >> * Fixed some outdated comments about rx queue configuration. >> >> >> Daniele Di Proietto (15): >> dpif-netdev: Destroy 'port_mutex' in dp_netdev_free(). >> dpif-netdev: Remove unused 'index' in dp_netdev_pmd_thread. >> dpif-netdev: Factor out port_create() from do_add_port(). >> dpif-netdev: Add functions to modify rxq without reloading pmd >> threads. >> dpif-netdev: Fix race condition in pmd thread initialization. >> hmap: Add HMAP_FOR_EACH_POP. >> dpif-netdev: Add pmd thread local port cache for transmission. >> hmap: Use struct for hmap_at_position(). >> dpif-netdev: Use hmap for ports. >> ovs-thread: Do not quiesce in ovs_mutex_cond_wait(). >> ofproto-dpif: Call dpif_poll_threads_set() before dpi
