The patch adds OVS-DPDK supports on Windows. Motivation ---------- Currently OVS supports multiple datapath implementations: Linux kernel, Windows kernel, userspace with OVS-DPDK Linux, and HW offload. Adding any new feature to OVS datapath requires OS-specific expertise and usually ends up with feature mismatch, ex: Linux kernel supports feature A, but Windows does not, and high maintenace cost [1].
It would be great if OVS just uses single datapath across different platforms, and the datapath is portatble, performant, and easy to maintain. The natural choice is the dpif-netdev, the usersapce netdev datapath currently used by OVS-DPDK and AF_XDP, currently only works on Linux platform. So the last piece is to make OVS-DPDK runs on Windows. With this work, the OVS userspace datapath becomes the one runs on Windows/Linux/FreeBSD and any new datapath feature should naturally work on different OSes. Performance of userspace datapath should be equal or better to the kernel datapath, due to the kernel bypass design of DPDK/AF_XDP, and optimizations in OVS userspace datapath. OVS also translates its datapath flows into HW offload-capable NICs, such as Mellanox CX6. With userspace datapath, HW offload works by using tc-flower (when using AF_XDP netdev) or using rte_flow API, (when using DPDK netdev). We've tried non-triival OVS actions such as conntrack or tunnel actions offloaded into Mellanox card. Moving forward, OVS will have better cross-platform support, better performance, and easier to maintain. So far I haven't seen any virtual switch capable of doing all of the above. Implementation on Windows ------------------------- It's harder than I thought due to my Linux only background. Sergey and I first need to add meson build support to OVS, in order to make compiling and linking to Windows DPDK library easier[2]. In this patch, we use clang compiler on Windows to compile DPDK and OVS, and OVS links to DPDK static library (Dynamic lib still not supported yet in DPDK Windows). Windows DPDK library hasn't finished all its library/driver porting to Windows. So a couple of DPDK library code used by OVS doesn't compile. For examples: 1) ovs-numa.c: used to detect lcores used by DPDK. 2) open_memstream, fopencookies: used to redirect DPDK logs. 3) vhostuser doesn't support: used to connect VM/container. 4) dpdk-socket-mem not support: configuration I simply remove them in this patch. In the future, I will probably refactor or #ifdef. In addition, only a few DPDK PMD drivers are supported. (please check the DPDK doc[3]) 1) Physical: ice, i40e, mlx5, ixgbe 3) Virtual: e1000, vmxnet3 For ice, i40, ixgbe, you will need additional two Windows drivers, netuio and virt2phys. For mlx5, you will need WINOF-2. I tested this patch using mlx5 pmd on Azure cloud and on-prem and ixgbevf on AWS EC2, using Windows server 2019. And I only tested basic OpenFlow actions such as matching ipv4 headers, drop and output. Build and Start --------------- see Dockerfile for Building the binaries I have two servers connected back-to-back using dual port CX6 25Gbps cards. Both server have Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 12 core. One server runs testpmd as packet generator, sending 64B UDP packet. The other server runs OVS-DPDK on Windows server 2019 Datacenter. 1. Make sure dpdk-testpmd.exe works on your system. 2. Start ovsdb server Here, we are using "datapath_type=netdev" ovsdb-server.exe C:\openvswitch\etc\openvswitch\conf.db -vfile:info --remote=punix:db.sock --remote=ptcp:6640 --log-file --pidfile ovs-vsctl --no-wait init ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true ovs-vsctl --set Bridge br0 datapath_type=netdev other_config:pmd-cpu-mask=0xfff ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xfff ovs-vsctl set Open_vSwitch . other_config:hw-offload=true 3. Start ovs-vswitchd, assume your device is 02:00.0 and 02:00.1 Due to no numa support, we manually configure it using --dummy-numa ovs-vswitchd.exe --pidfile -vfile:info --log-file --disable-system \ --dummy-numa=”0,0,0,0,0,0,0,0,0,0,0,0” ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev ovs-vsctl add-port br0 p0 -- set int p0 type=dpdk options:dpdk-devargs=0000:02:00.0 ovs-vsctl add-port br0 p1 -- set int p1 type=dpdk options:dpdk-devargs=0000:02:00.1 4. Program OpenFlow rules - rxdrop ovs-ofctl add-flows br0 "in_port=p0, actions=drop" - l2fwd ovs-ofctl add-flows br0 "in_port=p0, actions=output:p1" 5. At the other Linux server, generate packets by ./dpdk-testpmd -a 0000:02:00.0 -- -i --port-topology=chained \ --forward-mode=txonly --eth-peer=0,1C:34:DA:64:3B:A8 Performance ----------- I compared three solutions A) OVS Windows kernel module B) XDP example program in ebpf-for-windows github repo C) Windows DPDK l2fwd example app D) HW offload using rte_flow from OVS (other_config:hw-offload=true) E) OVS-DPDK (this project) rxdrop A) 2Mpps B) NA (there is no such example code) C) 31Mpps D) Not supported yet (no action=drop action support) E) 23Mpps l2fwd A) 1.2Mpps B) 120Kpps C) 29Mpps D) Not supported yet (no action=port id support) E) 23Mpps Deploying on the Cloud ---------------------- Currently I can only make it works on my bare-metal setup. I tried the three cloud providers: a. AWS EC2: AWS supports ixgbevf and DPDK supports it. However, I can load the dpdk-testpmd but the traffic doesn't go through. Otherwise, we need ENA driver support. b. Azure: Partially works with MLX5 VF, I tested using Standard D2sv3 VM. Other than VF, there is no virtual NIC support at this moment. c. Google cloud: There is no VF support, need to add virtio-net PMD support. Thoughts and Future Work ------------------------ a. Resolve all the compiler warnings b. Can we run ovs-dpdk on public cloud c. Interrupt mode support Reference --------- [1] https://dl.acm.org/doi/10.1145/3452296.3472914 [2] http://patchwork.ozlabs.org/project/openvswitch/cover/20210808014931.320242-1-sergey.madami...@gmail.com/ [3] https://doc.dpdk.org/guides/windows_gsg/ [4] Porting OvS-DPDK to Windows with Meson https://github.com/smadaminov/ovs-dpdk-meson-issues [5] ebpf-for-windows: https://github.com/microsoft/ebpf-for-windows Signed-off-by: William Tu <u9012...@gmail.com> --- v2: * add Dockerfile for building the project * add performance number and how to start ovs * enable HW rte_flow offload --- ovs/Dockerfile | 90 +++++++++++++++++++++++++++ ovs/config.h.meson | 7 --- ovs/include/openvswitch/compiler.h | 4 +- ovs/include/openvswitch/util.h | 2 +- ovs/lib/dpdk.c | 23 ++++--- ovs/lib/dpif-netdev.c | 4 ++ ovs/lib/meson.build | 13 +++- ovs/lib/netdev-dpdk.c | 99 ++++++++++++++++++++---------- ovs/lib/sflow_api.h | 1 + ovs/lib/util.c | 4 -- ovs/lib/vlog.c | 7 +++ ovs/meson.build | 88 ++++++++++++++++++++++---- ovs/meson_options.txt | 2 + ovs/ofproto/meson.build | 3 + ovs/ovsdb/meson.build | 3 + ovs/utilities/meson.build | 3 + ovs/vswitchd/meson.build | 10 +++ ovs/vtep/meson.build | 3 + 18 files changed, 298 insertions(+), 68 deletions(-) create mode 100644 ovs/Dockerfile diff --git a/ovs/Dockerfile b/ovs/Dockerfile new file mode 100644 index 0000000..6467fab --- /dev/null +++ b/ovs/Dockerfile @@ -0,0 +1,90 @@ +# escape=` +# I use some of the content from: +# https://github.com/aserdean/ovs_docker_file/blob/master/Dockerfile +# and add the DPDK windows built with OVS + +ARG WIN_VER="ltsc2019" +FROM mcr.microsoft.com/windows/servercore:$WIN_VER AS BUILDTOOLS + +# Downloa0d VS buildtool, choco installer, and WDK +ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe +ADD https://chocolatey.org/install.ps1 C:\TEMP\choco-install.ps1 +ADD https://go.microsoft.com/fwlink/?linkid=2085767 C:\TEMP\wdksetup.exe + +# Let's be explicit about the shell that we're going to use. +SHELL ["cmd", "/S", "/C"] + +# Download pthread4w +RUN mkdir c:\pthreads +RUN powershell Invoke-WebRequest 'ftp://sourceware.org/pub/pthreads-win32/pthreads-w32-2-9-1-release.zip' -OutFile 'C:\pthreads\pthreads-win32.zip' + +# Install Build Tools. A 3010 error signals that requested operation is +# successfull but changes will not be effective until the system is rebooted. +RUN C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache --installPath C:\BuildTools --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Component.VC.Runtimes.x86.x64.Spectre --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows10SDK.18362 --add Microsoft.VisualStudio.Component.VC.14.24.x86.x64 --add Microsoft.VisualStudio.Component.VC.14.24.x86.x64.Spectre || IF "%ERRORLEVEL%"=="3010" EXIT 0 + +# Install choco +RUN powershell C:\TEMP\choco-install.ps1 + +RUN choco install git -y +#RUN choco install msys2 --params "/NoUpdate /InstallDir:C:\msys2" -y +RUN choco install python3 -y +RUN choco install 7zip.install -y + +RUN 7z x C:\pthreads\pthreads-win32.zip -OC:\pthreads + +# OVS build using mingw and msys2 +#RUN msys2_shell.cmd -defterm -no-start -use-full-path -c "pacman --noconfirm -S automake autoconf libtool make patch" +#RUN msys2_shell.cmd -defterm -no-start -use-full-path -c "cp `which python` `which python`3" +#RUN msys2_shell.cmd -defterm -no-start -use-full-path -c "mv /usr/bin/link /usr/bin/link_bkup" +RUN python -m pip install pypiwin32 --disable-pip-version-check + +# Install WDK excluding WDK.vsix. +RUN C:\TEMP\wdksetup.exe /q + +# Install WDK.vsix in manual manner. +RUN copy "C:\Program Files (x86)\Windows Kits\10\Vsix\VS2019\WDK.vsix" C:\TEMP\wdkvsix.zip +RUN powershell Expand-Archive C:\TEMP\wdkvsix.zip -DestinationPath C:\TEMP\wdkvsix +RUN robocopy.exe /e "C:\temp\wdkvsix\$MSBuild\Microsoft\VC\v160" "C:\BuildTools\MSBuild\Microsoft\VC\v160" || EXIT 0 + +#VCRUNTIME 2010 +ADD https://download.microsoft.com/download/3/2/2/3224B87F-CFA0-4E70-BDA3-3DE650EFEBA5/vcredist_x64.exe C:\TEMP\vc_2010_x64.exe +RUN C:\TEMP\vc_2010_x64.exe /quiet /install + +RUN choco install meson -y +RUN choco install llvm -y +RUN choco install pkgconfiglite -y +RUN choco install ctags -y + +# ------------------- DPDK BUILD ---------------------- +FROM BUILDTOOLS AS DPDK-BUILD +# checkout DPDK commit e9123c467dbb due to API breakage on latest main +RUN cd C:\ && git clone --branch main https://dpdk.org/git/dpdk +RUN cd C:\dpdk && git checkout -b ovs-dpdk e9123c467dbb && meson build && ninja -C build + +# Build netuio and virtio +RUN cd C:\ && git clone git://dpdk.org/dpdk-kmods +RUN cd C:\dpdk-kmods\windows\netuio && c:\\BuildTools\\VC\\Auxiliary\\Build\\vcvarsall.bat x86_x64 10.0.18362.0 && msbuild +RUN cd C:\dpdk-kmods\windows\virt2phys && c:\\BuildTools\\VC\\Auxiliary\\Build\\vcvarsall.bat x86_x64 10.0.18362.0 && msbuild + +# ------------------- OVS-DPDK BUILD ---------------------- +FROM DPDK-BUILD AS OVS-DPDK-BUILD +# clone OVS +RUN cd C:\ && git clone --branch rfc2 https://github.com/smadaminov/ovs-dpdk-meson-issues.git + +# install dpdk +RUN mkdir C:\TEMP\dpdk && powershell "$Env:DESTDIR=\"C:\TEMP\dpdk\"; cd C:\dpdk\; ninja -C build install" + +# install pthread4w +RUN cd C:\ && git clone https://git.code.sf.net/p/pthreads4w/code pthreads4w-code && cd pthreads4w-code && C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvarsall.bat x86_x64 10.0.18362.0 && nmake all install + +RUN choco install vim -y + +# OVS needs to use meson 0.59.2 +RUN pip3 install meson==0.59.2 --user -U +RUN powershell New-Item -Type symbolicLink -Path C:\meson59.exe -value C:\Users\ContainerAdministrator\AppData\Roaming\Python\Python310\Scripts\meson.exe +RUN powershell "(Get-Content c:\PTHREADS-BUILT\include\_ptw32.h).replace('error \"Please upgrade', 'warning \"Please upgrade') | Set-Content c:\PTHREADS-BUILT\include\_ptw32.h" +RUN cd C:\ovs-dpdk-meson-issues\ovs && C:\meson59.exe build -Dwith-pthread=C:\PTHREADS-BUILT -Dwith-dpdk=C:\temp\dpdk\ -Dpkg_config_path=C:\temp\dpdk\lib\pkgconfig\ && ninja -C build + +SHELL ["cmd"] + +CMD [ "cmd","/k","c:\\BuildTools\\VC\\Auxiliary\\Build\\vcvarsall.bat", "x86_x64", "10.0.18362.0", "&&", "msys2_shell.cmd", "-no-start", "-defterm", "-msys2", "-use-full-path" ] diff --git a/ovs/config.h.meson b/ovs/config.h.meson index 3aaa1cb..fd6d79c 100644 --- a/ovs/config.h.meson +++ b/ovs/config.h.meson @@ -377,11 +377,4 @@ #include "include/windows/windefs.h" #endif -/* MSR: use meson checks to enable/disable this - and still not clear why it does not work on Linux - or maybe I should include this header only in Windows */ -#if defined(__clang__) && defined(WIN32) -#include <intrin.h> -#endif - #mesondefine HAVE_THREAD_SAFETY diff --git a/ovs/include/openvswitch/compiler.h b/ovs/include/openvswitch/compiler.h index 997d5eb..23b088a 100644 --- a/ovs/include/openvswitch/compiler.h +++ b/ovs/include/openvswitch/compiler.h @@ -172,7 +172,7 @@ * OVS_PACKED_ENUM is intended for use only as a space optimization, since it * only works with GCC. That means that it must not be used in wire protocols * or otherwise exposed outside of a single process. */ -#if __GNUC__ && !__CHECKER__ +#if __GNUC__ && !__CHECKER__ && !__clang__ #define OVS_PACKED_ENUM __attribute__((__packed__)) #define HAVE_PACKED_ENUM #else @@ -278,7 +278,7 @@ extern int (*build_assert(void))[BUILD_ASSERT__(EXPR)] #endif -#ifdef __GNUC__ +#if defined(__GNUC__) && !defined(__clang__) #define BUILD_ASSERT_GCCONLY(EXPR) BUILD_ASSERT(EXPR) #define BUILD_ASSERT_DECL_GCCONLY(EXPR) BUILD_ASSERT_DECL(EXPR) #else diff --git a/ovs/include/openvswitch/util.h b/ovs/include/openvswitch/util.h index 228b185..d73634f 100644 --- a/ovs/include/openvswitch/util.h +++ b/ovs/include/openvswitch/util.h @@ -100,7 +100,7 @@ OVS_NO_RETURN void ovs_assert_failure(const char *, const char *, const char *); * dereference any pointer, so it would be surprising for it to cause any * problems in practice. */ -#ifdef __GNUC__ +#if defined(__GNUC__) && !defined(__clang__) #define OBJECT_OFFSETOF(OBJECT, MEMBER) offsetof(typeof(*(OBJECT)), MEMBER) #else #define OBJECT_OFFSETOF(OBJECT, MEMBER) \ diff --git a/ovs/lib/dpdk.c b/ovs/lib/dpdk.c index 855467b..fd84ad2 100644 --- a/ovs/lib/dpdk.c +++ b/ovs/lib/dpdk.c @@ -274,9 +274,9 @@ construct_dpdk_args(const struct smap *ovs_other_config, struct svec *args) } construct_dpdk_options(ovs_other_config, args); - construct_dpdk_mutex_options(ovs_other_config, args); +// construct_dpdk_mutex_options(ovs_other_config, args); } - +/* static ssize_t dpdk_log_write(void *c OVS_UNUSED, const char *buf, size_t size) { @@ -312,7 +312,7 @@ dpdk_log_write(void *c OVS_UNUSED, const char *buf, size_t size) static cookie_io_functions_t dpdk_log_func = { .write = dpdk_log_write, }; - +*/ static void dpdk_unixctl_mem_stream(struct unixctl_conn *conn, int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, void *aux) @@ -423,6 +423,11 @@ dpdk_init__(const struct smap *ovs_other_config) struct ovs_numa_dump *affinity = NULL; struct svec args = SVEC_EMPTY_INITIALIZER; + VLOG_ERR("Can't redirect DPDK log: %s.", ovs_strerror(errno)); + + rte_openlog_stream(log_stream); + +#ifndef WIN32 log_stream = fopencookie(NULL, "w+", dpdk_log_func); if (log_stream == NULL) { VLOG_ERR("Can't redirect DPDK log: %s.", ovs_strerror(errno)); @@ -469,6 +474,7 @@ dpdk_init__(const struct smap *ovs_other_config) VLOG_INFO("POSTCOPY support for vhost-user-client %s.", vhost_postcopy_enabled ? "enabled" : "disabled"); +#endif per_port_memory = smap_get_bool(ovs_other_config, "per-port-memory", false); VLOG_INFO("Per port memory for DPDK devices %s.", @@ -531,7 +537,8 @@ dpdk_init__(const struct smap *ovs_other_config) optind = 1; - if (VLOG_IS_INFO_ENABLED()) { + //if (VLOG_IS_INFO_ENABLED()) { + if (true) { struct ds eal_args = DS_EMPTY_INITIALIZER; char *joined_args = svec_join(&args, " ", "."); @@ -545,6 +552,7 @@ dpdk_init__(const struct smap *ovs_other_config) * some arguments from it. '+1' to copy the terminating NULL. */ argv = xmemdup(args.names, (args.n + 1) * sizeof args.names[0]); + VLOG_INFO("XXX: Make sure things are initialized ..."); /* Make sure things are initialized ... */ result = rte_eal_init(args.n, argv); @@ -561,7 +569,7 @@ dpdk_init__(const struct smap *ovs_other_config) VLOG_EMER("Unable to initialize DPDK: %s", ovs_strerror(rte_errno)); return false; } - +#ifndef WIN32 if (VLOG_IS_DBG_ENABLED()) { size_t size; char *response = NULL; @@ -588,10 +596,11 @@ dpdk_init__(const struct smap *ovs_other_config) unixctl_command_register("dpdk/get-malloc-stats", "", 0, 0, dpdk_unixctl_mem_stream, malloc_dump_stats_wrapper); - +#endif /* We are called from the main thread here */ RTE_PER_LCORE(_lcore_id) = NON_PMD_CORE_ID; + VLOG_INFO("Register the dpdk classes"); /* Finally, register the dpdk classes */ netdev_dpdk_register(); netdev_register_flow_api_provider(&netdev_offload_dpdk); @@ -615,7 +624,7 @@ dpdk_init(const struct smap *ovs_other_config) static struct ovsthread_once once_enable = OVSTHREAD_ONCE_INITIALIZER; if (ovsthread_once_start(&once_enable)) { - VLOG_INFO("Using %s", rte_version()); + VLOG_INFO("Using %s", "123"); VLOG_INFO("DPDK Enabled - initializing..."); enabled = dpdk_init__(ovs_other_config); if (enabled) { diff --git a/ovs/lib/dpif-netdev.c b/ovs/lib/dpif-netdev.c index 59e326f..0f986ce 100644 --- a/ovs/lib/dpif-netdev.c +++ b/ovs/lib/dpif-netdev.c @@ -1575,7 +1575,11 @@ dpif_netdev_port_open_type(const struct dpif_class *class, const char *type) { return strcmp(type, "internal") ? type : dpif_netdev_class_is_dummy(class) ? "dummy-internal" +#ifdef _WIN32 + : "internal"; +#else : "tap"; +#endif } static struct dpif * diff --git a/ovs/lib/meson.build b/ovs/lib/meson.build index 42e0bbf..e828139 100644 --- a/ovs/lib/meson.build +++ b/ovs/lib/meson.build @@ -370,7 +370,6 @@ sources = files( 'lldp/lldp.c', 'lldp/lldpd.c', 'lldp/lldpd-structs.c', - 'dpdk-stub.c', 'async-append-null.c', # no POSIX_AIO 'stream-nossl.c', 'dns-resolve.h', @@ -389,11 +388,23 @@ if build_machine.system() == 'windows' 'strsep.c', 'dpif-netlink.c', 'netdev-windows.c', + 'netdev-offload-dpdk.c', 'netlink-conntrack.c', 'netlink-notifier.c', 'netlink-socket.c', 'wmi.c', +# 'syslog-direct.c', +# 'syslog-libc.c', + 'syslog-null.c', ) + + if enable_dpdk + sources += files( + 'dpdk.c', + 'netdev-dpdk.c', + ) + deps += dpdk_lib_deps + endif endif if build_machine.system() == 'linux' diff --git a/ovs/lib/netdev-dpdk.c b/ovs/lib/netdev-dpdk.c index 45a96b9..6b3a92d 100644 --- a/ovs/lib/netdev-dpdk.c +++ b/ovs/lib/netdev-dpdk.c @@ -22,9 +22,11 @@ #include <stdlib.h> #include <string.h> #include <unistd.h> +#ifndef WIN32 #include <linux/virtio_net.h> -#include <sys/socket.h> #include <linux/if.h> +#endif +#include <sys/socket.h> #include <rte_bus_pci.h> #include <rte_config.h> @@ -37,7 +39,9 @@ #include <rte_meter.h> #include <rte_pci.h> #include <rte_version.h> +#ifndef WIN32 #include <rte_vhost.h> +#endif #include "cmap.h" #include "coverage.h" @@ -180,7 +184,7 @@ static void destroy_device(int vid); static int vring_state_changed(int vid, uint16_t queue_id, int enable); static void destroy_connection(int vid); static void vhost_guest_notified(int vid); - +/* static const struct vhost_device_ops virtio_net_device_ops = { .new_device = new_device, @@ -191,7 +195,7 @@ static const struct vhost_device_ops virtio_net_device_ops = .destroy_connection = destroy_connection, .guest_notified = vhost_guest_notified, }; - +*/ /* Custom software stats for dpdk ports */ struct netdev_dpdk_sw_stats { /* No. of retries when unable to transmit. */ @@ -536,7 +540,9 @@ struct netdev_rxq_dpdk { }; static void netdev_dpdk_destruct(struct netdev *netdev); +#ifndef WIN32 static void netdev_dpdk_vhost_destruct(struct netdev *netdev); +#endif static int netdev_dpdk_get_sw_custom_stats(const struct netdev *, struct netdev_custom_stats *); @@ -550,8 +556,12 @@ netdev_dpdk_get_ingress_policer(const struct netdev_dpdk *dev); static bool is_dpdk_class(const struct netdev_class *class) { +#ifdef WIN32 + return class->destruct == netdev_dpdk_destruct; +#else return class->destruct == netdev_dpdk_destruct || class->destruct == netdev_dpdk_vhost_destruct; +#endif } /* DPDK NIC drivers allocate RX buffers at a particular granularity, typically @@ -1165,8 +1175,10 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) dev->started = true; rte_eth_promiscuous_enable(dev->port_id); +#if 0 + /* This is a bug. */ rte_eth_allmulticast_enable(dev->port_id); - +#endif memset(ð_addr, 0x0, sizeof(eth_addr)); rte_eth_macaddr_get(dev->port_id, ð_addr); VLOG_INFO_RL(&rl, "Port "DPDK_PORT_ID_FMT": "ETH_ADDR_FMT, @@ -1301,7 +1313,7 @@ netdev_dpdk_get_num_ports(struct rte_device *device) } return count; } - +#ifndef WIN32 static int vhost_common_construct(struct netdev *netdev) OVS_REQUIRES(dpdk_mutex) @@ -1423,7 +1435,7 @@ netdev_dpdk_vhost_client_construct(struct netdev *netdev) ovs_mutex_unlock(&dpdk_mutex); return err; } - +#endif static int netdev_dpdk_construct(struct netdev *netdev) { @@ -1499,7 +1511,7 @@ netdev_dpdk_destruct(struct netdev *netdev) ovs_mutex_unlock(&dpdk_mutex); } - +#ifndef WIN32 /* rte_vhost_driver_unregister() can call back destroy_device(), which will * try to acquire 'dpdk_mutex' and possibly 'dev->mutex'. To avoid a * deadlock, none of the mutexes must be held while calling this function. */ @@ -1551,7 +1563,7 @@ netdev_dpdk_vhost_destruct(struct netdev *netdev) out: free(vhost_id); } - +#endif static void netdev_dpdk_dealloc(struct netdev *netdev) { @@ -1759,6 +1771,7 @@ netdev_dpdk_get_port_by_mac(const char *mac_str) dpdk_port_t port_id; struct eth_addr mac, port_mac; + VLOG_WARN("mac: %s", mac_str); if (!eth_addr_from_string(mac_str, &mac)) { VLOG_ERR("invalid mac: %s", mac_str); return DPDK_ETH_PORT_ID_INVALID; @@ -1767,13 +1780,17 @@ netdev_dpdk_get_port_by_mac(const char *mac_str) RTE_ETH_FOREACH_DEV (port_id) { struct rte_ether_addr ea; + VLOG_WARN("mac: %d", (int)port_id); rte_eth_macaddr_get(port_id, &ea); memcpy(port_mac.ea, ea.addr_bytes, ETH_ADDR_LEN); if (eth_addr_equals(mac, port_mac)) { + + VLOG_WARN("found port"); return port_id; } } + VLOG_WARN("port not found!"); return DPDK_ETH_PORT_ID_INVALID; } @@ -1812,6 +1829,7 @@ netdev_dpdk_process_devargs(struct netdev_dpdk *dev, dpdk_port_t new_port_id; if (strncmp(devargs, "class=eth,mac=", 14) == 0) { + VLOG_WARN("get port by mac %s\n", &devargs[14]); new_port_id = netdev_dpdk_get_port_by_mac(&devargs[14]); } else { new_port_id = netdev_dpdk_get_port_by_devargs(devargs); @@ -2052,7 +2070,7 @@ out: return err; } - +#ifndef WIN32 static int netdev_dpdk_vhost_client_set_config(struct netdev *netdev, const struct smap *args, @@ -2088,7 +2106,7 @@ netdev_dpdk_vhost_client_set_config(struct netdev *netdev, return 0; } - +#endif static int netdev_dpdk_get_numa_id(const struct netdev *netdev) { @@ -2326,7 +2344,11 @@ ingress_policer_run(struct ingress_policer *policer, struct rte_mbuf **pkts, static bool is_vhost_running(struct netdev_dpdk *dev) { +#ifdef WIN32 + return false; +#else return (netdev_dpdk_get_vid(dev) >= 0 && dev->vhost_reconfigured); +#endif } static inline void @@ -2398,6 +2420,7 @@ netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, /* * The receive path for the vhost port is the TX path out from guest. */ +#ifndef WIN32 static int netdev_dpdk_vhost_rxq_recv(struct netdev_rxq *rxq, struct dp_packet_batch *batch, int *qfill) @@ -2457,7 +2480,7 @@ netdev_dpdk_vhost_rxq_enabled(struct netdev_rxq *rxq) return dev->vhost_rxq_enabled[rxq->queue_id]; } - +#endif static int netdev_dpdk_rxq_recv(struct netdev_rxq *rxq, struct dp_packet_batch *batch, int *qfill) @@ -2552,7 +2575,7 @@ netdev_dpdk_filter_packet_len(struct netdev_dpdk *dev, struct rte_mbuf **pkts, return cnt; } - +#ifndef WIN32 static inline void netdev_dpdk_vhost_update_tx_counters(struct netdev_dpdk *dev, struct dp_packet **packets, @@ -2668,7 +2691,7 @@ out: dp_packet_delete(pkts[i]); } } - +#endif static void netdev_dpdk_extbuf_free(void *addr OVS_UNUSED, void *opaque) { @@ -2825,13 +2848,13 @@ dpdk_do_tx_copy(struct netdev *netdev, int qid, struct dp_packet_batch *batch) } if (OVS_LIKELY(txcnt)) { - if (dev->type == DPDK_DEV_VHOST) { - __netdev_dpdk_vhost_send(netdev, qid, pkts, txcnt); - } else { +// if (dev->type == DPDK_DEV_VHOST) { +// __netdev_dpdk_vhost_send(netdev, qid, pkts, txcnt); +// } else { tx_failure += netdev_dpdk_eth_tx_burst(dev, qid, (struct rte_mbuf **)pkts, txcnt); - } +// } } dropped += qos_drops + mtu_drops + tx_failure; @@ -2844,7 +2867,7 @@ dpdk_do_tx_copy(struct netdev *netdev, int qid, struct dp_packet_batch *batch) rte_spinlock_unlock(&dev->stats_lock); } } - +#ifndef WIN32 static int netdev_dpdk_vhost_send(struct netdev *netdev, int qid, struct dp_packet_batch *batch, @@ -2860,7 +2883,7 @@ netdev_dpdk_vhost_send(struct netdev *netdev, int qid, } return 0; } - +#endif static inline void netdev_dpdk_send__(struct netdev_dpdk *dev, int qid, struct dp_packet_batch *batch, @@ -3580,7 +3603,7 @@ netdev_dpdk_update_flags(struct netdev *netdev, return error; } - +#ifndef WIN32 static int netdev_dpdk_vhost_user_get_status(const struct netdev *netdev, struct smap *args) @@ -3637,7 +3660,7 @@ netdev_dpdk_vhost_user_get_status(const struct netdev *netdev, ovs_mutex_unlock(&dev->mutex); return 0; } - +#endif /* * Convert a given uint32_t link speed defined in DPDK to a string * equivalent. @@ -3712,8 +3735,9 @@ netdev_dpdk_get_status(const struct netdev *netdev, struct smap *args) * support; cf. RFC 3635 Section 3.2.4. */ enum { IF_TYPE_ETHERNETCSMACD = 6 }; - smap_add_format(args, "if_type", "%"PRIu32, IF_TYPE_ETHERNETCSMACD); - smap_add_format(args, "if_descr", "%s %s", rte_version(), + smap_add_format(args, "if_type", "%"PRIu32, IF_TYPE_ETHERNETCSMACD); + smap_add_format(args, "if_descr", "%s %s","123", +// smap_add_format(args, "if_descr", "%s %s", rte_version(), dev_info.driver_name); smap_add_format(args, "pci-vendor_id", "0x%x", vendor_id); smap_add_format(args, "pci-device_id", "0x%x", device_id); @@ -3871,7 +3895,7 @@ netdev_dpdk_get_mempool_info(struct unixctl_conn *conn, goto out; } } - +#ifndef WIN32 stream = open_memstream(&response, &size); if (!stream) { response = xasprintf("Unable to open memstream: %s.", @@ -3879,7 +3903,7 @@ netdev_dpdk_get_mempool_info(struct unixctl_conn *conn, unixctl_command_reply_error(conn, response); goto out; } - +#endif if (netdev) { struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); @@ -3907,6 +3931,7 @@ out: /* * Set virtqueue flags so that we do not receive interrupts. */ +#ifndef WIN32 static void set_irq_status(int vid) { @@ -3916,7 +3941,7 @@ set_irq_status(int vid) rte_vhost_enable_guest_notification(vid, i, 0); } } - +#endif /* * Fixes mapping for vhost-user tx queues. Must be called after each * enabling/disabling of queues and n_txq modifications. @@ -3969,6 +3994,7 @@ netdev_dpdk_remap_txqs(struct netdev_dpdk *dev) /* * A new virtio-net device is added to a vhost port. */ +#ifndef WIN32 static int new_device(int vid) { @@ -3977,8 +4003,9 @@ new_device(int vid) int newnode = 0; char ifname[IF_NAME_SZ]; +#ifndef WIN32 rte_vhost_get_ifname(vid, ifname, sizeof ifname); - +#endif ovs_mutex_lock(&dpdk_mutex); /* Add device to the vhost port with the same name as that passed down. */ LIST_FOR_EACH(dev, list_node, &dpdk_list) { @@ -4032,7 +4059,7 @@ new_device(int vid) return 0; } - +#endif /* Clears mapping for all available queues of vhost interface. */ static void netdev_dpdk_txq_map_clear(struct netdev_dpdk *dev) @@ -4051,6 +4078,7 @@ netdev_dpdk_txq_map_clear(struct netdev_dpdk *dev) * ensure all currently queued packets have been sent/received before removing * the device. */ +#ifndef WIN32 static void destroy_device(int vid) { @@ -4214,6 +4242,7 @@ netdev_dpdk_get_vid(const struct netdev_dpdk *dev) return ovsrcu_index_get(&dev->vid); } +#endif struct ingress_policer * netdev_dpdk_get_ingress_policer(const struct netdev_dpdk *dev) { @@ -4229,7 +4258,7 @@ netdev_dpdk_class_init(void) * needs to be done only once */ if (ovsthread_once_start(&once)) { int ret; - +#ifndef WIN32 ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); unixctl_command_register("netdev-dpdk/set-admin-state", "[netdev] up|down", 1, 2, @@ -4250,7 +4279,7 @@ netdev_dpdk_class_init(void) VLOG_ERR("Ethernet device callback register error: %s", rte_strerror(-ret)); } - +#endif ovsthread_once_done(&once); } @@ -5041,7 +5070,7 @@ out: ovs_mutex_unlock(&dev->mutex); return err; } - +#ifndef WIN32 static int dpdk_vhost_reconfigure_helper(struct netdev_dpdk *dev) OVS_REQUIRES(dev->mutex) @@ -5193,7 +5222,7 @@ unlock: return err; } - +#endif int netdev_dpdk_get_port_id(struct netdev *netdev) { @@ -5461,7 +5490,7 @@ static const struct netdev_class dpdk_class = { .set_config = netdev_dpdk_set_config, .send = netdev_dpdk_eth_send, }; - +#ifndef WIN32 static const struct netdev_class dpdk_vhost_class = { .type = "dpdkvhostuser", NETDEV_DPDK_CLASS_COMMON, @@ -5492,11 +5521,13 @@ static const struct netdev_class dpdk_vhost_client_class = { .rxq_recv = netdev_dpdk_vhost_rxq_recv, .rxq_enabled = netdev_dpdk_vhost_rxq_enabled, }; - +#endif void netdev_dpdk_register(void) { netdev_register_provider(&dpdk_class); +#ifndef WIN32 netdev_register_provider(&dpdk_vhost_class); netdev_register_provider(&dpdk_vhost_client_class); +#endif } diff --git a/ovs/lib/sflow_api.h b/ovs/lib/sflow_api.h index 7264fc1..180648d 100644 --- a/ovs/lib/sflow_api.h +++ b/ovs/lib/sflow_api.h @@ -21,6 +21,7 @@ #include <unistd.h> #include <errno.h> #include <string.h> +#include <time.h> #ifdef SFLOW_DO_SOCKET #include <sys/socket.h> diff --git a/ovs/lib/util.c b/ovs/lib/util.c index 1195c79..3542f48 100644 --- a/ovs/lib/util.c +++ b/ovs/lib/util.c @@ -591,10 +591,6 @@ ovs_set_program_name(const char *argv0, const char *version) size_t max_len = strlen(argv0) + 1; SetErrorMode(GetErrorMode() | SEM_NOGPFAULTERRORBOX); -#if _MSC_VER < 1900 - /* This function is deprecated from 1900 (Visual Studio 2015) */ - _set_output_format(_TWO_DIGIT_EXPONENT); -#endif basename = xmalloc(max_len); _splitpath_s(argv0, NULL, 0, NULL, 0, basename, max_len, NULL, 0); diff --git a/ovs/lib/vlog.c b/ovs/lib/vlog.c index 533f937..00d5d92 100644 --- a/ovs/lib/vlog.c +++ b/ovs/lib/vlog.c @@ -215,9 +215,16 @@ vlog_get_destination_val(const char *name) void vlog_insert_module(struct ovs_list *vlog) { +/* There is a bug here related to linking pthread. + * I think it's related to WHOLEARCHIVE option. + * */ +#ifndef _WIN32 ovs_mutex_lock(&log_file_mutex); +#endif ovs_list_insert(&vlog_modules, vlog); +#ifndef _WIN32 ovs_mutex_unlock(&log_file_mutex); +#endif } /* Returns the name for logging module 'module'. */ diff --git a/ovs/meson.build b/ovs/meson.build index 72d20f2..9d57716 100644 --- a/ovs/meson.build +++ b/ovs/meson.build @@ -9,7 +9,7 @@ project('openvswitch', 'C', version: '2.16.90', license: 'Apache2.0', - default_options: ['buildtype=release', 'default_library=static'], + default_options: ['buildtype=debug', 'default_library=static'], meson_version: '>= 0.59.0' ) @@ -25,6 +25,8 @@ conf_soversion = configuration_data() conf_soversion.set('SOVERSION', as_soversion) cc = meson.get_compiler('c') +enable_dpdk = false +global_inc = [] prog_python = import('python').find_installation('python3') python_env = environment() @@ -42,15 +44,47 @@ if build_machine.system() == 'windows' if get_option('with-pthread') == 'optval' error('"with-pthread" is a required option on Windows') endif + + # suppress all warnings + # add_global_arguments('-w', language : 'c') + + add_global_arguments('-U_MSC_VER', language : 'c') + add_global_arguments('-D__GNUC__', language : 'c') + # add_global_arguments('-std=c11', language : 'c') + add_global_arguments('-std=gnu99', language : 'c') + add_global_arguments('-D_TIMESPEC_DEFINED', language : 'c') + # add_global_arguments('-D_CRT_NO_TIME_T', language : 'c') + add_global_arguments('-D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS', language : 'c') + add_global_arguments('-D_CRT_SECURE_NO_DEPRECATE', language : 'c') + + + if get_option('with-dpdk') != 'optval' + dpdk_lib_path = get_option('with-dpdk') + message('DPDK path=', dpdk_lib_path) + cdata.set('DPDK_NETDEV', 1) + cdata.set('HAVE_RTE_CONFIG_H', 1) + dpdk_lib_deps = dependency('libdpdk', required : true, static : true) + enable_dpdk = true + add_global_arguments('-msse3', language : 'c') + add_global_arguments('-mssse3', language : 'c') + add_global_arguments('-U__AVX512F__', language : 'c') + add_project_link_arguments('-Wl,/FORCE:MULTIPLE', language: 'c') + add_project_link_arguments('-Wl,/WHOLEARCHIVE', language: 'c') + add_project_link_arguments('-lmincore', language: 'c') + add_project_link_arguments('-ladvapi32', '-lsetupapi', language: 'c') + add_project_link_arguments('-ldbghelp', language: 'c') + endif + win_pthread_path = get_option('with-pthread') thread_dep = cc.find_library( - 'pthreadVC3-w32', + 'pthreadVC3', dirs : [ # Some may place DLL and LIB files under 'lib' subfolder win_pthread_path, win_pthread_path + '/lib', ], required : true, + static: true, ) windows_libs = [ 'Ws2_32', # Resolve symbols for Winsock @@ -65,6 +99,19 @@ if build_machine.system() == 'windows' required : true, ) endforeach + + enable_mlx5 = false + if enable_mlx5 + res_lib = run_command(prog_python, '-c', 'import os; print(os.environ["DEVX_LIB_PATH"])') + devx_lib_dir = res_lib.stdout().strip() + mlx5devx_lib = cc.find_library('mlx5devx', dirs: devx_lib_dir, required: true, static : false) + windows_deps += mlx5devx_lib + + res_inc = run_command(prog_python, '-c', 'import os; print(os.environ["DEVX_INC_PATH"])') + devx_inc_dir = res_inc.stdout().strip() + global_inc += include_directories(devx_inc_dir) + endif + endif check_headers = [ @@ -236,12 +283,25 @@ subdir('include') cin_processing = configuration_data() cin_processing.set('BANNER', '/* -*- mode: c; buffer-read-only: t -*- */') # this a temporary hack and may break backwards compatability with autotools cin_processing.set('srcdir', meson.project_source_root()) -cin_processing.set_quoted('LOGDIR', '/var/log/openvswitch') -cin_processing.set_quoted('RUNDIR', '/var/run/openvswitch') -cin_processing.set_quoted('DBDIR', '/etc/openvswitch') -cin_processing.set_quoted('bindir', '/bin') -cin_processing.set_quoted('sysconfdir', '/etc') -cin_processing.set_quoted('pkgdatadir', '/usr/share/openvswitch') + + +if build_machine.system() == 'windows' + install_prefix = 'c:/openvswitch/usr/' + cin_processing.set_quoted('LOGDIR', 'c:/openvswitch/var/log/') + cin_processing.set_quoted('RUNDIR', 'c:/openvswitch/var/run/') + cin_processing.set_quoted('DBDIR', 'c:/openvswitch/etc/') + cin_processing.set_quoted('bindir', 'c:/openvswitch/bin') + cin_processing.set_quoted('sysconfdir', 'c:/openvswitch/etc') + cin_processing.set_quoted('pkgdatadir', 'c:/openvswitch/usr/share/') +else + install_prefix = '/usr/local/' + cin_processing.set_quoted('LOGDIR', '/var/log/openvswitch') + cin_processing.set_quoted('RUNDIR', '/var/run/openvswitch') + cin_processing.set_quoted('DBDIR', '/etc/openvswitch') + cin_processing.set_quoted('bindir', '/bin') + cin_processing.set_quoted('sysconfdir', '/etc') + cin_processing.set_quoted('pkgdatadir', '/usr/share/openvswitch') +endif subdir('python/ovs') @@ -257,14 +317,14 @@ foreach cin : c_in_files ) endforeach -global_inc = include_directories( +global_inc += include_directories( '.', 'include', 'lib', ) if build_machine.system() == 'windows' - global_inc = include_directories( + global_inc += include_directories( '.', 'include', 'lib', @@ -274,6 +334,12 @@ if build_machine.system() == 'windows' win_pthread_path, win_pthread_path + '/include', ) + if enable_dpdk + dpdk_lib_path = get_option('with-dpdk') + global_inc += include_directories( + dpdk_lib_path + '/include', + ) + endif endif global_libs = [] @@ -306,8 +372,6 @@ add_project_arguments( language : 'c', ) -install_prefix = '/usr/local/' - lib_ovsdb_server_idl_ovsidl = custom_target( 'ovsdb-server-idl.ovsidl', input : [ diff --git a/ovs/meson_options.txt b/ovs/meson_options.txt index 2631b26..e857de9 100644 --- a/ovs/meson_options.txt +++ b/ovs/meson_options.txt @@ -1,2 +1,4 @@ option('with-pthread', type : 'string', value : 'optval', description : 'Path to the POSIX threads API library for Windows') +option('with-dpdk', type : 'string', value : 'optval', + description : 'Path to DPDK library for Windows') diff --git a/ovs/ofproto/meson.build b/ovs/ofproto/meson.build index 4f3b3fe..f318660 100644 --- a/ovs/ofproto/meson.build +++ b/ovs/ofproto/meson.build @@ -44,6 +44,9 @@ sources += [ ] deps = [] +if enable_dpdk + deps += dpdk_lib_deps +endif mapfile = meson.project_build_root() + '/ofproto.map' vflag = '-Wl'.format(meson.current_source_dir(), mapfile) diff --git a/ovs/ovsdb/meson.build b/ovs/ovsdb/meson.build index 1f7299e..206708e 100644 --- a/ovs/ovsdb/meson.build +++ b/ovs/ovsdb/meson.build @@ -43,6 +43,9 @@ sources = files( ) deps = [] +if enable_dpdk + deps += dpdk_lib_deps +endif mapfile = meson.project_build_root() + '/ovsdb.map' vflag = '-Wl'.format(meson.current_source_dir(), mapfile) diff --git a/ovs/utilities/meson.build b/ovs/utilities/meson.build index 49c7200..f4e62e5 100644 --- a/ovs/utilities/meson.build +++ b/ovs/utilities/meson.build @@ -6,6 +6,9 @@ sources = files( ) deps = [] +if enable_dpdk + deps += dpdk_lib_deps +endif executable('ovs-appctl', sources, dependencies : deps, diff --git a/ovs/vswitchd/meson.build b/ovs/vswitchd/meson.build index b8e3298..b438009 100644 --- a/ovs/vswitchd/meson.build +++ b/ovs/vswitchd/meson.build @@ -16,9 +16,19 @@ sources += lib_vswitch_idl_h deps = [] + +if enable_dpdk + deps += dpdk_lib_deps +endif + if build_machine.system() == 'linux' deps += m_dep endif +if build_machine.system() == 'windows' + deps += windows_deps +# deps += mlx5devx_lib + message('vswitchd: add windows_deps') +endif executable('ovs-vswitchd', sources, dependencies : deps, diff --git a/ovs/vtep/meson.build b/ovs/vtep/meson.build index abdae7e..7f9f3c1 100644 --- a/ovs/vtep/meson.build +++ b/ovs/vtep/meson.build @@ -40,6 +40,9 @@ sources = [ ] deps = [] +if enable_dpdk + deps += dpdk_lib_deps +endif mapfile = meson.project_build_root() + '/vtep.map' vflag = '-Wl'.format(meson.current_source_dir(), mapfile) -- 2.33.0.windows.2 _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
