On Wed, Jun 8, 2022 at 5:10 PM Van Haaren, Harry <harry.van.haa...@intel.com> wrote: > > Hi Li, Sunil and OVS Discuss mailing list, > > Answering the direction question inline: > > Any ideas? -DHAVE_AVX512F and -DHAVE_LD_AVX512_GOOD are always enabled. > They are always enabled *assuming* your binutils version does not have bugs. > > Why are you trying to disable AVX512? It will not run by default, what is the > end-goal here? > Below a large amount of detail around CPU ISA enabling, and how it > technically all works & why :) >
Thanks a lot for your clarification. Recently, I encountered a bug. I found that ovs crashes when executing avx instructions. It's really like this bug: https://inbox.dpdk.org/dev/31482910.FCGgztJ3Sx@xps/T/#m95492563f7e2819395e00e56d18f19e9911d2370 I am not quite sure if avx512 in ovs should be disabled or not. I want to make sure that ovs/dpdk disables avx512. If it won't run by default, I think I can leave it alone. > Hope the information here is helpful! Regards, -Harry > > > # Details around CPU ISA enabling > Let me try to explain the approach taken by both OVS and DPDK around AVX512 > compiling and running. > There are logically two "stages" to enabling AVX512 code, compile time and > runtime. It helps to treat them > individually, as they can occur on different machines (resulting in different > CPU flags being available, and > even "cross-compilations" where the resulting binary targets a different CPU > altogether!) > > # Compile Time; > At compile time, the OVS checks (using ./boot.sh and ./configure) a large > number of things: compiler versions, > compiler capabilities, CFLAGS, etc, to adjust the build parameters. It is > important to note for CPU ISA enabling > (e.g. AVX512, and others) that the CPU that does the configuring, and > compiling of OVS source code into the > binary, is *not* guaranteed to be the same as the one that *runs* the OVS > binaries in the end. > > As a result, the binary that is built must run for the CPU given by "-march" > in CFLAGS. This is often "corei7" or > "Nehalem" in practice, which enables ISA up to e.g. SSE4.2. This means that > AVX512 is not used by the compiler > in the *generic* scalar parts of OVS. > > Note that specific parts of the OVS codebase (DPIF, DPCLS, MiniflowExtract, > and Actions is WIP for 2.18 release) > are optimized using functions explicitly optimized for a specifically for the > AVX512 CPU ISA. These functions are > *compiled* by the compiler, and included in the binary (sometimes called a > "fat binary" as it includes multiple > versions of functions, which are selected based on CPU capabilities at > runtime). > > Very important; these "explicitly optimized" functions (using AVX512) are > *never* executed automatically. There > is *always* a runtime check to ensure that AVX512 is available on the runtime > CPU *before* any AVX512 instruction > executes. > > > # Runtime > At runtime, (on x86) the "CPUID" instruction is used to identify what ISA is > available. If it reports AVX512 is available, > then OVS is able to select the "explicitly optimized" functions from the "fat > binary". Note that at runtime, it is only > possible to execute code that was already compiled into the binary. As a > result, it is important to *always* include the > explicitly optimized functions at *compile time*, even if at compile-time the > CPU doesn't support the ISA! > > The technical method of enabling different functions at runtime is achieved > using "function pointers", a common concept > in e.g C or C++ codebases. DPDK uses it extensively e.g. at the ethdev layer > for different PMDs rx-routines, and even multiple > implementations of the *same* RX routine but optimized for different ISAs! > C++ also uses it (behind the scenes) for virtual > functions, and many other programming languages use them too (for context; > this is not "magic", its just not always visible > at the user-written code level.) > > > # OVS and DPDK together > Things become a little more complex, as there were specific bugs in a version > of binutils around assembling AVX512 code > years ago. This has been mitigated in OVS and DPDK, by checking a known "bad" > instruction sequence (on that assembler version) > and testing for the resulting invalid instruction sequence. As a result, both > DPDK and OVS are capable of *disabling* AVX512 > if the binutils version has this known bug. > > DPDK futher provides "-march" settings in its pkg-config file, as DPDK > requires SSE4.2 support as a baseline (on x86). This > setting is stripped from the pkg-config file in OVS's build configuration > stage, as OVS would like control over "-march" explicitly. > > Lastly, as DPDK and OVS could be built with *different* binutils versions (or > even on different compile machines!), both OVS > and DPDK check for the buggy-binutils themselves. As a result, it is possible > to have a DPDK/OVS combo with/without AVX512 > enabled. > > # Concerns around AVX512 enabling > Note that neither DPDK nor OVS enables AVX512 without user input. DPDK > defaults to AVX2/256 bit wide (ymm) registers, > and only uses AVX512 if the --force-max-simd-bitwidth=512 EAL argument is > passed: no reason for concern here. > > OVS will use scalar implementations, unless the appropriate "ovs-appctl > dpif-netdev/*" command is run to enable AVX512 > optimized routines (documentation here; > https://docs.openvswitch.org/en/latest/topics/dpdk/bridge/#datapath-classifier-performance) > > All in all, CPU ISA is compiled in by default, and not enabled. It can be > enabled manually by the user, and will then ensure that > the required CPU ISA is available on the runtime CPU, before actually > executing the optimized instruction sequence. As a result > binaries can happily contain AVX512 optimized code, it just won't be executed > unless the user requests it. > > > > -----Original Message----- > > From: Li Zhang <zhlci...@gmail.com> > > Sent: Wednesday, June 8, 2022 3:51 PM > > To: Pai G, Sunil <sunil.pa...@intel.com> > > Cc: ovs-discuss@openvswitch.org; Van Haaren, Harry > > <harry.van.haa...@intel.com> > > Subject: Re: [ovs-discuss] One question about AVX512 support > > > > Thanks a lot. > > > > Hi Harry, > > Any ideas? -DHAVE_AVX512F and -DHAVE_LD_AVX512_GOOD are always enabled. > > > > On Wed, Jun 8, 2022 at 3:15 PM Pai G, Sunil <sunil.pa...@intel.com> wrote: > > > > > > Adding Harry to help answer these questions on DPDK and OVS building with > > AVX512. > > > > > > Thanks and Regards, > > > Sunil > > > > > > > -----Original Message----- > > > > From: Li Zhang <zhlci...@gmail.com> > > > > Sent: Wednesday, June 8, 2022 4:26 PM > > > > To: Pai G, Sunil <sunil.pa...@intel.com> > > > > Cc: ovs-discuss@openvswitch.org > > > > Subject: Re: [ovs-discuss] One question about AVX512 support > > > > > > > > Hi Pai, > > > > > > > > I have been trying to disable avx512 in OVS for the platform which > > > > doesn't > > > > support avx512. > > > > Building fails and it seems that it is not disabled. Any idea about it? > > > > > > > > # ./configure --with-dpdk=yes --prefix=/usr --localstatedir=/var -- > > > > sysconfdir=/etc CFLAGS="-mno-avx512f" > > > > # make > > > > > > > > libtool: compile: gcc -DHAVE_CONFIG_H -I. -I ./include -I ./include -I > > > > ./lib -I ./lib -Wstrict-prototypes -Wall -Wextra -Wno-sign-compare - > > > > Wpointer-arith -Wformat -Wformat-security -Wswitch-enum -Wunused- > > parameter > > > > -Wbad-function-cast -Wcast-align -Wstrict-prototypes -Wold-style- > > > > definition -Wmissing-prototypes -Wmissing-field-initializers > > > > -fno-strict- > > > > aliasing -Wswitch-bool -Wlogical-not-parentheses > > > > -Wsizeof-array-argument - > > > > Wbool-compare -Wshift-negative-value -Wduplicated-cond -Wshadow -mssse3 > > > > - > > > > include rte_config.h -DOPENSSL_LOAD_CONF -I/usr/local/include > > > > -D_FILE_OFFSET_BITS=64 -mno-avx512f -DHAVE_AVX512F - > > DHAVE_LD_AVX512_GOOD - > > > > MT lib/netdev-dpdk.lo -MD -MP -MF lib/.deps/netdev-dpdk.Tpo -c > > > > lib/netdev- > > > > dpdk.c -o lib/netdev-dpdk.o In file included from /usr/lib64/gcc/x86_64- > > > > suse-linux/7/include/immintrin.h:41:0, > > > > from /usr/lib64/gcc/x86_64-suse- > > > > linux/7/include/x86intrin.h:48, > > > > from /usr/local/include/rte_vect.h:28, > > > > from /usr/local/include/rte_memcpy.h:17, > > > > from /usr/local/include/rte_ether.h:21, > > > > from /usr/local/include/rte_ethdev.h:159, > > > > from lib/netdev-dpdk.c:39: > > > > /usr/local/include/rte_memcpy.h: In function βrte_mov32β: > > > > /usr/lib64/gcc/x86_64-suse-linux/7/include/avxintrin.h:926:1: error: > > > > inlining failed in call to always_inline β_mm256_storeu_si256β: target > > > > specific option mismatch > > > > _mm256_storeu_si256 (__m256i_u *__P, __m256i __A) ^~~~~~~~~~~~~~~~~~~ > > In > > > > file included from /usr/local/include/rte_ether.h:21:0, > > > > from /usr/local/include/rte_ethdev.h:159, > > > > from lib/netdev-dpdk.c:39: > > > > /usr/local/include/rte_memcpy.h:320:2: note: called from here > > > > _mm256_storeu_si256((__m256i *)dst, ymm0); > > > > > > > > On Thu, Jun 2, 2022 at 4:00 PM Pai G, Sunil <sunil.pa...@intel.com> > > > > wrote: > > > > > > > > > > Hi Li, > > > > > > > > > > The assumption of ovs being dependent on dpdk for avx512 might not be > > > > true. > > > > > I found these two commits below in ovs-2.14.2 which strips out the "- > > > > march" and "-mno-avx512f" flags exported by dpdk i.e removes dependency > > > > on > > > > DPDK. The reason for this is rightly mentioned below as well. Hope this > > > > helps. > > > > > > > > > > > > > > > commit bb8f0e2a810889241f1d886d160ccee9b96c4d63 > > > > > Author: Ian Stokes <ian.sto...@intel.com> > > > > > Date: Fri Jan 15 15:46:02 2021 +0000 > > > > > > > > > > acinclude: Strip out -mno-avx512f provided by DPDK. > > > > > > > > > > DPDK forces '-mno-avx512f' flag for the application if the > > > > > toolchain > > > > > used to build DPDK had broken AVX512 support. > > > > > > > > > > DPDK forces '-mno-avx512f' flag for the application if the > > > > > toolchain > > > > > used to build DPDK had broken AVX512 support. But OVS could be > > > > built > > > > > with a completely different or fixed toolchain with correct avx512 > > > > > support. > > > > > > > > > > Fix that by stripping out `-mno-avx512f` as we already do for '- > > > > march'. > > > > > This will allow the OVS to decide if the AVX512 can be used. > > > > > > > > > > Reordering of CFLAGS (i.e. adding DPDK flags before OVS ones) is > > > > > not > > > > an > > > > > option since autotools might reorder them back later and it's very > > > > > unpredictable. > > > > > > > > > > Reported-at: https://github.com/openvswitch/ovs-issues/issues/201 > > > > > Signed-off-by: Ilya Maximets <i.maxim...@ovn.org> > > > > > Co-authored-by: Ilya Maximets <i.maxim...@ovn.org> > > > > > Signed-off-by: Ian Stokes <ian.sto...@intel.com> > > > > > > > > > > commit e9f9104d00006a83ce7efd702120171835991779 > > > > > Author: Ian Stokes <ian.sto...@intel.com> > > > > > Date: Fri Jan 15 14:54:04 2021 +0000 > > > > > > > > > > acinclude: Strip out -march provided by DPDK. > > > > > > > > > > DPDK flags may include -march. Forcing -march could be > > > > > considered too heavy a requirement when users compile OVS from > > > > > source and could override user provided options. > > > > > > > > > > Resolve this by stripping -march from provided DPDK flags. > > > > > > > > > > Signed-off-by: Ian Stokes <ian.sto...@intel.com> > > > > > > > > > > > > > > > > > > > > Thanks and Regards, > > > > > Sunil > > > > > > > > > > > -----Original Message----- > > > > > > From: discuss <ovs-discuss-boun...@openvswitch.org> On Behalf Of Li > > > > > > Zhang > > > > > > Sent: Thursday, June 2, 2022 6:35 PM > > > > > > To: ovs-discuss@openvswitch.org > > > > > > Subject: [ovs-discuss] One question about AVX512 support > > > > > > > > > > > > Hi all, > > > > > > > > > > > > We are using openvswitch 2.14.2, and dpdk-19.11.4. I found avx512 is > > > > > > enabled by default but it's disabled in DPDK. But I think ovs is > > > > > > dependent on the dpdk library, right? But why does ovs work with > > > > > > avx512 disabled in DPDK? > > > > > > > > > > > > I am not quite sure about the relationship between OVS and DPDK, any > > > > > > suggestions? > > > > > > > > > > > > -- > > > > > > > > > > > > Best Regards > > > > > > -Li > > > > > > _______________________________________________ > > > > > > discuss mailing list > > > > > > disc...@openvswitch.org > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > > > > > > > > > > > > > -- > > > > > > > > Best Regards > > > > -Li > > > > > > > > -- > > > > Best Regards > > -Li -- Best Regards -Li _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
