> -----Original Message----- > From: Ferruh Yigit <ferruh.yi...@intel.com> > Sent: Monday, June 8, 2020 1:39 PM > To: Juraj Linkeš <juraj.lin...@pantheon.tech>; Thomas Monjalon > <tho...@monjalon.net> > Cc: arybche...@solarflare.com; dev@dpdk.org; Honnappa Nagarahalli > <honnappa.nagaraha...@arm.com> > Subject: Re: [dpdk-dev] [PATCH] ethdev: fix dpdk gcc build on Arm > > On 6/8/2020 9:17 AM, Juraj Linkeš wrote: > > > > > >> -----Original Message----- > >> From: Ferruh Yigit <ferruh.yi...@intel.com> > >> Sent: Thursday, June 4, 2020 3:30 PM > >> To: Juraj Linkeš <juraj.lin...@pantheon.tech>; Thomas Monjalon > >> <tho...@monjalon.net> > >> Cc: arybche...@solarflare.com; dev@dpdk.org; Honnappa Nagarahalli > >> <honnappa.nagaraha...@arm.com> > >> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm > >> > >> On 6/4/2020 1:55 PM, Juraj Linkeš wrote: > >>>> -----Original Message----- > >>>> From: Ferruh Yigit <ferruh.yi...@intel.com> > >>>> Sent: Thursday, June 4, 2020 2:23 PM > >>>> To: Juraj Linkeš <juraj.lin...@pantheon.tech>; Thomas Monjalon > >>>> <tho...@monjalon.net> > >>>> Cc: arybche...@solarflare.com; dev@dpdk.org; Honnappa Nagarahalli > >>>> <honnappa.nagaraha...@arm.com> > >>>> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm > >>>> > >>>> On 6/4/2020 11:36 AM, Juraj Linkeš wrote: > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Ferruh Yigit <ferruh.yi...@intel.com> > >>>>>> Sent: Wednesday, June 3, 2020 1:41 PM > >>>>>> To: Thomas Monjalon <tho...@monjalon.net>; Juraj Linkeš > >>>>>> <juraj.lin...@pantheon.tech> > >>>>>> Cc: arybche...@solarflare.com; dev@dpdk.org > >>>>>> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm > >>>>>> > >>>>>> On 6/3/2020 11:16 AM, Thomas Monjalon wrote: > >>>>>>> 03/06/2020 11:48, Juraj Linkeš: > >>>>>>>> Directive #include <file> in gcc implementation searches for > >>>>>>>> files in a standard list of system directories, which leads to > >>>>>>>> a sporadici build error on Taishan arm machines: > >>>>>>>> /tmp/openvpp-testing/dpdk/lib/librte_ethdev/rte_ethdev.h:4287:10: > >>>>>>>> fatal error: rte_ethdev_core.h: > >>>>>>>> No such file or directory #include <rte_ethdev_core.h> > >>>>>>> > >>>>>>> Would be interesting to know why nobody else hit such error? > >>>>>> > >>>>>> > >>>>>> I can't see why this is happening, in the 'mk/rte.lib.mk' we have > following: > >>>>>> > >>>>>> " > >>>>>> install: _preinstall build _postinstall > >>>>>> build: _preinstall > >>>>>> " > >>>>>> > >>>>>> Which should cause the library header files installed before > >>>>>> building .c files in that library. > >>>>>> So when compiling 'rte_class_eth.c', the header files should be > >>>>>> already in install folder. > >>>>>> > >>>>>> > >>>>>> I can see how/why changing to "" fixes the issue but I am not > >>>>>> sure about this > >>>> fix. > >>>>>> "rte_ethdev.h" is a public header file, that applications will > >>>>>> include it in their applications. In the public library it is > >>>>>> more proper to have other includes from system folder, using format <>. > >>>>>> Again, I can't see why it is failing but I believe we should find > >>>>>> another solution for _internal_ build error. > >>>>>> > >>>>>> > >>>>>> A very simple solution can be following, but that is also not > >>>>>> good, since it solves the issue by creating a dependency to the > >>>>>> order of the header > >>>> includes: > >>>>>> diff --git a/lib/librte_ethdev/rte_class_eth.c > >>>>>> b/lib/librte_ethdev/rte_class_eth.c > >>>>>> index 6338355e25..3030c49020 100644 > >>>>>> --- a/lib/librte_ethdev/rte_class_eth.c > >>>>>> +++ b/lib/librte_ethdev/rte_class_eth.c > >>>>>> @@ -10,8 +10,8 @@ > >>>>>> #include <rte_kvargs.h> > >>>>>> #include <rte_log.h> > >>>>>> > >>>>>> -#include "rte_ethdev.h" > >>>>>> #include "rte_ethdev_core.h" > >>>>>> +#include "rte_ethdev.h" > >>>>>> #include "rte_ethdev_driver.h" > >>>>>> #include "ethdev_private.h" > >>>>>> > >>>>> > >>>>> Thomas, Ferruh, what should be the solution? I'm not an expert on > >>>>> this and I > >>>> can't really offer anything better, but I'd like that this gets fixed. > >>>> > >>>> First we need to root cause this before trying to solve it. > >>>> Honnappa also said he can reproduce this but our CI builds can't > >>>> (we are talking about tens of builds daily on various platforms), > >>>> need to understand > >> why. > >>>> Also from Makefile I can't see how this is happening, I am feeling > >>>> uneasy to fix something before figuring out how/why it is failing. > >>>> > >>>> Can you please try to collect more data on when/how this happens, > >>>> initial questions I can think of: > >>>> - Can you reproduce this with meson build? > >>>> - Is it bare DPDK build, or build part of other project (I guess I > >>>> saw fd.io on the > >>>> link) > >>>> - - If this is not bare DPDK build what changes has been done to build > system? > >>>> - Do you see this with fresh build (new clone) or rebuild of existing > >>>> clone? > >>>> - Can you confirm you have correct RTE_SDK and RTE_TARGET > >>>> environment variables? > >>>> - Can you please share your build command? > >>>> > >>> > >>> I sent an e-mail to dpdk dev a few days back asking for help where I > >>> outlined > >> what we're doing: > >>> We're not doing anything special, just downloading and extracting > >>> the archive, > >> then setting CONFIG_RTE_LIBRTE_MLX5_PMD and > >> CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC to y in config/common_base > and > >> then running make install T=arm64-armv8a-linuxapp-gcc -j. As > >> mentioned in the subject, the build server is a Taishan ARM server. > >>> > >>> We're doing a fresh rebuild everytime. The error doesn't happen > >>> everytime, > >> just sometimes - it seems to be random. > >>> > >>> We don't set RTE_SDK nor RTE_TARGET since > >> https://doc.dpdk.org/guides/linux_gsg/build_dpdk.html#installation-of > >> -dpdk- target-environment-using-make doesn't mention those. > >> > >> Can you please try with "V=1", > >> "make install T=arm64-armv8a-linuxapp-gcc -j V=1", this will generate > >> verbose output. > >> > >> And _before_ lines compiling the 'rte_class_eth.c' you should see > >> something like following lines, can you please check: > >> " > >> SYMLINK-FILE include/rte_ethdev_core.h > >> > >> ln -nsf `.../buildtools/relpath.sh > >> .../lib/librte_ethdev/rte_ethdev_core.h > >> .../x86_64-native-linux-gcc/include` > >> .../x86_64-native-linux-gcc/include > >> " > >> > > > > This is the log with V=1 build (a failed build): > > == Build lib/librte_ethdev > > == Build lib/librte_cmdline > > ar crDs librte_lpm.a rte_lpm.o rte_lpm6.o > > INSTALL-LIB librte_lpm.a > > ar crDs librte_member.a rte_member.o rte_member_ht.o rte_member_vbf.o > > INSTALL-LIB librte_member.a > > ar crDs librte_cryptodev.a rte_cryptodev.o rte_cryptodev_pmd.o > > INSTALL-LIB librte_cryptodev.a > > ar crDs librte_bbdev.a rte_bbdev.o > > ar crDs librte_compressdev.a rte_compressdev.o rte_compressdev_pmd.o > rte_comp.o > > INSTALL-LIB librte_compressdev.a > > INSTALL-LIB librte_bbdev.a > > SYMLINK-FILE include/rte_ethdev_pci.h > > SYMLINK-FILE include/rte_tm.h > > SYMLINK-FILE include/rte_tm_driver.h > > SYMLINK-FILE include/rte_eth_ctrl.h > > SYMLINK-FILE include/rte_flow_driver.h > > SYMLINK-FILE include/rte_ethdev_vdev.h > > SYMLINK-FILE include/rte_mtr_driver.h > > SYMLINK-FILE include/rte_ethdev_core.h > > SYMLINK-FILE include/rte_flow.h > > SYMLINK-FILE include/rte_mtr.h > > SYMLINK-FILE include/rte_ethdev_driver.h > > SYMLINK-FILE include/rte_ethdev.h > > SYMLINK-FILE include/rte_dev_info.h > > gcc -Wp,-MD,./.ethdev_private.o.d.tmp -pthread -I/tmp/dpdk- > 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - > DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - > I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - > DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- > linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - > DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- > prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - > Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- > security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers > - > Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- > member -o ethdev_private.o -c /tmp/dpdk- > 20.02/lib/librte_ethdev/ethdev_private.c > > gcc -Wp,-MD,./.rte_ethdev.o.d.tmp -pthread -I/tmp/dpdk- > 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - > DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - > I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - > DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- > linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - > DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- > prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - > Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- > security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers > - > Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- > member -o rte_ethdev.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev.c > > gcc -Wp,-MD,./.rte_class_eth.o.d.tmp -pthread -I/tmp/dpdk- > 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - > DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - > I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - > DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- > linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - > DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- > prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - > Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- > security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers > - > Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- > member -o rte_class_eth.o -c /tmp/dpdk- > 20.02/lib/librte_ethdev/rte_class_eth.c > > gcc -Wp,-MD,./.rte_flow.o.d.tmp -pthread -I/tmp/dpdk- > 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - > DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - > I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - > DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- > linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - > DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- > prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - > Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- > security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers > - > Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- > member -o rte_flow.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_flow.c > > gcc -Wp,-MD,./.rte_tm.o.d.tmp -pthread -I/tmp/dpdk- > 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - > DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - > I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - > DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- > linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - > DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- > prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - > Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- > security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers > - > Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- > member -o rte_tm.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_tm.c > > SYMLINK-FILE include/cmdline.h > > SYMLINK-FILE include/cmdline_parse.h > > SYMLINK-FILE include/cmdline_parse_num.h > > gcc -Wp,-MD,./.ethdev_profile.o.d.tmp -pthread -I/tmp/dpdk- > 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - > DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - > I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - > DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- > linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - > DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- > prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - > Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- > security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers > - > Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- > member -o ethdev_profile.o -c /tmp/dpdk- > 20.02/lib/librte_ethdev/ethdev_profile.c > > gcc -Wp,-MD,./.rte_mtr.o.d.tmp -pthread -I/tmp/dpdk- > 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - > DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - > I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - > DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- > linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - > DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- > prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - > Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- > security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers > - > Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- > member -o rte_mtr.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_mtr.c > > SYMLINK-FILE include/rte_sched.h > > SYMLINK-FILE include/rte_red.h > > SYMLINK-FILE include/rte_approx.h > > In file included from /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev.c:43: > > /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev_driver.h:18:10: fatal > > error: rte_ethdev.h: No such file or directory #include <rte_ethdev.h> > > ^~~~~~~~~~~~~~ > > compilation terminated. > > /tmp/dpdk-20.02/mk/internal/rte.compile-pre.mk:114: recipe for target > > 'rte_ethdev.o' failed > > > >> Also for the failed case, after fail can you please check if > >> a) > >> '.../arm64-armv8a-linuxapp-gcc/build/lib/librte_ethdev/_preinstall' > >> file exists > >> b) '.../x86_64-native-linux-gcc/include/rte_ethdev_core.h' exists and > >> it linked to correct file '../../lib/librte_ethdev/rte_ethdev_core.h' > >> > > > > a) it doesn't exist > > b) all of these exist: > > rte_ethdev_core.h -> ../../lib/librte_ethdev/rte_ethdev_core.h > > rte_ethdev_driver.h -> ../../lib/librte_ethdev/rte_ethdev_driver.h > > rte_ethdev.h -> ../../lib/librte_ethdev/rte_ethdev.h > > rte_ethdev_pci.h -> ../../lib/librte_ethdev/rte_ethdev_pci.h > > rte_ethdev_vdev.h -> ../../lib/librte_ethdev/rte_ethdev_vdev.h > > > > I was able to reproduce the failure after 7 successful builds. > > Thanks for details. > > This time it seems build complaining about not able to find "rte_ethdev.h", so > your patch won't fix this case and build still will fail, looks like we need > different > fix. >
Right, Thomas already pointed this out - I haven't submitted a new patch version since it didn't make much sense before we hash this all out. > And in the build command for 'rte_ethdev.c', as you can see there is '- > I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include' paramter, and you > mentioned above "rte_ethdev.h -> ../../lib/librte_ethdev/rte_ethdev.h" link > exist > under that filder, so still not clear why compiler can't find the files. > > There is a dependency of build for the '_preinstall' file, so buildling .c > files in that > library shouldn't start before all header files are installed and > '_preinstall' file is > created. This is also not clear how '_preinstall' can be missing. > > Can you please double check above links are valid, I mean you can access to > the > files from links? > The links are pointing to existing files: testuser@s17-t33-sut1:/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include$ ll rte_ethdev.h lrwxrwxrwx 1 testuser testuser 36 Jun 5 11:01 rte_ethdev.h -> ../../lib/librte_ethdev/rte_ethdev.h testuser@s17-t33-sut1:/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include$ ll ../../lib/librte_ethdev/rte_ethdev.h -rw-rw-r-- 1 testuser testuser 171636 Feb 25 15:46 ../../lib/librte_ethdev/rte_ethdev.h This is a complete mystery to me. Maybe there are some implementation details in the arm version of gcc or make? > > > >> > >>> > >>> I'll try Meson build a few times. How can I enable those two config > >>> options in > >> Meson? > >>> > >>>> Thanks, > >>>> ferruh > >>>> > >>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Fix this by replacing the directive with #include "file" which > >>>>>>>> searches for files first in the directory containing the > >>>>>>>> current file and other directories only after that. > >>>>>>>> > >>>>>>>> Signed-off-by: Juraj Linkeš <juraj.lin...@pantheon.tech> > >>>>>>> > >>>>>>> What about #include <rte_ethdev_trace.h> ? > >>>>>>> > >>>>>>> Except this, > >>>>>>> Acked-by: Thomas Monjalon <tho...@monjalon.net> > >>>>>>> > >>>>>>> > >>>>>>> I would change the title to be more generic (nothing specific to > >>>>>>> gcc or Arm > >>>>>> here): > >>>>>>> ethdev: fix local include > >>>>>>> > >>>>>>> It needs to be backported, so these lines are required: > >>>>>>> Fixes: 7f0a669e7b04 ("ethdev: add allocation helper for virtual > drivers") > >>>>>>> Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers") > >>>>>>> Fixes: ffc905f3b856 ("ethdev: separate driver APIs") > >>>>>>> Fixes: 331c447ad913 ("ethdev: separate internal structures into > >>>>>>> own > >>>>>> header") > >>>>>>> Cc: sta...@dpdk.org > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>> > > >