On 6/8/2020 1:03 PM, Juraj Linkeš wrote: > > >> -----Original Message----- >> From: Ferruh Yigit <ferruh.yi...@intel.com> >> Sent: Monday, June 8, 2020 1:39 PM >> To: Juraj Linkeš <juraj.lin...@pantheon.tech>; Thomas Monjalon >> <tho...@monjalon.net> >> Cc: arybche...@solarflare.com; dev@dpdk.org; Honnappa Nagarahalli >> <honnappa.nagaraha...@arm.com> >> Subject: Re: [dpdk-dev] [PATCH] ethdev: fix dpdk gcc build on Arm >> >> On 6/8/2020 9:17 AM, Juraj Linkeš wrote: >>> >>> >>>> -----Original Message----- >>>> From: Ferruh Yigit <ferruh.yi...@intel.com> >>>> Sent: Thursday, June 4, 2020 3:30 PM >>>> To: Juraj Linkeš <juraj.lin...@pantheon.tech>; Thomas Monjalon >>>> <tho...@monjalon.net> >>>> Cc: arybche...@solarflare.com; dev@dpdk.org; Honnappa Nagarahalli >>>> <honnappa.nagaraha...@arm.com> >>>> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm >>>> >>>> On 6/4/2020 1:55 PM, Juraj Linkeš wrote: >>>>>> -----Original Message----- >>>>>> From: Ferruh Yigit <ferruh.yi...@intel.com> >>>>>> Sent: Thursday, June 4, 2020 2:23 PM >>>>>> To: Juraj Linkeš <juraj.lin...@pantheon.tech>; Thomas Monjalon >>>>>> <tho...@monjalon.net> >>>>>> Cc: arybche...@solarflare.com; dev@dpdk.org; Honnappa Nagarahalli >>>>>> <honnappa.nagaraha...@arm.com> >>>>>> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm >>>>>> >>>>>> On 6/4/2020 11:36 AM, Juraj Linkeš wrote: >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Ferruh Yigit <ferruh.yi...@intel.com> >>>>>>>> Sent: Wednesday, June 3, 2020 1:41 PM >>>>>>>> To: Thomas Monjalon <tho...@monjalon.net>; Juraj Linkeš >>>>>>>> <juraj.lin...@pantheon.tech> >>>>>>>> Cc: arybche...@solarflare.com; dev@dpdk.org >>>>>>>> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm >>>>>>>> >>>>>>>> On 6/3/2020 11:16 AM, Thomas Monjalon wrote: >>>>>>>>> 03/06/2020 11:48, Juraj Linkeš: >>>>>>>>>> Directive #include <file> in gcc implementation searches for >>>>>>>>>> files in a standard list of system directories, which leads to >>>>>>>>>> a sporadici build error on Taishan arm machines: >>>>>>>>>> /tmp/openvpp-testing/dpdk/lib/librte_ethdev/rte_ethdev.h:4287:10: >>>>>>>>>> fatal error: rte_ethdev_core.h: >>>>>>>>>> No such file or directory #include <rte_ethdev_core.h> >>>>>>>>> >>>>>>>>> Would be interesting to know why nobody else hit such error? >>>>>>>> >>>>>>>> >>>>>>>> I can't see why this is happening, in the 'mk/rte.lib.mk' we have >> following: >>>>>>>> >>>>>>>> " >>>>>>>> install: _preinstall build _postinstall >>>>>>>> build: _preinstall >>>>>>>> " >>>>>>>> >>>>>>>> Which should cause the library header files installed before >>>>>>>> building .c files in that library. >>>>>>>> So when compiling 'rte_class_eth.c', the header files should be >>>>>>>> already in install folder. >>>>>>>> >>>>>>>> >>>>>>>> I can see how/why changing to "" fixes the issue but I am not >>>>>>>> sure about this >>>>>> fix. >>>>>>>> "rte_ethdev.h" is a public header file, that applications will >>>>>>>> include it in their applications. In the public library it is >>>>>>>> more proper to have other includes from system folder, using format <>. >>>>>>>> Again, I can't see why it is failing but I believe we should find >>>>>>>> another solution for _internal_ build error. >>>>>>>> >>>>>>>> >>>>>>>> A very simple solution can be following, but that is also not >>>>>>>> good, since it solves the issue by creating a dependency to the >>>>>>>> order of the header >>>>>> includes: >>>>>>>> diff --git a/lib/librte_ethdev/rte_class_eth.c >>>>>>>> b/lib/librte_ethdev/rte_class_eth.c >>>>>>>> index 6338355e25..3030c49020 100644 >>>>>>>> --- a/lib/librte_ethdev/rte_class_eth.c >>>>>>>> +++ b/lib/librte_ethdev/rte_class_eth.c >>>>>>>> @@ -10,8 +10,8 @@ >>>>>>>> #include <rte_kvargs.h> >>>>>>>> #include <rte_log.h> >>>>>>>> >>>>>>>> -#include "rte_ethdev.h" >>>>>>>> #include "rte_ethdev_core.h" >>>>>>>> +#include "rte_ethdev.h" >>>>>>>> #include "rte_ethdev_driver.h" >>>>>>>> #include "ethdev_private.h" >>>>>>>> >>>>>>> >>>>>>> Thomas, Ferruh, what should be the solution? I'm not an expert on >>>>>>> this and I >>>>>> can't really offer anything better, but I'd like that this gets fixed. >>>>>> >>>>>> First we need to root cause this before trying to solve it. >>>>>> Honnappa also said he can reproduce this but our CI builds can't >>>>>> (we are talking about tens of builds daily on various platforms), >>>>>> need to understand >>>> why. >>>>>> Also from Makefile I can't see how this is happening, I am feeling >>>>>> uneasy to fix something before figuring out how/why it is failing. >>>>>> >>>>>> Can you please try to collect more data on when/how this happens, >>>>>> initial questions I can think of: >>>>>> - Can you reproduce this with meson build? >>>>>> - Is it bare DPDK build, or build part of other project (I guess I >>>>>> saw fd.io on the >>>>>> link) >>>>>> - - If this is not bare DPDK build what changes has been done to build >> system? >>>>>> - Do you see this with fresh build (new clone) or rebuild of existing >>>>>> clone? >>>>>> - Can you confirm you have correct RTE_SDK and RTE_TARGET >>>>>> environment variables? >>>>>> - Can you please share your build command? >>>>>> >>>>> >>>>> I sent an e-mail to dpdk dev a few days back asking for help where I >>>>> outlined >>>> what we're doing: >>>>> We're not doing anything special, just downloading and extracting >>>>> the archive, >>>> then setting CONFIG_RTE_LIBRTE_MLX5_PMD and >>>> CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC to y in config/common_base >> and >>>> then running make install T=arm64-armv8a-linuxapp-gcc -j. As >>>> mentioned in the subject, the build server is a Taishan ARM server. >>>>> >>>>> We're doing a fresh rebuild everytime. The error doesn't happen >>>>> everytime, >>>> just sometimes - it seems to be random. >>>>> >>>>> We don't set RTE_SDK nor RTE_TARGET since >>>> https://doc.dpdk.org/guides/linux_gsg/build_dpdk.html#installation-of >>>> -dpdk- target-environment-using-make doesn't mention those. >>>> >>>> Can you please try with "V=1", >>>> "make install T=arm64-armv8a-linuxapp-gcc -j V=1", this will generate >>>> verbose output. >>>> >>>> And _before_ lines compiling the 'rte_class_eth.c' you should see >>>> something like following lines, can you please check: >>>> " >>>> SYMLINK-FILE include/rte_ethdev_core.h >>>> >>>> ln -nsf `.../buildtools/relpath.sh >>>> .../lib/librte_ethdev/rte_ethdev_core.h >>>> .../x86_64-native-linux-gcc/include` >>>> .../x86_64-native-linux-gcc/include >>>> " >>>> >>> >>> This is the log with V=1 build (a failed build): >>> == Build lib/librte_ethdev >>> == Build lib/librte_cmdline >>> ar crDs librte_lpm.a rte_lpm.o rte_lpm6.o >>> INSTALL-LIB librte_lpm.a >>> ar crDs librte_member.a rte_member.o rte_member_ht.o rte_member_vbf.o >>> INSTALL-LIB librte_member.a >>> ar crDs librte_cryptodev.a rte_cryptodev.o rte_cryptodev_pmd.o >>> INSTALL-LIB librte_cryptodev.a >>> ar crDs librte_bbdev.a rte_bbdev.o >>> ar crDs librte_compressdev.a rte_compressdev.o rte_compressdev_pmd.o >> rte_comp.o >>> INSTALL-LIB librte_compressdev.a >>> INSTALL-LIB librte_bbdev.a >>> SYMLINK-FILE include/rte_ethdev_pci.h >>> SYMLINK-FILE include/rte_tm.h >>> SYMLINK-FILE include/rte_tm_driver.h >>> SYMLINK-FILE include/rte_eth_ctrl.h >>> SYMLINK-FILE include/rte_flow_driver.h >>> SYMLINK-FILE include/rte_ethdev_vdev.h >>> SYMLINK-FILE include/rte_mtr_driver.h >>> SYMLINK-FILE include/rte_ethdev_core.h >>> SYMLINK-FILE include/rte_flow.h >>> SYMLINK-FILE include/rte_mtr.h >>> SYMLINK-FILE include/rte_ethdev_driver.h >>> SYMLINK-FILE include/rte_ethdev.h >>> SYMLINK-FILE include/rte_dev_info.h >>> gcc -Wp,-MD,./.ethdev_private.o.d.tmp -pthread -I/tmp/dpdk- >> 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - >> DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - >> I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - >> DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- >> linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - >> DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- >> prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - >> Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- >> security -Wundef -Wwrite-strings -Wdeprecated >> -Wno-missing-field-initializers - >> Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- >> member -o ethdev_private.o -c /tmp/dpdk- >> 20.02/lib/librte_ethdev/ethdev_private.c >>> gcc -Wp,-MD,./.rte_ethdev.o.d.tmp -pthread -I/tmp/dpdk- >> 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - >> DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - >> I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - >> DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- >> linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - >> DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- >> prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - >> Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- >> security -Wundef -Wwrite-strings -Wdeprecated >> -Wno-missing-field-initializers - >> Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- >> member -o rte_ethdev.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev.c >>> gcc -Wp,-MD,./.rte_class_eth.o.d.tmp -pthread -I/tmp/dpdk- >> 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - >> DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - >> I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - >> DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- >> linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - >> DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- >> prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - >> Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- >> security -Wundef -Wwrite-strings -Wdeprecated >> -Wno-missing-field-initializers - >> Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- >> member -o rte_class_eth.o -c /tmp/dpdk- >> 20.02/lib/librte_ethdev/rte_class_eth.c >>> gcc -Wp,-MD,./.rte_flow.o.d.tmp -pthread -I/tmp/dpdk- >> 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - >> DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - >> I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - >> DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- >> linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - >> DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- >> prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - >> Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- >> security -Wundef -Wwrite-strings -Wdeprecated >> -Wno-missing-field-initializers - >> Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- >> member -o rte_flow.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_flow.c >>> gcc -Wp,-MD,./.rte_tm.o.d.tmp -pthread -I/tmp/dpdk- >> 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - >> DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - >> I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - >> DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- >> linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - >> DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- >> prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - >> Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- >> security -Wundef -Wwrite-strings -Wdeprecated >> -Wno-missing-field-initializers - >> Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- >> member -o rte_tm.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_tm.c >>> SYMLINK-FILE include/cmdline.h >>> SYMLINK-FILE include/cmdline_parse.h >>> SYMLINK-FILE include/cmdline_parse_num.h >>> gcc -Wp,-MD,./.ethdev_profile.o.d.tmp -pthread -I/tmp/dpdk- >> 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - >> DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - >> I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - >> DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- >> linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - >> DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- >> prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - >> Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- >> security -Wundef -Wwrite-strings -Wdeprecated >> -Wno-missing-field-initializers - >> Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- >> member -o ethdev_profile.o -c /tmp/dpdk- >> 20.02/lib/librte_ethdev/ethdev_profile.c >>> gcc -Wp,-MD,./.rte_mtr.o.d.tmp -pthread -I/tmp/dpdk- >> 20.02/lib/librte_eal/linux/eal/include -march=armv8-a+crc - >> DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32 - >> I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include - >> DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a- >> linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE - >> DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing- >> prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith - >> Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat- >> security -Wundef -Wwrite-strings -Wdeprecated >> -Wno-missing-field-initializers - >> Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed- >> member -o rte_mtr.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_mtr.c >>> SYMLINK-FILE include/rte_sched.h >>> SYMLINK-FILE include/rte_red.h >>> SYMLINK-FILE include/rte_approx.h >>> In file included from /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev.c:43: >>> /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev_driver.h:18:10: fatal >>> error: rte_ethdev.h: No such file or directory #include <rte_ethdev.h> >>> ^~~~~~~~~~~~~~ >>> compilation terminated. >>> /tmp/dpdk-20.02/mk/internal/rte.compile-pre.mk:114: recipe for target >>> 'rte_ethdev.o' failed >>> >>>> Also for the failed case, after fail can you please check if >>>> a) >>>> '.../arm64-armv8a-linuxapp-gcc/build/lib/librte_ethdev/_preinstall' >>>> file exists >>>> b) '.../x86_64-native-linux-gcc/include/rte_ethdev_core.h' exists and >>>> it linked to correct file '../../lib/librte_ethdev/rte_ethdev_core.h' >>>> >>> >>> a) it doesn't exist >>> b) all of these exist: >>> rte_ethdev_core.h -> ../../lib/librte_ethdev/rte_ethdev_core.h >>> rte_ethdev_driver.h -> ../../lib/librte_ethdev/rte_ethdev_driver.h >>> rte_ethdev.h -> ../../lib/librte_ethdev/rte_ethdev.h >>> rte_ethdev_pci.h -> ../../lib/librte_ethdev/rte_ethdev_pci.h >>> rte_ethdev_vdev.h -> ../../lib/librte_ethdev/rte_ethdev_vdev.h >>> >>> I was able to reproduce the failure after 7 successful builds. >> >> Thanks for details. >> >> This time it seems build complaining about not able to find "rte_ethdev.h", >> so >> your patch won't fix this case and build still will fail, looks like we need >> different >> fix. >> > > Right, Thomas already pointed this out - I haven't submitted a new patch > version since it didn't make much sense before we hash this all out. > >> And in the build command for 'rte_ethdev.c', as you can see there is '- >> I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include' paramter, and you >> mentioned above "rte_ethdev.h -> ../../lib/librte_ethdev/rte_ethdev.h" link >> exist >> under that filder, so still not clear why compiler can't find the files. >> >> There is a dependency of build for the '_preinstall' file, so buildling .c >> files in that >> library shouldn't start before all header files are installed and >> '_preinstall' file is >> created. This is also not clear how '_preinstall' can be missing. >> >> Can you please double check above links are valid, I mean you can access to >> the >> files from links? >> > > The links are pointing to existing files: > testuser@s17-t33-sut1:/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include$ ll > rte_ethdev.h > lrwxrwxrwx 1 testuser testuser 36 Jun 5 11:01 rte_ethdev.h -> > ../../lib/librte_ethdev/rte_ethdev.h > testuser@s17-t33-sut1:/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include$ ll > ../../lib/librte_ethdev/rte_ethdev.h > -rw-rw-r-- 1 testuser testuser 171636 Feb 25 15:46 > ../../lib/librte_ethdev/rte_ethdev.h > > This is a complete mystery to me. Maybe there are some implementation details > in the arm version of gcc or make?
I don't see the lines as following, for symbolic link, did you strip them or not gettim them at all: ln -nsf `/tmp/build/dpdk-20.05/buildtools/relpath.sh ..... > >>> >>>> >>>>> >>>>> I'll try Meson build a few times. How can I enable those two config >>>>> options in >>>> Meson? >>>>> >>>>>> Thanks, >>>>>> ferruh >>>>>> >>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Fix this by replacing the directive with #include "file" which >>>>>>>>>> searches for files first in the directory containing the >>>>>>>>>> current file and other directories only after that. >>>>>>>>>> >>>>>>>>>> Signed-off-by: Juraj Linkeš <juraj.lin...@pantheon.tech> >>>>>>>>> >>>>>>>>> What about #include <rte_ethdev_trace.h> ? >>>>>>>>> >>>>>>>>> Except this, >>>>>>>>> Acked-by: Thomas Monjalon <tho...@monjalon.net> >>>>>>>>> >>>>>>>>> >>>>>>>>> I would change the title to be more generic (nothing specific to >>>>>>>>> gcc or Arm >>>>>>>> here): >>>>>>>>> ethdev: fix local include >>>>>>>>> >>>>>>>>> It needs to be backported, so these lines are required: >>>>>>>>> Fixes: 7f0a669e7b04 ("ethdev: add allocation helper for virtual >> drivers") >>>>>>>>> Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers") >>>>>>>>> Fixes: ffc905f3b856 ("ethdev: separate driver APIs") >>>>>>>>> Fixes: 331c447ad913 ("ethdev: separate internal structures into >>>>>>>>> own >>>>>>>> header") >>>>>>>>> Cc: sta...@dpdk.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> >> >