Lee, I don't have a multi numa system neither an AMD CPP device. I am not able to reproduce similar behavior. For instance, I can run a single worker with multiple vdev crypto devices and they all show whith the 'show dpdk crypto devices' command.
I would suggest to run a dpdk test app with a similar configuration to double check that everything is working there. On Mon, Feb 4, 2019 at 11:49 PM Roberts, Lee A. <lee.robe...@hpe.com> wrote: > ‘show dpdk crypto device’ shows just the one device. > > > > - Lee > > > > *From:* Sergio G.M. [mailto:sgmon...@gmail.com] > *Sent:* Monday, February 04, 2019 3:48 PM > *To:* Roberts, Lee A. <lee.robe...@hpe.com> > *Cc:* manuel.alo...@cavium.com; vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp > > > > When you run with a single worker and using multiple crypto devices, does > 'show dpdk crypto device' display all the devices or just the one being > used? > > > > On Mon, Feb 4, 2019 at 11:41 PM Roberts, Lee A. <lee.robe...@hpe.com> > wrote: > > Sergio, > > > > Yes, there are a pair of encryption controllers in each NUMA zone. > > I have tried configurations with all eight devices assigned to > uio_pci_generic, > > or with the configuration limited to four devices of one type or the other. > > > > - Lee > > > > > > [root@cl41 ~]# lspci -vv | egrep 'Encryption|NUMA' > > ... > > 03:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family > 17h (Models 00h-0fh) Platform Security Processor > > NUMA node: 0 > > ... > > 04:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device > 1468 > > NUMA node: 0 > > ... > > 21:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family > 17h (Models 00h-0fh) Platform Security Processor > > NUMA node: 1 > > ... > > 22:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device > 1468 > > NUMA node: 1 > > ... > > 42:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family > 17h (Models 00h-0fh) Platform Security Processor > > NUMA node: 2 > > ... > > 43:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device > 1468 > > NUMA node: 2 > > ... > > 62:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family > 17h (Models 00h-0fh) Platform Security Processor > > NUMA node: 3 > > ... > > 63:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device > 1468 > > NUMA node: 3 > > > > > > > > > > *From:* Sergio G.M. [mailto:sgmon...@gmail.com] > *Sent:* Monday, February 04, 2019 3:27 PM > *To:* Roberts, Lee A. <lee.robe...@hpe.com> > *Cc:* manuel.alo...@cavium.com; vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp > > > > I was assuming socket_id == numa, so not the issue there. > > > > Are the crypto devices also in each numa node? > > > > On Mon, Feb 4, 2019 at 11:22 PM Roberts, Lee A. <lee.robe...@hpe.com> > wrote: > > Sergio, > > > > The corelist-workers do translate to single cores in each of the NUMA > zones, > > as shown below: > > > > cpu { > > main-core 1 > > corelist-workers 2,6,10,14 > > } > > > > > > [root@cl41 ~]# numactl -H > > available: 4 nodes (0-3) > > node 0 cpus: 0 1 2 3 > > node 0 size: 32656 MB > > node 0 free: 12133 MB > > node 1 cpus: 4 5 6 7 > > node 1 size: 32767 MB > > node 1 free: 11632 MB > > node 2 cpus: 8 9 10 11 > > node 2 size: 32767 MB > > node 2 free: 13486 MB > > node 3 cpus: 12 13 14 15 > > node 3 size: 32767 MB > > node 3 free: 14229 MB > > node distances: > > node 0 1 2 3 > > 0: 10 16 16 16 > > 1: 16 10 16 16 > > 2: 16 16 10 16 > > 3: 16 16 16 10 > > [root@cl41 ~]# > > > > Might there be confusion between “NUMA node” and “socket_id” definitions? > > My configuration file assumes that “socket_id” ==> NUMA, but this is > actually > > a single-socket system with four NUMA zones. > > > > - Lee > > > > > > *From:* Sergio G.M. [mailto:sgmon...@gmail.com] > *Sent:* Monday, February 04, 2019 2:55 PM > *To:* Roberts, Lee A. <lee.robe...@hpe.com> > *Cc:* manuel.alo...@cavium.com; vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp > > > > Hi Lee, > > > > Looking at your startup.conf below, are you sure those parameters are > correct? are you running 4 numa node platform? > > If that is the case, you should also use cores from each numa node. > > > > vdev crypto_ccp0,socket_id=0 > > vdev crypto_ccp1,socket_id=1 > > vdev crypto_ccp2,socket_id=2 > > vdev crypto_ccp3,socket_id=3 > > > > I think the auto mapping is trying to map core and crypto devices from the > same numa node. > > > > On Fri, Feb 1, 2019 at 10:16 PM Roberts, Lee A. <lee.robe...@hpe.com> > wrote: > > Sergio, > > > > I’m having partial success. I can get a single “vdev crypto_ccp” to be > > recognized with a single worker core, but I’m not able to get multiple > > devices. > > > > In my /etc/vpp/startup.conf, I have the following: > > > > ... > > cpu { > > main-core 1 > > corelist-workers 2,6,10,14 > > } > > > > dpdk { > > socket-mem 2048,2048,2048,2048 > > dev default { > > num-rx-queues 2 > > num-tx-queues 2 > > num-tx-desc 2048 > > num-rx-desc 2048 > > } > > uio-driver uio-pci-generic > > dev 0000:41:00.0 > > dev 0000:41:00.1 > > vdev crypto_ccp0,socket_id=0 > > vdev crypto_ccp1,socket_id=1 > > vdev crypto_ccp2,socket_id=2 > > vdev crypto_ccp3,socket_id=3 > > } > > ... > > > > When I start VPP using this configuration, I see a message about “not > enough DPDK crypto resources, > > default to OpenSSL”: > > > > Feb 1 13:52:38 cl41 vpp: /usr/bin/vpp[15173]: dpdk_config:1344: > socket-mem argument is deprecated > > Feb 1 13:52:38 cl41 vpp: /usr/bin/vpp[15173]: dpdk: EAL init args: -c > 4446 -n 4 --in-memory --vdev crypto_ccp0,socket_id=0 --vdev > crypto_ccp1,socket_id=1 --vdev crypto_ccp2,socket_id=2 --vdev > crypto_ccp3,socket_id=3 --file-prefix vpp -w 0000:41:00.0 -w 0000:41:00.1 > --master-lcore 1 > > Feb 1 13:52:38 cl41 /usr/bin/vpp[15173]: dpdk: EAL init args: -c 4446 -n > 4 --in-memory --vdev crypto_ccp0,socket_id=0 --vdev > crypto_ccp1,socket_id=1 --vdev crypto_ccp2,socket_id=2 --vdev > crypto_ccp3,socket_id=3 --file-prefix vpp -w 0000:41:00.0 -w 0000:41:00.1 > --master-lcore 1 > > Feb 1 13:52:38 cl41 vnet[15173]: dpdk_ipsec_process:1014: not enough > DPDK crypto resources, default to OpenSSL > > Feb 1 13:53:13 cl41 vnet[15173]: unix_signal_handler:159: received > signal SIGCONT, PC 0x7f9aa76687e0 > > Feb 1 13:53:13 cl41 systemd: Stopping Vector Packet Processing Process... > > Feb 1 13:53:13 cl41 vnet[15173]: received SIGTERM, exiting... > > Feb 1 13:53:13 cl41 vnet[15173]: unix_signal_handler:159: received > signal SIGCONT, PC 0x7f9aa76687e0 > > Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1456 > > Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1468 > > Feb 1 13:53:13 cl41 vpp: Queue 0 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 1 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 2 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 3 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 4 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1456 > > Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1468 > > Feb 1 13:53:13 cl41 vpp: Queue 0 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 1 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 2 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 3 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 4 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1456 > > Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1468 > > Feb 1 13:53:13 cl41 vpp: Queue 0 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 1 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 2 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 3 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 4 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1456 > > Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1468 > > Feb 1 13:53:13 cl41 vpp: Queue 0 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 1 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 2 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 3 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: Queue 4 can access 7 LSB regions of mask 254 > > Feb 1 13:53:13 cl41 vpp: CCP : Crypto device count = 4 > > Feb 1 13:53:13 cl41 systemd: Stopped Vector Packet Processing Process. > > > > > > If I modify the startup.conf to only have one worker core (“corelist-workers > 2”), > > I don’t get the message about “not enough DPDK crypto resources”, and I > see: > > > > vpp# show dpdk crypto device > > crypto_ccp0 crypto_ccp up > > numa_node 0, max_queues 1 > > free_resources 0, used_resources 1 > > SYMMETRIC_CRYPTO, SYM_OPERATION_CHAINING, HW_ACCELERATED > > Cipher: aes-cbc-128, aes-cbc-192, aes-cbc-256, aes-ctr-128, aes-ctr-192, > aes-ctr-256, aes-gcm-128, aes-gcm-192, aes-gcm-256 > > Auth: > > > > vpp# > > > > > > When I look at the source code in crypto_parse_capabilities(), I am > concerned about the lines > > “ dcm->enabled |= (alg->resources >= n_mains);” in the > following code: > > > > static void > > crypto_parse_capabilities (crypto_dev_t * dev, > > const struct rte_cryptodev_capabilities *cap, > > u32 n_mains) > > { > > dpdk_crypto_main_t *dcm = &dpdk_crypto_main; > > crypto_alg_t *alg; > > u8 len, inc; > > > > for (; cap->op != RTE_CRYPTO_OP_TYPE_UNDEFINED; cap++) > > { > > /* A single capability maps to multiple cipher/auth algorithms */ > > switch (cap->sym.xform_type) > > { > > case RTE_CRYPTO_SYM_XFORM_AEAD: > > case RTE_CRYPTO_SYM_XFORM_CIPHER: > > inc = cap->sym.cipher.key_size.increment; > > inc = inc ? inc : 1; > > for (len = cap->sym.cipher.key_size.min; > > len <= cap->sym.cipher.key_size.max; len += inc) > > { > > alg = cipher_cap_to_alg (cap, len); > > if (!alg) > > continue; > > dev->cipher_support[cipher_alg_index (alg)] = 1; > > alg->resources += vec_len (dev->free_resources); > > /* At least enough resources to support one algo */ > > dcm->enabled |= (alg->resources >= n_mains); > > } > > break; > > case RTE_CRYPTO_SYM_XFORM_AUTH: > > inc = cap->sym.auth.digest_size.increment; > > inc = inc ? inc : 1; > > for (len = cap->sym.auth.digest_size.min; > > len <= cap->sym.auth.digest_size.max; len += inc) > > { > > alg = auth_cap_to_alg (cap, len); > > if (!alg) > > continue; > > dev->auth_support[auth_alg_index (alg)] = 1; > > alg->resources += vec_len (dev->free_resources); > > /* At least enough resources to support one algo */ > > dcm->enabled |= (alg->resources >= n_mains); > > } > > break; > > default: > > ; > > } > > } > > } > > > > crypto_scan_devs() calls crypto_parse_capabilities() within a loop over > cryptodevs. > > The code appears to be comparing the resources of each cryptodev to the > total number, > > n_mains. Should the code be comparing the sum of resources (from all of > the cryptodevs) > > to the total value, n_mains? > > > > - Lee > > > > > > *From:* vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io] *On Behalf Of *Sergio > Gonzalez Monroy > *Sent:* Wednesday, January 30, 2019 1:59 PM > *To:* Roberts, Lee A. <lee.robe...@hpe.com>; manuel.alo...@cavium.com; > vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp > > > > Hi folks, > > > > Give the following patch a try and let me know. > > https://gerrit.fd.io/r/#/c/17184/ > > > > I have done some very basic testing with SW cryptodev with single queue > pair and all seems fine. > > > > Let me know if there is any issue with it. > > > > Regards, > > Sergio > ------------------------------ > > *From:* Roberts, Lee A. <lee.robe...@hpe.com> > *Sent:* Tuesday, January 29, 2019 3:49 PM > *To:* Sergio Gonzalez Monroy; manuel.alo...@cavium.com; > vpp-dev@lists.fd.io > *Subject:* RE: [vpp-dev] Question about crypto dev queue pairs #vpp > > > > Sergio, > > > > I encountered the same problem when attempting to enable the AMD CCP poll > mode driver > > in VPP 18.10. > > > > As mentioned earlier in this e-mail thread, with max_qp = 1, max_res_idx > becomes 65535 > > in the following statement: > > > > max_res_idx = (dev->max_qp / 2) - 1; > > > > I hadn’t found the time to study/debug the code to understand whether this > was a > > VPP or DPDK issue. If you have a patch available in the next few days, I > could > > test it with the AMD CCP device. > > > > - Lee Roberts > > > > > > *From:* vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io > <vpp-dev@lists.fd.io>] *On Behalf Of *Sergio Gonzalez Monroy > *Sent:* Tuesday, January 29, 2019 2:18 AM > *To:* manuel.alo...@cavium.com; vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp > > > > Hi Manuel, > > > > This is likely a mismatch in VPP side. I only tested it with QAT (2 qps > per VF) and SW cryptodevs (default 8 qps) at the time (over a year ago). > > > > I only tested it with SW cryptodevs and QAT, that was the HW I had access > to. > > > > So like I mentioned before, if you do not want to rework the code to > support 1 qp per resource, then a check for at least 2 qps per device is > required to use that device. > > > > I could provide a patch to use 1 pq per resource over the next few days if > you are interested in it or could review if you decide to do the work. > > > > Which device do you want to use? > > > > Regards, > > Sergio > ------------------------------ > > *From:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> on behalf of > manuel.alo...@cavium.com <manuel.alo...@cavium.com> > *Sent:* Monday, January 28, 2019 4:15 PM > *To:* vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp > > > > Hi Sergio, > > thank you for the explanation, I see that there are 2 (or more qps). My > concern was due to dpdk, since there are a few device drivers exporting > only one queue pair for their crypto devices. > (I followed the code assuming one qps, based on a dpdk-18.11 exported > value) > So I do not know where is the mismatching, vpp or dpdk? > > > BR, > Manuel > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12173): https://lists.fd.io/g/vpp-dev/message/12173 Mute This Topic: https://lists.fd.io/mt/29538345/21656 Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
