Hi Akhil, Please see my answer below.
Thanks, Lukasz On 24.02.2020 14:40, Akhil Goyal wrote: > External Email > > ---------------------------------------------------------------------- > Hi Anoob/Lukasz, > >> >> This series introduces event-mode additions to ipsec-secgw. >> >> With this series, ipsec-secgw would be able to run in eventmode. The >> worker thread (executing loop) would be receiving events and would be >> submitting it back to the eventdev after the processing. This way, >> multicore scaling and h/w assisted scheduling is achieved by making use >> of the eventdev capabilities. >> >> Since the underlying event device would be having varying capabilities, >> the worker thread could be drafted differently to maximize performance. >> This series introduces usage of multiple worker threads, among which the >> one to be used will be determined by the operating conditions and the >> underlying device capabilities. >> >> For example, if an event device - eth device pair has Tx internal port, >> then application can do tx_adapter_enqueue() instead of regular >> event_enqueue(). So a thread making an assumption that the device pair >> has internal port will not be the right solution for another pair. The >> infrastructure added with these patches aims to help application to have >> multiple worker threads, there by extracting maximum performance from >> every device without affecting existing paths/use cases. >> >> The eventmode configuration is predefined. All packets reaching one eth >> port will hit one event queue. All event queues will be mapped to all >> event ports. So all cores will be able to receive traffic from all ports. >> When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event >> device >> will ensure the ordering. Ordering would be lost when tried in PARALLEL. >> >> Following command line options are introduced, >> >> --transfer-mode: to choose between poll mode & event mode >> --event-schedule-type: to specify the scheduling type >> (RTE_SCHED_TYPE_ORDERED/ >> RTE_SCHED_TYPE_ATOMIC/ >> RTE_SCHED_TYPE_PARALLEL) >> >> Additionally the event mode introduces two modes of processing packets: >> >> Driver-mode: This mode will have bare minimum changes in the application >> to support ipsec. There woudn't be any lookup etc done in >> the application. And for inline-protocol use case, the >> thread would resemble l2fwd as the ipsec processing would be >> done entirely in the h/w. This mode can be used to benchmark >> the raw performance of the h/w. All the application side >> steps (like lookup) can be redone based on the requirement >> of the end user. Hence the need for a mode which would >> report the raw performance. >> >> App-mode: This mode will have all the features currently implemented with >> ipsec-secgw (non librte_ipsec mode). All the lookups etc >> would follow the existing methods and would report numbers >> that can be compared against regular ipsec-secgw benchmark >> numbers. >> >> The driver mode is selected with existing --single-sa option >> (used also by poll mode). When --single-sa option is used >> in conjution with event mode then index passed to --single-sa >> is ignored. >> >> Example commands to execute ipsec-secgw in various modes on OCTEON TX2 >> platform, >> >> #Inbound and outbound app mode >> ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w >> 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log- >> level=8 -c 0x1 -- -P -p 0x3 -u 0x1 --config "(1,0,0),(0,0,0)" -f aes-gcm.cfg >> -- >> transfer-mode event --event-schedule-type parallel >> > > What is the need of adding the port queue core mapping in case of event. > In case of event, all queues are given to eventdev and there is no need for > specifying such specific mapping. In l3fwd also this was not done. > [Lukasz] You are right port,queue,core mapping is not needed in case of event mode. I will remove --config option from being used in event mode. Number of ports to be initialized will be derived from the port mask and nb_rx_queues per port will be set to 1. >> #Inbound and outbound driver mode >> ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w >> 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log- >> level=8 -c 0x1 -- -P -p 0x3 -u 0x1 --config "(1,0,0),(0,0,0)" -f aes-gcm.cfg >> -- >> transfer-mode event --event-schedule-type parallel --single-sa 0 >> >> This series adds non burst tx internal port workers only. It provides >> infrastructure >> for non internal port workers, however does not define any. Also, only inline >> ipsec >> protocol mode is supported by the worker threads added. >> >> Following are planned features, >> 1. Add burst mode workers. >> 2. Add non internal port workers. >> 3. Verify support for Rx core (the support is added but lack of h/w to >> verify). >> 4. Add lookaside protocol support. >> >> Following are features that Marvell won't be attempting. >> 1. Inline crypto support. >> 2. Lookaside crypto support. >> >> For the features that Marvell won't be attempting, new workers can be >> introduced by the respective stake holders. >> >> This series is tested on Marvell OCTEON TX2. >> This series is targeted for 20.05 release. >> >> Changes in v4: >> * Update ipsec-secgw documentation to describe the new options as well as >> event mode support. >> * In event mode reserve number of crypto queues equal to number of eth ports >> in order to meet inline protocol offload requirements. >> * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool >> and include fragments table size into the calculation. >> * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static keyword >> from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c. >> * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), check_sp() >> and prepare_out_sessions_tbl() functions as a result of changes introduced >> by SAD feature. >> * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx >> is created with rte_zmalloc. >> * Minor cleanup enhancements: >> - In eh_set_default_conf_eventdev() function in event_helper.c put definition >> of int local vars in one line, remove invalid comment, put >> "eventdev_config->ev_queue_mode = RTE_EVENT_QUEUE_CFG_ALL_TYPES" >> in one line >> instead of two. >> - Remove extern "C" from event_helper.h. >> - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() >> and >> eh_dev_has_tx_internal_port() functions in event_helper.c. >> - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c. >> - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec- >> secgw.h, >> remove #include <rte_hash.h>. >> - Remove not needed includes in ipsec_worker.c. >> - Remove expired todo from ipsec_worker.h. >> >> Changes in v3: >> * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c >> including minor rework. >> * Rename --schedule-type option to --event-schedule-type. >> * Replace macro UNPROTECTED_PORT with static inline function >> is_unprotected_port(). >> * Move definitions of global variables used by multiple modules >> to .c files and add externs in .h headers. >> * Add eh_check_conf() which validates ipsec-secgw configuration >> for event mode. >> * Add dynamic calculation of number of buffers in a pool based >> on number of cores, ports and crypto queues. >> * Fix segmentation fault in event mode driver worker which happens >> when there are no inline outbound sessions configured. >> * Remove change related to updating number of crypto queues >> in cryptodevs_init(). The update of crypto queues will be handled >> in a separate patch. >> * Fix compilation error on 32-bit platforms by using userdata instead >> of udata64 from rte_mbuf. >> >> Changes in v2: >> * Remove --process-dir option. Instead use existing unprotected port mask >> option (-u) to decide wheter port handles inbound or outbound traffic. >> * Remove --process-mode option. Instead use existing --single-sa option >> to select between app and driver modes. >> * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. >> * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). >> * Move destruction of flows to a location where eth ports are stopped >> and closed. >> * Print error and exit when event mode --schedule-type option is used >> in poll mode. >> * Reduce number of goto statements replacing them with loop constructs. >> * Remove sec_session_fixed table and replace it with locally build >> table in driver worker thread. Table is indexed by port identifier >> and holds first inline session pointer found for a given port. >> * Print error and exit when sessions other than inline are configured >> in event mode. >> * When number of event queues is less than number of eth ports then >> map all eth ports to one event queue. >> * Cleanup and minor improvements in code as suggested by Konstantin >> >> Ankur Dwivedi (1): >> examples/ipsec-secgw: add default rte flow for inline Rx >> >> Anoob Joseph (5): >> examples/ipsec-secgw: add framework for eventmode helper >> examples/ipsec-secgw: add eventdev port-lcore link >> examples/ipsec-secgw: add Rx adapter support >> examples/ipsec-secgw: add Tx adapter support >> examples/ipsec-secgw: add routines to display config >> >> Lukasz Bartosik (9): >> examples/ipsec-secgw: add routines to launch workers >> examples/ipsec-secgw: add support for internal ports >> examples/ipsec-secgw: add event helper config init/uninit >> examples/ipsec-secgw: add eventmode to ipsec-secgw >> examples/ipsec-secgw: add driver mode worker >> examples/ipsec-secgw: add app mode worker >> examples/ipsec-secgw: make number of buffers dynamic >> doc: add event mode support to ipsec-secgw >> examples/ipsec-secgw: reserve crypto queues in event mode >> >> doc/guides/sample_app_ug/ipsec_secgw.rst | 138 ++- >> examples/ipsec-secgw/Makefile | 2 + >> examples/ipsec-secgw/event_helper.c | 1812 >> ++++++++++++++++++++++++++++++ >> examples/ipsec-secgw/event_helper.h | 327 ++++++ >> examples/ipsec-secgw/ipsec-secgw.c | 463 ++++++-- >> examples/ipsec-secgw/ipsec-secgw.h | 88 ++ >> examples/ipsec-secgw/ipsec.c | 5 +- >> examples/ipsec-secgw/ipsec.h | 53 +- >> examples/ipsec-secgw/ipsec_worker.c | 638 +++++++++++ >> examples/ipsec-secgw/ipsec_worker.h | 35 + >> examples/ipsec-secgw/meson.build | 6 +- >> examples/ipsec-secgw/sa.c | 21 +- >> examples/ipsec-secgw/sad.h | 5 - >> 13 files changed, 3464 insertions(+), 129 deletions(-) >> create mode 100644 examples/ipsec-secgw/event_helper.c >> create mode 100644 examples/ipsec-secgw/event_helper.h >> create mode 100644 examples/ipsec-secgw/ipsec-secgw.h >> create mode 100644 examples/ipsec-secgw/ipsec_worker.c >> create mode 100644 examples/ipsec-secgw/ipsec_worker.h >> >> -- >> 2.7.4 >
