JannePeltonen replied on github web page:
doc/users-guide/users-guide-ipsec.adoc
line 294
@@ -0,0 +1,443 @@
+== IPsec services
+
+In addition to general cryptographic services, ODP offers offload support for
+the IPsec protocol. IPsec is a general term referencing a suite of protocols
+and packet formats and as such a full discussion of IPsec is beyond the scope
+of this document. See https://tools.ietf.org/html/rfc4301[RFC 4301] and
+related RFCs for more detail. This section assumes the reader is already
+familiar with IPsec and focuses on explaining the ODP APIs that support it.
+
+ODP provides APIs for the following IPsec services:
+
+* General IPsec configuration
+* Security Association (SA) configuration and lifecycle management
+* Synchronous and Asynchronous IPsec lookaside processing
+* Inline processing for full IPsec RX and/or TX offload
+* Pipelining for RX traffic
+* Fragmentation support for TX traffic
+* IPsec event management
+
+=== IPsec Capabilities and Configuration
+As with other features, ODP provides APIs that permit applications to query
+platform-specific IPsec capabilities. The `odp_ipsec_capability()` API queries
+the general IPsec features available while the `odp_ipsec_cipher_capability()`
+and `odp_ipsec_auth_capability()` APIs provide detail on the range of
+cipher and authentication algorithms supported by IPsec on this platform.
+
+General IPsec capabilities that are reported include:
+
+* The IPsec operation modes supported by this implementation. Different
+operation modes may be _not supported_, _supported_, or _preferred_. A
+preferred form means that this mode takes advantage of hardware
+acceleration features to achieve best performance.
+* Whether IPsec AH processing is supported. All ODP platforms must provide
+support for IPsec ESP processing, however since AH is relatively rare, it
+may not be supported, or supported only via software emulation (_e.g.,_ be
+non-preferred).
+* Whether IPsec headers should be retained on decrypt for inbound inline
+operations.
+* Whether classification pipelining is supported (to be discussed below).
+
+In addition, capabilities also inform the application of the maximum number
+of destination queues and classification CoS targets supported. These
+will be discussed further later.
+
+==== IPsec Operation Modes
+IPsec operates in one of three modes: Synchronous, Asynchronous, and Inline.
+
+==== Lookaside Processing
+Synchronous and Asynchronous are types of _lookaside_ processing. In lookaside
+mode, the application receives (or creates) an IPsec packet and then uses ODP
+to perform one of two functions:
+
+* To decrypt an IPsec packet into a "normal" packet
+* To take a "normal" packet and encrypt it into an IPsec packet.
+
+This process may be performed _synchronously_ with the APIs `odp_ipsec_in()`
+(to decrypt) and `odp_ipsec_out()` (to encrypt). Upon return from these calls
+the requested packet transformation is complete, or an error return code
+indicates that it could not be performed (_e.g.,_ packet decryption failed).
+
+Synchronous processing may be preferred if the application has a large number
+of worker threads so that blocking any individual worker while IPsec processing
+is performed represents a reasonable design. The alternative is to use
+_asynchronous_ forms of these APIs:
+
+* `odp_ipsec_in_enq()` for decrypt
+* `odp_ipsec_out_enq()` for encrypt
+
+These simply pass packets to IPsec for processing. When this processing is
+complete, the resulting packets are sent to the completion queue associated
+with the SA used by the operation, serving as IPsec completion events as
+shown here:
+
+image::ipsec-lookaside.svg[align="center"]
+
+If the operation fails because SA lookup failed for inbound processing, then
+these result packets are sent to the default queue specified as part of the
+`odp_ipsec_inbound_config_t` used in the `odp_ipsec_config()` call.
+
+Following an asynchronous IPsec call, the worker thread moves on to process
+other events until the IPsec completion shows up. At that point the worker
+thread sees whether the operation was successful or not and continues
+processing for that packet. These events may be direct-polled with
+`odp_queue_deq()` if the completion queue was created as a plain queue, or
+processed via the ODP scheduler if the completion queue was created as a
+scheduled queue.
+
+==== Inline Processing
+While lookaside processing offers flexibility, it still requires extra
+processing steps not required by modern hardware. To avoid this overhead
+ODP also offers _inline_ processing support for IPsec. In this mode the
+processing of IPsec packets on the RX and TX paths is fully offloaded as
+shown here:
+
+image::ipsec-inline.svg[align="center"]
+
+It is worth noting that, depending on the implementation and application
+needs, inline processing may be enabled only for one direction (inbound or
+outbound) or for both directions.
+
+On the receive side, once configured for inline processing, arriving IPsec
+packets that are recognized at the PktIO interface are decrypted automatically
+before the application ever sees them. On the transmit side, the application
+calls `odp_ipsec_out_inline()` and the packet is encrypted and queued for
+transmission as a single operation without further application involvement.
+Note that if an inbound IPsec packet is not recognized (_e.g.,_ it belongs to
+an unknown SA) then it will be presented to the application as-is without
+further processing. The application may then use a lookaside call to process
+the packet if it is able to supply a matching SA by other means.
+
+On the receive side, after an IPsec packet is decrypted, it may be
+_pipelined_ to the ODP classifier or added to a poll queue, as the
+application wishes. The advantage of classification pipelining is that inbound
+IPsec traffic is automatically decrypted and classified into appropriate
+flow-based queues for ease of processing.
+
+On the transmit side, since IPsec encryption and tunneling may exceed an
+output MTU, ODP also offers support for MTU configuration and automatic IPsec
+TX fragmentation.
+
+Both classification pipelining and TX fragmentation support are support
+features that are indicated by `odp_ipsec_capability()`.
+
+Note that at present inline IPsec output support sends resulting packets
+directly to an output PktIO. If it's desired to send them to the ODP
+Traffic Manager for shaping prior to transmission, use the lookaside APIs
+to perform the IPsec encrypt and then call `odp_tm_enq()` on the resulting
+packet.
+
+=== IPsec Configuration
+Prior to making use of IPsec services, the `odp_ipsec_config()` API is used to
+configure IPsec processing options. This API takes a pointer to an
+`odp_ipsec_config_t` struct as its argument. SAs in ODP are represented by the
+abstract type `odp_ipsec_sa_t`.
+
+The `odp_ipsec_config_t` struct specifies the inbound and outbound
+processing modes that the application plans to use, the maximum number of
+Security Associations it will use, and sets inbound and outbound
+processing options.
+
+==== IPsec Inbound Configuration
+Inbound configuration options for IPsec specify the default `odp_queue_t` to
+be used for processing global events like SA lookup failures, how Security
+Parameter Index (SPI) lookup is to be performed, and whether the application
+requires ODP to retain outer headers for decrypted IPsec packets.
+
+Parsing options specify how "deep" decrypted packets are to be parsed
+after IPsec processing by specifying the packet layers of interest to the
+application (None, L2, L3, L4, or All). And which checksums should be verified
+on decrypted packets.
+
+==== IPsec Outbound Configuration
+Outbound configuration options for IPsec specify checksum insertion processing
+that should be performed prior to encryption.
+
+=== IPsec Events
+IPsec introduces one new event type and one new event subtype. These are:
+
+* IPsec packet events. These are events of type `ODP_EVENT_PACKET` that have
+subtype `ODP_EVENT_PACKET_IPSEC`. These are packets that carry additional
+IPsec-related metadata in the form of an `odp_ipsec_packet_result_t` struct
+that can be retrieved from the packet via the `odp_ipsec_result()` API.
+
+* IPsec status notifications. These are events of type `ODP_EVENT_IPSEC_STATUS`
+that indicate status events not associated with any particular IPsec
+packet. Such events carry status in the form of an `odp_ipsec_status_t`
+struct that is retrieved from the event via the `odp_ipsec_status()` API.
+
+IPsec-related events are thus part of normal and exception processing when
+working with IPsec.
+
+=== Security Associations (SAs)
+The fundamental "building block" for IPsec processing is the _Security
+Association (SA)_. Similar to a crypto session, the SA encapsulates the keying
+material and context needed to perform IPsec protocol processing for inbound
+or outbound packets on a given flow, as well as additional processing options
+that control how IPsec is to be used for packets processed under this
+SA. Security Associations are unidirectional (RX or TX) so a flow that
+requires both inbound (decrypt) and outbound (encrypt) IPsec functions will
+have two SAs associated with it.
+
+After ODP initialization, IPsec support is dormant until it is configured
+by a call to `odp_ipsec_config()` as described earlier. Once configured,
+SAs may be created by calling `odp_ipsec_sa_create()`.
+
+==== SA Creation and Configuration
+The `odp_ipsec_sa_create()` API takes an `odp_ipsec_sa_param_t` argument that
+describes the SA to be created. Use the `odp_ipsec_sa_param_init()` API to
+initialize this to its default state and then override selected fields within
+the param struct as needed.
+
+Items specified in the `odp_ipsec_sa_param_t` struct include:
+
+* The direction of the SA (inbound or outbound).
+
+* The IPsec protocol being used (ESP or AH).
+
+* The IPsec protocol mode (Transport or Tunnel).
+
+* The parameters needed for the crypto and authentication algorithms to be
+used by this SA.
+
+* Miscellaneous SA options that control behavior such as use of Extended
+Sequence Numbers (ESNs), the use of UDP encapsulation, various copy
+options for header fields, and whether the TTL (Hop Limit) field should be
+decremented when operating in tunnel mode.
+
+* Parameters controlling the SA lifetime.
+
+* The Security Parameter Index (SPI) that packets will use to indicate that
+they belong to this SA.
+
+* The pipeline mode used by this SA.
+
+* The destination `odp_queue_t` to be used for status events associated with
+this SA.
+
+* The user context pointer (and length) associated with this SA for
+application use.
+
+In addition, there are specific direction-specific parameters that vary
+based on whether the SA is for inbound or outbound use. For inbound SAs:
+
+* Controls for how this SA is to be looked up.
+
+* The size of the anti-replay window to be used.
+
+* The default CoS to use when classification pipelining packets matching this
+SA.
+
+For outbound SAs:
+
+* Tunnel parameters to use when doing outbound processing in tunnel mode.
+
+* The fragmentation mode to be used.
+
+* The MTU to be used to control the maximum length IP packets that outbound
+IPsec operations may produce. This can be changed dynamically by the
+`odp_ipsec_sa_mtu_update()` API.
+
+As can be seen, SAs have a large degree of configurability.
+
+==== SA Lifecycle Management
+In discussing the lifecycle of an SA, it is useful to refer to the following
+state diagram:
+
+image::ipsec_fsm.svg[align="center"]
+
+After creation, IPsec services are active for this security association. How
+this is done depends on the IPsec operating mode being used
+
+===== IPsec Lookaside Processing
+If the SA is operating in lookaside mode (the `odp_ipsec_mode_t` is
+`ODP_IPSEC_OP_MODE_SYNC` or `ODP_IPSEC_OP_MODE_ASYNC`), then inbound or
+outbound lookaside operations may be performed.
+
+The synchronous forms of these APIs are:
+
+* `odp_ipsec_in()`
+* `odp_ipsec_out()`
+
+Upon return from these calls, the return code tells the application whether
+the application was successful and if not the reason is contained in the
+`odp_ipsec_result_t` struct that is attached to the failing packet. This
+struct is retrieved via the `odp_ipsec_result()` API.
+
+The asynchronous forms of these APIs are:
+
+* `odp_ipsec_in_enq()`
+* `odp_ipsec_out_enq()`
+
+Here, a return code of 0 simply indicates that the requested IPsec operation
+was successfully initiated.
+
+For both synchronous and asynchronous IPsec operations an input packet array
+is transformed into an output packet array as specified by a controlling
+parameter struct. For inbound operations, the `odp_ipsec_in_param_t` is
+used to specify how SA processing is to be performed for the requested
+operation. The caller may say that SA lookup processing should be performed
+for each input packet, a single (specified) SA should be used for all packets,
+or that each packet has a specified individual SA.
+
+For outbound lookaside operations, a corresponding `odp_ipsec_out_param_t`
+serves a similar role, but here the SA must be specified since the input
+packet(s) are non-IPsec packets. Again the option is to use a single SA for
+all input packets or one per input packet.
+
+For outbound operations, an associated array of `odp_ipsec_out_opt_t` structs
+is also used to control the fragmentation mode to be used as part of the
+outbound processing. Options here are to not fragment, to fragment before
+IPsec processing, after IPsec processing, or to only check whether IP
+fragmentation is needed but not to perform it. The `mtu` status error bit
+in the `odp_ipsec_packet_result_t` is set if check processing detects that
+the resulting packet will not fit into the configured MTU. Note that the MTU
Comment:
Is it clear enough here that the mtu error bit is set only when MTU checking
was requested and not in case fragmentation offload was requested?
> JannePeltonen wrote
> Same comment as for odp_ipsec_{in,out}(), the description of the return code
> is not correct.
>> JannePeltonen wrote
>> The return code does not tell whether the application of the SA was
>> successful but how many input packets were consumed by the operation. The
>> per-packet operation status is retrieved via the odp_ipsec_result() api for
>> the outputted packets, which will be available only when the function
>> succeeds.
>>> JannePeltonen wrote
>>> The operating mode is a global setting, not per-SA setting as the text
>>> implies. And the relevant type is odp_ipsec_op_mode_t, not odp_ipsec_mode_t
>>> which is for selecting between tunnel and transport mode.
>>>> JannePeltonen wrote
>>>> "How this is done" sounds a bit awkward as it is not clear what "this"
>>>> refers to (the way the IPsec services are used). Maybe the sentence should
>>>> be reworded.
>>>>> JannePeltonen wrote
>>>>> Would "minimum size" be better than "size" here?
>>>>>> JannePeltonen wrote
>>>>>> The queue is not only for status events but for ipsec packet events too.
>>>>>> Maybe just remove the word "status"?
>>>>>>> JannePeltonen wrote
>>>>>>> This should be: "...can be retained..." instead of "...should be
>>>>>>> retained..." as this is a capability and application chooses whether it
>>>>>>> wants to retain the headers if the ODP is capable.
>>>>>>>> Bill Fischofer(Bill-Fischofer-Linaro) wrote:
>>>>>>>> I'd like to see us all make it a priority to close on #197 next week.
>>>>>>>> I'll post a doc update once we're all agreed on the precise semantics
>>>>>>>> and operation flow. So I'm fine with holding off merging this PR until
>>>>>>>> then.
>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>> Any update here? @NikhilA-Linaro, @bala-manoharan can you please
>>>>>>>>> comment wrt your implementations?
>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>> Well, we do not have #197 merged, so it is too early to depend on
>>>>>>>>>> that.
>>>>>>>>>> Also, frankly speaking, this `sa_disabled` warning inside
>>>>>>>>>> `odp_ipsec_result_t` is a backup plan. It is expected that most of
>>>>>>>>>> the implementations will report this as a proper `ODP_IPSEC_STATUS`
>>>>>>>>>> event, carrying this warning bit inside.
>>>>>>>>>>> Bill Fischofer(Bill-Fischofer-Linaro) wrote:
>>>>>>>>>>> OK, changed in v6.
>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>> s/default //
>>>>>>>>>>>>> Bill Fischofer(Bill-Fischofer-Linaro) wrote:
>>>>>>>>>>>>> OK, corrected in v5.
>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>> In lookaside mode soft limit expiration is reported as `warn`
>>>>>>>>>>>>>> part of `ipsec_op_status` packet metadata.
>>>>>>>>>>>>>>> Bill Fischofer(Bill-Fischofer-Linaro) wrote:
>>>>>>>>>>>>>>> OK, in v4 I've added a new terminal state `SA_Expired` to the
>>>>>>>>>>>>>>> FSM and have updated the doc to say "expired" rather than
>>>>>>>>>>>>>>> "disabled". From the expired state the only valid operation is
>>>>>>>>>>>>>>> `odp_ipsec_sa_destroy()`.
>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>> Yep. It is just not a 'disabled' state, because we have
>>>>>>>>>>>>>>>> separate definition for 'disabled SA'.
>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>> IIRC, one can call it only once in NXP case. @NikhilA-Linaro,
>>>>>>>>>>>>>>>>> could you please comment?
>>>>>>>>>>>>>>>>>> Bill Fischofer(Bill-Fischofer-Linaro) wrote:
>>>>>>>>>>>>>>>>>> So how is this indicated in lookaside mode? The whole point
>>>>>>>>>>>>>>>>>> of ODP providing the limit support is so the application
>>>>>>>>>>>>>>>>>> doesn't have to track byte/packet counts itself, so it's
>>>>>>>>>>>>>>>>>> expected that soft limit overruns will happen as part of
>>>>>>>>>>>>>>>>>> lookaside processing as well.
>>>>>>>>>>>>>>>>>>> Bill Fischofer(Bill-Fischofer-Linaro) wrote:
>>>>>>>>>>>>>>>>>>> If you let yourself run out of gas, the car can stop at
>>>>>>>>>>>>>>>>>>> inconvenient times, which is why one pays attention to that
>>>>>>>>>>>>>>>>>>> low fuel warning light. A hard limit is a hard limit.
>>>>>>>>>>>>>>>>>>> That's what makes it hard. Any other definition seems
>>>>>>>>>>>>>>>>>>> confusingly fuzzy and unnecessary.
>>>>>>>>>>>>>>>>>>>> Bill Fischofer(Bill-Fischofer-Linaro) wrote:
>>>>>>>>>>>>>>>>>>>> I thought the restriction is that you can call this
>>>>>>>>>>>>>>>>>>>> repeatedly as long as an SA hasn't yet been created. I can
>>>>>>>>>>>>>>>>>>>> change this (and the state diagram) if that's not the case.
>>>>>>>>>>>>>>>>>>>>> Bill Fischofer(Bill-Fischofer-Linaro) wrote:
>>>>>>>>>>>>>>>>>>>>> It's part of the `enum`. In this case L2 would
>>>>>>>>>>>>>>>>>>>>> effectively be None.
>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>> Only in OUT INLINE mode, if I remember the outcome of
>>>>>>>>>>>>>>>>>>>>>> discussions correctly.
>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>> Ahem. It does not enter disabled state per se:
>>>>>>>>>>>>>>>>>>>>>>> - It is an undefined behaviour (iow, an application
>>>>>>>>>>>>>>>>>>>>>>> error) to submit packets to disabled SA
>>>>>>>>>>>>>>>>>>>>>>> - It is a perfectly valid to submit packets to SA after
>>>>>>>>>>>>>>>>>>>>>>> hard limit overrun (e.g. because other packets might be
>>>>>>>>>>>>>>>>>>>>>>> already queued at this moment).
>>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>>> It is worth mentioning that depending on the
>>>>>>>>>>>>>>>>>>>>>>>> implementation and application needs, inline
>>>>>>>>>>>>>>>>>>>>>>>> processing might be enabled either in both directions
>>>>>>>>>>>>>>>>>>>>>>>> or in just one direction.
>>>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> TBD: mention that it MUST be called at most once per
>>>>>>>>>>>>>>>>>>>>>>>>> IPsec application.
>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> L2 does not make sense here, does it?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> ... parsed after IPsec processing ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> TBD: describe that some IPsec packets still might
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be reported via plain PktIO interface (e.g.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> because of SA lookup failure). They can be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> resubmitted to IPsec in lookaside mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If SA was not determined (because SA lookup
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed for inbound packet), event will be sent to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the default queue.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... resulting packet is sent back serving as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IPsec completion event ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitry Eremin-Solenikov(lumag) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... in inbound inline operations.
https://github.com/Linaro/odp/pull/185#discussion_r148316384
updated_at 2017-11-01 16:48:05
