Please see in-line.
-----Original Message-----
From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Bogdan
Pricope
Sent: 22 March 2017 14:17
To: Mahipal Challa <mahipalreddy2...@gmail.com>
Cc: Narayana, Prasad Athreya <prasad.athr...@cavium.com>; Masood, Faisal
<faisal.mas...@cavium.com>; Challa, Mahipal <mahipal.cha...@cavium.com>;
Challa, Mahipal <mahipal.cha...@cavium.com>; lng-odp-forward
<lng-odp@lists.linaro.org>
Subject: Re: [lng-odp] [RFC, API-NEXT v1 1/1] comp:compression interface
Hi,
My understanding is that:
- compression can be used in stateless or statefull mode. How can we
differentiate between the two configurations: by specifying a special algorithm
on session creation or by setting "last" field to "true" for every operation?
Do we need this setting per operation or per session?
>> 'last' or 'current enumerated algo' are not sufficient parameters to decide
>> stateless operation. Pure stateless operation depends on App that it should
>> pass data which is guaranteed to decompress into a known size and thus pass
>> an sufficient output buffer size.
Take a case where user want stateless operation and set parameters:
In_pkt length 32KB,
Out_pkt length 64KB
Last=true // indicating this is first and Only data to process,
Call odp_comp_operation() to decompress,
Now, during decompress operation at any point if odp_comp_operation () returns
ODP_COMP_ERR_OUT_OF_SPACE(in case if uncompressed data is > 64kB) , it enter
into statefull mode.
Though in IPComp context, since IP packets are known to be max 64KB and need to
be stateless, thus every call should have with last=1 indicating this is last
chunk.
- stateless mode may be used by IPComp (RFC3173) - it will be application
job to add IPComp header between IP header and payload
>>yes
- statefull mode requires a connection oriented protocol (TCP) support to
take care of delivering date in order and without drops.
How will receiver know if payload is clear text or compressed? Should use a
well-known port?
>> In such case, the protocol must have capability to distinguish
>> compressed/uncompressed text (could be use of dedicated port/use of some
>> negotiation protocol). The Comp Spec is oblivious of protocol/networking
>> aspects of the data.
- on statefull mode, it will always be an output for every input or algorithm
may decide to concat consecutive packets and output fewer?
>> These details are implementation dependent. Implementation may accumulate
>> more data in order to provide better results, in such case Its
>> implementation responsibility to return ODP_COMP_ERR_NONE to let user
>> continue feed more data and keep track of output buffer where it starts
>> producing data (as Application may change output packet with each subsequent
>> call of odp_comp_operation()).
- in-place vs. new packet: in-place processing will be added later - you should
take care on what you to copy between input and output packets - user area, pkt
user pointer, etc.
>> Yes. However, In-place support require bit more feasibility study as it
>> heavily depend on implementation. This becomes an issue if
- implementation may accumulate data as you mentioned above
- if app does not know length of decompressed data beforehand. In such case,
output buffer may grow invariably.
- also algo like deflate,lzs are based on sliding window in which case
implementation would need to keep track how much to retain in input buffer
Thus, we have left it open to discussion subject to requirement/platform
support.
Is my understanding correct?
BR,
Bogdan
On 17 March 2017 at 14:20, Mahipal Challa <mahipalreddy2...@gmail.com> wrote:
> From: Shally Verma <shally.ve...@cavium.com>
>
> An API set to add compression/decompression support in ODP interface.
>
> Signed-off-by: Shally Verma <shally.ve...@cavium.com>
> Signed-off-by: Mahipal Challa <mahipal.cha...@cavium.com>
> ---
> include/odp/api/spec/comp.h | 668
> ++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 668 insertions(+)
>
> diff --git a/include/odp/api/spec/comp.h b/include/odp/api/spec/comp.h
> new file mode 100644 index 0000000..d8f6c68
> --- /dev/null
> +++ b/include/odp/api/spec/comp.h
> @@ -0,0 +1,668 @@
> +/* Copyright (c) 2017, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +/**
> + * @file
> + *
> + * ODP Compression
> + */
> +
> +#ifndef ODP_API_COMP_H_
> +#define ODP_API_COMP_H_
> +#include <odp/visibility_begin.h>
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/** @defgroup odp_compression ODP COMP
> + * ODP Compression defines interface to compress/decompress and
> +authenticate
> + * data.
> + *
> + * Compression here implicilty refer to both compression and decompression.
> + * Example, compression algo 'deflate' mean both 'deflate' and 'inflate'.
> + *
> + * if opcode = ODP_COMP_COMPRESS, then it will Compress,
> + * if opcode = ODP_COMP_DECOMPRESS, then it will Decompress.
> + *
> + * Current version of Interface allow Compression
> +ONLY,Authentication ONLY or
> + * both Compression + Auth ONLY sessions.
> + *
> + * Macros, enums, types and operations to utilise compression.
> + * @{
> + */
> +
> +/**
> + * @def ODP_COMP_SESSION_INVALID
> + * Invalid session handle
> + */
> +
> +/**
> + * @typedef odp_comp_session_t (platform dependent)
> + * Comp API opaque session handle
> + */
> +
> +/**
> + * @typedef odp_comp_compl_t
> +* Compression API completion event (platform dependent) */
> +
> +/**
> + * Compression API operation mode
> + */
> +typedef enum {
> + /** Synchronous, return results immediately */
> + ODP_COMP_SYNC,
> + /** Asynchronous, return results via event */
> + ODP_COMP_ASYNC,
> +} odp_comp_op_mode_t;
> +
> +/**
> + * Comp API operation type
> + */
> +typedef enum {
> + /** Compress and/or Compute ICV */
> + ODP_COMP_OP_COMPRESS,
> + /** Decompress and/or Compute ICV */
> + ODP_COMP_OP_DECOMPRESS,
> +} odp_comp_op_t;
> +
> +/**
> + * Comp API compression algorithm
> + *
> + * Enum listing support Compression algo. Currently one
> + * Compressor corresponds to 1 compliant decompressor.
> + *
> + */
> +typedef enum {
> + /** No algorithm specified */
> + ODP_COMP_ALG_NULL,
> + /** DEFLATE -
> + *
> + * implicit Inflate in case of decode operation
> + *
> + */
> + ODP_COMP_ALG_DEFLATE,
> + /** ZLIB */
> + ODP_COMP_ALG_ZLIB,
> + /** LZS*/
> + ODP_COMP_ALG_LZS,
> + /** SHA1
> + *
> + * When given, imply Authentication ONLY operation
> + *
> + */
> + ODP_COMP_ALG_SHA1,
> + /** SHA256
> + *
> + * When given, imply Authentication ONLY operation
> + *
> + */
> + ODP_COMP_ALG_SHA256,
> + /** DEFLATE+SHA1 */
> + ODP_COMP_ALG_DEFLATE_SHA1,
> + /** DEFLATE+SHA256 */
> + ODP_COMP_ALG_DEFLATE_SHA256,
> + /** ZLIB+SHA */
> + ODP_COMP_ALG_ZLIB_SHA1,
> + /** ZLIB+SHA256 */
> + ODP_COMP_ALG_ZLIB_SHA256,
> + /** LZS+SHA1 */
> + ODP_COMP_ALG_LZS_SHA1,
> + /** LZS+SHA256 */
> + ODP_COMP_ALG_LZS_SHA256
> +} odp_comp_alg_t;
> +
> +/**
> + * Comp API session creation return code */ typedef enum {
> + /** Session created */
> + ODP_COMP_SES_CREATE_ERR_NONE,
> + /** Creation failed, no resources */
> + ODP_COMP_SES_CREATE_ERR_ENOMEM,
> + /** Creation failed, bad compression params */
> + ODP_COMP_SES_CREATE_ERR_INV_COMP,
> + /** Creation failed,requested configuration not supported*/
> + ODP_COMP_SES_CREATE_ERR_NOT_SUPPORTED
> +} odp_comp_ses_create_err_t;
> +
> +/** Comp API operation return codes */ typedef enum {
> + /** Operation completed successfully */
> + ODP_COMP_ERR_NONE,
> + /** Invalid user data pointers */
> + ODP_COMP_ERR_DATA_PTR,
> + /** Invalid input data size */
> + ODP_COMP_ERR_DATA_SIZE,
> + /** Compression Algo fail error */
> + ODP_COMP_ERR_COMP_FAIL,
> + /** Error detected during DMA of data */
> + ODP_COMP_ERR_DMA,
> + /** Operation failed due to insufficient output buffer */
> + ODP_COMP_ERR_OUT_OF_SPACE,
> +} odp_comp_err_t;
> +
> +/** Comp API enumeration for preferred compression level/speed.
> +*
> +* trade-off between speed and compression ratio.
> +*
> +* Please note this enumeration is only a peferential selection
> +* but may not guarantee operation to committed level depending
> +* on implementation support. Ex. SPEED_FASTEST and SPEED_FAST may
> +* give same result on some platforms.
> +*
> +*/
> +typedef enum {
> + /* Use implementaion default between ratio and speed */
> + ODP_COMP_SPEED_DEFAULT,
> + /** fastest speed, lowest compression */
> + ODP_COMP_SPEED_FASTEST,
> + /** fast speed, lower compression */
> + ODP_COMP_SPEED_FAST,
> + /** medium speed, medium compression */
> + ODP_COMP_SPEED_MED,
> + /** slowest speed, maximum compression */
> + ODP_COMP_SPEED_SLOWEST,
> +
> +} odp_comp_speed_t;
> +
> +/** Comp API enumeration for encoding type
> +*
> +*/
> +typedef enum {
> + /** use implementation default to choose between compression codes */
> + ODP_COMP_CODE_DEFAULT,
> + /** use fixed huffman codes */
> + ODP_COMP_CODE_FIX_HUFFMAN,
> + /** use dynamic huffman coding */
> + ODP_COMP_CODE_DYN_HUFFMAN,
> +} odp_comp_code_t;
> +
> +/**
> + * Comp algorithms in a bit field structure
> + *
> + */
> +typedef union odp_comp_algos_t {
> + /** Compression algorithms */
> + struct {
> + /** ODP_COMP_ALG_NULL */
> + uint32_t null : 1;
> +
> + /** ODP_COMP_ALG_DEFLATE */
> + uint32_t deflate : 1;
> +
> + /** ODP_COMP_ALG_ZLIB */
> + uint32_t zlib : 1;
> +
> + /** ODP_COMP_ALG_LZS */
> + uint32_t lzs :1;
> +
> + /** ODP_COMP_ALG_SHA1 */
> + uint32_t sha1 :1;
> +
> + /** ODP_COMP_ALG_SHA256 */
> + uint32_t sha256 :1;
> +
> + /** ODP_COMP_ALG_DEFLATE_SHA1 */
> + uint32_t deflate_sha1:1;
> +
> + /** ODP_COMP_ALG_DEFLATE_SHA256 */
> + uint32_t deflate_sha256:1;
> +
> + /** ODP_COMP_ALG_ZLIB_SHA1 */
> + uint32_t zlib_sha1:1;
> +
> + /** ODP_COMP_ALG_ZLIB_SHA256 */
> + uint32_t zlib_sha256:1;
> +
> + /** ODP_COMP_ALG_LZS */
> + uint32_t lzs_sha1:1;
> +
> + /** ODP_COMP_ALG_LZS */
> + uint32_t lzs_sha256:1;
> + } bit;
> +
> + /** All bits of the bit field structure
> + *
> + * This field can be used to set/clear all flags, or bitwise
> + * operations over the entire structure. */
> + uint32_t all_bits;
> +} odp_comp_algos_t;
> +
> +/**
> + * Compression Interface Capabilities
> + *
> + */
> +typedef struct odp_comp_capability_t {
> + /** Maximum number of sessions */
> + uint32_t max_sessions;
> +
> + /** Supported compression algorithms */
> + odp_comp_algos_t comps;
> +
> +} odp_comp_capability_t;
> +
> +/**
> + * Compression algorithm capabilities
> + * structure for each algorithm.
> + *
> + */
> +typedef struct odp_comp_alg_capability_t {
> + /** Boolean indicating alg support dictionary load
> + *
> + * true: yes
> + * false : no
> + *
> + */
> + odp_bool_t support_dict;
> +
> + /** Optional Maximum length of dictionary supported
> + * by implementation of the algorithm.
> + *
> + * Invalid if support_dict == false.
> + *
> + * Implementation not supporting this may still allow a dictionary
> + * load during odp_comp_operation() where Implementation may
> + * use allowed length. In such case, implementation keep
> responsibility
> + * to update user discarded dictionary length.
> + *
> + */
> + uint32_t dict_len;
> + /** key size in bytes for hash */
> + uint32_t key_size;
> +
> + /** Digest length in bytes for hash */
> + uint32_t digest_len;
> +
> + /** NULL-terminated Optional identifier of the algo
> (platform-dependent)
> + */
> + void *name;
> +
> + /** NULL-terminated optional description message (platform-dependent)
> */
> + void *description;
> +
> +} odp_comp_alg_capability_t;
> +
> +/**
> + * Comp API data range specifier
> + */
> +typedef struct odp_comp_data_range_t {
> + /** Offset from beginning of data pointer */
> + uint32_t offset;
> +
> + /** Length of data to operate on */
> + uint32_t length;
> +
> +} odp_comp_data_range_t;
> +
> +/**
> + * Comp API key structure
> + */
> +typedef struct odp_comp_auth_key_t {
> + /** Key data */
> + uint8_t *data;
> +
> + /** Key length in bytes */
> + uint32_t length;
> +} odp_comp_auth_key_t;
> +
> + /**
> + * Comp API session creation parameters */ typedef struct
> +odp_comp_session_param_t {
> + /** Compress vs. Decompress operation */
> + odp_comp_op_t op;
> +
> + /** Preferred sync vs. async
> + *
> + * Indicate user preferred mode of operartion.
> + * Implementation may still choose sync or async depending
> + * upon which one gives better performance in a particular
> + * operational environment
> + *
> + */
> + odp_comp_op_mode_t pref_mode;
> +
> + /** force the requested op_mode.
> + *
> + * Indicates user request to enforce requested
> + * operation mode.
> + */
> + odp_bool_t force_mode;
> +
> + /** Compression algorithm
> + *
> + * Use odp_comp_capability() for supported algorithms.
> + */
> + odp_comp_alg_t comp_alg;
> +
> + /** Compression preferred encoding mode
> + *
> + */
> +
> + odp_comp_code_t comp_code;
> +
> + /** Compression preferred speed
> + *
> + */
> + odp_comp_speed_t comp_speed;
> +
> + /** auth parameters during session creation
> + * These will remain valid till session life duration.
> + *
> + */
> + odp_comp_auth_key_t auth_key;
> +
> + /** Async mode completion event queue
> + *
> + * When odp_comp_operation() returns result with 'posted' set to
> 'true',
> + * the completion queue is
> + * used to return the completion status of the operation to the
> + * application.
> + */
> + odp_queue_t compl_queue;
> +
> + /** Pool used by user for input/output data
> + *
> + * used by implementation to have user pool information.
> + * Helps to map input/output data pointer to appropriate type
> + * and other pool related operations.
> + *
> + */
> +
> + odp_pool_t pool;
> +} odp_comp_session_param_t;
> +
> +/**
> + * Comp API operation parameters.
> + * Called to process each data unit
> + * Data unit can be packet type/buffer type
> + * determined based on pool used by app
> + *
> + */
> +typedef struct odp_comp_op_param_t {
> + /** Session handle from creation */
> + odp_comp_session_t session;
> +
> + /** User context */
> + void *ctx;
> +
> + /** Boolean indicating End of data
> + * for stateless compressions (ex ipcomp), it should always be 'true'
> + *
> + * true : last chunk
> + * false: more to follow
> + *
> + */
> + odp_bool_t last;
> +
> + /** Input packet */
> + odp_packet_t in_data;
> +
> + /** input data range w.r.t in_data */
> + odp_comp_data_range_t in_data_range;
> +
> + /** Output packet.
> + * Stores both processed and hashed output (for Auth and Comp+Auth
> algo)
> + *
> + * User is expected to pass a valid output packet in
> + * current version of API set.
> + *
> + * In case of output packet is ODP_PACKET_INVALID, then
> + * Auto-allocation and/or in-place support for output packet
> + * to be considered later.
> + *
> + */
> + odp_packet_t out_data;
> +
> + /** output data range w.r.t out_data.
> + * length at input specifies length of available output buffer,
> + * offset marks beginning of output buffer.
> + * This may be updated after operation completion.
> + *
> + */
> + odp_comp_data_range_t out_data_range;
> +
> + /** dictionary - Optional Pointer to dictionary as passed by user.*/
> + void *dict;
> +
> + /** Length of dictionary */
> + uint32_t dict_len;
> +
> +} odp_comp_op_param_t;
> +
> +/**
> + * Comp API per operation result
> + */
> +typedef struct odp_comp_op_result_t {
> + /** Indicate state of packet operation. Must be checked
> + * at the return of odp_comp_operation()
> + *
> + * TRUE- Submitted but not completed (typical async op)
> + * FALSE - Completed
> + *
> + */
> + odp_bool_t posted;
> +
> + /** User context from request */
> + void *ctx;
> +
> + /** Operation Return Code */
> + odp_comp_err_t err;
> +
> + /** pointer to output buffer.
> + * Contains both processed output and hash for
> + * Authentication and Compression+Authentication
> + * Operations.
> + * Valid when odp_comp_err_t is ODP_COMP_ERR_NONE
> + */
> + odp_packet_t out_data;
> +
> + /** length of processed data */
> + odp_comp_data_range_t out_data_range;
> +
> +} odp_comp_op_result_t;
> +
> +/**
> + * Query comp capabilities
> + *
> + * Outputs comp capabilities on success.
> + *
> + * @param[out] capa Pointer to capability structure for output
> + *
> + * @retval 0 on success
> + * @retval <0 on failure
> + */
> +int odp_comp_capability(odp_comp_capability_t *capa);
> +
> +/**
> + * Query supported compression algorithm capabilities
> + *
> + * Outputs all supported configuration options for the algorithm.
> + *
> + * @param comp Compression algorithm
> + * @param[out] capa Array of capability structures for output
> + * @param num Maximum number of capability structures to output
> + *
> + * @return Number of capability structures for the algorithm. If this is
> larger
> + * than 'num', only 'num' first structures were output and
> application
> + * may call the function again with a larger value of 'num'.
> + * @retval <0 on failure
> + */
> +int odp_comp_alg_capability(odp_comp_alg_t comp,
> + odp_comp_alg_capability_t capa[], int
> +num);
> +
> + /**
> + * Initialize comp session parameters
> + *
> + * Initialize an odp_comp_session_param_t to its default values for
> + * all fields.
> + *
> + * @param param Pointer to odp_comp_session_param_t to be initialized
> + */
> +void odp_comp_session_param_init(odp_comp_session_param_t *param);
> +
> + /**
> + * Compression session creation
> + *
> + * Create a comp session according to the session parameters. Use
> + * odp_comp_session_param_init() to initialize parameters into their
> + * default values.
> + *
> + * @param param Session parameters
> + * @param session Created session else ODP_COMP_SESSION_INVALID
> + * @param status Failure code if unsuccessful
> + *
> + * @retval 0 on success
> + * @retval <0 on failure
> + */
> +int odp_comp_session_create(odp_comp_session_param_t *param,
> + odp_comp_session_t *session,
> + odp_comp_ses_create_err_t *status);
> +
> + /**
> + * Comp session destroy
> + *
> + * Destroy an unused session. Result is undefined if session is being
> +used
> + * (i.e. asynchronous operation is in progress).
> + *
> + * @param session Session handle
> + *
> + * @retval 0 on success
> + * @retval <0 on failure
> + */
> +int odp_comp_session_destroy(odp_comp_session_t session);
> +
> + /**
> + * Compression operation
> + *
> + * Each data operation call will route data
> + * through each components set up in engine. For example if
> + * user created Compression+Authentication(example, zlib_sha1)
> +session, then
> + * data will be passed to both compression and authentication
> + * block.
> + *
> + * After every API call , if result parameter has 'posted' set to
> +'false',
> + * means operation is complete and results are immediately available
> +in
> + * odp_comp_op_result_t.
> + *
> + * If 'posted' set to 'true' , means operation request has been
> +submitted but
> + * not completed. operation result will be delivered by completion queue.
> + *
> + * Application should always check 'posted' regardless of operation
> +mode type
> + * (sync/async) as some implementation may do some operation
> +synchornously even
> + * for async requests.
> + *
> + * if operation returns ODP_COMP_ERR_OUT_OF_SPACE, then application
> +should call
> + * API again with valid output buffer to continue processing of last
> +input
> + * request.
> + *
> + * @param param[in] Operation parameters.
> + * @param result[out] Results of operation. Successful and Completed
> + * call returns length of processed data in result
> + *
> + * @retval 0 on success
> + * @retval <0 on failure
> + */
> +int odp_comp_operation(odp_comp_op_param_t *param,
> + odp_comp_op_result_t *result);
> +
> + /**
> + * Return Compression completion handle that is associated with
> +event
> + *
> + * Note: any invalid parameters will cause undefined behavior and
> +may cause
> + * the application to abort or crash.
> + *
> + * @param ev ODP_EVENT_COMP_COMPL Event Handle
> + *
> + * @return Compression Completion Handle
> + *
> + */
> +odp_comp_compl_t odp_comp_compl_from_event(odp_event_t ev);
> +
> + /**
> + * Query result from completion event
> + *
> + * @param completion_event Event containing operation results
> + * @param result[out] Pointer to result structure
> + *
> + * Example usecase by user
> + *
> + * odp_event_t ev;
> + * odp_comp_op_result_t result;
> + * odp_comp_compl_t compl;
> + * odp_queue_t ev_queue;
> + * odp_queue_param_t qparam;
> + *
> + * qparam.type = ODP_QUEUE_TYPE_PLAIN;
> + * ev_queue = odp_queue_create("comp-out", &qparam);
> + *
> + * // do some work.. after it is completed
> + *
> + * ev = odp_queue_deq(ev_queue);
> + *
> + * compl=odp_comp_compl_from_event(ev);
> + *
> + * odp_comp_compl_result(compl,&result);
> + *
> + *
> + */
> +void odp_comp_compl_result(odp_comp_compl_t event,
> + odp_comp_op_result_t *result);
> +
> + /**
> + * Convert comp completion handle to event handle
> + *
> + * @param completion_event Completion event to convert to generic
> +event
> + *
> + * @return Event handle
> + */
> +odp_event_t odp_comp_compl_to_event(odp_comp_compl_t
> +completion_event);
> +
> + /**
> + * Release comp completion event
> + *
> + * @param completion_event Completion event we are done accessing
> + */
> +void odp_comp_compl_free(odp_comp_compl_t completion_event);
> +
> +/**
> + * Get printable value for an odp_comp_session_t
> + *
> + * @param hdl odp_comp_session_t handle to be printed
> + * @return uint64_t value that can be used to print/display this
> + * handle
> + *
> + * @note This routine is intended to be used for diagnostic purposes
> + * to enable applications to generate a printable value that
> +represents
> + * an odp_comp_session_t handle.
> + */
> +uint64_t odp_comp_session_to_u64(odp_comp_session_t hdl);
> +
> +/**
> + * Get printable value for an odp_comp_compl_t
> + *
> + * @param hdl odp_comp_compl_t handle to be printed
> + * @return uint64_t value that can be used to print/display this
> + * handle
> + *
> + * @note This routine is intended to be used for diagnostic purposes
> + * to enable applications to generate a printable value that
> +represents
> + * an odp_comp_compl_t handle.
> + */
> +uint64_t odp_comp_compl_to_u64(odp_comp_compl_t hdl);
> +
> +/**
> + * @}
> + */
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#include <odp/visibility_end.h>
> +#endif
> --
> 1.9.1
>
