Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
On Tue, Aug 29, 2017 at 10:41 PM, Dmitry Eremin-Solenikov wrote: > On 29/08/17 19:18, shally verma wrote: >> Alongside a question of retrieving results for synchronous operations, >> have one more questions - in crypto, I dont see separate output packet >> data range. So, does it mean it writes to same offset and uses same >> length as mentioned for input packet? > > Yes. Aside of padding, encryption does not change data length (and we do > not support padding ATM). Hash is written at hash_result_offset offset. > > -- Ok. and could you also clarify on this " retrieving results for synchronous call odp_crypto_op(const pkt_in[] ...)" . How do user get result per packet after this call is over ? Thanks Shally > With best wishes > Dmitry
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
On 29/08/17 19:18, shally verma wrote: > Alongside a question of retrieving results for synchronous operations, > have one more questions - in crypto, I dont see separate output packet > data range. So, does it mean it writes to same offset and uses same > length as mentioned for input packet? Yes. Aside of padding, encryption does not change data length (and we do not support padding ATM). Hash is written at hash_result_offset offset. -- With best wishes Dmitry
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
On Tue, Aug 29, 2017 at 6:06 PM, shally verma
wrote:
> On Tue, Aug 29, 2017 at 5:02 PM, Savolainen, Petri (Nokia - FI/Espoo)
> wrote:
>>
>>
>>> -Original Message-
>>> From: shally verma [mailto:shallyvermacav...@gmail.com]
>>> Sent: Tuesday, August 29, 2017 10:26 AM
>>> To: Narayana Prasad Athreya
>>> Cc: Savolainen, Petri (Nokia - FI/Espoo) ;
>>> Github ODP bot ; lng-odp@lists.linaro.org; Narayana,
>>> Prasad Athreya ; Mahipal Challa
>>> ; Verma, Shally
>>> Subject: Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
>>>
>>> Based on last discussion, I was reworking to add odp_comp_op_pkt ()
>>> API based on Crypto design. Help me to answer with these questions:
>>>
>>> 1. Current crypto packet base API is not giving Error code as an
>>> output to its sync version i.e. in int odp_crypto_op(const
>>> odp_packet_t pkt_in[],..), I do not see where it is returning
>>> odp_crypto_packet_result_t. Can anyone help?
>>
>> Error codes are part of operation results:
>>
>> /**
>> * Get crypto operation results from an crypto processed packet
>> *
>> * Successful crypto operations of all types (SYNC and ASYNC) produce packets
>> * which contain crypto result metadata. This function copies the operation
>> * results from an crypto processed packet. Event subtype of this kind of
>> * packet is ODP_EVENT_PACKET_crypto. Results are undefined if a non-crypto
>> * processed packet is passed as input.
>> *
>> * @param packet An crypto processed packet
>> (ODP_EVENT_PACKET_CRYPTO)
>> * @param[out]result Pointer to operation result for output
>> *
>> * @retval 0 On success
>> * @retval <0 On failure
>> */
>> int odp_crypto_result(odp_crypto_packet_result_t *result,
>> odp_packet_t packet);
>
> Ok. That seems user need to make explicit call to this API to get
> result, if he want.
> So this is optional call in crypto context?
>
Alongside a question of retrieving results for synchronous operations,
have one more questions - in crypto, I dont see separate output packet
data range. So, does it mean it writes to same offset and uses same
length as mentioned for input packet?
Thanks
Shally
>>
>> /**
>> * Crypto packet API operation result
>> */
>> typedef struct odp_crypto_packet_result_t {
>> /** Request completed successfully */
>> odp_bool_t ok;
>>
>> /** Cipher status */
>> odp_crypto_op_status_t cipher_status;
>>
>> /** Authentication status */
>> odp_crypto_op_status_t auth_status;
>>
>> } odp_crypto_packet_result_t;
>>
>> /**
>> * Cryto API per packet operation completion status
>> */
>> typedef struct odp_crypto_op_status {
>> /** Algorithm specific return code */
>> odp_crypto_alg_err_t alg_err;
>>
>> /** Hardware specific return code */
>> odp_crypto_hw_err_t hw_err;
>>
>> } odp_crypto_op_status_t;
>>
>> /**
>> * Crypto API algorithm return code
>> */
>> typedef enum {
>> /** Algorithm successful */
>> ODP_CRYPTO_ALG_ERR_NONE,
>> /** Invalid data block size */
>> ODP_CRYPTO_ALG_ERR_DATA_SIZE,
>> /** Key size invalid for algorithm */
>> ODP_CRYPTO_ALG_ERR_KEY_SIZE,
>> /** Computed ICV value mismatch */
>> ODP_CRYPTO_ALG_ERR_ICV_CHECK,
>> /** IV value not specified */
>> ODP_CRYPTO_ALG_ERR_IV_INVALID,
>> } odp_crypto_alg_err_t;
>>
>>
>>>
>>> 2. Current crypto version of odp_crypto_op(odp_pkt_t pkt_in[] ...)
>>> does not have two separate version for encryption and decryption where
>>> as in Compression, we added two. One for compress and another for
>>> decompress.
>>> So do we want to retain two separate flavor or unify like crypto
>>> packet based api? Ex.
>>> odp_comp_op_pkt ( ... ) OR
>>> odp_comp_compress_pkt( ...),
>>> odp_comp_decompress_pkt(),
>>> odp_comp_compress_pkt_enq() and so on...?
>>
>> Crypto has single operation, IPSEC has two operations (inbound vs outbound).
>> So, both styles are used today. Benefits of an operation per direction are:
>> * more readable code: odp_comp_compress() vs odp_comp_op()
>> * possibility to have different set of arguments (parameters) for each
>> direction. E.g. IPSEC does IP fragmentation on output direction and thus
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
On Tue, Aug 29, 2017 at 5:02 PM, Savolainen, Petri (Nokia - FI/Espoo)
wrote:
>
>
>> -Original Message-
>> From: shally verma [mailto:shallyvermacav...@gmail.com]
>> Sent: Tuesday, August 29, 2017 10:26 AM
>> To: Narayana Prasad Athreya
>> Cc: Savolainen, Petri (Nokia - FI/Espoo) ;
>> Github ODP bot ; lng-odp@lists.linaro.org; Narayana,
>> Prasad Athreya ; Mahipal Challa
>> ; Verma, Shally
>> Subject: Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
>>
>> Based on last discussion, I was reworking to add odp_comp_op_pkt ()
>> API based on Crypto design. Help me to answer with these questions:
>>
>> 1. Current crypto packet base API is not giving Error code as an
>> output to its sync version i.e. in int odp_crypto_op(const
>> odp_packet_t pkt_in[],..), I do not see where it is returning
>> odp_crypto_packet_result_t. Can anyone help?
>
> Error codes are part of operation results:
>
> /**
> * Get crypto operation results from an crypto processed packet
> *
> * Successful crypto operations of all types (SYNC and ASYNC) produce packets
> * which contain crypto result metadata. This function copies the operation
> * results from an crypto processed packet. Event subtype of this kind of
> * packet is ODP_EVENT_PACKET_crypto. Results are undefined if a non-crypto
> * processed packet is passed as input.
> *
> * @param packet An crypto processed packet (ODP_EVENT_PACKET_CRYPTO)
> * @param[out]result Pointer to operation result for output
> *
> * @retval 0 On success
> * @retval <0 On failure
> */
> int odp_crypto_result(odp_crypto_packet_result_t *result,
> odp_packet_t packet);
Ok. That seems user need to make explicit call to this API to get
result, if he want.
So this is optional call in crypto context?
>
> /**
> * Crypto packet API operation result
> */
> typedef struct odp_crypto_packet_result_t {
> /** Request completed successfully */
> odp_bool_t ok;
>
> /** Cipher status */
> odp_crypto_op_status_t cipher_status;
>
> /** Authentication status */
> odp_crypto_op_status_t auth_status;
>
> } odp_crypto_packet_result_t;
>
> /**
> * Cryto API per packet operation completion status
> */
> typedef struct odp_crypto_op_status {
> /** Algorithm specific return code */
> odp_crypto_alg_err_t alg_err;
>
> /** Hardware specific return code */
> odp_crypto_hw_err_t hw_err;
>
> } odp_crypto_op_status_t;
>
> /**
> * Crypto API algorithm return code
> */
> typedef enum {
> /** Algorithm successful */
> ODP_CRYPTO_ALG_ERR_NONE,
> /** Invalid data block size */
> ODP_CRYPTO_ALG_ERR_DATA_SIZE,
> /** Key size invalid for algorithm */
> ODP_CRYPTO_ALG_ERR_KEY_SIZE,
> /** Computed ICV value mismatch */
> ODP_CRYPTO_ALG_ERR_ICV_CHECK,
> /** IV value not specified */
> ODP_CRYPTO_ALG_ERR_IV_INVALID,
> } odp_crypto_alg_err_t;
>
>
>>
>> 2. Current crypto version of odp_crypto_op(odp_pkt_t pkt_in[] ...)
>> does not have two separate version for encryption and decryption where
>> as in Compression, we added two. One for compress and another for
>> decompress.
>> So do we want to retain two separate flavor or unify like crypto
>> packet based api? Ex.
>> odp_comp_op_pkt ( ... ) OR
>> odp_comp_compress_pkt( ...),
>> odp_comp_decompress_pkt(),
>> odp_comp_compress_pkt_enq() and so on...?
>
> Crypto has single operation, IPSEC has two operations (inbound vs outbound).
> So, both styles are used today. Benefits of an operation per direction are:
> * more readable code: odp_comp_compress() vs odp_comp_op()
> * possibility to have different set of arguments (parameters) for each
> direction. E.g. IPSEC does IP fragmentation on output direction and thus
> needs extra parameters for that, those params are not defined on inbound
> direction.
> * cleaner specification of different operations e.g. "... output of
> odp_comp_compress()..." vs "... output of odp_comp_op() in compress mode "
> * easier to extend since a new feature can be added to one side without
> changing the spec for the other side
>
>
> BTW, since most of our operations process packets, we don't need to highlight
> it with "pkt". I'd name odp_comp_compress() for packets, and then later on
> add odp_comp_compress_mem(), odp_comp_compress_from_mem(), etc for mem ->
> mem, pkt -> mem operations.
Ok. no issues. I will retain separate flavor and keep API name in sync
to crypto.
Thanks
Shally
>
> -Petri
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
> -Original Message-
> From: shally verma [mailto:shallyvermacav...@gmail.com]
> Sent: Tuesday, August 29, 2017 10:26 AM
> To: Narayana Prasad Athreya
> Cc: Savolainen, Petri (Nokia - FI/Espoo) ;
> Github ODP bot ; lng-odp@lists.linaro.org; Narayana,
> Prasad Athreya ; Mahipal Challa
> ; Verma, Shally
> Subject: Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
>
> Based on last discussion, I was reworking to add odp_comp_op_pkt ()
> API based on Crypto design. Help me to answer with these questions:
>
> 1. Current crypto packet base API is not giving Error code as an
> output to its sync version i.e. in int odp_crypto_op(const
> odp_packet_t pkt_in[],..), I do not see where it is returning
> odp_crypto_packet_result_t. Can anyone help?
Error codes are part of operation results:
/**
* Get crypto operation results from an crypto processed packet
*
* Successful crypto operations of all types (SYNC and ASYNC) produce packets
* which contain crypto result metadata. This function copies the operation
* results from an crypto processed packet. Event subtype of this kind of
* packet is ODP_EVENT_PACKET_crypto. Results are undefined if a non-crypto
* processed packet is passed as input.
*
* @param packet An crypto processed packet (ODP_EVENT_PACKET_CRYPTO)
* @param[out]result Pointer to operation result for output
*
* @retval 0 On success
* @retval <0 On failure
*/
int odp_crypto_result(odp_crypto_packet_result_t *result,
odp_packet_t packet);
/**
* Crypto packet API operation result
*/
typedef struct odp_crypto_packet_result_t {
/** Request completed successfully */
odp_bool_t ok;
/** Cipher status */
odp_crypto_op_status_t cipher_status;
/** Authentication status */
odp_crypto_op_status_t auth_status;
} odp_crypto_packet_result_t;
/**
* Cryto API per packet operation completion status
*/
typedef struct odp_crypto_op_status {
/** Algorithm specific return code */
odp_crypto_alg_err_t alg_err;
/** Hardware specific return code */
odp_crypto_hw_err_t hw_err;
} odp_crypto_op_status_t;
/**
* Crypto API algorithm return code
*/
typedef enum {
/** Algorithm successful */
ODP_CRYPTO_ALG_ERR_NONE,
/** Invalid data block size */
ODP_CRYPTO_ALG_ERR_DATA_SIZE,
/** Key size invalid for algorithm */
ODP_CRYPTO_ALG_ERR_KEY_SIZE,
/** Computed ICV value mismatch */
ODP_CRYPTO_ALG_ERR_ICV_CHECK,
/** IV value not specified */
ODP_CRYPTO_ALG_ERR_IV_INVALID,
} odp_crypto_alg_err_t;
>
> 2. Current crypto version of odp_crypto_op(odp_pkt_t pkt_in[] ...)
> does not have two separate version for encryption and decryption where
> as in Compression, we added two. One for compress and another for
> decompress.
> So do we want to retain two separate flavor or unify like crypto
> packet based api? Ex.
> odp_comp_op_pkt ( ... ) OR
> odp_comp_compress_pkt( ...),
> odp_comp_decompress_pkt(),
> odp_comp_compress_pkt_enq() and so on...?
Crypto has single operation, IPSEC has two operations (inbound vs outbound).
So, both styles are used today. Benefits of an operation per direction are:
* more readable code: odp_comp_compress() vs odp_comp_op()
* possibility to have different set of arguments (parameters) for each
direction. E.g. IPSEC does IP fragmentation on output direction and thus needs
extra parameters for that, those params are not defined on inbound direction.
* cleaner specification of different operations e.g. "... output of
odp_comp_compress()..." vs "... output of odp_comp_op() in compress mode "
* easier to extend since a new feature can be added to one side without
changing the spec for the other side
BTW, since most of our operations process packets, we don't need to highlight
it with "pkt". I'd name odp_comp_compress() for packets, and then later on add
odp_comp_compress_mem(), odp_comp_compress_from_mem(), etc for mem -> mem, pkt
-> mem operations.
-Petri
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
Based on last discussion, I was reworking to add odp_comp_op_pkt ()
API based on Crypto design. Help me to answer with these questions:
1. Current crypto packet base API is not giving Error code as an
output to its sync version i.e. in int odp_crypto_op(const
odp_packet_t pkt_in[],..), I do not see where it is returning
odp_crypto_packet_result_t. Can anyone help?
2. Current crypto version of odp_crypto_op(odp_pkt_t pkt_in[] ...)
does not have two separate version for encryption and decryption where
as in Compression, we added two. One for compress and another for
decompress.
So do we want to retain two separate flavor or unify like crypto
packet based api? Ex.
odp_comp_op_pkt ( ... ) OR
odp_comp_compress_pkt( ...),
odp_comp_decompress_pkt(),
odp_comp_compress_pkt_enq() and so on...?
Thanks
Shally
On Tue, Aug 8, 2017 at 11:36 PM, Narayana Prasad Athreya
wrote:
> Can someone explain what is the use-case of #1 and why the use-case cannot
> be met with proposed API?
>
> PRasad
>
> On Tuesday 08 August 2017 10:21 PM, shally verma wrote:
>
> Petri/Berry
>
> As per discussion in today's call, this is what I summarize :
>
> Two new requirements added:
>
> 1. Support compression / decompression of multiple ranges with in one
> single packet and
> 2. Operating on multiple packets with in call where each packet may
> further carry multiple range.
>
> To meet 1st requirement, current proposal says, calling
> odp_packet_compress/decomp() for each packet per range as it allows
> easy and clean management of output buffer and also more flexibility
> per various possible application use cases:.
> - call can be made in statefull or stateless mode per each range, Or
> - Modify headers according to each range output,
> - Application manage output buffers at their end,
> - Async notification on out of space condition will be easily maintained.
>
> Currently , we do support file-based compression using ODP APIs, but
> i.e. on stream of data bytes where 1 chunk occupy - 1 packet (which
> can be segmented/unsegmented) not on stream of Packets.
> Is there a use-case to have a file which consists of stream of
> Packets? (where packet is of type HTTP Packet ?)
>
> New proposal says "allow multiple ranges with in one single call". Few
> concerns raised for this, key one include:
>
> - How do we know each range be compressed in stateful or stateless
> mode i.e. each range is independent Or dependent?
> - How do we handle out_of_space error while operating on individual
> range? Especially for the case when Implementation try to use HW
> parallelization for better throughput?
> - How do we support same design in asynchronous mode ??
> - How do we see it - as performance gain ? Or ease-of-use? Or it may
> end up introducing more overhead to implementation.
>
> Please feedback your inputs to design issues as envisioned.
>
> For now we can focus on requirement #1 as design for requirement # 2
> will be based on outcome of #1.
>
> Thanks
> Shally
>
> On Tue, Aug 8, 2017 at 6:38 PM, Savolainen, Petri (Nokia - FI/Espoo)
> wrote:
>
> -Original Message-
> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
> Github ODP bot
> Sent: Friday, August 04, 2017 4:00 PM
> To: lng-odp@lists.linaro.org
> Subject: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
>
> From: Shally Verma
>
> Signed-off-by: Shally Verma
> Signed-off-by: Mahipal Challa Cc
> prasadathreya.naray...@cavium.com
> ---
> /** Email created from pull request 102 (1234sv:api-next)
> ** https://github.com/Linaro/odp/pull/102
> ** Patch: https://github.com/Linaro/odp/pull/102.patch
> ** Base sha: 8390f890d4bd2babb63a24f7b15d2f4763e44050
> ** Merge commit sha: fbdff8c82a19f5b640ae299204b3bb1bbbefdccb
> **/
> include/odp/api/spec/comp.h | 815
>
> 1 file changed, 815 insertions(+)
> create mode 100644 include/odp/api/spec/comp.h
>
> diff --git a/include/odp/api/spec/comp.h b/include/odp/api/spec/comp.h
> new file mode 100644
> index ..2956094c
> --- /dev/null
> +++ b/include/odp/api/spec/comp.h
> @@ -0,0 +1,815 @@
> +/* Copyright (c) 2013, Linaro Limited
>
> Year 2017
>
> +
> +/**
> + * Comp API hash algorithm
> + *
> + */
> +typedef enum {
> + /** ODP_COMP_HASH_ALG_NONE*/
>
> This kind of comment is not very helpful. Each enumeration needs explanation
> - like odp_comp_alg_t under.
>
> + ODP_COMP_HASH_ALG_NONE,
> + /** ODP_COMP_HASH_ALG_SHA1*/
> + ODP_COMP_HASH_ALG_SHA1,
> + /** ODP_COMP_HASH_ALG_SHA256*/
> + ODP_COMP_HASH_ALG_SHA256
> +} odp_comp_hash_alg_t;
> +
> +/**
> + * Comp API compression algorithm
>
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
Can someone explain what is the use-case of #1 and why the use-case
cannot be met with proposed API?
PRasad
On Tuesday 08 August 2017 10:21 PM, shally verma wrote:
Petri/Berry
As per discussion in today's call, this is what I summarize :
Two new requirements added:
1. Support compression / decompression of multiple ranges with in one
single packet and
2. Operating on multiple packets with in call where each packet may
further carry multiple range.
To meet 1st requirement, current proposal says, calling
odp_packet_compress/decomp() for each packet per range as it allows
easy and clean management of output buffer and also more flexibility
per various possible application use cases:.
- call can be made in statefull or stateless mode per each range, Or
- Modify headers according to each range output,
- Application manage output buffers at their end,
- Async notification on out of space condition will be easily maintained.
Currently , we do support file-based compression using ODP APIs, but
i.e. on stream of data bytes where 1 chunk occupy - 1 packet (which
can be segmented/unsegmented) not on stream of Packets.
Is there a use-case to have a file which consists of stream of
Packets? (where packet is of type HTTP Packet ?)
New proposal says "allow multiple ranges with in one single call". Few
concerns raised for this, key one include:
- How do we know each range be compressed in stateful or stateless
mode i.e. each range is independent Or dependent?
- How do we handle out_of_space error while operating on individual
range? Especially for the case when Implementation try to use HW
parallelization for better throughput?
- How do we support same design in asynchronous mode ??
- How do we see it - as performance gain ? Or ease-of-use? Or it may
end up introducing more overhead to implementation.
Please feedback your inputs to design issues as envisioned.
For now we can focus on requirement #1 as design for requirement # 2
will be based on outcome of #1.
Thanks
Shally
On Tue, Aug 8, 2017 at 6:38 PM, Savolainen, Petri (Nokia - FI/Espoo)
wrote:
-Original Message-
From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
Github ODP bot
Sent: Friday, August 04, 2017 4:00 PM
To: lng-odp@lists.linaro.org
Subject: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
From: Shally Verma
Signed-off-by: Shally Verma
Signed-off-by: Mahipal Challa Cc
prasadathreya.naray...@cavium.com
---
/** Email created from pull request 102 (1234sv:api-next)
** https://github.com/Linaro/odp/pull/102
** Patch: https://github.com/Linaro/odp/pull/102.patch
** Base sha: 8390f890d4bd2babb63a24f7b15d2f4763e44050
** Merge commit sha: fbdff8c82a19f5b640ae299204b3bb1bbbefdccb
**/
include/odp/api/spec/comp.h | 815
1 file changed, 815 insertions(+)
create mode 100644 include/odp/api/spec/comp.h
diff --git a/include/odp/api/spec/comp.h b/include/odp/api/spec/comp.h
new file mode 100644
index ..2956094c
--- /dev/null
+++ b/include/odp/api/spec/comp.h
@@ -0,0 +1,815 @@
+/* Copyright (c) 2013, Linaro Limited
Year 2017
+
+/**
+ * Comp API hash algorithm
+ *
+ */
+typedef enum {
+ /** ODP_COMP_HASH_ALG_NONE*/
This kind of comment is not very helpful. Each enumeration needs explanation -
like odp_comp_alg_t under.
+ ODP_COMP_HASH_ALG_NONE,
+ /** ODP_COMP_HASH_ALG_SHA1*/
+ ODP_COMP_HASH_ALG_SHA1,
+ /** ODP_COMP_HASH_ALG_SHA256*/
+ ODP_COMP_HASH_ALG_SHA256
+} odp_comp_hash_alg_t;
+
+/**
+ * Comp API compression algorithm
+ *
+ */
+typedef enum {
+ /** No algorithm specified.
+ * Means no compression, output == input.
+ * if provided, no operation (compression/decompression or
hash)
+ * applied on input. Added for testing purpose.
+ */
+ ODP_COMP_ALG_NULL,
+ /** DEFLATE - RFC1951 */
+ ODP_COMP_ALG_DEFLATE,
+ /** ZLIB - RFC1950 */
+ ODP_COMP_ALG_ZLIB,
+ /** LZS */
+ ODP_COMP_ALG_LZS
+} odp_comp_alg_t;
+
+
+/**
+ * Hash algorithms in a bit field structure
+ *
+ */
+typedef union odp_comp_hash_algos_t {
+ /** hash algorithms */
+ struct {
+ /** SHA-1 */
+ uint32_t sha1 : 1;
+
+ /** SHA with 256 bits of Message Digest */
+ uint32_t sha256 : 1;
Need to be more explicit in algorithm definition: SHA-1, SHA-256, ... algorithm
(SHA-2 would also do, but we use SHA-256 in crypto API since it seems to be
used by standards).
Actually, these explanations should go under enum definitions and then just
refer to those enums here - like odp_comp_algos_t under.
+
+ } bit;
+
+ /** All bits of the bit field structure
+ *
+ * This field can be used to set/clear all flags, or bitwise
+ * operations over the entire structure.
+ */
+ uint32_t all_bits;
+} odp_comp_hash_algos_t;
+
+/**
+ * Comp algorithms in a bit field structure
+ *
+ */
+typedef union o
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
Just a Resend.
On Tue, Aug 8, 2017 at 10:21 PM, shally verma
wrote:
> Petri/Berry
>
> As per discussion in today's call, this is what I summarize :
>
> Two new requirements added:
>
> 1. Support compression / decompression of multiple ranges with in one
> single packet and
> 2. Operating on multiple packets with in call where each packet may
> further carry multiple range.
>
> To meet 1st requirement, current proposal says, calling
> odp_packet_compress/decomp() for each packet per range as it allows
> easy and clean management of output buffer and also more flexibility
> per various possible application use cases:.
> - call can be made in statefull or stateless mode per each range, Or
> - Modify headers according to each range output,
> - Application manage output buffers at their end,
> - Async notification on out of space condition will be easily maintained.
>
> Currently , we do support file-based compression using ODP APIs, but
> i.e. on stream of data bytes where 1 chunk occupy - 1 packet (which
> can be segmented/unsegmented) not on stream of Packets.
> Is there a use-case to have a file which consists of stream of
> Packets? (where packet is of type HTTP Packet ?)
>
> New proposal says "allow multiple ranges with in one single call". Few
> concerns raised for this, key one include:
>
> - How do we know each range be compressed in stateful or stateless
> mode i.e. each range is independent Or dependent?
> - How do we handle out_of_space error while operating on individual
> range? Especially for the case when Implementation try to use HW
> parallelization for better throughput?
> - How do we support same design in asynchronous mode ??
> - How do we see it - as performance gain ? Or ease-of-use? Or it may
> end up introducing more overhead to implementation.
>
> Please feedback your inputs to design issues as envisioned.
>
> For now we can focus on requirement #1 as design for requirement # 2
> will be based on outcome of #1.
>
> Thanks
> Shally
>
> On Tue, Aug 8, 2017 at 6:38 PM, Savolainen, Petri (Nokia - FI/Espoo)
> wrote:
>>
>>
>>> -Original Message-
>>> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
>>> Github ODP bot
>>> Sent: Friday, August 04, 2017 4:00 PM
>>> To: lng-odp@lists.linaro.org
>>> Subject: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
>>>
>>> From: Shally Verma
>>>
>>> Signed-off-by: Shally Verma
>>> Signed-off-by: Mahipal Challa Cc
>>> prasadathreya.naray...@cavium.com
>>> ---
>>> /** Email created from pull request 102 (1234sv:api-next)
>>> ** https://github.com/Linaro/odp/pull/102
>>> ** Patch: https://github.com/Linaro/odp/pull/102.patch
>>> ** Base sha: 8390f890d4bd2babb63a24f7b15d2f4763e44050
>>> ** Merge commit sha: fbdff8c82a19f5b640ae299204b3bb1bbbefdccb
>>> **/
>>> include/odp/api/spec/comp.h | 815
>>>
>>> 1 file changed, 815 insertions(+)
>>> create mode 100644 include/odp/api/spec/comp.h
>>>
>>> diff --git a/include/odp/api/spec/comp.h b/include/odp/api/spec/comp.h
>>> new file mode 100644
>>> index ..2956094c
>>> --- /dev/null
>>> +++ b/include/odp/api/spec/comp.h
>>> @@ -0,0 +1,815 @@
>>> +/* Copyright (c) 2013, Linaro Limited
>>
>> Year 2017
>>
>>> +
>>> +/**
>>> + * Comp API hash algorithm
>>> + *
>>> + */
>>> +typedef enum {
>>> + /** ODP_COMP_HASH_ALG_NONE*/
>>
>> This kind of comment is not very helpful. Each enumeration needs explanation
>> - like odp_comp_alg_t under.
>>
>>> + ODP_COMP_HASH_ALG_NONE,
>>> + /** ODP_COMP_HASH_ALG_SHA1*/
>>> + ODP_COMP_HASH_ALG_SHA1,
>>> + /** ODP_COMP_HASH_ALG_SHA256*/
>>> + ODP_COMP_HASH_ALG_SHA256
>>> +} odp_comp_hash_alg_t;
>>> +
>>> +/**
>>> + * Comp API compression algorithm
>>> + *
>>> + */
>>> +typedef enum {
>>> + /** No algorithm specified.
>>> + * Means no compression, output == input.
>>> + * if provided, no operation (compression/decompression or
>>> hash)
>>> + * applied on input. Added for testing purpose.
>>> + */
>>> + ODP_COMP_ALG_NULL,
>>> + /** DEFLATE - RFC1951 */
>>> + ODP_COMP_ALG_DEFLATE,
>>> + /** ZLIB - RFC1950 */
>>> + ODP_COMP_ALG_ZLIB,
>>> + /** LZS */
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
Petri/Berry
As per discussion in today's call, this is what I summarize :
Two new requirements added:
1. Support compression / decompression of multiple ranges with in one
single packet and
2. Operating on multiple packets with in call where each packet may
further carry multiple range.
To meet 1st requirement, current proposal says, calling
odp_packet_compress/decomp() for each packet per range as it allows
easy and clean management of output buffer and also more flexibility
per various possible application use cases:.
- call can be made in statefull or stateless mode per each range, Or
- Modify headers according to each range output,
- Application manage output buffers at their end,
- Async notification on out of space condition will be easily maintained.
Currently , we do support file-based compression using ODP APIs, but
i.e. on stream of data bytes where 1 chunk occupy - 1 packet (which
can be segmented/unsegmented) not on stream of Packets.
Is there a use-case to have a file which consists of stream of
Packets? (where packet is of type HTTP Packet ?)
New proposal says "allow multiple ranges with in one single call". Few
concerns raised for this, key one include:
- How do we know each range be compressed in stateful or stateless
mode i.e. each range is independent Or dependent?
- How do we handle out_of_space error while operating on individual
range? Especially for the case when Implementation try to use HW
parallelization for better throughput?
- How do we support same design in asynchronous mode ??
- How do we see it - as performance gain ? Or ease-of-use? Or it may
end up introducing more overhead to implementation.
Please feedback your inputs to design issues as envisioned.
For now we can focus on requirement #1 as design for requirement # 2
will be based on outcome of #1.
Thanks
Shally
On Tue, Aug 8, 2017 at 6:38 PM, Savolainen, Petri (Nokia - FI/Espoo)
wrote:
>
>
>> -Original Message-
>> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
>> Github ODP bot
>> Sent: Friday, August 04, 2017 4:00 PM
>> To: lng-odp@lists.linaro.org
>> Subject: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
>>
>> From: Shally Verma
>>
>> Signed-off-by: Shally Verma
>> Signed-off-by: Mahipal Challa Cc
>> prasadathreya.naray...@cavium.com
>> ---
>> /** Email created from pull request 102 (1234sv:api-next)
>> ** https://github.com/Linaro/odp/pull/102
>> ** Patch: https://github.com/Linaro/odp/pull/102.patch
>> ** Base sha: 8390f890d4bd2babb63a24f7b15d2f4763e44050
>> ** Merge commit sha: fbdff8c82a19f5b640ae299204b3bb1bbbefdccb
>> **/
>> include/odp/api/spec/comp.h | 815
>>
>> 1 file changed, 815 insertions(+)
>> create mode 100644 include/odp/api/spec/comp.h
>>
>> diff --git a/include/odp/api/spec/comp.h b/include/odp/api/spec/comp.h
>> new file mode 100644
>> index ..2956094c
>> --- /dev/null
>> +++ b/include/odp/api/spec/comp.h
>> @@ -0,0 +1,815 @@
>> +/* Copyright (c) 2013, Linaro Limited
>
> Year 2017
>
>> +
>> +/**
>> + * Comp API hash algorithm
>> + *
>> + */
>> +typedef enum {
>> + /** ODP_COMP_HASH_ALG_NONE*/
>
> This kind of comment is not very helpful. Each enumeration needs explanation
> - like odp_comp_alg_t under.
>
>> + ODP_COMP_HASH_ALG_NONE,
>> + /** ODP_COMP_HASH_ALG_SHA1*/
>> + ODP_COMP_HASH_ALG_SHA1,
>> + /** ODP_COMP_HASH_ALG_SHA256*/
>> + ODP_COMP_HASH_ALG_SHA256
>> +} odp_comp_hash_alg_t;
>> +
>> +/**
>> + * Comp API compression algorithm
>> + *
>> + */
>> +typedef enum {
>> + /** No algorithm specified.
>> + * Means no compression, output == input.
>> + * if provided, no operation (compression/decompression or
>> hash)
>> + * applied on input. Added for testing purpose.
>> + */
>> + ODP_COMP_ALG_NULL,
>> + /** DEFLATE - RFC1951 */
>> + ODP_COMP_ALG_DEFLATE,
>> + /** ZLIB - RFC1950 */
>> + ODP_COMP_ALG_ZLIB,
>> + /** LZS */
>> + ODP_COMP_ALG_LZS
>> +} odp_comp_alg_t;
>> +
>> +
>> +/**
>> + * Hash algorithms in a bit field structure
>> + *
>> + */
>> +typedef union odp_comp_hash_algos_t {
>> + /** hash algorithms */
>> + struct {
>> + /** SHA-1 */
>> + uint32_t sha1 : 1;
>> +
>> + /** SHA with 256 bits of Message Digest */
>> + uint32_t sha256 : 1;
>
>
> Need to be more explicit in algorithm definition: SHA-1, SHA-256, ...
> algori
Re: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
> -Original Message-
> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
> Github ODP bot
> Sent: Friday, August 04, 2017 4:00 PM
> To: lng-odp@lists.linaro.org
> Subject: [lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
>
> From: Shally Verma
>
> Signed-off-by: Shally Verma
> Signed-off-by: Mahipal Challa Cc
> prasadathreya.naray...@cavium.com
> ---
> /** Email created from pull request 102 (1234sv:api-next)
> ** https://github.com/Linaro/odp/pull/102
> ** Patch: https://github.com/Linaro/odp/pull/102.patch
> ** Base sha: 8390f890d4bd2babb63a24f7b15d2f4763e44050
> ** Merge commit sha: fbdff8c82a19f5b640ae299204b3bb1bbbefdccb
> **/
> include/odp/api/spec/comp.h | 815
>
> 1 file changed, 815 insertions(+)
> create mode 100644 include/odp/api/spec/comp.h
>
> diff --git a/include/odp/api/spec/comp.h b/include/odp/api/spec/comp.h
> new file mode 100644
> index ..2956094c
> --- /dev/null
> +++ b/include/odp/api/spec/comp.h
> @@ -0,0 +1,815 @@
> +/* Copyright (c) 2013, Linaro Limited
Year 2017
> +
> +/**
> + * Comp API hash algorithm
> + *
> + */
> +typedef enum {
> + /** ODP_COMP_HASH_ALG_NONE*/
This kind of comment is not very helpful. Each enumeration needs explanation -
like odp_comp_alg_t under.
> + ODP_COMP_HASH_ALG_NONE,
> + /** ODP_COMP_HASH_ALG_SHA1*/
> + ODP_COMP_HASH_ALG_SHA1,
> + /** ODP_COMP_HASH_ALG_SHA256*/
> + ODP_COMP_HASH_ALG_SHA256
> +} odp_comp_hash_alg_t;
> +
> +/**
> + * Comp API compression algorithm
> + *
> + */
> +typedef enum {
> + /** No algorithm specified.
> + * Means no compression, output == input.
> + * if provided, no operation (compression/decompression or
> hash)
> + * applied on input. Added for testing purpose.
> + */
> + ODP_COMP_ALG_NULL,
> + /** DEFLATE - RFC1951 */
> + ODP_COMP_ALG_DEFLATE,
> + /** ZLIB - RFC1950 */
> + ODP_COMP_ALG_ZLIB,
> + /** LZS */
> + ODP_COMP_ALG_LZS
> +} odp_comp_alg_t;
> +
> +
> +/**
> + * Hash algorithms in a bit field structure
> + *
> + */
> +typedef union odp_comp_hash_algos_t {
> + /** hash algorithms */
> + struct {
> + /** SHA-1 */
> + uint32_t sha1 : 1;
> +
> + /** SHA with 256 bits of Message Digest */
> + uint32_t sha256 : 1;
Need to be more explicit in algorithm definition: SHA-1, SHA-256, ... algorithm
(SHA-2 would also do, but we use SHA-256 in crypto API since it seems to be
used by standards).
Actually, these explanations should go under enum definitions and then just
refer to those enums here - like odp_comp_algos_t under.
> +
> + } bit;
> +
> + /** All bits of the bit field structure
> + *
> + * This field can be used to set/clear all flags, or bitwise
> + * operations over the entire structure.
> + */
> + uint32_t all_bits;
> +} odp_comp_hash_algos_t;
> +
> +/**
> + * Comp algorithms in a bit field structure
> + *
> + */
> +typedef union odp_comp_algos_t {
> + /** Compression algorithms */
> + struct {
> + /** ODP_COMP_ALG_NULL */
> + uint32_t null : 1;
> +
> + /** ODP_COMP_ALG_DEFLATE */
> + uint32_t deflate: 1;
> +
> + /** ODP_COMP_ALG_ZLIB */
> + uint32_t zlib : 1;
> +
> + /** ODP_COMP_ALG_LZS */
> + uint32_t lzs:1;
> + } bit;
> +
> + /** All bits of the bit field structure
> + * This field can be used to set/clear all flags, or bitwise
> + * operations over the entire structure.
> + */
> + uint32_t all_bits;
> +} odp_comp_algos_t;
> +
> +/**
> + * Compression Interface Capabilities
> + *
> + */
> +typedef struct odp_comp_capability_t {
> + /** Maximum number of sessions */
> + uint32_t max_sessions;
> +
> + /** Supported compression algorithms */
> + odp_comp_algos_t comp_algs;
No need to save one character => comp_algos
> +
> + /** Supported hash algorithms. */
> + odp_comp_hash_algos_t hash_algs;
hash_algos
> +
> + /* sync/async mode of operation support.
> + * Implementation should support atleast one of the mode.
> + */
"mode" field definition missing on this line ?
> +
> + /** Support type for synchronous operation mode
> (ODP_COMP_SYNC).
> + * User should set odp_comp_session_param_t:mode based on
> + * support level as indicated by this param.
> + */
> + odp_support_t sync;
> +
&g
[lng-odp] [PATCH API-NEXT v8 1/1] comp: compression spec
From: Shally Verma
Signed-off-by: Shally Verma
Signed-off-by: Mahipal Challa Cc
prasadathreya.naray...@cavium.com
---
/** Email created from pull request 102 (1234sv:api-next)
** https://github.com/Linaro/odp/pull/102
** Patch: https://github.com/Linaro/odp/pull/102.patch
** Base sha: 8390f890d4bd2babb63a24f7b15d2f4763e44050
** Merge commit sha: fbdff8c82a19f5b640ae299204b3bb1bbbefdccb
**/
include/odp/api/spec/comp.h | 815
1 file changed, 815 insertions(+)
create mode 100644 include/odp/api/spec/comp.h
diff --git a/include/odp/api/spec/comp.h b/include/odp/api/spec/comp.h
new file mode 100644
index ..2956094c
--- /dev/null
+++ b/include/odp/api/spec/comp.h
@@ -0,0 +1,815 @@
+/* Copyright (c) 2013, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier:BSD-3-Clause
+ */
+
+/**
+ * @file
+ *
+ * ODP Compression
+ */
+
+#ifndef ODP_API_COMP_H_
+#define ODP_API_COMP_H_
+
+#include
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include
+#include
+
+/** @defgroup odp_compression ODP COMP
+ * ODP Compression defines API set to compress/decompress along with hash
+ * operations on data. Hash is calculated on plaintext.
+ *
+ * if opcode = ODP_COMP_COMPRESS, then it will Compress and apply hash,
+ * if opcode = ODP_COMP_DECOMPRESS, then it will Decompress and apply
+ * hash.
+ * Independent hash-only operations are not supported. Implementation should
+ * perform hash along with valid compression algo.
+ * Macros, enums, types and operations to utilize compression.
+ * @{
+ */
+
+/**
+ * @def ODP_COMP_SESSION_INVALID
+ * Invalid session handle
+ */
+
+/**
+ * @typedef odp_comp_session_t
+ * Compression/Decompression session handle
+ */
+
+/**
+ * Compression API operation mode
+ */
+typedef enum {
+ /** Synchronous, return results immediately */
+ ODP_COMP_SYNC,
+ /** Asynchronous, return results via event queue */
+ ODP_COMP_ASYNC
+} odp_comp_op_mode_t;
+
+/**
+ * Comp API operation type.
+ *
+ */
+typedef enum {
+ /** Compress */
+ ODP_COMP_OP_COMPRESS,
+ /** Decompress */
+ ODP_COMP_OP_DECOMPRESS
+} odp_comp_op_t;
+
+/**
+ * Comp API hash algorithm
+ *
+ */
+typedef enum {
+ /** ODP_COMP_HASH_ALG_NONE*/
+ ODP_COMP_HASH_ALG_NONE,
+ /** ODP_COMP_HASH_ALG_SHA1*/
+ ODP_COMP_HASH_ALG_SHA1,
+ /** ODP_COMP_HASH_ALG_SHA256*/
+ ODP_COMP_HASH_ALG_SHA256
+} odp_comp_hash_alg_t;
+
+/**
+ * Comp API compression algorithm
+ *
+ */
+typedef enum {
+ /** No algorithm specified.
+* Means no compression, output == input.
+* if provided, no operation (compression/decompression or hash)
+* applied on input. Added for testing purpose.
+*/
+ ODP_COMP_ALG_NULL,
+ /** DEFLATE - RFC1951 */
+ ODP_COMP_ALG_DEFLATE,
+ /** ZLIB - RFC1950 */
+ ODP_COMP_ALG_ZLIB,
+ /** LZS */
+ ODP_COMP_ALG_LZS
+} odp_comp_alg_t;
+
+/**
+ * Comp API session creation return code
+ *
+ */
+typedef enum {
+ /** Session created */
+ ODP_COMP_SES_CREATE_ERR_NONE,
+ /** Creation failed, no resources */
+ ODP_COMP_SES_CREATE_ERR_ENOMEM,
+ /** Creation failed, bad compression params */
+ ODP_COMP_SES_CREATE_ERR_INV_COMP,
+ /** Creation failed, bad hash params */
+ ODP_COMP_SES_CREATE_ERR_INV_HASH,
+ /** Creation failed,requested configuration not supported*/
+ ODP_COMP_SES_CREATE_ERR_NOT_SUPPORTED
+} odp_comp_ses_create_err_t;
+
+/**
+ * Comp API operation return codes
+ *
+ */
+typedef enum {
+ /** Operation completed successfully*/
+ ODP_COMP_ERR_NONE,
+ /** Operation paused due to insufficient output buffer.
+ *
+ * This is not an error condition. On seeing this situation,
+ * Implementation should maintain context of in-progress operation and
+ * application should call packet processing API again with valid
+ * output buffer but no other alteration to operation params
+ * (odp_comp_op_param_t).
+ *
+ * if using async mode, application should either make sure to
+ * provide sufficient output buffer size OR maintain relevant
+ * context (or ordering) information with respect to each input packet
+ * en-queued for processing.
+ *
+ */
+ ODP_COMP_ERR_OUT_OF_SPACE,
+ /** Invalid user data pointers*/
+ ODP_COMP_ERR_DATA_PTR,
+ /** Invalid input data size*/
+ ODP_COMP_ERR_DATA_SIZE,
+ /** Compression and/or hash Algo failure*/
+ ODP_COMP_ERR_ALGO_FAIL,
+ /** Error if operation has been requested in an invalid state */
+ ODP_COMP_ERR_INV_STATE,
+ /** Error if API does not support any of the operational parameter. */
+ ODP_COMP_ERR_NOT_SUPPORTED,
+ /** Error if session is invalid. */
+ ODP_COMP_ERR_INV_SESS
+} odp_comp_err_t;
+
+/**
+ * Comp API enumeration for preferred compres
