First off, thank you very much for this review.
Please note that this code has been streamlined in patch
http://patches.opendataplane.org/patch/7879/ and has been further
refined with patch http://patches.opendataplane.org/patch/8145/ but
the exposure you identify still exists in that code.
On Fri, Feb 17, 2017 at 11:31 AM, Peltonen, Janne (Nokia - FI/Espoo)
<janne.pelto...@nokia.com> wrote:
> Hi,
>
> I took a look at the packet references and it seems to me that
> either the implementation is a bit racy or I confused myself
> when reading the code. Or maybe I got the intended concurrency
> semantics of the packet references wrong?
>
> My first issue is that packet_free() may access freed packet
> header or corrupt unshared_len.
>
> The packet free function looks like this:
>
> static inline void packet_free(odp_packet_hdr_t *pkt_hdr)
> {
> odp_packet_hdr_t *ref_hdr;
> uint32_t ref_count;
>
> do {
> ref_hdr = pkt_hdr->ref_hdr;
> ref_count = packet_ref_count(pkt_hdr) - 1;
> free_bufs(pkt_hdr, 0, pkt_hdr->buf_hdr.segcount);
>
> if (ref_count == 1)
> pkt_hdr->unshared_len = pkt_hdr->frame_len;
>
> pkt_hdr = ref_hdr;
> } while (pkt_hdr);
> }
>
> The problem here is that decrementing the ref_count, checking
> its value and updating unshared_len is not single atomic
> operation. By the time packet_free() checks if ref_count == 1
> (i.e. if there is exactly one other reference left somewhere),
> the true ref_count may have already been changed by another
> thread doing packet_free() or packet_ref().
>
> For example, if two threads have a reference to the same packet
> then execution (or the relevant memory ops) may get "interleaved"
> as follows:
>
> T1: call packet_free()
> T1: ref_count = packet_ref_count(pkt_hdr) - 1;
> At this point ref_count variable is 1
> T1: call free_bufs()
> T1: call packet_ref_dec()
> Now the ref_count of the packet header is 1.
> T2: call and complete packet_free()
> Thread 2 sees refcount 1 in the packet and frees the buffers
> T1: pkt_hdr->unshared_len = pkt_hdr->frame_len;
> Thread 1 accesses freed buffer for reading and writing.
I agree. These steps should be reversed so that the code should read:
if (ref_count == 1)
pkt_hdr->unshared_len = pkt_hdr->frame_len;
free_bufs(pkt_hdr, 0, pkt_hdr->buf_hdr.segcount);
Or using the code with the above two patches applied, the code should read:
static inline void packet_free(odp_packet_hdr_t *pkt_hdr)
{
odp_packet_hdr_t *ref_hdr;
uint32_t ref_count;
int num_seg;
do {
ref_count = packet_ref_count(pkt_hdr);
num_seg = pkt_hdr->buf_hdr.segcount;
ref_hdr = pkt_hdr->ref_hdr;
if (odp_likely((CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1) &&
ref_count == 1)) {
buffer_free_multi((odp_buffer_t
*)&pkt_hdr->buf_hdr.handle.handle, 1);
} else {
if (ref_count == 2)
pkt_hdr->unshared_len = pkt_hdr->frame_len;
free_bufs(pkt_hdr, 0, num_seg);
}
pkt_hdr = ref_hdr;
} while (pkt_hdr);
}
The mistake was trying to optimize things so that unshared_len is not
set if the buffers are being freed, but that exposes these race
conditions. So the worst that should now happen is that it is set
unnecessarily before being freed.
If you concur I'll fold this fix into a v3 for patch
http://patches.opendataplane.org/patch/8145/
>
> Similarly, if T2 created a new reference, T1 would have
> a wrong idea of the number of remaining references and
> would adjust the unshared_len to an incorrect value.
>
> Right?
>
> Maybe other modifications of unshared_len are also racy.
I don't believe so, because references do not change the existing ODP
restriction that two threads cannot share the same odp_packet_t. When
a packet reference is created it returns a separate odp_packet_t that
has its own metadata. So unshared_len is always private to an
individual odp_packet_t. The exception is static references but in
this case the entire
packet along with its metadata must be treated as read only so
operations like odp_packet_push_head() that would try to modify
unshared_len are prohibited.
>
>
>
> The second issue is that the atomic ops for setting and
> reading the ref count seem to have too relaxed memory
> ordering. In particular, packet_ref_dec() must not happen
> (be visible to other threads) before its caller is done
> with the packet and the related memory accesses have
> completed. Now there does not seem to be any optimization
> and memory barrier to prevent the ref count decrementing
> happening too early. So I think it is at least theoretically
> possible that a thread e.g. reads from a packet buffer
> after it has already been freed by another thread, somehow
> like this:
>
> Source code order:
> T1: interesting_data = read_from_pkt(pkt)
> T1: packet_free(pkt)
>
> Order visible to T2:
> 1: ref count decr
> 2: read from pkt
>
> Now if T2 goes and frees the remaining reference between
> steps 1 and 2, T1 may get even more interesting data.
>
> Right?
I don't believe so. The semantics of odp_atomic_fetch_dec_u32(), which
is what packet_ref_dec() uses, says that no two calls can see the same
fetched value, so only one thread will return ref_count == 1 and free
the buffer. Note that if I see ref_count == 1 no other thread can be
trying to increment it via a concurrent odp_packet_ref() call because
that would mean that two threads were trying to manipulate the same
odp_packet_t, which is prohibited.
For CPUs that support out of order instruction execution, this is only
permitted providing the reordering and speculative executions are
semantically consistent with sequential execution. If this were not
the case you'd constantly have to worry about a processor turning
T1: interesting_data = read_from_pkt(pkt)
T1: packet_free(pkt)
into
T1: packet_free(pkt)
T1: interesting_data = read_from_pkt(pkt)
In your scenario above: T2 cannot be issuing a read to pkt after
ref_count is decremented because the only way that T2 could be
decrementing ref_count would be if T2 issued an odp_packet_free() call
for it. Obviously if it tries to reference pkt after such a call that
is an application error.
Thanks again for your much-appreciated help in looking at this!
>
> Janne
>
>
>> -----Original Message-----
>> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Bill
>> Fischofer
>> Sent: Wednesday, January 11, 2017 4:34 AM
>> To: lng-odp@lists.linaro.org
>> Subject: [lng-odp] [API-NEXT PATCHv7 2/5] linux-generic: packet: implement
>> reference apis
>>
>> Implement the APIs:
>> - odp_packet_ref_static()
>> - odp_packet_ref()
>> - odp_packet_ref_pkt()
>> - odp_packet_has_ref()
>> - odp_packet_unshared_len()
>>
>> This also involves functional upgrades to the existing packet manipulation
>> APIs to work with packet references as input arguments.
>>
>> Signed-off-by: Bill Fischofer <bill.fischo...@linaro.org>
>> ---
>> .../linux-generic/include/odp_packet_internal.h | 87 +++-
>> platform/linux-generic/odp_packet.c | 536
>> +++++++++++++++++----
>> 2 files changed, 516 insertions(+), 107 deletions(-)
>>
>> diff --git a/platform/linux-generic/include/odp_packet_internal.h
>> b/platform/linux-
>> generic/include/odp_packet_internal.h
>> index e6e9d74..607560d 100644
>> --- a/platform/linux-generic/include/odp_packet_internal.h
>> +++ b/platform/linux-generic/include/odp_packet_internal.h
>> @@ -19,6 +19,7 @@ extern "C" {
>>
>> #include <odp/api/align.h>
>> #include <odp/api/debug.h>
>> +#include <odp_debug_internal.h>
>> #include <odp_buffer_internal.h>
>> #include <odp_pool_internal.h>
>> #include <odp_buffer_inlines.h>
>> @@ -168,7 +169,7 @@ typedef struct {
>> * packet_init(). Because of this any new fields added must be reviewed for
>> * initialization requirements.
>> */
>> -typedef struct {
>> +typedef struct odp_packet_hdr_t {
>> /* common buffer header */
>> odp_buffer_hdr_t buf_hdr;
>>
>> @@ -184,6 +185,13 @@ typedef struct {
>> uint32_t headroom;
>> uint32_t tailroom;
>>
>> + /* Fields used to support packet references */
>> + uint32_t unshared_len;
>> + struct odp_packet_hdr_t *ref_hdr;
>> + uint32_t ref_offset;
>> + uint32_t ref_len;
>> + odp_atomic_u32_t ref_count;
>> +
>> /*
>> * Members below are not initialized by packet_init()
>> */
>> @@ -212,6 +220,55 @@ static inline odp_packet_hdr_t
>> *odp_packet_hdr(odp_packet_t pkt)
>> return (odp_packet_hdr_t *)buf_hdl_to_hdr((odp_buffer_t)pkt);
>> }
>>
>> +static inline odp_packet_hdr_t *odp_packet_last_hdr(odp_packet_t pkt,
>> + uint32_t *offset)
>> +{
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> + odp_packet_hdr_t *prev_hdr = pkt_hdr;
>> + uint32_t ref_offset = 0;
>> +
>> + while (pkt_hdr->ref_hdr) {
>> + ref_offset = pkt_hdr->ref_offset;
>> + prev_hdr = pkt_hdr;
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + }
>> +
>> + if (offset) {
>> + if (prev_hdr != pkt_hdr)
>> + ref_offset += pkt_hdr->frame_len - prev_hdr->ref_len;
>> + *offset = ref_offset;
>> + }
>> +
>> + return pkt_hdr;
>> +}
>> +
>> +static inline odp_packet_hdr_t *odp_packet_prev_hdr(odp_packet_hdr_t
>> *pkt_hdr,
>> + odp_packet_hdr_t *cur_hdr,
>> + uint32_t *offset)
>> +{
>> + uint32_t ref_offset = 0;
>> + odp_packet_hdr_t *prev_hdr = pkt_hdr;
>> +
>> + while (pkt_hdr->ref_hdr != cur_hdr) {
>> + ref_offset = pkt_hdr->ref_offset;
>> + prev_hdr = pkt_hdr;
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + }
>> +
>> + if (offset) {
>> + if (prev_hdr != pkt_hdr)
>> + ref_offset += pkt_hdr->frame_len - prev_hdr->ref_len;
>> + *offset = ref_offset;
>> + }
>> +
>> + return pkt_hdr;
>> +}
>> +
>> +static inline odp_packet_t _odp_packet_hdl(odp_packet_hdr_t *pkt_hdr)
>> +{
>> + return (odp_packet_t)odp_hdr_to_buf(&pkt_hdr->buf_hdr);
>> +}
>> +
>> static inline void copy_packet_parser_metadata(odp_packet_hdr_t *src_hdr,
>> odp_packet_hdr_t *dst_hdr)
>> {
>> @@ -234,17 +291,43 @@ static inline void pull_tail(odp_packet_hdr_t
>> *pkt_hdr, uint32_t
>> len)
>>
>> pkt_hdr->tailroom += len;
>> pkt_hdr->frame_len -= len;
>> + pkt_hdr->unshared_len -= len;
>> pkt_hdr->buf_hdr.seg[last].len -= len;
>> }
>>
>> static inline uint32_t packet_len(odp_packet_hdr_t *pkt_hdr)
>> {
>> - return pkt_hdr->frame_len;
>> + uint32_t pkt_len = 0;
>> + uint32_t offset = 0;
>> +
>> + do {
>> + pkt_len += pkt_hdr->frame_len - offset;
>> + offset = pkt_hdr->ref_offset;
>> + if (pkt_hdr->ref_hdr)
>> + offset += (pkt_hdr->ref_hdr->frame_len -
>> + pkt_hdr->ref_len);
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + } while (pkt_hdr);
>> +
>> + return pkt_len;
>> +}
>> +
>> +static inline uint32_t packet_ref_count(odp_packet_hdr_t *pkt_hdr)
>> +{
>> + return odp_atomic_load_u32(&pkt_hdr->ref_count);
>> +}
>> +
>> +static inline void packet_ref_count_set(odp_packet_hdr_t *pkt_hdr, uint32_t
>> n)
>> +{
>> + odp_atomic_init_u32(&pkt_hdr->ref_count, n);
>> }
>>
>> static inline void packet_set_len(odp_packet_hdr_t *pkt_hdr, uint32_t len)
>> {
>> + ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
>> +
>> pkt_hdr->frame_len = len;
>> + pkt_hdr->unshared_len = len;
>> }
>>
>> static inline int packet_parse_l2_not_done(packet_parser_t *prs)
>> diff --git a/platform/linux-generic/odp_packet.c
>> b/platform/linux-generic/odp_packet.c
>> index f632a51..170965a 100644
>> --- a/platform/linux-generic/odp_packet.c
>> +++ b/platform/linux-generic/odp_packet.c
>> @@ -33,13 +33,24 @@ static inline odp_buffer_t
>> buffer_handle(odp_packet_hdr_t *pkt_hdr)
>> return pkt_hdr->buf_hdr.handle.handle;
>> }
>>
>> +static inline uint32_t packet_ref_inc(odp_packet_hdr_t *pkt_hdr)
>> +{
>> + return odp_atomic_fetch_inc_u32(&pkt_hdr->ref_count);
>> +}
>> +
>> +static inline uint32_t packet_ref_dec(odp_packet_hdr_t *pkt_hdr)
>> +{
>> + return odp_atomic_fetch_dec_u32(&pkt_hdr->ref_count);
>> +}
>> +
>> static inline uint32_t packet_seg_len(odp_packet_hdr_t *pkt_hdr,
>> uint32_t seg_idx)
>> {
>> return pkt_hdr->buf_hdr.seg[seg_idx].len;
>> }
>>
>> -static inline void *packet_seg_data(odp_packet_hdr_t *pkt_hdr, uint32_t
>> seg_idx)
>> +static inline uint8_t *packet_seg_data(odp_packet_hdr_t *pkt_hdr,
>> + uint32_t seg_idx)
>> {
>> return pkt_hdr->buf_hdr.seg[seg_idx].data;
>> }
>> @@ -52,6 +63,11 @@ static inline int packet_last_seg(odp_packet_hdr_t
>> *pkt_hdr)
>> return pkt_hdr->buf_hdr.segcount - 1;
>> }
>>
>> +static inline void *packet_data(odp_packet_hdr_t *pkt_hdr)
>> +{
>> + return pkt_hdr->buf_hdr.seg[0].data;
>> +}
>> +
>> static inline uint32_t packet_first_seg_len(odp_packet_hdr_t *pkt_hdr)
>> {
>> return packet_seg_len(pkt_hdr, 0);
>> @@ -64,11 +80,6 @@ static inline uint32_t
>> packet_last_seg_len(odp_packet_hdr_t *pkt_hdr)
>> return packet_seg_len(pkt_hdr, last);
>> }
>>
>> -static inline void *packet_data(odp_packet_hdr_t *pkt_hdr)
>> -{
>> - return pkt_hdr->buf_hdr.seg[0].data;
>> -}
>> -
>> static inline void *packet_tail(odp_packet_hdr_t *pkt_hdr)
>> {
>> int last = packet_last_seg(pkt_hdr);
>> @@ -99,6 +110,7 @@ static inline void push_head(odp_packet_hdr_t *pkt_hdr,
>> uint32_t len)
>> {
>> pkt_hdr->headroom -= len;
>> pkt_hdr->frame_len += len;
>> + pkt_hdr->unshared_len += len;
>> pkt_hdr->buf_hdr.seg[0].data -= len;
>> pkt_hdr->buf_hdr.seg[0].len += len;
>> }
>> @@ -107,6 +119,7 @@ static inline void pull_head(odp_packet_hdr_t *pkt_hdr,
>> uint32_t len)
>> {
>> pkt_hdr->headroom += len;
>> pkt_hdr->frame_len -= len;
>> + pkt_hdr->unshared_len -= len;
>> pkt_hdr->buf_hdr.seg[0].data += len;
>> pkt_hdr->buf_hdr.seg[0].len -= len;
>> }
>> @@ -117,6 +130,7 @@ static inline void push_tail(odp_packet_hdr_t *pkt_hdr,
>> uint32_t len)
>>
>> pkt_hdr->tailroom -= len;
>> pkt_hdr->frame_len += len;
>> + pkt_hdr->unshared_len += len;
>> pkt_hdr->buf_hdr.seg[last].len += len;
>> }
>>
>> @@ -144,6 +158,10 @@ static inline void packet_seg_copy_md(odp_packet_hdr_t
>> *dst,
>> dst->buf_hdr.uarea_addr = src->buf_hdr.uarea_addr;
>> dst->buf_hdr.uarea_size = src->buf_hdr.uarea_size;
>>
>> + /* reference related metadata */
>> + dst->ref_len = src->ref_len;
>> + dst->unshared_len = src->unshared_len;
>> +
>> /* segmentation data is not copied:
>> * buf_hdr.seg[]
>> * buf_hdr.segcount
>> @@ -158,7 +176,15 @@ static inline void *packet_map(odp_packet_hdr_t
>> *pkt_hdr,
>> int seg = 0;
>> int seg_count = pkt_hdr->buf_hdr.segcount;
>>
>> - if (odp_unlikely(offset >= pkt_hdr->frame_len))
>> + /* Special processing for references */
>> + while (offset >= pkt_hdr->frame_len && pkt_hdr->ref_hdr) {
>> + offset -= (pkt_hdr->frame_len - pkt_hdr->ref_offset);
>> + offset += (pkt_hdr->ref_hdr->frame_len - pkt_hdr->ref_len);
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + seg_count = pkt_hdr->buf_hdr.segcount;
>> + }
>> +
>> + if (odp_unlikely(offset > pkt_hdr->frame_len))
>> return NULL;
>>
>> if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || seg_count == 1)) {
>> @@ -207,6 +233,9 @@ void packet_parse_reset(odp_packet_hdr_t *pkt_hdr)
>> pkt_hdr->p.l2_offset = 0;
>> pkt_hdr->p.l3_offset = ODP_PACKET_OFFSET_INVALID;
>> pkt_hdr->p.l4_offset = ODP_PACKET_OFFSET_INVALID;
>> +
>> + /* Ensure dummy pkt_hdrs used in I/O recv classification are valid */
>> + pkt_hdr->ref_hdr = NULL;
>> }
>>
>> /**
>> @@ -252,6 +281,10 @@ static inline void packet_init(odp_packet_hdr_t
>> *pkt_hdr, uint32_t
>> len,
>> CONFIG_PACKET_TAILROOM;
>>
>> pkt_hdr->input = ODP_PKTIO_INVALID;
>> +
>> + /* By default packet has no references */
>> + pkt_hdr->unshared_len = len;
>> + pkt_hdr->ref_hdr = NULL;
>> }
>>
>> static inline void init_segments(odp_packet_hdr_t *pkt_hdr[], int num)
>> @@ -264,6 +297,7 @@ static inline void init_segments(odp_packet_hdr_t
>> *pkt_hdr[], int num)
>>
>> hdr->buf_hdr.seg[0].data = hdr->buf_hdr.base_data;
>> hdr->buf_hdr.seg[0].len = BASE_LEN;
>> + packet_ref_count_set(hdr, 1);
>>
>> /* Link segments */
>> if (CONFIG_PACKET_MAX_SEGS != 1) {
>> @@ -273,6 +307,7 @@ static inline void init_segments(odp_packet_hdr_t
>> *pkt_hdr[], int num)
>> for (i = 1; i < num; i++) {
>> odp_buffer_hdr_t *buf_hdr;
>>
>> + packet_ref_count_set(pkt_hdr[i], 1);
>> buf_hdr = &pkt_hdr[i]->buf_hdr;
>> hdr->buf_hdr.seg[i].hdr = buf_hdr;
>> hdr->buf_hdr.seg[i].data = buf_hdr->base_data;
>> @@ -376,9 +411,10 @@ static inline odp_packet_hdr_t
>> *add_segments(odp_packet_hdr_t
>> *pkt_hdr,
>> new_hdr->buf_hdr.seg[0].len = seg_len;
>>
>> packet_seg_copy_md(new_hdr, pkt_hdr);
>> - new_hdr->frame_len = pkt_hdr->frame_len + len;
>> - new_hdr->headroom = pool->headroom + offset;
>> - new_hdr->tailroom = pkt_hdr->tailroom;
>> + new_hdr->frame_len = pkt_hdr->frame_len + len;
>> + new_hdr->unshared_len = pkt_hdr->unshared_len + len;
>> + new_hdr->headroom = pool->headroom + offset;
>> + new_hdr->tailroom = pkt_hdr->tailroom;
>>
>> pkt_hdr = new_hdr;
>> } else {
>> @@ -391,8 +427,9 @@ static inline odp_packet_hdr_t
>> *add_segments(odp_packet_hdr_t
>> *pkt_hdr,
>> last = packet_last_seg(pkt_hdr);
>> pkt_hdr->buf_hdr.seg[last].len = seg_len;
>>
>> - pkt_hdr->frame_len += len;
>> - pkt_hdr->tailroom = pool->tailroom + offset;
>> + pkt_hdr->frame_len += len;
>> + pkt_hdr->unshared_len += len;
>> + pkt_hdr->tailroom = pool->tailroom + offset;
>> }
>>
>> return pkt_hdr;
>> @@ -400,13 +437,18 @@ static inline odp_packet_hdr_t
>> *add_segments(odp_packet_hdr_t
>> *pkt_hdr,
>>
>> static inline void free_bufs(odp_packet_hdr_t *pkt_hdr, int first, int num)
>> {
>> - int i;
>> + int i, nfree;
>> odp_buffer_t buf[num];
>>
>> - for (i = 0; i < num; i++)
>> - buf[i] = buffer_handle(pkt_hdr->buf_hdr.seg[first + i].hdr);
>> + for (i = 0, nfree = 0; i < num; i++) {
>> + odp_packet_hdr_t *hdr = pkt_hdr->buf_hdr.seg[first + i].hdr;
>> +
>> + if (packet_ref_dec(hdr) == 1)
>> + buf[nfree++] = buffer_handle(hdr);
>> + }
>>
>> - buffer_free_multi(buf, num);
>> + if (nfree > 0)
>> + buffer_free_multi(buf, nfree);
>> }
>>
>> static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr,
>> @@ -417,11 +459,15 @@ static inline odp_packet_hdr_t
>> *free_segments(odp_packet_hdr_t
>> *pkt_hdr,
>>
>> if (head) {
>> odp_packet_hdr_t *new_hdr;
>> - int i;
>> + int i, nfree;
>> odp_buffer_t buf[num];
>>
>> - for (i = 0; i < num; i++)
>> - buf[i] = buffer_handle(pkt_hdr->buf_hdr.seg[i].hdr);
>> + for (i = 0, nfree = 0; i < num; i++) {
>> + new_hdr = pkt_hdr->buf_hdr.seg[i].hdr;
>> +
>> + if (packet_ref_dec(new_hdr) == 1)
>> + buf[nfree++] = buffer_handle(new_hdr);
>> + }
>>
>> /* First remaining segment is the new packet descriptor */
>> new_hdr = pkt_hdr->buf_hdr.seg[num].hdr;
>> @@ -430,15 +476,17 @@ static inline odp_packet_hdr_t
>> *free_segments(odp_packet_hdr_t
>> *pkt_hdr,
>> packet_seg_copy_md(new_hdr, pkt_hdr);
>>
>> /* Tailroom not changed */
>> - new_hdr->tailroom = pkt_hdr->tailroom;
>> - new_hdr->headroom = seg_headroom(new_hdr, 0);
>> - new_hdr->frame_len = pkt_hdr->frame_len - free_len;
>> + new_hdr->tailroom = pkt_hdr->tailroom;
>> + new_hdr->headroom = seg_headroom(new_hdr, 0);
>> + new_hdr->frame_len = pkt_hdr->frame_len - free_len;
>> + new_hdr->unshared_len = pkt_hdr->unshared_len - free_len;
>>
>> pull_head(new_hdr, pull_len);
>>
>> pkt_hdr = new_hdr;
>>
>> - buffer_free_multi(buf, num);
>> + if (nfree > 0)
>> + buffer_free_multi(buf, nfree);
>> } else {
>> /* Free last 'num' bufs */
>> free_bufs(pkt_hdr, num_remain, num);
>> @@ -447,6 +495,7 @@ static inline odp_packet_hdr_t
>> *free_segments(odp_packet_hdr_t
>> *pkt_hdr,
>> * of the metadata. */
>> pkt_hdr->buf_hdr.segcount = num_remain;
>> pkt_hdr->frame_len -= free_len;
>> + pkt_hdr->unshared_len -= free_len;
>> pkt_hdr->tailroom = seg_tailroom(pkt_hdr, num_remain - 1);
>>
>> pull_tail(pkt_hdr, pull_len);
>> @@ -550,45 +599,34 @@ int odp_packet_alloc_multi(odp_pool_t pool_hdl,
>> uint32_t len,
>> return num;
>> }
>>
>> -void odp_packet_free(odp_packet_t pkt)
>> +static inline void packet_free(odp_packet_hdr_t *pkt_hdr)
>> {
>> - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> - int num_seg = pkt_hdr->buf_hdr.segcount;
>> + odp_packet_hdr_t *ref_hdr;
>> + uint32_t ref_count;
>>
>> - if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1))
>> - buffer_free_multi((odp_buffer_t *)&pkt, 1);
>> - else
>> - free_bufs(pkt_hdr, 0, num_seg);
>> -}
>> + do {
>> + ref_hdr = pkt_hdr->ref_hdr;
>> + ref_count = packet_ref_count(pkt_hdr) - 1;
>> + free_bufs(pkt_hdr, 0, pkt_hdr->buf_hdr.segcount);
>>
>> -void odp_packet_free_multi(const odp_packet_t pkt[], int num)
>> -{
>> - if (CONFIG_PACKET_MAX_SEGS == 1) {
>> - buffer_free_multi((const odp_buffer_t * const)pkt, num);
>> - } else {
>> - odp_buffer_t buf[num * CONFIG_PACKET_MAX_SEGS];
>> - int i, j;
>> - int bufs = 0;
>> + if (ref_count == 1)
>> + pkt_hdr->unshared_len = pkt_hdr->frame_len;
>>
>> - for (i = 0; i < num; i++) {
>> - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt[i]);
>> - int num_seg = pkt_hdr->buf_hdr.segcount;
>> - odp_buffer_hdr_t *buf_hdr = &pkt_hdr->buf_hdr;
>> -
>> - buf[bufs] = (odp_buffer_t)pkt[i];
>> - bufs++;
>> + pkt_hdr = ref_hdr;
>> + } while (pkt_hdr);
>> +}
>>
>> - if (odp_likely(num_seg == 1))
>> - continue;
>> +void odp_packet_free(odp_packet_t pkt)
>> +{
>> + packet_free(odp_packet_hdr(pkt));
>> +}
>>
>> - for (j = 1; j < num_seg; j++) {
>> - buf[bufs] = buffer_handle(buf_hdr->seg[j].hdr);
>> - bufs++;
>> - }
>> - }
>> +void odp_packet_free_multi(const odp_packet_t pkt[], int num)
>> +{
>> + int i;
>>
>> - buffer_free_multi(buf, bufs);
>> - }
>> + for (i = 0; i < num; i++)
>> + packet_free(odp_packet_hdr(pkt[i]));
>> }
>>
>> int odp_packet_reset(odp_packet_t pkt, uint32_t len)
>> @@ -599,6 +637,9 @@ int odp_packet_reset(odp_packet_t pkt, uint32_t len)
>> if (len > pool->headroom + pool->data_size + pool->tailroom)
>> return -1;
>>
>> + if (pkt_hdr->ref_hdr)
>> + packet_free(pkt_hdr->ref_hdr);
>> +
>> packet_init(pkt_hdr, len, 0);
>>
>> return 0;
>> @@ -641,15 +682,21 @@ void *odp_packet_head(odp_packet_t pkt)
>> uint32_t odp_packet_buf_len(odp_packet_t pkt)
>> {
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> + uint32_t buf_len = 0;
>>
>> - return pkt_hdr->buf_hdr.size * pkt_hdr->buf_hdr.segcount;
>> + do {
>> + buf_len += pkt_hdr->buf_hdr.size * pkt_hdr->buf_hdr.segcount;
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + } while (pkt_hdr);
>> +
>> + return buf_len;
>> }
>>
>> void *odp_packet_data(odp_packet_t pkt)
>> {
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>>
>> - return packet_data(pkt_hdr);
>> + return packet_map(pkt_hdr, 0, NULL, NULL);
>> }
>>
>> uint32_t odp_packet_seg_len(odp_packet_t pkt)
>> @@ -661,7 +708,32 @@ uint32_t odp_packet_seg_len(odp_packet_t pkt)
>>
>> uint32_t odp_packet_len(odp_packet_t pkt)
>> {
>> - return odp_packet_hdr(pkt)->frame_len;
>> + return packet_len(odp_packet_hdr(pkt));
>> +}
>> +
>> +uint32_t odp_packet_unshared_len(odp_packet_t pkt)
>> +{
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> + uint32_t pkt_len = 0, offset = 0;
>> +
>> + do {
>> + if (packet_ref_count(pkt_hdr) > 1) {
>> + if (offset == 0)
>> + pkt_len += pkt_hdr->unshared_len;
>> + break;
>> + }
>> +
>> + pkt_len += pkt_hdr->frame_len - offset;
>> + offset = pkt_hdr->ref_offset;
>> +
>> + if (pkt_hdr->ref_hdr)
>> + offset += (pkt_hdr->ref_hdr->frame_len -
>> + pkt_hdr->ref_len);
>> +
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + } while (pkt_hdr);
>> +
>> + return pkt_len;
>> }
>>
>> uint32_t odp_packet_headroom(odp_packet_t pkt)
>> @@ -671,12 +743,12 @@ uint32_t odp_packet_headroom(odp_packet_t pkt)
>>
>> uint32_t odp_packet_tailroom(odp_packet_t pkt)
>> {
>> - return odp_packet_hdr(pkt)->tailroom;
>> + return odp_packet_last_hdr(pkt, NULL)->tailroom;
>> }
>>
>> void *odp_packet_tail(odp_packet_t pkt)
>> {
>> - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_last_hdr(pkt, NULL);
>>
>> return packet_tail(pkt_hdr);
>> }
>> @@ -870,7 +942,7 @@ int odp_packet_extend_head(odp_packet_t *pkt, uint32_t
>> len,
>> {
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(*pkt);
>> uint32_t frame_len = pkt_hdr->frame_len;
>> - uint32_t headroom = pkt_hdr->headroom;
>> + uint32_t headroom = pkt_hdr->headroom;
>> int ret = 0;
>>
>> if (len > headroom) {
>> @@ -885,6 +957,46 @@ int odp_packet_extend_head(odp_packet_t *pkt, uint32_t
>> len,
>> segs = pkt_hdr->buf_hdr.segcount;
>>
>> if (odp_unlikely((segs + num) > CONFIG_PACKET_MAX_SEGS)) {
>> + /* Handle recursively via references when
>> + * working with referenced packets since another
>> + * thread may be accessing it concurrently via
>> + * its reference to it. */
>> + if (packet_ref_count(pkt_hdr) > 1) {
>> + odp_packet_t ref;
>> + uint32_t unshared_len;
>> +
>> + push_head(pkt_hdr, headroom);
>> + unshared_len = pkt_hdr->unshared_len;
>> + ref = odp_packet_ref(*pkt, 0);
>> +
>> + if (ref == ODP_PACKET_INVALID) {
>> + pull_head(pkt_hdr, headroom);
>> + return -1;
>> + }
>> +
>> + ret = odp_packet_extend_head(&ref,
>> + len - headroom,
>> + data_ptr,
>> + seg_len);
>> +
>> + if (ret < 0) {
>> + odp_packet_free(ref);
>> + pull_head(pkt_hdr, headroom);
>> + return -1;
>> + }
>> +
>> + /* Since this is a special ref, the
>> + * base pkt's unshared len is unchanged */
>> + pkt_hdr->unshared_len = unshared_len;
>> +
>> + /* Remove extra ref to the base pkt */
>> + odp_packet_free(*pkt);
>> +
>> + /* Return the ref as the extension result */
>> + *pkt = ref;
>> + return 1;
>> + }
>> +
>> /* Cannot directly add new segments */
>> odp_packet_hdr_t *new_hdr;
>> int new_segs = 0;
>> @@ -936,6 +1048,7 @@ int odp_packet_extend_head(odp_packet_t *pkt, uint32_t
>> len,
>>
>> pkt_hdr->buf_hdr.segcount = segs;
>> pkt_hdr->frame_len = frame_len;
>> + pkt_hdr->unshared_len = frame_len;
>> pkt_hdr->headroom = offset + pool->headroom;
>> pkt_hdr->tailroom = pool->tailroom;
>>
>> @@ -961,11 +1074,16 @@ int odp_packet_extend_head(odp_packet_t *pkt,
>> uint32_t len,
>> push_head(pkt_hdr, len);
>> }
>>
>> - if (data_ptr)
>> - *data_ptr = packet_data(pkt_hdr);
>> + if (data_ptr || seg_len) {
>> + uint32_t seg_ln = 0;
>> + void *data = packet_map(pkt_hdr, 0, &seg_ln, NULL);
>>
>> - if (seg_len)
>> - *seg_len = packet_first_seg_len(pkt_hdr);
>> + if (data_ptr)
>> + *data_ptr = data;
>> +
>> + if (seg_len)
>> + *seg_len = seg_ln;
>> + }
>>
>> return ret;
>> }
>> @@ -977,6 +1095,8 @@ void *odp_packet_pull_head(odp_packet_t pkt, uint32_t
>> len)
>> if (len > pkt_hdr->frame_len)
>> return NULL;
>>
>> + ODP_ASSERT(len <= pkt_hdr->unshared_len);
>> +
>> pull_head(pkt_hdr, len);
>> return packet_data(pkt_hdr);
>> }
>> @@ -984,15 +1104,35 @@ void *odp_packet_pull_head(odp_packet_t pkt, uint32_t
>> len)
>> int odp_packet_trunc_head(odp_packet_t *pkt, uint32_t len,
>> void **data_ptr, uint32_t *seg_len_out)
>> {
>> - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(*pkt);
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(*pkt), *nxt_hdr;
>> uint32_t seg_len = packet_first_seg_len(pkt_hdr);
>> + int ret = 0;
>>
>> - if (len > pkt_hdr->frame_len)
>> + if (len > packet_len(pkt_hdr))
>> return -1;
>>
>> - if (len < seg_len) {
>> + ODP_ASSERT(len <= odp_packet_unshared_len(*pkt));
>> +
>> + /* Special processing for references */
>> + while (len >= pkt_hdr->frame_len && pkt_hdr->ref_hdr) {
>> + ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
>> + nxt_hdr = pkt_hdr->ref_hdr;
>> + len -= pkt_hdr->frame_len;
>> + len += pkt_hdr->ref_offset +
>> + (nxt_hdr->frame_len - pkt_hdr->ref_len);
>> + pkt_hdr->ref_hdr = NULL;
>> + packet_free(pkt_hdr);
>> + pkt_hdr = nxt_hdr;
>> + seg_len = packet_first_seg_len(pkt_hdr);
>> + *pkt = packet_handle(pkt_hdr);
>> + ret = 1;
>> + }
>> +
>> + if (CONFIG_PACKET_MAX_SEGS == 1 ||
>> + len < seg_len ||
>> + pkt_hdr->buf_hdr.segcount == 1) {
>> pull_head(pkt_hdr, len);
>> - } else if (CONFIG_PACKET_MAX_SEGS != 1) {
>> + } else {
>> int num = 0;
>> uint32_t pull_len = 0;
>>
>> @@ -1007,23 +1147,29 @@ int odp_packet_trunc_head(odp_packet_t *pkt,
>> uint32_t len,
>> *pkt = packet_handle(pkt_hdr);
>> }
>>
>> - if (data_ptr)
>> - *data_ptr = packet_data(pkt_hdr);
>> + if (data_ptr || seg_len_out) {
>> + void *data_head = packet_map(pkt_hdr, 0, &seg_len, NULL);
>>
>> - if (seg_len_out)
>> - *seg_len_out = packet_first_seg_len(pkt_hdr);
>> + if (data_ptr)
>> + *data_ptr = data_head;
>>
>> - return 0;
>> + if (seg_len_out)
>> + *seg_len_out = seg_len;
>> + }
>> +
>> + return ret;
>> }
>>
>> void *odp_packet_push_tail(odp_packet_t pkt, uint32_t len)
>> {
>> - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_last_hdr(pkt, NULL);
>> void *old_tail;
>>
>> if (len > pkt_hdr->tailroom)
>> return NULL;
>>
>> + ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
>> +
>> old_tail = packet_tail(pkt_hdr);
>> push_tail(pkt_hdr, len);
>>
>> @@ -1033,12 +1179,14 @@ void *odp_packet_push_tail(odp_packet_t pkt,
>> uint32_t len)
>> int odp_packet_extend_tail(odp_packet_t *pkt, uint32_t len,
>> void **data_ptr, uint32_t *seg_len_out)
>> {
>> - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(*pkt);
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_last_hdr(*pkt, NULL);
>> uint32_t frame_len = pkt_hdr->frame_len;
>> uint32_t tailroom = pkt_hdr->tailroom;
>> uint32_t tail_off = frame_len;
>> int ret = 0;
>>
>> + ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
>> +
>> if (len > tailroom) {
>> pool_t *pool = pool_entry_from_hdl(pkt_hdr->buf_hdr.pool_hdl);
>> int num;
>> @@ -1129,6 +1277,7 @@ void *odp_packet_pull_tail(odp_packet_t pkt, uint32_t
>> len)
>> if (len > packet_last_seg_len(pkt_hdr))
>> return NULL;
>>
>> + ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
>> pull_tail(pkt_hdr, len);
>>
>> return packet_tail(pkt_hdr);
>> @@ -1139,17 +1288,34 @@ int odp_packet_trunc_tail(odp_packet_t *pkt,
>> uint32_t len,
>> {
>> int last;
>> uint32_t seg_len;
>> - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(*pkt);
>> + uint32_t offset;
>> + odp_packet_hdr_t *first_hdr = odp_packet_hdr(*pkt);
>> + odp_packet_hdr_t *pkt_hdr, *prev_hdr;
>>
>> - if (len > pkt_hdr->frame_len)
>> + if (len > packet_len(first_hdr))
>> return -1;
>>
>> + pkt_hdr = odp_packet_last_hdr(*pkt, &offset);
>> +
>> + /* Special processing for references */
>> + while (len >= pkt_hdr->frame_len - offset && first_hdr->ref_hdr) {
>> + len -= (pkt_hdr->frame_len - offset);
>> + prev_hdr = odp_packet_prev_hdr(first_hdr, pkt_hdr, &offset);
>> + ODP_ASSERT(packet_ref_count(prev_hdr) == 1);
>> + prev_hdr->ref_hdr = NULL;
>> + packet_free(pkt_hdr);
>> + pkt_hdr = prev_hdr;
>> + }
>> +
>> + ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
>> last = packet_last_seg(pkt_hdr);
>> seg_len = packet_seg_len(pkt_hdr, last);
>>
>> - if (len < seg_len) {
>> + if (CONFIG_PACKET_MAX_SEGS == 1 ||
>> + len < seg_len ||
>> + pkt_hdr->buf_hdr.segcount == 1) {
>> pull_tail(pkt_hdr, len);
>> - } else if (CONFIG_PACKET_MAX_SEGS != 1) {
>> + } else {
>> int num = 0;
>> uint32_t pull_len = 0;
>>
>> @@ -1356,35 +1522,50 @@ void odp_packet_ts_set(odp_packet_t pkt, odp_time_t
>> timestamp)
>>
>> int odp_packet_is_segmented(odp_packet_t pkt)
>> {
>> - return odp_packet_hdr(pkt)->buf_hdr.segcount > 1;
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> +
>> + return pkt_hdr->buf_hdr.segcount > 1 || pkt_hdr->ref_hdr != NULL;
>> }
>>
>> int odp_packet_num_segs(odp_packet_t pkt)
>> {
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> + uint32_t segcount = 0, i;
>> + uint32_t seg_offset = 0, offset;
>> +
>> + do {
>> + segcount += pkt_hdr->buf_hdr.segcount - seg_offset;
>> + offset = pkt_hdr->ref_offset;
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + if (pkt_hdr) {
>> + for (i = 0, seg_offset = 0;
>> + i < pkt_hdr->buf_hdr.segcount;
>> + i++, seg_offset++) {
>> + if (offset < pkt_hdr->buf_hdr.seg[i].len)
>> + break;
>> + offset -= pkt_hdr->buf_hdr.seg[i].len;
>> + }
>> + }
>> + } while (pkt_hdr);
>>
>> - return pkt_hdr->buf_hdr.segcount;
>> + return segcount;
>> }
>>
>> -odp_packet_seg_t odp_packet_first_seg(odp_packet_t pkt)
>> +odp_packet_seg_t odp_packet_first_seg(odp_packet_t pkt ODP_UNUSED)
>> {
>> - (void)pkt;
>> -
>> return 0;
>> }
>>
>> odp_packet_seg_t odp_packet_last_seg(odp_packet_t pkt)
>> {
>> - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> -
>> - return packet_last_seg(pkt_hdr);
>> + return (odp_packet_seg_t)(odp_packet_num_segs(pkt) - 1);
>> }
>>
>> odp_packet_seg_t odp_packet_next_seg(odp_packet_t pkt, odp_packet_seg_t seg)
>> {
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>>
>> - if (odp_unlikely(seg >= (odp_packet_seg_t)packet_last_seg(pkt_hdr)))
>> + if (odp_unlikely(seg >= packet_last_seg(pkt_hdr)))
>> return ODP_PACKET_SEG_INVALID;
>>
>> return seg + 1;
>> @@ -1400,21 +1581,51 @@ odp_packet_seg_t odp_packet_next_seg(odp_packet_t
>> pkt,
>> odp_packet_seg_t seg)
>> void *odp_packet_seg_data(odp_packet_t pkt, odp_packet_seg_t seg)
>> {
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> + uint32_t seg_offset = 0, offset = 0, i;
>> +
>> + while (seg >= pkt_hdr->buf_hdr.segcount - seg_offset &&
>> + pkt_hdr->ref_hdr) {
>> + seg -= (pkt_hdr->buf_hdr.segcount - seg_offset);
>> + offset = pkt_hdr->ref_offset;
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + for (i = 0, seg_offset = 0;
>> + i < pkt_hdr->buf_hdr.segcount;
>> + i++, seg_offset++) {
>> + if (offset < pkt_hdr->buf_hdr.seg[i].len)
>> + break;
>> + offset -= pkt_hdr->buf_hdr.seg[i].len;
>> + }
>> + }
>>
>> - if (odp_unlikely(seg >= pkt_hdr->buf_hdr.segcount))
>> + if (odp_unlikely(seg + seg_offset >= pkt_hdr->buf_hdr.segcount))
>> return NULL;
>>
>> - return packet_seg_data(pkt_hdr, seg);
>> + return packet_seg_data(pkt_hdr, seg + seg_offset) + offset;
>> }
>>
>> uint32_t odp_packet_seg_data_len(odp_packet_t pkt, odp_packet_seg_t seg)
>> {
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> + uint32_t seg_offset = 0, offset = 0, i;
>> +
>> + while (seg >= pkt_hdr->buf_hdr.segcount - seg_offset &&
>> + pkt_hdr->ref_hdr) {
>> + seg -= (pkt_hdr->buf_hdr.segcount - seg_offset);
>> + offset = pkt_hdr->ref_offset;
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + for (i = 0, seg_offset = 0;
>> + i < pkt_hdr->buf_hdr.segcount;
>> + i++, seg_offset++) {
>> + if (offset < pkt_hdr->buf_hdr.seg[i].len)
>> + break;
>> + offset -= pkt_hdr->buf_hdr.seg[i].len;
>> + }
>> + }
>>
>> - if (odp_unlikely(seg >= pkt_hdr->buf_hdr.segcount))
>> + if (odp_unlikely(seg + seg_offset >= pkt_hdr->buf_hdr.segcount))
>> return 0;
>>
>> - return packet_seg_len(pkt_hdr, seg);
>> + return packet_seg_len(pkt_hdr, seg + seg_offset) - offset;
>> }
>>
>> /*
>> @@ -1428,12 +1639,14 @@ int odp_packet_add_data(odp_packet_t *pkt_ptr,
>> uint32_t offset,
>> uint32_t len)
>> {
>> odp_packet_t pkt = *pkt_ptr;
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> - uint32_t pktlen = pkt_hdr->frame_len;
>> + uint32_t pktlen = packet_len(pkt_hdr);
>> odp_packet_t newpkt;
>>
>> if (offset > pktlen)
>> return -1;
>>
>> + ODP_ASSERT(odp_packet_unshared_len(*pkt_ptr) >= offset);
>> +
>> newpkt = odp_packet_alloc(pkt_hdr->buf_hdr.pool_hdl, pktlen + len);
>>
>> if (newpkt == ODP_PACKET_INVALID)
>> @@ -1496,6 +1709,8 @@ int odp_packet_align(odp_packet_t *pkt, uint32_t
>> offset, uint32_t
>> len,
>> if (align > ODP_CACHE_LINE_SIZE)
>> return -1;
>>
>> + ODP_ASSERT(odp_packet_has_ref(*pkt) == 0);
>> +
>> if (seglen >= len) {
>> misalign = align <= 1 ? 0 :
>> ODP_ALIGN_ROUNDUP(uaddr, align) - uaddr;
>> @@ -1535,10 +1750,13 @@ int odp_packet_concat(odp_packet_t *dst,
>> odp_packet_t src)
>> uint32_t dst_len = dst_hdr->frame_len;
>> uint32_t src_len = src_hdr->frame_len;
>>
>> + ODP_ASSERT(packet_ref_count(dst_hdr) == 1);
>> +
>> /* Do a copy if resulting packet would be out of segments or packets
>> - * are from different pools. */
>> + * are from different pools or src is a reference. */
>> if (odp_unlikely((dst_segs + src_segs) > CONFIG_PACKET_MAX_SEGS) ||
>> - odp_unlikely(dst_pool != src_pool)) {
>> + odp_unlikely(dst_pool != src_pool) ||
>> + odp_unlikely(packet_ref_count(src_hdr)) > 1) {
>> if (odp_packet_extend_tail(dst, src_len, NULL, NULL) >= 0) {
>> (void)odp_packet_copy_from_pkt(*dst, dst_len,
>> src, 0, src_len);
>> @@ -1553,8 +1771,9 @@ int odp_packet_concat(odp_packet_t *dst, odp_packet_t
>> src)
>>
>> add_all_segs(dst_hdr, src_hdr);
>>
>> - dst_hdr->frame_len = dst_len + src_len;
>> - dst_hdr->tailroom = src_hdr->tailroom;
>> + dst_hdr->frame_len = dst_len + src_len;
>> + dst_hdr->unshared_len = dst_len + src_len;
>> + dst_hdr->tailroom = src_hdr->tailroom;
>>
>> /* Data was not moved in memory */
>> return 0;
>> @@ -1567,6 +1786,7 @@ int odp_packet_split(odp_packet_t *pkt, uint32_t len,
>> odp_packet_t
>> *tail)
>> if (len >= pktlen || tail == NULL)
>> return -1;
>>
>> + ODP_ASSERT(odp_packet_unshared_len(*pkt) >= len);
>> *tail = odp_packet_copy_part(*pkt, len, pktlen - len,
>> odp_packet_pool(*pkt));
>>
>> @@ -1577,6 +1797,109 @@ int odp_packet_split(odp_packet_t *pkt, uint32_t
>> len, odp_packet_t
>> *tail)
>> }
>>
>> /*
>> + * References
>> + */
>> +
>> +static inline void packet_ref(odp_packet_hdr_t *pkt_hdr)
>> +{
>> + uint32_t i;
>> + odp_packet_hdr_t *hdr;
>> +
>> + do {
>> + for (i = 0; i < pkt_hdr->buf_hdr.segcount; i++) {
>> + hdr = pkt_hdr->buf_hdr.seg[i].hdr;
>> + packet_ref_inc(hdr);
>> + }
>> +
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + } while (pkt_hdr);
>> +}
>> +
>> +static inline odp_packet_t packet_splice(odp_packet_hdr_t *pkt_hdr,
>> + uint32_t offset,
>> + odp_packet_hdr_t *ref_hdr)
>> +{
>> + /* Catch attempted references to stale handles in debug builds */
>> + ODP_ASSERT(packet_ref_count(pkt_hdr) > 0);
>> +
>> + /* Splicing is from the last section of src pkt */
>> + while (ref_hdr->ref_hdr)
>> + ref_hdr = ref_hdr->ref_hdr;
>> +
>> + /* Find section where splice begins */
>> + while (offset >= pkt_hdr->frame_len && pkt_hdr->ref_hdr) {
>> + offset -= (pkt_hdr->frame_len - pkt_hdr->ref_offset);
>> + offset += (pkt_hdr->ref_hdr->frame_len - pkt_hdr->ref_len);
>> + pkt_hdr = pkt_hdr->ref_hdr;
>> + }
>> +
>> + ref_hdr->ref_hdr = pkt_hdr;
>> + ref_hdr->ref_offset = offset;
>> + ref_hdr->ref_len = pkt_hdr->frame_len;
>> +
>> + if (offset < pkt_hdr->unshared_len)
>> + pkt_hdr->unshared_len = offset;
>> +
>> + packet_ref(pkt_hdr);
>> + return _odp_packet_hdl(ref_hdr);
>> +}
>> +
>> +odp_packet_t odp_packet_ref_static(odp_packet_t pkt)
>> +{
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> +
>> + pkt_hdr->unshared_len = 0;
>> + packet_ref(pkt_hdr);
>> + return pkt;
>> +}
>> +
>> +odp_packet_t odp_packet_ref(odp_packet_t pkt, uint32_t offset)
>> +{
>> + odp_packet_t hdr;
>> + odp_packet_hdr_t *pkt_hdr;
>> +
>> + if (pkt == ODP_PACKET_INVALID)
>> + return ODP_PACKET_INVALID;
>> +
>> + pkt_hdr = odp_packet_hdr(pkt);
>> + if (offset >= packet_len(pkt_hdr))
>> + return ODP_PACKET_INVALID;
>> +
>> + hdr = odp_packet_alloc(odp_packet_pool(pkt), 0);
>> +
>> + if (hdr == ODP_PACKET_INVALID)
>> + return ODP_PACKET_INVALID;
>> +
>> + return packet_splice(pkt_hdr, offset, odp_packet_hdr(hdr));
>> +}
>> +
>> +odp_packet_t odp_packet_ref_pkt(odp_packet_t pkt, uint32_t offset,
>> + odp_packet_t hdr)
>> +{
>> + odp_packet_hdr_t *pkt_hdr;
>> +
>> + if (pkt == ODP_PACKET_INVALID ||
>> + hdr == ODP_PACKET_INVALID ||
>> + pkt == hdr)
>> + return ODP_PACKET_INVALID;
>> +
>> + ODP_ASSERT(odp_packet_has_ref(hdr) == 0);
>> +
>> + pkt_hdr = odp_packet_hdr(pkt);
>> + if (offset >= packet_len(pkt_hdr))
>> + return ODP_PACKET_INVALID;
>> +
>> + return packet_splice(pkt_hdr, offset, odp_packet_hdr(hdr));
>> +}
>> +
>> +int odp_packet_has_ref(odp_packet_t pkt)
>> +{
>> + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>> +
>> + return pkt_hdr->ref_hdr != NULL || packet_ref_count(pkt_hdr) > 1;
>> +}
>> +
>> +/*
>> *
>> * Copy
>> * ********************************************************
>> @@ -1585,8 +1908,7 @@ int odp_packet_split(odp_packet_t *pkt, uint32_t len,
>> odp_packet_t
>> *tail)
>>
>> odp_packet_t odp_packet_copy(odp_packet_t pkt, odp_pool_t pool)
>> {
>> - odp_packet_hdr_t *srchdr = odp_packet_hdr(pkt);
>> - uint32_t pktlen = srchdr->frame_len;
>> + uint32_t pktlen = odp_packet_len(pkt);
>> odp_packet_t newpkt = odp_packet_alloc(pool, pktlen);
>>
>> if (newpkt != ODP_PACKET_INVALID) {
>> @@ -1625,7 +1947,7 @@ int odp_packet_copy_to_mem(odp_packet_t pkt, uint32_t
>> offset,
>> uint8_t *dstaddr = (uint8_t *)dst;
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>>
>> - if (offset + len > pkt_hdr->frame_len)
>> + if (offset + len > packet_len(pkt_hdr))
>> return -1;
>>
>> while (len > 0) {
>> @@ -1649,9 +1971,11 @@ int odp_packet_copy_from_mem(odp_packet_t pkt,
>> uint32_t offset,
>> const uint8_t *srcaddr = (const uint8_t *)src;
>> odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt);
>>
>> - if (offset + len > pkt_hdr->frame_len)
>> + if (offset + len > packet_len(pkt_hdr))
>> return -1;
>>
>> + ODP_ASSERT(odp_packet_unshared_len(pkt) >= offset + len);
>> +
>> while (len > 0) {
>> mapaddr = packet_map(pkt_hdr, offset, &seglen, NULL);
>> cpylen = len > seglen ? seglen : len;
>> @@ -1677,10 +2001,12 @@ int odp_packet_copy_from_pkt(odp_packet_t dst,
>> uint32_t
>> dst_offset,
>> uint32_t src_seglen = 0; /* GCC */
>> int overlap;
>>
>> - if (dst_offset + len > dst_hdr->frame_len ||
>> - src_offset + len > src_hdr->frame_len)
>> + if (dst_offset + len > packet_len(dst_hdr) ||
>> + src_offset + len > packet_len(src_hdr))
>> return -1;
>>
>> + ODP_ASSERT(odp_packet_unshared_len(dst) >= dst_offset + len);
>> +
>> overlap = (dst_hdr == src_hdr &&
>> ((dst_offset <= src_offset &&
>> dst_offset + len >= src_offset) ||
>> @@ -1764,7 +2090,7 @@ void odp_packet_print(odp_packet_t pkt)
>> len += snprintf(&str[len], n - len,
>> " l4_offset %" PRIu32 "\n", hdr->p.l4_offset);
>> len += snprintf(&str[len], n - len,
>> - " frame_len %" PRIu32 "\n", hdr->frame_len);
>> + " frame_len %" PRIu32 "\n", packet_len(hdr));
>> len += snprintf(&str[len], n - len,
>> " input %" PRIu64 "\n",
>> odp_pktio_to_u64(hdr->input));
>> --
>> 2.9.3
>
