Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Ola Liljedahl]

On 3 June 2015 at 14:31, Benoît Ganne  wrote:

> If I use D-BUS or ZeroMQ, I get a socket descriptor. I don't want to
 block on this descriptor, my application is likely blocking in
 odp_schedule() or polling some queues. I could allocate an event (of
 whatever type) and associate that with the socket descriptor.
 When the socket can read or written, the event will be enqueued
 on some queue (also specified). My application will receive the
 notification event and will perform all message operations on the
 non-blocking socket.
 When no more messages to read or message write would block, the
 application re-registers the event and queue.
 The only API we would need is:
 void odp_notification_register(int sd, odp_event_t evt,
 odp_queue_t queue);
 Perhaps also a call to unregister the notification, to be used
 at termination. This would be a nice and generic design (not only
 for this specific use case). But it also seems limited to e.g.
 Linux and similar operating systems. What about other operating
 environments (e.g. bare metal)?
 Would those environments not implement this API? Can we have an
 ODP API which only is implemented on some platforms/environments?

>>>
>  Why not using standard odp_queue_enq() then?
>>> I mean, you could create a thread which spin over recv() and enqueue
>>> an event with the received message to your queue. In that case your
>>> datapath threads would just have to deal with queues the usual way.
>>> And this pattern would work with the current API and for any
>>> notification mechanism, socket, ZeroMQ, POSIX signals.
>>> Any thoughts?
>>>
>>
>  I want my worker threads to have full and exclusive control of the
>> dataplane CPU's, no time slicing here, no OS invocation at all. Any
>> additional threads would have to run on some additional CPU which shares
>> memory and queues with the dataplane CPU's. (I suddenly see a use for
>> Petri's thread type proposal). Are such CPU's always going to be
>> available?
>> So I can see other (simpler) solutions but no guaranteed portability
>> between different ODP platforms. You people (specifically you users)
>> think this is a good trade-off?
>>
>
> I think Petri's proposal could take care of this, along with its new
> addition about cpumasks we could have a nice solution.
> Regarding portability, you won't necessarily have sockets in bare metal
> environment anyway.
> In our solution for example, we have those extra CPUs for control-plane,
> but you don't have sockets on bare metal (at least for now).



> The only "ODP portable" communication so far is pktio.
>
Which can always be used as the implementation for a MSGIO API.

But for now I will continue my prototyping using a non-worker thread in the
dataplane application that interacts with the environment (OS/kernel) and
performs any blocking system calls. Incoming messages can be passed as
events on queues to the worker threads. I will have to see what works for
outgoing messages (sent by the worker threads), possibly the non-worker
thread will have to act as an intermediary here as well.


>
> ben
>
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Benoît Ganne]

If I use D-BUS or ZeroMQ, I get a socket descriptor. I don't want to
block on this descriptor, my application is likely blocking in
odp_schedule() or polling some queues. I could allocate an event (of
whatever type) and associate that with the socket descriptor.
When the socket can read or written, the event will be enqueued
on some queue (also specified). My application will receive the
notification event and will perform all message operations on the
non-blocking socket.
When no more messages to read or message write would block, the
application re-registers the event and queue.
The only API we would need is:
void odp_notification_register(int sd, odp_event_t evt,
odp_queue_t queue);
Perhaps also a call to unregister the notification, to be used
at termination. This would be a nice and generic design (not only
for this specific use case). But it also seems limited to e.g.
Linux and similar operating systems. What about other operating
environments (e.g. bare metal)?
Would those environments not implement this API? Can we have an
ODP API which only is implemented on some platforms/environments?



Why not using standard odp_queue_enq() then?
I mean, you could create a thread which spin over recv() and enqueue
an event with the received message to your queue. In that case your
datapath threads would just have to deal with queues the usual way.
And this pattern would work with the current API and for any
notification mechanism, socket, ZeroMQ, POSIX signals.
Any thoughts?



I want my worker threads to have full and exclusive control of the
dataplane CPU's, no time slicing here, no OS invocation at all. Any
additional threads would have to run on some additional CPU which shares
memory and queues with the dataplane CPU's. (I suddenly see a use for
Petri's thread type proposal). Are such CPU's always going to be available?
So I can see other (simpler) solutions but no guaranteed portability
between different ODP platforms. You people (specifically you users)
think this is a good trade-off?


I think Petri's proposal could take care of this, along with its new 
addition about cpumasks we could have a nice solution.
Regarding portability, you won't necessarily have sockets in bare metal 
environment anyway.
In our solution for example, we have those extra CPUs for control-plane, 
but you don't have sockets on bare metal (at least for now). The only 
"ODP portable" communication so far is pktio.


ben
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Ola Liljedahl]

On 2 June 2015 at 16:46, Benoît Ganne  wrote:

> Thinking about using notification events instead of the messages
>> themselves...
>> If I use D-BUS or ZeroMQ, I get a socket descriptor. I don't want to
>> block on this descriptor, my application is likely blocking in
>> odp_schedule() or polling some queues. I could allocate an event (of
>> whatever type) and associate that with the socket descriptor. When the
>> socket can read or written, the event will be enqueued on some queue
>> (also specified). My application will receive the notification event and
>> will perform all message operations on the non-blocking socket. When no
>> more messages to read or message write would block, the application
>> re-registers the event and queue.
>>
>> The only API we would need is:
>> void odp_notification_register(int sd, odp_event_t evt, odp_queue_t
>> queue);
>> Perhaps also a call to unregister the notification, to be used at
>> termination.
>>
>> This would be a nice and generic design (not only for this specific use
>> case). But it also seems limited to e.g. Linux and similar operating
>> systems. What about other operating environments (e.g. bare metal)?
>> Would those environments not implement this API? Can we have an ODP API
>> which only is implemented on some platforms/environments?
>>
>
> Why not using standard odp_queue_enq() then?
> I mean, you could create a thread which spin over recv() and enqueue an
> event with the received message to your queue. In that case your datapath
> threads would just have to deal with queues the usual way.
> And this pattern would work with the current API and for any notification
> mechanism, socket, ZeroMQ, POSIX signals.
> Any thoughts?
>
I want my worker threads to have full and exclusive control of the
dataplane CPU's, no time slicing here, no OS invocation at all. Any
additional threads would have to run on some additional CPU which shares
memory and queues with the dataplane CPU's. (I suddenly see a use for
Petri's thread type proposal). Are such CPU's always going to be available?

So I can see other (simpler) solutions but no guaranteed portability
between different ODP platforms. You people (specifically you users) think
this is a good trade-off?

-- Ola

>
> ben
>
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Benoît Ganne]

Thinking about using notification events instead of the messages
themselves...
If I use D-BUS or ZeroMQ, I get a socket descriptor. I don't want to
block on this descriptor, my application is likely blocking in
odp_schedule() or polling some queues. I could allocate an event (of
whatever type) and associate that with the socket descriptor. When the
socket can read or written, the event will be enqueued on some queue
(also specified). My application will receive the notification event and
will perform all message operations on the non-blocking socket. When no
more messages to read or message write would block, the application
re-registers the event and queue.

The only API we would need is:
void odp_notification_register(int sd, odp_event_t evt, odp_queue_t queue);
Perhaps also a call to unregister the notification, to be used at
termination.

This would be a nice and generic design (not only for this specific use
case). But it also seems limited to e.g. Linux and similar operating
systems. What about other operating environments (e.g. bare metal)?
Would those environments not implement this API? Can we have an ODP API
which only is implemented on some platforms/environments?


Why not using standard odp_queue_enq() then?
I mean, you could create a thread which spin over recv() and enqueue an 
event with the received message to your queue. In that case your 
datapath threads would just have to deal with queues the usual way.
And this pattern would work with the current API and for any 
notification mechanism, socket, ZeroMQ, POSIX signals.

Any thoughts?

ben
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Ola Liljedahl]

On 2 June 2015 at 15:35, Ola Liljedahl  wrote:

>
>
> On 2 June 2015 at 14:36, Benoît Ganne  wrote:
>
>> Hi Ola, all,
>>
>> On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
>>
>>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>>> passing
>>> based IPC for a shared nothing architecture.
>>>
>>
>> I understand the value of this, but I feel I need to ask the question:
>> does it really belong to ODP ? My understanding so far was that ODP should
>> not try to replace OS services: for example, thread or process related API
>> were ruled out. This looks to me a lot like D-BUS or ZeroMQ (or > favorite solution here>). Wouldn't it be better to let the application
>> select what they want to use instead of providing our own?
>>
> Yes but I want MSGIO IPC to be part of the ODP functionality such as
> scheduling, not something completely separate. So messages need to be some
> type of events (or at least incoming messages notified using events) and
> when you set up your message IO, input messages (or notifications) are put
> on some queue. Perhaps this can be done differently. It would be great of
> there was a generic way of including alien functionality (e.g. D-BUS or
> ZeroMQ as you mention) in the ODP framework. Any suggestions how this could
> be done?
>

Thinking about using notification events instead of the messages
themselves...

If I use D-BUS or ZeroMQ, I get a socket descriptor. I don't want to block
on this descriptor, my application is likely blocking in odp_schedule() or
polling some queues. I could allocate an event (of whatever type) and
associate that with the socket descriptor. When the socket can read or
written, the event will be enqueued on some queue (also specified). My
application will receive the notification event and will perform all
message operations on the non-blocking socket. When no more messages to
read or message write would block, the application re-registers the event
and queue.

The only API we would need is:
void odp_notification_register(int sd, odp_event_t evt, odp_queue_t queue);
Perhaps also a call to unregister the notification, to be used at
termination.

This would be a nice and generic design (not only for this specific use
case). But it also seems limited to e.g. Linux and similar operating
systems. What about other operating environments (e.g. bare metal)? Would
those environments not implement this API? Can we have an ODP API which
only is implemented on some platforms/environments?


>
>> ben
>>
>>
>>  IPC/message passing can be separated into a number of different layers.
>>> 1) Message formats and protocols (as seen by the user).
>>> 2) API - defines syntax and semantics of MSGIO operations.
>>> 3) Transport interface - defines a binary packet format used by MSGIO
>>> endpoints/implementations. Cf. OSI network layer.
>>> 4) Transport layer - implements the transport of message packet between
>>> different endpoints. Cf. OSI link layer.
>>>
>>> #1 is application specific and there are many different standards (e.g.
>>> Linux
>>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>>> #3 may be implementation specific but is important for interoperatibility
>>> (binary compatibility). If the user message (payload), operation and
>>> parameters are encoded in a well-defined way (e.g. some kind of packet
>>> format
>>> which includes operation, sender and receiver addresses, payload etc),
>>> inter-operatibility between different ODP/MSGIO implementations may be
>>> achieved or simplified.
>>> #4 is implementation (platform) specific.
>>>
>>> This proposal focuses on the API which is fundamental for source code
>>> compatibility between different platforms (ODP implementations) - this
>>> being
>>> the primary purpose of ODP. A packet format for messages will be
>>> specified as
>>> part of the reference implementation and may be seen as a proposal for a
>>> standard packet format for binary compatibility between different
>>> ODP/MSGIO
>>> implementations.
>>>
>>> All MSGIO calls are non-blocking (with the possible exception of
>>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>>> transfer is asynchronous so it is not possible to return any status
>>> operation concerning the eventual (or remote) result of an operation.
>>>
>>> The semantics of MSGIO message passing is that sending a message to an
>>> endpoint
>>> will always look like it succeeds. The appearance of endpoints is
>>> explicitly
>>> notified through user-defined messages specified in the
>>> odp_msgio_lookup()
>>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>>> connection
>>> to endpoint) is also explicitly notified through user-defined messages
>>> specified in the odp_msgio_monitor() call. The send call does not fail
>>> because
>>> the addressed endpoint has disappeared, the message is just silently
>>> discarded.
>>>
>>> Message delivery into the recipient address space is ordered (per
>>> priority)
>>>

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Ola Liljedahl]

On 2 June 2015 at 14:53, Jerin Jacob  wrote:

> On Tue, Jun 02, 2015 at 02:36:47PM +0200, Benoît Ganne wrote:
> > Hi Ola, all,
> >
> > On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
> > >Here is my third attempt at a ODP API for MSGIO - asynchronous message
> passing
> > >based IPC for a shared nothing architecture.
> >
> > I understand the value of this, but I feel I need to ask the question:
> does
> > it really belong to ODP ? My understanding so far was that ODP should not
> > try to replace OS services: for example, thread or process related API
> were
> > ruled out. This looks to me a lot like D-BUS or ZeroMQ (or  > favorite solution here>). Wouldn't it be better to let the application
> > select what they want to use instead of providing our own?
>
> +1 and I am not sure how much value addition it has in hardware
> abstraction perspective.
> Most of the HW accelerated platforms will map odp_msg_t as packet and
> odp_msgio_t as pktio
> then it will boils down to a helper function that creates the
> vlan packet/message from given message(uint8_t *), src, dst 48 bit address
>
That's a perfectly valid implementation strategy. But as I wanted to make
the API independent of the underlying HW and ODP implementation I didn't
want to mandate that specific design.

Bala suggested that we actually use odp_packet_t as the message type and
that could work from an API perspective. I don't know if this would
introduce some limitations (packet pools may be special HW constructs (e.g.
limited in number or other characteristics) on some target, can they then
be used for MSGIO as well?).


> >
> > ben
> >
> > >IPC/message passing can be separated into a number of different layers.
> > >1) Message formats and protocols (as seen by the user).
> > >2) API - defines syntax and semantics of MSGIO operations.
> > >3) Transport interface - defines a binary packet format used by MSGIO
> > >endpoints/implementations. Cf. OSI network layer.
> > >4) Transport layer - implements the transport of message packet between
> > >different endpoints. Cf. OSI link layer.
> > >
> > >#1 is application specific and there are many different standards (e.g.
> Linux
> > >Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
> > >#3 may be implementation specific but is important for
> interoperatibility
> > >(binary compatibility). If the user message (payload), operation and
> > >parameters are encoded in a well-defined way (e.g. some kind of packet
> format
> > >which includes operation, sender and receiver addresses, payload etc),
> > >inter-operatibility between different ODP/MSGIO implementations may be
> > >achieved or simplified.
> > >#4 is implementation (platform) specific.
> > >
> > >This proposal focuses on the API which is fundamental for source code
> > >compatibility between different platforms (ODP implementations) - this
> being
> > >the primary purpose of ODP. A packet format for messages will be
> specified as
> > >part of the reference implementation and may be seen as a proposal for a
> > >standard packet format for binary compatibility between different
> ODP/MSGIO
> > >implementations.
> > >
> > >All MSGIO calls are non-blocking (with the possible exception of
> > >odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
> > >transfer is asynchronous so it is not possible to return any status
> > >operation concerning the eventual (or remote) result of an operation.
> > >
> > >The semantics of MSGIO message passing is that sending a message to an
> endpoint
> > >will always look like it succeeds. The appearance of endpoints is
> explicitly
> > >notified through user-defined messages specified in the
> odp_msgio_lookup()
> > >call. Similarly, the disappearance (e.g. death of or otherwise lost
> connection
> > >to endpoint) is also explicitly notified through user-defined messages
> > >specified in the odp_msgio_monitor() call. The send call does not fail
> because
> > >the addressed endpoint has disappeared, the message is just silently
> discarded.
> > >
> > >Message delivery into the recipient address space is ordered (per
> priority)
> > >and reliable. Delivery of message N implies delivery of all messages  > >(of the same priority). All messages (accepted by MSGIO send operation)
> will
> > >be delivered up to the point of endpoint termination or lost connection
> > >where no more messages will be delivered. Actual reception (dequeueing)
> and
> > >processing by the recipient is not guaranteed (use end-to-end
> acknowledgements
> > >for that).
> > >
> > >MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
> > >multidrop network where each endpoint has a unique address which is only
> > >valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
> > >and then recreated (with the same name), the new endpoint will have a
> > >new unique address (eventually endpoints addresses will have to be
> recycled
> > >but not for a very long time). Endpoints nam

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Ola Liljedahl]

On 2 June 2015 at 14:36, Benoît Ganne  wrote:

> Hi Ola, all,
>
> On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
>
>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>> passing
>> based IPC for a shared nothing architecture.
>>
>
> I understand the value of this, but I feel I need to ask the question:
> does it really belong to ODP ? My understanding so far was that ODP should
> not try to replace OS services: for example, thread or process related API
> were ruled out. This looks to me a lot like D-BUS or ZeroMQ (or  favorite solution here>). Wouldn't it be better to let the application
> select what they want to use instead of providing our own?
>
Yes but I want MSGIO IPC to be part of the ODP functionality such as
scheduling, not something completely separate. So messages need to be some
type of events (or at least incoming messages notified using events) and
when you set up your message IO, input messages (or notifications) are put
on some queue. Perhaps this can be done differently. It would be great of
there was a generic way of including alien functionality (e.g. D-BUS or
ZeroMQ as you mention) in the ODP framework. Any suggestions how this could
be done?


> ben
>
>
>  IPC/message passing can be separated into a number of different layers.
>> 1) Message formats and protocols (as seen by the user).
>> 2) API - defines syntax and semantics of MSGIO operations.
>> 3) Transport interface - defines a binary packet format used by MSGIO
>> endpoints/implementations. Cf. OSI network layer.
>> 4) Transport layer - implements the transport of message packet between
>> different endpoints. Cf. OSI link layer.
>>
>> #1 is application specific and there are many different standards (e.g.
>> Linux
>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>> #3 may be implementation specific but is important for interoperatibility
>> (binary compatibility). If the user message (payload), operation and
>> parameters are encoded in a well-defined way (e.g. some kind of packet
>> format
>> which includes operation, sender and receiver addresses, payload etc),
>> inter-operatibility between different ODP/MSGIO implementations may be
>> achieved or simplified.
>> #4 is implementation (platform) specific.
>>
>> This proposal focuses on the API which is fundamental for source code
>> compatibility between different platforms (ODP implementations) - this
>> being
>> the primary purpose of ODP. A packet format for messages will be
>> specified as
>> part of the reference implementation and may be seen as a proposal for a
>> standard packet format for binary compatibility between different
>> ODP/MSGIO
>> implementations.
>>
>> All MSGIO calls are non-blocking (with the possible exception of
>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>> transfer is asynchronous so it is not possible to return any status
>> operation concerning the eventual (or remote) result of an operation.
>>
>> The semantics of MSGIO message passing is that sending a message to an
>> endpoint
>> will always look like it succeeds. The appearance of endpoints is
>> explicitly
>> notified through user-defined messages specified in the odp_msgio_lookup()
>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>> connection
>> to endpoint) is also explicitly notified through user-defined messages
>> specified in the odp_msgio_monitor() call. The send call does not fail
>> because
>> the addressed endpoint has disappeared, the message is just silently
>> discarded.
>>
>> Message delivery into the recipient address space is ordered (per
>> priority)
>> and reliable. Delivery of message N implies delivery of all messages > (of the same priority). All messages (accepted by MSGIO send operation)
>> will
>> be delivered up to the point of endpoint termination or lost connection
>> where no more messages will be delivered. Actual reception (dequeueing)
>> and
>> processing by the recipient is not guaranteed (use end-to-end
>> acknowledgements
>> for that).
>>
>> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
>> multidrop network where each endpoint has a unique address which is only
>> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
>> and then recreated (with the same name), the new endpoint will have a
>> new unique address (eventually endpoints addresses will have to be
>> recycled
>> but not for a very long time). Endpoints names do not necessarily have to
>> be
>> unique. The scope of message network is not defined but it is expected
>> that
>> the scope by default corresponds to an OS instance (e.g. virtual machine).
>>
>> Proposed transport packet for message I/O (not visible to the user):
>> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
>> uint8_t src_addr[6]; /* Unique address of source endpoint */
>> uint16_t frametype_o; /* Outer frametype, 0x8100 */
>> uint16_t pcp_vid; /* 3 bits of Priority C

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Mike Holmes]

On 2 June 2015 at 08:53, Jerin Jacob  wrote:

> On Tue, Jun 02, 2015 at 02:36:47PM +0200, Benoît Ganne wrote:
> > Hi Ola, all,
> >
> > On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
> > >Here is my third attempt at a ODP API for MSGIO - asynchronous message
> passing
> > >based IPC for a shared nothing architecture.
> >
> > I understand the value of this, but I feel I need to ask the question:
> does
> > it really belong to ODP ? My understanding so far was that ODP should not
> > try to replace OS services: for example, thread or process related API
> were
> > ruled out. This looks to me a lot like D-BUS or ZeroMQ (or  > favorite solution here>). Wouldn't it be better to let the application
> > select what they want to use instead of providing our own?
>
> +1 and I am not sure how much value addition it has in hardware
> abstraction perspective.
>

Just a thought, but if it is not a HW abstraction that makes it a candidate
for a helper, assuming the API can support capabilities the helper will
need.


> Most of the HW accelerated platforms will map odp_msg_t as packet and
> odp_msgio_t as pktio
> then it will boils down to a helper function that creates the
> vlan packet/message from given message(uint8_t *), src, dst 48 bit address
>
> >
> > ben
> >
> > >IPC/message passing can be separated into a number of different layers.
> > >1) Message formats and protocols (as seen by the user).
> > >2) API - defines syntax and semantics of MSGIO operations.
> > >3) Transport interface - defines a binary packet format used by MSGIO
> > >endpoints/implementations. Cf. OSI network layer.
> > >4) Transport layer - implements the transport of message packet between
> > >different endpoints. Cf. OSI link layer.
> > >
> > >#1 is application specific and there are many different standards (e.g.
> Linux
> > >Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
> > >#3 may be implementation specific but is important for
> interoperatibility
> > >(binary compatibility). If the user message (payload), operation and
> > >parameters are encoded in a well-defined way (e.g. some kind of packet
> format
> > >which includes operation, sender and receiver addresses, payload etc),
> > >inter-operatibility between different ODP/MSGIO implementations may be
> > >achieved or simplified.
> > >#4 is implementation (platform) specific.
> > >
> > >This proposal focuses on the API which is fundamental for source code
> > >compatibility between different platforms (ODP implementations) - this
> being
> > >the primary purpose of ODP. A packet format for messages will be
> specified as
> > >part of the reference implementation and may be seen as a proposal for a
> > >standard packet format for binary compatibility between different
> ODP/MSGIO
> > >implementations.
> > >
> > >All MSGIO calls are non-blocking (with the possible exception of
> > >odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
> > >transfer is asynchronous so it is not possible to return any status
> > >operation concerning the eventual (or remote) result of an operation.
> > >
> > >The semantics of MSGIO message passing is that sending a message to an
> endpoint
> > >will always look like it succeeds. The appearance of endpoints is
> explicitly
> > >notified through user-defined messages specified in the
> odp_msgio_lookup()
> > >call. Similarly, the disappearance (e.g. death of or otherwise lost
> connection
> > >to endpoint) is also explicitly notified through user-defined messages
> > >specified in the odp_msgio_monitor() call. The send call does not fail
> because
> > >the addressed endpoint has disappeared, the message is just silently
> discarded.
> > >
> > >Message delivery into the recipient address space is ordered (per
> priority)
> > >and reliable. Delivery of message N implies delivery of all messages  > >(of the same priority). All messages (accepted by MSGIO send operation)
> will
> > >be delivered up to the point of endpoint termination or lost connection
> > >where no more messages will be delivered. Actual reception (dequeueing)
> and
> > >processing by the recipient is not guaranteed (use end-to-end
> acknowledgements
> > >for that).
> > >
> > >MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
> > >multidrop network where each endpoint has a unique address which is only
> > >valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
> > >and then recreated (with the same name), the new endpoint will have a
> > >new unique address (eventually endpoints addresses will have to be
> recycled
> > >but not for a very long time). Endpoints names do not necessarily have
> to be
> > >unique. The scope of message network is not defined but it is expected
> that
> > >the scope by default corresponds to an OS instance (e.g. virtual
> machine).
> > >
> > >Proposed transport packet for message I/O (not visible to the user):
> > >uint8_t dest_addr[6]; /* Unique address of destination endpoint 

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Jerin Jacob]

On Tue, Jun 02, 2015 at 02:36:47PM +0200, Benoît Ganne wrote:
> Hi Ola, all,
> 
> On 05/27/2015 10:08 AM, Ola Liljedahl wrote:
> >Here is my third attempt at a ODP API for MSGIO - asynchronous message 
> >passing
> >based IPC for a shared nothing architecture.
> 
> I understand the value of this, but I feel I need to ask the question: does
> it really belong to ODP ? My understanding so far was that ODP should not
> try to replace OS services: for example, thread or process related API were
> ruled out. This looks to me a lot like D-BUS or ZeroMQ (or  favorite solution here>). Wouldn't it be better to let the application
> select what they want to use instead of providing our own?

+1 and I am not sure how much value addition it has in hardware abstraction 
perspective.
Most of the HW accelerated platforms will map odp_msg_t as packet and 
odp_msgio_t as pktio
then it will boils down to a helper function that creates the
vlan packet/message from given message(uint8_t *), src, dst 48 bit address

> 
> ben
> 
> >IPC/message passing can be separated into a number of different layers.
> >1) Message formats and protocols (as seen by the user).
> >2) API - defines syntax and semantics of MSGIO operations.
> >3) Transport interface - defines a binary packet format used by MSGIO
> >endpoints/implementations. Cf. OSI network layer.
> >4) Transport layer - implements the transport of message packet between
> >different endpoints. Cf. OSI link layer.
> >
> >#1 is application specific and there are many different standards (e.g. Linux
> >Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
> >#3 may be implementation specific but is important for interoperatibility
> >(binary compatibility). If the user message (payload), operation and
> >parameters are encoded in a well-defined way (e.g. some kind of packet format
> >which includes operation, sender and receiver addresses, payload etc),
> >inter-operatibility between different ODP/MSGIO implementations may be
> >achieved or simplified.
> >#4 is implementation (platform) specific.
> >
> >This proposal focuses on the API which is fundamental for source code
> >compatibility between different platforms (ODP implementations) - this being
> >the primary purpose of ODP. A packet format for messages will be specified as
> >part of the reference implementation and may be seen as a proposal for a
> >standard packet format for binary compatibility between different ODP/MSGIO
> >implementations.
> >
> >All MSGIO calls are non-blocking (with the possible exception of
> >odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
> >transfer is asynchronous so it is not possible to return any status
> >operation concerning the eventual (or remote) result of an operation.
> >
> >The semantics of MSGIO message passing is that sending a message to an 
> >endpoint
> >will always look like it succeeds. The appearance of endpoints is explicitly
> >notified through user-defined messages specified in the odp_msgio_lookup()
> >call. Similarly, the disappearance (e.g. death of or otherwise lost 
> >connection
> >to endpoint) is also explicitly notified through user-defined messages
> >specified in the odp_msgio_monitor() call. The send call does not fail 
> >because
> >the addressed endpoint has disappeared, the message is just silently 
> >discarded.
> >
> >Message delivery into the recipient address space is ordered (per priority)
> >and reliable. Delivery of message N implies delivery of all messages  >(of the same priority). All messages (accepted by MSGIO send operation) will
> >be delivered up to the point of endpoint termination or lost connection
> >where no more messages will be delivered. Actual reception (dequeueing) and
> >processing by the recipient is not guaranteed (use end-to-end 
> >acknowledgements
> >for that).
> >
> >MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
> >multidrop network where each endpoint has a unique address which is only
> >valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
> >and then recreated (with the same name), the new endpoint will have a
> >new unique address (eventually endpoints addresses will have to be recycled
> >but not for a very long time). Endpoints names do not necessarily have to be
> >unique. The scope of message network is not defined but it is expected that
> >the scope by default corresponds to an OS instance (e.g. virtual machine).
> >
> >Proposed transport packet for message I/O (not visible to the user):
> >uint8_t dest_addr[6]; /* Unique address of destination endpoint */
> >uint8_t src_addr[6]; /* Unique address of source endpoint */
> >uint16_t frametype_o; /* Outer frametype, 0x8100 */
> >uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
> >uint16_t frametype_i; /* Inner frametype, TBD */
> >uint8_t payload[]; /* User message */
> >Looks very much like a VLAN-tagged Ethernet frame so should easily be
> >transported by a

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Benoît Ganne]

Hi Ola, all,

On 05/27/2015 10:08 AM, Ola Liljedahl wrote:

Here is my third attempt at a ODP API for MSGIO - asynchronous message passing
based IPC for a shared nothing architecture.


I understand the value of this, but I feel I need to ask the question: 
does it really belong to ODP ? My understanding so far was that ODP 
should not try to replace OS services: for example, thread or process 
related API were ruled out. This looks to me a lot like D-BUS or ZeroMQ 
(or ). Wouldn't it be better to let 
the application select what they want to use instead of providing our own?


ben


IPC/message passing can be separated into a number of different layers.
1) Message formats and protocols (as seen by the user).
2) API - defines syntax and semantics of MSGIO operations.
3) Transport interface - defines a binary packet format used by MSGIO
endpoints/implementations. Cf. OSI network layer.
4) Transport layer - implements the transport of message packet between
different endpoints. Cf. OSI link layer.

#1 is application specific and there are many different standards (e.g. Linux
Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
#3 may be implementation specific but is important for interoperatibility
(binary compatibility). If the user message (payload), operation and
parameters are encoded in a well-defined way (e.g. some kind of packet format
which includes operation, sender and receiver addresses, payload etc),
inter-operatibility between different ODP/MSGIO implementations may be
achieved or simplified.
#4 is implementation (platform) specific.

This proposal focuses on the API which is fundamental for source code
compatibility between different platforms (ODP implementations) - this being
the primary purpose of ODP. A packet format for messages will be specified as
part of the reference implementation and may be seen as a proposal for a
standard packet format for binary compatibility between different ODP/MSGIO
implementations.

All MSGIO calls are non-blocking (with the possible exception of
odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
transfer is asynchronous so it is not possible to return any status
operation concerning the eventual (or remote) result of an operation.

The semantics of MSGIO message passing is that sending a message to an endpoint
will always look like it succeeds. The appearance of endpoints is explicitly
notified through user-defined messages specified in the odp_msgio_lookup()
call. Similarly, the disappearance (e.g. death of or otherwise lost connection
to endpoint) is also explicitly notified through user-defined messages
specified in the odp_msgio_monitor() call. The send call does not fail because
the addressed endpoint has disappeared, the message is just silently discarded.

Message delivery into the recipient address space is ordered (per priority)
and reliable. Delivery of message N implies delivery of all messages 
---
(This document/code contribution attached is provided under the terms of
agreement LES-LTM-21309)

  include/odp/api/message.h  | 199 ++
  include/odp/api/msgio.h| 222 +
  platform/linux-generic/include/odp/message.h   |  39 
  platform/linux-generic/include/odp/msgio.h |  40 
  .../linux-generic/include/odp/plat/message_types.h |  47 +
  .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++
  6 files changed, 606 insertions(+)
  create mode 100644 include/odp/api/message.h
  create mode 100644 include/odp/api/msgio.h
  create mode 100644 platform/linux-generic/include/odp/message.h
  create mode 100644 platform/linux-generic/include/odp/msgio.h
  create mode 100644 platform/linux-generic/include/odp/plat/message_types.h
  create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h

diff --git a/include/odp/api/message.h b/include/odp/api/message.h
new file mode 100644
index 000..76697fb
--- /dev/null
+++ b/include/odp/api/message.h
@@ -0,0 +1,199 @@
+/* Copyright (c) 2015, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+
+/**
+ * @file
+ *
+ * ODP Message event type
+ */
+
+#ifndef ODP_API_MESSAGE_H_
+#define ODP_API_MESSAGE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/** @defgroup odp_message ODP MESSAGE
+ *  @{
+ */
+
+/**
+ * @typedef odp_message_t
+ * ODP message handle
+ */
+
+
+/**
+ * Get address of sender (source) of message
+ *
+ * @param  msg  Message handle
+ * @param[out] addr Buffer to hold sender address
+ */
+void odp_message_sender(odp_message_t msg,
+   uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
+
+/**
+ * Message data pointer
+ *
+ * Return a pointer to the message data
+ *
+ * @param msg Message handle
+ *
+ * @return Pointer to the message data
+ */
+void *odp_message_data(odp_message_t msg);
+
+/**
+ * Message data length
+ *
+ * Return length of the message data.
+ *
+ * @pa

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Ola Liljedahl]

On 27 May 2015 at 14:39, Bill Fischofer  wrote:

> If you want to use an Ethernet-like internal transport, you might was well
> use a standard one like TIPC  rather than
> reinventing this particular wheel.
>
I don't want to use or enforce an Ethernet-like internal transport. I am
just proposing a standardized message format (user payload + metadata) for
connecting to the message network/bus so that ODP applications can be
binary interoperable regardless of which ODP implementation they are using.
Everyone seems to be misunderstanding my intention here. I could define
this standard format without trying to emulate any known packet format,
would that be better?

I am aware of TIPC on a high level but isn't the semantics a bit different
from what I propose? E.g. it uses well-known (numeric) ports for
applications to listen to? Does it have a suitable packet format that can
be reused?



>
> On Wed, May 27, 2015 at 4:49 AM, Ola Liljedahl 
> wrote:
>
>> On 27 May 2015 at 11:10, Alexandru Badicioiu <
>> alexandru.badici...@linaro.org> wrote:
>>
>>>
>>>
>>> On 27 May 2015 at 11:08, Ola Liljedahl  wrote:
>>>
 Here is my third attempt at a ODP API for MSGIO - asynchronous message
 passing
 based IPC for a shared nothing architecture.

 IPC/message passing can be separated into a number of different layers.
 1) Message formats and protocols (as seen by the user).
 2) API - defines syntax and semantics of MSGIO operations.
 3) Transport interface - defines a binary packet format used by MSGIO
endpoints/implementations. Cf. OSI network layer.
 4) Transport layer - implements the transport of message packet between
different endpoints. Cf. OSI link layer.

 #1 is application specific and there are many different standards (e.g.
 Linux
 Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
 #3 may be implementation specific but is important for
 interoperatibility
 (binary compatibility). If the user message (payload), operation and
 parameters are encoded in a well-defined way (e.g. some kind of packet
 format
 which includes operation, sender and receiver addresses, payload etc),
 inter-operatibility between different ODP/MSGIO implementations may be
 achieved or simplified.
 #4 is implementation (platform) specific.

 This proposal focuses on the API which is fundamental for source code
 compatibility between different platforms (ODP implementations) - this
 being
 the primary purpose of ODP. A packet format for messages will be
 specified as
 part of the reference implementation and may be seen as a proposal for a
 standard packet format for binary compatibility between different
 ODP/MSGIO
 implementations.

 All MSGIO calls are non-blocking (with the possible exception of
 odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
 transfer is asynchronous so it is not possible to return any status
 operation concerning the eventual (or remote) result of an operation.

 The semantics of MSGIO message passing is that sending a message to an
 endpoint
 will always look like it succeeds. The appearance of endpoints is
 explicitly
 notified through user-defined messages specified in the
 odp_msgio_lookup()
 call. Similarly, the disappearance (e.g. death of or otherwise lost
 connection
 to endpoint) is also explicitly notified through user-defined messages
 specified in the odp_msgio_monitor() call. The send call does not fail
 because
 the addressed endpoint has disappeared, the message is just silently
 discarded.

 Message delivery into the recipient address space is ordered (per
 priority)
 and reliable. Delivery of message N implies delivery of all messages >>> (of the same priority). All messages (accepted by MSGIO send operation)
 will
 be delivered up to the point of endpoint termination or lost connection
 where no more messages will be delivered. Actual reception (dequeueing)
 and
 processing by the recipient is not guaranteed (use end-to-end
 acknowledgements
 for that).

 MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
 multidrop network where each endpoint has a unique address which is only
 valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
 and then recreated (with the same name), the new endpoint will have a
 new unique address (eventually endpoints addresses will have to be
 recycled
 but not for a very long time). Endpoints names do not necessarily have
 to be
 unique. The scope of message network is not defined but it is expected
 that
 the scope by default corresponds to an OS instance (e.g. virtual
 machine).

 Proposed transport packet for message I/O (not visible to the us

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Bill Fischofer]

If you want to use an Ethernet-like internal transport, you might was well
use a standard one like TIPC  rather than
reinventing this particular wheel.

On Wed, May 27, 2015 at 4:49 AM, Ola Liljedahl 
wrote:

> On 27 May 2015 at 11:10, Alexandru Badicioiu <
> alexandru.badici...@linaro.org> wrote:
>
>>
>>
>> On 27 May 2015 at 11:08, Ola Liljedahl  wrote:
>>
>>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>>> passing
>>> based IPC for a shared nothing architecture.
>>>
>>> IPC/message passing can be separated into a number of different layers.
>>> 1) Message formats and protocols (as seen by the user).
>>> 2) API - defines syntax and semantics of MSGIO operations.
>>> 3) Transport interface - defines a binary packet format used by MSGIO
>>>endpoints/implementations. Cf. OSI network layer.
>>> 4) Transport layer - implements the transport of message packet between
>>>different endpoints. Cf. OSI link layer.
>>>
>>> #1 is application specific and there are many different standards (e.g.
>>> Linux
>>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>>> #3 may be implementation specific but is important for interoperatibility
>>> (binary compatibility). If the user message (payload), operation and
>>> parameters are encoded in a well-defined way (e.g. some kind of packet
>>> format
>>> which includes operation, sender and receiver addresses, payload etc),
>>> inter-operatibility between different ODP/MSGIO implementations may be
>>> achieved or simplified.
>>> #4 is implementation (platform) specific.
>>>
>>> This proposal focuses on the API which is fundamental for source code
>>> compatibility between different platforms (ODP implementations) - this
>>> being
>>> the primary purpose of ODP. A packet format for messages will be
>>> specified as
>>> part of the reference implementation and may be seen as a proposal for a
>>> standard packet format for binary compatibility between different
>>> ODP/MSGIO
>>> implementations.
>>>
>>> All MSGIO calls are non-blocking (with the possible exception of
>>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>>> transfer is asynchronous so it is not possible to return any status
>>> operation concerning the eventual (or remote) result of an operation.
>>>
>>> The semantics of MSGIO message passing is that sending a message to an
>>> endpoint
>>> will always look like it succeeds. The appearance of endpoints is
>>> explicitly
>>> notified through user-defined messages specified in the
>>> odp_msgio_lookup()
>>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>>> connection
>>> to endpoint) is also explicitly notified through user-defined messages
>>> specified in the odp_msgio_monitor() call. The send call does not fail
>>> because
>>> the addressed endpoint has disappeared, the message is just silently
>>> discarded.
>>>
>>> Message delivery into the recipient address space is ordered (per
>>> priority)
>>> and reliable. Delivery of message N implies delivery of all messages >> (of the same priority). All messages (accepted by MSGIO send operation)
>>> will
>>> be delivered up to the point of endpoint termination or lost connection
>>> where no more messages will be delivered. Actual reception (dequeueing)
>>> and
>>> processing by the recipient is not guaranteed (use end-to-end
>>> acknowledgements
>>> for that).
>>>
>>> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
>>> multidrop network where each endpoint has a unique address which is only
>>> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
>>> and then recreated (with the same name), the new endpoint will have a
>>> new unique address (eventually endpoints addresses will have to be
>>> recycled
>>> but not for a very long time). Endpoints names do not necessarily have
>>> to be
>>> unique. The scope of message network is not defined but it is expected
>>> that
>>> the scope by default corresponds to an OS instance (e.g. virtual
>>> machine).
>>>
>>> Proposed transport packet for message I/O (not visible to the user):
>>> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
>>> uint8_t src_addr[6]; /* Unique address of source endpoint */
>>> uint16_t frametype_o; /* Outer frametype, 0x8100 */
>>> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
>>> uint16_t frametype_i; /* Inner frametype, TBD */
>>> uint8_t payload[]; /* User message */
>>> Looks very much like a VLAN-tagged Ethernet frame so should easily be
>>> transported by and processed by Ethernet HW & SW
>>
>> Why to use Ethernet HW & SW to transport messages inside an OS instance?
>> Usually Ethernet HW & SW (drivers?) deal with sending/receiving to/from the
>> wire.
>>
> I don't say you have to. I am just suggesting that *if* we standardize the
> "packet" (message payload + metadata) format, we might as well use a
> well-known format that likely

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Ola Liljedahl]

On 27 May 2015 at 11:10, Alexandru Badicioiu  wrote:

>
>
> On 27 May 2015 at 11:08, Ola Liljedahl  wrote:
>
>> Here is my third attempt at a ODP API for MSGIO - asynchronous message
>> passing
>> based IPC for a shared nothing architecture.
>>
>> IPC/message passing can be separated into a number of different layers.
>> 1) Message formats and protocols (as seen by the user).
>> 2) API - defines syntax and semantics of MSGIO operations.
>> 3) Transport interface - defines a binary packet format used by MSGIO
>>endpoints/implementations. Cf. OSI network layer.
>> 4) Transport layer - implements the transport of message packet between
>>different endpoints. Cf. OSI link layer.
>>
>> #1 is application specific and there are many different standards (e.g.
>> Linux
>> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
>> #3 may be implementation specific but is important for interoperatibility
>> (binary compatibility). If the user message (payload), operation and
>> parameters are encoded in a well-defined way (e.g. some kind of packet
>> format
>> which includes operation, sender and receiver addresses, payload etc),
>> inter-operatibility between different ODP/MSGIO implementations may be
>> achieved or simplified.
>> #4 is implementation (platform) specific.
>>
>> This proposal focuses on the API which is fundamental for source code
>> compatibility between different platforms (ODP implementations) - this
>> being
>> the primary purpose of ODP. A packet format for messages will be
>> specified as
>> part of the reference implementation and may be seen as a proposal for a
>> standard packet format for binary compatibility between different
>> ODP/MSGIO
>> implementations.
>>
>> All MSGIO calls are non-blocking (with the possible exception of
>> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
>> transfer is asynchronous so it is not possible to return any status
>> operation concerning the eventual (or remote) result of an operation.
>>
>> The semantics of MSGIO message passing is that sending a message to an
>> endpoint
>> will always look like it succeeds. The appearance of endpoints is
>> explicitly
>> notified through user-defined messages specified in the odp_msgio_lookup()
>> call. Similarly, the disappearance (e.g. death of or otherwise lost
>> connection
>> to endpoint) is also explicitly notified through user-defined messages
>> specified in the odp_msgio_monitor() call. The send call does not fail
>> because
>> the addressed endpoint has disappeared, the message is just silently
>> discarded.
>>
>> Message delivery into the recipient address space is ordered (per
>> priority)
>> and reliable. Delivery of message N implies delivery of all messages > (of the same priority). All messages (accepted by MSGIO send operation)
>> will
>> be delivered up to the point of endpoint termination or lost connection
>> where no more messages will be delivered. Actual reception (dequeueing)
>> and
>> processing by the recipient is not guaranteed (use end-to-end
>> acknowledgements
>> for that).
>>
>> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
>> multidrop network where each endpoint has a unique address which is only
>> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
>> and then recreated (with the same name), the new endpoint will have a
>> new unique address (eventually endpoints addresses will have to be
>> recycled
>> but not for a very long time). Endpoints names do not necessarily have to
>> be
>> unique. The scope of message network is not defined but it is expected
>> that
>> the scope by default corresponds to an OS instance (e.g. virtual machine).
>>
>> Proposed transport packet for message I/O (not visible to the user):
>> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
>> uint8_t src_addr[6]; /* Unique address of source endpoint */
>> uint16_t frametype_o; /* Outer frametype, 0x8100 */
>> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
>> uint16_t frametype_i; /* Inner frametype, TBD */
>> uint8_t payload[]; /* User message */
>> Looks very much like a VLAN-tagged Ethernet frame so should easily be
>> transported by and processed by Ethernet HW & SW
>
> Why to use Ethernet HW & SW to transport messages inside an OS instance?
> Usually Ethernet HW & SW (drivers?) deal with sending/receiving to/from the
> wire.
>
I don't say you have to. I am just suggesting that *if* we standardize the
"packet" (message payload + metadata) format, we might as well use a
well-known format that likely can be processed by preexisting HW and SW
blocks. Possibly this will simplify classification, queueing and scheduling
of messages.



> An above requirement was referring to ordering which must be defined by
> the transport layer - how this format ensures ordered delivery at the
> destination?
>
The packet format is just the format of the message payload + metadata as
passed to and

Re: [lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Alexandru Badicioiu]

On 27 May 2015 at 11:08, Ola Liljedahl  wrote:

> Here is my third attempt at a ODP API for MSGIO - asynchronous message
> passing
> based IPC for a shared nothing architecture.
>
> IPC/message passing can be separated into a number of different layers.
> 1) Message formats and protocols (as seen by the user).
> 2) API - defines syntax and semantics of MSGIO operations.
> 3) Transport interface - defines a binary packet format used by MSGIO
>endpoints/implementations. Cf. OSI network layer.
> 4) Transport layer - implements the transport of message packet between
>different endpoints. Cf. OSI link layer.
>
> #1 is application specific and there are many different standards (e.g.
> Linux
> Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
> #3 may be implementation specific but is important for interoperatibility
> (binary compatibility). If the user message (payload), operation and
> parameters are encoded in a well-defined way (e.g. some kind of packet
> format
> which includes operation, sender and receiver addresses, payload etc),
> inter-operatibility between different ODP/MSGIO implementations may be
> achieved or simplified.
> #4 is implementation (platform) specific.
>
> This proposal focuses on the API which is fundamental for source code
> compatibility between different platforms (ODP implementations) - this
> being
> the primary purpose of ODP. A packet format for messages will be specified
> as
> part of the reference implementation and may be seen as a proposal for a
> standard packet format for binary compatibility between different ODP/MSGIO
> implementations.
>
> All MSGIO calls are non-blocking (with the possible exception of
> odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
> transfer is asynchronous so it is not possible to return any status
> operation concerning the eventual (or remote) result of an operation.
>
> The semantics of MSGIO message passing is that sending a message to an
> endpoint
> will always look like it succeeds. The appearance of endpoints is
> explicitly
> notified through user-defined messages specified in the odp_msgio_lookup()
> call. Similarly, the disappearance (e.g. death of or otherwise lost
> connection
> to endpoint) is also explicitly notified through user-defined messages
> specified in the odp_msgio_monitor() call. The send call does not fail
> because
> the addressed endpoint has disappeared, the message is just silently
> discarded.
>
> Message delivery into the recipient address space is ordered (per priority)
> and reliable. Delivery of message N implies delivery of all messages  (of the same priority). All messages (accepted by MSGIO send operation)
> will
> be delivered up to the point of endpoint termination or lost connection
> where no more messages will be delivered. Actual reception (dequeueing) and
> processing by the recipient is not guaranteed (use end-to-end
> acknowledgements
> for that).
>
> MSGIO endpoints can be seen as interfaces (taps) to an internal reliable
> multidrop network where each endpoint has a unique address which is only
> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
> and then recreated (with the same name), the new endpoint will have a
> new unique address (eventually endpoints addresses will have to be recycled
> but not for a very long time). Endpoints names do not necessarily have to
> be
> unique. The scope of message network is not defined but it is expected that
> the scope by default corresponds to an OS instance (e.g. virtual machine).
>
> Proposed transport packet for message I/O (not visible to the user):
> uint8_t dest_addr[6]; /* Unique address of destination endpoint */
> uint8_t src_addr[6]; /* Unique address of source endpoint */
> uint16_t frametype_o; /* Outer frametype, 0x8100 */
> uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */
> uint16_t frametype_i; /* Inner frametype, TBD */
> uint8_t payload[]; /* User message */
> Looks very much like a VLAN-tagged Ethernet frame so should easily be
> transported by and processed by Ethernet HW & SW

Why to use Ethernet HW & SW to transport messages inside an OS instance?
Usually Ethernet HW & SW (drivers?) deal with sending/receiving to/from the
wire.
An above requirement was referring to ordering which must be defined by the
transport layer - how this format ensures ordered delivery at the
destination?

> .
> Encoding of publish/lookup/monitor operations TBD.
>
> v3:
> Renamed Message Bus (MBUS) to Message I/O (MSGIO).
> Changed all definitions to use "msgio" instead of "mbus" prefix/infix.
> odp_msgio_create(): removed default input queue parameter.
> Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef().
> All input queues must now be specified using odp_msgio_inq_set() before
> the MSGIO endpoint is activated.
> Added odp_msgio_activate().
> Added odp_message_push_head() and odp_message_pull_head().
> Updated some function descriptions in msgio.h.
>
> v2:

[lng-odp] [RFCv3] Add message I/O (MSGIO) API's

[Ola Liljedahl]

Here is my third attempt at a ODP API for MSGIO - asynchronous message passing
based IPC for a shared nothing architecture.

IPC/message passing can be separated into a number of different layers.
1) Message formats and protocols (as seen by the user).
2) API - defines syntax and semantics of MSGIO operations.
3) Transport interface - defines a binary packet format used by MSGIO
   endpoints/implementations. Cf. OSI network layer.
4) Transport layer - implements the transport of message packet between
   different endpoints. Cf. OSI link layer.

#1 is application specific and there are many different standards (e.g. Linux
Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases.
#3 may be implementation specific but is important for interoperatibility
(binary compatibility). If the user message (payload), operation and
parameters are encoded in a well-defined way (e.g. some kind of packet format
which includes operation, sender and receiver addresses, payload etc),
inter-operatibility between different ODP/MSGIO implementations may be
achieved or simplified.
#4 is implementation (platform) specific.

This proposal focuses on the API which is fundamental for source code
compatibility between different platforms (ODP implementations) - this being
the primary purpose of ODP. A packet format for messages will be specified as
part of the reference implementation and may be seen as a proposal for a
standard packet format for binary compatibility between different ODP/MSGIO
implementations.

All MSGIO calls are non-blocking (with the possible exception of
odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message
transfer is asynchronous so it is not possible to return any status
operation concerning the eventual (or remote) result of an operation.

The semantics of MSGIO message passing is that sending a message to an endpoint
will always look like it succeeds. The appearance of endpoints is explicitly
notified through user-defined messages specified in the odp_msgio_lookup()
call. Similarly, the disappearance (e.g. death of or otherwise lost connection
to endpoint) is also explicitly notified through user-defined messages
specified in the odp_msgio_monitor() call. The send call does not fail because
the addressed endpoint has disappeared, the message is just silently discarded.

Message delivery into the recipient address space is ordered (per priority)
and reliable. Delivery of message N implies delivery of all messages 
---
(This document/code contribution attached is provided under the terms of
agreement LES-LTM-21309)

 include/odp/api/message.h  | 199 ++
 include/odp/api/msgio.h| 222 +
 platform/linux-generic/include/odp/message.h   |  39 
 platform/linux-generic/include/odp/msgio.h |  40 
 .../linux-generic/include/odp/plat/message_types.h |  47 +
 .../linux-generic/include/odp/plat/msgio_types.h   |  59 ++
 6 files changed, 606 insertions(+)
 create mode 100644 include/odp/api/message.h
 create mode 100644 include/odp/api/msgio.h
 create mode 100644 platform/linux-generic/include/odp/message.h
 create mode 100644 platform/linux-generic/include/odp/msgio.h
 create mode 100644 platform/linux-generic/include/odp/plat/message_types.h
 create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h

diff --git a/include/odp/api/message.h b/include/odp/api/message.h
new file mode 100644
index 000..76697fb
--- /dev/null
+++ b/include/odp/api/message.h
@@ -0,0 +1,199 @@
+/* Copyright (c) 2015, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+
+/**
+ * @file
+ *
+ * ODP Message event type
+ */
+
+#ifndef ODP_API_MESSAGE_H_
+#define ODP_API_MESSAGE_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/** @defgroup odp_message ODP MESSAGE
+ *  @{
+ */
+
+/**
+ * @typedef odp_message_t
+ * ODP message handle
+ */
+
+
+/**
+ * Get address of sender (source) of message
+ *
+ * @param  msg  Message handle
+ * @param[out] addr Buffer to hold sender address
+ */
+void odp_message_sender(odp_message_t msg,
+   uint8_t addr[ODP_MSGIO_ADDR_SIZE]);
+
+/**
+ * Message data pointer
+ *
+ * Return a pointer to the message data
+ *
+ * @param msg Message handle
+ *
+ * @return Pointer to the message data
+ */
+void *odp_message_data(odp_message_t msg);
+
+/**
+ * Message data length
+ *
+ * Return length of the message data.
+ *
+ * @param msg Message handle
+ *
+ * @return Message length
+ */
+uint32_t odp_message_length(const odp_message_t msg);
+
+/**
+ * Set message length
+ *
+ * Set length of the message data.
+ * Increase message data length by moving message tail into message tailroom.
+ * The actual message data is not modified.
+ *
+ * @param msg Message handle
+ * @param len New length
+ *
+ * @retval 0 on success
+ * @retval <0 on error (e.g. no enough tailroom)
+ */
+int odp_message_length_set(co