On 20 May 2015 at 13:21, Ola Liljedahl <ola.liljed...@linaro.org> wrote:
> On 20 May 2015 at 10:25, Maxim Uvarov <maxim.uva...@linaro.org> wrote:
> > Hello Ola,
> >
> > please find some replays bellow.
> >
> > On 19 May 2015 at 01:03, Ola Liljedahl <ola.liljed...@linaro.org> wrote:
> >>
> >> As promised, here is my first attempt at a standalone API for IPC -
> inter
> >> process communication in a shared nothing architecture (message passing
> >> between processes which do not share memory).
> >>
> >> Currently all definitions are in the file ipc.h but it is possible to
> >> break out some message/event related definitions (everything from
> >> odp_ipc_sender) in a separate file message.h. This would mimic the
> >> packet_io.h/packet.h separation.
> >>
> >> The semantics of message passing is that sending a message to an
> endpoint
> >> will always look like it succeeds. The appearance of endpoints is
> >> explicitly
> >> notified through user-defined messages specified in the
> odp_ipc_resolve()
> >> call. Similarly, the disappearance (e.g. death or otherwise lost
> >> connection)
> >> is also explicitly notified through user-defined messages specified in
> the
> >> odp_ipc_monitor() call. The send call does not fail because the
> addressed
> >> endpoints has disappeared.
> >>
> >> Messages (from endpoint A to endpoint B) are delivered in order. If
> >> message
> >> N sent to an endpoint is delivered, then all messages <N have also been
> >> delivered. Message delivery does not guarantee actual processing by the
> >> recipient. End-to-end acknowledgements (using messages) should be used
> if
> >> this guarantee is important to the user.
> >>
> >> IPC endpoints can be seen as interfaces (taps) to an internal reliable
> >> multidrop network where each endpoint has a unique address which is only
> >> valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
> >> and then recreated (with the same name), the new endpoint will have a
> >> new address (eventually endpoints addresses will have to be recycled but
> >> not for a very long time). Endpoints names do not necessarily have to be
> >> unique.
> >>
> >> Signed-off-by: Ola Liljedahl <ola.liljed...@linaro.org>
> >> ---
> >> (This document/code contribution attached is provided under the terms of
> >> agreement LES-LTM-21309)
> >>
> >> include/odp/api/ipc.h | 261
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 261 insertions(+)
> >> create mode 100644 include/odp/api/ipc.h
> >>
> >> diff --git a/include/odp/api/ipc.h b/include/odp/api/ipc.h
> >> new file mode 100644
> >> index 0000000..3395a34
> >> --- /dev/null
> >> +++ b/include/odp/api/ipc.h
> >> @@ -0,0 +1,261 @@
> >> +/* Copyright (c) 2015, Linaro Limited
> >> + * All rights reserved.
> >> + *
> >> + * SPDX-License-Identifier: BSD-3-Clause
> >> + */
> >> +
> >> +
> >> +/**
> >> + * @file
> >> + *
> >> + * ODP IPC API
> >> + */
> >> +
> >> +#ifndef ODP_API_IPC_H_
> >> +#define ODP_API_IPC_H_
> >> +
> >> +#ifdef __cplusplus
> >> +extern "C" {
> >> +#endif
> >> +
> >> +/** @defgroup odp_ipc ODP IPC
> >> + * @{
> >> + */
> >> +
> >> +/**
> >> + * @typedef odp_ipc_t
> >> + * ODP IPC handle
> >> + */
> >> +
> >> +/**
> >> + * @typedef odp_ipc_msg_t
> >> + * ODP IPC message handle
> >> + */
> >> +
> >> +
> >> +/**
> >> + * @def ODP_IPC_ADDR_SIZE
> >> + * Size of the address of an IPC endpoint
> >> + */
> >> +
> >> +/**
> >> + * Create IPC endpoint
> >> + *
> >> + * @param name Name of local IPC endpoint
> >> + * @param pool Pool for incoming messages
> >> + *
> >> + * @return IPC handle on success
> >> + * @retval ODP_IPC_INVALID on failure and errno set
> >> + */
> >> +odp_ipc_t odp_ipc_create(const char *name, odp_pool_t pool);
> >> +
> >
> >
> > Will it be separate pool or you want to attach existing pool to your ipc?
> It should be a pool with events of type messages (ODP_EVENT_MESSAGE)
> so in practice it will likely be a separate pool.
>
> >
> >
> >>
> >> +/**
> >> + * Destroy IPC endpoint
> >> + *
> >> + * @param ipc IPC handle
> >> + *
> >> + * @retval 0 on success
> >> + * @retval <0 on failure
> >> + */
> >> +int odp_ipc_destroy(odp_ipc_t ipc);
> >> +
> >> +/**
> >> + * Set the default input queue for an IPC endpoint
> >> + *
> >> + * @param ipc IPC handle
> >> + * @param queue Queue handle
> >> + *
> >> + * @retval 0 on success
> >> + * @retval <0 on failure
> >> + */
> >> +int odp_ipc_inq_setdef(odp_ipc_t ipc, odp_queue_t queue);
> >> +
> >> +/**
> >> + * Remove the default input queue
> >> + *
> >> + * Remove (disassociate) the default input queue from an IPC endpoint.
> >> + * The queue itself is not touched.
> >> + *
> >> + * @param ipc IPC handle
> >> + *
> >> + * @retval 0 on success
> >> + * @retval <0 on failure
> >> + */
> >> +int odp_ipc_inq_remdef(odp_ipc_t ipc);
> >> +
> >> +/**
> >> + * Resolve endpoint by name
> >> + *
> >> + * Look up an existing or future endpoint by name.
> >> + * When the endpoint exists, return the specified message with the
> >> endpoint
> >> + * as the sender.
> >> + *
> >> + * @param ipc IPC handle
> >> + * @param name Name to resolve
> >> + * @param msg Message to return
> >> + */
> >> +void odp_ipc_resolve(odp_ipc_t ipc,
> >> + const char *name,
> >> + odp_ipc_msg_t msg);
> >> +
> >> +/**
> >> + * Monitor endpoint
> >> + *
> >> + * Monitor an existing (potentially already dead) endpoint.
> >> + * When the endpoint is dead, return the specified message with the
> >> endpoint
> >> + * as the sender.
> >> + *
> >> + * Unrecognized or invalid endpoint addresses are treated as dead
> >> endpoints.
> >> + *
> >> + * @param ipc IPC handle
> >> + * @param addr Address of monitored endpoint
> >> + * @param msg Message to return
> >> + */
> >> +void odp_ipc_monitor(odp_ipc_t ipc,
> >> + const uint8_t addr[ODP_IPC_ADDR_SIZE],
> >> + odp_ipc_msg_t msg);
> >> +
> >> +/**
> >> + * Send message
> >> + *
> >> + * Send a message to an endpoint (which may already be dead).
> >> + * Message delivery is ordered and reliable. All (accepted) messages
> will
> >> be
> >> + * delivered up to the point of endpoint death or lost connection.
> >> + * Actual reception and processing is not guaranteed (use end-to-end
> >> + * acknowledgements for that).
> >> + * Monitor the remote endpoint to detect death or lost connection.
> >> + *
> >> + * @param ipc IPC handle
> >> + * @param msg Message to send
> >> + * @param addr Address of remote endpoint
> >> + *
> >> + * @retval 0 on success
> >> + * @retval <0 on error
> >> + */
> >> +int odp_ipc_send(odp_ipc_t ipc,
> >> + odp_ipc_msg_t msg,
> >> + const uint8_t addr[ODP_IPC_ADDR_SIZE]);
> >> +
> >> +/**
> >> + * Get address of sender (source) of message
> >> + *
> >> + * @param msg Message handle
> >> + * @param addr Address of sender endpoint
> >> + */
> >> +void odp_ipc_sender(odp_ipc_msg_t msg,
> >> + uint8_t addr[ODP_IPC_ADDR_SIZE]);
> >> +
> >> +/**
> >> + * Message data pointer
> >> + *
> >> + * Return a pointer to the message data
> >> + *
> >> + * @param msg Message handle
> >> + *
> >> + * @return Pointer to the message data
> >> + */
> >> +void *odp_ipc_data(odp_ipc_msg_t msg);
> >> +
> >> +/**
> >> + * Message data length
> >> + *
> >> + * Return length of the message data.
> >> + *
> >> + * @param msg Message handle
> >> + *
> >> + * @return Message length
> >> + */
> >> +uint32_t odp_ipc_length(const odp_ipc_msg_t msg);
> >> +
> >> +/**
> >> + * Set message length
> >> + *
> >> + * Set length of the message data.
> >> + *
> >> + * @param msg Message handle
> >> + * @param len New length
> >> + *
> >> + * @retval 0 on success
> >> + * @retval <0 on error
> >> + */
> >> +int odp_ipc_reset(const odp_ipc_msg_t msg, uint32_t len);
> >
> >
> > reset usually stand to reset to some default value. In that case it's
> better
> > to be odp_ipc_len_set(msg, len).
> Well I copied this from packet.h where odp_packet_reset(pkt, len)
> seemed to be equivalent to what I want to achieve but I agree that
> odp_ipc_len_set() is more direct. I can change.
>
> > And reset is just void inline function to reset to default value
> > odp_ipc_reset(msg).
> > And len might be is not needed here because you transfer events. So just
> get
> The IPC mechanism transfers messages. And different messages have
> different lengths.
> > size from event.
> The message size specified when creating a message pool that is
> associated with an IPC endpoint is basically an MTU (the largest
> message size that can be sent or received).
>
> >
> > Might be something like:
> >
> > msg.event = ev; // len will be get from event
> > msg.remote = addr;
> I don't understand what you are trying to describe here.
>
> >
> >
> >
> >>
> >> +
> >> +/**
> >> + * Allocate message
> >> + *
> >> + * Allocate a message of a specific size.
> >> + *
> >> + * @param pool Message pool to allocate message from
> >> + * @param len Length of the allocated message
> >> + *
> >> + * @return IPC message handle on success
> >> + * @retval ODP_IPC_MSG_INVALID on failure and errno set
> >> + */
> >> +odp_ipc_msg_t odp_ipc_alloc(odp_pool_t pool, uint32_t len);
> >> +
> >
> >
> > is len sizeof(odp_ipc_msg_t) here?
> No. odp_ipc_msg_t is a handle to a message, just like odp_packet_t is
> a handle to a packet.
> The implementation of odp_ipc_msg_t is platform specific and thus
> sizeof(odp_ipc_msg_t) as well (but 32 or 64 bits is a good guess).
>
> The len parameter in odp_ipc_alloc() specifies the current/actual
> length of the message which depends on what the application wants to
> use that specific message for.
>
> >
> >
> >>
> >> +/**
> >> + * Free message
> >> + *
> >> + * Free message back to the message pool it was allocated from.
> >> + *
> >> + * @param msg Handle of message to free
> >> + */
> >> +void odp_ipc_free(odp_ipc_msg_t msg);
> >> +
> >> +/**
> >> + * Get message handle from event
> >> + *
> >> + * Converts an ODP_EVENT_MESSAGE type event to a message.
> >> + *
> >> + * @param ev Event handle
> >> + *
> >> + * @return Message handle
> >> + *
> >> + * @see odp_event_type()
> >> + */
> >> +odp_ipc_msg_t odp_message_from_event(odp_event_t ev);
> >> +
> >> +/**
> >> + * Convert message handle to event
> >> + *
> >> + * @param msg Message handle
> >> + *
> >> + * @return Event handle
> >> + */
> >> +odp_event_t odp_message_to_event(odp_ipc_msg_t msg);
> >> +
> >
> >
> > Most of functions look like pktio. But that version of IPC looks like
> fully
> > software implementation.
> I think some SoC's could use HW mechanism to transfer messages (which
> are basically buffers with individual lengths) between endpoints (e.g.
> different address spaces).
>
> > Idea is good - send and receive odp events.
> Send and receive ODP *messages*, not any type of ODP events.
>
> > It looks like odp_ipc_t ipc
> > object also connected to
> > some odp_pktio_t where actual packets will be sent. And it will be
> > interesting to make queues interface
> > work with that ipc.
> >
> > Might be:
> > odp_pktio_t pktio can have ipc atribute like
> >
> > odp_ipc_pktio_set(odp_ipc_t ipc, odp_pktio_t pktio);
> Now you are down to implementation details. Why should this be part of
> the public API?
>
>
> >
> > then queue created for that pktio can send/recv packets from outside.
> I think you can already do this with the current packet_io API. No
> need to conflate this with IPC/message passing.
>
> >
> > If I understand right, that functions use queue only for ipc messages:
> > int odp_ipc_inq_setdef(odp_ipc_t ipc, odp_queue_t queue);
> > int odp_ipc_inq_remdef(odp_ipc_t ipc);
> You want to be able to receive and dispatch IPC messages just like any
> other type of events so incoming messages should be put on queues. The
> queue could be specified when the IPC endpoint is created but I copied
> the packet_io API where it was possible.
>
> > and they are not linked with outside pktio. Earlier you said that
> propose
> > to send some packets to some internal tap device.
> No not packets, messages. And the reference to "tap" was in a general
> sense, not related to Linux tap interfaces.
>
> > And in that case tap
> > device can be pktio. And then this pktio-tap connected to ipc object to
> > dispatch events from it.
>
> >
> > And probably we need better names for our ipc implementation. Mine can be
> > odp_pktio_ipc and your can be odp_ipcmsg_ because the propose is
> different.
> Actually I don't understand why your packet pipes need to change the
> ODP API. They are just pktio interfaces with different semantics. By
> using different pktio device names (which anyway are platform
> specific), you create packet pipes instead of opening physical network
> interfaces. But they behave the same from the user perspective. Or?
>
>
Yes in my case is only pipes and it's just pktio with different name. No
difference to application how to use this pktio.
> >
> >
> >
> >
> >>
> >> +/**
> >> + * Get printable value for an odp_ipc_t
> >> + *
> >> + * @param ipc IPC handle to be printed
> >> + * @return uint64_t value that can be used to print/display this
> >> + * handle
> >> + *
> >> + * @note This routine is intended to be used for diagnostic purposes
> >> + * to enable applications to generate a printable value that represents
> >> + * an odp_ipc_t handle.
> >> + */
> >> +uint64_t odp_ipc_to_u64(odp_ipc_t ipc);
> >> +
> >> +/**
> >> + * Get printable value for an odp_ipc_msg_t
> >> + *
> >> + * @param msg Message handle to be printed
> >> + * @return uint64_t value that can be used to print/display this
> >> + * handle
> >> + *
> >> + * @note This routine is intended to be used for diagnostic purposes
> >> + * to enable applications to generate a printable value that represents
> >> + * an odp_ipc_msg_t handle.
> >> + */
> >> +uint64_t odp_ipc_msg_to_u64(odp_ipc_msg_t msg);
> >> +
> >> +/**
> >> + * @}
> >> + */
> >> +
> >> +#ifdef __cplusplus
> >> +}
> >> +#endif
> >> +
> >> +#endif
> >> --
> >> 1.9.1
> >
> >
> >
> > After writing replay I came with idea why not build one pktio based on
> other
> > pktio?
> > In that case eth0_pktio = ocp_create_pktio("eth0") can be used for
> external
> > packets exchange and
> > ipc_eth0_pktio = odp_create_pktio(eth0_pktio, ..).
> > So read from eth0_pktio will get only outside packets and read from
> > ipc_eth0_pktio can deliver only
> > ipc packets.
> There are no IPC packets, just IPC messages.
>
> > In that case you do not need to implement transport layer and
> > just do functionality above
> > on current pktio.
> IPC/message passing has different semantics (e.g. reliable/in-order)
> from network interfaces.
> IPC/message passing is for communication between local entities (e.g.
> applications on the same host), not for communicating with external
> entities over some physical network.
> So I see several serious incompatibilities and I see no reason to
> build IPC/message passing on top of some physical network interface
> and its representation in ODP.
>
>
Ok, I understand that it's exchange with some messages on one machine. I
just not sure why conrtol plane and data plane be different host. Like ODL
where OVS on one machine and controller on other.
As I understand you don't what not transfer any odp events. Just ipc
messages. Btw what is inside that messages? How that message looks like?
Maxim.
> >
> > Thanks,
> > Maxim.
> >
> >
> >
> >
> >
> >>
> >>
> >> _______________________________________________
> >> lng-odp mailing list
> >> lng-odp@lists.linaro.org
> >> https://lists.linaro.org/mailman/listinfo/lng-odp
> >
> >
>
_______________________________________________
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp
