On 22 October 2015 at 18:06, Nikita Kalyazin <n.kalya...@samsung.com> wrote:
> Hi,
>
> > [Alex] ODP queues are neither software nor hardware by definition, each
> > implementation is free to implement them as they see fit. Also PacketI/O
> > abstraction is not an abstraction for a NIC device. The ODP way to use
> multiple
> > queues for ingress is through the classifier. There was a former
> proposal of
> > enabling RSS at pktio level but it was not accepted. I also noticed that
> this
> > patch tries to introduce multiple default queues which is kind of
> confusing -
> > default traffic is all traffic which does not satisfy classifier
> criteria. For
> > output, multiple queues are used by the means of TM api.
> As I understand, ODP implies three options of ingress processing:
> - "recv" (poll pktio with raw recv() method);
> - "poll" (poll poll pktio via default input queue associated with the
> pktio);
> - "sched" (call schedule() which makes use of classifier and queue
> priorities).
> Looks like each ODP implementation should support all the three options.
>
> For "sched" option, classification may be implemented in hardware, and ODP
> queues, in their turn, can also be implemented in hardware. The hardware
> would
> distribute packets to its hardware queues according to PMRs or L2/L3 QoS.
>
> For "recv" option, the only way possible to access NIC's hardware queues
> (if any), is to call recv() on a specified queue explicitly. So there
> should be
> an API for it. In my patch I proposed such APIs (odp_pktio_recv_queue()).
> I admit, this might be not the best API name possible. However I don't
> see any
> other way to access NIC's hardware queues at "recv" level. We can add
> hw_queue_id parameter to existing odp_pktio_recv() alternatively (same way
> as
> DPDK does with rte_eth_rx_burst()). In case of using such an explicit
> hardware
> queue polling, RSS should be configured somehow to distribute packets
> across them.
>
> For "poll" option, it's not completely clear to me how it should relate to
> NIC's
> hardware queues. It looks like classification is not involved here, and
> this
> option isn't really different from "recv". If this is correct, presence of
> multiple default input queues is reasonable, since they're directly mapped
> to
> NIC's hardware queues, and odp_queue_deq() simply extracts a packet from
> the
> appropriate HW queue.
> [Alex]
>
Hi Nikita,
I understand your reasoning but ODP Packet I/O (pktio) is _not_ (yet?) a
NIC abstraction. The fact is that in current ODP design ingress queues are
"connected" to the pktio device by the configuration of classifier or as
default/error CoS queues. Classifier can be changed dynamically, e.g,
adding or removing CoSes or queues without stopping/re-configuring the
pktio device. I don't see in DPDK such features in PMD, all queues are
configured and setup at initialization time and they don't change
thereafter.
> --
>
> Best regards,
>
> Nikita Kalyazin,
> n.kalya...@samsung.com
>
> Software Engineer
> Virtualization Group
> Samsung R&D Institute Russia
> Tel: +7 (495) 797-25-00 #3816
> Tel: +7 (495) 797-25-03
> Office #1501, 12-1, Dvintsev str.,
> Moscow, 127018, Russia
>
> On Mon, Oct 19, 2015 at 08:17:17PM +0300, Alexandru Badicioiu wrote:
> >
> >
> > On 19 October 2015 at 19:16, Nikita Kalyazin <[1]n.kalya...@samsung.com>
> wrote:
> >
> > Hi Stuart,
> >
> >
> > Thanks for your feedback.
> >
> > > One thing that is missing here is a method for the application to
> > > configure how packets are distributed, i.e. which fields in the
> packet
> > > are used when calculating the flow hash (not necessarily the hash
> > > algorithm itself, that can be implementation defined).
> > >
> > > For example with the netmap implementation here, if you have a
> > > compatible interface, I assume the user can control distribution
> to RX
> > > queues using something like;
> > >
> > > ethtool --config-ntuple rx-flow-hash esp4
> > >
> > > But it would be better if this were configurable via the ODP API.
> Petri
> > > had a suggestion a while back related to this to add a structure
> to the
> > > odp_pktio_params_t;
> > >
> > > enum odp_pktio_input_hash {
> > > /** No specific fields defined */
> > > ODP_PKTIN_HASH_NONE = 0,
> > > /** IPv4/v6 addresses */
> > > ODP_PKTIN_HASH_IP,
> > > /** UDP ports and IPv4/v6 addresses */
> > > ODP_PKTIN_HASH_UDP_IP,
> > > /** TCP ports and IPv4/v6 addresses */
> > > ODP_PKTIN_HASH_TCP_IP
> > > };
> > Thanks, I plan to think about it.
> > Btw, NICs provide wide variety of settings regarding distribution of
> > packets across the queues. Is it supposed to extend the proposed
> > structure to support more flexible configuration?
> >
> > > How about:
> > >
> > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name,
> > > const odp_queue_param_t *param,
> > > odp_queue_t queues[], int num);
> > >
> > > Which would return the number of queues created, and ensure that
> all
> > > queues had the same parameters. Or, maybe it's time to start using
> > > odp_queue_group_t, which we have always intended to be a group of
> queues
> > > that all have the same properties.
> > Good point. I'll consider the cleaner approach.
> >
> > > I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues
> > > referred to there are not ODP queues so it makes things a bit less
> clear,
> > > and in general we should be moving away from using the direct
> _send/_recv
> > > APIs. If we're explicitly using queues then we should use the
> queue and/
> > or
> > > scheduler APIs.
> > Right, these API names look a bit confusing, because same word is
> used to
> > refer to both hardware NIC queues and ODP (software) queues.
> >
> > [Alex] ODP queues are neither software nor hardware by definition, each
> > implementation is free to implement them as they see fit. Also PacketI/O
> > abstraction is not an abstraction for a NIC device. The ODP way to use
> multiple
> > queues for ingress is through the classifier. There was a former
> proposal of
> > enabling RSS at pktio level but it was not accepted. I also noticed that
> this
> > patch tries to introduce multiple default queues which is kind of
> confusing -
> > default traffic is all traffic which does not satisfy classifier
> criteria. For
> > output, multiple queues are used by the means of TM api.
> >
> > However, I think it would be useful to preserve explicit send/recv
> APIs,
> > since in many cases they might appear more efficient (odp-ovs is an
> example
> > of such applications), and rename those to make them more obvious.
> >
> > > Are you able to join the meeting on Tuesday to discuss?
> > >
> > > [2]http://www.opendataplane.org/meetings/
> > Thanks, I'm going to join the meeting.
> >
> > --
> >
> > Best regards,
> >
> > Nikita Kalyazin,
> > [3]n.kalya...@samsung.com
> >
> > Software Engineer
> > Virtualization Group
> > Samsung R&D Institute Russia
> > Tel: [4]+7 (495) 797-25-00 #3816
> > Tel: [5]+7 (495) 797-25-03
> > Office #1501, 12-1, Dvintsev str.,
> > Moscow, 127018, Russia
> >
> > On Fri, Oct 16, 2015 at 07:15:43PM +0100, Stuart Haslam wrote:
> > > On Wed, Oct 14, 2015 at 02:32:37PM +0300, Nikita Kalyazin wrote:
> > > > This series is a proposal of multiqueue API for ODP.
> > > > Modern network adapters (both physical and virtual) support
> multiple
> > > > hardware queues. Assigning separate CPU core for processing of
> each
> > > > queue allows to scale throughput (in best case - linearly).
> > > >
> > >
> > > Thanks for the contribution, we do need something like this, in
> > > particular hash based distribution has been on the todo list for a
> > > little while. I've not reviewed the implementation in detail but
> have
> > > focused on the API changes specifically and have a few initial
> thoughts
> > > below.
> > >
> > > One thing that is missing here is a method for the application to
> > > configure how packets are distributed, i.e. which fields in the
> packet
> > > are used when calculating the flow hash (not necessarily the hash
> > > algorithm itself, that can be implementation defined).
> > >
> > > For example with the netmap implementation here, if you have a
> > > compatible interface, I assume the user can control distribution
> to RX
> > > queues using something like;
> > >
> > > ethtool --config-ntuple rx-flow-hash esp4
> > >
> > > But it would be better if this were configurable via the ODP API.
> Petri
> > > had a suggestion a while back related to this to add a structure
> to the
> > > odp_pktio_params_t;
> > >
> > > enum odp_pktio_input_hash {
> > > /** No specific fields defined */
> > > ODP_PKTIN_HASH_NONE = 0,
> > > /** IPv4/v6 addresses */
> > > ODP_PKTIN_HASH_IP,
> > > /** UDP ports and IPv4/v6 addresses */
> > > ODP_PKTIN_HASH_UDP_IP,
> > > /** TCP ports and IPv4/v6 addresses */
> > > ODP_PKTIN_HASH_TCP_IP
> > > };
> > >
> > > As for RX queue creation, rather than proposed sequence that looks
> like
> > > this:
> > >
> > > odp_pktio_max_num_queues()
> > > odp_pktio_configure()
> > > foreach inq
> > > odp_queue_create()
> > >
> > > How about:
> > >
> > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name,
> > > const odp_queue_param_t *param,
> > > odp_queue_t queues[], int num);
> > >
> > > Which would return the number of queues created, and ensure that
> all
> > > queues had the same parameters. Or, maybe it's time to start using
> > > odp_queue_group_t, which we have always intended to be a group of
> queues
> > > that all have the same properties.
> > >
> > > I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues
> > > referred to there are not ODP queues so it makes things a bit less
> clear,
> > > and in general we should be moving away from using the direct
> _send/_recv
> > > APIs. If we're explicitly using queues then we should use the
> queue and/
> > or
> > > scheduler APIs.
> > >
> > > Are you able to join the meeting on Tuesday to discuss?
> > >
> > > [6]http://www.opendataplane.org/meetings/
> > >
> > > > This series should be considered a prototype.
> > > > This implementation of multiqueue API is not complete (not all
> the
> > > > code has been migrated to multiqueue API). Existing packet IO
> API has
> > > > been left untouched and reimplemented using mutliqueue API
> (queue 0 is
> > > > always used in such cases). The existing API is supposed to be
> > > > removed in the future, as soon as all the remaining code
> migrates to
> > > > the multiqueue API.
> > > > Since single lock for all the queues makes all pktio Rx and Tx
> calls
> > > > sequential, per queue locks have been added.
> > > > For the purpose of evaluation, multiqueue support has been added
> to
> > > > netmap pktio.
> > > >
> > > > Patch 1 adds multiqueue prototypes of pktio API functions.
> > > > Patches 2-4 contain common implementation of the multiqueue API.
> > > > Patch 5 adds multiqueue support in netmap pktio.
> > > > Patch 6 moves odp_generator to multiqueue API.
> > > >
> > > > In our tests (odp_generator sends packets, odp_generator receives
> > > > packets), the following results have been obtained:
> > > > +----------+------------------------+-------+
> > > > | # queues | kpps (64-byte packets) | boost |
> > > > +----------+------------------------+-------+
> > > > | 1 | 853 | 1.00x |
> > > > | 2 | 1404 | 1.64x |
> > > > | 4 | 2374 | 2.78x |
> > > > | 8 | 4345 | 5.09x |
> > > > +----------+------------------------+-------+
> > > >
> > > > Nikita Kalyazin (6):
> > > > api: pktio: add multiqueue API
> > > > linux-generic: pktio: add per queue pktio locks
> > > > linux-generic: pktio: implement multiqueue API (part I)
> > > > linux-generic: pktio: imlement multiqueue API (part II)
> > > > linux-generic: pktio: multiqueue support in netmap
> > > > example: generator: use multiqueue API
> > > >
> > > > example/generator/odp_generator.c | 63 +++-
> > > > include/odp/api/config.h | 7 +-
> > > > include/odp/api/packet_io.h | 118 ++++++
> > > > .../linux-generic/include/odp_packet_io_internal.h | 19 +-
> > > > platform/linux-generic/include/odp_packet_netmap.h | 6 +-
> > > > .../linux-generic/include/odp_queue_internal.h | 2 +
> > > > .../linux-generic/include/odp_schedule_internal.h | 1 +
> > > > platform/linux-generic/odp_packet_io.c | 405
> > +++++++++++++++++----
> > > > platform/linux-generic/odp_schedule.c | 26 +-
> > > > platform/linux-generic/pktio/netmap.c | 218
> +++++++++--
> > > > 10 files changed, 725 insertions(+), 140 deletions(-)
> > > >
> > > > --
> > > > 2.5.3
> > > >
> > _______________________________________________
> > lng-odp mailing list
> > [7]lng-odp@lists.linaro.org
> > [8]https://lists.linaro.org/mailman/listinfo/lng-odp
> >
> >
> >
> > References:
> >
> > [1] mailto:n.kalya...@samsung.com
> > [2] http://www.opendataplane.org/meetings/
> > [3] mailto:n.kalya...@samsung.com
> > [4] tel:%2B7%20%28495%29%20797-25-00%20%233816
> > [5] tel:%2B7%20%28495%29%20797-25-03
> > [6] http://www.opendataplane.org/meetings/
> > [7] mailto:lng-odp@lists.linaro.org
> > [8] https://lists.linaro.org/mailman/listinfo/lng-odp
>
_______________________________________________
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp
