>
> Do you mean this line?
> vpp_transport_shmem.VppTransport.read = read_map_domains_get_mock
>
> I do not like it that much.
> I mean, it works for make test, as it is in the same git repository as
> PAPI code,
> so make test can be accommodated for any big PAPI edit in the same Gerrit
> Change.
> We want to avoid such tight coupling with any code outside VPP repo
> (which includes code inside CSIT repo).
That is actually not tightly coupled. I just selected that as a sample of
a stream type message so folks could see I tested samples of all the
variants:
"map_domains_get": {
"reply": "map_domains_get_reply",
"stream": true,
"stream_msg": "map_domain_details"
}
We could rename them as NOT_map_domains_get. It was selected because the
map api uses the new stream/cursor logic.
Yes, the monkey patching is tightly coupled to the transport code. It could
be replaced with a MockTransport but that code isn't present here.
On Wed, Apr 7, 2021 at 11:38 AM Vratko Polak -X (vrpolak - PANTHEON TECH
SRO at Cisco) <vrpo...@cisco.com> wrote:
> Finally, this got back high enough in my TODO list.
>
>
>
> > Have you looked at [0]?
>
> > [0] https://gerrit.fd.io/r/c/vpp/+/30350
>
>
>
> Do you mean this line?
>
> vpp_transport_shmem.VppTransport.read = read_map_domains_get_mock
>
>
>
> I do not like it that much.
>
> I mean, it works for make test, as it is in the same git repository as
> PAPI code,
>
> so make test can be accommodated for any big PAPI edit in the same Gerrit
> Change.
>
> We want to avoid such tight coupling with any code outside VPP repo
>
> (which includes code inside CSIT repo).
>
>
> As you said:
>
> Poking in the values is an ingenious way to go about solving the problem,
> but it is going to make the systems too tightly coupled and create a
> maintenance headache.
>
>
>
> > Have you seen [0]?
>
> > [0] https://gerrit.fd.io/r/c/vpp/+/30361
>
>
>
> That looks better.
>
> You can have MakeTestMockTransport defined somewhere in tests/,
>
> and use that when appropriate.
>
> It may take a while to make vpp_papi APIs clean,
>
> as different transports have different requirements and capabilities,
>
> but definitely a step in the right direction.
>
>
>
> For a socket transport variant suitable for high performance,
>
> I imagine something like [3].
>
> Marked as WiP, as it is poorly documented and not tested at all yet.
>
> Based on master, as 30361 is not merged yet (and needs a rebase).
>
>
>
> With 30361, the new transport code from 31920 could be in CSIT,
>
> but I imagine other users may want to use it too
>
> (not probable for MakeTestMockTransport).
>
>
>
> What do you think?
>
>
>
> > Can I suggest that if you want to use the binary api (vapi) instead of
> papi, that you look at swig. [1]
>
> > [1] http://www.swig.org/
>
>
>
> Not sure vapi+swig gives big enough performance benefit.
>
> In CSIT we want to keep all Python stuff in one machine (with Robot tests),
>
> and using socket (forwarded via SSH) to talk to a plain VPP machine.
>
>
>
> If we consider putting custom code onto the VPP machine,
>
> the range of possibilities increase considerably
>
> (extreme example: CSIT could start building its own VPP plugin
>
> to do direct C calls, similarly to what CLI and VAT code in VPP does).
>
>
>
> Vratko.
>
>
>
> [3] https://gerrit.fd.io/r/c/vpp/+/31920
>
>
>
> *From:* Paul Vinciguerra <pvi...@vinciconsulting.com>
> *Sent:* Thursday, 2021-February-04 05:09
> *To:* Paul Vinciguerra <pvi...@vinciconsulting.com>
> *Cc:* Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco) <
> vrpo...@cisco.com>; Ole Troan (otroan) <otr...@cisco.com>; Peter Mikus -X
> (pmikus - PANTHEON TECH SRO at Cisco) <pmi...@cisco.com>;
> csit-...@lists.fd.io; vpp-dev@lists.fd.io
> *Subject:* Re: [vpp-dev] Faster PAPI
>
>
>
> Following up on Peter's comments in the changeset,
>
>
>
> Have you seen [0]? It separates out the transports, so that you can
> implement/subclass your own without poking in values from your executor.
> Poking in the values is an ingenious way to go about solving the problem,
> but it is going to make the systems too tightly coupled and create a
> maintenance headache. Even if we write unit tests against papi, we cannot
> anticipate how you might redefine the logic. It looks relatively
> straightforward to move your methods into a new or subclassed transport.
>
>
>
> Can I suggest that if you want to use the binary api (vapi) instead of
> papi, that you look at swig. [1]
>
>
>
> [0] https://gerrit.fd.io/r/c/vpp/+/30361
>
> [1] http://www.swig.org/
>
>
>
> On Wed, Feb 3, 2021 at 11:08 AM Paul Vinciguerra via lists.fd.io <pvinci=
> vinciconsulting....@lists.fd.io> wrote:
>
> Hi Vratko.
>
>
>
> Have you looked at [0]?
>
>
>
> [0] https://gerrit.fd.io/r/c/vpp/+/30350
>
>
>
> On Wed, Feb 3, 2021 at 9:44 AM Vratko Polak -X (vrpolak - PANTHEON TECH
> SRO at Cisco) <vrpo...@cisco.com> wrote:
>
> > Hello people interested in PAPI (VPP's Python API client library).
>
>
>
> Hello again.
>
> This is an update e-mail, adding some information,
>
> while still asking basically the same questions.
>
>
>
> Since my first e-mail, there was some private communication,
>
> mostly related to reasons the vanilla performance is not good,
>
> and how improvements to VAT [2] can help.
>
>
>
> > The exact code change is [0], but that may be hard to review.
>
>
>
> The current patch set [3] is a little better.
>
>
>
> > For that I have created [1], which shows the changed VPP PAPI code.
>
>
>
> Still mostly unfinished, I need to familiarize better with shmem transport.
>
>
>
> The main inputs came from Peter, who expressed
>
> dislike [4] on how brittle the fast binary message generation is,
>
> and he prefers "we will call a vector operation and PAPI just executes it".
>
>
>
> Let me summarize the current options as I see them.
>
>
>
> 1. Keep the status quo.
>
> That means using VAT for some operations (e.g. adding multiple routes [5]),
>
> and creating "exec scripts" [6] for operations without VAT one-liner.
>
> Pros: No work needed, good speed, old VPP versions are supported.
>
> Cons: Relying on VAT functionality (outside API compatibility rules).
>
>
>
> 2. Support "vector operations" in VPP via binary API.
>
> This will probably need a new VPP plugin to host the implementations.
>
> Pros: Fast speed, small CSIT work, guarded by API compatibility rules.
>
> Cons: New VPP plugin of questionable usefulness outside CSIT,
>
> plugin maintainer needed, old VPP versions not supported.
>
>
>
> 3. VPP PAPI improvements only.
>
> No changes to VPP API, just changes to PAPI to allow better speed for
> socket interactions.
>
> CSIT would need a fast way to synthetize binary messages.
>
> Pros: Small VPP work, good speed, only "official" VPP API is used.
>
> Cons: Brittle CSIT message handling, old VPP versions not supported.
>
>
>
> 4. CSIT hacks only (Gerrit 26019).
>
> No changes to VPP API nor PAPI. CSIT code messes with PAPI internals.
>
> CSIT needs a fast way to synthetize binary messages.
>
> Pros: Code is ready, good speed, old VPP versions are supported.
>
> Cons: Brittle CSIT message handling, risky with respect to VPP PAPI
> changes.
>
>
>
> The open questions:
>
> Do you see any other options?
>
> Did I miss some important pros or cons?
>
> Which option do you prefer?
>
>
>
> Vratko.
>
>
>
> [2] https://lists.fd.io/g/vpp-dev/topic/78362835#18092
>
> [3] https://gerrit.fd.io/r/c/csit/+/26019/140
>
> [4]
> https://gerrit.fd.io/r/c/csit/+/26019/140#message-314d168d8951b539e588e644a875624f5ca3fb77
>
> [5]
> https://github.com/FDio/csit/blob/b5073afc4a941ea33ce874e016fe86384ae7a60d/resources/templates/vat/vpp_route_add.vat
>
> [6]
> https://github.com/FDio/csit/blob/b5073afc4a941ea33ce874e016fe86384ae7a60d/resources/libraries/python/TestConfig.py#L121-L150
>
>
>
> *From:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> *On Behalf Of *Vratko
> Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) via lists.fd.io
> *Sent:* Thursday, 2020-May-07 18:35
> *To:* vpp-dev@lists.fd.io
> *Cc:* csit-...@lists.fd.io
> *Subject:* [vpp-dev] Faster PAPI
>
>
>
> Hello people interested in PAPI (VPP's Python API client library).
>
>
>
> In CSIT, our tests are using PAPI to interact with VPP.
>
> We are using socket transport (instead of shared memory transport),
>
> as VPP is running on machines separate from machines running the tests.
>
> We use SSH to forward the socket between the machines.
>
>
>
> Some of our scale tests need to send high number of commands towards VPP.
>
> The largest test sends 4 million commands (ip_route_add_del with ip6
> addresses).
>
> You can imagine that can take a while.
>
> Even using PAPI in asynchronous mode, it takes tens of minutes per million
> commands.
>
>
>
> I was able to speed that up considerably, just by changing code on CSIT
> side.
>
> The exact code change is [0], but that may be hard to review.
>
> Gerrit does not even recognize the new PapiSocketExecutor.py
>
> to be an edited copy of the old PapiExecutor.py file.
>
>
>
> That code relies on the fact that Python is quite permissive language,
>
> not really distinguishing private fields and methods from public ones.
>
> So the current code is vulnerable to refactors of VPP PAPI code.
>
> Also, pylint (static code analysis tool CSIT uses) is complaining.
>
>
>
> The proper fix is to change the VPP PAPI code,
>
> so that it exposes the inner parts the new CSIT code needs to access
>
> (or some abstractions of them).
>
>
>
> For that I have created [1], which shows the changed VPP PAPI code.
>
> Commit message contains a simplified example of how the new features can
> be used.
>
>
>
> The changed VPP code allows three performance improvements.
>
>
>
> 1. Capturing raw bytes sent.
>
> For complicated commands, many CPU cycles are spent serializing
>
> command arguments (basically nested python dicts) into bytes.
>
> If user (CSIT code) has access to the message as serialized by PAPI (VPP
> code),
>
> the user can choose a faster method to create subsequent serialized data.
>
> Implementing this on CSIT side improved the speed, but not greatly enough.
>
> (See bytes_template.py in [0] for the faster data generator.)
>
> The VPP code [1] introduces fields remember_sent and last_sent.
>
>
>
> 2. Reading replies without de-serializing them.
>
> This was already possible by calling transport.q.get(),
>
> but had next to no effect on PAPI speed.
>
> Replies are usually short, so deserialization does not take too many
> cycles.
>
>
>
> 3. Disabling the background reader thread.
>
> By default, socket transport creates (upon connect) a background thread,
>
> which select()s on the socket, reads any messages,
>
> and put()s them to transport.q (multiprocessing.Queue).
>
> I am not sure whether it is the multithreading (waiting for Python
> interpreter
>
> to switch between threads), or Queue (locks, its own reader thread),
>
> but overall this was the remaining bottleneck.
>
> The VPP code exposes public methods for stopping and starting the thread.
>
>
>
> Back to point 2:
>
> With the reading thread stopped, transport.q is not filled,
>
> so another way to read the reply is needed.
>
> The VPP code contained transport._read(),
>
> the underscore hinting this is an internal method
>
> (leading to the abovementioned pylint complaints).
>
> The VPP change [1] renames that method to read_message(),
>
> adding a docstring explaining it has to be used
>
> when the reading thread is stopped.
>
>
>
> Finally, with all 3 improvements, CSIT will be able
>
> to execute million PAPI commands in around 15 seconds.
>
>
>
> Even if something like [1] is merged to VPP,
>
> CSIT will still use [0] for some time,
>
> so we are able to test older VPP versions.
>
>
>
> So, any comments on [1], or other ideas
>
> on what changes are needed on VPP side
>
> so users can achieve good PAPI speed using public PAPI methods?
>
>
>
> Vratko.
>
>
>
> [0] https://gerrit.fd.io/r/c/csit/+/26019/108
>
> [1] https://gerrit.fd.io/r/c/vpp/+/26946/1
>
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19127): https://lists.fd.io/g/vpp-dev/message/19127
Mute This Topic: https://lists.fd.io/mt/80354555/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
