[
https://issues.apache.org/jira/browse/S4-95?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497292#comment-13497292
]
Matthieu Morel edited comment on S4-95 at 11/14/12 6:10 PM:
------------------------------------------------------------
I just updated more improvements in commit f9689ea
In addition to improvements from previous commits, we further and better
decoupled the processing pipeline into asynchronous stages with configurable
executors: reception, deserialization, processing, serialization and event
emission.
We can reach 250+ thousands _remote_ events/s per node with a sufficient number
of injectors using not the most recent hardware and OS, and that translates
into 1M+ _remote_ events per seconds with only 4 S4 nodes.
Since remote events require going through the communication layer and the
network, we expect even higher rates when combining with internal events as in
a standard application. (Internal events are passed by reference).
See the README.md file in s4-benchmarks subprojects for instructions to
reproduce the tests.
was (Author: mmorel):
I just updated more improvements in commit f9689ea
In addition to improvements from previous commits, we further and better
decoupled the processing pipeline into asynchronous stages with configurable
executors: reception, deserialization, processing, serialization and event
emission.
We can reach 250+ thousands _remote_ events/s per node with a sufficient number
of injectors using not the most recent hardware and OS, and that translates
into 1M+ _remote_ events per seconds with only 4 S4 nodes.
Since remote events require going through the communication layer and the
network, we expect even higher rates when combining with internal events as in
a standard application. (Internal events are passed by reference).
> Add reproducible performance benchmarks
> ---------------------------------------
>
> Key: S4-95
> URL: https://issues.apache.org/jira/browse/S4-95
> Project: Apache S4
> Issue Type: Test
> Affects Versions: 0.6
> Reporter: Matthieu Morel
> Assignee: Matthieu Morel
>
> In order to track performance improvements, we need some reproducible
> performance benchmarks. Here are some ideas of what we'd need:
> - use PEs that do nothing but create a new message and forward. Allows us to
> focus on the overhead of the platform
> - what is the maximum throughput without dropping messages, in a given host
> (in a setup with 1 adapter node and 1 or 2 app nodes)
> - what is the latency for end to end processing (avg, median, etc...)
> - using a very simple app, with only 1 PE prototype
> - varying the number of keys
> - using a slightly more complex app (at least 2 communicating prototypes), in
> order to take into account inter-PE communications and related optimizations
> - start measurements after a warmup phase
> Some tests could be part of the test suite (by specifying a given option for
> those performance-related tests). That would allow some tracking of the
> performance.
> We could also add a simple injection mechanism that would work out of the box
> with the example bundled with new S4 apps (through "s4 newApp" command).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
