[
https://issues.apache.org/jira/browse/ARTEMIS-3289?focusedWorklogId=597360&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597360
]
ASF GitHub Bot logged work on ARTEMIS-3289:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 16/May/21 09:27
Start Date: 16/May/21 09:27
Worklog Time Spent: 10m
Work Description: franz1981 edited a comment on pull request #3572:
URL: https://github.com/apache/activemq-artemis/pull/3572#issuecomment-841788187
@clebertsuconic @michaelandrepearce @gtully
One important note related the default thread pool(s) configuration of the
broker.
Just to have some fun, I've played a bit with the thread configuration in
order to reduce the number of context switched of master and this is the
improvement I got:
```
**************
RUN 1 EndToEnd Throughput: 51031 ops/sec
**************
EndToEnd SERVICE-TIME Latencies distribution in MICROSECONDS
mean 346.02
min 92.67
50.00% 321.54
90.00% 444.42
99.00% 626.69
99.90% 4014.08
99.99% 14942.21
max 22020.10
count 1600000
```
Same test as
https://github.com/apache/activemq-artemis/pull/3572#issuecomment-841677475,
where I was getting `37770 ops/sec`
Just as a side note `main` behaviour after fine tuning the box is getting
```
**************
RUN 1 EndToEnd Throughput: 46643 ops/sec
**************
EndToEnd SERVICE-TIME Latencies distribution in MICROSECONDS
mean 358.92
min 98.30
50.00% 344.06
90.00% 458.75
99.00% 618.50
99.90% 3276.80
99.99% 11599.87
max 27394.05
count 1600000
```
vs `33265 ops/sec`
The improvement delivered by this PR is still there `51031 ops/sec vs 46643
ops/sec`
I've just reduced the acceptors/netty thread pool size to be 1/2 of the
available cores (instead of 3X), while leaving the rest to the
`thread-pool-max-size` (instead of having it to be `30`).
eg on a 16 cores machines -> 8 threads for Netty + 8 threads for Artemis
generic thread pool.
Given that core protocol perform very few operations on the Netty threads
I've decided to split the CPU capacity in 2.
The boost is A LOT, hence I think we should re-think the default parameters
we set for the broker wdyt? any concern?
I'm going to file a new issue if you agree: both public benchmarks and cloud
instances will get a huge boost by having proper default values, because users
won't know upfront the machines they're going to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 597360)
Time Spent: 4h (was: 3h 50m)
> Reduce journal appender executor Thread wakeup cost
> ---------------------------------------------------
>
> Key: ARTEMIS-3289
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3289
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Reporter: Francesco Nigro
> Assignee: Francesco Nigro
> Priority: Major
> Attachments: 3289_backup.html, image-2021-05-11-09-32-15-538.png,
> main_backup.html
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> As mentioned in https://issues.apache.org/jira/browse/ARTEMIS-2877 one of the
> major factors that contribute to reduce the scalability of a shared-nothing
> replication setup is the thread wake-up cost of the {{JournalImpl}}'s
> {{appendExecutor}} I/O threads.
> See the flamegraph below for a busy replica while appending replicated
> journal record:
> !image-2021-05-11-09-32-15-538.png|width=966,height=313!
> The violet bars represent the CPU cycles spent to awake the Journal appender
> thread(s): despite https://issues.apache.org/jira/browse/ARTEMIS-2877 allow
> backup to batch append tasks as much as possible, it seems the signaling cost
> is still too high, if compared with the rest of replica packet processing.
> Given that the append executor is an ordered executor built on top of I/O
> thread pool, see {{ActiveMQServerImpl}}:
> {code:java}
> if (serviceRegistry.getIOExecutorService() != null) {
> this.ioExecutorFactory = new
> OrderedExecutorFactory(serviceRegistry.getIOExecutorService());
> } else {
> ThreadFactory tFactory = AccessController.doPrivileged(new
> PrivilegedAction<ThreadFactory>() {
> @Override
> public ThreadFactory run() {
> return new ActiveMQThreadFactory("ActiveMQ-IO-server-" +
> this.toString(), false, ClientSessionFactoryImpl.class.getClassLoader());
> }
> });
> this.ioExecutorPool = new ThreadPoolExecutor(0, Integer.MAX_VALUE,
> 60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>(), tFactory);
> this.ioExecutorFactory = new OrderedExecutorFactory(ioExecutorPool);
> }
> {code}
> And it's using a {{SynchronousQueue}} to submit/take new wakeup tasks, it
> worths investigate if using a different thread pool, executor or a different
> "sleeping" strategy could reduce such cost under heavy load and improve
> response time with/without replication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
