I must say the improvements are pretty impressive!
Earlier number reported with 24 processes: 519K
Earlier number reported with 24 threads: 79K
New RPS with system irq tuning, today's git,
configuration changes, 24 threads: 353K
Old code with same tuning gave: 290K
My test machine is a 2 NUMA node server, CPU's 0, 2, ..., 22 on node 0, and
1, 3, ..., 23 on node 1. That adds to 24 cpus. The remaining 24 cores are
HT.
Node 0 has 24, 26, ..., 46 and node 1 has 25, 27, ..., 47. This may explain
why
it scales well to 24. The 16 irq's of the NIC and pinned to cpus 0, 2, ...,
30.
Hoping performance further improves.
12 threads: 280K
16 threads: 318K
24 threads: 353K (occasional drop till 330K)
32 threads: 238K
I am attaching 2 text files of the lock metrics for 24 and 32 threads. A
vimdiff shows the differences nicely (fd, task_rq, task_wq, proxy, server,
lbprm and buf_wq increased significantly).
Thanks!
On Thu, Oct 11, 2018 at 8:53 AM Krishna Kumar (Engineering) <
krishna...@flipkart.com> wrote:
> Thanks, will do that.
>
> On Thu, Oct 11, 2018 at 8:37 AM Willy Tarreau <w...@1wt.eu> wrote:
>
>> On Thu, Oct 11, 2018 at 08:18:21AM +0530, Krishna Kumar (Engineering)
>> wrote:
>> > Hi Willy,
>> >
>> > Thank you very much for the in-depth analysis and configuration setting
>> > suggestions.
>> > I believe I have got the 3 key items to continue based on your mail:
>> >
>> > 1. Thread pinning
>> > 2. Fix system irq pinning accordingly
>> > 3. Listen on all threads.
>> >
>> > I will post the configuration changes and the results.
>>
>> By the way, please pull the latest master fixes. I've addressed two issues
>> there with locking :
>> - one where the scheduler work was slightly too high, increasing the
>> time
>> spent on RQ lock
>> - another one where I messed up on a fix, causing lock-free pools to be
>> disabled (as seen in your output, where the POOL lock appears a lot)
>>
>> On some tests I've run here, I've found the stick-tables lock to be a
>> bottleneck when tracking is enabled. I don't have a short-term solution
>> to this, but looking at the code it's obvious that it can significantly
>> be improved (though it will take quite some time). I'll probably at least
>> try to replace it with an RW lock as I think it could improve the
>> situation.
>>
>> The FD lock is another one requiring some lift-up. I'm certain it's
>> possible,
>> I just don't know if it will not degrade low-thread count performance by
>> using too many atomic ops instead. We'll have to experiment.
>>
>> Cheers,
>> Willy
>>
>
Stats about Lock FD:
# write lock : 407304162
# write unlock: 407304155 (-7)
# wait time for write : 39627.620 msec
# wait time for write/lock: 97.292 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock TASK_RQ:
# write lock : 2230051
# write unlock: 2230051 (0)
# wait time for write : 63163.277 msec
# wait time for write/lock: 28323.692 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock TASK_WQ:
# write lock : 14897430
# write unlock: 14897430 (0)
# wait time for write : 49136.313 msec
# wait time for write/lock: 3298.308 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock POOL:
# write lock : 0
# write unlock: 0 (0)
# wait time for write : 0.000 msec
# wait time for write/lock: 0.000 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock LISTENER:
# write lock : 5500
# write unlock: 5500 (0)
# wait time for write : 0.076 msec
# wait time for write/lock: 13.734 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock PROXY:
# write lock : 7368276
# write unlock: 7368276 (0)
# wait time for write : 16768.394 msec
# wait time for write/lock: 2275.755 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock SERVER:
# write lock : 29505947
# write unlock: 29505947 (0)
# wait time for write : 38455.325 msec
# wait time for write/lock: 1303.308 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock LBPRM:
# write lock : 29504402
# write unlock: 29504402 (0)
# wait time for write : 228173.315 msec
# wait time for write/lock: 7733.535 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock BUF_WQ:
# write lock : 15392931
# write unlock: 15392931 (0)
# wait time for write : 35582.037 msec
# wait time for write/lock: 2311.583 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock STRMS:
# write lock : 14770
# write unlock: 14770 (0)
# wait time for write : 7.343 msec
# wait time for write/lock: 497.153 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock VARS:
# write lock : 22140518
# write unlock: 22140518 (0)
# wait time for write : 1676.562 msec
# wait time for write/lock: 75.724 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock FD:
# write lock : 432225022
# write unlock: 432225017 (-5)
# wait time for write : 12957.242 msec
# wait time for write/lock: 29.978 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock TASK_RQ:
# write lock : 1327383
# write unlock: 1327383 (0)
# wait time for write : 1860.444 msec
# wait time for write/lock: 1401.588 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock TASK_WQ:
# write lock : 24034868
# write unlock: 24034868 (0)
# wait time for write : 2908.506 msec
# wait time for write/lock: 121.012 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock POOL:
# write lock : 0
# write unlock: 0 (0)
# wait time for write : 0.000 msec
# wait time for write/lock: 0.000 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock LISTENER:
# write lock : 4940
# write unlock: 4940 (0)
# wait time for write : 0.046 msec
# wait time for write/lock: 9.331 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock PROXY:
# write lock : 13591357
# write unlock: 13591357 (0)
# wait time for write : 897.820 msec
# wait time for write/lock: 66.058 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock SERVER:
# write lock : 54395593
# write unlock: 54395593 (0)
# wait time for write : 3302.803 msec
# wait time for write/lock: 60.718 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock LBPRM:
# write lock : 54393800
# write unlock: 54393800 (0)
# wait time for write : 57584.576 msec
# wait time for write/lock: 1058.661 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock BUF_WQ:
# write lock : 27439616
# write unlock: 27439616 (0)
# wait time for write : 2684.139 msec
# wait time for write/lock: 97.820 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock STRMS:
# write lock : 14568
# write unlock: 14568 (0)
# wait time for write : 5.893 msec
# wait time for write/lock: 404.491 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
Stats about Lock VARS:
# write lock : 40809547
# write unlock: 40809547 (0)
# wait time for write : 2036.149 msec
# wait time for write/lock: 49.894 nsec
# read lock : 0
# read unlock : 0 (0)
# wait time for read : 0.000 msec
# wait time for read/lock : 0.000 nsec
