c-taylor opened a new issue #7903:
URL: https://github.com/apache/trafficserver/issues/7903
Whilst chasing down another problem I noticed that the affinity rules are
'broken'.
ATS version 8.1.2
Issues:
1. Accept threads
1. Multiple accept threads are all pushed to same core
2. Accept thread core is not excluded from other masks and shared with a
NET thread
2. AIO threads have no affinity
3. NET threads are biased toward siblings over ‘proper’ cores
1. 0,32 are different threads on the same ‘core’
2. (12-30),(44-63) have nothing allocated
4. LOG_(FLUSH|PREPROC) may benefit from affinity (especially flush)
This leads to oversubscribed cores and likely explains why I see increased
handshake std_dev when applied.
----
System:
1x 32Co (64 HT)
CPU layout:
0-31 are all unique cores.
32-63 are siblings of 0-31.
```grep -H . /sys/devices/system/node/node0/cpu*/topology/core_cpus_list
/sys/devices/system/node/node0/cpu0/topology/core_cpus_list:0,32
/sys/devices/system/node/node0/cpu1/topology/core_cpus_list:1,33
/sys/devices/system/node/node0/cpu10/topology/core_cpus_list:10,42
/sys/devices/system/node/node0/cpu11/topology/core_cpus_list:11,43
/sys/devices/system/node/node0/cpu12/topology/core_cpus_list:12,44
/sys/devices/system/node/node0/cpu13/topology/core_cpus_list:13,45
/sys/devices/system/node/node0/cpu14/topology/core_cpus_list:14,46
/sys/devices/system/node/node0/cpu15/topology/core_cpus_list:15,47
/sys/devices/system/node/node0/cpu16/topology/core_cpus_list:16,48
/sys/devices/system/node/node0/cpu17/topology/core_cpus_list:17,49
/sys/devices/system/node/node0/cpu18/topology/core_cpus_list:18,50
/sys/devices/system/node/node0/cpu19/topology/core_cpus_list:19,51
/sys/devices/system/node/node0/cpu2/topology/core_cpus_list:2,34
/sys/devices/system/node/node0/cpu20/topology/core_cpus_list:20,52
/sys/devices/system/node/node0/cpu21/topology/core_cpus_list:21,53
/sys/devices/system/node/node0/cpu22/topology/core_cpus_list:22,54
/sys/devices/system/node/node0/cpu23/topology/core_cpus_list:23,55
/sys/devices/system/node/node0/cpu24/topology/core_cpus_list:24,56
/sys/devices/system/node/node0/cpu25/topology/core_cpus_list:25,57
/sys/devices/system/node/node0/cpu26/topology/core_cpus_list:26,58
/sys/devices/system/node/node0/cpu27/topology/core_cpus_list:27,59
/sys/devices/system/node/node0/cpu28/topology/core_cpus_list:28,60
/sys/devices/system/node/node0/cpu29/topology/core_cpus_list:29,61
/sys/devices/system/node/node0/cpu3/topology/core_cpus_list:3,35
/sys/devices/system/node/node0/cpu30/topology/core_cpus_list:30,62
/sys/devices/system/node/node0/cpu31/topology/core_cpus_list:31,63
/sys/devices/system/node/node0/cpu32/topology/core_cpus_list:0,32
/sys/devices/system/node/node0/cpu33/topology/core_cpus_list:1,33
/sys/devices/system/node/node0/cpu34/topology/core_cpus_list:2,34
/sys/devices/system/node/node0/cpu35/topology/core_cpus_list:3,35
/sys/devices/system/node/node0/cpu36/topology/core_cpus_list:4,36
/sys/devices/system/node/node0/cpu37/topology/core_cpus_list:5,37
/sys/devices/system/node/node0/cpu38/topology/core_cpus_list:6,38
/sys/devices/system/node/node0/cpu39/topology/core_cpus_list:7,39
/sys/devices/system/node/node0/cpu4/topology/core_cpus_list:4,36
/sys/devices/system/node/node0/cpu40/topology/core_cpus_list:8,40
/sys/devices/system/node/node0/cpu41/topology/core_cpus_list:9,41
/sys/devices/system/node/node0/cpu42/topology/core_cpus_list:10,42
/sys/devices/system/node/node0/cpu43/topology/core_cpus_list:11,43
/sys/devices/system/node/node0/cpu44/topology/core_cpus_list:12,44
/sys/devices/system/node/node0/cpu45/topology/core_cpus_list:13,45
/sys/devices/system/node/node0/cpu46/topology/core_cpus_list:14,46
/sys/devices/system/node/node0/cpu47/topology/core_cpus_list:15,47
/sys/devices/system/node/node0/cpu48/topology/core_cpus_list:16,48
/sys/devices/system/node/node0/cpu49/topology/core_cpus_list:17,49
/sys/devices/system/node/node0/cpu5/topology/core_cpus_list:5,37
/sys/devices/system/node/node0/cpu50/topology/core_cpus_list:18,50
/sys/devices/system/node/node0/cpu51/topology/core_cpus_list:19,51
/sys/devices/system/node/node0/cpu52/topology/core_cpus_list:20,52
/sys/devices/system/node/node0/cpu53/topology/core_cpus_list:21,53
/sys/devices/system/node/node0/cpu54/topology/core_cpus_list:22,54
/sys/devices/system/node/node0/cpu55/topology/core_cpus_list:23,55
/sys/devices/system/node/node0/cpu56/topology/core_cpus_list:24,56
/sys/devices/system/node/node0/cpu57/topology/core_cpus_list:25,57
/sys/devices/system/node/node0/cpu58/topology/core_cpus_list:26,58
/sys/devices/system/node/node0/cpu59/topology/core_cpus_list:27,59
/sys/devices/system/node/node0/cpu6/topology/core_cpus_list:6,38
/sys/devices/system/node/node0/cpu60/topology/core_cpus_list:28,60
/sys/devices/system/node/node0/cpu61/topology/core_cpus_list:29,61
/sys/devices/system/node/node0/cpu62/topology/core_cpus_list:30,62
/sys/devices/system/node/node0/cpu63/topology/core_cpus_list:31,63
/sys/devices/system/node/node0/cpu7/topology/core_cpus_list:7,39
/sys/devices/system/node/node0/cpu8/topology/core_cpus_list:8,40
/sys/devices/system/node/node0/cpu9/topology/core_cpus_list:9,41```
NUMA config
```CONFIG proxy.config.exec_thread.affinity INT 1```
Thread layout
```
8839
Name: [TS_MAIN]
Cpus_allowed_list: 0-63
8840
Name: traffic_server
Cpus_allowed_list: 0-63
8847
Name: [ET_NET 0]
Cpus_allowed_list: 0-63
8848
Name: [ET_NET 1]
Cpus_allowed_list: 0-63
8849
Name: [ET_NET 2]
Cpus_allowed_list: 0-63
8850
Name: [ET_NET 3]
Cpus_allowed_list: 0-63
8851
Name: [ET_NET 4]
Cpus_allowed_list: 0-63
8852
Name: [ET_NET 5]
Cpus_allowed_list: 0-63
8853
Name: [ET_NET 6]
Cpus_allowed_list: 0-63
8854
Name: [ET_NET 7]
Cpus_allowed_list: 0-63
8855
Name: [ET_NET 8]
Cpus_allowed_list: 0-63
8856
Name: [ET_NET 9]
Cpus_allowed_list: 0-63
8857
Name: [ET_NET 10]
Cpus_allowed_list: 0-63
8858
Name: [ET_NET 11]
Cpus_allowed_list: 0-63
8859
Name: [ET_NET 12]
Cpus_allowed_list: 0-63
8860
Name: [ET_NET 13]
Cpus_allowed_list: 0-63
8861
Name: [ET_NET 14]
Cpus_allowed_list: 0-63
8862
Name: [ET_NET 15]
Cpus_allowed_list: 0-63
8863
Name: [ET_NET 16]
Cpus_allowed_list: 0-63
8864
Name: [ET_NET 17]
Cpus_allowed_list: 0-63
8865
Name: [ET_NET 18]
Cpus_allowed_list: 0-63
8866
Name: [ET_NET 19]
Cpus_allowed_list: 0-63
8867
Name: [ET_NET 20]
Cpus_allowed_list: 0-63
8868
Name: [ET_NET 21]
Cpus_allowed_list: 0-63
8869
Name: [ET_NET 22]
Cpus_allowed_list: 0-63
8870
Name: [ET_NET 23]
Cpus_allowed_list: 0-63
8871
Name: [ET_DNS 0]
Cpus_allowed_list: 0-63
8872
Name: [LOG_PREPROC 0]
Cpus_allowed_list: 0-63
8873
Name: [LOG_FLUSH]
Cpus_allowed_list: 0-63
8874
Name: traffic_server
Cpus_allowed_list: 0-63
8888
Name: [ET_AIO 0:78]
Cpus_allowed_list: 0-63
8889
Name: [ET_AIO 1:78]
Cpus_allowed_list: 0-63
8890
Name: [ET_AIO 2:78]
Cpus_allowed_list: 0-63
8891
Name: [ET_AIO 3:78]
Cpus_allowed_list: 0-63
8892
Name: [ET_AIO 4:78]
Cpus_allowed_list: 0-63
8893
Name: [ET_AIO 5:78]
Cpus_allowed_list: 0-63
8894
Name: [ET_AIO 6:78]
Cpus_allowed_list: 0-63
8896
Name: [ET_AIO 7:78]
Cpus_allowed_list: 0-63
8897
Name: [ET_AIO 0:83]
Cpus_allowed_list: 0-63
8898
Name: [ET_AIO 1:83]
Cpus_allowed_list: 0-63
8899
Name: [ET_AIO 2:83]
Cpus_allowed_list: 0-63
8900
Name: [ET_AIO 3:83]
Cpus_allowed_list: 0-63
8901
Name: [ET_AIO 4:83]
Cpus_allowed_list: 0-63
8902
Name: [ET_AIO 5:83]
Cpus_allowed_list: 0-63
8903
Name: [ET_AIO 6:83]
Cpus_allowed_list: 0-63
8904
Name: [ET_AIO 7:83]
Cpus_allowed_list: 0-63
8905
Name: [ET_AIO 0:84]
Cpus_allowed_list: 0-63
8906
Name: [ET_AIO 1:84]
Cpus_allowed_list: 0-63
8907
Name: [ET_AIO 2:84]
Cpus_allowed_list: 0-63
8908
Name: [ET_AIO 3:84]
Cpus_allowed_list: 0-63
8909
Name: [ET_AIO 4:84]
Cpus_allowed_list: 0-63
8910
Name: [ET_AIO 5:84]
Cpus_allowed_list: 0-63
8911
Name: [ET_AIO 6:84]
Cpus_allowed_list: 0-63
8912
Name: [ET_AIO 7:84]
Cpus_allowed_list: 0-63
8913
Name: [ET_AIO 0:85]
Cpus_allowed_list: 0-63
8914
Name: [ET_AIO 1:85]
Cpus_allowed_list: 0-63
8915
Name: [ET_AIO 2:85]
Cpus_allowed_list: 0-63
8916
Name: [ET_AIO 3:85]
Cpus_allowed_list: 0-63
8917
Name: [ET_AIO 4:85]
Cpus_allowed_list: 0-63
8918
Name: [ET_AIO 5:85]
Cpus_allowed_list: 0-63
8919
Name: [ET_AIO 6:85]
Cpus_allowed_list: 0-63
8920
Name: [ET_AIO 7:85]
Cpus_allowed_list: 0-63
8921
Name: [ET_AIO 0:86]
Cpus_allowed_list: 0-63
8922
Name: [ET_AIO 1:86]
Cpus_allowed_list: 0-63
8923
Name: [ET_AIO 2:86]
Cpus_allowed_list: 0-63
8924
Name: [ET_AIO 3:86]
Cpus_allowed_list: 0-63
8925
Name: [ET_AIO 4:86]
Cpus_allowed_list: 0-63
8926
Name: [ET_AIO 5:86]
Cpus_allowed_list: 0-63
8927
Name: [ET_AIO 6:86]
Cpus_allowed_list: 0-63
8928
Name: [ET_AIO 7:86]
Cpus_allowed_list: 0-63
8929
Name: [ET_TASK 0]
Cpus_allowed_list: 0-63
8930
Name: [ET_TASK 1]
Cpus_allowed_list: 0-63
8931
Name: [ET_TASK 2]
Cpus_allowed_list: 0-63
8933
Name: [ET_TASK 3]
Cpus_allowed_list: 0-63
8935
Name: [ACCEPT 0:80]
Cpus_allowed_list: 0-63
8936
Name: [ACCEPT 1:80]
Cpus_allowed_list: 0-63
8939
Name: [ACCEPT 0:80]
Cpus_allowed_list: 0-63
8941
Name: [ACCEPT 1:80]
Cpus_allowed_list: 0-63
8942
Name: [ACCEPT 0:443]
Cpus_allowed_list: 0-63
8943
Name: [ACCEPT 1:443]
Cpus_allowed_list: 0-63
8945
Name: [ACCEPT 0:443]
Cpus_allowed_list: 0-63
8946
Name: [ACCEPT 1:443]
Cpus_allowed_list: 0-63
````
Affinity config
```CONFIG proxy.config.exec_thread.affinity INT 3```
In the layout below note that:
* CPU 10 is used by all 8 ACCEPT threads
* CPU 10 is used by 'ET_NET 20'
* ET_NET 21 is using '42' the sibling of CPU 10
* ET_AIO* use mask 0-63
Thread layout
```
46619
Name: [TS_MAIN]
Cpus_allowed_list: 0-63
46620
Name: traffic_server
Cpus_allowed_list: 0-63
46627
Name: [ET_NET 0]
Cpus_allowed_list: 0
46628
Name: [ET_NET 1]
Cpus_allowed_list: 32
46629
Name: [ET_NET 2]
Cpus_allowed_list: 1
46630
Name: [ET_NET 3]
Cpus_allowed_list: 33
46631
Name: [ET_NET 4]
Cpus_allowed_list: 2
46632
Name: [ET_NET 5]
Cpus_allowed_list: 34
46633
Name: [ET_NET 6]
Cpus_allowed_list: 3
46634
Name: [ET_NET 7]
Cpus_allowed_list: 35
46635
Name: [ET_NET 8]
Cpus_allowed_list: 4
46636
Name: [ET_NET 9]
Cpus_allowed_list: 36
46637
Name: [ET_NET 10]
Cpus_allowed_list: 5
46638
Name: [ET_NET 11]
Cpus_allowed_list: 37
46639
Name: [ET_NET 12]
Cpus_allowed_list: 6
46640
Name: [ET_NET 13]
Cpus_allowed_list: 38
46641
Name: [ET_NET 14]
Cpus_allowed_list: 7
46642
Name: [ET_NET 15]
Cpus_allowed_list: 39
46643
Name: [ET_NET 16]
Cpus_allowed_list: 8
46644
Name: [ET_NET 17]
Cpus_allowed_list: 40
46645
Name: [ET_NET 18]
Cpus_allowed_list: 9
46646
Name: [ET_NET 19]
Cpus_allowed_list: 41
46647
Name: [ET_NET 20]
Cpus_allowed_list: 10
46648
Name: [ET_NET 21]
Cpus_allowed_list: 42
46649
Name: [ET_NET 22]
Cpus_allowed_list: 11
46650
Name: [ET_NET 23]
Cpus_allowed_list: 43
46651
Name: [ET_DNS 0]
Cpus_allowed_list: 0-63
46652
Name: [LOG_PREPROC 0]
Cpus_allowed_list: 0-63
46653
Name: [LOG_FLUSH]
Cpus_allowed_list: 0-63
46654
Name: traffic_server
Cpus_allowed_list: 0-63
46666
Name: [ET_AIO 0:79]
Cpus_allowed_list: 0-63
46667
Name: [ET_AIO 1:79]
Cpus_allowed_list: 0-63
46668
Name: [ET_AIO 2:79]
Cpus_allowed_list: 0-63
46669
Name: [ET_AIO 3:79]
Cpus_allowed_list: 0-63
46670
Name: [ET_AIO 4:79]
Cpus_allowed_list: 0-63
46671
Name: [ET_AIO 5:79]
Cpus_allowed_list: 0-63
46672
Name: [ET_AIO 6:79]
Cpus_allowed_list: 0-63
46673
Name: [ET_AIO 7:79]
Cpus_allowed_list: 0-63
46674
Name: [ET_AIO 0:83]
Cpus_allowed_list: 0-63
46675
Name: [ET_AIO 1:83]
Cpus_allowed_list: 0-63
46676
Name: [ET_AIO 2:83]
Cpus_allowed_list: 0-63
46677
Name: [ET_AIO 3:83]
Cpus_allowed_list: 0-63
46678
Name: [ET_AIO 4:83]
Cpus_allowed_list: 0-63
46679
Name: [ET_AIO 5:83]
Cpus_allowed_list: 0-63
46680
Name: [ET_AIO 6:83]
Cpus_allowed_list: 0-63
46681
Name: [ET_AIO 7:83]
Cpus_allowed_list: 0-63
46682
Name: [ET_AIO 0:84]
Cpus_allowed_list: 0-63
46683
Name: [ET_AIO 1:84]
Cpus_allowed_list: 0-63
46684
Name: [ET_AIO 2:84]
Cpus_allowed_list: 0-63
46685
Name: [ET_AIO 3:84]
Cpus_allowed_list: 0-63
46686
Name: [ET_AIO 4:84]
Cpus_allowed_list: 0-63
46687
Name: [ET_AIO 5:84]
Cpus_allowed_list: 0-63
46688
Name: [ET_AIO 6:84]
Cpus_allowed_list: 0-63
46689
Name: [ET_AIO 7:84]
Cpus_allowed_list: 0-63
46691
Name: [ET_AIO 0:85]
Cpus_allowed_list: 0-63
46692
Name: [ET_AIO 1:85]
Cpus_allowed_list: 0-63
46693
Name: [ET_AIO 2:85]
Cpus_allowed_list: 0-63
46694
Name: [ET_AIO 3:85]
Cpus_allowed_list: 0-63
46695
Name: [ET_AIO 4:85]
Cpus_allowed_list: 0-63
46696
Name: [ET_AIO 5:85]
Cpus_allowed_list: 0-63
46697
Name: [ET_AIO 6:85]
Cpus_allowed_list: 0-63
46698
Name: [ET_AIO 7:85]
Cpus_allowed_list: 0-63
46699
Name: [ET_AIO 0:86]
Cpus_allowed_list: 0-63
46700
Name: [ET_AIO 1:86]
Cpus_allowed_list: 0-63
46701
Name: [ET_AIO 2:86]
Cpus_allowed_list: 0-63
46702
Name: [ET_AIO 3:86]
Cpus_allowed_list: 0-63
46703
Name: [ET_AIO 4:86]
Cpus_allowed_list: 0-63
46704
Name: [ET_AIO 5:86]
Cpus_allowed_list: 0-63
46705
Name: [ET_AIO 6:86]
Cpus_allowed_list: 0-63
46706
Name: [ET_AIO 7:86]
Cpus_allowed_list: 0-63
46708
Name: [ET_TASK 0]
Cpus_allowed_list: 0-63
46709
Name: [ET_TASK 1]
Cpus_allowed_list: 0-63
46710
Name: [ET_TASK 2]
Cpus_allowed_list: 0-63
46711
Name: [ET_TASK 3]
Cpus_allowed_list: 0-63
46713
Name: [ACCEPT 0:80]
Cpus_allowed_list: 10
46714
Name: [ACCEPT 1:80]
Cpus_allowed_list: 10
46715
Name: [ACCEPT 0:80]
Cpus_allowed_list: 10
46716
Name: [ACCEPT 1:80]
Cpus_allowed_list: 10
46717
Name: [ACCEPT 0:443]
Cpus_allowed_list: 10
46718
Name: [ACCEPT 1:443]
Cpus_allowed_list: 10
46719
Name: [ACCEPT 0:443]
Cpus_allowed_list: 10
46720
Name: [ACCEPT 1:443]
Cpus_allowed_list: 10
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
