[ https://issues.apache.org/jira/browse/ROCKETMQ-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976178#comment-15976178 ]
yukon commented on ROCKETMQ-152: -------------------------------- If you set a large sendMessageThreadPoolNums, please use ReentrantLock instead of spin lock. > lockForPutMessage CAS may cause system load very high and putMessage cost > more than 500ms if the sendMessageThread num is very much > -------------------------------------------------------------------------------------------------------------------------------------- > > Key: ROCKETMQ-152 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-152 > Project: Apache RocketMQ > Issue Type: Improvement > Components: rocketmq-store > Affects Versions: 4.0.0-incubating > Environment: sendMessageThreadPoolNums=128 and > JDK jdk1.7.0_80 linux 2.6.32-504.el6.x86_64 and have more than 50 g free > mem > Reporter: zhaoziyan > Assignee: yukon > Attachments: QQ截图20170322152327.png > > > broker store sendMessageThreadPoolNums=128 > 16 thread 1000bytes benchmark test , System load to 100 > put message acquire the lock may spin,cause System load high and putMessage > cost more than 500ms > /** > * Spin util acquired the lock. > */ > private void lockForPutMessage() { > if > (this.defaultMessageStore.getMessageStoreConfig().isUseReentrantLockWhenPutMessage()) > { > putMessageNormalLock.lock(); > } else { > boolean flag; > do { > flag = this.putMessageSpinLock.compareAndSet(true, false); > } > while (!flag); > } > } > 2017-03-22 14:40:16 WARN SendMessageThread_101 - putMessage not in lock > eclipse time(ms)=1030, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_34 - putMessage not in lock > eclipse time(ms)=1932, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_3 - putMessage not in lock eclipse > time(ms)=581, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_57 - putMessage not in lock > eclipse time(ms)=583, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_123 - putMessage not in lock > eclipse time(ms)=2225, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_1 - putMessage not in lock eclipse > time(ms)=1642, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_58 - putMessage not in lock > eclipse time(ms)=587, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_19 - putMessage not in lock > eclipse time(ms)=1369, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_32 - putMessage not in lock > eclipse time(ms)=1896, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_2 - putMessage not in lock eclipse > time(ms)=1018, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_122 - putMessage not in lock > eclipse time(ms)=657, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_120 - putMessage not in lock > eclipse time(ms)=592, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_88 - putMessage not in lock > eclipse time(ms)=559, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_44 - putMessage not in lock > eclipse time(ms)=892, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_79 - putMessage not in lock > eclipse time(ms)=699, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_84 - putMessage not in lock > eclipse time(ms)=616, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_112 - putMessage not in lock > eclipse time(ms)=515, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_59 - putMessage not in lock > eclipse time(ms)=1301, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_96 - putMessage not in lock > eclipse time(ms)=635, bodyLength=10013 > 2017-03-22 14:40:16 WARN SendMessageThread_90 - ********lockForPutMessage > cost:317 > 2017-03-22 14:40:16 WARN SendMessageThread_10 - ********lockForPutMessage > cost:447 > 2017-03-22 14:40:16 WARN SendMessageThread_62 - ********lockForPutMessage > cost:450 > 2017-03-22 14:40:16 WARN SendMessageThread_93 - ********lockForPutMessage > cost:131 > 2017-03-22 14:40:16 WARN SendMessageThread_8 - ********lockForPutMessage > cost:171 > 2017-03-22 14:40:16 WARN SendMessageThread_93 - ********lockForPutMessage > cost:4 > 2017-03-22 14:40:16 WARN SendMessageThread_105 - ********lockForPutMessage > cost:666 > 2017-03-22 14:40:16 WARN SendMessageThread_127 - ********lockForPutMessage > cost:689 > 2017-03-22 14:40:16 WARN SendMessageThread_30 - ********lockForPutMessage > cost:341 > 2017-03-22 14:40:16 WARN SendMessageThread_70 - ********lockForPutMessage > cost:317 > 2017-03-22 14:40:16 WARN SendMessageThread_92 - ********lockForPutMessage > cost:58 > 2017-03-22 14:40:16 WARN SendMessageThread_63 - ********lockForPutMessage > cost:261 > 2017-03-22 14:40:16 WARN SendMessageThread_98 - ********lockForPutMessage > cost:261 > 2017-03-22 14:40:16 WARN SendMessageThread_27 - ********lockForPutMessage > cost:546 > 2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage > cost:257 > 2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage > cost:1047 > 2017-03-22 14:40:16 WARN SendMessageThread_82 - ********lockForPutMessage > cost:1891 > 2017-03-22 14:40:16 WARN SendMessageThread_57 - ********lockForPutMessage > cost:559 > 2017-03-22 14:40:16 WARN SendMessageThread_40 - ********lockForPutMessage > cost:235 > 2017-03-22 14:40:16 WARN SendMessageThread_77 - ********lockForPutMessage > cost:265 > 2017-03-22 14:40:16 WARN SendMessageThread_35 - ********lockForPutMessage > cost:909 > 2017-03-22 14:40:16 WARN SendMessageThread_113 - ********lockForPutMessage > cost:244 > 2017-03-22 14:40:16 WARN SendMessageThread_97 - ********lockForPutMessage > cost:922 > 2017-03-22 14:40:16 WARN SendMessageThread_114 - ********lockForPutMessage > cost:40 > 2017-03-22 14:40:16 WARN SendMessageThread_43 - ********lockForPutMessage > cost:351 > 2017-03-22 14:40:16 WARN SendMessageThread_126 - ********lockForPutMessage > cost:346 > 2017-03-22 14:40:16 WARN SendMessageThread_110 - ********lockForPutMessage > cost:887 > 2017-03-22 14:40:16 WARN SendMessageThread_15 - ********lockForPutMessage > cost:320 > 2017-03-22 14:40:16 WARN SendMessageThread_3 - ********lockForPutMessage > cost:566 > 2017-03-22 14:40:16 WARN SendMessageThread_68 - ********lockForPutMessage > cost:748 > 2017-03-22 14:40:16 WARN SendMessageThread_22 - ********lockForPutMessage > cost:308 > 2017-03-22 14:40:16 WARN SendMessageThread_34 - ********lockForPutMessage > cost:1926 > 2017-03-22 14:40:16 WARN SendMessageThread_91 - ********lockForPutMessage > cost:358 > 2017-03-22 14:40:16 WARN SendMessageThread_25 - ********lockForPutMessage > cost:107 > 2017-03-22 14:40:16 WARN SendMessageThread_101 - ********lockForPutMessage > cost:1024 > 2017-03-22 14:40:16 WARN SendMessageThread_89 - ********lockForPutMessage > cost:326 > 2017-03-22 14:40:16 WARN SendMessageThread_43 - ********lockForPutMessage > cost:5 > 2017-03-22 14:40:16 WARN SendMessageThread_30 - ********lockForPutMessage > cost:1 > 2017-03-22 14:40:16 WARN SendMessageThread_42 - ********lockForPutMessage > cost:326 > 2017-03-22 14:40:16 WARN SendMessageThread_95 - ********lockForPutMessage > cost:58 > 2017-03-22 14:40:16 WARN SendMessageThread_19 - ********lockForPutMessage > cost:1363 > 2017-03-22 14:40:16 WARN SendMessageThread_86 - ********lockForPutMessage > cost:92 > 2017-03-22 14:40:16 WARN SendMessageThread_1 - ********lockForPutMessage > cost:1640 > 2017-03-22 14:40:16 WARN SendMessageThread_82 - ********lockForPutMessage > cost:30 > 2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage > cost:33 > 2017-03-22 14:40:16 WARN SendMessageThread_58 - ********lockForPutMessage > cost:582 > 2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage > cost:34 > 2017-03-22 14:40:16 WARN SendMessageThread_123 - ********lockForPutMessage > cost:2224 > 2017-03-22 14:40:16 WARN SendMessageThread_78 - ********lockForPutMessage > cost:286 > 2017-03-22 14:40:16 WARN SendMessageThread_63 - ********lockForPutMessage > cost:27 > 2017-03-22 14:40:16 WARN SendMessageThread_32 - ********lockForPutMessage > cost:1874 > 2017-03-22 14:40:16 WARN SendMessageThread_87 - ********lockForPutMessage > cost:312 > 2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage > cost:14 > 2017-03-22 14:40:16 WARN SendMessageThread_123 - ********lockForPutMessage > cost:21 > 2017-03-22 14:40:16 WARN SendMessageThread_45 - ********lockForPutMessage > cost:382 > 2017-03-22 14:40:16 WARN SendMessageThread_89 - ********lockForPutMessage > cost:37 > 2017-03-22 14:40:16 WARN SendMessageThread_15 - ********lockForPutMessage > cost:51 > 2017-03-22 14:40:16 WARN SendMessageThread_2 - ********lockForPutMessage > cost:1015 > 2017-03-22 14:40:16 WARN SendMessageThread_68 - ********lockForPutMessage > cost:50 > 2017-03-22 14:40:16 WARN SendMessageThread_22 - ********lockForPutMessage > cost:51 > 2017-03-22 14:40:16 WARN SendMessageThread_84 - ********lockForPutMessage > cost:606 > 2017-03-22 14:40:16 WARN SendMessageThread_3 - ********lockForPutMessage > cost:42 > 2017-03-22 14:40:16 WARN SendMessageThread_122 - ********lockForPutMessage > cost:652 > 2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage > cost:19 > 2017-03-22 14:40:16 WARN SendMessageThread_126 - ********lockForPutMessage > cost:24 > 2017-03-22 14:40:16 WARN SendMessageThread_57 - ********lockForPutMessage > cost:3 > 2017-03-22 14:40:16 WARN SendMessageThread_110 - ********lockForPutMessage > cost:3 > 2017-03-22 14:40:16 WARN SendMessageThread_70 - ********lockForPutMessage > cost:40 > 2017-03-22 14:40:16 WARN SendMessageThread_118 - ********lockForPutMessage > cost:279 > 2017-03-22 14:40:16 WARN SendMessageThread_103 - ********lockForPutMessage > cost:301 > 2017-03-22 14:40:16 WARN SendMessageThread_36 - ********lockForPutMessage > cost:384 > 2017-03-22 14:40:16 WARN SendMessageThread_120 - ********lockForPutMessage > cost:587 > 2017-03-22 14:40:16 WARN SendMessageThread_88 - ********lockForPutMessage > cost:554 > 2017-03-22 14:40:16 WARN SendMessageThread_124 - ********lockForPutMessage > cost:324 > 2017-03-22 14:40:16 WARN SendMessageThread_44 - ********lockForPutMessage > cost:888 > 2017-03-22 14:40:16 WARN SendMessageThread_49 - ********lockForPutMessage > cost:311 > 2017-03-22 14:40:16 WARN SendMessageThread_41 - ********lockForPutMessage > cost:312 > 2017-03-22 14:40:16 WARN SendMessageThread_5 - ********lockForPutMessage > cost:115 > 2017-03-22 14:40:16 WARN SendMessageThread_4 - ********lockForPutMessage > cost:151 > 2017-03-22 14:40:16 WARN SendMessageThread_119 - ********lockForPutMessage > cost:79 > 2017-03-22 14:40:16 WARN SendMessageThread_65 - ********lockForPutMessage > cost:107 > 2017-03-22 14:40:16 WARN SendMessageThread_128 - ********lockForPutMessage > cost:112 > 2017-03-22 14:40:16 WARN SendMessageThread_96 - ********lockForPutMessage > cost:609 > 2017-03-22 14:40:16 WARN SendMessageThread_79 - ********lockForPutMessage > cost:697 > 2017-03-22 14:40:16 WARN SendMessageThread_7 - ********lockForPutMessage > cost:417 > 2017-03-22 14:40:16 WARN SendMessageThread_94 - ********lockForPutMessage > cost:419 > 2017-03-22 14:40:16 WARN SendMessageThread_72 - ********lockForPutMessage > cost:208 > 2017-03-22 14:40:16 WARN SendMessageThread_8 - ********lockForPutMessage > cost:42 > 2017-03-22 14:40:16 WARN SendMessageThread_99 - ********lockForPutMessage > cost:237 > 2017-03-22 14:40:16 WARN SendMessageThread_16 - ********lockForPutMessage > cost:123 > 2017-03-22 14:40:16 WARN SendMessageThread_85 - ********lockForPutMessage > cost:133 > 2017-03-22 14:40:16 WARN SendMessageThread_111 - ********lockForPutMessage > cost:139 > 2017-03-22 14:40:16 WARN SendMessageThread_112 - ********lockForPutMessage > cost:510 > 2017-03-22 14:40:16 WARN SendMessageThread_45 - ********lockForPutMessage > cost:2 > 2017-03-22 14:40:16 WARN SendMessageThread_2 - ********lockForPutMessage > cost:3 > 2017-03-22 14:40:16 WARN SendMessageThread_59 - ********lockForPutMessage > cost:1291 > 2017-03-22 14:40:16 WARN SendMessageThread_64 - ********lockForPutMessage > cost:105 > 2017-03-22 14:40:16 WARN SendMessageThread_59 - ********lockForPutMessage > cost:48 > 2017-03-22 14:40:16 WARN SendMessageThread_46 - ********lockForPutMessage > cost:255 > 2017-03-22 14:40:16 WARN SendMessageThread_49 - ********lockForPutMessage > cost:95 > 2017-03-22 14:40:16 WARN SendMessageThread_44 - ********lockForPutMessage > cost:7 > 2017-03-22 14:40:16 WARN SendMessageThread_112 - ********lockForPutMessage > cost:37 > 2017-03-22 14:40:16 WARN SendMessageThread_7 - ********lockForPutMessage > cost:75 -- This message was sent by Atlassian JIRA (v6.3.15#6346)