top - 12:25:30 up 8 days, 21:57, 6 users, load average: 16.77, 17.89, 15.97 Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 12.9%sy, 0.0%ni, 87.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 131999964k total, 125632212k used, 6367752k free, 207536k buffers Swap: 6291448k total, 16352k used, 6275096k free, 6655152k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6915 mysql 20 0 34.2g 1.5g 5372 S 59.7 1.2 0:55.34 mysqld 7012 root 20 0 3189m 1.3g 1468 S 15.1 1.1 2:40.51 I really think there is something wrong on your system related to this migration threads. You still have a load of 17 with CPU 87% idle... Strange. And even if they are more active now, mysql and robinhood only produce a load of 0.7. It sounds more like a driver or hardware issue, or a RT kernel mode... Do you run a specific kernel? or with realtime options? If you have a spare node, it would be worthwhile to run robinhood on it and see if you have the same strange load. Regards Thomas. On 04/23/15 12:49, Carmelo Ponti (CSCS) wrote: > I divided the two processes between the two sockets and now I can see > them using some CPU time to time: > > # top -p 7012,6915 -b > > top - 12:25:27 up 8 days, 21:57, 6 users, load average: 16.77, 17.89, > 15.97 > Tasks: 2 total, 1 running, 1 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.0%us, 12.7%sy, 0.0%ni, 87.2%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 131999964k total, 125626996k used, 6372968k free, 207532k > buffers > Swap: 6291448k total, 16352k used, 6275096k free, 6655084k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 7012 root 20 0 3189m 1.3g 1468 S 1.6 1.1 2:40.02 > robinhood > 6915 mysql 20 0 34.2g 1.5g 5372 R 0.0 1.2 0:53.40 > mysqld > > top - 12:25:30 up 8 days, 21:57, 6 users, load average: 16.77, 17.89, > 15.97 > Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.0%us, 12.9%sy, 0.0%ni, 87.0%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 131999964k total, 125632212k used, 6367752k free, 207536k > buffers > Swap: 6291448k total, 16352k used, 6275096k free, 6655152k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 6915 mysql 20 0 34.2g 1.5g 5372 S 59.7 1.2 0:55.34 > mysqld > 7012 root 20 0 3189m 1.3g 1468 S 15.1 1.1 2:40.51 > robinhood > > top - 12:25:33 up 8 days, 21:57, 6 users, load average: 16.39, 17.79, > 15.94 > Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.0%us, 13.7%sy, 0.0%ni, 86.2%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 131999964k total, 125631972k used, 6367992k free, 207540k > buffers > Swap: 6291448k total, 16352k used, 6275096k free, 6655116k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 7012 root 20 0 3189m 1.3g 1468 S 21.3 1.1 2:41.17 > robinhood > 6915 mysql 20 0 34.2g 1.5g 5372 S 0.0 1.2 0:55.34 > mysqld > > In this moment we have 24 million of Changelog lines so I guest we need > some time to see if there is an improvement. By sure the load average > decreased a lot. > > Today I also noticed many messages as the following on dmesg and > on /var/log/messages: > > Lustre: 24416:0:(kernel_user_comm.c:201:libcfs_kkuc_msg_put()) message > send failed (-32) > Lustre: 24416:0:(kernel_user_comm.c:201:libcfs_kkuc_msg_put()) Skipped 1 > previous similar message > > I searched on google and I found an old request on robinhood-support > (http://sourceforge.net/p/robinhood/mailman/message/31162194/) which > explain the messages and how to fix it. Could these messages explain in > part the problem we have or it a consequence of the problem? > > Carmelo > > On Thu, 2015-04-23 at 10:00 +0200, LEIBOVICI Thomas wrote: >> On 04/22/15 16:26, Carmelo Ponti (CSCS) wrote: >>> I will wait until tomorrow to see if the situation will go better but I >>> can immediately noticed that the cpu usage of robinhood now is between >>> 40% and 100%. The load of mysql didn't change: >>> >>> 2847 root 20 0 3867m 1.7g 1528 S 63.9 1.4 47:41.73 robinhood >>> 3217 mysql 20 0 37.6g 1.3g 4500 S 0.0 1.0 1603:03 mysqld >> It may be a good sign that robinhood now does something :) >> >> mysqld should be much more active however. >> What about pinning it too? >> >> Thomas. ------------------------------------------------------------------------------ BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF _______________________________________________ robinhood-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/robinhood-support
