> > what kind of load do you have? Puppet? Facter? Is that ENC? Something > else? > > Can you tell which requests are slow from logs or monitoring? > > Yes, I should have mentioned that - there's very little puppet and ENC work done by this cluster at this point (more is coming soon though). Host discovery is by far the largest workload - 7600 discovered systems at this point. The last spike that I saw the impact to overall flows was when 300-400 systems were trying to register at the same time. Because of the deadlocks, about 200-300 systems could not register repeatedly and had to keep retrying for a rather long time. Rather often these registration attempts would end up creating either duplicate entries with the same "mac<mac>" but different IDs in a DB or an "empty" discovery host entry. Both of these would prevent a system successfully register unless they are removed (I had to write a little script that runs from the cron to do so). Here are the examples of an "empty" record and duplicate ones (as they get deleted):
{ "id": 437923, "name": "mac90e2bae6cc70", "last_compile": null, "last_report": null, "updated_at": "2017-08-22T07:08:54.000Z", "created_at": "2017-08-22T07:08:54.000Z", "root_pass": "<removed>", "architecture_id": null, "operatingsystem_id": null, "environment_id": null, "ptable_id": null, "medium_id": null, "build": false, "comment": null, "disk": null, "installed_at": null, "model_id": null, "hostgroup_id": null, "owner_id": null, "owner_type": null, "enabled": true, "puppet_ca_proxy_id": null, "managed": false, "use_image": null, "image_file": null, "uuid": null, "compute_resource_id": null, "puppet_proxy_id": null, "certname": null, "image_id": null, "organization_id": null, "location_id": null, "otp": null, "realm_id": null, "compute_profile_id": null, "provision_method": null, "grub_pass": "", "global_status": 0, "lookup_value_matcher": null, "discovery_rule_id": null, "salt_proxy_id": null, "salt_environment_id": null, "pxe_loader": null } Duplicates (usually the later duplicate would be an empty one as well, but not all the time): { "id": 430090, "name": "mac3417ebe3f8f1", "last_compile": null, "last_report": "2017-09-14T19:47:55.000Z", "updated_at": "2017-09-14T19:47:57.000Z", "created_at": "2017-03-08T20:24:05.000Z", "root_pass": "<removed>", "architecture_id": null, "operatingsystem_id": null, "environment_id": null, "ptable_id": null, "medium_id": null, "build": false, "comment": null, "disk": null, "installed_at": null, "model_id": 3, "hostgroup_id": null, "owner_id": 10, "owner_type": "User", "enabled": true, "puppet_ca_proxy_id": null, "managed": false, "use_image": null, "image_file": null, "uuid": null, "compute_resource_id": null, "puppet_proxy_id": null, "certname": null, "image_id": null, "organization_id": null, "location_id": null, "otp": null, "realm_id": null, "compute_profile_id": null, "provision_method": null, "grub_pass": "", "global_status": 0, "lookup_value_matcher": null, "discovery_rule_id": null, "salt_proxy_id": null, "salt_environment_id": null, "pxe_loader": null } { "id": 438146, "name": "mac3417ebe3f8f1", "last_compile": null, "last_report": "2017-09-11T08:58:05.000Z", "updated_at": "2017-09-11T08:58:07.000Z", "created_at": "2017-08-24T19:44:23.000Z", "root_pass": "<removed>", "architecture_id": null, "operatingsystem_id": null, "environment_id": null, "ptable_id": null, "medium_id": null, "build": false, "comment": null, "disk": null, "installed_at": null, "model_id": null, "hostgroup_id": null, "owner_id": null, "owner_type": null, "enabled": true, "puppet_ca_proxy_id": null, "managed": false, "use_image": null, "image_file": null, "uuid": null, "compute_resource_id": null, "puppet_proxy_id": null, "certname": null, "image_id": null, "organization_id": null, "location_id": null, "otp": null, "realm_id": null, "compute_profile_id": null, "provision_method": null, "grub_pass": "", "global_status": 0, "lookup_value_matcher": null, "discovery_rule_id": null, "salt_proxy_id": null, "salt_environment_id": null, "pxe_loader": null } I can't tell if any queries are slow - can you remind me how to do that? Thanks! -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscr...@googlegroups.com. To post to this group, send email to foreman-users@googlegroups.com. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.