On Fri, Sep 15, 2017 at 8:56 PM, 'Konstantin Orekhov' via Foreman users <
foreman-users@googlegroups.com> wrote:
>
>
>> what kind of load do you have? Puppet? Facter? Is that ENC? Something
>> else?
>>
>> Can you tell which requests are slow from logs or monitoring?
>>
>>
> Yes, I should have mentioned that - there's very little puppet and ENC
> work done by this cluster at this point (more is coming soon though). Host
> discovery is by far the largest workload - 7600 discovered systems at this
> point. The last spike that I saw the impact to overall flows was when
> 300-400 systems were trying to register at the same time. Because of the
> deadlocks, about 200-300 systems could not register repeatedly and had to
> keep retrying for a rather long time.
> Rather often these registration attempts would end up creating either
> duplicate entries with the same "mac<mac>" but different IDs in a DB or an
> "empty" discovery host entry. Both of these would prevent a system
> successfully register unless they are removed (I had to write a little
> script that runs from the cron to do so). Here are the examples of an
> "empty" record and duplicate ones (as they get deleted):
>
>
Lukas - how about we change discovery to be async? e.g. import all new
discovered systems into active job and than process than one / multiple at
a time? I assume this would require a image change too (so it knows when
the discovery"job" is done)
> {
> "id": 437923,
> "name": "mac90e2bae6cc70",
> "last_compile": null,
> "last_report": null,
> "updated_at": "2017-08-22T07:08:54.000Z",
> "created_at": "2017-08-22T07:08:54.000Z",
> "root_pass": "<removed>",
> "architecture_id": null,
> "operatingsystem_id": null,
> "environment_id": null,
> "ptable_id": null,
> "medium_id": null,
> "build": false,
> "comment": null,
> "disk": null,
> "installed_at": null,
> "model_id": null,
> "hostgroup_id": null,
> "owner_id": null,
> "owner_type": null,
> "enabled": true,
> "puppet_ca_proxy_id": null,
> "managed": false,
> "use_image": null,
> "image_file": null,
> "uuid": null,
> "compute_resource_id": null,
> "puppet_proxy_id": null,
> "certname": null,
> "image_id": null,
> "organization_id": null,
> "location_id": null,
> "otp": null,
> "realm_id": null,
> "compute_profile_id": null,
> "provision_method": null,
> "grub_pass": "",
> "global_status": 0,
> "lookup_value_matcher": null,
> "discovery_rule_id": null,
> "salt_proxy_id": null,
> "salt_environment_id": null,
> "pxe_loader": null
> }
>
> Duplicates (usually the later duplicate would be an empty one as well, but
> not all the time):
>
> {
> "id": 430090,
> "name": "mac3417ebe3f8f1",
> "last_compile": null,
> "last_report": "2017-09-14T19:47:55.000Z",
> "updated_at": "2017-09-14T19:47:57.000Z",
> "created_at": "2017-03-08T20:24:05.000Z",
> "root_pass": "<removed>",
> "architecture_id": null,
> "operatingsystem_id": null,
> "environment_id": null,
> "ptable_id": null,
> "medium_id": null,
> "build": false,
> "comment": null,
> "disk": null,
> "installed_at": null,
> "model_id": 3,
> "hostgroup_id": null,
> "owner_id": 10,
> "owner_type": "User",
> "enabled": true,
> "puppet_ca_proxy_id": null,
> "managed": false,
> "use_image": null,
> "image_file": null,
> "uuid": null,
> "compute_resource_id": null,
> "puppet_proxy_id": null,
> "certname": null,
> "image_id": null,
> "organization_id": null,
> "location_id": null,
> "otp": null,
> "realm_id": null,
> "compute_profile_id": null,
> "provision_method": null,
> "grub_pass": "",
> "global_status": 0,
> "lookup_value_matcher": null,
> "discovery_rule_id": null,
> "salt_proxy_id": null,
> "salt_environment_id": null,
> "pxe_loader": null
> }
> {
> "id": 438146,
> "name": "mac3417ebe3f8f1",
> "last_compile": null,
> "last_report": "2017-09-11T08:58:05.000Z",
> "updated_at": "2017-09-11T08:58:07.000Z",
> "created_at": "2017-08-24T19:44:23.000Z",
> "root_pass": "<removed>",
> "architecture_id": null,
> "operatingsystem_id": null,
> "environment_id": null,
> "ptable_id": null,
> "medium_id": null,
> "build": false,
> "comment": null,
> "disk": null,
> "installed_at": null,
> "model_id": null,
> "hostgroup_id": null,
> "owner_id": null,
> "owner_type": null,
> "enabled": true,
> "puppet_ca_proxy_id": null,
> "managed": false,
> "use_image": null,
> "image_file": null,
> "uuid": null,
> "compute_resource_id": null,
> "puppet_proxy_id": null,
> "certname": null,
> "image_id": null,
> "organization_id": null,
> "location_id": null,
> "otp": null,
> "realm_id": null,
> "compute_profile_id": null,
> "provision_method": null,
> "grub_pass": "",
> "global_status": 0,
> "lookup_value_matcher": null,
> "discovery_rule_id": null,
> "salt_proxy_id": null,
> "salt_environment_id": null,
> "pxe_loader": null
> }
>
> I can't tell if any queries are slow - can you remind me how to do that?
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-users+unsubscr...@googlegroups.com.
> To post to this group, send email to foreman-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
