Were you running the "foreman-trace rails calls" utility when you searched the problematic "puppet" query? Because the output looks like sane output, I can see one hundred of NIC models allocated, but this should not take 6 Gbs at all.
I've slightly enhanced the foreman-tracer, can you also try to run this while doing the problematic query: foreman-tracer rails objects-total all foreman-tracer rails objects all foreman-tracer rails calls all We must be allocating some number of objects somewhere, this should reveal them. LZ On Wed, Nov 2, 2016 at 11:55 AM, Erez Zarum <erezza...@gmail.com> wrote: > This is the problem, it's seems like running a query without any field is > causing this problem, it became more aggressive now probably because i am > running it under the "Classes" dashboard which now have much much more > objects than before. > > Running a query without a specific field: "puppet", memory usage from 1G to > 5G, a ruby instance takes 100% CPU for about a minute. > Running a query with a specific field: "name = puppet", memory usage remains > quite the same, small spike in CPU, returns immediately (less than a > second). > > > On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum wrote: >> >> I have the "theforeman/puppet" module, when i search under the "Classes" >> dashboard the word "puppet" it's seems to cause a very high CPU and memory >> starts to increase up to 6GB from almost 1GB. >> >> On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum wrote: >>> >>> rails objects-total >>> OBJECT COUNT >>> ERB 826 >>> Nic::Managed 104 >>> Foreman::Parameters::Caster 38 >>> IPAddr 24 >>> ConfigReportStatusCalculator 24 >>> ActiveSupport::HashWithIndifferentAccess 18 >>> SafeRender 10 >>> Range 9 >>> ActionView::OutputBuffer 8 >>> Classification::GlobalParam 6 >>> ActionDispatch::Request 6 >>> Foreman::Parameters::Validator 6 >>> OpenSSL::ASN1::ASN1Data 6 >>> OpenSSL::ASN1::Sequence 6 >>> Authorizer 5 >>> CertificateExtract 3 >>> OpenSSL::ASN1::ObjectId 3 >>> OpenSSL::ASN1::OctetString 3 >>> OpenSSL::X509::Certificate 3 >>> Classification::ClassParam 2 >>> ActiveModel::Name 2 >>> PuppetFactParser 1 >>> HostStatus::Global 1 >>> PuppetFactImporter 1 >>> >>> rails objects >>> FILENAME >>> LINE OBJECT COUNT >>> /usr/share/foreman/app/services/safe_render.rb >>> 39 ERB 826 >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb >>> 10 Nic::Managed 96 >>> /usr/share/foreman/app/services/classification/base.rb >>> 151 Foreman::Parameters::Caster 38 >>> /usr/share/foreman/app/models/config_report.rb >>> 88 ConfigReportStatusCalculator 18 >>> /usr/share/foreman/app/models/host/base.rb >>> 390 ActiveSupport::HashWithIndifferentAccess 11 >>> /usr/share/foreman/app/models/concerns/encryptable.rb >>> 50 Range 8 >>> /usr/share/foreman/app/models/host/base.rb >>> 78 Nic::Managed 8 >>> /usr/share/foreman/lib/net/validations.rb >>> 31 IPAddr 8 >>> /usr/share/foreman/lib/net/validations.rb >>> 18 IPAddr 8 >>> /usr/share/foreman/app/services/classification/base.rb >>> 8 SafeRender 8 >>> /usr/share/foreman/lib/net/validations.rb >>> 102 IPAddr 8 >>> /usr/share/foreman/app/models/lookup_value.rb >>> 51 Foreman::Parameters::Validator 6 >>> /usr/share/foreman/lib/middleware/tagged_logging.rb >>> 10 ActionDispatch::Request 6 >>> /usr/share/foreman/app/models/host/managed.rb >>> 934 Classification::GlobalParam 6 >>> /usr/share/foreman/app/models/host_status/configuration_status.rb >>> 112 ConfigReportStatusCalculator 6 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 17 OpenSSL::ASN1::ASN1Data 6 >>> /usr/share/foreman/app/helpers/application_helper.rb >>> 104 Authorizer 5 >>> /usr/share/foreman/app/services/authorizer_cache.rb >>> 3 ActiveSupport::HashWithIndifferentAccess 5 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 5 OpenSSL::X509::Certificate 3 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 16 OpenSSL::ASN1::OctetString 3 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 16 OpenSSL::ASN1::Sequence 3 >>> >>> /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb >>> 59 CertificateExtract 3 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 16 OpenSSL::ASN1::ObjectId 3 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 17 OpenSSL::ASN1::Sequence 3 >>> /usr/share/foreman/app/models/host/managed.rb >>> 420 SafeRender 2 >>> >>> rails calls >>> FILENAME LINE >>> METHOD CALLS >>> /usr/share/foreman/app/models/setting.rb 218 >>> cache 2120 >>> /usr/share/foreman/app/models/setting.rb 72 >>> [] 2086 >>> /usr/share/foreman/app/models/concerns/exportable.rb 55 >>> exportable_attributes 2054 >>> /usr/share/foreman/app/services/safe_render.rb 12 >>> parse 1138 >>> /usr/share/foreman/app/services/safe_render.rb 31 >>> parse_string 826 >>> /usr/share/foreman/app/models/concerns/exportable.rb 33 >>> export_attr 810 >>> /usr/share/foreman/app/models/fact_value.rb 7 >>> name 509 >>> /usr/share/foreman/app/models/user.rb 137 >>> admin? 180 >>> /usr/share/foreman/app/models/taxonomy.rb 100 >>> expand 180 >>> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92 >>> to_s 112 >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9 >>> setup_object_clone 104 >>> /usr/share/foreman/app/models/nic/managed.rb 45 >>> hostname 96 >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 34 >>> with_taxonomy_scope 84 >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 115 >>> scope_by_taxable_ids 84 >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 75 >>> taxable_ids 84 >>> /usr/share/foreman/app/models/host/base.rb 441 >>> get_interface_by_flag 74 >>> /usr/share/foreman/app/models/concerns/exportable.rb 35 >>> call 66 >>> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12 >>> dns_ready? 64 >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 131 >>> set_current_taxonomy 62 >>> /usr/share/foreman/app/models/concerns/exportable.rb 14 >>> to_export 62 >>> /usr/share/foreman/app/services/structured_fact_importer.rb 9 >>> normalize_recurse 60 >>> /usr/share/foreman/app/models/host/base.rb 237 >>> primary_interface 54 >>> /usr/share/foreman/app/services/classification/base.rb 167 >>> update_generic_matcher 54 >>> /usr/share/foreman/app/services/classification/base.rb 72 >>> value_of_key 54 >>> /usr/share/foreman/app/services/classification/base.rb 73 >>> yield 44 >>> >>> >>> >>> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal wrote: >>>> >>>> Two or three minutes is just fine, counters do reset after 5 minutes >>>> so not longer. >>>> >>>> You can also add "rails calls" output as well, that might be interesting >>>> too. >>>> >>>> LZ >>>> >>>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <erez...@gmail.com> wrote: >>>> > No problem, for how long you think i should have it running to give >>>> > valuable >>>> > data to work with? >>>> > I am running only on objects-total and objects (as you mentioned) >>>> > >>>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas Zapletal wrote: >>>> >> >>>> >> Can you guys try foreman-tracer utility (SystemTap based, will only >>>> >> work on CentOS 7 or higher or ST-enabled kernels) on your production >>>> >> instance? No changes required in Foreman, setup is quite easy: >>>> >> >>>> >> https://github.com/lzap/foreman-tracer >>>> >> >>>> >> Interesting statistics would be >>>> >> >>>> >> foreman-tracer rails objects-total >>>> >> >>>> >> and >>>> >> >>>> >> foreman-tracer rails objects >>>> >> >>>> >> It's like "top" utility experience, pastebin the bottlenecks please. >>>> >> >>>> >> Thanks >>>> >> >>>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum <erez...@gmail.com> >>>> >> wrote: >>>> >> > I have just upgraded a foreman deployment to 1.13.1 from 1.12.1 and >>>> >> > experience the same performance degradation on the foreman web app, >>>> >> > ruby >>>> >> > processes taking a lot of memory, no new plugins were introduced. >>>> >> > Same as what was written above, I suddenly see ruby instances using >>>> >> > of >>>> >> > up to >>>> >> > 4GB RAM/instance as of before i never saw this happens. >>>> >> > >>>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad wrote: >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh <guywa...@gmail.com> >>>> >> >> wrote: >>>> >> >>> >>>> >> >>> Hi all, >>>> >> >>> >>>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 and are >>>> >> >>> finding >>>> >> >>> that Foreman's ENC/facts/reports components are requiring a lot >>>> >> >>> more >>>> >> >>> resources than the old version. >>>> >> >>> >>>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them checking >>>> >> >>> in >>>> >> >>> with >>>> >> >>> external puppet masters every 30 minutes. Foreman runs under >>>> >> >>> passenger >>>> >> >>> and >>>> >> >>> has a mysql database. >>>> >> >>> >>>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server handling the >>>> >> >>> web >>>> >> >>> UI, >>>> >> >>> database and ENC/facts/reports. On 1.12.3, we have expanded to 4 >>>> >> >>> x 8 >>>> >> >>> vCPU, >>>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and 1 x 8 >>>> >> >>> vCPU, >>>> >> >>> 32GB >>>> >> >>> RAM Foreman server handling the web UI and database. We are >>>> >> >>> currently >>>> >> >>> restarting apache on the ENC/fact/report servers every 20 minutes >>>> >> >>> to >>>> >> >>> stop >>>> >> >>> them running out of memory (if we tune passenger more >>>> >> >>> aggressively, >>>> >> >>> the >>>> >> >>> request queue fills up). >>>> >> >>> >>>> >> >>> I noticed a couple of other posts in the last few days regarding >>>> >> >>> performance. Is anyone running a similarly-sized installation on >>>> >> >>> version >>>> >> >>> 1.12/1.13? If so, what performance are you getting? Any tips for >>>> >> >>> tuning >>>> >> >>> passenger or other components? >>>> >> >> >>>> >> >> >>>> >> >> Are you using a new plugins? or just plain foreman? any >>>> >> >> information you >>>> >> >> could share that would provide some light on where things are >>>> >> >> slow? >>>> >> >> (e.g. >>>> >> >> slow operations, etc). >>>> >> >> >>>> >> >> thanks, >>>> >> >> Ohad >>>> >> >>> >>>> >> >>> >>>> >> >>> Regards >>>> >> >>> Guy. >>>> >> >>> >>>> >> >>> -- >>>> >> >>> You received this message because you are subscribed to the >>>> >> >>> Google >>>> >> >>> Groups >>>> >> >>> "Foreman users" group. >>>> >> >>> To unsubscribe from this group and stop receiving emails from it, >>>> >> >>> send >>>> >> >>> an >>>> >> >>> email to foreman-user...@googlegroups.com. >>>> >> >>> To post to this group, send email to forema...@googlegroups.com. >>>> >> >>> Visit this group at >>>> >> >>> https://groups.google.com/group/foreman-users. >>>> >> >>> For more options, visit https://groups.google.com/d/optout. >>>> >> >> >>>> >> >> >>>> >> > -- >>>> >> > You received this message because you are subscribed to the Google >>>> >> > Groups >>>> >> > "Foreman users" group. >>>> >> > To unsubscribe from this group and stop receiving emails from it, >>>> >> > send >>>> >> > an >>>> >> > email to foreman-user...@googlegroups.com. >>>> >> > To post to this group, send email to forema...@googlegroups.com. >>>> >> > Visit this group at https://groups.google.com/group/foreman-users. >>>> >> > For more options, visit https://groups.google.com/d/optout. >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Later, >>>> >> Lukas @lzap Zapletal >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> > Groups >>>> > "Foreman users" group. >>>> > To unsubscribe from this group and stop receiving emails from it, send >>>> > an >>>> > email to foreman-user...@googlegroups.com. >>>> > To post to this group, send email to forema...@googlegroups.com. >>>> > Visit this group at https://groups.google.com/group/foreman-users. >>>> > For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>>> -- >>>> Later, >>>> Lukas @lzap Zapletal > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscr...@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscr...@googlegroups.com. To post to this group, send email to foreman-users@googlegroups.com. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
