That's expected, the tracer is topped to prevent eating lot of memory, so it rather crash than consume more. I have slightly increased the limit, can you update the script and run once again, now for the last time:
rails objects-total app 100 and rails calls all 100 Please set minimum font size for your terminal to fit 100 rows. I am just curious if we can spot the Model that is causing this. But I suspect this is the similar issue that someone else just had (see replies from today in the list). Hosts were created with too many NICs. The suggestion was: "See the 'ignore interfaces with matching identifier' option under Settings -> Provisioning." Then you need to delete all the NICs from the database to get things rolling again. LZ On Wed, Nov 2, 2016 at 2:50 PM, Erez Zarum <erezza...@gmail.com> wrote: > Hey, > I think you nailed it :) > > rails calls all probe failed at some point though. > > rails calls all > > FILENAME LINE METHOD CALLS > gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb > 260 hash 86006 > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb > 171 association_instance_get 76602 > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb > 157 association 76602 > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb > 160 nil? 76602 > are/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/association.rb > 63 loaded! 76422 > e/gems/gems/activesupport-4.2.5.1/lib/active_support/core_ext/class/attribute.rb > 86 _reflections 60166 > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb > 115 _reflect_on_association 60165 > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb > 116 to_s 60164 > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb > 889 name 60162 > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb > 889 options 60086 > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb > 686 through_reflection 60065 > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb > 889 active_record 60065 > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb > 442 belongs_to? 59980 > /gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb > 67 stale_state 59979 > /gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb > 6 through_reflection 59979 > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb > 592 collection? 59894 > ems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/primary_key.rb > 72 primary_key 59802 > activerecord-4.2.5.1/lib/active_record/associations/join_dependency/join_part.rb > 18 primary_key 59712 > gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb > 31 column_alias 59711 > gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb > 254 nil? 59710 > gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb > 32 hash 59710 > /share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb > 39 [] 58136 > root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb > 9 [] 58136 > r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb > 35 value 58078 > root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb > 30 fetch_value 58042 > WARNING: Number of errors: 0, skipped probes: 101 > WARNING: /usr/bin/staprun exited with status: 1 > Pass 5: run failed. [man error::pass5] > > rails objects all > FILENAME > LINE OBJECT COUNT > cord-4.2.5.1/lib/active_record/connection_adapters/postgresql/utils.rb > 72 rd::ConnectionAdapters::PostgreSQL::Name 7037 > pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/table.rb > 100 Arel::Attributes::Attribute 4317 > /opt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel.rb > 27 Arel::Nodes::SqlLiteral 4174 > t/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/result.rb > 110 String 3668 > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/table_alias.rb > 9 Arel::Attributes::Attribute 3095 > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb > 4 Arel::Nodes::As 3091 > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb > 4 Arel::Nodes::SqlLiteral 3091 > activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb > 126 iations::JoinDependency::Aliases::Column 3091 > usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation.rb > 35 Array 2753 > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb > 10 Taxonomy::ActiveRecord_Relation 2655 > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb > 44 ActiveRecord::Relation::Merger 2589 > hare/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/route.rb > 84 Hash 1825 > s/activerecord-4.2.5.1/lib/active_record/associations/alias_tracker.rb > 63 Arel::Table 1666 > clo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/predications.rb > 16 Arel::Nodes::Equality 1436 > ems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb > 106 Taxonomy::ActiveRecord_Relation 1180 > sr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb > 5 ActiveRecord::Attribute::FromDatabase 1180 > pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes.rb > 88 Arel::Nodes::Casted 1152 > o-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/select_manager.rb > 11 Arel::Nodes::SelectStatement 1088 > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_core.rb > 9 Arel::Nodes::JoinSource 1088 > root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_statement.rb > 7 Arel::Nodes::SelectCore 1088 > /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb > 864 Arel::SelectManager 1088 > /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb > 81 Hash 992 > hare/gems/gems/activerecord-4.2.5.1/lib/active_record/scoping/named.rb > 26 Taxonomy::ActiveRecord_Relation 885 > /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb > 944 Arel::Nodes::And 880 > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb > 10 Hostgroup::ActiveRecord_Relation 869 > > rails objects-total all > OBJECT COUNT > Hash 11318 > Arel::Nodes::SqlLiteral 7566 > Arel::Attributes::Attribute 7453 > ActiveRecord::ConnectionAdapters::PostgreSQL::Name 7037 > Array 6752 > Taxonomy::ActiveRecord_Relation 4720 > String 4651 > ActiveRecord::Associations::JoinDependency::Aliases::Column 3091 > Arel::Nodes::As 3091 > ActiveRecord::Relation::Merger 2589 > Arel::Table 1670 > Time 1597 > Arel::Nodes::Equality 1436 > Hostgroup::ActiveRecord_Relation 1283 > ActiveRecord::Attribute::FromDatabase 1180 > Arel::Nodes::Casted 1152 > Arel::SelectManager 1088 > Arel::Nodes::JoinSource 1088 > Arel::Nodes::SelectCore 1088 > Arel::Nodes::SelectStatement 1088 > TaxableTaxonomy::ActiveRecord_Relation 1065 > Arel::Nodes::And 1052 > Set 992 > VariableLookupKey::ActiveRecord_Relation 984 > ActiveRecord::Result 949 > > > > On Wednesday, November 2, 2016 at 3:17:09 PM UTC+2, Lukas Zapletal wrote: >> >> Were you running the "foreman-trace rails calls" utility when you >> searched the problematic "puppet" query? Because the output looks like >> sane output, I can see one hundred of NIC models allocated, but this >> should not take 6 Gbs at all. >> >> I've slightly enhanced the foreman-tracer, can you also try to run >> this while doing the problematic query: >> >> foreman-tracer rails objects-total all >> >> foreman-tracer rails objects all >> >> foreman-tracer rails calls all >> >> We must be allocating some number of objects somewhere, this should reveal >> them. >> >> LZ >> >> On Wed, Nov 2, 2016 at 11:55 AM, Erez Zarum <erez...@gmail.com> wrote: >> > This is the problem, it's seems like running a query without any field >> > is >> > causing this problem, it became more aggressive now probably because i >> > am >> > running it under the "Classes" dashboard which now have much much more >> > objects than before. >> > >> > Running a query without a specific field: "puppet", memory usage from 1G >> > to >> > 5G, a ruby instance takes 100% CPU for about a minute. >> > Running a query with a specific field: "name = puppet", memory usage >> > remains >> > quite the same, small spike in CPU, returns immediately (less than a >> > second). >> > >> > >> > On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum wrote: >> >> >> >> I have the "theforeman/puppet" module, when i search under the >> >> "Classes" >> >> dashboard the word "puppet" it's seems to cause a very high CPU and >> >> memory >> >> starts to increase up to 6GB from almost 1GB. >> >> >> >> On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum wrote: >> >>> >> >>> rails objects-total >> >>> OBJECT COUNT >> >>> ERB 826 >> >>> Nic::Managed 104 >> >>> Foreman::Parameters::Caster 38 >> >>> IPAddr 24 >> >>> ConfigReportStatusCalculator 24 >> >>> ActiveSupport::HashWithIndifferentAccess 18 >> >>> SafeRender 10 >> >>> Range 9 >> >>> ActionView::OutputBuffer 8 >> >>> Classification::GlobalParam 6 >> >>> ActionDispatch::Request 6 >> >>> Foreman::Parameters::Validator 6 >> >>> OpenSSL::ASN1::ASN1Data 6 >> >>> OpenSSL::ASN1::Sequence 6 >> >>> Authorizer 5 >> >>> CertificateExtract 3 >> >>> OpenSSL::ASN1::ObjectId 3 >> >>> OpenSSL::ASN1::OctetString 3 >> >>> OpenSSL::X509::Certificate 3 >> >>> Classification::ClassParam 2 >> >>> ActiveModel::Name 2 >> >>> PuppetFactParser 1 >> >>> HostStatus::Global 1 >> >>> PuppetFactImporter 1 >> >>> >> >>> rails objects >> >>> FILENAME >> >>> LINE OBJECT COUNT >> >>> /usr/share/foreman/app/services/safe_render.rb >> >>> 39 ERB 826 >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb >> >>> 10 Nic::Managed 96 >> >>> /usr/share/foreman/app/services/classification/base.rb >> >>> 151 Foreman::Parameters::Caster 38 >> >>> /usr/share/foreman/app/models/config_report.rb >> >>> 88 ConfigReportStatusCalculator 18 >> >>> /usr/share/foreman/app/models/host/base.rb >> >>> 390 ActiveSupport::HashWithIndifferentAccess 11 >> >>> /usr/share/foreman/app/models/concerns/encryptable.rb >> >>> 50 Range 8 >> >>> /usr/share/foreman/app/models/host/base.rb >> >>> 78 Nic::Managed 8 >> >>> /usr/share/foreman/lib/net/validations.rb >> >>> 31 IPAddr 8 >> >>> /usr/share/foreman/lib/net/validations.rb >> >>> 18 IPAddr 8 >> >>> /usr/share/foreman/app/services/classification/base.rb >> >>> 8 SafeRender 8 >> >>> /usr/share/foreman/lib/net/validations.rb >> >>> 102 IPAddr 8 >> >>> /usr/share/foreman/app/models/lookup_value.rb >> >>> 51 Foreman::Parameters::Validator 6 >> >>> /usr/share/foreman/lib/middleware/tagged_logging.rb >> >>> 10 ActionDispatch::Request 6 >> >>> /usr/share/foreman/app/models/host/managed.rb >> >>> 934 Classification::GlobalParam 6 >> >>> /usr/share/foreman/app/models/host_status/configuration_status.rb >> >>> 112 ConfigReportStatusCalculator 6 >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >>> 17 OpenSSL::ASN1::ASN1Data 6 >> >>> /usr/share/foreman/app/helpers/application_helper.rb >> >>> 104 Authorizer 5 >> >>> /usr/share/foreman/app/services/authorizer_cache.rb >> >>> 3 ActiveSupport::HashWithIndifferentAccess 5 >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >>> 5 OpenSSL::X509::Certificate 3 >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >>> 16 OpenSSL::ASN1::OctetString 3 >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >>> 16 OpenSSL::ASN1::Sequence 3 >> >>> >> >>> >> >>> /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb >> >>> 59 CertificateExtract 3 >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >>> 16 OpenSSL::ASN1::ObjectId 3 >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >>> 17 OpenSSL::ASN1::Sequence 3 >> >>> /usr/share/foreman/app/models/host/managed.rb >> >>> 420 SafeRender 2 >> >>> >> >>> rails calls >> >>> FILENAME LINE >> >>> METHOD CALLS >> >>> /usr/share/foreman/app/models/setting.rb 218 >> >>> cache 2120 >> >>> /usr/share/foreman/app/models/setting.rb 72 >> >>> [] 2086 >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 55 >> >>> exportable_attributes 2054 >> >>> /usr/share/foreman/app/services/safe_render.rb 12 >> >>> parse 1138 >> >>> /usr/share/foreman/app/services/safe_render.rb 31 >> >>> parse_string 826 >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 33 >> >>> export_attr 810 >> >>> /usr/share/foreman/app/models/fact_value.rb 7 >> >>> name 509 >> >>> /usr/share/foreman/app/models/user.rb 137 >> >>> admin? 180 >> >>> /usr/share/foreman/app/models/taxonomy.rb 100 >> >>> expand 180 >> >>> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92 >> >>> to_s 112 >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9 >> >>> setup_object_clone 104 >> >>> /usr/share/foreman/app/models/nic/managed.rb 45 >> >>> hostname 96 >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 34 >> >>> with_taxonomy_scope 84 >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 115 >> >>> scope_by_taxable_ids 84 >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 75 >> >>> taxable_ids 84 >> >>> /usr/share/foreman/app/models/host/base.rb 441 >> >>> get_interface_by_flag 74 >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 35 >> >>> call 66 >> >>> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12 >> >>> dns_ready? 64 >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 131 >> >>> set_current_taxonomy 62 >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 14 >> >>> to_export 62 >> >>> /usr/share/foreman/app/services/structured_fact_importer.rb 9 >> >>> normalize_recurse 60 >> >>> /usr/share/foreman/app/models/host/base.rb 237 >> >>> primary_interface 54 >> >>> /usr/share/foreman/app/services/classification/base.rb 167 >> >>> update_generic_matcher 54 >> >>> /usr/share/foreman/app/services/classification/base.rb 72 >> >>> value_of_key 54 >> >>> /usr/share/foreman/app/services/classification/base.rb 73 >> >>> yield 44 >> >>> >> >>> >> >>> >> >>> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal >> >>> wrote: >> >>>> >> >>>> Two or three minutes is just fine, counters do reset after 5 minutes >> >>>> so not longer. >> >>>> >> >>>> You can also add "rails calls" output as well, that might be >> >>>> interesting >> >>>> too. >> >>>> >> >>>> LZ >> >>>> >> >>>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <erez...@gmail.com> >> >>>> wrote: >> >>>> > No problem, for how long you think i should have it running to give >> >>>> > valuable >> >>>> > data to work with? >> >>>> > I am running only on objects-total and objects (as you mentioned) >> >>>> > >> >>>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas Zapletal >> >>>> > wrote: >> >>>> >> >> >>>> >> Can you guys try foreman-tracer utility (SystemTap based, will >> >>>> >> only >> >>>> >> work on CentOS 7 or higher or ST-enabled kernels) on your >> >>>> >> production >> >>>> >> instance? No changes required in Foreman, setup is quite easy: >> >>>> >> >> >>>> >> https://github.com/lzap/foreman-tracer >> >>>> >> >> >>>> >> Interesting statistics would be >> >>>> >> >> >>>> >> foreman-tracer rails objects-total >> >>>> >> >> >>>> >> and >> >>>> >> >> >>>> >> foreman-tracer rails objects >> >>>> >> >> >>>> >> It's like "top" utility experience, pastebin the bottlenecks >> >>>> >> please. >> >>>> >> >> >>>> >> Thanks >> >>>> >> >> >>>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum <erez...@gmail.com> >> >>>> >> wrote: >> >>>> >> > I have just upgraded a foreman deployment to 1.13.1 from 1.12.1 >> >>>> >> > and >> >>>> >> > experience the same performance degradation on the foreman web >> >>>> >> > app, >> >>>> >> > ruby >> >>>> >> > processes taking a lot of memory, no new plugins were >> >>>> >> > introduced. >> >>>> >> > Same as what was written above, I suddenly see ruby instances >> >>>> >> > using >> >>>> >> > of >> >>>> >> > up to >> >>>> >> > 4GB RAM/instance as of before i never saw this happens. >> >>>> >> > >> >>>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad wrote: >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh <guywa...@gmail.com> >> >>>> >> >> wrote: >> >>>> >> >>> >> >>>> >> >>> Hi all, >> >>>> >> >>> >> >>>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 and are >> >>>> >> >>> finding >> >>>> >> >>> that Foreman's ENC/facts/reports components are requiring a >> >>>> >> >>> lot >> >>>> >> >>> more >> >>>> >> >>> resources than the old version. >> >>>> >> >>> >> >>>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them >> >>>> >> >>> checking >> >>>> >> >>> in >> >>>> >> >>> with >> >>>> >> >>> external puppet masters every 30 minutes. Foreman runs under >> >>>> >> >>> passenger >> >>>> >> >>> and >> >>>> >> >>> has a mysql database. >> >>>> >> >>> >> >>>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server handling >> >>>> >> >>> the >> >>>> >> >>> web >> >>>> >> >>> UI, >> >>>> >> >>> database and ENC/facts/reports. On 1.12.3, we have expanded to >> >>>> >> >>> 4 >> >>>> >> >>> x 8 >> >>>> >> >>> vCPU, >> >>>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and 1 x 8 >> >>>> >> >>> vCPU, >> >>>> >> >>> 32GB >> >>>> >> >>> RAM Foreman server handling the web UI and database. We are >> >>>> >> >>> currently >> >>>> >> >>> restarting apache on the ENC/fact/report servers every 20 >> >>>> >> >>> minutes >> >>>> >> >>> to >> >>>> >> >>> stop >> >>>> >> >>> them running out of memory (if we tune passenger more >> >>>> >> >>> aggressively, >> >>>> >> >>> the >> >>>> >> >>> request queue fills up). >> >>>> >> >>> >> >>>> >> >>> I noticed a couple of other posts in the last few days >> >>>> >> >>> regarding >> >>>> >> >>> performance. Is anyone running a similarly-sized installation >> >>>> >> >>> on >> >>>> >> >>> version >> >>>> >> >>> 1.12/1.13? If so, what performance are you getting? Any tips >> >>>> >> >>> for >> >>>> >> >>> tuning >> >>>> >> >>> passenger or other components? >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> Are you using a new plugins? or just plain foreman? any >> >>>> >> >> information you >> >>>> >> >> could share that would provide some light on where things are >> >>>> >> >> slow? >> >>>> >> >> (e.g. >> >>>> >> >> slow operations, etc). >> >>>> >> >> >> >>>> >> >> thanks, >> >>>> >> >> Ohad >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> Regards >> >>>> >> >>> Guy. >> >>>> >> >>> >> >>>> >> >>> -- >> >>>> >> >>> You received this message because you are subscribed to the >> >>>> >> >>> Google >> >>>> >> >>> Groups >> >>>> >> >>> "Foreman users" group. >> >>>> >> >>> To unsubscribe from this group and stop receiving emails from >> >>>> >> >>> it, >> >>>> >> >>> send >> >>>> >> >>> an >> >>>> >> >>> email to foreman-user...@googlegroups.com. >> >>>> >> >>> To post to this group, send email to >> >>>> >> >>> forema...@googlegroups.com. >> >>>> >> >>> Visit this group at >> >>>> >> >>> https://groups.google.com/group/foreman-users. >> >>>> >> >>> For more options, visit https://groups.google.com/d/optout. >> >>>> >> >> >> >>>> >> >> >> >>>> >> > -- >> >>>> >> > You received this message because you are subscribed to the >> >>>> >> > Google >> >>>> >> > Groups >> >>>> >> > "Foreman users" group. >> >>>> >> > To unsubscribe from this group and stop receiving emails from >> >>>> >> > it, >> >>>> >> > send >> >>>> >> > an >> >>>> >> > email to foreman-user...@googlegroups.com. >> >>>> >> > To post to this group, send email to forema...@googlegroups.com. >> >>>> >> > Visit this group at >> >>>> >> > https://groups.google.com/group/foreman-users. >> >>>> >> > For more options, visit https://groups.google.com/d/optout. >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> -- >> >>>> >> Later, >> >>>> >> Lukas @lzap Zapletal >> >>>> > >> >>>> > -- >> >>>> > You received this message because you are subscribed to the Google >> >>>> > Groups >> >>>> > "Foreman users" group. >> >>>> > To unsubscribe from this group and stop receiving emails from it, >> >>>> > send >> >>>> > an >> >>>> > email to foreman-user...@googlegroups.com. >> >>>> > To post to this group, send email to forema...@googlegroups.com. >> >>>> > Visit this group at https://groups.google.com/group/foreman-users. >> >>>> > For more options, visit https://groups.google.com/d/optout. >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Later, >> >>>> Lukas @lzap Zapletal >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Foreman users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to foreman-user...@googlegroups.com. >> > To post to this group, send email to forema...@googlegroups.com. >> > Visit this group at https://groups.google.com/group/foreman-users. >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Later, >> Lukas @lzap Zapletal > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscr...@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscr...@googlegroups.com. To post to this group, send email to foreman-users@googlegroups.com. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
