[foreman-dev] Notifications / monitoring for infra
Lukas asked me to raise this after Koji ran out of space, so ... :) Currently there are two forms of notification from our infra. Firstly there is the private infra Google Group, but no-one seems to know who maintains is (Dominic?). Secondly, there is a list of email addresses written by Puppet to /etc/aliases (current it's me, Ohad and Michael). While notifications from Koji would have at least reached more people if it was in Puppet (which we discussed briefly, but no one had time for), I think it makes sense to think about what we want to do more generally about this stuff. My *long term* preference would be proper monitoring - something like Icinga or Zabbix, properly deployed with the appropriate checks, and the ability to alert the right people if a check fails. That's the right thing to do, and is flexibile in who gets a particular alert. In the short term, though, we could use either *another* mailing list (since the existing Google Group seems lost) or a private infra group on Discourse (I tested it briefly, you can email in to a group and keep the messages private to the group members). Either of these would work in the short term. We could also use an alias on the root DNS (in...@theforeman.org or something) but I don't think Ohad can set up multiple forwards for that very easily... and it's harder to manage when the list of forwards changes (only Ohad can do it). Thoughts? I lean to a temporary solution on Discourse (but not strongly, a mailing list is fine here), and proper monitoring to be added to Eric's infra roadmap for later on. Greg -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] RFC: File audit for template rendering
I like this. I agree the renderred template should not be stored along audit records we have today, it does not represent any data change. Also it would be a lot of data in our db and would make searching of other audit records hard. Having long kickstart files in log files is neither a good idea. One small comment, Foreman must remain operational if directory can't be written to and a directory must be configurable (developer setups). The only downside is related to HA Foreman setups. Users must share the dir among all Foreman instances. We should probably limit this to provisioning templates and partition tables only in terms of templates rendering. Job templates can grow rapidly, thousands of files per one invocation. The job template result is different for each host. Btw this would be also great for host classification auditing. That answers the question, what parameter values were available for the host when ENC/info method was called during provisioning. It'd also be great if foreman-debug would be gathering this, but I suppose that's already on your radar. -- Marek On December 5, 2017 6:02:14 PM Lukas Zapletal wrote: Hello, Foreman does have a nice audit trail for many operations, what's missing is ability to find how a template (e.g. kickstart) was rendered. Storing whole template text in audit table is probably not the best thing to do, production.log is also not a good fit, so here is a proposal. I want to create a small API called File audit (*) with ability to store arbitrary files into /var/log/foreman/file_audit/id/text-timestamp where "id" is record id (or multiple ids concatenated with dots), "text" is arbitrary alphanum text and "timestamp" is unix epoch number. Example: /var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603 That API will be used to store contents of all templates rendered, so users can easily go "back in time" and see how templates are being rendered. The directory would be root-only reading and files will be created with restricted permissions (foreman user rw only). On system with SELinux, security would be more tightened allowing writing only to foreman_t domain and reading to nobody. For the example above, this would mean: 1.33.7/my.host.lan-pxelinux-1512492603 1 - file audit type (static list, 1 stands for "template audit entry") 33 - host id 7 - template id my.host.lan-pxelinux - extra data so users can work and search from command line 1512492603 - UNIX epoch timestamp Everytime new record is added, a log entry is created into production.log containing file path. By default, there will be a cron job deleting all files older than one month. In documentation, we will ask users to rsync the directory to different location if long-term archival is needed. This API could be used for other audit logging, for example when user uploads a manifest ZIP file in katello or new version of RPM/Puppet file. This will be the first step to improve audit around templates, later on we can create a plugin showing the data in audit/host pages if needed. But in the first phase, administrators could easily search/grep/diff those files when necessary. (*) if you have a better name, please do propose -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] RFC: File audit for template rendering
I don't think, this is a good idea. Storing files on the local filesystem is generally consideres an anti-pattern imho. Both for web applications and expecially in a container world. In a clustered setup, you'd have to attach some kind of shared-storage to the foreman hosts to make this work. Possibly via NFS or something. This is meant to cause trouble. I generally advise not to have any state on an application server. I think, this can and should be done with a database. I agree, that the audit model might not be the best place for this. But a database definitely is. Just my 2 cents. - Timo Am 05.12.17 um 18:01 schrieb Lukas Zapletal: Hello, Foreman does have a nice audit trail for many operations, what's missing is ability to find how a template (e.g. kickstart) was rendered. Storing whole template text in audit table is probably not the best thing to do, production.log is also not a good fit, so here is a proposal. I want to create a small API called File audit (*) with ability to store arbitrary files into /var/log/foreman/file_audit/id/text-timestamp where "id" is record id (or multiple ids concatenated with dots), "text" is arbitrary alphanum text and "timestamp" is unix epoch number. Example: /var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603 That API will be used to store contents of all templates rendered, so users can easily go "back in time" and see how templates are being rendered. The directory would be root-only reading and files will be created with restricted permissions (foreman user rw only). On system with SELinux, security would be more tightened allowing writing only to foreman_t domain and reading to nobody. For the example above, this would mean: 1.33.7/my.host.lan-pxelinux-1512492603 1 - file audit type (static list, 1 stands for "template audit entry") 33 - host id 7 - template id my.host.lan-pxelinux - extra data so users can work and search from command line 1512492603 - UNIX epoch timestamp Everytime new record is added, a log entry is created into production.log containing file path. By default, there will be a cron job deleting all files older than one month. In documentation, we will ask users to rsync the directory to different location if long-term archival is needed. This API could be used for other audit logging, for example when user uploads a manifest ZIP file in katello or new version of RPM/Puppet file. This will be the first step to improve audit around templates, later on we can create a plugin showing the data in audit/host pages if needed. But in the first phase, administrators could easily search/grep/diff those files when necessary. (*) if you have a better name, please do propose -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[foreman-dev] RFC: File audit for template rendering
Hello, Foreman does have a nice audit trail for many operations, what's missing is ability to find how a template (e.g. kickstart) was rendered. Storing whole template text in audit table is probably not the best thing to do, production.log is also not a good fit, so here is a proposal. I want to create a small API called File audit (*) with ability to store arbitrary files into /var/log/foreman/file_audit/id/text-timestamp where "id" is record id (or multiple ids concatenated with dots), "text" is arbitrary alphanum text and "timestamp" is unix epoch number. Example: /var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603 That API will be used to store contents of all templates rendered, so users can easily go "back in time" and see how templates are being rendered. The directory would be root-only reading and files will be created with restricted permissions (foreman user rw only). On system with SELinux, security would be more tightened allowing writing only to foreman_t domain and reading to nobody. For the example above, this would mean: 1.33.7/my.host.lan-pxelinux-1512492603 1 - file audit type (static list, 1 stands for "template audit entry") 33 - host id 7 - template id my.host.lan-pxelinux - extra data so users can work and search from command line 1512492603 - UNIX epoch timestamp Everytime new record is added, a log entry is created into production.log containing file path. By default, there will be a cron job deleting all files older than one month. In documentation, we will ask users to rsync the directory to different location if long-term archival is needed. This API could be used for other audit logging, for example when user uploads a manifest ZIP file in katello or new version of RPM/Puppet file. This will be the first step to improve audit around templates, later on we can create a plugin showing the data in audit/host pages if needed. But in the first phase, administrators could easily search/grep/diff those files when necessary. (*) if you have a better name, please do propose -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[foreman-dev] Speed up Foreman development booting by 50%
I just filed this PR which speeds booting up quite a lot, from 10s to 6s on my system: https://github.com/theforeman/foreman/pull/5045 Feel free to test, I haven't tested with Katello but it should give even bigger boost with it. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Firefox 57 and noVNC
This is Sat 6.3 install (Katello 3.4), no changes here:
# Websockets
:websockets_encrypt: on
:websockets_ssl_key: /etc/pki/katello/private/katello-apache.key
:websockets_ssl_cert: /etc/pki/katello/certs/katello-apache.crt
# SSL-settings
:ssl_certificate: /etc/foreman/client_cert.pem
:ssl_ca_file: /etc/foreman/proxy_ca.pem
:ssl_priv_key: /etc/foreman/client_key.pem
On Tue, Dec 5, 2017 at 12:06 PM, Ewoud Kohl van Wijngaarden
wrote:
> On Tue, Dec 05, 2017 at 09:43:20AM +0100, Lukas Zapletal wrote:
>>
>> I was trying noVNC the other day on FF 57. Imported katello CA (it was
>> a katello intance), but it didn't work until I flipped flag
>> "network.websocket.allowInsecureFromHTTPS" on and then it worked.
>>
>> I remember that Firefox always worked out of box - when server CA was
>> imported, it worked like charm. In the documentation we have "FF might
>> need to turn on this flag". Anyone knows under which circumstances we
>> need to flip this on?
>>
>> Any luck setting up FF 57 without any hacking with noVNC?
>>
>> https://theforeman.org/manuals/1.15/index.html#7.1NoVNC
>
>
> There are settings (websockets_ssl_{key,cert}) to choose which certificate
> is presented by the VNC websockets proxy. Are you sure the correct
> certificates are used by it?
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
Later,
Lukas @lzap Zapletal
--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Firefox 57 and noVNC
On Tue, Dec 05, 2017 at 09:43:20AM +0100, Lukas Zapletal wrote:
I was trying noVNC the other day on FF 57. Imported katello CA (it was
a katello intance), but it didn't work until I flipped flag
"network.websocket.allowInsecureFromHTTPS" on and then it worked.
I remember that Firefox always worked out of box - when server CA was
imported, it worked like charm. In the documentation we have "FF might
need to turn on this flag". Anyone knows under which circumstances we
need to flip this on?
Any luck setting up FF 57 without any hacking with noVNC?
https://theforeman.org/manuals/1.15/index.html#7.1NoVNC
There are settings (websockets_ssl_{key,cert}) to choose which
certificate is presented by the VNC websockets proxy. Are you sure the
correct certificates are used by it?
--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Koji Space on /
My 2C: 1) We need to setup foreman-infra mailing list, we had one set by Dominic but I am not sure we still have access to this. Greg, can you set up something for us and add our infra team? Whatever this is, but personally I'd prefer emails :) I had an email about this on Saturday but I haven't read it until today. I am bad in this. 2) By default httpd rotation has delayed compression so on busy day, it can create one or two huge files until this is compressed. Although we can tune logrotate, I've done this: I moved and symlinked the following directories to local ephemeral volume: * /var/cache/mrepo -> /mnt/tmp/cache/mrepo * /var/log/httpd -> /mnt/tmp/log/httpd Restarted httpd making sure there are no open jobs. Everything is fine now, report issues. On Mon, Dec 4, 2017 at 4:25 PM, Eric D Helms wrote: > Today Koji filled up its root partition which is designed to be small. In > part, there was 2.0 GB of httpd logs due to the API intensive use of Koji. > These logs are currently rotated and not compressed in anyway. > > How should we handle this? A few ideas for discussion: > > 1) Modify logrotate configuration to try to compress the logs > 2) Setup a cron job to compress the rotated logs > 3) Rotate the logs less > > We also noticed about 850MB of mrepo cache. > > -- > Eric D. Helms > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[foreman-dev] Firefox 57 and noVNC
Hello, I was trying noVNC the other day on FF 57. Imported katello CA (it was a katello intance), but it didn't work until I flipped flag "network.websocket.allowInsecureFromHTTPS" on and then it worked. I remember that Firefox always worked out of box - when server CA was imported, it worked like charm. In the documentation we have "FF might need to turn on this flag". Anyone knows under which circumstances we need to flip this on? Any luck setting up FF 57 without any hacking with noVNC? https://theforeman.org/manuals/1.15/index.html#7.1NoVNC -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
