Re: Audit firewall changes in RHEL 8
On 2020-12-05 00:45, Smith, Gary R wrote: > Good afternoon, > > I have RHEL 7 systems set up to emit audit records when the firewall rules > with iptables change. I do it with a single audit command: > > -a always,exit -F arch=b64 -S setsockopt -F a2=0x40 -F key=IPTablesChange > > And it works great. I get audit logs like this: > > type=PROCTITLE msg=audit(12/04/2020 11:04:58.840:3334178) : > proctitle=iptables -D INPUT 2 > type=SYSCALL msg=audit(12/04/2020 11:04:58.840:3334178) : arch=x86_64 > syscall=setsockopt success=yes exit=0 a0=0x4 a1=ip a2=IPT_SO_SET_REPLACE > a3=0x1009ca0 items=0 ppid=154754 pid=160855 auid=DrEvil uid=root gid=root > euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 > ses=198995 comm=iptables exe=/usr/sbin/xtables-multi key=IPtablesChange > type=NETFILTER_CFG msg=audit(12/04/2020 11:04:58.840:3334178) : table=filter > family=ipv4 entries=48 > > I want to do the same thing with RHEL 8 and nftables. I tried the same audit > rule but nothing happens. I tried using firewall-cmd to change the rules. The > rules changed, but no audit records. I fat fingered rules using nft but no > audit record. I suspect that I’m not writing the audit rule correctly. I > looked around to see if a2 needed to be something other than 0x040 > (IPT_SO_SET_REPLACE) but I couldn’t find anything. The hooks were missing for nftables and were the subject of the recent upstream patches to address that, covered by issue: https://github.com/linux-audit/audit-kernel/issues/124 The patches in question went into: 2020-08-04 fd76a74d940a Merge tag 'audit-pr-20200803' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit 2020-07-08 68df2ed54487 audit: use the proper gfp flags in the audit_log_nfcfg() calls 2020-06-29 142240398e50 audit: add gfp parameter to audit_log_nfcfg 2020-06-23 8e6cf365e1d5 audit: log nftables configuration change events > Any suggestions on how to do this in RHEL 8 would be appreciated. That is a distro-specific question that should be asked in the appropriate vendor forum, but are expected to be backported. > Gary Smith - RGB -- Richard Guy Briggs Sr. S/W Engineer, Kernel Security, Base Operating Systems Remote, Ottawa, Red Hat Canada IRC: rgb, SunRaycer Voice: +1.647.777.2635, Internal: (81) 32635 -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On 7/2/20 2:42 PM, Paul Moore wrote:
#define AUDIT_FEATURE_BITMAP_BACKLOG_LIMIT 0x0001
#define AUDIT_FEATURE_BITMAP_BACKLOG_WAIT_TIME 0x0002
@@ -348,6 +349,7 @@ enum {
#define AUDIT_FEATURE_BITMAP_SESSIONID_FILTER 0x0010
#define AUDIT_FEATURE_BITMAP_LOST_RESET0x0020
#define AUDIT_FEATURE_BITMAP_FILTER_FS 0x0040
+#define AUDIT_FEATURE_BITMAP_BACKLOG_WAIT_SUM 0x0080
In an effort not to exhaust the feature bitmap too quickly, I've been
restricting it to only those features that would cause breakage with
userspace. I haven't looked closely at Steve's userspace in quite a
while, but I'm guessing it can key off the structure size and doesn't
need this entry in the bitmap, right? Let me rephrase, if userspace
needs to key off anything, it*should* key off the structure size and
not a new flag in the bitmask;)
Also, I'm assuming that older userspace doesn't blow-up if it sees the
larger structure size? That's even more important.
Paul,
This change does seem to the untrained eye to be in line with the
existing FEATURE_BITMAP definitions. I appreciate your intent on not
exhausting the available space, but at some point if that happens is
there any reasonable way to expand? I'm sure you have some thoughts, or
is this "it" as far as features could go (the last available bits)?
Max,
It's a pretty good feature. I agree with your original problem
assessment; this is an area I'm always looking at. I've got questions
I'll post separately as they are not germane to this thread.
As an interested user I'm hoping for a resolution on this, so that the
userspace release can happen, as this seems to be a beneficial change
which I could make use of when available.
Thx,
LCB
--
Lenny Bruzenak
MagitekLTD
--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On Fri, Dec 4, 2020 at 3:41 PM Paul Moore wrote:
> On Thu, Dec 3, 2020 at 9:47 PM Steve Grubb wrote:
> > On Thursday, December 3, 2020 9:16:52 PM EST Paul Moore wrote:
> > > > > > Author: Richard Guy Briggs
> > > > > > AuthorDate: 2014-11-17 15:51:01 -0500
> > > > > > Commit: Paul Moore
> > > > > > CommitDate: 2014-11-17 16:53:51 -0500
> > > > > > ("audit: convert status version to a feature bitmap")
> > > > > > It was introduced specifically to enable distributions to
> selectively
> > > > > > backport features. It was converted away from AUDIT_VERSION.
> > > > > >
> > > > > > There are other ways to detect the presence of
> > > > > > backlog_wait_time_actual
> > > > > > as I mentioned above.
> > > > >
> > > > > Let me be blunt - I honestly don't care what Steve's audit
> userspace
> > > > > does to detect this. I've got my own opinion, but Steve's audit
> > > > > userspace is not my project to manage and I think we've established
> > > > > over the years that Steve and I have very different views on what
> > > > > constitutes good design.
> > > >
> > > > And guessing what might be in buffers of different sizes is good
> design?
> > > > The FEATURE_BITMAP was introduced to get rid of this ambiguity.
> > >
> > > There is just soo much to unpack in your comment Steve, but let me
> > > keep it short ...
> > >
> > > - This is an enterprise distro problem, not an upstream problem. The
> > > problems you are talking about are not a problem for upstream.
> >
> > You may look at it that way. I do not. Audit -userspace is also an
> upstream
> > for a lot of distros and I need to make this painless for them. So,
> while you
> > may think of this being a backport problem for Red Hat to solve, I think
> of
> > this as a generic problem that I'd like to solve for Debian, Suse,
> Ubuntu,
> > Arch, Gentoo, anyone using audit. We both are upstream.
>
> I intentionally said "enterprise Linux distributions", I never singled
> out RH/IBM. Contrary to what RH/IBM marketing may have me believe, I
> don't consider RHEL to be the only "enterprise Linux distribution" :)
>
> Beyond that, while I haven't looked at all of the distros you list
> above, I know a few of them typically only backport fixes, not new
> features. Further, as I mentioned previously in this thread, there is
> a way to backport this feature in a safe manner without using the
> feature bits. Even further, if there wasn't a way to backport
> this feature safely (and let me stress agai that you can backport this
> safely), I would still consider that to be a distro problem and not an
> upstream kernel problem. The upstream kernel is not responsible for
> enabling or supporting arbitrary combinations of patches.
>
> --
> paul moore
> www.paul-moore.com
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
>
>
Hi Steve, Paul,
I'm replying with the Gmail UI since I don't have my Mutt setup handy, so
apologies for any formatting which doesn't align with the mailing list best
practices!
First off, my apologies for being late to the thread, and for submitting
code
to the kernel and user space which aren't playing nicely with each other.
It sounds like there's a decision to be made around whether or not to use
the bitmap feature flags which I probably am probably not in a position to
help decide. However, I'm more than happy to fix my userspace PR so
that it does not rely on the feature flag space using the approach Paul
outlined, in spite of the drawbacks, if that ends up being the decision.
Steve, I understand your preference to rely on the feature bitmap since it
is a more reliable way to determine the availability of a feature than
key size, but if you're open to Paul's recommendations in spite of the
drawbacks, I'll make the changes to my patch as soon as I can to unblock
your work.
Separately, since there is tension between these two approaches
(structure size and bitmap), I wonder if Paul/Steve you would be open
to a third way.
For example, I can imagine adding additional bitmap
spaces (FEATURE_BITMAP_2, FEATURE_BITMAP_3, etc.).
Alternately, I can imagine each feature being assigned a unique u64
ID, and user space programs querying the kernel to see whether a
a particular feature is enabled.
I'm not familiar enough with the kernel to be able to judge how sound
either idea is (or if these have been considered and rejected in the past)
but if you all think a third way is viable, I'd be happy to start a separate
mailing thread to try to thread the competing requirements of the kernel
and userspace, and contribute code if we can find a solution.
Max
--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On Mon, Dec 7, 2020 at 2:43 PM Lenny Bruzenak wrote: > Paul, > > This change does seem to the untrained eye to be in line with the existing > FEATURE_BITMAP definitions. I appreciate your intent on not exhausting the > available space, but at some point if that happens is there any reasonable > way to expand? I'm sure you have some thoughts, or is this "it" as far as > features could go (the last available bits)? I haven't given it a lot of thought, but I'm sure there are ways we could expand the bitmap if/when it is needed. I'm also sure it would be ugly; it almost always is a pain. My hope is that we've changed the API before then so it is a non-issue. -- paul moore www.paul-moore.com -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On Mon, Dec 7, 2020 at 4:13 PM Max Englander wrote: > It sounds like there's a decision to be made around whether or not to use > the bitmap feature flags which I probably am probably not in a position to > help decide. However, I'm more than happy to fix my userspace PR so > that it does not rely on the feature flag space using the approach Paul > outlined, in spite of the drawbacks, if that ends up being the decision. As mentioned several times in the past, I'm not merging a patch which allocates a bitmap entry for this feature. > Separately, since there is tension between these two approaches > (structure size and bitmap), I wonder if Paul/Steve you would be open > to a third way. > > For example, I can imagine adding additional bitmap > spaces (FEATURE_BITMAP_2, FEATURE_BITMAP_3, etc.). > Alternately, I can imagine each feature being assigned a unique u64 > ID, and user space programs querying the kernel to see whether a > a particular feature is enabled. This isn't attractive to me at this point in time. NACK. -- paul moore www.paul-moore.com -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On 2020-12-07 16:13, Max Englander wrote:
> On Fri, Dec 4, 2020 at 3:41 PM Paul Moore wrote:
>
> > On Thu, Dec 3, 2020 at 9:47 PM Steve Grubb wrote:
> > > On Thursday, December 3, 2020 9:16:52 PM EST Paul Moore wrote:
> > > > > > > Author: Richard Guy Briggs
> > > > > > > AuthorDate: 2014-11-17 15:51:01 -0500
> > > > > > > Commit: Paul Moore
> > > > > > > CommitDate: 2014-11-17 16:53:51 -0500
> > > > > > > ("audit: convert status version to a feature bitmap")
> > > > > > > It was introduced specifically to enable distributions to
> > selectively
> > > > > > > backport features. It was converted away from AUDIT_VERSION.
> > > > > > >
> > > > > > > There are other ways to detect the presence of
> > > > > > > backlog_wait_time_actual
> > > > > > > as I mentioned above.
> > > > > >
> > > > > > Let me be blunt - I honestly don't care what Steve's audit
> > userspace
> > > > > > does to detect this. I've got my own opinion, but Steve's audit
> > > > > > userspace is not my project to manage and I think we've established
> > > > > > over the years that Steve and I have very different views on what
> > > > > > constitutes good design.
> > > > >
> > > > > And guessing what might be in buffers of different sizes is good
> > design?
> > > > > The FEATURE_BITMAP was introduced to get rid of this ambiguity.
> > > >
> > > > There is just soo much to unpack in your comment Steve, but let me
> > > > keep it short ...
> > > >
> > > > - This is an enterprise distro problem, not an upstream problem. The
> > > > problems you are talking about are not a problem for upstream.
> > >
> > > You may look at it that way. I do not. Audit -userspace is also an
> > upstream
> > > for a lot of distros and I need to make this painless for them. So,
> > while you
> > > may think of this being a backport problem for Red Hat to solve, I think
> > of
> > > this as a generic problem that I'd like to solve for Debian, Suse,
> > Ubuntu,
> > > Arch, Gentoo, anyone using audit. We both are upstream.
> >
> > I intentionally said "enterprise Linux distributions", I never singled
> > out RH/IBM. Contrary to what RH/IBM marketing may have me believe, I
> > don't consider RHEL to be the only "enterprise Linux distribution" :)
> >
> > Beyond that, while I haven't looked at all of the distros you list
> > above, I know a few of them typically only backport fixes, not new
> > features. Further, as I mentioned previously in this thread, there is
> > a way to backport this feature in a safe manner without using the
> > feature bits. Even further, if there wasn't a way to backport
> > this feature safely (and let me stress agai that you can backport this
> > safely), I would still consider that to be a distro problem and not an
> > upstream kernel problem. The upstream kernel is not responsible for
> > enabling or supporting arbitrary combinations of patches.
> >
> > --
> > paul moore
> > www.paul-moore.com
> >
> > --
> > Linux-audit mailing list
> > Linux-audit@redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-audit
> >
> >
> Hi Steve, Paul,
>
> I'm replying with the Gmail UI since I don't have my Mutt setup handy, so
> apologies for any formatting which doesn't align with the mailing list best
> practices!
>
> First off, my apologies for being late to the thread, and for submitting
> code
> to the kernel and user space which aren't playing nicely with each other.
>
> It sounds like there's a decision to be made around whether or not to use
> the bitmap feature flags which I probably am probably not in a position to
> help decide. However, I'm more than happy to fix my userspace PR so
> that it does not rely on the feature flag space using the approach Paul
> outlined, in spite of the drawbacks, if that ends up being the decision.
>
> Steve, I understand your preference to rely on the feature bitmap since it
> is a more reliable way to determine the availability of a feature than
> key size, but if you're open to Paul's recommendations in spite of the
> drawbacks, I'll make the changes to my patch as soon as I can to unblock
> your work.
>
> Separately, since there is tension between these two approaches
> (structure size and bitmap), I wonder if Paul/Steve you would be open
> to a third way.
Max, Steve: have you looked at my reply in this thread from 2020-12-03 18:10?
There are two solutions in there that should work without relying on the
unreliable test for structure size that work without changing the
currently commited kernel code. Or have I missed something?
> For example, I can imagine adding additional bitmap
> spaces (FEATURE_BITMAP_2, FEATURE_BITMAP_3, etc.).
> Alternately, I can imagine each feature being assigned a unique u64
> ID, and user space programs querying the kernel to see whether a
> a particular feature is enabled.
>
> I'm not familiar enough with the kernel to be able to judge how sound
> either idea is (or if these have been considered and rejected in the past)
> but if you all think a
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On Mon, Dec 7, 2020 at 4:21 PM Richard Guy Briggs wrote:
> On 2020-12-07 16:13, Max Englander wrote:
> > On Fri, Dec 4, 2020 at 3:41 PM Paul Moore wrote:
> >
> > > On Thu, Dec 3, 2020 at 9:47 PM Steve Grubb wrote:
> > > > On Thursday, December 3, 2020 9:16:52 PM EST Paul Moore wrote:
> > > > > > > > Author: Richard Guy Briggs
> > > > > > > > AuthorDate: 2014-11-17 15:51:01 -0500
> > > > > > > > Commit: Paul Moore
> > > > > > > > CommitDate: 2014-11-17 16:53:51 -0500
> > > > > > > > ("audit: convert status version to a feature bitmap")
> > > > > > > > It was introduced specifically to enable distributions to
> > > selectively
> > > > > > > > backport features. It was converted away from AUDIT_VERSION.
> > > > > > > >
> > > > > > > > There are other ways to detect the presence of
> > > > > > > > backlog_wait_time_actual
> > > > > > > > as I mentioned above.
> > > > > > >
> > > > > > > Let me be blunt - I honestly don't care what Steve's audit
> > > userspace
> > > > > > > does to detect this. I've got my own opinion, but Steve's
> audit
> > > > > > > userspace is not my project to manage and I think we've
> established
> > > > > > > over the years that Steve and I have very different views on
> what
> > > > > > > constitutes good design.
> > > > > >
> > > > > > And guessing what might be in buffers of different sizes is good
> > > design?
> > > > > > The FEATURE_BITMAP was introduced to get rid of this ambiguity.
> > > > >
> > > > > There is just soo much to unpack in your comment Steve, but let me
> > > > > keep it short ...
> > > > >
> > > > > - This is an enterprise distro problem, not an upstream problem.
> The
> > > > > problems you are talking about are not a problem for upstream.
> > > >
> > > > You may look at it that way. I do not. Audit -userspace is also an
> > > upstream
> > > > for a lot of distros and I need to make this painless for them. So,
> > > while you
> > > > may think of this being a backport problem for Red Hat to solve, I
> think
> > > of
> > > > this as a generic problem that I'd like to solve for Debian, Suse,
> > > Ubuntu,
> > > > Arch, Gentoo, anyone using audit. We both are upstream.
> > >
> > > I intentionally said "enterprise Linux distributions", I never singled
> > > out RH/IBM. Contrary to what RH/IBM marketing may have me believe, I
> > > don't consider RHEL to be the only "enterprise Linux distribution" :)
> > >
> > > Beyond that, while I haven't looked at all of the distros you list
> > > above, I know a few of them typically only backport fixes, not new
> > > features. Further, as I mentioned previously in this thread, there is
> > > a way to backport this feature in a safe manner without using the
> > > feature bits. Even further, if there wasn't a way to backport
> > > this feature safely (and let me stress agai that you can backport this
> > > safely), I would still consider that to be a distro problem and not an
> > > upstream kernel problem. The upstream kernel is not responsible for
> > > enabling or supporting arbitrary combinations of patches.
> > >
> > > --
> > > paul moore
> > > www.paul-moore.com
> > >
> > > --
> > > Linux-audit mailing list
> > > Linux-audit@redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-audit
> > >
> > >
> > Hi Steve, Paul,
> >
> > I'm replying with the Gmail UI since I don't have my Mutt setup handy, so
> > apologies for any formatting which doesn't align with the mailing list
> best
> > practices!
> >
> > First off, my apologies for being late to the thread, and for submitting
> > code
> > to the kernel and user space which aren't playing nicely with each other.
> >
> > It sounds like there's a decision to be made around whether or not to use
> > the bitmap feature flags which I probably am probably not in a position
> to
> > help decide. However, I'm more than happy to fix my userspace PR so
> > that it does not rely on the feature flag space using the approach Paul
> > outlined, in spite of the drawbacks, if that ends up being the decision.
> >
> > Steve, I understand your preference to rely on the feature bitmap since
> it
> > is a more reliable way to determine the availability of a feature than
> > key size, but if you're open to Paul's recommendations in spite of the
> > drawbacks, I'll make the changes to my patch as soon as I can to unblock
> > your work.
> >
> > Separately, since there is tension between these two approaches
> > (structure size and bitmap), I wonder if Paul/Steve you would be open
> > to a third way.
>
> Max, Steve: have you looked at my reply in this thread from 2020-12-03
> 18:10?
>
> There are two solutions in there that should work without relying on the
> unreliable test for structure size that work without changing the
> currently commited kernel code. Or have I missed something?
>
> > For example, I can imagine adding additional bitmap
> > spaces (FEATURE_BITMAP_2, FEATURE_BITMAP_3, etc.).
> > Alternately, I can imagine each feature being assigned a unique u64
> >
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On Wed, Dec 2, 2020 at 11:33 PM Joe Wulf wrote: > I would like to suggest providing a mechanism where admins can query the > status or state of backlog issues (wait time, sums, etc...). Maybe the > intent is to expand the output of status checking of auditd. > > I believe further clarity is beneficial on the setting of the > 'backlog_wait_sum' (or to whatever the name evolves to) initially. > - How it evolves over time > - What the conditions in the system, or auditing, would change it > - What conditions admins should pay attention to for informational > understanding of status > - What conditions admins should realize exist such that adjustments are > needed >(and suggestions to what those adjustments should be) > - What new guidance will admins have for building adjusting audit.rules > around this > > Consider the scenario where auditing has been 'working fine' for days. > Little to no active admin monitoring. > Events occur to spike the auditing such that backloging of audit records > dramatically increases. > (for some reason) admins now come looking to investigate. > Assuming they do: 'systemctl status auditd' the newly presented 'state' > of the 'backlog_wait_sum' will show some evidence. > Q: Is that just a moment in time? > Q: What information here will give the perspective things are good/ok > 'now', versus some action needs to be taken? > > Maybe that isn't a great scenario, or good questionsit is what occurs > to me at the moment. > > Thank you. > > R, > -Joe Wulf > > > On Wednesday, July 1, 2020, 5:33:14 PM EDT, Max Englander < > max.englan...@gmail.com> wrote: > > > In environments where the preservation of audit events and predictable > > usage of system memory are prioritized, admins may use a combination of > > --backlog_wait_time and -b options at the risk of degraded performance > > resulting from backlog waiting. In some cases, this risk may be > > preferred to lost events or unbounded memory usage. Ideally, this risk > > can be mitigated by making adjustments when backlog waiting is > detected. > > > > However, detection can be diffult using the currently available > metrics. > > For example, an admin attempting to debug degraded performance may > > falsely believe a full backlog indicates backlog waiting. It may turn > > out the backlog frequently fills up but drains quickly. > > > > To make it easier to reliably track degraded performance to backlog > > waiting, this patch makes the following changes: > > > > Add a new field backlog_wait_sum to the audit status reply. Initialize > > this field to zero. Add to this field the total time spent by the > > current task on scheduled timeouts while the backlog limit is exceeded. > > > > Tested on Ubuntu 18.04 using complementary changes to the audit > > userspace: https://github.com/linux-audit/audit-userspace/pull/134. > > > Hi Joe, Not sure I can address all your points above, but the way that we monitor Linux audit internals at my employer is to continuously monitor the audit status response with short evaluation windows. - We compute a rate of change on the lost field, and alert if the there are more than N lost records per second on average - We compute the backlog utilization by computing backlog/backlog_limit, and alert if that goes above 75% at any point in time - If/when we run on a kernel that has backlog_wait_time_actual, we'll monitor on that as well, setting thresholds around where we'd expect growth in this value to result in service degradation. If we get an alert, and it is just a blip that goes away and doesn't come back, we probably won't spend a lot of time investigating. However, if we see that the alert is frequently active across multiple hosts, that will prompt us to investigate. As far as what action we would take, it would depend on the precise values in the audit status reply, as well as other information we had gathered from our system. For example, if we observed elevated values for backlog and backlog_wait_time_actual, we might first investigate other environmental factors such as whether the auditd daemon was crashed or starved for CPU time. If we saw that lost was high but backlog was low that might indicate to us that the rate limit is being exceeded, or that the kernel is out of memory. I agree with you that it would help to expand the metrics reported in audit status. For example, reporting the number of times an audit record was lost due to rate limit being exceeded would help. Not sure how responsive this is to your questions. Hope it helps some. Thanks, Max -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
Re: [PATCH v2] audit: report audit wait metric in audit status reply
Hello Max, On Monday, December 7, 2020 4:28:14 PM EST Max Englander wrote: > Steve, I'm happy to make changes to the userspace PR based on > Richard's suggestions, if that sounds good to you. I'll follow up in > the PR to discuss it more The only issue is new userspace on old kernel. I think if we use both the configure macro in addition to a size check, then it will at least allow forward and backward compatibility. Other metrics would be good. I'd like to see a max_backlog to know if we are wasting memory. It would just record the highwater mark since auditing was enabled. -Steve -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
lost events on boot
Apologies if this has been answered. I searched and found some relevant-looking dialog 2 years ago (on 12/14/2018) that Paul/RGB/Ondrej were discussing, however I do not see the answer. I'm running userspace 2.8.5 , kernel 3.10.0-1160. I have boot parameters "audit=1 ... audit_backlog_limit=8192" . Immediately after boot, I use "auditctl -s and see hundreds (varies, between 119-330) of lost records. So I cleaned out all the audit data, rebooted again and examined the events. They are numbered sequentially 1-515. I counted the events and they match (515). So my questions are these: * Is this "lost" value accurate? * If the numbering doesn't indicate any gaps, what does that tell me? The kernel is supplying the serial number (right?), so is it discarding the events without assigning a serial number? * Do I have something wrong with my kernel boot parameters? I'd have thought that 8k buffers would be enough, and certainly if I only have 515 events, should be. Unless, each record inside the event is adding. I also then counted each record, not just events, and got around 1600, so I'd have thought that even multi-record events would have fit. I guess that depends on the buffer size. Appreciate the help in advance; thanks. LCB -- Lenny Bruzenak MagitekLTD -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
Re: lost events on boot
On 2020-12-07 16:28, Lenny Bruzenak wrote:
> Apologies if this has been answered. I searched and found some
> relevant-looking dialog 2 years ago (on 12/14/2018) that Paul/RGB/Ondrej
> were discussing, however I do not see the answer.
>
> I'm running userspace 2.8.5 , kernel 3.10.0-1160.
>
> I have boot parameters "audit=1 ... audit_backlog_limit=8192" .
>
> Immediately after boot, I use "auditctl -s and see hundreds (varies, between
> 119-330) of lost records.
>
>
> So I cleaned out all the audit data, rebooted again and examined the events.
>
> They are numbered sequentially 1-515. I counted the events and they match
> (515).
>
>
> So my questions are these:
>
> * Is this "lost" value accurate?
Not entirely on that vintage of kernel. It counted a lost message even
if it was later delivered via the audit_skb_hold_queue, IIRC. Paul
re-did the queues to avoid this false report. That change went into
v4.10-rc1:
2016-12-14 c6480207fdf7 ("audit: rework the audit queue handling")
It was too disruptive to backport to the 3.10.0-xxx vintage kernel you
are running.
> * If the numbering doesn't indicate any gaps, what does that tell me?
Messages that went through the hold queue, IIRC.
>The kernel is supplying the serial number (right?), so is it
>discarding the events without assigning a serial number?
Yes, the kernel assigns the serial numbers. Sometimes. Some buffers
never get allocated and therefore no serial number assigned due to full
queues or memory pressure. Other buffers get dropped when queues are
full and there is no choice but to drop a message. This is true before
and after Paul's queue re-write.
> * Do I have something wrong with my kernel boot parameters?
Not likely. From what you have described above it sounds like you have
done what you can.
> I'd have thought that 8k buffers would be enough, and certainly if I only
> have 515 events, should be. Unless, each record inside the event is adding.
If your kernel command line is larger than your lost count and your
serial number when you check it after boot, you should be in good shape.
> I also then counted each record, not just events, and got around 1600, so
> I'd have thought that even multi-record events would have fit. I guess that
> depends on the buffer size.
Good thinking, and you are correct. That backlog limit may need to be
increased for more recent kernels since there are more events caught and
some events have more records.
> Appreciate the help in advance; thanks.
I hope this helps.
> LCB
- RGB
--
Richard Guy Briggs
Sr. S/W Engineer, Kernel Security, Base Operating Systems
Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635
--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On 2020-12-07 18:28, Steve Grubb wrote: > Hello Max, > > On Monday, December 7, 2020 4:28:14 PM EST Max Englander wrote: > > Steve, I'm happy to make changes to the userspace PR based on > > Richard's suggestions, if that sounds good to you. I'll follow up in > > the PR to discuss it more > > The only issue is new userspace on old kernel. I think if we use both the > configure macro in addition to a size check, then it will at least allow > forward and backward compatibility. Are you talking about a new userspace compiled on a new kernel header file run on an old kernel? That would be less reliable and need the size check. The bitmap would be the most reliable in that scenario. By configure macro are you talking about the presence of that audit status mask bit, or the presence of that struct audit_status member? > Other metrics would be good. I'd like to see a max_backlog to know if we are > wasting memory. It would just record the highwater mark since auditing was > enabled. That would be covered with this issue: https://github.com/linux-audit/audit-kernel/issues/63 > -Steve - RGB -- Richard Guy Briggs Sr. S/W Engineer, Kernel Security, Base Operating Systems Remote, Ottawa, Red Hat Canada IRC: rgb, SunRaycer Voice: +1.647.777.2635, Internal: (81) 32635 -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
Re: [PATCH v2] audit: report audit wait metric in audit status reply
On Monday, December 7, 2020 8:34:35 PM EST Richard Guy Briggs wrote: > On 2020-12-07 18:28, Steve Grubb wrote: > > Hello Max, > > > > On Monday, December 7, 2020 4:28:14 PM EST Max Englander wrote: > > > Steve, I'm happy to make changes to the userspace PR based on > > > Richard's suggestions, if that sounds good to you. I'll follow up in > > > the PR to discuss it more > > > > The only issue is new userspace on old kernel. I think if we use both the > > configure macro in addition to a size check, then it will at least allow > > forward and backward compatibility. > > Are you talking about a new userspace compiled on a new kernel header > file run on an old kernel? Yes. This is my worry. Someone compiles the code and the does a roll back. It can happen because the new kernel has some problems that a driver cannot handle. > That would be less reliable and need the > size check. The bitmap would be the most reliable in that scenario. Right, but the person that can make that happen doesn't want to use this facility for what it was intended for. So, we are all trying to do the best. > By configure macro are you talking about the presence of that audit > status mask bit, or the presence of that struct audit_status member? Yes. But it doesn't apply to old kernels. -Steve -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit
