On 06/29/2018 08:16 PM, David Lang wrote:
On Fri, 29 Jun 2018, Rich Megginson via rsyslog wrote:
On 06/29/2018 06:49 PM, David Lang wrote:
On Fri, 29 Jun 2018, Rich Megginson via rsyslog wrote:
On 06/29/2018 03:08 PM, David Lang wrote:
the filename I have is:
$!:{ "metadata": { "filename":
"\/var\/log\/containers\/calico-node-jlhg5_kube-system_calico-node-e9abcaf4367e0264243c955699cd0987ff610df33fc9299facb97d1cb97aca2b.log",
"fileoffset": "553373" } }
the rule that should match it is:
rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log
but since the node name contains a - and the separater betweeen
the contain name and container id is also a -, the char-to:- is
not matching enough of the name
Is there any way in liblognorm to properly match this? or do we
need to say that - is not an allowed character?
I changed mmkubernetes to match on container_name_and_id:
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/k8s_filename.rulebase
what would we need to do to have these files show up in the OS
packages? (or do they, and I just missed them?)
Should show up in the next Fedora rebase (f28 is currently on 8.35.0,
should rebase to 8.36.0 soon).
EL7 . . . working on it . . .
Ok, this was centos 7, so it's in the works to make these files part
of the package, thanks.
other platforms - I have no idea
We need to update the doc to match the new rulebase
and the code too :-(
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L78
but at least the actual rulebase files included with rsyslog are
correct:
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/k8s_filename.rulebase
I just went through the excercise of trying to use mmkubernetes with
v8.36 and had to explicitly create the rulebase files and point to
them,
You could have used the rulebase files included with rsyslog, but
perhaps the documentation was not clear.
I didn't see the files included with the syslog packages, and so I
created files by doing a cut-n-paste from the docs, I later saw they
were in the source but then assumed that they matched the docs and
didn't even look at them :-(
I have submitted PRs to address these.
trying to let it use the default rules (as shown in the examples)
did not work. Because of the parsing problem above, I ended up
routing things through journald, and that also failed until I
created a ruleset with the rules that are in the documentation and
explicitly pointed at it.
and now I understand why the file version didn't work. I still don't
understand why the journald version didn't work.
I don't either, but that is also tricky, because the rsyslog container
must have /run/log/journal and /var/log/journal from the host mapped
into the container.
One other thing that I would like to see is an example of "I have
the namespace and pod name for this log, now go fetch the data from
the API for me".
When does that happen? That is, what is the use case when you are
not immediately trying to annotate the log record after reading it
from the kubernetes/container log file, or from journal?
I'm thinking that this happens at pod/container creation time, there
should be a log message (or some hook to create an action) that has
all three data items there at the same time.
I'm still not sure how you would use that. You still need some way to
associate every container log message read with the container name, pod
name, and namespace name.
Right now, extracting this is done inside the module, and I think I
see ways to dummy up a variable, tell mmkubernetes to use that as
the file metadata, set a custom ruleset to parse the variable, and I
can probably trick mmkubernetes into using my values. But that's a
pretty ugly way to do the job :-)
Or dummy up a CONTAINER_NAME and CONTAINER_ID_FULL, but yes, it isn't
ideal.
I'm thinking along the lines of watching the k8s master or docker
logs to maintain a lookup table that allows looking up the container
ID (which docker will put in a log message) and retrieve the pod and
namespace names.
What is the use case for this?
The case I ran into here is that we are running all the kubernetes
software in containers/pods, so /var/log/containers is something
inside the container, as is /var/log/pods, it's only the third file in
the symlink chain that is written outside the container where the
syslog on the host can see it (work-around, mount /var/log/pods to the
host)
my container expert wasnt to have rsyslog containerised as well, which
makes it harder to share filesystems.
If we move to cloud based systems, being able to share a filesystem
between pods or route through journald on a potentially shared host
seems even more of a problem.
being able to run one instance (or a couple redundant instances) per
cluster also seems like a much better option than needing to run this
on each node to be able to access the shared filesystem or journald
yes - the rsyslog running on each node would not do the kubernetes
metadata lookup, but instead just read from the json-file/journald
source, and send the logs to a cluster of rsyslog aggregators which use
mmkubernetes to do the annotation. This also saves kubernetes resources
on large clusters, where you might run into the limit on kubernetes api
connections. However - currently the only way to get log records with
the correct kubernetes metadata is to use either docker with
--log-driver=json-file or --log-driver=journald, or use cri-o with the
log file driver. This is the only way currently to get docker to
annotate each log record with the pod name, namespace name, container
name, and container id.
If using the file log drivers, you will need to use imfile with
addmetadata="on" so that you have $!metadata!filename in order to
extract the kubernetes metadata from the filename. So you might as well
send that information to the aggregators (and in fact, right now, you
have to let mmkubernetes parse out the container_name_and_id field due
to the lognorm limitation). Being able to get the name of the file
under /var/log/containers and parse it correctly is an absolute
necessity - no way currently around this.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
