Hi Rich,
Indexed pre_tag_map could fit the bill (and if going down this route i
do recommend to perform a proof-of-concept using latest & greatest code
in master since it has been a recent area of growth / improvement).
Example of how you could populate the pre_tag_map (centric on IPv4 but
you can similarly throw IPv6 in the mix):
set_label=dns dst_net=X.X.X.X/32
set_label=dns dst_net=Y.Y.Y.Y/32
...
set_label=ntp dst_net=J.J.J.J/32
set_label=ntp dst_net=K.K.K.K/32
...
set_label=xyz dst_net=W.W.W.W/32
set_label=xyz dst_net=Z.Z.Z.Z/32
And so forth. That simple if there is guarantee of no overlaps, ie. an
host serving, say, both dns and ntp.
In case of overlaps you could make the workflow more complex (and
computationally more expensive as it's not going to be a O(1) hit
anymore) with JEQs in the map but i'd rather recommend you determine
upfront - when composing the map - whether this is the case and make
entries, say, 'set_label=dns,ntp dst_net=<..>'. It is a classic case of
what you prefer to make more complex / what to optimize - and the good
answer most often should be to optimize what happens at runtime.
So you have a pre_tag_map composed, you can reload it at runtime for
updates sending a SIGUSR2 to the daemon, you have labels issued for
flows. Now onto the filtering part & putting ports in the mix:
pre_tag_map: /path/to/pretag.map
maps_index: true
maps_entries: <real or estimate amount of rows>
!
plugins: print[dns], print[ntp], print[xyz]
!
pre_tag_label_filter[dns]: dns
aggregate_filter[dns]: dst port 53
!
pre_tag_label_filter[ntp]: ntp
aggregate_filter[ntp]: dst port 123
!
pre_tag_label_filter[xyz]: xyz
aggregate_filter[xyz]: dst port <..>
Then each plugin can have its own faith: output file / kafka topic, log
interval, historical time bucketing, etc.
This use-case is a classic, i have helped others optimizing and making
it sure that maps with entries in the order of tens / hundreds of
millions would just go through without problem (of course, in certain
cases, given the right amount of memory). Feel free to reach out here or
by unicast email if you need further help with this!
Paolo
On 15/12/22 16:03, Compton, Rich A wrote:
Hi, I have a few (~20) lists of IPs provided by Shadowserver
(https://www.shadowserver.org <https://www.shadowserver.org>) on a daily
basis. Some lists contain a few hundred IPs and some contain tens of
thousands of IPs. I want to have pmacct filter out netflow records that
do not have a destination IP contained in these lists.
Example logic would be:
If the netflow record is destined to an IP in the open DNS server list
and on UDP dst port 53
Then store netflow record
Else If the netflow record is destined to an IP in the open NTP server
list and on UDP dst port 123
Then store netflow record
..additional lists...
Else drop netflow record
Is there a way to do this? It seems like there would be too many
entries for BPF. Also, I want to dynamically update these lists every
night.
Thanks!
signature_1767717039
Rich Compton | Principal Eng | 314.596.2828
8560 Upland Drive, Suite B | Englewood, CO 80112
The contents of this e-mail message and
any attachments are intended solely for the
addressee(s) and may contain confidential
and/or legally privileged information. If you
are not the intended recipient of this message
or if this message has been addressed to you
in error, please immediately alert the sender
by reply e-mail and then delete this message
and any attachments. If you are not the
intended recipient, you are notified that
any use, dissemination, distribution, copying,
or storage of this message or any attachment
is strictly prohibited.
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
