Thank you for the feedback. Glad to hear that the changes are working well.
> On 28 Jan 2026, at 16:33, Matthias Fischer <[email protected]> > wrote: > > On 26.01.2026 18:18, Michael Tremer wrote: >> Hello Matthias, > > Hi, > >> Are you running this on Core Update 200? > > Now running on Core 199 - after applying... > >> There were some changes required so that we will extract the datasets from >> the rules tarballs. I am using a special feature in Suricata so that the >> lists won’t be using too much memory and will be quickly searchable: >> >> >> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f0b43241a501f7c545e3cb15f6989e945c60b3e2 > > ...these patches. > > No more errors. ;-) > > Best > Matthias > >> -Michael >> >>> On 25 Jan 2026, at 17:50, Matthias Fischer <[email protected]> >>> wrote: >>> >>> On 25.01.2026 15:40, Michael Tremer wrote: >>>> Hello Matthias, >>> >>> Hi Michael, >>> >>>> Nice catch! >>>> >>>> I fixed it here and added the missing “;”: >>> >>> Yep. The missing ";" seems to be fixed, but 'suricata' still doesn't >>> like our rules. I added 'violence' as an attachment... ;-) >>> >>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b >>>> >>>> If you go to the provider settings there is a button to force a ruleset >>>> update which should give you the fixed version. Please let me know if this >>>> works. >>> >>> I did that - rules were updated, but no luck: >>> >>> ***SNIP*** >>> ... >>> 18:37:37 suricata: [2577] <Info> -- Including configuration file >>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml. >>> 18:37:37 suricata: [2577] <Error> -- failed to set up dataset 'violence'. >>> 18:37:37 suricata: [2577] <Error> -- error parsing signature "drop dns >>> any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query"; >>> dns.query; domain; dataset:isset,violence,type string,load >>> datasets/violence.txt; classtype:policy-violation; priority:2; >>> sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence; >>> metadata:dbl violence.dbl.ipfire.org;)" from file >>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 39 >>> 18:37:37 suricata: [2577] <Error> -- failed to set up dataset 'violence'. >>> 18:37:37 suricata: [2577] <Error> -- error parsing signature "drop >>> http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP >>> Request"; http.host; domain; dataset:isset,violence,type string,load >>> datasets/violence.txt; classtype:policy-violation; priority:2; >>> sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence; >>> metadata:dbl violence.dbl.ipfire.org;)" from file >>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 40 >>> 18:37:37 suricata: [2577] <Error> -- failed to set up dataset 'violence'. >>> 18:37:37 suricata: [2577] <Error> -- error parsing signature "drop tls >>> any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection"; >>> tls.sni; domain; dataset:isset,violence,type string,load >>> datasets/violence.txt; classtype:policy-violation; priority:2; >>> sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence; >>> metadata:dbl violence.dbl.ipfire.org;)" from file >>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 41 >>> 18:37:37 suricata: [2577] <Error> -- failed to set up dataset 'violence'. >>> 18:37:37 suricata: [2577] <Error> -- error parsing signature "drop >>> quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC >>> Connection"; quic.sni; domain; dataset:isset,violence,type string,load >>> datasets/violence.txt; classtype:policy-violation; priority:2; >>> sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence; >>> metadata:dbl violence.dbl.ipfire.org;)" from file >>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 42 >>> ... >>> ***SNAP*** >>> >>> For better reading - see attached screenshot. >>> >>> Best >>> Matthias >>> >>>> Best, >>>> -Michael >>>> >>>>> On 24 Jan 2026, at 23:41, Matthias Fischer <[email protected]> >>>>> wrote: >>>>> >>>>> On 23.01.2026 17:39, Michael Tremer wrote: >>>>>> Hello Matthias, >>>>> >>>>> Hi Michael, >>>>> >>>>>> Thank you very much for testing IPFire DBL. >>>>> >>>>> No problem - I have news: >>>>> >>>>> After taking a closer look to the IPS system logs, unfortunately I found >>>>> some parsing errors: >>>>> >>>>> 'suricata' complains about missing ";". >>>>> >>>>> ***SNIP*** >>>>> ... >>>>> 00:32:40 suricata: [13343] <Info> -- Including configuration file >>>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml. >>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found >>>>> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop >>>>> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS >>>>> Query"; dns.query; domain; dataset:isset,ads,type string,load >>>>> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041; >>>>> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl >>>>> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules >>>>> at line 72 >>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found >>>>> ... >>>>> ***SNAP*** >>>>> >>>>> I tried, but didn't find the right place for any missing ";". >>>>> >>>>> Can "anyone" confirm? >>>>> >>>>> Best >>>>> Matthias >>>>> >>>>>>> On 23 Jan 2026, at 15:02, Matthias Fischer >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>> On 22.01.2026 12:33, Michael Tremer wrote: >>>>>>>> Hello everyone, >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> short feedback from me: >>>>>>> >>>>>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and >>>>>>> the URLfilter lists from 'dbl.ipfire.org'. >>>>>> >>>>>> This is an interesting case. What I didn’t manage to test yet is what >>>>>> happens when Suricata blocks the connection first. If URL Filter sees a >>>>>> domain that is being blocked it will either send you an error page if >>>>>> you are using HTTP, or simply close the connection if it is HTTPS. >>>>>> However, when Suricata comes first in the chain (and it will), it might >>>>>> close the connection because URL Filter has received the request. In the >>>>>> case of HTTPS this does not make any difference because the connection >>>>>> will be closed, but in the HTTP case you won’t see an error page any >>>>>> more and instead have the connection closed, too. You are basically >>>>>> losing the explicit error notification which is a little bit annoying. >>>>>> >>>>>> We could have the same when we are doing the same with Unbound and DNS >>>>>> filtering. Potentially we would need to whitelist the local DNS resolver >>>>>> then, but how is Suricata supposed to know that the same categories are >>>>>> activated in both places? >>>>>> >>>>>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and >>>>>>> copied/pasted them in my fritzbox filter lists. >>>>>> >>>>>> LOL Why not use IPFire to filter this as well? >>>>>> >>>>>>> Everything works as expected. Besides, the download of the IPFire >>>>>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-) >>>>>> >>>>>> Yes, we don’t have much traffic on the server, yet. >>>>>> >>>>>>> Functionality is good - no false positives or seen problems. Good work - >>>>>>> thanks! >>>>>> >>>>>> Nice. We need to distinguish a little between what is a technical issue >>>>>> and what is a false-positive/missing domain on the list. However, >>>>>> testing both at the same time is something we will all cope quite well >>>>>> with :) >>>>>> >>>>>> -Michael >>>>>> >>>>>>> Best >>>>>>> Matthias >>>>>>> >>>>>>>> Over the past few weeks I have made significant progress on this all, >>>>>>>> and I think we're getting close to something the community will be >>>>>>>> really happy with. I'd love to get feedback from the team before we >>>>>>>> finalise things. >>>>>>>> >>>>>>>> So what has happened? >>>>>>>> >>>>>>>> First of all, the entire project has been renamed. DNSBL is not >>>>>>>> entirely what this is. Although the lists can be thrown into DNS, they >>>>>>>> have much more use outside of it that I thought we should simply go >>>>>>>> with DBL, short for Domain Blocklist. After all, we are only importing >>>>>>>> domains. The new home of the project therefore is >>>>>>>> https://www.ipfire.org/dbl >>>>>>>> >>>>>>>> I have added a couple more lists that I thought interesting and I have >>>>>>>> added a couple more sources that I considered a good start. Hopefully, >>>>>>>> we will soon gather some more feedback on how well this is all holding >>>>>>>> up. My main focus has however been on the technology that will power >>>>>>>> this project. >>>>>>>> >>>>>>>> One of the bigger challenges was to create Suricata rules from the >>>>>>>> lists. Initially I tried to create a ton of rules but since our lists >>>>>>>> are so large, this quickly became too complicated. I have now settled >>>>>>>> on using a feature that is only available in more recent versions of >>>>>>>> Suricata (I believe 7 and later), but since we are already on Suricata >>>>>>>> 8 in IPFire this won’t be a problem for us. All domains for each list >>>>>>>> are basically compiled into one massively large dataset and one single >>>>>>>> rule is referring to that dataset. This way, we won’t have the option >>>>>>>> to remove any false-positives, but at least Suricata and the GUI won’t >>>>>>>> starve a really bad death when loading millions of rules. >>>>>>>> >>>>>>>> Suricata will now be able to use our rules to block access to any >>>>>>>> listed domains of each of the categories over DNS, HTTP, TLS or QUIC. >>>>>>>> Although I don’t expect many users to use Suricata to block porn or >>>>>>>> other things, this is a great backstop to enforce any policy like >>>>>>>> that. For example, if there is a user on the network who is trying to >>>>>>>> circumvent the DNS server that might filter out certain domains, even >>>>>>>> after getting an IP address resolved through other means, they won’t >>>>>>>> be able to open a TLS/QUIC connection or send a HTTP request to all >>>>>>>> blocked domains. Some people have said they were interested in >>>>>>>> blocking DNS-over-HTTPS and this is a perfect way to do this and >>>>>>>> actually be sure that any server that is being blocked on the list >>>>>>>> will actually be completely inaccessible. >>>>>>>> >>>>>>>> Those Suricata rules are already available for testing in Core Update >>>>>>>> 200: >>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85 >>>>>>>> >>>>>>>> I have chosen various severities for the lists. If someone was to >>>>>>>> block advertising using DBL, this is fine, but not a very severe >>>>>>>> alert. If someone chooses to block malware and there is a system on >>>>>>>> the network trying to access those domains, this is an alert worth >>>>>>>> being investigated by an admin. Our new Suricata Reporter will show >>>>>>>> those violations in different colours based on the severity which >>>>>>>> helps to identify the right alerts to further investigate. >>>>>>>> >>>>>>>> Formerly I have asked you to test the lists using URL Filter. Those >>>>>>>> rules are now available as well in Core Update 200: >>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a >>>>>>>> >>>>>>>> I talked about a method to remove any dead domains from any sources >>>>>>>> which is a great way to keep our lists smaller. The pure size of them >>>>>>>> is a problem in so many ways. That check was however a little bit too >>>>>>>> ambitious and I had to make it a little bit less eager. Basically if >>>>>>>> we are in doubt, we need to still list the domain because it might be >>>>>>>> resolvable by a user. >>>>>>>> >>>>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7 >>>>>>>> >>>>>>>> So how else could we make the lists smaller without losing any actual >>>>>>>> data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there >>>>>>>> is very little point in listing any domains of this TLD. They will >>>>>>>> always be caught anyways. So I built a check that marks all domains >>>>>>>> that don’t need to be included on the exported lists because they will >>>>>>>> never be needed and was able to shrink the size of the lists by a lot >>>>>>>> again. >>>>>>>> >>>>>>>> The website does not show this data, but the API returns the number of >>>>>>>> “subsumed” domains (I didn’t have a better name): >>>>>>>> >>>>>>>> curl https://api.dbl.ipfire.org/lists | jq . >>>>>>>> >>>>>>>> The number shown would normally be added to the total number of >>>>>>>> domains and usually cuts the size of the list by 50-200%. >>>>>>>> >>>>>>>> Those stats will now also be stored in a history table so that we will >>>>>>>> be able to track growth of all lists. >>>>>>>> >>>>>>>> Furthermore, the application will now send email notifications for any >>>>>>>> incoming reports. This way, we will be able to stay in close touch >>>>>>>> with the reporters and keep them up to date on their submissions as >>>>>>>> well as inform moderators that there is something to have a look at. >>>>>>>> >>>>>>>> The search has been refactored as well, so that we can show clearly >>>>>>>> whether something is blocked or not at one glance: >>>>>>>> https://www.ipfire.org/dbl/search?q=github.com. There is detailed >>>>>>>> information available on all domains and what happened to them. In >>>>>>>> case of GitHub.com, this seems to be blocked and unblocked by someone >>>>>>>> all of the time and we can see a clear audit trail of that: >>>>>>>> https://www.ipfire.org/dbl/lists/malware/domains/github.com >>>>>>>> >>>>>>>> On the DNS front, I have added some metadata to the zones so that >>>>>>>> people can programmatically request some data, like when it has been >>>>>>>> last updated (in a human-friendly timestamp and not only the serial), >>>>>>>> license, description and so on: >>>>>>>> >>>>>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org >>>>>>>> "total-domains=42226" >>>>>>>> "license=CC BY-SA 4.0" >>>>>>>> "updated-at=2026-01-20T22:17:02.409933+00:00" >>>>>>>> "description=Blocks domains used for ads, tracking, and ad delivery” >>>>>>>> >>>>>>>> Now, I would like to hear more feedback from you. I know we've all >>>>>>>> been stretched thin lately, so I especially appreciate anyone who has >>>>>>>> time to review and provide input. Ideas, just say if you like it or >>>>>>>> not. Where this could go in the future? >>>>>>>> >>>>>>>> Looking ahead, I would like us to start thinking about the RPZ feature >>>>>>>> that has been on the wishlist. IPFire DBL has been a bigger piece of >>>>>>>> work, and I think it's worth having a conversation about >>>>>>>> sustainability. Resources for this need to be allocated and paid for. >>>>>>>> Open source is about freedom, not free beer — and to keep building >>>>>>>> features like this, we will need to explore some funding options. I >>>>>>>> would be interested to hear any ideas you might have that could work >>>>>>>> for IPFire. >>>>>>>> >>>>>>>> Please share your thoughts on the mailing list when you can — even a >>>>>>>> quick 'looks good' or 'I have concerns about X' is valuable. Public >>>>>>>> discussion helps everyone stay in the loop and contribute. >>>>>>>> >>>>>>>> I am aiming to move forward with this in a week's time, so if you have >>>>>>>> input, now would be a good time to share it. >>>>>>>> >>>>>>>> Best, >>>>>>>> -Michael >>>>>>>> >>>>>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Good Morning Adolf, >>>>>>>>> >>>>>>>>> I had a look at this problem yesterday and it seems that parsing the >>>>>>>>> format is becoming a little bit difficult this way. Since this is >>>>>>>>> only affecting very few domains, I have simply whitelisted them all >>>>>>>>> manually and duckduckgo.com <http://duckduckgo.com/> and others >>>>>>>>> should now be easily reachable again. >>>>>>>>> >>>>>>>>> Please let me know if you have any more findings. >>>>>>>>> >>>>>>>>> All the best, >>>>>>>>> -Michael >>>>>>>>> >>>>>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hello Adolf, >>>>>>>>>> >>>>>>>>>> This is a good find. >>>>>>>>>> >>>>>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will >>>>>>>>>> have to have a source somewhere that blocks that domain. Not only a >>>>>>>>>> sub-domain of it. Otherwise we have a bug somewhere. >>>>>>>>>> >>>>>>>>>> This is most likely as the domain is listed here, but with some >>>>>>>>>> stuff afterwards: >>>>>>>>>> >>>>>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo >>>>>>>>>> >>>>>>>>>> We strip everything after a # away because we consider it a comment. >>>>>>>>>> However, that causes that there is only a line with the domain left >>>>>>>>>> which will cause it being listed. >>>>>>>>>> >>>>>>>>>> The # sign is used as some special character but at the same time it >>>>>>>>>> is being used for comments. >>>>>>>>>> >>>>>>>>>> I will fix this and then refresh the list. >>>>>>>>>> >>>>>>>>>> -Michael >>>>>>>>>> >>>>>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Michael, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote: >>>>>>>>>>>> Hi Michael, >>>>>>>>>>>> >>>>>>>>>>>> I have found that the malware list includes duckduckgo.com >>>>>>>>>>>> >>>>>>>>>>> I have checked through the various sources used for the malware >>>>>>>>>>> list. >>>>>>>>>>> >>>>>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in >>>>>>>>>>> its list. I suspect that this one is the one causing the problem. >>>>>>>>>>> >>>>>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 >>>>>>>>>>> times but not directly as a domain name - looks more like a >>>>>>>>>>> reference. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Adolf. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote: >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <[email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Michael, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote: >>>>>>>>>>>>>>>> Hello everyone, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet >>>>>>>>>>>>>>>> days to relax from all the stress that was the year 2025. >>>>>>>>>>>>>>> Still relaxing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Very good, so let’s have a strong start into 2026 now! >>>>>>>>>>>>> >>>>>>>>>>>>> Starting next week, yes. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, >>>>>>>>>>>>>>>> little (hopefully) side project that has probably been high up >>>>>>>>>>>>>>>> on our radar since the Shalla list has shut down in 2020, or >>>>>>>>>>>>>>>> maybe even earlier. The goal of the project is to provide good >>>>>>>>>>>>>>>> lists with categories of domain names which are usually used >>>>>>>>>>>>>>>> to block access to these domains. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS >>>>>>>>>>>>>>>> Blocklists. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> How did we get here? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the >>>>>>>>>>>>>>>> problem that there are not many good blocklists available any >>>>>>>>>>>>>>>> more. There used to be a couple more - most famously the >>>>>>>>>>>>>>>> Shalla list - but we are now down to a single list from the >>>>>>>>>>>>>>>> University of Toulouse. It is a great list, but it is not >>>>>>>>>>>>>>>> always the best fit for all users. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Then there has been talk about whether we could implement more >>>>>>>>>>>>>>>> blocking features into IPFire that don’t involve the proxy. >>>>>>>>>>>>>>>> Most famously blocking over DNS. The problem here remains a >>>>>>>>>>>>>>>> the blocking feature is only as good as the data that is fed >>>>>>>>>>>>>>>> into it. Some people have been putting forward a number of >>>>>>>>>>>>>>>> lists that were suitable for them, but they would not have >>>>>>>>>>>>>>>> replaced the blocking functionality as we know it. Their aim >>>>>>>>>>>>>>>> is to provide “one list for everything” but that is not what >>>>>>>>>>>>>>>> people usually want. It is targeted at a classic home user and >>>>>>>>>>>>>>>> the only separation that is being made is any adult/porn/NSFW >>>>>>>>>>>>>>>> content which usually is put into a separate list. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It would have been technically possible to include these lists >>>>>>>>>>>>>>>> and let the users decide, but that is not the aim of IPFire. >>>>>>>>>>>>>>>> We want to do the job for the user so that their job is >>>>>>>>>>>>>>>> getting easier. Including obscure lists that don’t have a >>>>>>>>>>>>>>>> clear outline of what they actually want to block (“bad >>>>>>>>>>>>>>>> content” is not a category) and passing the burden of figuring >>>>>>>>>>>>>>>> out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, >>>>>>>>>>>>>>>> “Ultimate” or even a “Venti” list with cream on top is really >>>>>>>>>>>>>>>> not going to work. It is all confusing and will lead to a bad >>>>>>>>>>>>>>>> user experience. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> An even bigger problem that is however completely impossible >>>>>>>>>>>>>>>> to solve is bad licensing of these lists. A user has asked the >>>>>>>>>>>>>>>> publisher of the HaGeZi list whether they could be included in >>>>>>>>>>>>>>>> IPFire and under what terms. The response was that the list is >>>>>>>>>>>>>>>> available under the terms of the GNU General Public License >>>>>>>>>>>>>>>> v3, but that does not seem to be true. The list contains data >>>>>>>>>>>>>>>> from various sources. Many of them are licensed under >>>>>>>>>>>>>>>> incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and >>>>>>>>>>>>>>>> unless there is a non-public agreement that this data may be >>>>>>>>>>>>>>>> redistributed, there is a huge legal issue here. We would >>>>>>>>>>>>>>>> expose our users to potential copyright infringement which we >>>>>>>>>>>>>>>> cannot do under any circumstances. Furthermore many lists are >>>>>>>>>>>>>>>> available under a non-commercial license which excludes them >>>>>>>>>>>>>>>> from being used in any kind of business. Plenty of IPFire >>>>>>>>>>>>>>>> systems are running in businesses, if not even the vast >>>>>>>>>>>>>>>> majority. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart >>>>>>>>>>>>>>>> from HaGeZi, I consider OISD to have the same problem. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the >>>>>>>>>>>>>>>> new, good things: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other >>>>>>>>>>>>>>>> lists. These lists vary in quality with some of them being not >>>>>>>>>>>>>>>> that good and without a clear focus and others being excellent >>>>>>>>>>>>>>>> data. Since we don’t have the man power to start from scratch, >>>>>>>>>>>>>>>> I felt that we can copy the concept that HaGeZi and OISD have >>>>>>>>>>>>>>>> started and simply create a new list that is based on other >>>>>>>>>>>>>>>> lists at the beginning to have a good starting point. That >>>>>>>>>>>>>>>> way, we have much better control over what is going on these >>>>>>>>>>>>>>>> lists and we can shape and mould them as we need them. Most >>>>>>>>>>>>>>>> importantly, we don’t create a single lists, but many lists >>>>>>>>>>>>>>>> that have a clear focus and allow users to choose what they >>>>>>>>>>>>>>>> want to block and what not. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So the current experimental stage that I am in has these lists: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> * Ads >>>>>>>>>>>>>>>> * Dating >>>>>>>>>>>>>>>> * DoH >>>>>>>>>>>>>>>> * Gambling >>>>>>>>>>>>>>>> * Malware >>>>>>>>>>>>>>>> * Porn >>>>>>>>>>>>>>>> * Social >>>>>>>>>>>>>>>> * Violence >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The categories have been determined by what source lists we >>>>>>>>>>>>>>>> have available with good data and are compatible with our >>>>>>>>>>>>>>>> chosen license CC BY-SA 4.0. This is the same license that we >>>>>>>>>>>>>>>> are using for the IPFire Location database, too. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with >>>>>>>>>>>>>>>> legal requirements in networks with children (i.e. schools) to >>>>>>>>>>>>>>>> remove any kind of pornographic content, sometimes block >>>>>>>>>>>>>>>> social media as well. Gambling and violence are commonly >>>>>>>>>>>>>>>> blocked, too. Even more common would be filtering advertising >>>>>>>>>>>>>>>> and any malicious content. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The latter is especially difficult because so many source >>>>>>>>>>>>>>>> lists throw phishing, spyware, malvertising, tracking and >>>>>>>>>>>>>>>> other things into the same bucket. Here this is currently all >>>>>>>>>>>>>>>> in the malware list which has therefore become quite large. I >>>>>>>>>>>>>>>> am not sure whether this will stay like this in the future or >>>>>>>>>>>>>>>> if we will have to make some adjustments, but that is exactly >>>>>>>>>>>>>>>> why this is now entering some larger testing. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What has been built so far? In order to put these lists >>>>>>>>>>>>>>>> together properly, track any data about where it is coming >>>>>>>>>>>>>>>> from, I have built a tool in Python available here: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This tool will automatically update all lists once an hour if >>>>>>>>>>>>>>>> there have been any changes and export them in various >>>>>>>>>>>>>>>> formats. The exported lists are available for download here: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/ >>>>>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as >>>>>>>>>>>>>>> the custom url works fine. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> However you need to remember not to put the https:// at the >>>>>>>>>>>>>>> front of the url otherwise the WUI page completes without any >>>>>>>>>>>>>>> error messages but leaves an error message in the system logs >>>>>>>>>>>>>>> saying >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I found this out the hard way. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that >>>>>>>>>>>>>> does not accept https:// as a prefix, please file a bug and we >>>>>>>>>>>>>> will fix it. >>>>>>>>>>>>> >>>>>>>>>>>>> I will confirm it and raise a bug. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> The other thing I noticed is that if you already have the >>>>>>>>>>>>>>> Toulouse University list downloaded and you then change to the >>>>>>>>>>>>>>> ipfire custom url then all the existing Toulouse blocklists >>>>>>>>>>>>>>> stay in the directory on IPFire and so you end up with a huge >>>>>>>>>>>>>>> number of category tick boxes, most of which are the old >>>>>>>>>>>>>>> Toulouse ones, which are still available to select and it is >>>>>>>>>>>>>>> not clear which ones are from Toulouse and which ones from >>>>>>>>>>>>>>> IPFire. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, >>>>>>>>>>>>>> because otherwise you would have a lot of unused categories >>>>>>>>>>>>>> lying around that will never be updated. You cannot even tell >>>>>>>>>>>>>> which ones are from the current list and which ones from the old >>>>>>>>>>>>>> list. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse >>>>>>>>>>>>>> list entirely and only have our own lists available which would >>>>>>>>>>>>>> make the problem go away. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url >>>>>>>>>>>>>>> is provided the first step should be to remove the old ones >>>>>>>>>>>>>>> already existing. >>>>>>>>>>>>>>> That might be a problem because users can also create their own >>>>>>>>>>>>>>> blocklists and I believe those go into the same directory. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Good thought. We of course cannot delete the custom lists. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge >>>>>>>>>>>>>>> number of checkboxes for lists but it is not clear what happens >>>>>>>>>>>>>>> if there is a category that has the same name for the Toulouse >>>>>>>>>>>>>>> list and the IPFire list such as gambling. I will have a look >>>>>>>>>>>>>>> at that and see what happens. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Not sure what the best approach to this is. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I believe it is removing all old content. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ >>>>>>>>>>>>>>> directory and then selecting the IPFire blocklist url for the >>>>>>>>>>>>>>> custom url I end up with only the 8 categories from the IPFire >>>>>>>>>>>>>>> list. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the >>>>>>>>>>>>>>> block worked on some. On others the site no longer exists so >>>>>>>>>>>>>>> there is nothing to block or has been changed to an https site >>>>>>>>>>>>>>> and in that case it went straight through. Also if I chose the >>>>>>>>>>>>>>> http version of the link, it was automatically changed to https >>>>>>>>>>>>>>> and went through without being blocked. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you >>>>>>>>>>>>>> start using HTTP, you will be automatically redirected. It is >>>>>>>>>>>>>> 2026 and we don’t need to talk HTTP any more :) >>>>>>>>>>>>> >>>>>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem >>>>>>>>>>>>> to only have an http access. If I tried https it came back with >>>>>>>>>>>>> the fact that it couldn't find it. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am glad to hear that the list is actually blocking. It would >>>>>>>>>>>>>> have been bad if it didn’t. Now we have the big task to check >>>>>>>>>>>>>> out the “quality” - however that can be determined. I think this >>>>>>>>>>>>>> is what needs some time… >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the meantime I have set up a small page on our website: >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://www.ipfire.org/dnsbl >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would like to run this as a first-class project inside IPFire >>>>>>>>>>>>>> like we are doing with IPFire Location. That means that we need >>>>>>>>>>>>>> to tell people about what we are doing. Hopefully this page is a >>>>>>>>>>>>>> little start. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Initially it has a couple of high-level bullet points about what >>>>>>>>>>>>>> we are trying to achieve. I don’t think the text is very good, >>>>>>>>>>>>>> yet, but it is the best I had in that moment. There is then also >>>>>>>>>>>>>> a list of the lists that we currently offer. For each list, a >>>>>>>>>>>>>> detailed page will tell you about the license, how many domains >>>>>>>>>>>>>> are listed, when the last update has been, the sources and even >>>>>>>>>>>>>> there is a history page that shows all the changes whenever they >>>>>>>>>>>>>> have happened. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Finally there is a section that explains “How To Use?” the list >>>>>>>>>>>>>> which I would love to extend to include AdGuard Plus and things >>>>>>>>>>>>>> like that as well as Pi-Hole and whatever else could use the >>>>>>>>>>>>>> list. In a later step we should go ahead and talk to any >>>>>>>>>>>>>> projects to include our list(s) into their dropdown so that >>>>>>>>>>>>>> people can enable them nice and easy. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Behind the web page there is an API service that is running on >>>>>>>>>>>>>> the host that is running the DNSBL. The frontend web app that is >>>>>>>>>>>>>> running www.ipfire.org <http://www.ipfire.org/> is connecting to >>>>>>>>>>>>>> that API service to fetch the current lists, any details and so >>>>>>>>>>>>>> on. That way, we can split the logic and avoid creating a huge >>>>>>>>>>>>>> monolith of a web app. This also means that page could be down a >>>>>>>>>>>>>> little as I am still working on the entire thing and will >>>>>>>>>>>>>> frequently restart it. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The API documentation is available here and the API is publicly >>>>>>>>>>>>>> available: https://api.dnsbl.ipfire.org/docs >>>>>>>>>>>>>> >>>>>>>>>>>>>> The website/API allows to file reports for anything that does >>>>>>>>>>>>>> not seem to be right on any of the lists. I would like to keep >>>>>>>>>>>>>> it as an open process, however, long-term, this cannot cost us >>>>>>>>>>>>>> any time. In the current stage, the reports are getting filed >>>>>>>>>>>>>> and that is about it. I still need to build out some way for >>>>>>>>>>>>>> admins or moderators (I am not sure what kind of roles I want to >>>>>>>>>>>>>> have here) to accept or reject those reports. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In case of us receiving a domain from a source list, I would >>>>>>>>>>>>>> rather like to submit a report to upstream for them to de-list. >>>>>>>>>>>>>> That way, we don’t have any admin to do and we are contributing >>>>>>>>>>>>>> back to other list. That would be a very good thing to do. We >>>>>>>>>>>>>> cannot however throw tons of emails at some random upstream >>>>>>>>>>>>>> projects without co-ordinating this first. By not reporting >>>>>>>>>>>>>> upstream, we will probably over time create large whitelists and >>>>>>>>>>>>>> I am not sure if that is a good thing to do. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Finally, there is a search box that can be used to find out if a >>>>>>>>>>>>>> domain is listed on any of the lists. >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If you download and open any of the files, you will see a >>>>>>>>>>>>>>>> large header that includes copyright information and lists all >>>>>>>>>>>>>>>> sources that have been used to create the individual lists. >>>>>>>>>>>>>>>> This way we ensure maximum transparency, comply with the terms >>>>>>>>>>>>>>>> of the individual licenses of the source lists and give credit >>>>>>>>>>>>>>>> to the people who help us to put together the most perfect >>>>>>>>>>>>>>>> list for our users. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would like this to become a project that is not only being >>>>>>>>>>>>>>>> used in IPFire. We can and will be compatible with other >>>>>>>>>>>>>>>> solutions like AdGuard, PiHole so that people can use our >>>>>>>>>>>>>>>> lists if they would like to even though they are not using >>>>>>>>>>>>>>>> IPFire. Hopefully, these users will also feed back to us so >>>>>>>>>>>>>>>> that we can improve our lists over time and make them one of >>>>>>>>>>>>>>>> the best options out there. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> All lists are available as a simple text file that lists the >>>>>>>>>>>>>>>> domains. Then there is a hosts file available as well as a DNS >>>>>>>>>>>>>>>> zone file and an RPZ file. Each list is individually available >>>>>>>>>>>>>>>> to be used in squidGuard and there is a larger tarball >>>>>>>>>>>>>>>> available with all lists that can be used in IPFire’s URL >>>>>>>>>>>>>>>> Filter. I am planning to add Suricata/Snort signatures >>>>>>>>>>>>>>>> whenever I have time to do so. Even though it is not a good >>>>>>>>>>>>>>>> idea to filter pornographic content this way, I suppose that >>>>>>>>>>>>>>>> catching malware and blocking DoH are good use-cases for an >>>>>>>>>>>>>>>> IPS. Time will tell… >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL >>>>>>>>>>>>>>>> Filter and collect some feedback about how we are doing. >>>>>>>>>>>>>>>> Afterwards, we can see where else we can take this project. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If you want to enable this on your system, simply add the URL >>>>>>>>>>>>>>>> to your autoupdate.urls file like here: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f >>>>>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and >>>>>>>>>>>>>>> that also worked fine for me. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Very good. Should we include this already with Core Update 200? >>>>>>>>>>>>>> I don’t think we would break anything, but we might already gain >>>>>>>>>>>>>> a couple more people who are helping us to test this all? >>>>>>>>>>>>> >>>>>>>>>>>>> I think that would be a good idea. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The next step would be to build and test our DNS infrastructure. >>>>>>>>>>>>>> In the “How To Use?” Section on the pages of the individual >>>>>>>>>>>>>> lists, you can already see some instructions on how to use the >>>>>>>>>>>>>> lists as an RPZ. In comparison to other “providers”, I would >>>>>>>>>>>>>> prefer if people would be using DNS to fetch the lists. This is >>>>>>>>>>>>>> simply to push out updates in a cheap way for us and also do it >>>>>>>>>>>>>> very regularly. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There >>>>>>>>>>>>>> is no way around this as they need to have the data in the first >>>>>>>>>>>>>> place. After that, clients will only need the changes. As you >>>>>>>>>>>>>> can see in the history, the lists don’t actually change that >>>>>>>>>>>>>> often. Sometimes only once a day and therefore downloading the >>>>>>>>>>>>>> entire list again would be a huge waste of data, both on the >>>>>>>>>>>>>> client side, but also for us hosting then. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and >>>>>>>>>>>>>> there won't be any changes whatsoever. We don’t do that. We will >>>>>>>>>>>>>> only export the lists again when they have actually changed. The >>>>>>>>>>>>>> timestamps on the files that we offer using HTTPS can be checked >>>>>>>>>>>>>> by clients so that they won’t re-download the list again if it >>>>>>>>>>>>>> has not been changed. But using HTTPS still means that we would >>>>>>>>>>>>>> have to re-download the entire list and not only the changes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a >>>>>>>>>>>>>> few kilobytes and therefore we can have clients check once an >>>>>>>>>>>>>> hour if a list has actually changed and only send out the raw >>>>>>>>>>>>>> changes. That way, we will be able to serve millions of clients >>>>>>>>>>>>>> at very cheap cost and they will always have a very up to date >>>>>>>>>>>>>> list. >>>>>>>>>>>>>> >>>>>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports >>>>>>>>>>>>>> AXFR/IXFR with exception of Knot Resolver which expects the zone >>>>>>>>>>>>>> to be downloaded externally. There is a ticket for AXFR/IXFR >>>>>>>>>>>>>> support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Initially, some of the lists have been *huge* which is why a >>>>>>>>>>>>>> simple HTTP download is not feasible. The porn list was over 100 >>>>>>>>>>>>>> MiB. We could have spent thousands on just traffic alone which I >>>>>>>>>>>>>> don’t have for this kind of project. It would also be >>>>>>>>>>>>>> unnecessary money being spent. There are simply better solutions >>>>>>>>>>>>>> out there. But then I built something that basically tests the >>>>>>>>>>>>>> data that we are receiving from upstream but simply checking if >>>>>>>>>>>>>> a listed domain still exists. The result was very astonishing to >>>>>>>>>>>>>> me. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So whenever someone adds a domain to the list, we will >>>>>>>>>>>>>> (eventually, but not immediately) check if we can resolve the >>>>>>>>>>>>>> domain’s SOA record. If not, we mark the domain as non-active >>>>>>>>>>>>>> and will no longer include them in the exported data. This >>>>>>>>>>>>>> brought down the porn list from just under 5 million domains to >>>>>>>>>>>>>> just 421k. On the sources page >>>>>>>>>>>>>> (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing >>>>>>>>>>>>>> the percentage of dead domains from each of them and the UT1 >>>>>>>>>>>>>> list has 94% dead domains. Wow. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we >>>>>>>>>>>>>> would otherwise fill the lists with tons of domains that simply >>>>>>>>>>>>>> could never be reached. And if they cannot be reached, why would >>>>>>>>>>>>>> we block them? We would waste bandwidth and a lot of memory on >>>>>>>>>>>>>> each single client. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The other sources have similarly high rations of dead domains. >>>>>>>>>>>>>> Most of them are in the 50-80% range. Therefore I am happy that >>>>>>>>>>>>>> we are doing some extra work here to give our users much better >>>>>>>>>>>>>> data for their filtering. >>>>>>>>>>>>> >>>>>>>>>>>>> Removing all dead entries sounds like an excellent step. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with >>>>>>>>>>>>>> Unbound. Instructions are on the page. I would be happy to hear >>>>>>>>>>>>>> how this is turning out. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please let me know if there are any more questions, and I would >>>>>>>>>>>>>> be glad to answer them. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Happy New Year, >>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>> This email is just a brain dump from me to this list. I would >>>>>>>>>>>>>>>> be happy to answer any questions about implementation details, >>>>>>>>>>>>>>>> etc. if people are interested. Right now, this email is long >>>>>>>>>>>>>>>> enough already… >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> All the best, >>>>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Sent from my laptop >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Sent from my laptop >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> <suricata-log.jpg><ipfire_dnsbl-violence.rules> >> >
