Re: [Dnsmasq-discuss] Feature request = block-conf
> OK, but as you rightly pointed out a couple of messages ago, all these > operations are performed before the service dnsmasq restart. > > Because of this I have performed an additional test to complete the > picture: > > my usual 650K list of domains is now unzipped and I have simply scripted > the usual formatting with address=/domain/. So no compression involved > this time. e.g. > > /bin/cat /mnt/USB/adblock/adblock.domains | /bin/sed -e "s:^:address=/:" > -e "s:$:/:" > > > The service dnsmasq restart went from 15.something sec of the script+zip > version to 14.4 sec of this script only version. > root at sparrow:/tmp/mnt/USB/adblock# time service dnsmasq restart > > Done. > real0m 14.43s > user0m 0.00s > sys 0m 0.00s > > What I was trying to say is that feeding pre-formatted config (as per > prev message posted here) takes 4.2 sec on the very same file. > > And you're right the aim here is to minimize storage hence RAM demand, > but the dnsmasq restart time must also stay within acceptable levels as > nobody wants the DNS resolution to be down for too long. > > So to summarize: Because the conf-script option works great on RAM > reduction but takes a big hit on the restart time, I was suggesting if a > different approach could/should be considered. Since we learnt from this > last test that the zip operation add only 1-2 sec of delay on the restart > > root at sparrow:/mnt/USB/adblock# time gzip -d adblock.domains.unloaded.gz > real0m 1.76s > user0m 0.67s > sys 0m 0.18s > > perhaps allowing the conf-file to process zipped content (internal zcat > or something) would suffice to achieve the desired result to minimize > storage demand and retain restart time within acceptable levels. > Allowing something like conf-block and import domain only is indeed > nice-to-have but secondary in my opinion compared to support for > compressed config: > > root at sparrow:/mnt/USB/adblock# ls -lh adblock.domains.unloaded > -rwxrwxrwx1 root root 14.3M Mar 31 18:42 > root at sparrow:/mnt/USB/adblock# gzip adblock.domains.unloaded > root at sparrow:/mnt/USB/adblock# ls -lh adblock.domains.unloaded.gz > -rwxrwxrwx1 root root4.2M Mar 31 18:42 Simon, it is not my intention to put pressure or else but I just wanted to give a final feedback on this topic. The conf-script is not usable on a router for large lists/config. It simply takes dnsmasq far too long to restart. There are a series of event that restart dnsmasq from a simple Save Config from the GUI to an automatic event e.g. a multiWAN swap due to external factors and anything in between. on a 1M record blocklist it takes dnsmasq almost 30 before before it can resolve domains again. On one side I'm grateful this was considered by the dnsmasq team, on the other side I still believe (as per message above) that allowing dnsmasq to read natively a conf-file in .gz would actually be the simplest and possibly most effective of the solutions to resolve the large list issue. On plain config (conf-file) even a huge file takes dnsmasq 5-6 seconds to restart. The gzip operation on such file are about 1 sec on top which is totally an acceptable compromise. Finally the size of a 1M records conf-file compresses down from 40M to 8Mb. A win win. I'm not sure if you still have interest in tackling this topic but I felt a feedback is expected as a minimum given all the work done so far. It would be good to know either ways if you don't mind. Thanks! ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
My understanding is that the script-conf feature was to be used to reduce filesystem storage usage, ie instead of a conf-file full of lines like address=/domain1.com/ address=/domain2.com/ Juts the list of domains could be stored, in compressed form, and then they would be decompressed and decorated with the address=/./ stuff on the fly before being fed into the dnsmasq configuration parser. The scripts you posted earlier seemed to 1) Download the block list. 2) compress it 3) feed it to dnsmasq which decompresses it Which is the worst of all possible worlds, since it uses more storage for the compressed AND uncompressed versions, and more CPU to the compression and decompression. It also make the down-time depend on how fast the block list downloads. OK, but as you rightly pointed out a couple of messages ago, all these operations are performed before the service dnsmasq restart. Because of this I have performed an additional test to complete the picture: my usual 650K list of domains is now unzipped and I have simply scripted the usual formatting with address=/domain/. So no compression involved this time. e.g. /bin/cat /mnt/USB/adblock/adblock.domains | /bin/sed -e "s:^:address=/:" -e "s:$:/:" The service dnsmasq restart went from 15.something sec of the script+zip version to 14.4 sec of this script only version. root@sparrow:/tmp/mnt/USB/adblock# time service dnsmasq restart Done. real0m 14.43s user0m 0.00s sys 0m 0.00s What I was trying to say is that feeding pre-formatted config (as per prev message posted here) takes 4.2 sec on the very same file. And you're right the aim here is to minimize storage hence RAM demand, but the dnsmasq restart time must also stay within acceptable levels as nobody wants the DNS resolution to be down for too long. So to summarize: Because the conf-script option works great on RAM reduction but takes a big hit on the restart time, I was suggesting if a different approach could/should be considered. Since we learnt from this last test that the zip operation add only 1-2 sec of delay on the restart root@sparrow:/mnt/USB/adblock# time gzip -d adblock.domains.unloaded.gz real0m 1.76s user0m 0.67s sys 0m 0.18s perhaps allowing the conf-file to process zipped content (internal zcat or something) would suffice to achieve the desired result to minimize storage demand and retain restart time within acceptable levels. Allowing something like conf-block and import domain only is indeed nice-to-have but secondary in my opinion compared to support for compressed config: root@sparrow:/mnt/USB/adblock# ls -lh adblock.domains.unloaded -rwxrwxrwx1 root root 14.3M Mar 31 18:42 root@sparrow:/mnt/USB/adblock# gzip adblock.domains.unloaded root@sparrow:/mnt/USB/adblock# ls -lh adblock.domains.unloaded.gz -rwxrwxrwx1 root root4.2M Mar 31 18:42 My 2 cents ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
On 30/03/2022 12:13, Ercolino de Spiacico wrote: It looks like your script which downloads the blocked domains file and compresses it takes 15s, then dnsmasq takes 15s to uncompress the list and load it into memory and sort. The first delay can be solved by doing the download before stopping the old dnsmasq process. The second is amenable the new option to SIGTERM the old dnsmasq _after_ parsing the new config. I run some additional test to verify the impact. I got the very same list (650K records) unzipped and pre-formatted (address=/$domain/) to be used by the standard conf-file directive. As we used to do before the conf-script was introduced. Here the result: root@sparrow:/tmp/mnt/USB/adblock# time service dnsmasq restart .. Done. real 0m 4.21s user 0m 0.00s sys 0m 0.01s The amazing part is that there's no measurable delay in the change of ownership from root to nobody. So 4.2 seconds and that's it which is completely acceptable for this list size. But then we're back to square 1 before this conversation ever happened. So I'm starting to conclude that the scripting part (script-conf) and it's interpretation internally, despite being flexible in terms of wanted output, it doesn't scale well on large block-files. Fair enough admittedly this has never been the reason why dnsmasq was originally created. This also make me think further about two (potentially even overlapping) ways to progress: A- Going back to my original idea to have a new conf-block (or block-conf whichever) directive where any domain defined in the target file will be blocked (only domains without formatting) dnsmasq would do an internal interpretation of this list of domains based on the directive. So smaller list size and no extra processing needed. This perhaps could be an option to have the cake and eat it? B- Allow the conf-block= and/or the existing conf-file= to read a zipped source e.g. conf-block=z,/bla/bla/file.gz conf-file=z,/bla/bla/file.gz Again no scripting/manipulation but taking advantage of the on-the-fly decompression My understanding is that the script-conf feature was to be used to reduce filesystem storage usage, ie instead of a conf-file full of lines like address=/domain1.com/ address=/domain2.com/ Juts the list of domains could be stored, in compressed form, and then they would be decompressed and decorated with the address=/./ stuff on the fly before being fed into the dnsmasq configuration parser. The scripts you posted earlier seemed to 1) Download the block list. 2) compress it 3) feed it to dnsmasq which decompresses it Which is the worst of all possible worlds, since it uses more storage for the compressed AND uncompressed versions, and more CPU to the compression and decompression. It also make the down-time depend on how fast the block list downloads. Simon. ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
It looks like your script which downloads the blocked domains file and compresses it takes 15s, then dnsmasq takes 15s to uncompress the list and load it into memory and sort. The first delay can be solved by doing the download before stopping the old dnsmasq process. The second is amenable the new option to SIGTERM the old dnsmasq _after_ parsing the new config. I run some additional test to verify the impact. I got the very same list (650K records) unzipped and pre-formatted (address=/$domain/) to be used by the standard conf-file directive. As we used to do before the conf-script was introduced. Here the result: root@sparrow:/tmp/mnt/USB/adblock# time service dnsmasq restart .. Done. real0m 4.21s user0m 0.00s sys 0m 0.01s The amazing part is that there's no measurable delay in the change of ownership from root to nobody. So 4.2 seconds and that's it which is completely acceptable for this list size. But then we're back to square 1 before this conversation ever happened. So I'm starting to conclude that the scripting part (script-conf) and it's interpretation internally, despite being flexible in terms of wanted output, it doesn't scale well on large block-files. Fair enough admittedly this has never been the reason why dnsmasq was originally created. This also make me think further about two (potentially even overlapping) ways to progress: A- Going back to my original idea to have a new conf-block (or block-conf whichever) directive where any domain defined in the target file will be blocked (only domains without formatting) dnsmasq would do an internal interpretation of this list of domains based on the directive. So smaller list size and no extra processing needed. This perhaps could be an option to have the cake and eat it? B- Allow the conf-block= and/or the existing conf-file= to read a zipped source e.g. conf-block=z,/bla/bla/file.gz conf-file=z,/bla/bla/file.gz Again no scripting/manipulation but taking advantage of the on-the-fly decompression thanks ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
On 28/03/2022 11:49, Ercolino de Spiacico wrote: This Internet list above (https://hosts.oisd.nl) is 40MB uncompressed, the regex extracts domains-only so shrinking it to 60% of its original size and the gzip compression shrinks much further. Decompressing and scripting it up of course takes time. All of that looks like stuff which can be done before stopping dnsmasq, right. SO how long it takes makes no difference to how long DNS and DHCP service is interrupted for? I just reported what's done to potentially try to replicate that's all. So going straight to the point: root@sparrow:/# zcat /mnt/USB/adblock/adblock.domains.gz | wc -l 658839 root@sparrow:/# time service dnsmasq restart . Done. real 0m 15.76s user 0m 0.01s sys 0m 0.00s But please note, at this point the process is still owned by root root@sparrow:/# ps | grep [d]nsmasq 31137 root 13264 S dnsmasq -c 4096 --log-async Until this is delegated to nobody (15 more seconds I'd say) name resolution is not performed. So 30sec in total? And this is with a list of 650K records, well below the maximum we managed to push on the same router with unscripted config which accounted for 1.8M. It looks like your script which downloads the blocked domains file and compresses it takes 15s, then dnsmasq takes 15s to uncompress the list and load it into memory and sort. The first delay can be solved by doing the download before stopping the old dnsmasq process. The second is amenable the new option to SIGTERM the old dnsmasq _after_ parsing the new config. Right! Could you please share more details on this idea? It could be a smart workaround indeed. When dnsmasq starts up, it does roughly the following things: 1) Read config files (including execute script-conf 2) organise the data from the config into in-memory data structures 3) Open listening sockets on DNS and DHCP ports. 4) Enter the event loop. 1) and 2) are likely to take an appreciaable amount of time with big block lists. 3) and 4) never do. A restart of dnsmasq consists of 1) send SIGTERM to existing dnsmasq process to cause it to halt. 2) Run new dnsmasq instance, which goes through the four steps above. If reading the congfig takes time, that causes service interruption. Note that you can't do those two steps in the opposite order, since the old dnsmasq process will still be listening on the DNS and DHCP ports, and the new one will fail to start up. If we add an option to dnsmasq which takes a process-ID and sends a SIGTERM to that process-ID as step 2.5, the old dnsmasq process can continue to run during the parsing of the options, then the it gets toen down before opening listening sockets. The restart now just runs the new dnsmasq instance, passing the PID of the old dnsmasq instance if it exists. Ok understood. Interesting it raises 3 questions on my side: 1- say the old process is already handling 1M blocked domains, I suppose this means that for the time of the restarting the RAM demand will be pushed to the equivalent of 2M domains (1M old + 1M new), right? Right. 2- this will certainly help the restart but there's nothing that can save us from the first ever start after a device reboot I suppose No. 3- Is there any option to hand over the existing resolution cache between the "leaver" and "joiner" process? Like a memory mapping swap or something? It could be done, but it's a huge rewrite. Simon. ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
This Internet list above (https://hosts.oisd.nl) is 40MB uncompressed, the regex extracts domains-only so shrinking it to 60% of its original size and the gzip compression shrinks much further. Decompressing and scripting it up of course takes time. All of that looks like stuff which can be done before stopping dnsmasq, right. SO how long it takes makes no difference to how long DNS and DHCP service is interrupted for? I just reported what's done to potentially try to replicate that's all. So going straight to the point: root@sparrow:/# zcat /mnt/USB/adblock/adblock.domains.gz | wc -l 658839 root@sparrow:/# time service dnsmasq restart . Done. real0m 15.76s user0m 0.01s sys 0m 0.00s But please note, at this point the process is still owned by root root@sparrow:/# ps | grep [d]nsmasq 31137 root 13264 Sdnsmasq -c 4096 --log-async Until this is delegated to nobody (15 more seconds I'd say) name resolution is not performed. So 30sec in total? And this is with a list of 650K records, well below the maximum we managed to push on the same router with unscripted config which accounted for 1.8M. Right! Could you please share more details on this idea? It could be a smart workaround indeed. When dnsmasq starts up, it does roughly the following things: 1) Read config files (including execute script-conf 2) organise the data from the config into in-memory data structures 3) Open listening sockets on DNS and DHCP ports. 4) Enter the event loop. 1) and 2) are likely to take an appreciaable amount of time with big block lists. 3) and 4) never do. A restart of dnsmasq consists of 1) send SIGTERM to existing dnsmasq process to cause it to halt. 2) Run new dnsmasq instance, which goes through the four steps above. If reading the congfig takes time, that causes service interruption. Note that you can't do those two steps in the opposite order, since the old dnsmasq process will still be listening on the DNS and DHCP ports, and the new one will fail to start up. If we add an option to dnsmasq which takes a process-ID and sends a SIGTERM to that process-ID as step 2.5, the old dnsmasq process can continue to run during the parsing of the options, then the it gets toen down before opening listening sockets. The restart now just runs the new dnsmasq instance, passing the PID of the old dnsmasq instance if it exists. Ok understood. Interesting it raises 3 questions on my side: 1- say the old process is already handling 1M blocked domains, I suppose this means that for the time of the restarting the RAM demand will be pushed to the equivalent of 2M domains (1M old + 1M new), right? 2- this will certainly help the restart but there's nothing that can save us from the first ever start after a device reboot I suppose 3- Is there any option to hand over the existing resolution cache between the "leaver" and "joiner" process? Like a memory mapping swap or something? ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
On 27/03/2022 08:52, Ercolino de Spiacico wrote:
>> [ -f list.of.domain ] && { create the formatted configuration }
>>
>> would always fail at dnsmasq level if the list.of.domains doesn't exist
>> despite the file existence condition defined. I'm not sure this is
meant
>> to be and/or if there's a smart workaround that can be used.
>
> I don't understand this, could you give more details.
Ok I have progressed on this point since. Basically I worked out the
dnsmasq is checking the script exit code. as I had something like this
in the script:
[ -f list.of.domain ] && { create the formatted configuration }
I just had to "force" the exit 0 to the command so:
[ -f list.of.domain ] && { create the formatted configuration } || {
exit 0; }
This makes the script returning exit 0 regardless and have --test
ignoring (rightly) any issue with that part of the script. Wanted so
good enough for me.
> Note that having loaded the config, dnsmasq has to sort all the domains,
> which will be part of the time taken. Doing that once makes the lookups
> much faster.
As part of the adblock.domain formatting I feed it to dnsmasq after a
sort -u so somehow it's already sorted, Could this internal sorting be
optionally skipped?
Not safely. If it's not sorted or sorted wrong, dnsmasq will break.
> How long are you without DNS service?
>
It's proportional to the number of records in the adblock file. For
large lists, on a fast router it can easy go into the 15+ seconds. This
happens relatively frequently on a router where external conditions
restart dnsmasq "just in case". So answering your question in detail,
given a relatively basic scripting config:
###
DOMAINS="/tmp/adblock.domains"
/usr/bin/wget --no-check-certificate -T 15 -q -U "Mozilla/5.0 (X11;
Linux x86_64; rv:10.0) Gecko/20100101 Firefox/98.0.1" -O-
https://hosts.oisd.nl | grep -Ev
'^#.*|^!.*|^::|^\s*?$|^([a-f0-9:]+:+)+[a-f0-9]+' | grep -Eo
'((([a-zA-Z]{1,2})|([0-9]{1,2})|([a-zA-Z0-9]{1,2})|([a-zA-Z0-9][a-zA-Z0-9-]{1,61}[a-zA-Z0-9]))\.)+[a-zA-Z]{2,6}'
| sort -u > $DOMAINS
gzip /tmp/adblock.domains
echo "script-conf=/tmp/script.sh" >> /etc/dnsmasq.conf
echo -e "#!/bin/sh\nset -e\n[ -f ${DOMAINS} ] && /bin/zcat ${DOMAINS}.gz
| /bin/sed -e \"s:^:address=/:\" -e \"s:$:/:\" || exit 0" > /tmp/script.sh
chmod 777 /tmp/script.sh
###
At this point restarting should give you a magnitude of time. Of course
on a fast PC/VM this might not be that much, on a router where in the
best of the cases you have 2x 800/1000MHz the delay is surely noticeable.
This Internet list above (https://hosts.oisd.nl) is 40MB uncompressed,
the regex extracts domains-only so shrinking it to 60% of its original
size and the gzip compression shrinks much further. Decompressing and
scripting it up of course takes time.
All of that looks like stuff which can be done before stopping dnsmasq,
right. SO how long it takes makes no difference to how long DNS and DHCP
service is interrupted for?
> One possible solution is to add an option to dnsmasq which causes it to
> send SIGTERM is a process-id _after_ reading config and _before_ opening
> network sockets. That would delay stopping the old dnsmasq process until
> the new one is almost ready to go. Would require some clever scripting
> in the init system or systemd to make it work.
Right! Could you please share more details on this idea? It could be a
smart workaround indeed.
When dnsmasq starts up, it does roughly the following things:
1) Read config files (including execute script-conf
2) organise the data from the config into in-memory data structures
3) Open listening sockets on DNS and DHCP ports.
4) Enter the event loop.
1) and 2) are likely to take an appreciaable amount of time with big
block lists. 3) and 4) never do.
A restart of dnsmasq consists of
1) send SIGTERM to existing dnsmasq process to cause it to halt.
2) Run new dnsmasq instance, which goes through the four steps above.
If reading the congfig takes time, that causes service interruption.
Note that you can't do those two steps in the opposite order, since the
old dnsmasq process will still be listening on the DNS and DHCP ports,
and the new one will fail to start up.
If we add an option to dnsmasq which takes a process-ID and sends a
SIGTERM to that process-ID as step 2.5, the old dnsmasq process can
continue to run during the parsing of the options, then the it gets toen
down before opening listening sockets.
The restart now just runs the new dnsmasq instance, passing the PID of
the old dnsmasq instance if it exists.
Cheers,
Simon.
Thanks
___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
Re: [Dnsmasq-discuss] Feature request = block-conf
>> [ -f list.of.domain ] && { create the formatted configuration }
>>
>> would always fail at dnsmasq level if the list.of.domains doesn't exist
>> despite the file existence condition defined. I'm not sure this is
meant
>> to be and/or if there's a smart workaround that can be used.
>
> I don't understand this, could you give more details.
Ok I have progressed on this point since. Basically I worked out the
dnsmasq is checking the script exit code. as I had something like this
in the script:
[ -f list.of.domain ] && { create the formatted configuration }
I just had to "force" the exit 0 to the command so:
[ -f list.of.domain ] && { create the formatted configuration } || {
exit 0; }
This makes the script returning exit 0 regardless and have --test
ignoring (rightly) any issue with that part of the script. Wanted so
good enough for me.
> Note that having loaded the config, dnsmasq has to sort all the domains,
> which will be part of the time taken. Doing that once makes the lookups
> much faster.
As part of the adblock.domain formatting I feed it to dnsmasq after a
sort -u so somehow it's already sorted, Could this internal sorting be
optionally skipped?
> How long are you without DNS service?
>
It's proportional to the number of records in the adblock file. For
large lists, on a fast router it can easy go into the 15+ seconds. This
happens relatively frequently on a router where external conditions
restart dnsmasq "just in case". So answering your question in detail,
given a relatively basic scripting config:
###
DOMAINS="/tmp/adblock.domains"
/usr/bin/wget --no-check-certificate -T 15 -q -U "Mozilla/5.0 (X11;
Linux x86_64; rv:10.0) Gecko/20100101 Firefox/98.0.1" -O-
https://hosts.oisd.nl | grep -Ev
'^#.*|^!.*|^::|^\s*?$|^([a-f0-9:]+:+)+[a-f0-9]+' | grep -Eo
'((([a-zA-Z]{1,2})|([0-9]{1,2})|([a-zA-Z0-9]{1,2})|([a-zA-Z0-9][a-zA-Z0-9-]{1,61}[a-zA-Z0-9]))\.)+[a-zA-Z]{2,6}'
| sort -u > $DOMAINS
gzip /tmp/adblock.domains
echo "script-conf=/tmp/script.sh" >> /etc/dnsmasq.conf
echo -e "#!/bin/sh\nset -e\n[ -f ${DOMAINS} ] && /bin/zcat ${DOMAINS}.gz
| /bin/sed -e \"s:^:address=/:\" -e \"s:$:/:\" || exit 0" > /tmp/script.sh
chmod 777 /tmp/script.sh
###
At this point restarting should give you a magnitude of time. Of course
on a fast PC/VM this might not be that much, on a router where in the
best of the cases you have 2x 800/1000MHz the delay is surely noticeable.
This Internet list above (https://hosts.oisd.nl) is 40MB uncompressed,
the regex extracts domains-only so shrinking it to 60% of its original
size and the gzip compression shrinks much further. Decompressing and
scripting it up of course takes time.
> One possible solution is to add an option to dnsmasq which causes it to
> send SIGTERM is a process-id _after_ reading config and _before_ opening
> network sockets. That would delay stopping the old dnsmasq process until
> the new one is almost ready to go. Would require some clever scripting
> in the init system or systemd to make it work.
Right! Could you please share more details on this idea? It could be a
smart workaround indeed.
Thanks
___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
On 24/03/2022 09:28, Ercolino de Spiacico wrote:
I've just added it to 2.87test8
Please test and report back.
Ok after a first implementation on my test environment this is my feedback:
A- Great feature! the scripting + gzip + zcat is a godsend. Yes it
pushed the CPU up a bit, yes the the script takes longer to execute
(although this is once/day in the early morning so not important), but
most importantly the RAM demand decreased drastically.
B- Since conf-file and conf-script make dnsmasq fail if they are defined
but the target file doesn't exist I have found a simple workaround to
just touch the all the target files at the top of my adblock script and
that does the job nicely.
Now the potential further points of improvements I could identify are:
1- Within the conf-script target file I map the list of domains as
discussed e.g.:
address=/ + $domain + /
For some reason a condition like this:
[ -f list.of.domain ] && { create the formatted configuration }
would always fail at dnsmasq level if the list.of.domains doesn't exist
despite the file existence condition defined. I'm not sure this is meant
to be and/or if there's a smart workaround that can be used.
I don't understand this, could you give more details.
2- I have tried to push this into the million of records and I can
confirm it still perform very well. There one down side though, the time
needed to restart dnsmasq increased visibly. Somehow it takes too long.
I guess the conf-script is run at every restart or relevant SIGHUP
It will run during restart,, but not at SIGHUP.
given. Suppose I'm aware the scripting part is not modified is it worthy
allowing an extra parameter to the conf-script to retain this specific
part of the config? e.g.
script-conf=0,populate-config.sh # do not recreate as part of dnsmasq
restart
script-conf=1,populate-config.sh # (or "1," not defined) always recreate
as part of restart
Not really practical. The config only exists in the the memory of the
old dnsmasq process which is discarded as part of the restart.
This somehow also introduced a new minor feature request like allowing
to detach the conf-script directive (creation of) from any SIGHUP and
allow them to fully run in async. So that we could reload the dnsmasq
config without necessarily run the script or all the way around
regenerate the script manually simply asking dnsmasq to reload the
(pre-generated) scripted config.
I'm not so much concerned about the adblock script as this ideally will
run once/day; but I'm rather concerned about external factors triggering
a dnsmasq restart multiple times/day. This potentially can be very
disruptive. If this point 2) of mine is a possibility I see that as a
perfect solution as there's full control on what to trigger and when
without noticeable dnsmasq downtime.
Note that having loaded the config, dnsmasq has to sort all the domains,
which will be part of the time taken. Doing that once makes the lookups
much faster.
How long are you without DNS service?
One possible solution is to add an option to dnsmasq which causes it to
send SIGTERM is a process-id _after_ reading config and _before_ opening
network sockets. That would delay stopping the old dnsmasq process until
the new one is almost ready to go. Would require some clever scripting
in the init system or systemd to make it work.
Simon.
Thanks
___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
I've just added it to 2.87test8
Please test and report back.
Ok after a first implementation on my test environment this is my feedback:
A- Great feature! the scripting + gzip + zcat is a godsend. Yes it
pushed the CPU up a bit, yes the the script takes longer to execute
(although this is once/day in the early morning so not important), but
most importantly the RAM demand decreased drastically.
B- Since conf-file and conf-script make dnsmasq fail if they are defined
but the target file doesn't exist I have found a simple workaround to
just touch the all the target files at the top of my adblock script and
that does the job nicely.
Now the potential further points of improvements I could identify are:
1- Within the conf-script target file I map the list of domains as
discussed e.g.:
address=/ + $domain + /
For some reason a condition like this:
[ -f list.of.domain ] && { create the formatted configuration }
would always fail at dnsmasq level if the list.of.domains doesn't exist
despite the file existence condition defined. I'm not sure this is meant
to be and/or if there's a smart workaround that can be used.
2- I have tried to push this into the million of records and I can
confirm it still perform very well. There one down side though, the time
needed to restart dnsmasq increased visibly. Somehow it takes too long.
I guess the conf-script is run at every restart or relevant SIGHUP
given. Suppose I'm aware the scripting part is not modified is it worthy
allowing an extra parameter to the conf-script to retain this specific
part of the config? e.g.
script-conf=0,populate-config.sh # do not recreate as part of dnsmasq
restart
script-conf=1,populate-config.sh # (or "1," not defined) always recreate
as part of restart
This somehow also introduced a new minor feature request like allowing
to detach the conf-script directive (creation of) from any SIGHUP and
allow them to fully run in async. So that we could reload the dnsmasq
config without necessarily run the script or all the way around
regenerate the script manually simply asking dnsmasq to reload the
(pre-generated) scripted config.
I'm not so much concerned about the adblock script as this ideally will
run once/day; but I'm rather concerned about external factors triggering
a dnsmasq restart multiple times/day. This potentially can be very
disruptive. If this point 2) of mine is a possibility I see that as a
perfect solution as there's full control on what to trigger and when
without noticeable dnsmasq downtime.
Thanks
___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
Ercolino:I can't speak for Simon and the rest of the Dnsmasq team (mostly because I'm not on it) but I appreciate your discussion and explanation of your need. I would have responded sooner, but I've had a medical emergency with my wife and was off the net for a few days being with family in the hospital.Now your comparison to the state of TFTP in my judgement isn't of the same caliber. If the TFTP root is not present then the only issue is that a handful of netbooting clients wont work at all, and you'll get immediate feedback (on an impacted system) that you broke something, AND anything that booted on its own will be fine.If the supplemental config script were to not be present and skipped, you wouldnt get the immediate feedback that something wasn't working, AND you couldn't guarantee a safe state for the server instance.It seems to me that you have a legitimate issue, but there are other ways to implement what you need to happen that don't require changing Dnsmasq at all.1) manipulating the boot order such that Dnsmasq starts AFTER the USB subsystem is loaded and the supplemental file system is mounted.2) The file system on the embedded device shouldn't be read-only and you should be able to copy the supplemental config script from the USB key to the root filesystem of the device and then it would be available when the system booted and your mount sequencing issue would go away.RanceOn Mar 4, 2022, at 2:52 PM, Ercolino de Spiacico wrote:>How does dnsmasq behave if there is a configuration error in the config >file elsewhere? If the syntax is broken then it fails hard. Don't see >why this wouldn't be true of a suplemental config script being referred >to in the main one.And as to --fail-safe: I don't see how this is >reasonable, as it will lead to undesirable operation and possibly even >broken clients if the mistake includes part of the dhcp >configuration.Its annoying, but probably better for services not to >start if they can't interpret/understand their starting statI appreciate the reason why this was originally designed to be the default behavior however please allow me: this conf-script might be is another beast.I'm on a router developing this, the dnsmasq config is read at boot from the content of a nvram variable. By the time dnsmasq starts I must already have this conf-script target created, the USB mounting comes way after everything else and the script booting process is screwed; NTP doesn't sync, clients don't get an IP... you name it. Also if the device has no USB this needs to be referenced and created in /tmp (RAM) at boot, this is via the init script that again is coming in a bit too late in the SoE. Until this file is created dnsmasq fails. Moreover there's an additional risk here, part of the config content is coming from Internet so outside the administrative domain. A typo by the list maintainer might cause havoc, most importantly, this is not necessary when the device is initially set up, it can come after months and affect a large number of devices at one.I really don't want to sound insistent but let me put it this way, long time ago I brought up this very topic in the context of TFTP. If the destination folder of TFTP didn't exist it used to fail dnsmasq (big time on a router). Then fortunately the tftp-no-fail directive was introduced.This conf-script is pretty much the same case but in a different context. If this extra info here above is still not enough I'll drop the ball, but I'm just making a final effort because I see value in it, that's all.Regards___Dnsmasq-discuss mailing listdnsmasq-disc...@lists.thekelleys.org.ukhttps://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
On 04/03/2022 20:12, Ercolino de Spiacico wrote: >How does dnsmasq behave if there is a configuration error in the config >file elsewhere? If the syntax is broken then it fails hard. Don't see >why this wouldn't be true of a suplemental config script being referred >to in the main one.And as to --fail-safe: I don't see how this is >reasonable, as it will lead to undesirable operation and possibly even >broken clients if the mistake includes part of the dhcp >configuration.Its annoying, but probably better for services not to >start if they can't interpret/understand their starting stat I appreciate the reason why this was originally designed to be the default behavior however please allow me: this conf-script might be is another beast. I'm on a router developing this, the dnsmasq config is read at boot from the content of a nvram variable. By the time dnsmasq starts I must already have this conf-script target created, the USB mounting comes way after everything else and the script booting process is screwed; NTP doesn't sync, clients don't get an IP... you name it. Also if the device has no USB this needs to be referenced and created in /tmp (RAM) at boot, this is via the init script that again is coming in a bit too late in the SoE. Until this file is created dnsmasq fails. Moreover there's an additional risk here, part of the config content is coming from Internet so outside the administrative domain. A typo by the list maintainer might cause havoc, most importantly, this is not necessary when the device is initially set up, it can come after months and affect a large number of devices at one. I really don't want to sound insistent but let me put it this way, long time ago I brought up this very topic in the context of TFTP. If the destination folder of TFTP didn't exist it used to fail dnsmasq (big time on a router). Then fortunately the tftp-no-fail directive was introduced. This conf-script is pretty much the same case but in a different context. If this extra info here above is still not enough I'll drop the ball, but I'm just making a final effort because I see value in it, that's all. If the conf-script fails in a non-fatal manner, it has the option to return a zero exit code. It must be the case that a conf-script should be able to cause an abort on start up, but it has contrl over that. Another option would be to use the --test flag in the startup script. if dnsmasq --test --conf-script=. fails, then start up dnsmasq without the --conf-script option, maybe? Simon. Regards ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
>How does dnsmasq behave if there is a configuration error in the config >file elsewhere? If the syntax is broken then it fails hard. Don't see >why this wouldn't be true of a suplemental config script being referred >to in the main one.And as to --fail-safe: I don't see how this is >reasonable, as it will lead to undesirable operation and possibly even >broken clients if the mistake includes part of the dhcp >configuration.Its annoying, but probably better for services not to >start if they can't interpret/understand their starting stat I appreciate the reason why this was originally designed to be the default behavior however please allow me: this conf-script might be is another beast. I'm on a router developing this, the dnsmasq config is read at boot from the content of a nvram variable. By the time dnsmasq starts I must already have this conf-script target created, the USB mounting comes way after everything else and the script booting process is screwed; NTP doesn't sync, clients don't get an IP... you name it. Also if the device has no USB this needs to be referenced and created in /tmp (RAM) at boot, this is via the init script that again is coming in a bit too late in the SoE. Until this file is created dnsmasq fails. Moreover there's an additional risk here, part of the config content is coming from Internet so outside the administrative domain. A typo by the list maintainer might cause havoc, most importantly, this is not necessary when the device is initially set up, it can come after months and affect a large number of devices at one. I really don't want to sound insistent but let me put it this way, long time ago I brought up this very topic in the context of TFTP. If the destination folder of TFTP didn't exist it used to fail dnsmasq (big time on a router). Then fortunately the tftp-no-fail directive was introduced. This conf-script is pretty much the same case but in a different context. If this extra info here above is still not enough I'll drop the ball, but I'm just making a final effort because I see value in it, that's all. Regards ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
You are most welcome. Sent from my iPhone > On Mar 4, 2022, at 12:19 PM, Simon Kelley wrote: > > Thanks Rance, you saved me from writing the same answer. > > Simon. > > >> On 04/03/2022 17:00, Rance Hall via Dnsmasq-discuss wrote: >> How does dnsmasq behave if there is a configuration error in the config file >> elsewhere? If the syntax is broken then it fails hard. Don't see why this >> wouldn't be true of a suplemental config script being referred to in the >> main one. >> And as to --fail-safe: I don't see how this is reasonable, as it will lead >> to undesirable operation and possibly even broken clients if the mistake >> includes part of the dhcp configuration. >> Its annoying, but probably better for services not to start if they can't >> interpret/understand their starting state. >> Rance On Mar 4, 2022, at 4:16 AM, Ercolino de Spiacico wrote: >>> >>> >>> > I've just added it to 2.87test8 >>> >>> > Please test and report back. >>> >>> >>> >>> I'm finally managed to find a way to build from sources. One initial >>> feedback: >>> >>> I cross referenceed the conf script e.g. >>> >>> conf-scrip=/tmp/adblock-expander.sh >>> >>> If the file doesn't exists or has a broken syntax it will make the whole >>> dnsmasq process failing with a message like "/tmp/adblock-expander.sh >>> returns a non 0 exit code something" >>> >>> This is perhaps a wider topic and goes a bit out of scope for this >>> feature request, but perhaps we should: >>> >>> 1) remove this error control for conf-script and simply log+skip errors >>> rather than crash land the whole dnsmasq. >>> >>> 2) perhaps introducing a new "--fail-safe" option for dnsmasq to extend >>> point 1) to any broken directive in the configuration >>> >>> >>> Thanks! >>> >>> ___ >>> Dnsmasq-discuss mailing list >>> Dnsmasq-discuss@lists.thekelleys.org.uk >>> https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss >> ___ >> Dnsmasq-discuss mailing list >> Dnsmasq-discuss@lists.thekelleys.org.uk >> https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss > > ___ > Dnsmasq-discuss mailing list > Dnsmasq-discuss@lists.thekelleys.org.uk > https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
Thanks Rance, you saved me from writing the same answer. Simon. On 04/03/2022 17:00, Rance Hall via Dnsmasq-discuss wrote: How does dnsmasq behave if there is a configuration error in the config file elsewhere? If the syntax is broken then it fails hard. Don't see why this wouldn't be true of a suplemental config script being referred to in the main one. And as to --fail-safe: I don't see how this is reasonable, as it will lead to undesirable operation and possibly even broken clients if the mistake includes part of the dhcp configuration. Its annoying, but probably better for services not to start if they can't interpret/understand their starting state. Rance On Mar 4, 2022, at 4:16 AM, Ercolino de Spiacico wrote: > I've just added it to 2.87test8 > Please test and report back. I'm finally managed to find a way to build from sources. One initial feedback: I cross referenceed the conf script e.g. conf-scrip=/tmp/adblock-expander.sh If the file doesn't exists or has a broken syntax it will make the whole dnsmasq process failing with a message like "/tmp/adblock-expander.sh returns a non 0 exit code something" This is perhaps a wider topic and goes a bit out of scope for this feature request, but perhaps we should: 1) remove this error control for conf-script and simply log+skip errors rather than crash land the whole dnsmasq. 2) perhaps introducing a new "--fail-safe" option for dnsmasq to extend point 1) to any broken directive in the configuration Thanks! ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
How does dnsmasq behave if there is a configuration error in the config file elsewhere? If the syntax is broken then it fails hard. Don't see why this wouldn't be true of a suplemental config script being referred to in the main one.And as to --fail-safe: I don't see how this is reasonable, as it will lead to undesirable operation and possibly even broken clients if the mistake includes part of the dhcp configuration.Its annoying, but probably better for services not to start if they can't interpret/understand their starting state.RanceOn Mar 4, 2022, at 4:16 AM, Ercolino de Spiacico wrote:> I've just added it to 2.87test8> Please test and report back.I'm finally managed to find a way to build from sources. One initial feedback:I cross referenceed the conf script e.g.conf-scrip=/tmp/adblock-expander.shIf the file doesn't exists or has a broken syntax it will make the whole dnsmasq process failing with a message like "/tmp/adblock-expander.sh returns a non 0 exit code something"This is perhaps a wider topic and goes a bit out of scope for this feature request, but perhaps we should:1) remove this error control for conf-script and simply log+skip errors rather than crash land the whole dnsmasq.2) perhaps introducing a new "--fail-safe" option for dnsmasq to extend point 1) to any broken directive in the configurationThanks!___Dnsmasq-discuss mailing listdnsmasq-disc...@lists.thekelleys.org.ukhttps://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
> I've just added it to 2.87test8 > Please test and report back. I'm finally managed to find a way to build from sources. One initial feedback: I cross referenceed the conf script e.g. conf-scrip=/tmp/adblock-expander.sh If the file doesn't exists or has a broken syntax it will make the whole dnsmasq process failing with a message like "/tmp/adblock-expander.sh returns a non 0 exit code something" This is perhaps a wider topic and goes a bit out of scope for this feature request, but perhaps we should: 1) remove this error control for conf-script and simply log+skip errors rather than crash land the whole dnsmasq. 2) perhaps introducing a new "--fail-safe" option for dnsmasq to extend point 1) to any broken directive in the configuration Thanks! ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
> I've just added it to 2.87test8 > Please test and report back. I unfortunately don't have a build environment, so I hope someone else here can help with the testing... Looking forward to the outcome. ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
> On Feb 6, 2022, at 15:55, Ercolino de Spiacico > wrote: > > From what version is this --conf-script made available? > > root@router:/tmp# dnsmasq -v > Dnsmasq version 2.86 Copyright (c) 2000-2021 Simon Kelley > Compile time options: IPv6 GNU-getopt no-RTC no-DBus no-UBus no-i18n no-IDN > DHCP DHCPv6 no-Lua TFTP no-conntrack ipset Tomato-helper auth cryptohash > DNSSEC no-ID loop-detect inotify no-dumpfile > > This software comes with ABSOLUTELY NO WARRANTY. > Dnsmasq is free software, and you are welcome to redistribute it > under the terms of the GNU General Public License, version 2 or 3. > root@router:/tmp# dnsmasq --help | grep conf-script > root@router:/tmp# > > ___ > Dnsmasq-discuss mailing list > Dnsmasq-discuss@lists.thekelleys.org.uk > https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss I don’t think it is actually available. I believe they were discussing the idea of such a feature. ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
From what version is this --conf-script made available? root@router:/tmp# dnsmasq -v Dnsmasq version 2.86 Copyright (c) 2000-2021 Simon Kelley Compile time options: IPv6 GNU-getopt no-RTC no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset Tomato-helper auth cryptohash DNSSEC no-ID loop-detect inotify no-dumpfile This software comes with ABSOLUTELY NO WARRANTY. Dnsmasq is free software, and you are welcome to redistribute it under the terms of the GNU General Public License, version 2 or 3. root@router:/tmp# dnsmasq --help | grep conf-script root@router:/tmp# ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
On 06/02/2022 05:54, Ercolino de Spiacico wrote: > Opinions, all? The scripting solution would indeed solve the original feature request in my view. But there's a new point that is now coming into scope. Just to give the magnitude here I came across lists with 1.2M domains, 40MB uncompressed pre 'address=//' formatting. Loading such a list consumes about 100MB of RAM. For the sake of ads/domain blocking I have identified two metrics: a) storage b) RAM We normally have two common scenarios: 1) routers without mass storage 2) routers with mass storage available (e.g. USB) for case 1) the script solution would help both a) and partially b) as storage is essentially RAM on those devices. for case 2) though only a) is affected and somehow not that important here. Using the script solution I have the feeling that we are still going to get the full syntax 'address=/domain.com/' in RAM. But if it was dnsmasq to give a different interpretation to the cached data though, so that only domains or perhaps a very shorten syntax (e.g. -domain.com) can be used, I would expect this to result into a demand reduction on b). No. The RAM usage once the rule is parsed is always the same. (And you really need to be using 2.86 or later, which optimised both RAM and CPU for large lists.) Simon. ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
> Opinions, all? The scripting solution would indeed solve the original feature request in my view. But there's a new point that is now coming into scope. Just to give the magnitude here I came across lists with 1.2M domains, 40MB uncompressed pre 'address=//' formatting. Loading such a list consumes about 100MB of RAM. For the sake of ads/domain blocking I have identified two metrics: a) storage b) RAM We normally have two common scenarios: 1) routers without mass storage 2) routers with mass storage available (e.g. USB) for case 1) the script solution would help both a) and partially b) as storage is essentially RAM on those devices. for case 2) though only a) is affected and somehow not that important here. Using the script solution I have the feeling that we are still going to get the full syntax 'address=/domain.com/' in RAM. But if it was dnsmasq to give a different interpretation to the cached data though, so that only domains or perhaps a very shorten syntax (e.g. -domain.com) can be used, I would expect this to result into a demand reduction on b). ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
> On Feb 5, 2022, at 5:32 AM, Simon Kelley wrote: > > Let's try thinking out of the box here. Given the motivation to save storage, > I was wondering if there could be a way to use compression, gzip etc to save > more space. > > Building a decompressor into dnsmasq seems ugly, but then I came up with the > option of --conf-script=/path/to/script. The 'zcat' utility is ubiquitous, no command-line options required ... could be called by dnsmasq directly with a "block-conf" file suffix of .gz, .bz2 or .xz . Also 'zcat' returns non-zero for "no gzip/bzip2/xz magic" > Opinions, all? Consider processing a "block-conf" file (no text other than one domain/line) and with the proper file suffix pre-processed using 'zcat'. Lonnie ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
On Sat, Feb 05, 2022 at 11:32:44AM +, Simon Kelley wrote: > On 04/02/2022 09:46, Ercolino de Spiacico wrote: > > I have a feature request. > > > > Since dnsmasq is extremely popular on embedded devices with limited > > resources (e.g. router) it would be a huge improvement to have a new > > directive to block a list of domains. > > > > > > > > Could we please have a new directive e.g. block-file > > > > block-conf=/etc/dnsmasq.adblock > > > > where only domains are needed in the mapped file? > > > > baddomain.com > > baddomain2.com > > > > > > This would reduce drastically the demand on storage (so memory). Some of > > the public adblock lists are huge hence having the adblock file > > literally 1/2 of the original size would be amazing. > > > Let's try thinking out of the box here. Given the motivation to save > storage, I was wondering if there could be a way to use compression, gzip > etc to save more space. > > Building a decompressor into dnsmasq seems ugly, but then I came up with the > option of --conf-script=/path/to/script. > > Analogous to --conf-file, except that the standard output of the script is > used as the configuration. > > > So you could just have a script which consisted of > > #!/bin/sh > set -e > > gunzip block-domains.gz | sed -e "s:^:address=/:" -e "s:$:/#:" > > > The sed stuff might not be necessary: a decent compression algo should hide > the repeated information pretty successfully itself. > > > That solves you problem, and is more generally useful: It's also pretty > trivial to implement. > > Opinions, all? > Another effect of `--conf-script` is that the script could be ``` #!/bin/sh set -e curl http://server.lan:4242/config/magic ``` Groeten Geert Stappers -- Silence is hard to parse ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Feature request = block-conf
Let's try thinking out of the box here. Given the motivation to save storage, I was wondering if there could be a way to use compression, gzip etc to save more space. Building a decompressor into dnsmasq seems ugly, but then I came up with the option of --conf-script=/path/to/script. Analogous to --conf-file, except that the standard output of the script is used as the configuration. So you could just have a script which consisted of #!/bin/sh set -e gunzip block-domains.gz | sed -e "s:^:address=/:" -e "s:$:/#:" The sed stuff might not be necessary: a decent compression algo should hide the repeated information pretty successfully itself. That solves you problem, and is more generally useful: It's also pretty trivial to implement. Opinions, all? Simon. On 04/02/2022 09:46, Ercolino de Spiacico wrote: I have a feature request. Since dnsmasq is extremely popular on embedded devices with limited resources (e.g. router) it would be a huge improvement to have a new directive to block a list of domains. Practical example: We currently need to add an external file to the main config (preferred) e.g.: conf-file=/etc/dnsmasq.adblock and define the syntax in a relatively long way address=/baddomain.com/# address=/baddomain2.com/# Suggestion - Could we please have a new directive e.g. block-file block-conf=/etc/dnsmasq.adblock where only domains are needed in the mapped file? baddomain.com baddomain2.com This would reduce drastically the demand on storage (so memory). Some of the public adblock lists are huge hence having the adblock file literally 1/2 of the original size would be amazing. Thanks ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
