Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
Hi Simon, I am happy to tell you that our new open source IaaS project -- ZStack (http://zstack.org/) is released. Now with a single networking node (we called Virtual Router VM) with Dnsmasq, we can achieve creating 10,000 VMs within 23 minutes; and with a patched Dnsmasq based on your work of inotify, we can even improve the performance to 11 minutes! Dnsmasq is awesome! We can't archive such a good performance, without Dnsmasq. We will continue working with Dnsmasq community closely to improve our project and tell you our thoughts about Dnsmasq. Your suggestions for our project are always welcome! Thank you. On Wed, Feb 18, 2015 at 4:59 AM, Simon Kelley si...@thekelleys.org.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 16/02/15 06:56, Yongkang You wrote: Hi Simon, Can you tell me if `dhcp_release eth1 ip mac` is still needed after send SIGHUP, or why `dhcp_release eth1 ip mac` can't release the binding without send SIGHUP? Changing the dnsmasq configuration, specifically dhcp-host entries, won't affect any current DHCP leases, so if you want to end a DHCP lease early (maybe to re-use the address) then you'll need to call use dhcp_release. That can be before or after sending SIGHUP it doesn't matter. BTW, I am not using openstack. Sorry for the confusion. My mistake sorry, I saw talk about VMs and assumed openstack. I'd be interested to hear more about the system you are working on, on or off list. If you need to have configuration deleted automatically when the file is deleted using inotify, I can look at doing that. It's significant extra complexity, so it makes sense to determine if it's needed first. Cheers, Simon. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Mon, Feb 16, 2015 at 6:49 AM, Simon Kelley si...@thekelleys.org.uk wrote: The current code doesn't do anything when a file is deleted. To make that have an affect, you're back to sending SIGHUP to clean out the data and re-read the current set of files. Okay. I'm pretty sure I mentioned that way back. If deleting lots of VMs one-at-time is something that will happen, then we need to look at solving that, but my impressiion is that it's not something openstack does often? Am I wrong? I don't know if openstack does that or not. If that's the right solution, I will follow it. Anyway, thanks for the inotify feature. It is great! -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
Hi Simon, Can you tell me if `dhcp_release eth1 ip mac` is still needed after send SIGHUP, or why `dhcp_release eth1 ip mac` can't release the binding without send SIGHUP? BTW, I am not using openstack. Sorry for the confusion. Thanks, Yongkang On Mon, Feb 16, 2015 at 1:23 PM, Yongkang You yongkang@gmail.com wrote: On Mon, Feb 16, 2015 at 6:49 AM, Simon Kelley si...@thekelleys.org.uk wrote: The current code doesn't do anything when a file is deleted. To make that have an affect, you're back to sending SIGHUP to clean out the data and re-read the current set of files. Okay. I'm pretty sure I mentioned that way back. If deleting lots of VMs one-at-time is something that will happen, then we need to look at solving that, but my impressiion is that it's not something openstack does often? Am I wrong? I don't know if openstack does that or not. If that's the right solution, I will follow it. Anyway, thanks for the inotify feature. It is great! -- Yongkang You -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Sun, Feb 15, 2015 at 4:11 AM, Simon Kelley si...@thekelleys.org.uk wrote: Got it. Do you have --no-hosts set? That would explain why it was working for me and not for you. I just pushed the fix to git. Your guessing is completely right! I am using no-hosts in dnsmasq.conf. Now the new codes work greatly in my environment. But another issue I found is, the IP binding can't be really released even after remove the config files and manually call dhcp_release. So when I create another VM with the same IP address, it will report duplication. dnsmasq.log is something like: --- Feb 15 06:12:45 dnsmasq-dhcp[3521]: DHCPRELEASE(eth2) 10.11.0.3 fa:e1:51:0b:a5:00 ... Feb 15 06:13:13 dnsmasq[3521]: inotify, new or changed file /etc/hosts.dhcps//fa:e9:4a:40:7f:00 Feb 15 06:13:13 dnsmasq-dhcp[3521]: read /etc/hosts.dhcps//fa:e9:4a:40:7f:00 Feb 15 06:13:13 dnsmasq[3521]: duplicate dhcp-host IP address 10.11.0.3 at line 1 of /etc/hosts.dhcps//fa:e9:4a:40:7f:00 Feb 15 06:13:23 dnsmasq-dhcp[3521]: DHCPDISCOVER(eth2) fa:e9:4a:40:7f:00 no address available Feb 15 06:13:26 dnsmasq-dhcp[3521]: DHCPDISCOVER(eth2) fa:e9:4a:40:7f:00 no address available --- Anything wrong? -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Mon, Feb 9, 2015 at 11:49 PM, Simon Kelley si...@thekelleys.org.uk wrote: You should see something like dnsmasq: inotify, new or changed file /tmp/hosts/testone dnsmasq: read /tmp/hosts/testone - 1 addresses Just to check the argument to hostsdir is a directory, and not a file? Any errors logged at startup? Simon, I used the config like this: -dnsmasq.conf-- ... dhcp-hostsdir=/etc/hosts.dhcps/ dhcp-optsdir=/etc/hosts.options/ hostsdir=/etc/hosts.dnss/ ... --end- -bash-4.1# ls /etc/hosts.dnss/ fa:cd:42:cf:50:00 fa:ed:12:0d:ad:00 -bash-4.1# cat /etc/hosts.dnss/* 10.11.30.33 10-11-30-33.test1.org 10.11.109.127 10-11-109-127.test1.org From the dnsmasq.log: -bash-4.1# tail -f /var/log/dnsmasq.log Feb 10 15:49:56 dnsmasq[24347]: started, version 2.73test5-12-g2941d3a cachesize 150 Feb 10 15:49:56 dnsmasq[24347]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify Feb 10 15:49:56 dnsmasq-dhcp[24347]: DHCP, static leases only on 10.11.0.1, lease time 1h Feb 10 15:49:56 dnsmasq[24347]: reading /etc/resolv.conf Feb 10 15:49:56 dnsmasq[24347]: using nameserver 10.0.101.1#53 Feb 10 15:49:56 dnsmasq[24347]: cleared cache Feb 10 15:49:56 dnsmasq-dhcp[24347]: read /etc/hosts.options//fa:ed:12:0d:ad:00 Feb 10 15:49:56 dnsmasq-dhcp[24347]: read /etc/hosts.options//fa:cd:42:cf:50:00 Feb 10 15:49:56 dnsmasq-dhcp[24347]: read /etc/hosts.dhcps//fa:ed:12:0d:ad:00 Feb 10 15:49:56 dnsmasq-dhcp[24347]: read /etc/hosts.dhcps//fa:cd:42:cf:50:00 Feb 10 15:50:27 dnsmasq[24347]: inotify, new or changed file /etc/hosts.dhcps//fa:ec:59:d0:15:00 Feb 10 15:50:27 dnsmasq-dhcp[24347]: read /etc/hosts.dhcps//fa:ec:59:d0:15:00 Feb 10 15:50:27 dnsmasq[24347]: inotify, new or changed file /etc/hosts.options//fa:ec:59:d0:15:00 Feb 10 15:50:27 dnsmasq-dhcp[24347]: read /etc/hosts.options//fa:ec:59:d0:15:00 The last 4 lines show a new VM is added, configs in dhcp-hostsdir and dhcp-optsdir are found. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Sun, Feb 8, 2015 at 6:35 AM, Simon Kelley si...@thekelleys.org.uk wrote: Are you using the new config keywords dhcp-hostsdir dhcp-optsdir hostsdir These are equivalent to dhcp-hostfile, dhcp-optsfile and addn-hosts when provided with a directory, except that they use inotify and don't need SIGUSR1. It's a bit of a pain to have yet more options, but the semantics are different than the existing ones, and we need to be able to error if someone asks for the new inotify behaviour when support isn't compiled in, or we're on a platform that doesn't support it. okay. After I switched to dhcp-hostsdir and dhcp-optsdir, the magic inotifies are coming! I can see related logs in dnsmasq.log and VM does receive IP address successfully. But hostsdir doesn't work. When new config file created, there isn't log tracked in dsmasq.log and VMs can not use hostname to ping each other. Even if I restart dnsmasq process, the config also work. The same config file works when using addn-hosts option, when reboot dnsmasq process. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Wed, Feb 4, 2015 at 5:54 AM, Simon Kelley si...@thekelleys.org.uk wrote: Done. I'm not sure how much logging is appropriate long term, but it's good to have at least for now. Well, actually nothing was trapped when new VM's config files are added in config folder. Could you please provide your dnsmasq.conf to me to try or do I need to pass some parameters to dnsmasq? I am worrying about if something wrong in my configurations. But those configs do work, if I just restart dnsmasq process. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Tue, Feb 3, 2015 at 6:38 AM, Simon Kelley si...@thekelleys.org.uk wrote: Could you try the latest code again? I just checked in a fix. Still doesn't work. I still need to restart dnsmasq to let the new VM to acquire IP address. How about adding some logs for tracking new added config files, it might be easily for me to figure out whether the service find the those new items. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Thu, Jan 22, 2015 at 12:10 AM, Yongkang You yongkang@gmail.com wrote: I am trying to do: 1. change dnsmasq.conf to: dhcp-hostsfile=/etc/hosts.dhcp-dir dhcp-optsfile=/etc/hosts.option-dir addn-hosts=/etc/hosts.dns-dir After I add new config files into above folders, will dnsmasq.log record some logs to track whether it is functional? -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Wed, Jan 21, 2015 at 4:57 AM, Simon Kelley si...@thekelleys.org.uk wrote: There's some code in the git repo now, or download the 2.73test6 tarball. I've added a new config option --dhcp-hostsdir, which reads all the files in a directory in the same was as --dhcp-hostsfile, but doesn't need SIGHUP to do it. Let me know how you get on. Hi Simon, I am trying. I want to confirm with you firstly what I am doing is correct. I was originally using following 3 config files for 1 VM's configuration in dnsmasq.conf: dhcp-hostsfile=/etc/hosts.dhcp dhcp-optsfile=/etc/hosts.option addn-hosts=/etc/hosts.dns The contents are something like: #cat /etc/hosts.dhcp fa:04:0a:d5:97:00,set:fa040ad59700,10.11.10.184,10-11-10-184.test.org ,infinite #cat /etc/hosts.option tag:fa040ad59700,option:router,10.11.0.1 tag:fa040ad59700,option:dns-server,10.11.0.1 tag:fa040ad59700,option:domain-name,test1.org tag:fa040ad59700,option:netmask,255.255.0.0 #cat /etc/hosts.dns 10.11.10.184 10-11-10-184.test1.org I am trying to do: 1. change dnsmasq.conf to: dhcp-hostsfile=/etc/hosts.dhcp-dir dhcp-optsfile=/etc/hosts.option-dir addn-hosts=/etc/hosts.dns-dir 2. put the VM's configurations into 3 files which named with VM's mac under corresponding config folders. e.g. #cat /etc/hosts.dhcp-dir/fa:04:0a:d5:97:00 fa:04:0a:d5:97:00,set:fa040ad59700,10.11.10.184,10-11-10-184.test.org ,infinite #cat /etc/hosts.option-dir/fa:04:0a:d5:97:00 tag:fa040ad59700,option:router,10.11.0.1 tag:fa040ad59700,option:dns-server,10.11.0.1 tag:fa040ad59700,option:domain-name,test1.org tag:fa040ad59700,option:netmask,255.255.0.0 #cat /etc/hosts.dns-dir/fa:04:0a:d5:97:00 10.11.10.184 10-11-10-184.test1.org And I will duplicate step #2 with 10k VMs. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Thu, Jan 22, 2015 at 4:48 AM, Simon Kelley si...@thekelleys.org.uk wrote: OK, the first problem: I didn't anticipate that you're using dhcp-optsfile, do there's no support for that in the code I checked in. That's easily fixed. After release new binary, I would love to try. There's no support for addn-hosts either. You're including the hostname on the dhcp-host stanza, so the hosts file may be redundant, but it does allow the hostname to resolve in the DNS even if the host doesn't have a DHCP lease. If that why you're using the hosts file? If We using leasefile-ro option in dnsmasq.conf. I am not sure if it means no DHCP lease, although we still have to call 'dhcp_release', after removing a VM's dns config. Is it a correct usage? so, I need to think about that a bit more. But we do need that files, since there are some hostname bindings will only appear in addn-hosts configs (and not in dhcp-host configs). Re: you second message. There's no logging for this, given we're trying to support lots of files, I didn't want to clog up the logs. If there are some debug option I can use to test, I could easily figure out if the new function works in my environment. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Thu, Jan 15, 2015 at 9:56 PM, Simon Kelley si...@thekelleys.org.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/01/15 07:47, Yongkang You wrote: Hi Simon, Thanks for replying. Batching the changes is a very good idea, but it is not easy to implement in IaaS, since the VMs creation processes are handled separately. From my observation, the burst 10k SIGUSER1 signals will cause dnsmasq occupying 100% CPUs for several hours. The workload comes from 10k reload events, although the 10k DHCP config entries have been successfully updated in first tens of minutes. Wow, that's a lot of CPU. How are you synchronising changes to the config files which you are re-reading? That last thing you want is race conditions where changes are missed, or half-written files read. Is there one file per VM (ie ~1 files, just some of which change)? we might have to get more smart about only readng stuff which changes, especially as I think there's processing going on at the moment which is O(n^2) in the number of configs. Cheers, Simon. Hi Simon, 10k VMs share same config file. The operation logic is like: 1. update the config file, by adding some specific mac/ip lines. (This step is mutual between different VM updating) 2. send SIGUSER1 to dnsmasq process. So, yes you are right. The step 2 might happen when some other VM is adding config. This might cause some problem. After all entries are recorded in config file, there might be 40k lines. Will it become slow when trigger SIGUSER1? -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
On Fri, Jan 16, 2015 at 7:24 PM, Simon Kelley si...@thekelleys.org.uk wrote: Here's one possible solution. dhcp-hostfile can take a directory instead of a filename, so instead of having one file with 40k lines in it, have a directory with a file for each host. (Aside, 40k files is a big directory, but it can be split into multiple directories, if needed). By itself, this will go backwards, since opening 40k small files to read them all will be even slower, so we need an addition to dnsmasq. Yeah, this could resolve the incomplete config file issue. But the slower issue is also bad. The addition to dnsmasq is to use inotify to detect when a file is added or written. The development branch of dnsmasq already has code to do this for resolv files. With this change, there's no need to us a signal, and no need to re-read all the files. When a file appears in the directory or is modified _and_is_closed_ dnsmasq will wake up and read just that file. Much better, this fixes the race condition too. It is very impressive! The simple way to implement this simply adds new configuration from new files, if a file gets changed or deleted, then the dhcp-hosts that were created by its old contents are not deleted, and SIGUSR1 is still needed to re-read the config from scratch and garbage collect old entries. More complex implementation would keep track which dhcp-hosts are associated with which files, and delete old versions when a file changes. I'd rather not have to do the more complex version, is it can be avoided. Comments? Thanks Simon! I will currently use some workaround and wait for the super inotify solution. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
Hi Simon, Thanks for replying. Batching the changes is a very good idea, but it is not easy to implement in IaaS, since the VMs creation processes are handled separately. From my observation, the burst 10k SIGUSER1 signals will cause dnsmasq occupying 100% CPUs for several hours. The workload comes from 10k reload events, although the 10k DHCP config entries have been successfully updated in first tens of minutes. I didn't read the code, but just guess. Do you think if there is any possibility that the event queue doesn't accept other reload events, when last reload event is not finished. On Thu, Jan 15, 2015 at 6:10 AM, Simon Kelley si...@thekelleys.org.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dnsmasq checks for incoming network packets and signals in the same event loop, so the maximum delay to DHCP packets will be the time to read a configuration file once, even if you send 1 SIGUSER1 signals, because the code will interleave handling DHCP and DNS packets with the config file reading. Batching changes would be good, if you can do it, but I'd not go looking for trouble here. Cheers, Simon. On 14/01/15 15:49, Brian Haley wrote: On 01/14/2015 04:48 AM, Yongkang You wrote: Hi, We have a cloud environment which uses DNSMASQ as DHCP/DNS server; in the cloud, it usually starts lots of VMs concurrently, for example, 1 VM; for each VM, we will configure a static DHCP entry in DNSMASQ configuration and send SIGUSER1 to reload. we observe DNSMASQ becomes very busy of reloading, for example, it seems to reload 1 times if we send for 1 SIGUSER1. Is this expected behavior? Is there any way to avoid this? because it keep reloading, we don't know if this will effect VM getting IP address. Just for completeness, is this on Openstack? I can see that code doesn't buffer anything when a port is plugged-in, so each triggers a reload. Could maybe tweak things there to wait a few seconds between the HUPs. I'm not sure it's a bug in dnsmasq though. -Brian ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss -BEGIN PGP SIGNATURE- Version: GnuPG v1 iEYEARECAAYFAlS26U0ACgkQKPyGmiibgrcZYACeIJqI+BnUB29af33HWFBTKlZ9 RZgAn1eTCs/D+jrXr7xTh1SNWqADAqn0 =hrhh -END PGP SIGNATURE- ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
[Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests
Hi, We have a cloud environment which uses DNSMASQ as DHCP/DNS server; in the cloud, it usually starts lots of VMs concurrently, for example, 1 VM; for each VM, we will configure a static DHCP entry in DNSMASQ configuration and send SIGUSER1 to reload. we observe DNSMASQ becomes very busy of reloading, for example, it seems to reload 1 times if we send for 1 SIGUSER1. Is this expected behavior? Is there any way to avoid this? because it keep reloading, we don't know if this will effect VM getting IP address. -- Yongkang You ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
