Hi Jerry, Thanks a lot for your long detailed email! My answers inline.
We are building a run-time orchestrater (honcho) that will manage the > running apps inside a kubernetes pod. We have large complex apps and we > often need to do various things that don't fit the typical "docker run" > model. Honcho also does all the monitoring of the app, so we will be using > the agent addr/port hooks for each host so that haproxy talks to honcho for > health checks but feeds connections to the app address/port. (thanks for > having the flexibility to do this.) When kubernetes starts up an instance > of an app, either a kubernetes hook or honcho will do a dynamic DNS insert > of the records into the zone for the data center. > > Is Honcho a DNS server by itself? by the way, is Honcho open source or a tool purely internal at your $WORKPLACE ? > TXT records are always a hack, but its the way I came up with to allow per > target information to get into haproxy from DNS. > > I did not plan to support TXT records soon... If we do it, we'll have to be very careful on the "language" or keywords or accepted format we'll use for these records. We use lots of alternate ports as well as the regulars, so we would expect > to use the nonstandard _portnum._tcp.name form as well as the _https._tcp > standard form. A couple examples of the names I expect to use: > > _12354._tcp.service1.datacenter.soundhound.com > _https._tcp.web2.datacenter.soundhound.com > > No problem with that. we don't care the first part and care only the name pointed by the record. > The entire record set might look like: > > ($origin datacenter.soundhound.com) > > _12354._tcp.service1 in SRV 10 50 23465 target1.service1 > target1.service1 in A 10.1.2.3 > t1honcho.service1 in A 10.1.2.2 > target1.service1._12354._tcp.service1 in TXT "addr t1honcho.service1" > "port 19999" > What type of query do you send to get such recordset??? Is it standard? I thought SRV records work differently, and would return something like: _12354._tcp.service1 in SRV 10 50 23465 10.1.2.3 _12354._tcp.service1 in SRV 10 50 23465 10.1.2.2 And the query would be an equivalent to "dig -t SRV service1" and HAProxy would configure 2 servers 10.1.2.3:23465 and 10.1.2.2:23465. In any case, HAProxy can (for now) autoconfigure a frontend or a bind at runtime. The config might look like this > > frontend service1fe > bind *.12354 > default_backend service1be > > backend service1be > txt_labels 2 # prepend this many labels from the name of the target to > look for txt records > server _12354._tcp.service1 > > You could even just do a one liner, since the srv record name specifies > what to list on and the targets. > listen xxx > srv _12354._tcp.service1 > > This would assume that you are listening on all interfaces... > > I don't understand the purpose of the txt_labels statement > One last thing. We want to be able to make it so as soon as the DNS zone > is updated, haproxy rereads the DNS for these. DNS has a feature called > notify where a zone master can tell the slaves to do zone transfer so you > can minimize the skew across a set of nameservers. > This feature is interesting! It means that HAProxy becomes a "slave" DNS server. Is this notification sent over UDP or TCP? (I noted it into my etherpad as something to keep in mind) > > As a simple first step, I can write a tiny python app that can receive DNS > notify messages when a zone is updated. It would then want to kick haproxy > to reread the DNS. I don't know haproxy well, there may already be a signal > or message that exists that can do this. It would be a small "nice to have" > to teach haproxy to accept notify messages and update automatically. (You > don't need to do the zone transfer, you just need to receive the message > for the new SOA and reply with the correct response.) > > For now, it will be able to enforce HAProxy to perform a new DNS query at the next health check. > I hope all this helps, > Yes, a lot, thank you! > jerry > > > On 12/19/16 4:15 AM, Baptiste wrote: > > Hi Jerry, > > Thanks a lot for your email and your nice feedback! > > When checking SRV records, I first did not really know what to do with the > "priority" field, then I though we could use it to know which set of > servers to use first (as for the 'backup' keyword), exactly as you > mentioned! > > Before being able to add SRV records in HAProxy, we had to had more > flexibility (IE: ability to change a server's port...). Now, everything > seems to be ready for SRV records and we can move forward on the design. > > I have an etherpad running somewhere where I'll summarize this design and > share it with the people interested (you, Conrad, Pavlos, HAProxy > Technologies devs, etc...). So we can modify it live, find an agreement and > let the people who want to code it, code it. > This will be "phase 1", using SRV records to update a unique server. > > Then, we'll have to work on phase 2: use a DNS query to fill up servers in > the backend. As Willy explained already, there will be a discussion on the > ML too, because the design is not that simple and we expect the people with > experience like you to help us on this stage! > > Baptiste > > > On Mon, Dec 19, 2016 at 6:29 AM, jerry <je...@soundhound.com> wrote: > >> Igor, >> >> thanks for the pointer tot he email thread. We have too many pieces of >> automation going on and adding consul really isn't an option. >> >> Baptiste, if you are open to discussing some requirements I would love to >> talk. I am an old hand at DNS and have some specific ideas of how it could >> work with haproxy, If you folks are working toward using SRV records as a >> way of specifying the current backend server set, I will take that as a >> significant opinion that this is not a bad idea. >> >> thanks again, >> >> jerry >> >> On 2016-12-18 8:45 pm, Igor Cicimov wrote: >> >> >> >> On Mon, Dec 19, 2016 at 11:43 AM, jerry <je...@soundhound.com> wrote: >> >>> Hi, >>> >>> We use haproxy quite a but at soundhound for fronting our various >>> external and internal services. We are i the process of moving to a >>> container based deployment model. With kubernetes in particular, it's as >>> easy as editing 1 line to add a new server to a replication set. In one of >>> our applications, we have 50+ partitions, so when a new instance is added, >>> it is actually 50+ containers and 50+ proxy configs to edit. Clearly we >>> need to automate this more than we have so far. >>> >>> I was looking around at how to do this and came up with an unusual idea. >>> Everybody uses DNS, but almost no one uses/knows about SRV records. SRV >>> records have the form of >>> _(service)._(protocol).name and return a target DNS name (like CNAME), >>> priority, weight and destination port. This is like MX records on steroids. >> >> >> For SRV records specifically, I'm using Consul and dnsmasq so on the >> haproxy instances I can do: >> >> $ dig +short srv tomcat.service.consul >> 1 1 8080 ip-192.168-0-45.node.dc-prod.consul. >> 1 1 8080 ip-192.168-1-226.node.dc-prod.consul. >> >> and update the appropriate backend section of the config file and reload >> in case of changes. I keep the config file "modular" meaning multiple parts >> (mostly for the backends) that I can update based on some criteria and then >> glue together to create the final haproxy.cfg. How do you implement the >> updates, the triggers and the config file setup depends on your >> architecture and needs. >> >> If you google for "consul-template + haproxy" I'm sure you will also come >> across several solutions. >> >> The dynamic DNS update is coming to haproxy in version 1.7, check this >> discussion for details https://www.mail-archive.com/h >> apr...@formilux.org/msg23579.html >> >> >>> >>> This seems to have everything I need for the dynamic parts of a >>> configuration. The original name defines the port to listen on (this can be >>> a name that gets looked up in /etc/services or _31112 for a hard port >>> number). The set of records returned is the backend set. The ones that have >>> the lowest priority are active and the rest are backups. If you want to do >>> per backend commands, those can be put in as TXT records for the name >>> backend.system.name._(service)._(protocol).name. Similarly, dynamic >>> parts of the frontend and backend could be done with TXT records for >>> _frontend and _backend prefixes to the SRV name. >>> >>> For now, we can assume that there is only one level of backup rather >>> than the arbitrary number that DNS allows. >>> >>> There is an extra little bit that DNS can do for us. DNS has a notify >>> capability. When a zone updates, the dns server can notify it's slaves with >>> the new SOA record. It turns out that all you need to do is generate a >>> response that is almost an exact copy of the query you got, in DNS wire >>> form to make DNS happy and then do the lookups again. >>> >>> My thought is that the typical thing you configure for a given proxy >>> would remain in the config file as now (what to check, connection mode...). >>> The front end becomes the SRV name and the backend is either empty of have >>> some command that says expand the SRV record and put it as the backends. >>> >>> >>> >>> The reason I like it is there are 75 different orchestration/automation >>> systems that could want to drive haproxy. They all talk to DNS already. >>> With this, there is a single interface that I need for the automation from >>> the view of both an orchestration system and haproxy. We have a custom load >>> balancer that we plan to drive the same way. >>> >>> >>> Here are my questions: >>> Do people think this is a interesting/good idea? >>> Is there anything that is missing and would make this a terrible idea? >>> If not fatally flawed, should this be run outside of haproxy as an >>> automation tool or should haproxy do this itself? >>> >>> thanks, >>> jerry >>> >>> >>> >> >> >> -- >> Igor Cicimov | DevOps >> >> >> >> >> p. +61 (0) 433 078 728 >> e. ig...@encompasscorporation.com <http://encompasscorporation.com/> >> w*.* www.encompasscorporation.com >> a. Level 4, 65 York Street, Sydney 2000 >> >> >> >> > > >
