Re: [OpenAFS] Re: Redundant Internet links
On Thu, Dec 16, 2010 at 11:12 AM, Andrew Deason wrote: > On Thu, 16 Dec 2010 01:55:31 +0100 > Jaap Winius wrote: > >> Regarding the use of the redundant links in the above scenario, would >> it be possible -- desirable or wise even -- to present the clients, >> which have empty CellServDB files, with DNS AFSDB RRs that point to >> five different IP addresses: one internal for the local machine and >> four external for the two remote ones? > > Is the "internal" address only routable from that site? Clients (OpenAFS > clients, anyway) will try to ask the dbservers for which dbserver is the > sync site when performing a write operation to whatever db; I'm not sure > how they behave if a client gets an address it doesn't recognize as an > answer. You can simulate this easily with no-AFSDB, and a sync site on a not-in-CellServDB server. You end up with UNOTSYNC, iirc, but basically "you lose" > dbservers are different, though; the specification in CellServDB affects > the voting algorithm Only for the server config of course, but yeah. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: Redundant Internet links
Quoting Andrew Deason : Is the "internal" address only routable from that site? No. Although each of the three sites will be connected to a small internal network with private IP addresses, the broadband modems will be in bridged mode and the server will also have two public IP addresses. Each server will also act as a router between the internal network and the external links (but not between the latter). I figure that it will be best if each OpenAFS server is only known by its public IP address(es), even to the clients on its internal network. This way, each OpenAFS client at each location in the cell will always contact the three OpenAFS servers using the same set of public IP addresses. If you're just taking the "real" sites and adding additional IPs, though, I think the clients should be fine. Aside from the aforementioned sync site determination, it's effectively just a list of sites to try to contact. Okay, so I could give all of the clients a list of six IP addresses for the three servers. I plan to use DNS with AFSDB RRs for this, although that doesn't provide a method for giving priority to the faster links. However, I recently noticed RFC5864 (good work, Russ!): what's the minimum OpenAFS version necessary for anyone wanting to use SRV RRs for OpenAFS instead? dbservers are different, though; the specification in CellServDB affects the voting algorithm, and is not as simple as a list of sites to try to reach. I don't think I'd recommend putting multiple IPs per site in there, but I don't really know what happens if you do that. Each site is aware of the local addresses of the other sites, and I'm not sure what a site does if it sees that another site claims ownership of more than one CellServDB server entry... But I don't imagine it ending well. Yeah, I suspected that would be a problem... despite the fact that the names following the "#" behind the IP addresses would be the same. So, I'll just use one IP address per server in the CellServDB files of the DB/file server machines and let the routing system take care of any communications problems. As to whether this is advisable in general... it's not immediately clear to me what failure case you're trying to cover here. If you have three geographically-diverse sites, losing access to one of those db sites will not significantly affect accessing AFS beyond an initial delay when the site goes away. If you lose two sites, then you lose the ability to make vldb or ptdb changes, but you can still access data (assuming the relevant fileservers are still accessible). The redundant links will significantly reduce the risk for each site that, if it temporarily loses its connection to the Internet, the local OpenAFS file server also becomes read-only until the link is restored. But, besides that, the business already grinds to a halt when Internet access is lost due to their dependence on a proprietary online database management solution. So, the redundant links will also address an existing problem. ... it doesn't seem worth the effort of running with such a non-typical configuration. While it may be possible to make it work, providing the redundancy at a lower level seems easier. Indeed. It was always the plan to implement that low-level redundancy anyway. What would seem to have more immediate benefit is to have your fileservers register (at least) two IP addresses in the vldb; one for each redundant link. That way, if one link to the site containing that fileserver goes down, clients should still be able to access the data on that fileserver. Would it then not be necessary to have the database and file servers installed on separate hosts? Otherwise they would share the same CellServDB files (even with a single physical machines, that's possible using virtual hosts, although I would not be able to afford two public IPv4 addresses for both hosts). Or is that not what you mean? Of course, this should already happen automatically if the two IPs for the fileserver are addresses known by its local interfaces (that is, they show up in 'ifconfig' or whatever). I plan to use the "iproute" package in Debian. Cheers, Jaap ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: Redundant Internet links
>> dbservers are different, though; the specification in CellServDB affects >> the voting algorithm, and is not as simple as a list of sites to try to >> reach. I don't think I'd recommend putting multiple IPs per site in >> there, but I don't really know what happens if you do that. Each site is >> aware of the local addresses of the other sites, and I'm not sure what a >> site does if it sees that another site claims ownership of more than one >> CellServDB server entry... But I don't imagine it ending well. > > Yeah, I suspected that would be a problem... despite the fact that the names > following the "#" behind the IP addresses would be the same. So, I'll just > use one IP address per server in the CellServDB files of the DB/file server > machines and let the routing system take care of any communications > problems. Careful here. Things get fun if more than one server ends up believing it is the "lowest host" (by IP, numerically) and thus has a tiebreaker vote. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: Redundant Internet links
Quoting Andrew Deason : ... We don't provide the tools for a split-horizon vldb (yet, anyway). Actually, if we're all going to move to IPv6 anyway, of what use would that be? To be clear, the fileserver does not become readonly; what becomes readonly are the databases that contain volume location information and authenticated user metadata. So, that means you can read and write to files to any fileserver you can reach, but you cannot create, remove, or release volumes, create, remove, or alter users/groups, or anything else that requires modifying those databases. Very interesting. So, I take it a different, local database is used to keep track of the changes made to individual files in local R/W volumes, and this database stays R/W even if the server it's on gets cut off? ... You contact the vlserver at site A, and it will tell you that the volume is on a fileserver at site B, and it will also tell you all known IP addresses for the fileserver at site B. Sounds like you're referring to the IP addresses for the servers that the clients are given. In that case I understand. I can do that with AFSDB RRs. What I meant, though, are the IP addresses that the servers have to contact each other. On Debian, these are in /etc/openafs/server/CellServDB. I'd like to use multiple IP addresses for each host in there too, but that would adversely affect the voting algorithm. On the other hand, what if I were to set up virtual hosts on which to run the file servers separately? In that case, each database server would still run on the bare metal OS and those CellServDB files would still contain only three IP addresses. Lower level routing would still take care of connectivity if one of the main links went down. The files servers, however, could each have a CellServDB file with five addresses: a local private range address and four public addresses for the two remote file servers (which would be reached through port-forwarding). Still, even if this would work, I no longer think I'd want to do it. That's because I'd rather have the AFS servers avoid the secondary links entirely unless the main links go down, and I can't instruct them to do that (yet) through prioritization. I can only do that with routing. Cheers, Jaap ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: Redundant Internet links
On Fri, Dec 17, 2010 at 9:41 PM, Jaap Winius wrote: > Quoting Andrew Deason : > >> ... We don't provide the tools for a split-horizon vldb (yet, anyway). > > Actually, if we're all going to move to IPv6 anyway, of what use would that > be? ipv4 isn't going away tomorrow... >> To be clear, the fileserver does not become readonly; what becomes >> readonly are the databases that contain volume location information and >> authenticated user metadata. So, that means you can read and write to >> files to any fileserver you can reach, but you cannot create, remove, or >> release volumes, create, remove, or alter users/groups, or anything else >> that requires modifying those databases. > > Very interesting. So, I take it a different, local database is used to keep > track of the changes made to individual files in local R/W volumes, and this > database stays R/W even if the server it's on gets cut off? each volume is tracked individually on the server hosting it (whether RW or RO; RO are just published, snapshot copies, of the RW) >> ... You contact the vlserver at site A, and >> it will tell you that the volume is on a fileserver at site B, and it >> will also tell you all known IP addresses for the fileserver at site B. > > Sounds like you're referring to the IP addresses for the servers that the > clients are given. In that case I understand. I can do that with AFSDB RRs. > > What I meant, though, are the IP addresses that the servers have to contact > each other. On Debian, these are in /etc/openafs/server/CellServDB. I'd like > to use multiple IP addresses for each host in there too, but that would > adversely affect the voting algorithm. can't do it. sorry. > On the other hand, what if I were to set up virtual hosts on which to run > the file servers separately? In that case, each database server would still > run on the bare metal OS and those CellServDB files would still contain only > three IP addresses. Lower level routing would still take care of > connectivity if one of the main links went down. The files servers, however, > could each have a CellServDB file with five addresses: a local private range > address and four public addresses for the two remote file servers (which > would be reached through port-forwarding). doable, albeit potentially fussy if there are issues with the routing. > Still, even if this would work, I no longer think I'd want to do it. That's > because I'd rather have the AFS servers avoid the secondary links entirely > unless the main links go down, and I can't instruct them to do that (yet) > through prioritization. I can only do that with routing. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: Redundant Internet links
Quoting Derrick Brashear : Yeah, I suspected that would be a problem... despite the fact that the names following the "#" behind the IP addresses would be the same. So, I'll just use one IP address per server in the CellServDB files of the DB/file server machines and let the routing system take care of any communications problems. Careful here. Things get fun if more than one server ends up believing it is the "lowest host" (by IP, numerically) and thus has a tiebreaker vote. I can imagine, but I don't think that will be a problem in this case. Each database server should never consider that it has any other IP address except one in particular. The single IP addresses that they each use to reach their remote companions will never change either. Only the routing in between will occasionally change as needed when the main links go down and up. Cheers, Jaap ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: Redundant Internet links
On Sat, Dec 18, 2010 at 9:16 AM, Jaap Winius wrote: > Quoting Derrick Brashear : > >>> Yeah, I suspected that would be a problem... despite the fact that the >>> names >>> following the "#" behind the IP addresses would be the same. So, I'll >>> just >>> use one IP address per server in the CellServDB files of the DB/file >>> server >>> machines and let the routing system take care of any communications >>> problems. >> >> Careful here. Things get fun if more than one server ends up believing >> it is the "lowest host" (by IP, numerically) and thus has a tiebreaker >> vote. > > I can imagine, but I don't think that will be a problem in this case. Each > database server should never consider that it has any other IP address > except one in particular. The single IP addresses that they each use to > reach their remote companions will never change either. Only the routing in > between will occasionally change as needed when the main links go down and > up. Sure, but will any host's address for some other host be lower than its own address while at the same time that host itself *should be* the lowest host? (It's too early; Does that make sense?) -- Derrick ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: Redundant Internet links
Quoting Derrick Brashear : ... Each database server should never consider that it has any other IP address except one in particular. The single IP addresses that they each use to reach their remote companions will never change either. Only the routing in between will occasionally change as needed when the main links go down and up. Sure, but will any host's address for some other host be lower than its own address while at the same time that host itself *should be* the lowest host? (It's too early; Does that make sense?) The addresses of the DB/file servers, as well as the addresses by which they are known each other, will always be the same. Originally I wondered whether it would be possible to let the DB servers decide for themselves the routes by which they would contact each other, since each of the hosts on which they will reside will be multi-homed, but the answer to that is no: they can only know and contact each other via a single IP address (not two) because of the way the voting algorithm works. So, now I'm just going to configure some lower-level routing to take advantage of the redundant links. I've set up a system with two default routes before, using Debian's iproute package, but this network will go one step further: when one site's main Internet link goes down, each of the other sites will still be able to reach it's usual IP address via an alternative static route. Cheers, Jaap ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: Redundant Internet links
On Sat, Dec 18, 2010 at 8:05 PM, Jaap Winius wrote: > Quoting Derrick Brashear : > >>> ... Each database server should never consider that it has any other IP >>> address except one in particular. The single IP addresses that they >>> each use to reach their remote companions will never change either. Only >>> the routing in between will occasionally change as needed when the >>> main links go down and up. >> >> Sure, but will any host's address for some other host be lower than >> its own address while >> at the same time that host itself *should be* the lowest host? >> >> (It's too early; Does that make sense?) > > The addresses of the DB/file servers, as well as the addresses by which they > are known each other, will always be the same. Good. Of note, there are cases where due to one DB server being behind an address-rewriting router (like a NAT) that a server's idea of its address does not match what its peers' ideas are. This can be a valid configuration, if care is taken, which is why I brought it up. -- Derrick ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
