[hackers] Re: Limiting clients per source IP address (ftpd, inetd,etc.)
Terry == Terry Lambert [EMAIL PROTECTED] writes: Actually I was thinking more of ReGet and Godzilla-style software used by some users to play unfair and suck more bandwidth out of an FTP server, by opening a zillion sockets and downloading a single file in chunks. Terry What a clever hack! Terry I don't know if I should revise my argument to include Terry per-IP-per-file, which would of necessity be user space, or Terry just admire it and say they *deserve* more bandwidth for being Terry smart... ... and axel for FreeBSD. I think this is another example of the iternet routing around problems. The problem is that small amounts of packet loss (etc) make is amazingly good to take the multiple stream approach. I believe axel's default approach is to open 4 connections. With the FreeBSD ISO, I get 100 to 250 KB/s with regular ftp (depending). With axel, I get 550 KB/s (on a 100M well-connected link). In fact, the worse one connection does, the more reason to get out the multi-connection tool. There are only two ways to attack this problem: make it less profitable or impossible. Making it impossible will catch all kinds of people in the same net. Making it less profitable will make the problem go away by itself. The only idea (off the top of my head) that will make it sufficiently less profitable is to ensure that the multiple streams behave exactly in the manner of a single stream. This would require the TCP retry mechanism to track the steams together and punish additional streams for each fault in the other streams. Dave. -- |David Gilbert, Velocet Communications. | Two things can only be | |Mail: [EMAIL PROTECTED] | equal if and only if they | |http://daveg.ca | are precisely opposite. | =GLO To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On Fri, Jun 21, 2002 at 03:09:25AM +0300, Giorgos Keramidas wrote: I've been thinking for quite some time to add per-client-IP limiting to ftpd, I needed to do this. Then I discovered that ipfw's limit directive lets you limit the number of incoming connections, which proved much more powerful. N -- FreeBSD: The Power to Serve http://www.freebsd.org/ (__) FreeBSD Documentation Projecthttp://www.freebsd.org/docproj/\\\'',) \/ \ ^ --- 15B8 3FFC DDB4 34B0 AA5F 94B7 93A8 0764 2C37 E375 --- .\._/_) msg35241/pgp0.pgp Description: PGP signature
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On Sun, Jun 23, 2002 at 06:19:51PM +0100, Nik Clayton wrote: On Fri, Jun 21, 2002 at 03:09:25AM +0300, Giorgos Keramidas wrote: I've been thinking for quite some time to add per-client-IP limiting to ftpd, I needed to do this. Then I discovered that ipfw's limit directive lets you limit the number of incoming connections, which proved much more powerful. the funny thing is that when i implemented it i thought it was completely useless :) As a matter of fact, I still think that, at least for resource management purposes. It may have its good use for protection against denial-of-service attacks though. cheers luigi To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
Hi, On Thu, 20 Jun 2002 20:25:28 -0700 Terry Lambert [EMAIL PROTECTED] said: tlambert2 Giorgos Keramidas wrote: I've been thinking for quite some time to add per-client-IP limiting to ftpd, and I had almost decided upon something like the following, where each child of ftpd has two numbers associated with it. The client IP address, and the PID of the ftpd child that serves it. The hash at the beginning of the lists serves as a minor assistance in splitting the 2^32 address space in smaller chunks so that we don't end up with a singly linked list of a few thousand entries. tlambert2 Someone just did something similar for inetd (per IP per port). Yes, it's me. I already rewrote my patch to use open hash as you mentioned. My patch is in testing on snapshots.jp.FreeBSD.org (thank you Matusita-san). You can find my patch from: http://www.imasy.or.jp/~ume/FreeBSD/inetd-perip-hash-5c.diff (for 5-CURRENT) http://www.imasy.or.jp/~ume/FreeBSD/inetd-perip-hash-4s.diff (for 4-STABLE) Sincerely, -- Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan [EMAIL PROTECTED] [EMAIL PROTECTED] ume@{,jp.}FreeBSD.org http://www.imasy.org/~ume/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
Luigi Rizzo wrote: On Thu, Jun 20, 2002 at 08:25:28PM -0700, Terry Lambert wrote: Giorgos Keramidas wrote: I've been thinking for quite some time to add per-client-IP limiting to ftpd, and I had almost decided upon something like the following, ... Someone just did something similar for inetd (per IP per port). The more I think about this, and the fact that there is code growing to do basically the same thing in every program, the more I think that the code to do this needs to be centralized. in fact there is an ipfw rule which does just this: ipfw add allow ip from any to any limit src-addr 5 and here you go... Can this be done per port? THis is what both the FTP and the inetd modification movements have been about... -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On Thu, Jun 20, 2002 at 11:58:10PM -0700, Terry Lambert wrote: ... in fact there is an ipfw rule which does just this: ipfw add allow ip from any to any limit src-addr 5 and here you go... Can this be done per port? THis is what both the FTP and the inetd modification movements have been about... ipfw add allow ip from any to any limit src-addr src-port 5 (you can select a subset of the src-addr src-port dst-addr dst-port as the match mask to determine if connections belong to the same group. With the new ipfw code that i have posted it should be trivial to extend the match mask to use real bitmasks (so you can limit per-subnet, per port ranges, etc etc.) BTW in terms of implementation efficiency: this limit thing uses the same hash table used by dynamic ipfw rules. There is currently an (arbitrary) limit of a total of 1000 dynamic entries in the table, but no reason not to raise it much higher if you have memory. cheers luigi To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On 2002-06-20 20:25 +, Terry Lambert wrote: Giorgos Keramidas wrote: I've been thinking for quite some time to add per-client-IP limiting to ftpd, and I had almost decided upon something like the following, where each child of ftpd has two numbers associated with it. Someone just did something similar for inetd (per IP per port). I know. I missed that post (probably deleted it accidentally), and a friend told me that this was being discussed. Since I had been giving the topic a big of thought, I thought I'd post what I had and ask for comments. The friend who notified me about this mentioned that a linear list was being used and what I had so far would probably be OK for inetd too. The more I think about this, and the fact that there is code growing to do basically the same thing in every program, the more I think that the code to do this needs to be centralized. A simple core reuse of ../../foo.c would probably be sufficient in this case, since the part that does the limiting only needs the following interface: find_client_addr(); find_client_pid(); add_client(addr, pid); del_client(pid);/* called by reapchild() */ Putting this code into a seperate daemon, or even natd, makes a lot more sense to me than hacking up the kernel, or every network application ever written. Sound very general to me, and I can't say I don't like the idea. - Giorgos To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On 2002-06-21 00:35 +, Luigi Rizzo wrote: On Thu, Jun 20, 2002 at 11:58:10PM -0700, Terry Lambert wrote: in fact there is an ipfw rule which does just this: ipfw add allow ip from any to any limit src-addr 5 and here you go... Can this be done per port? THis is what both the FTP and the inetd modification movements have been about... ipfw add allow ip from any to any limit src-addr src-port 5 ... BTW in terms of implementation efficiency: this limit thing uses the same hash table used by dynamic ipfw rules. There is currently an (arbitrary) limit of a total of 1000 dynamic entries in the table, but no reason not to raise it much higher if you have memory. The main reason I was looking for a userland implementation of this was that adding limiting to an FTP server that has an active number of a few thousand connections might be a little resource intensive to the kernel of the machine. It's probably OK to stay a bit to much within a userland function that searches a hash/list of addresses, but doing this in the kernel, is something I can't say I fully understand yet. I'm not familiar with the ipfw code. Would it be possible to limit the connections based on source address for a machine that has a few thousand connections and still not put a heavy load on the kernel? - Giorgos To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On Fri, Jun 21, 2002 at 04:36:26PM +0300, Giorgos Keramidas wrote: ... BTW in terms of implementation efficiency: this limit thing uses the same hash table used by dynamic ipfw rules. There is currently an (arbitrary) limit of a total of 1000 dynamic entries in the table, but no reason not to raise it much higher if you have memory. The main reason I was looking for a userland implementation of this was that adding limiting to an FTP server that has an active number of a few thousand connections might be a little resource intensive to the kernel of the machine. It's probably OK to stay a bit to much within a userland function that searches a hash/list of addresses, but doing this in the kernel, is something I can't say I fully understand yet. I'm not familiar with the ipfw code. Would it be possible to limit the connections based on source address for a machine that has a few thousand connections and still not put a heavy load on the kernel? i'd say yes, as long as you make the hash table size and number of buckets large enough. Both are configurable via sysctl variables: net.inet.ip.fw.dyn_buckets: 256 net.inet.ip.fw.dyn_max: 1000 cheers luigi To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
Hello, The main reason I was looking for a userland implementation of this was that adding limiting to an FTP server that has an active number of a few thousand connections might be a little resource intensive to the kernel of the machine. It's probably OK to stay a bit to much within a userland function that searches a hash/list of addresses, but doing this in the kernel, is something I can't say I fully understand yet. Not only this. For example take the normal inetd behaviour for an FTP server. If the ftpd child processes grow above the limit, inetd simply won't spawn others. The users think that the service is dying (because it can be pinged, but the client can't log on) and begin to flame the operator (such a lame service :). Imagine this with the per IP address limit (this will hit more users, because of proxies, NAT boxes, etc). I think it is much better if the daemon can report this via a simple text message. The user limit thing is the last which is necessary to the FreeBSD ftpd for running an anonymous server. [ Free Software ISOs - ftp://ftp.fsn.hu/pub/CDROM-Images/ ]--- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU)phone @work: +361 210 1415 (194) cell.: +3630 306 6758 To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
Wouter Van Hemel wrote: I agree with this... but I think that the reason many people like to do it by implementing the limitation in the daemon, is that they can send back some kind of reply, stating the reason of the refusal (which is a nice thing to do, since so many people are behind proxies, sharing the same ip). In that case, you need to speak the protocol of the specific service, even though most of it is plain text anyway. If not, the ipfw method works just fine. Just a thought. Attila Nagy makes a similar point. I think maybe the place for such a message is in a proxy server. The problem here is the point is to limit the number of connections by one IP address. The obvious reason for this constraint is to prevent a trivial denial of service attack occurring from a single IP address, so that clients from other IP address can get in. If you allow the connection to go through so that you can present a user with a capacity limitation message, then you are in much deeper trouble. You've already allowed your attacker to consume a connection resource, and enough of the protocol state engine has to be there for the inital connection negotiation, such that a failure message can be delivered down the protocol encapsulated channel (in the FTP case, the command channel, as a greetings message failure). After that happens, then you close the connection to the client. However, an attacking client can delay the handshake negotiation mechanism, thus consuming the per connection state information without a full connection. Alternately, for other protocols, immediately upon being connected to the server which is about to reject the connection, the client can wait for the server to do the connection close as part of the reject. This sends the socket into TIME-WAIT on the server, tying up the server resources for a full 2MSL. I think that permitting the connection to go through so that you can have a protocol rejection followed by a protocol close defeats much of the purpose of rate limiting based on IPs. If the patch were simply to set an overall connection limit, then it might make sense to allow the connection. Even so, you are then subject to attack. The ftp.cdrom.com server does a connection rejection based on protocol negotiation and a failure handshake message (Too many users; maximum of 5000 or whatever). Technically, you could DOS attack this server: it's relying on pool retention in order to not run out of connections. In other words, it's assuming that the people connecting to it are legitimate. So a connection followed by a protocol rejection is for people who are playing nice. A connection reject outright is for people who are attacking you. In other words, it's a circuit breaker, not door security. Someone made the comment about people sitting behind a NAT, so that the number of connections from a given IP is actually legitimate traffic. This rate limitation is targetted at an attacker. One would expect that the number of connections would be set high enough that it would not trigger for these people, and low enough to catch an attacker. Really, if you get to this point in dropping connections, you are pretty much screwed anyway. It's not going to protect you from partial connection attempts, which are one of the most common attach types these days (e.g. SYN-flooding), since by the time you get through the protocol state machine, the resources that you should be worrying about protecting have already been compromised. If the idea is to prevent an attack, then you should not even queue the SYN requests from an IP after you hit the connection limit -- you should drop them. The network card DMA'ing them into memory in the first place is where you want to squelch the problem. That's probably a firmware issue, where you give the card a list of squelch IP addresses, and it ignores packets from there for timeout. In any case, there's really little justification for accepting a connection and then immediately rejecting it with a protocol reject, since all of your overhead for protocol state, connection, etc., has already been consumed, if your intent is to guard against an attack. So I guess what needs to happen is a strict definition of what it is people expect to actually accomplish with this class of patches. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On 2002-06-21 15:12 +, Terry Lambert wrote: Someone made the comment about people sitting behind a NAT, so that the number of connections from a given IP is actually legitimate traffic. This rate limitation is targetted at an attacker. Actually I was thinking more of ReGet and Godzilla-style software used by some users to play unfair and suck more bandwidth out of an FTP server, by opening a zillion sockets and downloading a single file in chunks. - Giorgos To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
Giorgos Keramidas wrote: On 2002-06-21 15:12 +, Terry Lambert wrote: Someone made the comment about people sitting behind a NAT, so that the number of connections from a given IP is actually legitimate traffic. This rate limitation is targetted at an attacker. Actually I was thinking more of ReGet and Godzilla-style software used by some users to play unfair and suck more bandwidth out of an FTP server, by opening a zillion sockets and downloading a single file in chunks. What a clever hack! I don't know if I should revise my argument to include per-IP-per-file, which would of necessity be user space, or just admire it and say they *deserve* more bandwidth for being smart... I guess I'll argue that it's a different problem space, and limiting the number of connections for that reason is really easy to get around: 1) Open as many connections as you can 2) Divide the download between the connections In other words, your workaround only works if you take the file into account, or if you set your per IP connection limit to 1 connection per IP. The former is a totally different problem, while the latter can be done with ipfw or one of the other approaches already discussed. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Limiting clients per source IP address (ftpd, inetd, etc.)
Hello all, I've been thinking for quite some time to add per-client-IP limiting to ftpd, and I had almost decided upon something like the following, where each child of ftpd has two numbers associated with it. The client IP address, and the PID of the ftpd child that serves it. The hash at the beginning of the lists serves as a minor assistance in splitting the 2^32 address space in smaller chunks so that we don't end up with a singly linked list of a few thousand entries. addrhash % % .--..--. 0 []-| struct childaddr |---| struct childaddr |- ... 1 [] |==||==| 3 [] | in_addr ca_addr || in_addr ca_addr | 4 [] | int ca_count || int ca_count | 5 [] | LIST(pids) |--+ | LIST(pids) |--+ ... [] `--' | `--' | 65535 []| | .-. | .-. | | struct childpid |--+ | struct childpid |--+ +--|=| +--|=| | | pid_t cp_pid| | | pid_t cp_pid| | `-' | `-' | | +-.-. +- ... | struct childpid | +--|=| | | pid_t cp_pid| | `-' | +- ... A simple hash function, can be used in selecting which entry of the addrhash[] array points to the head of the proper list. The first level lists are lists of IP addresses, stored in ca_addr, and the second level of lists contain just the process ID of the child that serves a client from that address. The ca_count field of the address structure can be used to quickly find out if more clients are allowed or not, without doing a LIST_FOREACH() every time we want to see if another client socket can be accepted or dropped. I am not sure if using simple lists instead of some more sophisticated scheme is OK in this case, since I don't have a multi-thousand FTP server handy to test the patches. The inetd discussion brought this up from old things I had been thinking about, and I wondered, is there any case we could make ftpd limit client sockets per IP address too? And if that is feasible, could inetd and ftpd be made to use the same limiting code (instead of rolling two different versions), reducing code duplication? There is also the following problem, that I am not sure if I have solved it correctly. If the parent process plays around with those hashes and structures, then a signal handler that calls reapchild() to remove parts of those structures can probably do a lot of harm. The race condition seems easy to fix if I use something like: nosignals(); /* Fool around with the structures. */ signals(); Where signals() and nosignals() use sigaction() to block unblock the signals that ftpd already installs a handler for. Does this all sound reasonable? Below is a prototype I'm playing the last few days with, trying to make something that implements the above scheme using queue.h macros. Now, what do you all think about this? Does it sound like a nice idea to pursue further? - Giorgos msg35116/pgp0.pgp Description: PGP signature
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On 2002-06-21 03:09 +, Giorgos Keramidas wrote: Below is a prototype I'm playing the last few days with, trying to make something that implements the above scheme using queue.h macros. Now, what do you all think about this? Does it sound like a nice idea to pursue further? It would be nice if I also included the source *grin* %%% #include netinet/in.h #include sys/queue.h #include sys/types.h #include assert.h #include limits.h #include stdio.h #include stdlib.h #include unistd.h LIST_HEAD(cpidhead, childpid); struct childpid { pid_t cp_pid; /* Process ID of a child. */ LIST_ENTRY(childpid) cp_entry; /* Glue to the other list elements. */ }; LIST_HEAD(ciphead, childaddr); struct childaddr { struct in_addr ca_addr; /* Address of a host. */ int ca_count; /* Number of items in pid list. */ struct cpidhead ca_pid; /* Head of the PID list. */ LIST_ENTRY(childaddr) ca_entry; /* Glue to the other list elements. */ }; /* * This hash keeps a number of pointers to `ciphead' lists. * Use the iphash() function to find the proper element of this array * for the records related to an IP address. */ struct ciphead *ciphash[UINT16_MAX]; static unsigned short iphash(struct in_addr *paddr); static struct childaddr *cip_find(struct ciphead *ph, struct in_addr *addr); static struct childaddr *cip_add(struct ciphead *ph, struct in_addr *addr); static struct childpid *cpid_find(struct cpidhead *ph, pid_t pid); static struct childpid *cpid_add(struct childaddr *pa, pid_t pid); int main(void) { struct in_addr lo; struct childaddr *pa; struct childpid *pp; uint16_t hval; /* Add the address of localhost to the proper place in ciphash[]. */ lo.s_addr = htonl(0x7f01); hval = iphash(lo); if (ciphash[hval] == NULL) { ciphash[hval] = malloc(sizeof(struct ciphead)); if (ciphash[hval] == NULL) exit(1); } else LIST_INIT(ciphash[hval]); pa = cip_find(ciphash[hval], lo); if (pa == NULL (pa = cip_add(ciphash[hval], lo)) == NULL) { if (LIST_FIRST(ciphash[hval]) == NULL) free(ciphash[hval]); exit(1); } /* Add the PID to the list `pa-ca_pid'. */ cpid_add(pa, getpid()); /* Print the mess we created so far. */ printf(%p: struct ciphead {\n, ciphash[hval]); printf(lh_first = %p,\n, pa = LIST_FIRST((ciphash[hval]))); printf(};\n); printf(\n); printf(%p: struct childaddr {\n, pa); printf(ca_addr = 0x%08x,\n, (pa-ca_addr).s_addr); printf(ca_count = %d,\n, pa-ca_count); printf(ca_pid = %p,\n, (pa-ca_pid)); printf(};\n); printf(\n); printf(%p: struct cpidhead {\n, (pa-ca_pid)); printf(lh_first = %p,\n, pp = LIST_FIRST(((pa-ca_pid; printf(};\n); printf(\n); printf(%p: struct childpid {\n, pp); printf(pid = %d,\n, pp-cp_pid); printf(};\n); printf(\n); return (0); } /* * Return a very simple XOR-based hash value, derived from the bytes of a * `struct in_addr' structure. */ static uint16_t iphash(struct in_addr *paddr) { uint16_t *sp; uint16_t val; size_t len; size_t k; assert(paddr != NULL); sp = ((uint16_t *) paddr); val = 0; len = sizeof(struct in_addr) / sizeof(uint16_t); if (len == 0 || len == 1) { val = 0x; } else { for (k = 0; k len; k++) val ^= sp[k]; val = 0x; } return (val); } /* * Look in all the elements of `ph' and see if they match `addr'. * Return the address of the first match, or NULL if none is found. */ static struct childaddr * cip_find(struct ciphead *ph, struct in_addr *addr) { struct childaddr *pa; assert(ph != NULL addr != NULL); LIST_FOREACH(pa, ph, ca_entry) if ((pa-ca_addr).s_addr == (*addr).s_addr) return (pa); return (NULL); } /* * Add a new address structure, in the list of childaddr's pointed at by * the `ph' list head. This doesn't check for an existing match, so * duplicates might end up in your list, if you don't use cip_find() first to * look for older matches. */ struct childaddr * cip_add(struct ciphead *ph, struct in_addr *addr) { struct childaddr *pa; assert(ph != NULL addr != NULL); /* Try to allocate a new childaddr record. */ if ((pa = malloc(sizeof(struct childaddr))) == NULL) return (NULL); pa-ca_addr = *addr; pa-ca_count = 0; LIST_INIT((pa-ca_pid)); LIST_INSERT_HEAD(ph, pa, ca_entry);
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
Giorgos Keramidas wrote: I've been thinking for quite some time to add per-client-IP limiting to ftpd, and I had almost decided upon something like the following, where each child of ftpd has two numbers associated with it. The client IP address, and the PID of the ftpd child that serves it. The hash at the beginning of the lists serves as a minor assistance in splitting the 2^32 address space in smaller chunks so that we don't end up with a singly linked list of a few thousand entries. Someone just did something similar for inetd (per IP per port). The more I think about this, and the fact that there is code growing to do basically the same thing in every program, the more I think that the code to do this needs to be centralized. I would prefer a divert to an administrative daemon approach, using ipfw rules and exisitng code. You could also do it in the kernel, or you could do it by adding a wrapper library for accept and close, where the accounting on connections can be enforced. Putting this code into a seperate daemon, or even natd, makes a lot more sense to me than hacking up the kernel, or every network application ever written. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Limiting clients per source IP address (ftpd, inetd, etc.)
On Thu, Jun 20, 2002 at 08:25:28PM -0700, Terry Lambert wrote: Giorgos Keramidas wrote: I've been thinking for quite some time to add per-client-IP limiting to ftpd, and I had almost decided upon something like the following, ... Someone just did something similar for inetd (per IP per port). The more I think about this, and the fact that there is code growing to do basically the same thing in every program, the more I think that the code to do this needs to be centralized. in fact there is an ipfw rule which does just this: ipfw add allow ip from any to any limit src-addr 5 and here you go... cheers luigi To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message