Re: rogue Chinese crawler
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 OK, I've now been 24 hours without a hit, so I'm presuming I've got rid of all the crawlers. Thanks for all the help and advice from both lists. Resume: - - the openfind.com(.tw) 'bots don't respect the norobots conventions, so your robots.txt is useless, whatever its contents. (In fact, these 'bots don't even look for it.) - - the 'bots are no respecters of any other conventions, either -- they will stay on your machine for unlimited amounts of time, doing a constant recursive grab, and gradually freezing out all other activity. (Worst case on my machine -- 45 minutes) - - there are 16 'bots, none of which knows what the others are doing. This means you can have any number on your machine at any one time, each progressively slowing down the system. (Worst case on my machine -- 8 simultaneously, for over 30 minutes.) This can create a virtual DoS attack, or "paralysis" of services. - - they may not all come from the same address -- I've currently got two addresses in my rules/directives to drop packets. (Monitor where they're coming from.) - - the originators do NOT reply to e-mails or polite requests to fix their code to respect the norobots conventions. The DO respond to abusive e-mails by bouncing any further attempts at communication with them. - - as pointed out by almost everyone, the best method of dissuasion is to drop all packets from thisese sources as they come into the firewall/router. Failing that, a Deny from directive in httpd.conf fixes them good. - - if the above is implemented, it takes a while for all the 'bots to learn they're not welcome. I don't mind well-behaved spiders -- in fact, I welcome them, as no-one would be able to find some of my pages otherwise -- but these ones go beyond what is tolerable behaviour for me. I don't know whether it's due to bad code, or a "don't care" attitude to others; but I would advise anyone who finds them clogging up their system to ban them completely. Martin - -- - Share your knowledge. It's a way to achieve immortality - <[EMAIL PROTECTED]> pub 1024D/01269BEB 2001-09-29 Martin Wheeler (personal key) Key fingerprint = 6CAD BFFB DB11 653E B1B7 C62B AC93 0ED8 0126 9BEB -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE8AY5orJMO2AEmm+sRAuBRAJ9gRbweyJAsYn1dL1OWWYJcHg2x1ACgkFLe 4Af36/11JM3+bXXhtNNVFoU= =EBHS -END PGP SIGNATURE-
Re: virtual hosting methods
Gavin Hamill wrote: > This is my biggest problem and a significant security hole :/ > > I have a directory /www containing all the vhosting directories, named > domain.com, etc. > > the entire directory tree is owned by a user called virtual, and > everyone has CGI, PHP and SSI access. > > In this way it would be very easy for anyone to upload a 'file manager' > CGI and be able to change the documents of any other Vhost user :( Why not have the owner of the files be somethingelse, and "virtual" has group read rights; so to upload any file they would have an upload web page which passes the job to the owner, somethingelse. You wouldn't need to create multiple real users, just two for the job (which could then use sudo, and you'd have to lock-down that upload program well). Mark Aitchison -- phone:(064)3-364-5888 /\/\ _/\ /\ fax: (064)3-364-5835 _/\/ ^ \/\,__ System Administrator at: Plain Communications mailto:[EMAIL PROTECTED]">==
Installing PPP 2.4.0
Hi, Sorry for the cross posting, but I am really stuck! I am currently setting up my Debian machine to connect to my ADSL modem for internet access. I have had everything working before, but I am having problems this time with the PPP daemon! I am trying to install the ppp-2.4.0 tarball, but after I have extracted it I have gone into the directory and executed: ./configure ... then ran a: make but, when compiling the pppoatm.c file, it has complained about not finding the atm.h file in the /usr/src/linux/include directory. After checking there indeed is not an atm.h file there. However, there are no atm.h files on my system that will work! Anybody any ideas how to get it working, or to find the right atm.h file? Cheers, Ben --- b e n @ j a v a c o d e r . n e t - w w w . j a v a c o d e r . n e t "The reader is entertained by the journey of another, but the writer is the changer of worlds" - D'Ni proverb
Re: rogue Chinese crawler
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 OK, I've now been 24 hours without a hit, so I'm presuming I've got rid of all the crawlers. Thanks for all the help and advice from both lists. Resume: - - the openfind.com(.tw) 'bots don't respect the norobots conventions, so your robots.txt is useless, whatever its contents. (In fact, these 'bots don't even look for it.) - - the 'bots are no respecters of any other conventions, either -- they will stay on your machine for unlimited amounts of time, doing a constant recursive grab, and gradually freezing out all other activity. (Worst case on my machine -- 45 minutes) - - there are 16 'bots, none of which knows what the others are doing. This means you can have any number on your machine at any one time, each progressively slowing down the system. (Worst case on my machine -- 8 simultaneously, for over 30 minutes.) This can create a virtual DoS attack, or "paralysis" of services. - - they may not all come from the same address -- I've currently got two addresses in my rules/directives to drop packets. (Monitor where they're coming from.) - - the originators do NOT reply to e-mails or polite requests to fix their code to respect the norobots conventions. The DO respond to abusive e-mails by bouncing any further attempts at communication with them. - - as pointed out by almost everyone, the best method of dissuasion is to drop all packets from thisese sources as they come into the firewall/router. Failing that, a Deny from directive in httpd.conf fixes them good. - - if the above is implemented, it takes a while for all the 'bots to learn they're not welcome. I don't mind well-behaved spiders -- in fact, I welcome them, as no-one would be able to find some of my pages otherwise -- but these ones go beyond what is tolerable behaviour for me. I don't know whether it's due to bad code, or a "don't care" attitude to others; but I would advise anyone who finds them clogging up their system to ban them completely. Martin - -- - Share your knowledge. It's a way to achieve immortality - <[EMAIL PROTECTED]> pub 1024D/01269BEB 2001-09-29 Martin Wheeler (personal key) Key fingerprint = 6CAD BFFB DB11 653E B1B7 C62B AC93 0ED8 0126 9BEB -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE8AY5orJMO2AEmm+sRAuBRAJ9gRbweyJAsYn1dL1OWWYJcHg2x1ACgkFLe 4Af36/11JM3+bXXhtNNVFoU= =EBHS -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: virtual hosting methods
Gavin Hamill wrote: > This is my biggest problem and a significant security hole :/ > > I have a directory /www containing all the vhosting directories, named > domain.com, etc. > > the entire directory tree is owned by a user called virtual, and > everyone has CGI, PHP and SSI access. > > In this way it would be very easy for anyone to upload a 'file manager' > CGI and be able to change the documents of any other Vhost user :( Why not have the owner of the files be somethingelse, and "virtual" has group read rights; so to upload any file they would have an upload web page which passes the job to the owner, somethingelse. You wouldn't need to create multiple real users, just two for the job (which could then use sudo, and you'd have to lock-down that upload program well). Mark Aitchison -- phone:(064)3-364-5888 /\/\ _/\ /\ fax: (064)3-364-5835 _/\/ ^ \/\,__ System Administrator at: Plain Communications mailto:[EMAIL PROTECTED]";>== -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Installing PPP 2.4.0
Hi, Sorry for the cross posting, but I am really stuck! I am currently setting up my Debian machine to connect to my ADSL modem for internet access. I have had everything working before, but I am having problems this time with the PPP daemon! I am trying to install the ppp-2.4.0 tarball, but after I have extracted it I have gone into the directory and executed: ./configure ... then ran a: make but, when compiling the pppoatm.c file, it has complained about not finding the atm.h file in the /usr/src/linux/include directory. After checking there indeed is not an atm.h file there. However, there are no atm.h files on my system that will work! Anybody any ideas how to get it working, or to find the right atm.h file? Cheers, Ben --- b e n @ j a v a c o d e r . n e t - w w w . j a v a c o d e r . n e t "The reader is entertained by the journey of another, but the writer is the changer of worlds" - D'Ni proverb -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: virtual hosting methods
On Sat, Nov 24, 2001 at 06:44:02PM -0500, Kevin J. Menard, Jr. wrote: > > MpP> For simple masshosting I still suggest mod_vhost. > > Which brings me back to my original question. For simple masshosting, I > would agree. But what about a system where some vhosts have CGI or SSI > access for example, and some don't. Would the former setup be better, or > the latter? This is my biggest problem and a significant security hole :/ I have a directory /www containing all the vhosting directories, named domain.com, etc. the entire directory tree is owned by a user called virtual, and everyone has CGI, PHP and SSI access. In this way it would be very easy for anyone to upload a 'file manager' CGI and be able to change the documents of any other Vhost user :( People have pointed me at sudo in the past but I don't want to start creating /etc/passwd users - that was the whole point of the virtual system - no real system users for www, ftp or mail! Any ideas, anyone? We haven't had any problems to date because none of our clients know anything / much about scripting... Cheers, gdh
Re: virtual hosting methods
On Sat, Nov 24, 2001 at 06:44:02PM -0500, Kevin J. Menard, Jr. wrote: > > MpP> For simple masshosting I still suggest mod_vhost. > > Which brings me back to my original question. For simple masshosting, I > would agree. But what about a system where some vhosts have CGI or SSI > access for example, and some don't. Would the former setup be better, or > the latter? This is my biggest problem and a significant security hole :/ I have a directory /www containing all the vhosting directories, named domain.com, etc. the entire directory tree is owned by a user called virtual, and everyone has CGI, PHP and SSI access. In this way it would be very easy for anyone to upload a 'file manager' CGI and be able to change the documents of any other Vhost user :( People have pointed me at sudo in the past but I don't want to start creating /etc/passwd users - that was the whole point of the virtual system - no real system users for www, ftp or mail! Any ideas, anyone? We haven't had any problems to date because none of our clients know anything / much about scripting... Cheers, gdh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: rogue Chinese crawler
The best way would be to block it at your router with an access list. Blocking it at the box is ok too but that takes a little bit of your resources. And you have to do it on each box on your network you want protected. The router block will protect your entire network in one fell swoop and cost your boxes no resources. You can block just his ip address with a deny statement, or if he's scanning from multiple ip's you can chunk his whole network. But that ip (139.175.250.23) is under a huge Seed-net /16. You might end up blocking legitimate traffic. You can try to guess his local subnet mask and block that, like a /27 or something. On a related topic I've been receiving an enormous amount of spam coming through Asian mx's. Is there any effort underway to try and get these people to lock down their networks? We've got a bunch of rogue mailservers over there. At 05:32 PM 11/23/01 +, Martin WHEELER wrote: >Is anyone else having problems with the robot from > > openfind.com.tw > >-- an intrusive, irritating, hard-to-get-rid-of crawler that completely >paralyses my system *every day*? > >Despite what I put in any robots.txt, this one disregards all rules and >just jams up my system, downloading every damn' thing in sight. >Mails to the owners are totally disregarded. > >Anyone know of a sure-fire robot killer under woody? > >Who should this thing be reported to to get it stopped? PS, the first time around I accidently only sent this to debian-security. :) ---==--- ___/`< WTC 911 >`\___ 0100
Re: virtual hosting method
OK, I'll write a patch... you'' get it within an hour or so.. regards, -- Martin 'pisi' Paljak / freelancer consultant [EMAIL PROTECTED] / pisi.pisitek.com www.pisitek.com On Sun, 25 Nov 2001, Martin 'pisi' Paljak wrote: > As of 1.3.22 it reads everything .file and file~ :( Easy to fix but aint > got no time nor interest. > > -- > Martin 'pisi' Paljak / freelancer consultant > [EMAIL PROTECTED] / pisi.pisitek.com > www.pisitek.com > > > On 24 Nov 2001, Karl M. Hegbloom wrote: > > > > "Frank" == Frank Louwers <[EMAIL PROTECTED]> writes: > > > > Frank> On Sun, Nov 25, 2001 at 12:30:41AM +0200, Martin 'pisi' Paljak > > wrote: > > >> Actually there is a very nice and nifty feature in apache 1.3.19+ > > (or was > > >> it 20+) that allows an include filename to be a directory what will > > >> include all directories and subdirs of the named direcotry, and load > > all > > >> files in those dirs as config files. With some maintenance scripts it > > >> allows very easy maintenance of virtual hosts (configuration...) > > > > Frank> Only thing: remember NOT to leave temp/backup files in that > > directory, > > Frank> as EVERY file is read as a config file... > > > > That should be fixed. I think it ought to ignore dot files, "~" > > suffixed files, files that begin with "-" (so you can elide them > > without moving them elsewhere), and files with a ".dpkg-*" suffix. > > > > It should not descend a ".backup*/" directory created by emacs > > either, in case someone is using backup directories. > > > > -- > > I was Linux when Linux wasn't cool. > > > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > >
Re: virtual hosting methods
As of 1.3.22 it reads everything .file and file~ :( Easy to fix but aint got no time nor interest. -- Martin 'pisi' Paljak / freelancer consultant [EMAIL PROTECTED] / pisi.pisitek.com www.pisitek.com On 24 Nov 2001, Karl M. Hegbloom wrote: > > "Frank" == Frank Louwers <[EMAIL PROTECTED]> writes: > > Frank> On Sun, Nov 25, 2001 at 12:30:41AM +0200, Martin 'pisi' Paljak > wrote: > >> Actually there is a very nice and nifty feature in apache 1.3.19+ (or > was > >> it 20+) that allows an include filename to be a directory what will > >> include all directories and subdirs of the named direcotry, and load > all > >> files in those dirs as config files. With some maintenance scripts it > >> allows very easy maintenance of virtual hosts (configuration...) > > Frank> Only thing: remember NOT to leave temp/backup files in that > directory, > Frank> as EVERY file is read as a config file... > > That should be fixed. I think it ought to ignore dot files, "~" > suffixed files, files that begin with "-" (so you can elide them > without moving them elsewhere), and files with a ".dpkg-*" suffix. > > It should not descend a ".backup*/" directory created by emacs > either, in case someone is using backup directories. > > -- > I was Linux when Linux wasn't cool. >
Re: rogue Chinese crawler
The best way would be to block it at your router with an access list. Blocking it at the box is ok too but that takes a little bit of your resources. And you have to do it on each box on your network you want protected. The router block will protect your entire network in one fell swoop and cost your boxes no resources. You can block just his ip address with a deny statement, or if he's scanning from multiple ip's you can chunk his whole network. But that ip (139.175.250.23) is under a huge Seed-net /16. You might end up blocking legitimate traffic. You can try to guess his local subnet mask and block that, like a /27 or something. On a related topic I've been receiving an enormous amount of spam coming through Asian mx's. Is there any effort underway to try and get these people to lock down their networks? We've got a bunch of rogue mailservers over there. At 05:32 PM 11/23/01 +, Martin WHEELER wrote: >Is anyone else having problems with the robot from > > openfind.com.tw > >-- an intrusive, irritating, hard-to-get-rid-of crawler that completely >paralyses my system *every day*? > >Despite what I put in any robots.txt, this one disregards all rules and >just jams up my system, downloading every damn' thing in sight. >Mails to the owners are totally disregarded. > >Anyone know of a sure-fire robot killer under woody? > >Who should this thing be reported to to get it stopped? PS, the first time around I accidently only sent this to debian-security. :) ---==--- ___/`< WTC 911 >`\___ 0100 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: virtual hosting method
OK, I'll write a patch... you'' get it within an hour or so.. regards, -- Martin 'pisi' Paljak / freelancer consultant [EMAIL PROTECTED] / pisi.pisitek.com www.pisitek.com On Sun, 25 Nov 2001, Martin 'pisi' Paljak wrote: > As of 1.3.22 it reads everything .file and file~ :( Easy to fix but aint > got no time nor interest. > > -- > Martin 'pisi' Paljak / freelancer consultant > [EMAIL PROTECTED] / pisi.pisitek.com > www.pisitek.com > > > On 24 Nov 2001, Karl M. Hegbloom wrote: > > > > "Frank" == Frank Louwers <[EMAIL PROTECTED]> writes: > > > > Frank> On Sun, Nov 25, 2001 at 12:30:41AM +0200, Martin 'pisi' Paljak wrote: > > >> Actually there is a very nice and nifty feature in apache 1.3.19+ (or was > > >> it 20+) that allows an include filename to be a directory what will > > >> include all directories and subdirs of the named direcotry, and load all > > >> files in those dirs as config files. With some maintenance scripts it > > >> allows very easy maintenance of virtual hosts (configuration...) > > > > Frank> Only thing: remember NOT to leave temp/backup files in that directory, > > Frank> as EVERY file is read as a config file... > > > > That should be fixed. I think it ought to ignore dot files, "~" > > suffixed files, files that begin with "-" (so you can elide them > > without moving them elsewhere), and files with a ".dpkg-*" suffix. > > > > It should not descend a ".backup*/" directory created by emacs > > either, in case someone is using backup directories. > > > > -- > > I was Linux when Linux wasn't cool. > > > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: virtual hosting methods
As of 1.3.22 it reads everything .file and file~ :( Easy to fix but aint got no time nor interest. -- Martin 'pisi' Paljak / freelancer consultant [EMAIL PROTECTED] / pisi.pisitek.com www.pisitek.com On 24 Nov 2001, Karl M. Hegbloom wrote: > > "Frank" == Frank Louwers <[EMAIL PROTECTED]> writes: > > Frank> On Sun, Nov 25, 2001 at 12:30:41AM +0200, Martin 'pisi' Paljak wrote: > >> Actually there is a very nice and nifty feature in apache 1.3.19+ (or was > >> it 20+) that allows an include filename to be a directory what will > >> include all directories and subdirs of the named direcotry, and load all > >> files in those dirs as config files. With some maintenance scripts it > >> allows very easy maintenance of virtual hosts (configuration...) > > Frank> Only thing: remember NOT to leave temp/backup files in that directory, > Frank> as EVERY file is read as a config file... > > That should be fixed. I think it ought to ignore dot files, "~" > suffixed files, files that begin with "-" (so you can elide them > without moving them elsewhere), and files with a ".dpkg-*" suffix. > > It should not descend a ".backup*/" directory created by emacs > either, in case someone is using backup directories. > > -- > I was Linux when Linux wasn't cool. > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
