Re: Got a puzzle here

[tomas]

On Mon, Nov 04, 2019 at 12:48:00PM -0500, Gene Heskett wrote:
> On Monday 04 November 2019 09:29:48 Greg Wooledge wrote:
> 
> > On Mon, Nov 04, 2019 at 09:08:36AM -0500, Gene Heskett wrote:

[...]

> And that looks like nginx is a lot easier to program than apache2.

Nearly anything is -- except perhaps Sendmail.

Cheers
-- t


signature.asc
Description: Digital signature

Re: Got a puzzle here

[Gene Heskett]

On Monday 04 November 2019 09:29:48 Greg Wooledge wrote:

> On Mon, Nov 04, 2019 at 09:08:36AM -0500, Gene Heskett wrote:
> > Does apache2 have its own module that would prevent its responding
> > to an ipv4 address presented in a .conf file as "xx.xx.xx.xx/24"
> > format?
>
> Well, looking at your larger issue, you might find it more useful
> to block these bots based on their user-agent strings.
>
> The first thing you want to do is actually find a log entry from one
> of these bots, so you know what you're dealing with.  If you're not
> logging user-agent, then you'll want to turn that on first.
>
> Once you have that information, you can google "apache block user
> agent" or whatever search terms work best for you.
>
> I'm using nginx on my (real) web site, so I don't have the
> Apache-specific knowledge you're looking for.  I do block one type of
> bot based on its user-agent.  It's pretty simple in nginx:
>
> greg@remote:/etc/nginx$ cat sites-enabled/mywiki.wooledge.org
> server {
> listen 80;
> listen 443 ssl;
> server_name mywiki.wooledge.org;
>
> if ($http_user_agent ~ SemrushBot) {
> return 403;
> }
> ...

And that looks like nginx is a lot easier to program than apache2.  The 
above makes sense. Once I'm functioning again after tomorrows heart 
valve work, I'll investigate that, probably on a fresh drive and a 
buster 10.1 install.  Thanks Greg.

Cheers, Gene Heskett

Re: Got a puzzle here

[tomas]

On Mon, Nov 04, 2019 at 09:44:56AM -0500, Gene Heskett wrote:

[...]

> If its not built to use libwrap0, then I assume it has its own module to 
> similarly restrict its response to a specified incoming source address?
> 
> And it is?

See above :)

Much more flexible than tcpwrappers. And once you got that up and
running, you might escalate to fail2ban (hint: get your Apache to
recognize reliably those clients you don't want -- be it by host
name, IP address, user agent string, whatever else or a combination
thereof; then teach your apache to mumble something in its log
file which fail2ban understands. Then unleash fail2ban).

But first things first -- and perhaps the Apache config is enough
for your needs.

Cheers
-- t


signature.asc
Description: Digital signature

Re: Got a puzzle here

[Gene Heskett]

On Monday 04 November 2019 08:50:20 Greg Wooledge wrote:

> On Fri, Nov 01, 2019 at 06:46:25PM -0400, Gene Heskett wrote:
> > I'll make sure its installed. Right now. But that is a problem:
> > root@coyote:etc$ apt install tcpwrappers
>
> ... no, Gene.
>
> TCP wrappers is a *library*, and its package name in Debian is
> libwrap0.
>
Already installed, claims latest version.

> wooledg:~$ apt-cache search tcp wrappers
> fakeroot - tool for simulating superuser privileges
> libfakeroot - tool for simulating superuser privileges - shared
> libraries libauthen-libwrap-perl - module providing access to the TCP
> Wrappers library python-tcpwrap - Python interface for libwrap0 (TCP
> wrappers) ruby-proxifier - add support for HTTP or SOCKS proxies
> sendmail - powerful, efficient, and scalable Mail Transport Agent
> (metapackage) sendmail-bin - powerful, efficient, and scalable Mail
> Transport Agent libwrap0 - Wietse Venema's TCP wrappers library
> libwrap0-dev - Wietse Venema's TCP wrappers library, development files
> ucspi-tcp - command-line tools for building TCP client-server
> applications ucspi-tcp-ipv6 - command-line tools for building TCP
> client-server applications (IPv6)
>
> (At least learn how to use the basic Debian utilities.)
>
> A given program is either built with libwrap, or it isn't.  You can't
> just install it and have it affect programs that aren't built to use
> it.
>
> (It actually has a second mode of operation, though -- in a service
> manager like inetd or xinetd, you can use TCP wrappers as an actual
> wrapper program that inetd invokes.  Then the wrapper can validate
> whether it wants to continue this connection or not, and if it chooses
> to allow the connection, it will exec the actual daemon that it's
> wrapping, e.g. in.ftpd or in.telnetd or some other relic of the bronze
> age.)
>
> (None of this applies to Apache, which is NOT linked with libwrap0,
> and which is NOT launched by a service manager.  It's a standalone
> daemon that does its own socket listening, so there's no place to
> insert a chain-loading wrapper program.)

If its not built to use libwrap0, then I assume it has its own module to 
similarly restrict its response to a specified incoming source address?

And it is?

Thanks.

Gene Heskett

Re: Got a puzzle here

[tomas]

On Mon, Nov 04, 2019 at 09:08:36AM -0500, Gene Heskett wrote:
> On Monday 04 November 2019 08:45:42 Greg Wooledge wrote:
> 
> > On Fri, Nov 01, 2019 at 11:06:26PM +0100, to...@tuxteam.de wrote:
> > > That will depend on whether apache is compiled with tcpwrappers
> > > (that's the library implementing the hosts.{allow,deny} policies). I
> > > don't know whether Debian's distribution does that (perhaps others
> > > will).
> >
> > It's not.
> 
> Oh fudge, no wonder my mechinations with /etc/hosts.deny have zero long 
> term effect.
> 
> Does apache2 have its own module that would prevent its responding to an 
> ipv4 address presented in a .conf file as "xx.xx.xx.xx/24" format?

More or less (your request is too specific, the /24 can be an arbitrary
netmask). This has come up already in this thread.

See, e.g. https://httpd.apache.org/docs/2.4/howto/access.html for several
ways to skin that cat.

I can't tell you how to actually weave those configuration snippets
into the Debian-provided config -- it's a long time since I "did"
Apache myself.

I know that Debian breaks down the config into multiple files to
ease separate package configuration. All lives somewhere under
/etc/apache2, there are subdirectories for configuration snippets
(conf-available and conf-enabled -- the latter being just a link
farm to the former, to ease dis- and enabling of individual config
items: there are commands for that (a2enconf, a2disconf) -- likewise
for different sites (if your Apache is serving several sites).

It's bound to be a panoramic ride. Apache config is a heck of
a dungeon. But I think this is where you should start.

Cheers
-- t


signature.asc
Description: Digital signature

Re: Got a puzzle here

[Greg Wooledge]

On Mon, Nov 04, 2019 at 09:08:36AM -0500, Gene Heskett wrote:
> Does apache2 have its own module that would prevent its responding to an 
> ipv4 address presented in a .conf file as "xx.xx.xx.xx/24" format?

Well, looking at your larger issue, you might find it more useful
to block these bots based on their user-agent strings.

The first thing you want to do is actually find a log entry from one
of these bots, so you know what you're dealing with.  If you're not
logging user-agent, then you'll want to turn that on first.

Once you have that information, you can google "apache block user agent"
or whatever search terms work best for you.

I'm using nginx on my (real) web site, so I don't have the Apache-specific
knowledge you're looking for.  I do block one type of bot based on
its user-agent.  It's pretty simple in nginx:

greg@remote:/etc/nginx$ cat sites-enabled/mywiki.wooledge.org 
server {
listen 80;
listen 443 ssl;
server_name mywiki.wooledge.org;

if ($http_user_agent ~ SemrushBot) {
return 403;
}
...

Re: Got a puzzle here

[Gene Heskett]

On Monday 04 November 2019 08:45:42 Greg Wooledge wrote:

> On Fri, Nov 01, 2019 at 11:06:26PM +0100, to...@tuxteam.de wrote:
> > That will depend on whether apache is compiled with tcpwrappers
> > (that's the library implementing the hosts.{allow,deny} policies). I
> > don't know whether Debian's distribution does that (perhaps others
> > will).
>
> It's not.

Oh fudge, no wonder my mechinations with /etc/hosts.deny have zero long 
term effect.

Does apache2 have its own module that would prevent its responding to an 
ipv4 address presented in a .conf file as "xx.xx.xx.xx/24" format? These 
bots are not just indexing the site, they are downloading the whole site 
non-stop, repeatedly and have been for over a week now, burning up what 
little upload bandwidth I have, blocking access from folks who might 
have a legit reason to want this data.  The classic definition of a 
DDOS.

I've a request in to join the apache2 mailing list.  I've also emailed 
postmaster@offender's, but the only answer has been from yandex.ru, in 
russian of course.  That to me is kin of swahili.

> arc3:~$ dpkg -l \*apache\* | grep '^.i'
> ii  apache2  2.4.38-3+deb10u3 i386
> Apache HTTP Server ii  apache2-bin 
> 2.4.38-3+deb10u3 i386 Apache HTTP Server (modules and
> other binary files) ii  apache2-data
> 2.4.38-3+deb10u3 all  Apache HTTP Server (common files) ii
>  apache2-utils2.4.38-3+deb10u3 i386
> Apache HTTP Server (utility programs for web servers) ii 
> libapache2-mod-authnz-pam1.2.0-1  i386 PAM
> authorization checker and PAM Basic Authentication provider ii 
> libapache2-mod-php   2:7.3+69 all 
> server-side, HTML-embedded scripting language (Apache 2 module)
> (default) ii  libapache2-mod-php5  5.6.30+dfsg-0+deb8u1
> i386 server-side, HTML-embedded scripting language (Apache 2
> module) ii  libapache2-mod-php7.07.0.33-0+deb9u3  i386
> server-side, HTML-embedded scripting language (Apache 2
> module) ii  libapache2-mod-php7.37.3.9-1~deb10u1  i386
> server-side, HTML-embedded scripting language (Apache 2
> module) arc3:~$ for i in apache2 apache2-bin apache2-data
> apache2-utils; do apt-cache show "$i" | grep wrap; done arc3:~$

Re: Got a puzzle here

[Greg Wooledge]

On Fri, Nov 01, 2019 at 06:46:25PM -0400, Gene Heskett wrote:
> I'll make sure its installed. Right now. But that is a problem:
> root@coyote:etc$ apt install tcpwrappers

... no, Gene.

TCP wrappers is a *library*, and its package name in Debian is libwrap0.

wooledg:~$ apt-cache search tcp wrappers
fakeroot - tool for simulating superuser privileges
libfakeroot - tool for simulating superuser privileges - shared libraries
libauthen-libwrap-perl - module providing access to the TCP Wrappers library
python-tcpwrap - Python interface for libwrap0 (TCP wrappers)
ruby-proxifier - add support for HTTP or SOCKS proxies
sendmail - powerful, efficient, and scalable Mail Transport Agent (metapackage)
sendmail-bin - powerful, efficient, and scalable Mail Transport Agent
libwrap0 - Wietse Venema's TCP wrappers library
libwrap0-dev - Wietse Venema's TCP wrappers library, development files
ucspi-tcp - command-line tools for building TCP client-server applications
ucspi-tcp-ipv6 - command-line tools for building TCP client-server applications 
(IPv6)

(At least learn how to use the basic Debian utilities.)

A given program is either built with libwrap, or it isn't.  You can't
just install it and have it affect programs that aren't built to use it.

(It actually has a second mode of operation, though -- in a service
manager like inetd or xinetd, you can use TCP wrappers as an actual
wrapper program that inetd invokes.  Then the wrapper can validate
whether it wants to continue this connection or not, and if it chooses
to allow the connection, it will exec the actual daemon that it's
wrapping, e.g. in.ftpd or in.telnetd or some other relic of the bronze
age.)

(None of this applies to Apache, which is NOT linked with libwrap0, and
which is NOT launched by a service manager.  It's a standalone daemon
that does its own socket listening, so there's no place to insert a
chain-loading wrapper program.)

Re: Got a puzzle here

[Greg Wooledge]

On Fri, Nov 01, 2019 at 11:06:26PM +0100, to...@tuxteam.de wrote:
> That will depend on whether apache is compiled with tcpwrappers (that's
> the library implementing the hosts.{allow,deny} policies). I don't
> know whether Debian's distribution does that (perhaps others will).

It's not.

arc3:~$ dpkg -l \*apache\* | grep '^.i'
ii  apache2  2.4.38-3+deb10u3 i386 Apache 
HTTP Server
ii  apache2-bin  2.4.38-3+deb10u3 i386 Apache 
HTTP Server (modules and other binary files)
ii  apache2-data 2.4.38-3+deb10u3 all  Apache 
HTTP Server (common files)
ii  apache2-utils2.4.38-3+deb10u3 i386 Apache 
HTTP Server (utility programs for web servers)
ii  libapache2-mod-authnz-pam1.2.0-1  i386 PAM 
authorization checker and PAM Basic Authentication provider
ii  libapache2-mod-php   2:7.3+69 all  
server-side, HTML-embedded scripting language (Apache 2 module) (default)
ii  libapache2-mod-php5  5.6.30+dfsg-0+deb8u1 i386 
server-side, HTML-embedded scripting language (Apache 2 module)
ii  libapache2-mod-php7.07.0.33-0+deb9u3  i386 
server-side, HTML-embedded scripting language (Apache 2 module)
ii  libapache2-mod-php7.37.3.9-1~deb10u1  i386 
server-side, HTML-embedded scripting language (Apache 2 module)
arc3:~$ for i in apache2 apache2-bin apache2-data apache2-utils; do apt-cache 
show "$i" | grep wrap; done
arc3:~$ 

Re: Got a puzzle here

[Gene Heskett]

On Saturday 02 November 2019 08:55:51 Dan Purgert wrote:

>  wrote:
> > On Fri, Nov 01, 2019 at 06:46:25PM -0400, Gene Heskett wrote:
> >> [...]
> >> Humm, further exercise with a digital potatoe fork discloses that
> >> "tcpd" is the old tcpwappers, and its installed and running.
> >
> > Yikes. I didn't even know (or remember) that it runs under a
> > different name.
>
> Seems the actual package name is "tcp-wrappers", at least according to
> https://packages.debian.org

Where I was going by the name that synaptics shows me on this stretch 
install. Shrug, as long as it works.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[Reco]

Hi.

On Sat, Nov 02, 2019 at 12:55:51PM -, Dan Purgert wrote:
>  wrote:
> > On Fri, Nov 01, 2019 at 06:46:25PM -0400, Gene Heskett wrote:
> 
> >> [...]
> >> Humm, further exercise with a dital potatoe fork discloses that
> >> "tcpd" is the old tcpwappers, and its installed and running.
> >
> > Yikes. I didn't even know (or remember) that it runs under a different
> > name.
> 
> Seems the actual package name is "tcp-wrappers", at least according to
> https://packages.debian.org

It's a source package name. A binary packages are "tcpd" and "libwrap0".
Also, "apt rdepends libwrap0 | grep apache".

Reco

Re: Got a puzzle here

[Dan Purgert]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

 wrote:
> On Fri, Nov 01, 2019 at 06:46:25PM -0400, Gene Heskett wrote:

>> [...]
>> Humm, further exercise with a dital potatoe fork discloses that
>> "tcpd" is the old tcpwappers, and its installed and running.
>
> Yikes. I didn't even know (or remember) that it runs under a different
> name.

Seems the actual package name is "tcp-wrappers", at least according to
https://packages.debian.org


-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAl29fNcACgkQjhHd8xJ5
ooHJHAgAs3xdhVEa/kd5JOJROhjtd3/XOP36o4rxuCgqK1FlUFPgqiFxgohLitlJ
RXAT13rNyzQdrQ9OI0g80w8yr1cYZtVocVlOFlenDxGctra7fZgGr+379vNV/37W
PT5+HnRZCHN9i7fcgGj1RDqmHyG4py3VeZOPBZBRJj9BduTsgTKdcJrmBZQEoePc
ojm8a3P02flUg+8/LcNytW4WpBcwKh6+6ydvbs2W53JQZP3NWS1xfd2aRL5i1f7r
nSSNS9DOBg3H7jTe780/qqqM+DoRtuNUkxm6sKAb0bNkTdUaELSfoJBFvhPB2My0
0JIxoEcITTaSy9Kqbcjs/Wbo3aCidQ==
=6iYV
-END PGP SIGNATURE-

-- 
|_|O|_| 
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

Re: Got a puzzle here

[tomas]

On Fri, Nov 01, 2019 at 06:46:25PM -0400, Gene Heskett wrote:

[...]

> > For tcpwrappers... dunno.
> >
> I'll make sure its installed. Right now. But that is a problem:

[...]

> E: Unable to locate package tcpwrappers

Bummer.

> Humm, further exercise with a dital potatoe fork discloses that "tcpd" is 
> the old tcpwappers, and its installed and running.

Yikes. I didn't even know (or remember) that it runs under a different
name.

[...]

> Your trivia factoid for today I guess.  Thanks Tomas.

Thanks for the heads-up :-)

Cheers
-- t


signature.asc
Description: Digital signature

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 18:06:26 to...@tuxteam.de wrote:

> On Fri, Nov 01, 2019 at 04:35:24PM -0400, Gene Heskett wrote:
> > On Friday 01 November 2019 14:44:07 Gene Heskett wrote:
> > > On Friday 01 November 2019 13:43:04 to...@tuxteam.de wrote:
> > > > On Fri, Nov 01, 2019 at 01:12:46PM -0400, Gene Heskett wrote:
> > > > > On Friday 01 November 2019 12:42:21 to...@tuxteam.de wrote:
> > > >
> > > > [...]
> > > >
> > > > > > https://en.wikipedia.org/wiki/Robots.txt
> > > > >
> > > > > Did that, blanket Disallow for all didn't stop them. But they
> > > > > are spacing out the requests now, so the average traffic is
> > > > > very low. I can tolerate that.
> > > >
> > > > You can block by user agent, that's more drastic.
> > > >
> > > > If I've been paying attention, you are on apache. Then this
> > > > might be relevant:
> > > >
> > > >
> > > > https://httpd.apache.org/docs/2.4/rewrite/access.html#blocking-o
> > > >f-ro bo ts
> > >
> > > Unfortunately  its starts with a very fuzzy explanation of where
> > > to put all those examples. I suspect thats because I probably
> > > don't have near all of apache2 installed.
>
> Those are directives for the apache configuration, somewhere under
> /etc/apache2, I guess (it's a while since I did Apache -- these days I
> prefer something smaller, like lighttpd).
>
> > Nother question, perhaps back on thread. I have 30 or so copies of a
> > rotots.txt that should block the bots, but in half an hour after
> > I've done an apache2 restart, they are back again.
> >
> > Does apache2 pay any attention to hosts.deny, and does hosts.deny
> > honor CIDR addresses which would allow me to block the whole /24
> > they are coming from. What they are doing is, since they're using
> > all my upload bandwidth, qualifies as a DDOS.
>
> That will depend on whether apache is compiled with tcpwrappers
> (that's the library implementing the hosts.{allow,deny} policies). I
> don't know whether Debian's distribution does that (perhaps others
> will).
>
> > And at this point I don't care if I play dirty with stuff outside of
> > apache2's control.
> >
> > Ack the man page CIDR working is yes.  But there is no clue what log
> > file to look at to see if its working to control what apache2 does.
> > That would  be most helpfull.
> >
> > And a new one just showed up, a Mac known as
> > bytespi...@bytedance.com, and he/she got put in hosts.deny. tsk tsk.
>
> IMO Apache config is the cleaner option here, but hey, it's your box
> :)
That was my thinking also
>
> For apache, you'll find the log files in /var/log/apache2, I think.
> For tcpwrappers... dunno.
>
I'll make sure its installed. Right now. But that is a problem:
root@coyote:etc$ apt install tcpwrappers
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package tcpwrappers

Humm, further exercise with a dital potatoe fork discloses that "tcpd" is 
the old tcpwappers, and its installed and running. It 
uses /etc/hosts.allow and /etc/hosts.deny so I'd expect them to work. 
which they seem to be.

Your trivia factoid for today I guess.  Thanks Tomas.

> Cheers
> -- t


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 17:33:56 mick crane wrote:

> >> That was my assumption, and might help keep track of what apache2
> >> can see. Except I just ran it, and it didn't show to the full path
> >> depth. Perhaps it has a depth control?
> >
> > that's the "f" bit of the options
>
> Oh hang on, there's something about a limit to the depth but I can't
> remember what it was,
> maybe was a windows thing, maybe not.
And this just showed up in a dmesg report, never seen here before.

TCP: eth0: Driver has suspect GRO implementation, TCP performance may be 
compromised.

This is now an amd64 stretch install. Just updated today.

Anybody know what this is all about?

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[tomas]

On Fri, Nov 01, 2019 at 04:35:24PM -0400, Gene Heskett wrote:
> On Friday 01 November 2019 14:44:07 Gene Heskett wrote:
> 
> > On Friday 01 November 2019 13:43:04 to...@tuxteam.de wrote:
> > > On Fri, Nov 01, 2019 at 01:12:46PM -0400, Gene Heskett wrote:
> > > > On Friday 01 November 2019 12:42:21 to...@tuxteam.de wrote:
> > >
> > > [...]
> > >
> > > > > https://en.wikipedia.org/wiki/Robots.txt
> > > >
> > > > Did that, blanket Disallow for all didn't stop them. But they are
> > > > spacing out the requests now, so the average traffic is very low.
> > > > I can tolerate that.
> > >
> > > You can block by user agent, that's more drastic.
> > >
> > > If I've been paying attention, you are on apache. Then this might
> > > be relevant:
> > >
> > >
> > > https://httpd.apache.org/docs/2.4/rewrite/access.html#blocking-of-ro
> > >bo ts
> >
> > Unfortunately  its starts with a very fuzzy explanation of where to
> > put all those examples. I suspect thats because I probably don't have
> > near all of apache2 installed.

Those are directives for the apache configuration, somewhere under /etc/apache2,
I guess (it's a while since I did Apache -- these days I prefer something
smaller, like lighttpd).

> Nother question, perhaps back on thread. I have 30 or so copies of a 
> rotots.txt that should block the bots, but in half an hour after I've 
> done an apache2 restart, they are back again.
> 
> Does apache2 pay any attention to hosts.deny, and does hosts.deny honor 
> CIDR addresses which would allow me to block the whole /24 they are 
> coming from. What they are doing is, since they're using all my upload 
> bandwidth, qualifies as a DDOS.

That will depend on whether apache is compiled with tcpwrappers (that's
the library implementing the hosts.{allow,deny} policies). I don't
know whether Debian's distribution does that (perhaps others will).

> And at this point I don't care if I play dirty with stuff outside of 
> apache2's control.
> 
> Ack the man page CIDR working is yes.  But there is no clue what log file 
> to look at to see if its working to control what apache2 does. That 
> would  be most helpfull.
> 
> And a new one just showed up, a Mac known as bytespi...@bytedance.com, 
> and he/she got put in hosts.deny. tsk tsk.

IMO Apache config is the cleaner option here, but hey, it's your box :)

For apache, you'll find the log files in /var/log/apache2, I think.
For tcpwrappers... dunno.

Cheers
-- t


signature.asc
Description: Digital signature

Re: Got a puzzle here

[mick crane]

That was my assumption, and might help keep track of what apache2 can
see. Except I just ran it, and it didn't show to the full path depth.
Perhaps it has a depth control?


that's the "f" bit of the options


Oh hang on, there's something about a limit to the depth but I can't 
remember what it was,

maybe was a windows thing, maybe not.



--
Key ID4BFEBB31

Re: Got a puzzle here

[mick crane]

On 2019-11-01 18:18, Gene Heskett wrote:

On Friday 01 November 2019 13:37:02 mick crane wrote:


On 2019-11-01 17:20, Gene Heskett wrote:
> On Friday 01 November 2019 13:12:42 mick crane wrote:
>> On 2019-11-01 17:01, mick crane wrote:
>> > On 2019-11-01 16:47, mick crane wrote:
>> >> On 2019-11-01 16:18, Gene Heskett wrote:
>> >>> lsof |grep www-data shows me several hundred lines of "no
>> >>> perms" errors
>> >>> without nameing the source of the errors other than apache2.
>> >>> How do I
>> >>> find the file causing those errors?  Thats question #1 here.
>> >>
>> >> excuse my ignorance.
>> >> You've got files have no permissions ?
>> >>
>> >> "tree -pufid /var/www/html | less"
>> >
>> > Ok I read the man page a bit and is probably
>> > "tree -aguf /var/www/html | less"
>>
>> I'll get my coat
>> need the "p" if want to see if anything has funny permissions
>> "tree -pagu /var/www/html | less"
>> is more readable then if spot one can look where it is.
>
> But I find I need tree installed before it will work. :)

that's weird :)
I didn't know it was there.
Seems handy for recursively looking through a directory.


That was my assumption, and might help keep track of what apache2 can
see. Except I just ran it, and it didn't show to the full path depth.
Perhaps it has a depth control?


that's the "f" bit of the options

--
Key ID4BFEBB31

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 14:44:07 Gene Heskett wrote:

> On Friday 01 November 2019 13:43:04 to...@tuxteam.de wrote:
> > On Fri, Nov 01, 2019 at 01:12:46PM -0400, Gene Heskett wrote:
> > > On Friday 01 November 2019 12:42:21 to...@tuxteam.de wrote:
> >
> > [...]
> >
> > > > https://en.wikipedia.org/wiki/Robots.txt
> > >
> > > Did that, blanket Disallow for all didn't stop them. But they are
> > > spacing out the requests now, so the average traffic is very low.
> > > I can tolerate that.
> >
> > You can block by user agent, that's more drastic.
> >
> > If I've been paying attention, you are on apache. Then this might
> > be relevant:
> >
> >
> > https://httpd.apache.org/docs/2.4/rewrite/access.html#blocking-of-ro
> >bo ts
>
> Unfortunately  its starts with a very fuzzy explanation of where to
> put all those examples. I suspect thats because I probably don't have
> near all of apache2 installed.
>
> > Cheers
> > -- t
>
> Cheers, Gene Heskett

Nother question, perhaps back on thread. I have 30 or so copies of a 
rotots.txt that should block the bots, but in half an hour after I've 
done an apache2 restart, they are back again.

Does apache2 pay any attention to hosts.deny, and does hosts.deny honor 
CIDR addresses which would allow me to block the whole /24 they are 
coming from. What they are doing is, since they're using all my upload 
bandwidth, qualifies as a DDOS.

And at this point I don't care if I play dirty with stuff outside of 
apache2's control.

Ack the man page CIDR working is yes.  But there is no clue what log file 
to look at to see if its working to control what apache2 does. That 
would  be most helpfull.

And a new one just showed up, a Mac known as bytespi...@bytedance.com, 
and he/she got put in hosts.deny. tsk tsk.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 13:43:04 to...@tuxteam.de wrote:

> On Fri, Nov 01, 2019 at 01:12:46PM -0400, Gene Heskett wrote:
> > On Friday 01 November 2019 12:42:21 to...@tuxteam.de wrote:
>
> [...]
>
> > > https://en.wikipedia.org/wiki/Robots.txt
> >
> > Did that, blanket Disallow for all didn't stop them. But they are
> > spacing out the requests now, so the average traffic is very low. I
> > can tolerate that.
>
> You can block by user agent, that's more drastic.
>
> If I've been paying attention, you are on apache. Then this might
> be relevant:
>
>  
> https://httpd.apache.org/docs/2.4/rewrite/access.html#blocking-of-robo
>ts
>
Unfortunately  its starts with a very fuzzy explanation of where to put 
all those examples. I suspect thats because I probably don't have near 
all of apache2 installed. 

> Cheers
> -- t


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 13:37:02 mick crane wrote:

> On 2019-11-01 17:20, Gene Heskett wrote:
> > On Friday 01 November 2019 13:12:42 mick crane wrote:
> >> On 2019-11-01 17:01, mick crane wrote:
> >> > On 2019-11-01 16:47, mick crane wrote:
> >> >> On 2019-11-01 16:18, Gene Heskett wrote:
> >> >>> lsof |grep www-data shows me several hundred lines of "no
> >> >>> perms" errors
> >> >>> without nameing the source of the errors other than apache2. 
> >> >>> How do I
> >> >>> find the file causing those errors?  Thats question #1 here.
> >> >>
> >> >> excuse my ignorance.
> >> >> You've got files have no permissions ?
> >> >>
> >> >> "tree -pufid /var/www/html | less"
> >> >
> >> > Ok I read the man page a bit and is probably
> >> > "tree -aguf /var/www/html | less"
> >>
> >> I'll get my coat
> >> need the "p" if want to see if anything has funny permissions
> >> "tree -pagu /var/www/html | less"
> >> is more readable then if spot one can look where it is.
> >
> > But I find I need tree installed before it will work. :)
>
> that's weird :)
> I didn't know it was there.
> Seems handy for recursively looking through a directory.

That was my assumption, and might help keep track of what apache2 can 
see. Except I just ran it, and it didn't show to the full path depth. 
Perhaps it has a depth control?


> mick


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[tomas]

On Fri, Nov 01, 2019 at 01:12:46PM -0400, Gene Heskett wrote:
> On Friday 01 November 2019 12:42:21 to...@tuxteam.de wrote:

[...]

> > https://en.wikipedia.org/wiki/Robots.txt
> >
> Did that, blanket Disallow for all didn't stop them. But they are spacing 
> out the requests now, so the average traffic is very low. I can tolerate 
> that.

You can block by user agent, that's more drastic.

If I've been paying attention, you are on apache. Then this might
be relevant:

  https://httpd.apache.org/docs/2.4/rewrite/access.html#blocking-of-robots

Cheers
-- t


signature.asc
Description: Digital signature

Re: Got a puzzle here

[mick crane]

On 2019-11-01 17:20, Gene Heskett wrote:

On Friday 01 November 2019 13:12:42 mick crane wrote:


On 2019-11-01 17:01, mick crane wrote:
> On 2019-11-01 16:47, mick crane wrote:
>> On 2019-11-01 16:18, Gene Heskett wrote:
>>> lsof |grep www-data shows me several hundred lines of "no perms"
>>> errors
>>> without nameing the source of the errors other than apache2.  How
>>> do I
>>> find the file causing those errors?  Thats question #1 here.
>>
>> excuse my ignorance.
>> You've got files have no permissions ?
>>
>> "tree -pufid /var/www/html | less"
>
> Ok I read the man page a bit and is probably
> "tree -aguf /var/www/html | less"

I'll get my coat
need the "p" if want to see if anything has funny permissions
"tree -pagu /var/www/html | less"
is more readable then if spot one can look where it is.

But I find I need tree installed before it will work. :)


that's weird :)
I didn't know it was there.
Seems handy for recursively looking through a directory.

mick


--
Key ID4BFEBB31

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 13:12:42 mick crane wrote:

> On 2019-11-01 17:01, mick crane wrote:
> > On 2019-11-01 16:47, mick crane wrote:
> >> On 2019-11-01 16:18, Gene Heskett wrote:
> >>> lsof |grep www-data shows me several hundred lines of "no perms"
> >>> errors
> >>> without nameing the source of the errors other than apache2.  How
> >>> do I
> >>> find the file causing those errors?  Thats question #1 here.
> >>
> >> excuse my ignorance.
> >> You've got files have no permissions ?
> >>
> >> "tree -pufid /var/www/html | less"
> >
> > Ok I read the man page a bit and is probably
> > "tree -aguf /var/www/html | less"
>
> I'll get my coat
> need the "p" if want to see if anything has funny permissions
> "tree -pagu /var/www/html | less"
> is more readable then if spot one can look where it is.
But I find I need tree installed before it will work. :)

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 12:52:47 Dan Purgert wrote:

>  wrote:
> > On Fri, Nov 01, 2019 at 04:29:38PM -, Dan Purgert wrote:
> >> Gene Heskett wrote:
> >>> [...]
> >>> Humm. in /var/log/httpd/other_vhosts_access.log, is looks as if
> >>> both www.yandex.com/bots and www.semrush.com/bot.html are trying
> >>> to download all of my nitros9 stuff, which is a unix like OS for
> >>> the trs-80 color computer.
> >>
> >> Sounds like you might want a robots.txt file in there, to keep them
> >> from grabbing it.
> >
> > https://en.wikipedia.org/wiki/Robots.txt
>
> Whoops, could've sworn I had the link in there too.  Curse
> telemarketers.

Daily, if not more often, Dan. :(

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 12:47:46 mick crane wrote:

> On 2019-11-01 16:18, Gene Heskett wrote:
> > lsof |grep www-data shows me several hundred lines of "no perms"
> > errors without nameing the source of the errors other than apache2. 
> > How do I find the file causing those errors?  Thats question #1
> > here.
>
> excuse my ignorance.
> You've got files have no permissions ?
>
> "tree -pufid /var/www/html | less"
>
> mick

Miss understood, fumbled fingers.  No permissions to execute or access. 
www-data doesn't own the file IOW.

Thanks Mick.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[mick crane]

On 2019-11-01 17:01, mick crane wrote:

On 2019-11-01 16:47, mick crane wrote:

On 2019-11-01 16:18, Gene Heskett wrote:

lsof |grep www-data shows me several hundred lines of "no perms" 
errors
without nameing the source of the errors other than apache2.  How do 
I

find the file causing those errors?  Thats question #1 here.


excuse my ignorance.
You've got files have no permissions ?

"tree -pufid /var/www/html | less"


Ok I read the man page a bit and is probably
"tree -aguf /var/www/html | less"


I'll get my coat
need the "p" if want to see if anything has funny permissions
"tree -pagu /var/www/html | less"
is more readable then if spot one can look where it is.

--
Key ID4BFEBB31

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 12:42:21 to...@tuxteam.de wrote:

> On Fri, Nov 01, 2019 at 04:29:38PM -, Dan Purgert wrote:
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA256
> >
> > Gene Heskett wrote:
> > > [...]
> > > Humm. in /var/log/httpd/other_vhosts_access.log, is looks as if
> > > both www.yandex.com/bots and www.semrush.com/bot.html are trying
> > > to download all of my nitros9 stuff, which is a unix like OS for
> > > the trs-80 color computer.
> >
> > Sounds like you might want a robots.txt file in there, to keep them
> > from grabbing it.
>
> https://en.wikipedia.org/wiki/Robots.txt
>
Did that, blanket Disallow for all didn't stop them. But they are spacing 
out the requests now, so the average traffic is very low. I can tolerate 
that.

Thanks Tomas

> Cheers
> -- t


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[tomas]

On Fri, Nov 01, 2019 at 04:52:47PM -, Dan Purgert wrote:
[...]

> Whoops, could've sworn I had the link in there too.  Curse
> telemarketers.

:-)

Cheers
-- t


signature.asc
Description: Digital signature

Re: Got a puzzle here

[mick crane]

On 2019-11-01 16:47, mick crane wrote:

On 2019-11-01 16:18, Gene Heskett wrote:

lsof |grep www-data shows me several hundred lines of "no perms" 
errors

without nameing the source of the errors other than apache2.  How do I
find the file causing those errors?  Thats question #1 here.


excuse my ignorance.
You've got files have no permissions ?

"tree -pufid /var/www/html | less"


Ok I read the man page a bit and is probably
"tree -aguf /var/www/html | less"



--
Key ID4BFEBB31

Re: Got a puzzle here

[Dan Purgert]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

 wrote:
> On Fri, Nov 01, 2019 at 04:29:38PM -, Dan Purgert wrote:
>> Gene Heskett wrote:
>>> [...]
>>> Humm. in /var/log/httpd/other_vhosts_access.log, is looks as if both
>>> www.yandex.com/bots and www.semrush.com/bot.html are trying to
>>> download all of my nitros9 stuff, which is a unix like OS for the
>>> trs-80 color computer.
>>
>> Sounds like you might want a robots.txt file in there, to keep them from
>> grabbing it.
>
> https://en.wikipedia.org/wiki/Robots.txt

Whoops, could've sworn I had the link in there too.  Curse
telemarketers.

-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAl28Yt8ACgkQjhHd8xJ5
ooGoIwf+OR+RDKsR4YJ5UKnhCVceJWfM96h7oaszbQISfj7RMWa/h2pn0tvx633a
ufVG4WQKc02CKccUgtZbt2CXRNT60WGBy8FJe/65+wyOSTbUlkfyCD+UzZotABJx
3qojwcOKG8kPHf8992wirbzg6OOEOvPlnV5T0wWqmJSIwzlIA5KUKl924QcZuwg2
mDkGj3AaAqnLhB+5kLfFY9eVeDHLck4Jvja6dRZ/QHWz8614ai12T4Hf/Qa+AJDP
SpSfizYC68jSTVVFVTFbWEAvJk7I6/BDQsKNSOEgoHC0iNFOs9QfgjIcGfISdcFt
jR+WeF4GVh69dy8xxRTc/U/SkPWplQ==
=f5Am
-END PGP SIGNATURE-

-- 
|_|O|_| 
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

Re: Got a puzzle here

[Gene Heskett]

On Friday 01 November 2019 12:29:38 Dan Purgert wrote:

> Gene Heskett wrote:
> > [...]
> > Humm. in /var/log/httpd/other_vhosts_access.log, is looks as if both
> > www.yandex.com/bots and www.semrush.com/bot.html are trying to
> > download all of my nitros9 stuff, which is a unix like OS for the
> > trs-80 color computer.
>
> Sounds like you might want a robots.txt file in there, to keep them
> from grabbing it.

I have 2 or 3 in there now.  Where is the most effective place to put it?

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Got a puzzle here

[mick crane]

On 2019-11-01 16:18, Gene Heskett wrote:


lsof |grep www-data shows me several hundred lines of "no perms" errors
without nameing the source of the errors other than apache2.  How do I
find the file causing those errors?  Thats question #1 here.


excuse my ignorance.
You've got files have no permissions ?

"tree -pufid /var/www/html | less"

mick

--
Key ID4BFEBB31

Re: Got a puzzle here

[tomas]

On Fri, Nov 01, 2019 at 04:29:38PM -, Dan Purgert wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> Gene Heskett wrote:
> > [...]
> > Humm. in /var/log/httpd/other_vhosts_access.log, is looks as if both 
> > www.yandex.com/bots and www.semrush.com/bot.html are trying to download 
> > all of my nitros9 stuff, which is a unix like OS for the trs-80 color 
> > computer.
> 
> Sounds like you might want a robots.txt file in there, to keep them from
> grabbing it.

https://en.wikipedia.org/wiki/Robots.txt

Cheers
-- t


signature.asc
Description: Digital signature

Re: Got a puzzle here

[Dan Purgert]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Gene Heskett wrote:
> [...]
> Humm. in /var/log/httpd/other_vhosts_access.log, is looks as if both 
> www.yandex.com/bots and www.semrush.com/bot.html are trying to download 
> all of my nitros9 stuff, which is a unix like OS for the trs-80 color 
> computer.

Sounds like you might want a robots.txt file in there, to keep them from
grabbing it.


-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEBcqaUD8uEzVNxUrujhHd8xJ5ooEFAl28XXIACgkQjhHd8xJ5
ooGr0wf/Xfhmfi2q/5XFQUXwB35WgnwPF85aMq9W9Hmmv7AIcaciafA3zDRTmRRu
vzlYMGJbC6/o9V1yIh3mixjxSbsi/NUYAGent+fg6s5Oi96E9nOewd0ZCmVqvaPr
zGMLrJqEB2npB9K2EHvb5QKh+WnMli5V9ypQtfqsqyELneazpjrFCJm6j+SwWldd
/Ct8b5J2Rh95tH2l3xJCMsGbI+wNScVrZflefOOiFQ2+qxQX5O8g0dsyhCEQd4/b
XHNtXU1uwttK2ZU9F6rO3+gjoHJBYiwjD410LNpPFSY+GAPq/x0fTQ3E2IIHZiIU
CS47N2s/P2MUkGamQZ6giwZtlfDgFg==
=Jt75
-END PGP SIGNATURE-

-- 
|_|O|_| 
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

Got a puzzle here

[Gene Heskett]

I run apache2  in a www-data sandbux, so I occasionally see some network 
traffic as someone downloads what I offer on my web page.

gkrellm is showing heavy, at my upload bandwith limit traffic. and has 
been for a couple hours.

But that not enough to point htop fingers at the perp.

lsof |grep www-data shows me several hundred lines of "no perms" errors 
without nameing the source of the errors other than apache2.  How do I 
find the file causing those errors?  Thats question #1 here.

Humm. in /var/log/httpd/other_vhosts_access.log, is looks as if both 
www.yandex.com/bots and www.semrush.com/bot.html are trying to download 
all of my nitros9 stuff, which is a unix like OS for the trs-80 color 
computer.

If that was going to a private user, I'd not have a problem with it, but 
how do I disable the bots from sucking at all on a weekly basis? This is 
not the first time they used my bandwidth for many hours at a time.

Any help with apache2 rules would be much appreciated.

Thanks all.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page