Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Dennis Peterson]

Your proxy is not passing the request to the server. But never give up - try:

curl -H "Range: bytes=35-39" -s --proxy http://proxy:3128 
http://db.us.clamav.net/daily.cvd |strings


On 7/3/18 1:29 PM, SCOTT PACKARD wrote:

Hmm, I went to recreate both cases before replying, and I can get both to work, 
sort of.
I still can't resolve DNS TXT records, but I can it seems throw the URI
http://db.us.clamav.net/daily.cvd to the proxy server and it can handle it.
Beats me what IP db.us.clamav.net resolves to.
I get the whole daily.cvd, with either wget or curl.

curl's -r 35-39 isn't honored though, when fetching externally.  I get the 
whole daily.cvd.

(I swear  this doesn't work at 6am Monday morning though. :) )

Thanks, Scott


-Original Message-
From: clamav-users [mailto:clamav-users-boun...@lists.clamav.net] On Behalf Of 
Dennis Peterson
Sent: Tuesday, July 03, 2018 12:53 PM
To: clamav-users@lists.clamav.net
Subject: [External] Re: [clamav-users] We STILL cannot reliably get virus 
updates (since new mirrors)

Does your wget not support the -e args to access a proxy?

Example:
wget http://someurl.com/filename.html -e use_proxy=yes -e
http_proxy=xxx.xxx.xxx.xxx:3128

The proxy IP or hostname can be used.

dp

On 7/3/18 11:11 AM, SCOTT PACKARD wrote:

The current DNS TXT does not work within my company, as a firewall fully blocks 
things, including DNS.
(as an aside, curl works, with sufficient massaging, but wget cannot, as it 
does not have an option to work with a proxy).

I rely on someone in Arizona to pull definitions from, but sometimes their 
server goes out, other times clamav's content system

breaks,

and it's a pain to figure out which one is the culprit.

Regards, Scott


___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Joel Esler (jesler)]

On Jul 3, 2018, at 4:50 PM, Benny Pedersen mailto:m...@junc.eu>> 
wrote:

Joel Esler (jesler) skrev den 2018-07-03 22:42:

Yes.  But measuring those numbers is the difficult part.  A fresh
install of ClamAV is going to download the main, the daily, then all
the diffs since the last daily, which could be a ton.  It's the people
that are downloading the *same* diff 1000x an hour that are the
problem.

could this be solved in freshclam

maxdiffupdates 50 # number of diff to max update at once
minimalrechecktime 60 # minimal minutes before next recheck new diff updates

adjust as needed

would that atleast settle it a bit ?

Everything helps!  Lowers bandwidth costs!

--
Joel Esler
Sr. Manager
Open Source, Design, Web, and Education
Talos Group
http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Joel Esler (jesler)]

On Jul 3, 2018, at 4:46 PM, Reindl Harald 
mailto:h.rei...@thelounge.net>> wrote:

Am 03.07.2018 um 22:42 schrieb Joel Esler (jesler):
On Jul 3, 2018, at 3:59 PM, Reindl Harald 
mailto:h.rei...@thelounge.net>
> wrote:

voila - all new connections which are more than 5 per hour from the same
IP are dropped, i have similar rules for specific ports and max
connections per client for many years now - no rocket science

Yes.  But measuring those numbers is the difficult part.  A fresh
install of ClamAV is going to download the main, the daily, then all the
diffs since the last daily, which could be a ton.  It's the people that
are downloading the *same* diff 1000x an hour that are the problem.

but these idiots are not fixed by the DNS record at all otherwise that
won#t exist - so it shows once more how useless and in total complex the
DNS/mirror split is instead have just a "version.txt" directly on the mirror

that would likely even solve the problem at all when they have whatever
crap which ignores the DNS (maybe because they have a broken network
with no DNS requests to the world but obviosuly http access to the
mirrors and so download it everytime)

I appreciate your point, and I'd love to streamline it.  But I'd like to figure 
out how to balance the overhead of a TCP connection vs the overhead of a super 
fast UDP connection.  Maybe there is a different way we can do the DNS query to 
make it smarter.

--
Joel Esler
Sr. Manager
Open Source, Design, Web, and Education
Talos Group
http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Benny Pedersen]

Joel Esler (jesler) skrev den 2018-07-03 22:42:


Yes.  But measuring those numbers is the difficult part.  A fresh
install of ClamAV is going to download the main, the daily, then all
the diffs since the last daily, which could be a ton.  It's the people
that are downloading the *same* diff 1000x an hour that are the
problem.


could this be solved in freshclam

maxdiffupdates 50 # number of diff to max update at once
minimalrechecktime 60 # minimal minutes before next recheck new diff 
updates


adjust as needed

would that atleast settle it a bit ?
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Joel Esler (jesler)]

On Jul 3, 2018, at 3:59 PM, Reindl Harald 
mailto:h.rei...@thelounge.net>> wrote:

voila - all new connections which are more than 5 per hour from the same
IP are dropped, i have similar rules for specific ports and max
connections per client for many years now - no rocket science

Yes.  But measuring those numbers is the difficult part.  A fresh install of 
ClamAV is going to download the main, the daily, then all the diffs since the 
last daily, which could be a ton.  It's the people that are downloading the 
*same* diff 1000x an hour that are the problem.

--
Joel Esler
Sr. Manager
Open Source, Design, Web, and Education
Talos Group
http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[SCOTT PACKARD]

Hmm, I went to recreate both cases before replying, and I can get both to work, 
sort of.
I still can't resolve DNS TXT records, but I can it seems throw the URI
http://db.us.clamav.net/daily.cvd to the proxy server and it can handle it.
Beats me what IP db.us.clamav.net resolves to.
I get the whole daily.cvd, with either wget or curl.

curl's -r 35-39 isn't honored though, when fetching externally.  I get the 
whole daily.cvd.

(I swear  this doesn't work at 6am Monday morning though. :) )

Thanks, Scott

> -Original Message-
> From: clamav-users [mailto:clamav-users-boun...@lists.clamav.net] On Behalf 
> Of Dennis Peterson
> Sent: Tuesday, July 03, 2018 12:53 PM
> To: clamav-users@lists.clamav.net
> Subject: [External] Re: [clamav-users] We STILL cannot reliably get virus 
> updates (since new mirrors)
> 
> Does your wget not support the -e args to access a proxy?
> 
> Example:
> wget http://someurl.com/filename.html -e use_proxy=yes -e
> http_proxy=xxx.xxx.xxx.xxx:3128
> 
> The proxy IP or hostname can be used.
> 
> dp
> 
> On 7/3/18 11:11 AM, SCOTT PACKARD wrote:
> > The current DNS TXT does not work within my company, as a firewall fully 
> > blocks things, including DNS.
> > (as an aside, curl works, with sufficient massaging, but wget cannot, as it 
> > does not have an option to work with a proxy).
> >
> > I rely on someone in Arizona to pull definitions from, but sometimes their 
> > server goes out, other times clamav's content system
> breaks,
> > and it's a pain to figure out which one is the culprit.
> >
> > Regards, Scott
> >
> 
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Dennis Peterson]

Does your wget not support the -e args to access a proxy?

Example:
wget http://someurl.com/filename.html -e use_proxy=yes -e 
http_proxy=xxx.xxx.xxx.xxx:3128


The proxy IP or hostname can be used.

dp

On 7/3/18 11:11 AM, SCOTT PACKARD wrote:

The current DNS TXT does not work within my company, as a firewall fully blocks 
things, including DNS.
(as an aside, curl works, with sufficient massaging, but wget cannot, as it 
does not have an option to work with a proxy).

I rely on someone in Arizona to pull definitions from, but sometimes their 
server goes out, other times clamav's content system breaks,
and it's a pain to figure out which one is the culprit.

Regards, Scott



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Joel Esler (jesler)]

On Jul 3, 2018, at 2:11 PM, SCOTT PACKARD 
mailto:scott.pack...@raytheon.com>> wrote:

I rely on someone in Arizona to pull definitions from, but sometimes their 
server goes out, other times clamav's content system breaks,
and it's a pain to figure out which one is the culprit.

Well, hopefully, we have eliminated the content system problems.

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[SCOTT PACKARD]

The current DNS TXT does not work within my company, as a firewall fully blocks 
things, including DNS.
(as an aside, curl works, with sufficient massaging, but wget cannot, as it 
does not have an option to work with a proxy).

I rely on someone in Arizona to pull definitions from, but sometimes their 
server goes out, other times clamav's content system breaks,
and it's a pain to figure out which one is the culprit.

Regards, Scott

> -Original Message-
> From: clamav-users [mailto:clamav-users-boun...@lists.clamav.net] On Behalf 
> Of Christopher X. Candreva
> Sent: Tuesday, July 03, 2018 10:36 AM
> To: ClamAV users ML 
> Subject: [External] Re: [clamav-users] We STILL cannot reliably get virus 
> updates (since new mirrors)
> 
> 
> 
> For everyone (or maybe the one) asking why the DNS system exists, as the
> person who came up with the idea in the first place (or the idea of stealing
> it from the DNSbls ) I thought I would provide a link to the original
> discussion in which is was hashed out ( beaten to death) back in 2004:
> 
> https://lists.gt.net/clamav/users/11106?do=post_view_threaded
> 
> I thought the math was in this thread, but at some point the actual savings
> of being able to check for a new version with a UDP packet over a TCP/http
> HEAD command was calculated, and it was a significant amount of transfer,
> expensive at the time.
> 
> 
> I have to admit I've wondered if Cloudflare and the other CDN's meant it
> outlived it's usefullness, but it's a contribution I'm fairly proud of.
> 
> -Chris
> 
> 
> 
> On Tue, 3 Jul 2018, Joel Esler (jesler) wrote:
> 
> >
> >
> >   On Jul 2, 2018, at 1:17 PM, Reindl Harald
> >wrote:
> >
> > on a typical setup freshclam is running once or twice *daily* while a
> > webserver these days can spit out the same small static txt file many
> > thousands of times per seond with zero load
> >
> >
> > That is not the results we are seeing.  There are a LARGE amount of people
> > that check for updates once or twice a day, yes.  However, we have hundreds
> > of thousands of people that check for updates hundreds of times a day.  We
> > haven't started concentrating on these people yet (our biggest offender is
> > one IP that checks 100,000+ times a day), but clearly that's excessive.  We
> > publish approx 5-6 times a day.  So, let's say you check 50 times a day
> >  Clearly, that's enough.
> >
> > --
> > Joel Esler
> > Sr. Manager
> > Open Source, Design, Web, and Education
> > Talos Group
> > http://www.talosintelligence.com
> >
> >
> 
> ---
> 
> Chris Candreva  --  ch...@westnet.com  --  http://www.westnet.com/~chris
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Joel Esler (jesler)]

On Jul 3, 2018, at 1:36 PM, Christopher X. Candreva 
mailto:ch...@westnet.com>> wrote:

I have to admit I've wondered if Cloudflare and the other CDN's meant it
outlived it's usefullness, but it's a contribution I'm fairly proud of.

That's what we are evaluating.  It's a great system.  The problem is 
maintenance.  We spend a tremendous amount of time maintaining and grooming the 
mirror network (removing dead ones, removing ones that don't work, etc.).

It's more than one full time employee (FTE), let's put it that way.

--
Joel Esler
Sr. Manager
Open Source, Design, Web, and Education
Talos Group
http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Christopher X. Candreva]

For everyone (or maybe the one) asking why the DNS system exists, as the 
person who came up with the idea in the first place (or the idea of stealing 
it from the DNSbls ) I thought I would provide a link to the original 
discussion in which is was hashed out ( beaten to death) back in 2004:

https://lists.gt.net/clamav/users/11106?do=post_view_threaded

I thought the math was in this thread, but at some point the actual savings 
of being able to check for a new version with a UDP packet over a TCP/http 
HEAD command was calculated, and it was a significant amount of transfer, 
expensive at the time.


I have to admit I've wondered if Cloudflare and the other CDN's meant it 
outlived it's usefullness, but it's a contribution I'm fairly proud of.

-Chris



On Tue, 3 Jul 2018, Joel Esler (jesler) wrote:

> 
> 
>   On Jul 2, 2018, at 1:17 PM, Reindl Harald
>wrote:
> 
> on a typical setup freshclam is running once or twice *daily* while a
> webserver these days can spit out the same small static txt file many
> thousands of times per seond with zero load
> 
> 
> That is not the results we are seeing.  There are a LARGE amount of people
> that check for updates once or twice a day, yes.  However, we have hundreds
> of thousands of people that check for updates hundreds of times a day.  We
> haven't started concentrating on these people yet (our biggest offender is
> one IP that checks 100,000+ times a day), but clearly that's excessive.  We
> publish approx 5-6 times a day.  So, let's say you check 50 times a day
>  Clearly, that's enough.
> 
> --
> Joel Esler
> Sr. Manager
> Open Source, Design, Web, and Education
> Talos Group
> http://www.talosintelligence.com
> 
> 

---

Chris Candreva  --  ch...@westnet.com  --  http://www.westnet.com/~chris
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Paul Kosinski]

We used to check once every 90 minutes (16 per day). Plus, we run a
local proxy/mirror so the updates can be served to other machines on
our LAN without extra load on the ClamAV servers.

That was before the new mirroring scheme. Now we're checking several
times per hour in the (vain?) hope of getting something. And to make it
much worse, many of the cvd downloads are out of sync, and thus useless.

P.S. I could probably open a port (of our choosing) in our firewall to
receive some kind of notification and have *that* trigger a freshclam
run on our internal mirror. But I seriously doubt many users could.



On Tue, 3 Jul 2018 16:39:00 +
"Joel Esler (jesler)"  wrote:

> 
> 
> On Jul 2, 2018, at 1:17 PM, Reindl Harald
> mailto:h.rei...@thelounge.net>> wrote:
> 
> on a typical setup freshclam is running once or twice *daily* while a
> webserver these days can spit out the same small static txt file many
> thousands of times per seond with zero load
> 
> That is not the results we are seeing.  There are a LARGE amount of
> people that check for updates once or twice a day, yes.  However, we
> have hundreds of thousands of people that check for updates hundreds
> of times a day.  We haven't started concentrating on these people yet
> (our biggest offender is one IP that checks 100,000+ times a day),
> but clearly that's excessive.  We publish approx 5-6 times a day.
> So, let's say you check 50 times a day  Clearly, that's enough.
> 
> --
> Joel Esler
> Sr. Manager
> Open Source, Design, Web, and Education
> Talos Group
> http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] freshclam works for me

[Joel Esler (jesler)]

On Jul 3, 2018, at 11:38 AM, Noel Jones 
mailto:njo...@megan.vbhcs.org>> wrote:

Using Cloudflare changes the dynamics of updates.  I wonder if it
might be better if everyone pointed to db.clamav.net and 
all the
direct mirrors are dropped.  Let Cloudflare decide what is the
closest POP, that's kinda their job.

Regardless of where you are pointed, this is what is happening.  Effectively, 
everyone in the world is pointed at three mirrors.  Ours.

Cloudflare is caching the files once they are downloaded the first time, and 
you get them from cloudflare.  The first time someone requests a new file, each 
Cloudflare node fetches it from our server, then everyone else that requests 
that file, gets it from cloudflare.  It results in about 80GB to our servers 
every update... but once all the Cloudflare pops have the files, there are no 
more requests that hit our server for that file. (Unless I flush the cache or 
something).

--
Joel Esler
Sr. Manager
Open Source, Design, Web, and Education
Talos Group
http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Freddie Cash]

On Tue, Jul 3, 2018 at 9:28 AM, Paul Kosinski 
wrote:

> The way Linux updates are done in practice is significantly different
> from ClamAV virus signature updates.
>
> With ClamAV, freshclam is automatically run periodically, sees (by
> some low-cost means) that a new file version is *supposed* to be
> available and tries to download it. If either it can't, or worse yet,
> it's the wrong one, it tries the next mirror. This all takes time and
> bandwidth.
>
> With Linux updates, I explicitly ask (via aptitude) what new updates
> are available: It takes some time to retrieve the list. Then I select
> the ones I want and ask to install them. I have *never*, *ever* seen
> this mechanism deliver the wrong version and thus fail to install it.
>

​You obviously haven't tried very hard, then.  :)  Or you don't run a local
repo mirror, at least.

We've run into issues with our local Debian repo mirror.  Usually, it's
that we're asking to install an old version of something and it's no longer
available on the mirror (ie forgot to run "aptitude update" first).  Or the
mirror ran out of disk space, so it didn't actually download the new
packages, but the index files were correctly downloaded/loaded.  Thus,
running "aptitude update" works, but it can't find any of the new files to
download/install.  Or, the Debian project decided to change how things work
in the repo, and that change didn't get propagated to our repo, so aptitude
just stops working on all our servers (the localisation changes for Jessie
were the latest niggle​ to trip us up).  Or, or, or.

The Linux updating method (at least as used in Debian) is not bulletproof.
No update method every is.


> This is due to the fact that the same Debian mirror machine provides
> the new versions of a group of files as provides the list of new
> versions. Thus there is an almost zero chance of a race condition
> (unless some idiot adds a version to the list before uploading the
> actual deb file). Even if set to auto update, I think the *lists*
> always come from the same servers as the files.
>
> It's not a matter of using DNS TXT records, it's a matter of sourcing
> them on a *different* computer than the actual files. This separation
> virtually begs for synchronization problems.


​Or, it's a matter of everyone getting in a tizzy over something that's
really minor in the grand scheme of things.  They've migrated to a new
CDN.  There's going to be teething pains with any new infrastructure.
Instead of trying to "rip-a-new-one" in the devs and demanding everything
be redone from scratch, how about we wait a bit while they work out the
bugs in the new setup.

Are updates completely broken right now?  No.  Are there occasional
hiccups?  Sure.  Are things getting better?  Yeah, they are.  Are they
perfect?  Not yet.  Should they scrap everything and start over?  Hell no.

-- 
Freddie Cash
fjwc...@gmail.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Joel Esler (jesler)]

On Jul 2, 2018, at 1:17 PM, Reindl Harald 
mailto:h.rei...@thelounge.net>> wrote:

on a typical setup freshclam is running once or twice *daily* while a
webserver these days can spit out the same small static txt file many
thousands of times per seond with zero load

That is not the results we are seeing.  There are a LARGE amount of people that 
check for updates once or twice a day, yes.  However, we have hundreds of 
thousands of people that check for updates hundreds of times a day.  We haven't 
started concentrating on these people yet (our biggest offender is one IP that 
checks 100,000+ times a day), but clearly that's excessive.  We publish approx 
5-6 times a day.  So, let's say you check 50 times a day  Clearly, that's 
enough.

--
Joel Esler
Sr. Manager
Open Source, Design, Web, and Education
Talos Group
http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Paul Kosinski]

You are right!  Maybe it only rejects browser-ish headers.


On Tue, 3 Jul 2018 08:12:47 -0700
Dennis Peterson  wrote:

> If you run that curl command I provided it will return only the
> signature serial number.
> 
> dp
> 
> On 7/3/18 6:59 AM, Paul Kosinski wrote:
> > Determining what version a *mirror* has is a bit tricky. Looking at
> > the capture of the entire HTTP session with the new mirrors, they
> > seem to require some header magic to be acceptable:
> >
> >Host: db.us.clamav.net
> >User-Agent: ClamAV/0.99.4 (OS: linux-gnu, ARCH: x86_64, CPU:
> > x86_64)
> >
> > Simply trying to point your (e.g.) browser at a mirror's IP gets
> > you:
> >
> > Error 1003 Ray ID: 4349da2f33f4ae20 • 2018-07-03 13:55:52 UTC
> > Direct IP access not allowed
> >
> >
> > On Tue, 3 Jul 2018 00:11:06 -0700
> > Dennis Peterson  wrote:
> >
> >> Well damn - they say memory is the first thing to go...
> >>
> >> curl -s -r 35-39 http://db.us.clamav.net/daily.cvd |strings
> >>
> >> The -s (silent) inhibits stats.
> >>
> >> dp
> >>
> >> On 7/3/18 12:02 AM, Dennis Peterson wrote:
> >>> I had completely forgotten about freshclam grabbing the entire
> >>> file to determine currency. I recall knocking off a quick script
> >>> to avoid that which included:
> >>>
> >>> curl -q -r 35-39 http://db.us.clamav.net/daily.cvd |strings
> >>>
> >>> It returns the ID of what ever version is on the mirror. I've
> >>> added strings to the end as a safety valve in case someone wants
> >>> to try it with different arguments to the -r.
> >>>
> >>> Being retired I no longer sweat the small schtuff, but when I was
> >>> responsible for hundreds of servers I used every trick in the book
> >>> to avoid wasting time (CFengine was involved and freshclam was
> >>> not). Because the filename daily.xxx is overloaded (version
> >>> agnostic) this kind of trick was needed.
> >>>
> >>> dp

> 
> 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Paul Kosinski]

The way Linux updates are done in practice is significantly different
from ClamAV virus signature updates.  

With ClamAV, freshclam is automatically run periodically, sees (by
some low-cost means) that a new file version is *supposed* to be
available and tries to download it. If either it can't, or worse yet,
it's the wrong one, it tries the next mirror. This all takes time and
bandwidth.

With Linux updates, I explicitly ask (via aptitude) what new updates
are available: It takes some time to retrieve the list. Then I select
the ones I want and ask to install them. I have *never*, *ever* seen
this mechanism deliver the wrong version and thus fail to install it.
This is due to the fact that the same Debian mirror machine provides
the new versions of a group of files as provides the list of new
versions. Thus there is an almost zero chance of a race condition
(unless some idiot adds a version to the list before uploading the
actual deb file). Even if set to auto update, I think the *lists*
always come from the same servers as the files.

It's not a matter of using DNS TXT records, it's a matter of sourcing
them on a *different* computer than the actual files. This separation
virtually begs for synchronization problems.




On Tue, 3 Jul 2018 09:14:31 +0200
Matus UHLAR - fantomas  wrote:

> >> On Mon, 02 Jul 2018 04:02:58 -0700
> >> Al Varnell wrote:
> >>> Does the evidence available infivsyr that it's the mirrors that
> >>> are out-of-date or is it DNS? Everything I've seen shows that
> >>> they are not in sync, but I'm not sure which get's updated first.
> 
> >Am 02.07.2018 um 13:22 schrieb Brian Morrison:
> >> It should not matter if the mirrors are ahead of DNS, they will
> >> simply provide the latest version for download.
> >>
> >> The problem is when a given set of mirrors are not available for a
> >> particular requester, eventually you completely run out of mirrors
> >> and no updates happen at all. There should be fall backs to
> >> prevent this...
> 
> On 02.07.18 13:27, Reindl Harald wrote:
> >it's not rocket science to have a metafile on the mirror which
> >inicates the latest available version,
> 
> because it's much easier, faster and effective  to connect to
> mirror to check a metafile instead of checking single small DNS
> record.
> 
> > linux distributions doing that for decades
> >and they have way larger metadata
> 
> that's exactly because they have way larger metadata. If their
> metadata was as big as Clamav's, they'd use DNS too.
> 
> 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Reindl Harald]

Am 03.07.2018 um 09:14 schrieb Matus UHLAR - fantomas:
>>> On Mon, 02 Jul 2018 04:02:58 -0700
>>> Al Varnell wrote:
 Does the evidence available infivsyr that it's the mirrors that are
 out-of-date or is it DNS? Everything I've seen shows that they are
 not in sync, but I'm not sure which get's updated first.
> 
>> Am 02.07.2018 um 13:22 schrieb Brian Morrison:
>>> It should not matter if the mirrors are ahead of DNS, they will simply
>>> provide the latest version for download.
>>>
>>> The problem is when a given set of mirrors are not available for a
>>> particular requester, eventually you completely run out of mirrors and
>>> no updates happen at all. There should be fall backs to prevent this...
> 
> On 02.07.18 13:27, Reindl Harald wrote:
>> it's not rocket science to have a metafile on the mirror which inicates
>> the latest available version,
> 
> because it's much easier, faster and effective  to connect to
> mirror
> to check a metafile instead of checking single small DNS record.

but it does not help when the dns and the mirror are out-of-sync leading
to the whole topic - obviously it's *not* easy to deal with two distinct
services that way or w ewould not see the problems

>> linux distributions doing that for decades
>> and they have way larger metadata
> 
> that's exactly because they have way larger metadata. If their metadata was
> as big as Clamav's, they'd use DNS too

to introduce the same problems of metadata out-of-sync?

how can it be a f**g problem have a few bytes metadata on the mirror
when obviuosly hundrets of megabytes metadata for linux distributions
are no problem?

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Reindl Harald]

Am 03.07.2018 um 03:37 schrieb Paul Kosinski:
> Any system whereby new versions of files are announced before they are
> actually available to automated downloads is awkward (to say the least).
> 
> If, in addition, a server which doesn't have the announced version is
> blacklisted by the automated downloader, the whole mechanism can grind
> to a halt (as it has for us).
> 
> Even if a server which is out of sync (i.e., behind) is not
> blacklisted, but merely temporarily skipped, it uses extra bandwidth in
> the current scheme. In the case of daily.cvd, the only way freshclam
> detects that the server is out of sync is by downloading the whole file
> (currently about 47 MB) -- the waste of bandwidth is enormous. For
> example, our logs this afternoon show 15 complete downloads of
> daily.cvd over about 1 hour. Of these, all but the last failed due to
> out of sync. This is why we have recently taken to deleting mirrors.dat
> before each freshclam run -- to compensate for the blacklisting -- and
> running freshclam 3 times an hour hoping for sync.
> 
> This behavior is both unreasonable and inefficient
tell that the people who think the DNS nonsense instead a static
"daily.version" text-file gains anything

 Weitergeleitete Nachricht 
Betreff: Re: [clamav-users] We STILL cannot reliably get virus updates
(since new mirrors)
Datum: Mon, 2 Jul 2018 19:10:40 +0100
Von: Brian Morrison 
Antwort an: ClamAV users ML 
Organisation: The Fool and Bladder Face-Jumping Team
An: ClamAV users ML 

On Mon, 2 Jul 2018 19:50:55 +0200
Reindl Harald wrote:

> > For me freshclam runs roughly every 2 hours, so I think that the
> > load is an order of magnitude higher than you state. I will confess
> > that I don't know about the capability of web servers in this
> > regard, but the point that d.net made was that the DNS server would
> > always be more capable in this regard than a web server
> come on - our main-server running ina virtual machine spits out 3
> requests/sec. on our core-cms in case of cache-hits and even on a 7
> years old workstation far above 1/sec and that is *not* static
> content with a few bytes

How many requests/sec can a DNS server process?

Given that the clamav mirrors seem to be struggling (new system, I know)
I still think that anything that reduces the load they are serving ought
to be a good idea. Not my day job though...
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Reindl Harald]

Am 02.07.2018 um 20:10 schrieb Brian Morrison:
> On Mon, 2 Jul 2018 19:50:55 +0200
> Reindl Harald wrote:
> 
>>> For me freshclam runs roughly every 2 hours, so I think that the
>>> load is an order of magnitude higher than you state. I will confess
>>> that I don't know about the capability of web servers in this
>>> regard, but the point that d.net made was that the DNS server would
>>> always be more capable in this regard than a web server  
>> come on - our main-server running ina virtual machine spits out 3
>> requests/sec. on our core-cms in case of cache-hits and even on a 7
>> years old workstation far above 1/sec and that is *not* static
>> content with a few bytes
> 
> How many requests/sec can a DNS server process?
> 
> Given that the clamav mirrors seem to be struggling (new system, I know)
> I still think that anything that reduces the load they are serving ought
> to be a good idea. Not my day job though...

you gain nothing when it don't work relieable and when you have to serve
the large update-files the single textfile don't matter - especially in
case of a CDN
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Reindl Harald]

Am 02.07.2018 um 19:45 schrieb Brian Morrison:
> On Mon, 2 Jul 2018 19:17:32 +0200
> Reindl Harald wrote:
> 
>> Am 02.07.2018 um 19:07 schrieb Brian Morrison:
>>> On Mon, 2 Jul 2018 10:26:34 +0200
>>> Reindl Harald wrote:
>>>   
 Am 02.07.2018 um 08:44 schrieb Bill Maidment:  
> Maybe these are dumb questions; if so, please ignore.
> But doesn't it make more sense to update all the mirrors first,
> before changing the DNS? Is there some mechanism to do it that way
> round?

 i wonder why all the linux distributions with update mirrors don't
 need that DNS theatre to start with  
>>>
>>> Because the rate of updates is much less frequent, the more often
>>> you need to check the higher the mirror load becomes. Much of this
>>> load is telling people that there is no newer version...  
>>
>> says who?
> 
> I am basing my comments on the distributed.net experience during the
> mid to late 90s. At the time they used one of the first DNS-based TXT
> record update mechanisms, it's broadly similar to how all of these work.
> At the time this made a very big difference to the load their mirrors
> were dealing with.
> 
>>
>> on a typical setup freshclam is running once or twice *daily* while a
>> webserver these days can spit out the same small static txt file many
>> thousands of times per seond with zero load
>>
> 
> For me freshclam runs roughly every 2 hours, so I think that the load
> is an order of magnitude higher than you state. I will confess that I
> don't know about the capability of web servers in this regard, but the
> point that d.net made was that the DNS server would always be more
> capable in this regard than a web server
come on - our main-server running ina virtual machine spits out 3
requests/sec. on our core-cms in case of cache-hits and even on a 7
years old workstation far above 1/sec and that is *not* static
content with a few bytes
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] update report

[Reindl Harald]

Am 02.07.2018 um 19:38 schrieb Benny Pedersen:
> Gene Heskett skrev den 2018-07-02 19:20:
>> On Monday 02 July 2018 13:12:12 Gene Heskett wrote:
>> However, a network restart did not get rid of the ipv6 stuff in the
>> ifconfig lo report. ?  /etc/network/interfaces is also clean of any
>> ipv6 stuffs. ?
>>
> 
> if all else fails
> 
> check /etc/gai.conf
> change that conf to prefer ipv4 first

yeah distro specific crap besides that it does not help in case of
freshclam at all to disable ivp6 entirely

[root@srv-rhsoft:~]$ cat /etc/gai.conf
cat: /etc/gai.conf: No such file or directory

[root@srv-rhsoft:~]$ cat sysctl.conf | grep ipv6 | grep disable
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

[root@srv-rhsoft:~]$ ifconfig lo
lo: flags=73  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
loop  txqueuelen 1000  (Local Loopback)
RX packets 3019819  bytes 2513735808 (2.3 GiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 3019819  bytes 2513735808 (2.3 GiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] update report

[Reindl Harald]

Am 02.07.2018 um 19:20 schrieb Gene Heskett:
>> And since that stuff did exist in my /etc/hosts file, I just stuck a #
>> in front of all those, just for S&G of course. Watching log too. But
>> its seems like an every other update run, and since I am not a
>> paying/supporting customer, I only run it 2x daily. So the next run
>> will be just about 0:50 local time.
> 
> However, a network restart did not get rid of the ipv6 stuff in the 
> ifconfig lo report. ?  /etc/network/interfaces is also clean of any 
> ipv6 stuffs. ?

sysctl.conf:
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

sysctl -p

ifconfig no longer shows any ipv6 stuff
it's that easy

but even on my setups which have this for years freshclam repetaly
produces ipv6 crap-messages which is simpyl wrong
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Reindl Harald]

Am 02.07.2018 um 19:07 schrieb Brian Morrison:
> On Mon, 2 Jul 2018 10:26:34 +0200
> Reindl Harald wrote:
> 
>> Am 02.07.2018 um 08:44 schrieb Bill Maidment:
>>> Maybe these are dumb questions; if so, please ignore.
>>> But doesn't it make more sense to update all the mirrors first,
>>> before changing the DNS? Is there some mechanism to do it that way
>>> round?  
>>
>> i wonder why all the linux distributions with update mirrors don't
>> need that DNS theatre to start with
> 
> Because the rate of updates is much less frequent, the more often you
> need to check the higher the mirror load becomes. Much of this load is
> telling people that there is no newer version...

says who?

on a typical setup freshclam is running once or twice *daily* while a
webserver these days can spit out the same small static txt file many
thousands of times per seond with zero load
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] freshclam works for me

[Noel Jones]

I just wanted to chime in and say that freshclam continues to work
fine for me.

I have great sympathy for those having trouble, but I strongly
suspect they are the vocal minority.  I'd complain too if it seemed
unreliable, but it works fine here.

Before any changes are made to freshclam or the procedure to check
for updates, it's important to understand why some sites are
failing, so the right problem can be fixed.

This is a ipv4 site, and I occasionally get ipv6 error messages --
maybe 4 a week.  They don't seem to cause any particular problem.  A
freshclam.config option to disable ipv6 would fix that.  Or maybe a
"protocol {ipv4|ipv6|any}" option.

There are 6 servers here, running various versions of FreeBSD with
clam 0.100.0.   All are set to run freshclam as a daemon (not from
cron) with "checks 15" and "DatabaseMirror db.us.clamav.net".  I
don't mess with freshclam except to check the logs once in a while
for errors, which are rare.

These servers are at various sites with various internet providers,
but all in US/Tennessee.  Maybe my geographic region just happens to
point to a good mirror.

Using Cloudflare changes the dynamics of updates.  I wonder if it
might be better if everyone pointed to db.clamav.net and all the
direct mirrors are dropped.  Let Cloudflare decide what is the
closest POP, that's kinda their job.

Seems like the DNS record is still needed to announce what update is
supposed to be available.

Anyway, thanks for continuing to look at ways to improve this, and
thanks for listening.



  -- Noel Jones
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Joel Esler (jesler)]

On Jul 2, 2018, at 2:10 PM, Brian Morrison 
mailto:b...@fenrir.org.uk>> wrote:

On Mon, 2 Jul 2018 19:50:55 +0200
Reindl Harald wrote:

For me freshclam runs roughly every 2 hours, so I think that the
load is an order of magnitude higher than you state. I will confess
that I don't know about the capability of web servers in this
regard, but the point that d.net made was that the DNS server 
would
always be more capable in this regard than a web server
come on - our main-server running ina virtual machine spits out 3
requests/sec. on our core-cms in case of cache-hits and even on a 7
years old workstation far above 1/sec and that is *not* static
content with a few bytes

How many requests/sec can a DNS server process?

Given that the clamav mirrors seem to be struggling (new system, I know)
I still think that anything that reduces the load they are serving ought
to be a good idea. Not my day job though...

I made an adjustment yesterday.  Are people still seeing this error?

--
Joel Esler
Sr. Manager
Open Source, Design, Web, and Education
Talos Group
http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Joel Esler (jesler)]

On Jul 3, 2018, at 10:37 AM, Benoit Panizzon 
mailto:benoit.paniz...@imp.ch>> wrote:

Sorry I was not following that discussion...

 Host: db.us.clamav.net
 User-Agent: ClamAV/0.99.4 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

  Error 1003 Ray ID: 4349da2f33f4ae20 • 2018-07-03 13:55:52 UTC
  Direct IP access not allowed

But this cought my attention...

db.us.clamav.net is an alias for 
db.us.clamav.net.cdn.cloudflare.net.

Cloudflare uses some kind of DDOS protection to detect if the visitor
might be a 'malicious bot' or a 'human' with a 'propper' webbrowser.

I fear, FreshClam does not pass the 'human' test.

I would suggest to the ClamAV team to move away from Cloudflare. Such
issues are bound to occur with CloudFlare.

That feature is turned off for the mirror network.  So, no, those issues will 
not occur.  In fact, it was on, and yes, it was causing problems, which is why 
it's now off.

However, the ~60TB of traffic we are passing on a 24 hour basis tells me that 
freshclam is working fine. You can't hit the cloudflare IPs directly, which is 
what that error says.

--
Joel Esler
Sr. Manager
Open Source, Design, Web, and Education
Talos Group
http://www.talosintelligence.com
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Benoit Panizzon]

Hi List

Sorry I was not following that discussion...

>   Host: db.us.clamav.net
>   User-Agent: ClamAV/0.99.4 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
> 
>Error 1003 Ray ID: 4349da2f33f4ae20 • 2018-07-03 13:55:52 UTC
>Direct IP access not allowed

But this cought my attention...

db.us.clamav.net is an alias for db.us.clamav.net.cdn.cloudflare.net.

Cloudflare uses some kind of DDOS protection to detect if the visitor
might be a 'malicious bot' or a 'human' with a 'propper' webbrowser.

I fear, FreshClam does not pass the 'human' test.

I would suggest to the ClamAV team to move away from Cloudflare. Such
issues are bound to occur with CloudFlare.

Mit freundlichen Grüssen

-Benoît Panizzon-
-- 
I m p r o W a r e   A G-Leiter Commerce Kunden
__

Zurlindenstrasse 29 Tel  +41 61 826 93 00
CH-4133 PrattelnFax  +41 61 826 93 01
Schweiz Web  http://www.imp.ch
__
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Paul Kosinski]

Determining what version a *mirror* has is a bit tricky. Looking at the
capture of the entire HTTP session with the new mirrors, they seem to
require some header magic to be acceptable:

  Host: db.us.clamav.net
  User-Agent: ClamAV/0.99.4 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

Simply trying to point your (e.g.) browser at a mirror's IP gets you:

   Error 1003 Ray ID: 4349da2f33f4ae20 • 2018-07-03 13:55:52 UTC
   Direct IP access not allowed


On Tue, 3 Jul 2018 00:11:06 -0700
Dennis Peterson  wrote:

> Well damn - they say memory is the first thing to go...
> 
> curl -s -r 35-39 http://db.us.clamav.net/daily.cvd |strings
> 
> The -s (silent) inhibits stats.
> 
> dp
> 
> On 7/3/18 12:02 AM, Dennis Peterson wrote:
> > I had completely forgotten about freshclam grabbing the entire file
> > to determine currency. I recall knocking off a quick script to
> > avoid that which included:
> >
> > curl -q -r 35-39 http://db.us.clamav.net/daily.cvd |strings
> >
> > It returns the ID of what ever version is on the mirror. I've added
> > strings to the end as a safety valve in case someone wants to try
> > it with different arguments to the -r.
> >
> > Being retired I no longer sweat the small schtuff, but when I was
> > responsible for hundreds of servers I used every trick in the book
> > to avoid wasting time (CFengine was involved and freshclam was
> > not). Because the filename daily.xxx is overloaded (version
> > agnostic) this kind of trick was needed.
> >
> > dp

___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] lost the thread, but my ipv6 noise in the freshclam log has vanished

[Gene Heskett]

On Tuesday 03 July 2018 06:59:59 Joel Esler (jesler) wrote:

> What does that mean?
>
The failure to access the outside world via ipv6 report has now vanished 
from the logs.

Here is a paste from a couple days back:
nonblock_connect: connect(): fd=4 errno=101: Network is unreachable
Can't connect to port 80 of host db.us.clamav.net (IP: 
2400:cb00:2048:1::6810:bb8a)

So now I see only the real failures but there haven't been any since.
Normal log looks like this:

Tue Jul  3 06:50:51 2018 -> Received signal: wake up
Tue Jul  3 06:50:51 2018 -> ClamAV update process started at Tue Jul  3 
06:50:51 2018
Tue Jul  3 06:50:51 2018 -> WARNING: Your ClamAV installation is OUTDATED!
Tue Jul  3 06:50:51 2018 -> WARNING: Local version: 0.99.4 Recommended version: 
0.100.0
Tue Jul  3 06:50:51 2018 -> DON'T PANIC! Read 
http://www.clamav.net/documents/upgrading-clamav
Tue Jul  3 06:50:51 2018 -> main.cld is up to date (version: 58, sigs: 4566249, 
f-level: 60, builder: sigmgr)
Tue Jul  3 06:50:52 2018 -> Downloading daily-24718.cdiff [100%]
Tue Jul  3 06:50:59 2018 -> daily.cld updated (version: 24718, sigs: 2002049, 
f-level: 63, builder: neo)
Tue Jul  3 06:50:59 2018 -> bytecode.cld is up to date (version: 322, sigs: 90, 
f-level: 63, builder: neo)
Tue Jul  3 06:51:05 2018 -> Database updated (6568388 signatures) from 
db.us.clamav.net (IP: 104.16.187.138)
Tue Jul  3 06:51:05 2018 -> Clamd successfully notified about the update.

Thanks Joel.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] lost the thread, but my ipv6 noise in the freshclam log has vanished

[Joel Esler (jesler)]

What does that mean?

Sent from my iPhone

> On Jul 3, 2018, at 06:39, Gene Heskett  wrote:
> 
> 
> -- 
> Cheers, Gene Heskett
> --
> "There are four boxes to be used in defense of liberty:
> soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author)
> Genes Web page 
> ___
> clamav-users mailing list
> clamav-users@lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] lost the thread, but my ipv6 noise in the freshclam log has vanished

[Gene Heskett]

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Matus UHLAR - fantomas]

On Mon, 02 Jul 2018 04:02:58 -0700
Al Varnell wrote:

Does the evidence available infivsyr that it's the mirrors that are
out-of-date or is it DNS? Everything I've seen shows that they are
not in sync, but I'm not sure which get's updated first.



Am 02.07.2018 um 13:22 schrieb Brian Morrison:

It should not matter if the mirrors are ahead of DNS, they will simply
provide the latest version for download.

The problem is when a given set of mirrors are not available for a
particular requester, eventually you completely run out of mirrors and
no updates happen at all. There should be fall backs to prevent this...


On 02.07.18 13:27, Reindl Harald wrote:

it's not rocket science to have a metafile on the mirror which inicates
the latest available version,


because it's much easier, faster and effective  to connect to mirror
to check a metafile instead of checking single small DNS record.


linux distributions doing that for decades
and they have way larger metadata


that's exactly because they have way larger metadata. If their metadata was
as big as Clamav's, they'd use DNS too.


--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Enter any 12-digit prime number to continue.
___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Dennis Peterson]

Well damn - they say memory is the first thing to go...

curl -s -r 35-39 http://db.us.clamav.net/daily.cvd |strings

The -s (silent) inhibits stats.

dp

On 7/3/18 12:02 AM, Dennis Peterson wrote:
I had completely forgotten about freshclam grabbing the entire file to 
determine currency. I recall knocking off a quick script to avoid that which 
included:


curl -q -r 35-39 http://db.us.clamav.net/daily.cvd |strings

It returns the ID of what ever version is on the mirror. I've added strings to 
the end as a safety valve in case someone wants to try it with different 
arguments to the -r.


Being retired I no longer sweat the small schtuff, but when I was responsible 
for hundreds of servers I used every trick in the book to avoid wasting time 
(CFengine was involved and freshclam was not). Because the filename daily.xxx 
is overloaded (version agnostic) this kind of trick was needed.


dp

On 7/2/18 6:37 PM, Paul Kosinski wrote:

Any system whereby new versions of files are announced before they are
actually available to automated downloads is awkward (to say the least).

If, in addition, a server which doesn't have the announced version is
blacklisted by the automated downloader, the whole mechanism can grind
to a halt (as it has for us).

Even if a server which is out of sync (i.e., behind) is not
blacklisted, but merely temporarily skipped, it uses extra bandwidth in
the current scheme. In the case of daily.cvd, the only way freshclam
detects that the server is out of sync is by downloading the whole file
(currently about 47 MB) -- the waste of bandwidth is enormous. For
example, our logs this afternoon show 15 complete downloads of
daily.cvd over about 1 hour. Of these, all but the last failed due to
out of sync. This is why we have recently taken to deleting mirrors.dat
before each freshclam run -- to compensate for the blacklisting -- and
running freshclam 3 times an hour hoping for sync.

This behavior is both unreasonable and inefficient.

P.S. Just before I sent this mail, I sent some proposals for how ClamAV
might possibly avoid this behavior.



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] We STILL cannot reliably get virus updates (since new mirrors)

[Dennis Peterson]

I had completely forgotten about freshclam grabbing the entire file to determine 
currency. I recall knocking off a quick script to avoid that which included:


curl -q -r 35-39 http://db.us.clamav.net/daily.cvd |strings

It returns the ID of what ever version is on the mirror. I've added strings to 
the end as a safety valve in case someone wants to try it with different 
arguments to the -r.


Being retired I no longer sweat the small schtuff, but when I was responsible 
for hundreds of servers I used every trick in the book to avoid wasting time 
(CFengine was involved and freshclam was not). Because the filename daily.xxx is 
overloaded (version agnostic) this kind of trick was needed.


dp

On 7/2/18 6:37 PM, Paul Kosinski wrote:

Any system whereby new versions of files are announced before they are
actually available to automated downloads is awkward (to say the least).

If, in addition, a server which doesn't have the announced version is
blacklisted by the automated downloader, the whole mechanism can grind
to a halt (as it has for us).

Even if a server which is out of sync (i.e., behind) is not
blacklisted, but merely temporarily skipped, it uses extra bandwidth in
the current scheme. In the case of daily.cvd, the only way freshclam
detects that the server is out of sync is by downloading the whole file
(currently about 47 MB) -- the waste of bandwidth is enormous. For
example, our logs this afternoon show 15 complete downloads of
daily.cvd over about 1 hour. Of these, all but the last failed due to
out of sync. This is why we have recently taken to deleting mirrors.dat
before each freshclam run -- to compensate for the blacklisting -- and
running freshclam 3 times an hour hoping for sync.

This behavior is both unreasonable and inefficient.

P.S. Just before I sent this mail, I sent some proposals for how ClamAV
might possibly avoid this behavior.



___
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml