Yes, file synchronization is difficult. But we *started out* using the provided (i.e., standard) freshclam tool to update our daily.cvd (etc.). I only built our current non-standard tool (reading the file header) when the Cloudflare mirrors started serving out-of-date file versions which caused freshclam to fail and blacklist the mirror (which eventually resulted in all mirrors being blacklisted).
This says to me that the old, "standard", DNS TXT approach built in to freshclam doesn't play well with Cloudflare (or similar mirrors?). On Sat, 20 Oct 2018 06:57:55 -0700 Dennis Peterson <denni...@inetnw.com> wrote: > Caching file systems do validate the requested file against a master > file to see if there has been a change. De-dupe caches do the same. > It isn't instantaneous but they also don't have to wait for the cache > to refresh as they can deliver a pass through request at the same > time they're updating the cache. This is more expensive than > scheduled sync methods, but those necessarily have a delay. These > systems should reject requests for files they don't have but that is > difficult if the updated file has the same name as the one it > replaces. I know it was always a big deal for the dot com I worked > for to update Akamai because of sync problems around the world. > Atomic synchronized file updates are pretty much impossible when you > have a million page requests/minute. > > I agree with Joel about using non-standard tools to request > signatures and people that do so should have no expectation of > consistent high reliability, and support requests should go in the > bit bucket. The risk associated with self-service falls on the > operator, not the vendor. > > dp > > On 10/19/18 2:19 PM, Paul Kosinski wrote: > > I'm glad modern multi-core / multi-thread CPU's don't operate this > > way. > > > > Imagine if, when your code on CPU1 tried to access memory location > > M, your code got what CPU1 happened to have in its cache, instead > > of what CPU2 stored into M a few microseconds ago. Fortunately, > > with real CPUs, CPU2 invalidates the other CPUs' caches, and CPU1 > > takes the extra time to fetch the new and correct data from memory. > > > > Thus, what Cloudflare *should* have (if you can't explicitly upload > > a file), is a mechanism to tell it that a file is out of date. This > > mechanism could operate very quickly. Then, what Cloudflare would > > do is either to stall the HTTP response -- I doubt it would have to > > stall for long -- or reply with the appropriate HTTP status code > > warning the requester that something is amiss. (Codes 503, 504 or > > 409 might be applicable.) > > > > > > On Thu, 18 Oct 2018 22:34:03 +0000 > > "Joel Esler (jesler)" <jes...@cisco.com> wrote: > > > >> Cloudflare will grab the file from our infrastructure once it's > >> been requested. (Otherwise it wouldn't know it was there, we > >> can't push into Cloudflare.). But we have discussed a few ideas > >> internally that I think will fix this, let us try a couple things > >> and see if it cuts down on this. > >> > >> On Oct 18, 2018, at 1:55 PM, Eric Tykwinski > >> <eric-l...@truenet.com<mailto:eric-l...@truenet.com>> wrote: > >> > >> As far as I know you don't upload to cloudflare, it's more of how > >> often does cloudflare check to see if the files have changed. > >> So you setup a TTL on the check frequency on the cloudflare > >> website. > >> > >> Since updates are new they should just be pulled when you ask from > >> the main clam server. > >> So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's > >> main server for that file and cache it. > >> > >> So my guess would be same as the TTL on the DNS check: > >> current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800 > >> IN TXT "0.100.2:58:25048:1539883740:1:63:48006:327" > >> I.E. 30 minutes for older files, and new ones are when they come > >> in. > >> > >> Sound about right Joel, Micah? > >> > >> Sincerely, > >> > >> Eric Tykwinski > >> TrueNet, Inc. > >> P: 610-429-8300 > >> > >> -----Original Message----- > >> From: clamav-users [mailto:clamav-users-boun...@lists.clamav.net] > >> On Behalf Of Paul Kosinski > >> Sent: Thursday, October 18, 2018 1:23 PM > >> To: > >> clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net> > >> Subject: Re: [clamav-users] Latest report on update "delays" > >> > >> How can it take 10, 20 30 or more minutes (and I've seen well over > >> an hour at times) to upload the ClamAV database to Cloudflare? > >> Does it have to be uploaded separately (and maybe sequentially) > >> from Cisco to each Cloudflare mirror? Or is Cloudflare's automatic > >> propagation slow? > >> > >> > >> On Thu, 18 Oct 2018 16:07:38 +0000 > >> "Micah Snyder (micasnyd)" > >> <micas...@cisco.com<mailto:micas...@cisco.com>> wrote: > >> > >> Hi Paul, > >> > >> I realize it may look misleading to state that you're up to date > >> when a newer database has been announced. However, if the newer > >> database is still being uploaded to the CDN, it is more accurate > >> to say that the DNS announcement is premature. > >> > >> The change to freshclam is an effort to ignore potentially > >> premature database version numbers listed via DNS. > >> > >> Micah Snyder > >> ClamAV Development > >> Talos > >> Cisco Systems, Inc. _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
