Hi Micah, I appreciate your response. It has been driving me nuts since it started about a week ago. Things had been humming along nicely for over a year until ~22 Feb.
So, as best I can tell at this point, the mirror does not have a bytecode.cvd to serve up (0 length or otherwise). Here is a listing of /var/lib/clamav on the mirror. koconnor@ampion-clamav-mirror:~$ ls -l /var/lib/clamav total 226196 -rw-r--r-- 1 clamav clamav 314802 Feb 22 22:02 bytecode.cld -rw-r--r-- 1 clamav clamav 60814501 Mar 1 09:07 daily.cld -rw-r--r-- 1 clamav clamav 69 Jan 29 2022 freshclam.dat -rw-r--r-- 1 clamav clamav 170479789 Jan 29 2022 main.cvd -rw-r--r-- 1 clamav clamav 87 Jan 29 2022 test.html This is the version of freshclam on the mirror: koconnor@ampion-clamav-mirror:~$ freshclam -V ClamAV 0.103.8/26827/Wed Mar 1 08:28:49 2023 And the freshclam.conf on the mirror too. koconnor@ampion-clamav-mirror:~$ cat /etc/clamav/freshclam.conf # Automatically created by the clamav-freshclam postinst # Comments will get lost when you reconfigure the clamav-freshclam package DatabaseOwner clamav UpdateLogFile /var/log/clamav/freshclam.log LogVerbose false LogSyslog false LogFacility LOG_LOCAL6 LogFileMaxSize 0 LogRotate true LogTime true Foreground false Debug false MaxAttempts 5 DatabaseDirectory /var/lib/clamav DNSDatabaseInfo current.cvd.clamav.net ConnectTimeout 30 ReceiveTimeout 0 TestDatabases yes ScriptedUpdates yes CompressLocalDatabase yes Bytecode true NotifyClamd /etc/clamav/clamd.conf # Check for new database 24 times a day Checks 24 DatabaseMirror db.local.clamav.net DatabaseMirror database.clamav.net I did find something interesting in the logs on the mirror server. First a listing of the log directory: koconnor@ampion-clamav-mirror:~$ ls -l /var/log/clamav/* -rw-r----- 1 clamav clamav 57381 Mar 1 13:07 /var/log/clamav/freshclam.log -rw-r----- 1 clamav adm 142086 Feb 26 00:00 /var/log/clamav/freshclam.log.1 -rw-r----- 1 clamav clamav 5142 Dec 25 00:00 /var/log/clamav/freshclam.log.10.gz -rw-r----- 1 clamav adm 5002 Dec 18 00:00 /var/log/clamav/freshclam.log.11.gz -rw-r----- 1 clamav adm 5008 Dec 11 00:00 /var/log/clamav/freshclam.log.12.gz -rw-r----- 1 clamav adm 6158 Feb 19 00:00 /var/log/clamav/freshclam.log.2.gz -rw-r----- 1 clamav adm 4997 Feb 12 00:00 /var/log/clamav/freshclam.log.3.gz -rw-r----- 1 clamav clamav 5148 Feb 5 00:00 /var/log/clamav/freshclam.log.4.gz -rw-r----- 1 clamav adm 5023 Jan 29 00:00 /var/log/clamav/freshclam.log.5.gz -rw-r----- 1 clamav adm 5008 Jan 22 00:00 /var/log/clamav/freshclam.log.6.gz -rw-r----- 1 clamav adm 4990 Jan 15 00:00 /var/log/clamav/freshclam.log.7.gz -rw-r----- 1 clamav adm 5009 Jan 8 00:00 /var/log/clamav/freshclam.log.8.gz -rw-r----- 1 clamav clamav 5174 Jan 1 00:00 /var/log/clamav/freshclam.log.9.gz Then a search for bytecode.cvd in the most recent log file: koconnor@ampion-clamav-mirror:~$ sudo grep bytecode.cvd /var/log/clamav/freshclam.log koconnor@ampion-clamav-mirror:~$ Followed by a search for that string in the next most recent file: koconnor@ampion-clamav-mirror:~$ sudo grep bytecode.cvd /var/log/clamav/freshclam.log.1 Sun Feb 19 00:00:35 2023 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2) Sun Feb 19 01:00:35 2023 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2) Sun Feb 19 02:00:35 2023 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2) Sun Feb 19 03:00:35 2023 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2) <snip> this is repeated every hour <snip> Wed Feb 22 18:02:30 2023 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2) Wed Feb 22 19:02:31 2023 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2) Wed Feb 22 20:02:31 2023 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2) Wed Feb 22 21:02:31 2023 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2) koconnor@ampion-clamav-mirror:~$ This is particularly interesting as the end of that output is approximately the time when the problem started. Let me know if I should send you a copy of any of the log files from the mirror. I wasn't sure if that was appropriate for the listserv. Thanks again Kevin On Tue, Feb 28, 2023 at 1:31 PM Micah Snyder (micasnyd) <micas...@cisco.com> wrote: > The bytecode.cvd file is the original. > When there is an update, we publish two things: > > 1. a bytecode.cdiff patch file that will update the older bytecode.cvd > to the newest version. This is the "scripted update" mechanism. > > If using the .cdiff patch file to update, it should replace the old > bytecode.cvd with a new bytecode.cld. We may issue an empty patch > file (zero-bytes) to tell freshclam to download the whole bytecode.cvd > instead. We do this if the patch is so big it is better to just download > the whole file, or if is a bug preventing the patch file from working > correctly, which there presently is for bytecode signatures (sad!). > > This .cdiff update mechanism would not be used in your situation > because ScriptedUpdates is disabled. > > 2. a new bytecode.cvd. > > This should only be downloaded in two cases: A) If you do not have the > old bytecode.cvd (or cld) and thus cannot use the patch file to > update. And B) If the bytecode.cdiff patch file is empty. > > The issue you're facing feels to me like an issue with what the private > mirror is serving. Can you please check if it is serving an empty > bytecode.cvd? It feels like it may be serving both the empty > bytecode.cvd and a bytecode.cld. > > If that's not the case, then we may have a bug in freshclam and I would > love some more information on what freshclam is downloading when it runs > in order to get into this strange state. > > Best, > Micah > > Micah Snyder > ClamAV Development > Talos > Cisco Systems, Inc. > > ------------------------------ > *From:* clamav-users <clamav-users-boun...@lists.clamav.net> on behalf of > Kevin O'Connor via clamav-users <clamav-users@lists.clamav.net> > *Sent:* Monday, February 27, 2023 11:12 AM > *To:* ClamAV users ML <clamav-users@lists.clamav.net> > *Cc:* Kevin O'Connor <kocon...@ampion.net> > *Subject:* Re: [clamav-users] 0 length bytecode.cvd causing problems with > clamav daemon > > Marc, > > I had a similar understanding of that document. That is; if there is no > bytecode.cvd pushed by the ClamAV team, it should not exist on my local > scanners. When I checked the mirror and there was no bytecode.cvd file, yet > it appeared on my scanner machines with 0 length, I figured that the new > release had highlighted a misconfiguration in my freshclam.conf that the > earlier version was more forgiving of. However I have not found what that > might be. > > Your idea of removing all the files in the /var/lib/clamav directory is > what I found worked initially, but that seems like a poor workaround as I > need this running all the time. I don't know when our clients will drop > files on us that need a scan. > > Thanks for looking at it. > > Kevin > > On Mon, Feb 27, 2023 at 1:11 PM Marc via clamav-users < > clamav-users@lists.clamav.net> wrote: > > i would suggest, to delete alle libraries in /var/lib/clamav and download > all complete new. > CLD Files comes not regularly, normally we have CVD only. > > If i understand this well, CLD Files comes only when error occures while > updating. > https://blog.clamav.net/2021/03/clamav-cvds-cdiffs-and-magic-behind.html > <https://blog.clamav.net/2021/03/clamav-cvds-cdiffs-and-magic-behind.html> > > > Von / From: Kevin O'connor <mailto:kocon...@ampion.net> > An / To: Newcomer01 <mailto:newcome...@posteo.de> > Gesendet / Sent: Montag, Februar 27, 2023 um 18:38 (at 06:38 PM) +0100 > Betreff / Subject: Re: [clamav-users] 0 length bytecode.cvd causing > problems with clamav daemon > > Heh, good question. Just checked again, and it looks like that was a > copy-paste error. There is only one PrivateMirror line. > > Kevin > > > > On Mon, Feb 27, 2023 at 12:02 PM newcomer01 via clamav-users < > clamav-users@lists.clamav.net> wrote: > > > > why you have set two times the "PrivateMirror" with identically IP's? > > Can't believe that this happens with the automated PostInst 😉 > > > > > > Von / From: Clamav User Mailinglist <mailto: > clamav-users@lists.clamav.net> > > An / To: Newcomer01 <mailto:newcome...@posteo.de> > > CC / CC: Kevin O'connor <mailto:kocon...@ampion.net> > > Gesendet / Sent: Montag, Februar 27, 2023 um 16:58 (at 04:58 PM) +0100 > > Betreff / Subject: [clamav-users] 0 length bytecode.cvd causing problems > with clamav daemon > > > I am having an issue with 0 length bytecode.cvd files on my scanner > instances. This seems to have started sometime on 22 Feb, I'm afraid I > don't have an exact time. The clamav daemon produces logs like the > following: > > > > > > Feb 27 14:39:11 av-scan-wrhn clamd[163614]: LibClamAV Error: > cli_cvdverify: Can't read CVD header > > > Feb 27 14:39:11 av-scan-wrhn clamd[163614]: LibClamAV Error: Can't > load /var/lib/clamav/bytecode.cld: Broken or not a CVD file > > > Feb 27 14:39:11 av-scan-wrhn clamd[163614]: LibClamAV Error: > cli_loaddbdir(): error loading database /var/lib/clamav/bytecode.cld > > > Feb 27 14:39:11 av-scan-wrhn clamd[163614]: Mon Feb 27 14:39:11 2023 > -> !Broken or not a CVD file > > > Feb 27 14:39:11 av-scan-wrhn systemd[1]: clamav-daemon.service: Main > process exited, code=exited, status=1/FAILURE > > > Feb 27 14:39:11 av-scan-wrhn systemd[1]: clamav-daemon.service: Failed > with result 'exit-code'. > > > Feb 27 14:39:11 av-scan-wrhn systemd[1]: clamav-daemon.service: > Consumed 8.679s CPU time. > > > > > > > > > I feel like I have narrowed the problem down to a 0 length > 'bytecode.cvd' file. Here is a listing of the definitions directory: > > > > > > $ ls -l /var/lib/clamav > > > total 226168 > > > -rw-r--r-- 1 clamav clamav 314802 Feb 27 14:06 bytecode.cld > > > -rw-r--r-- 1 clamav clamav 0 Feb 27 02:00 bytecode.cvd > > > -rw-r--r-- 1 clamav clamav 60787973 Feb 27 10:01 daily.cld > > > -rw-r--r-- 1 clamav clamav 69 Feb 23 15:33 freshclam.dat > > > -rw-r--r-- 1 clamav clamav 170479789 Feb 27 02:00 main.cvd > > > > > > > > > My initial fix (before narrowing the problem down to bytecode.cvd) was > to > > > > > > 1. stop freshclam > > > 2. clean this directory > > > 3. restart freshclam > > > 4. give it time to get the definitions (from a private mirror) > > > 5. start clamav daemon > > > > > > This would work for maybe 1/2 day then the empty bytecode.cvd file > would reappear and the daemon would fail. > > > > > > This morning I was able to spend some more time and find that it was > just the one file that needed to be removed. > > > > > > I have a local mirror because there are several instances of this > scanner in use (at least 2 instances for several environments). I have > checked the mirror and it appears to be working fine and keeping the > definitions up to date inside our environment. In addition, the scanner > instances appear to be keeping the local set of definitions up to date with > the mirror. > > > > > > The mirror does not have a bytecode.cvd file on it (here is a listing > of its definitions directory) > > > > > > $ ls -l /var/lib/clamav > > > total 226172 > > > -rw-r--r-- 1 clamav clamav 314802 Feb 22 22:02 bytecode.cld > > > -rw-r--r-- 1 clamav clamav 60787973 Feb 27 09:06 daily.cld > > > -rw-r--r-- 1 clamav clamav 69 Jan 29 2022 freshclam.dat > > > -rw-r--r-- 1 clamav clamav 170479789 Jan 29 2022 main.cvd > > > -rw-r--r-- 1 clamav clamav 87 Jan 29 2022 test.html > > > > > > > > > To the best of my knowledge, the software is up to date: > > > > > > $ sudo freshclam -V > > > ClamAV 0.103.8/26825/Mon Feb 27 08:24:38 2023 > > > > > > > > > Here is the freshclam.conf used on all the local sanner instances > > > > > > $ cat /etc/clamav/freshclam.conf > > > # Automatically created by the clamav-freshclam postinst > > > # Comments will get lost when you reconfigure the clamav-freshclam > package > > > > > > DatabaseOwner clamav > > > UpdateLogFile /var/log/clamav/freshclam.log > > > LogVerbose false > > > LogSyslog false > > > LogFacility LOG_LOCAL6 > > > LogFileMaxSize 0 > > > LogRotate true > > > LogTime true > > > Foreground false > > > Debug false > > > MaxAttempts 5 > > > DatabaseDirectory /var/lib/clamav > > > DNSDatabaseInfo current.cvd.clamav.net > <http://current.cvd.clamav.net> > <http://current.cvd.clamav.net > <http://current.cvd.clamav.net>> > <http://current.cvd.clamav.net > <http://current.cvd.clamav.net> > > > > > ConnectTimeout 30 > > > ReceiveTimeout 0 > > > TestDatabases yes > > > CompressLocalDatabase no > > > Bytecode true > > > NotifyClamd /etc/clamav/clamd.conf > > > # Check for new database 24 times a day > > > Checks 24 > > > PrivateMirror http://10.50.0.2 > <http://10.50.0.2> > > > ScriptedUpdates no > > > PrivateMirror http://10.50.0.2 > <http://10.50.0.2> > > > > > > > > > The scanner has been working fine for about 12 months, keeping the > software and the definitions up to date. The only configuration item that > seems to relate is "Bytecode true", but the description seems to discuss > just the downloading of the file, not whether it is created on the local > instance. > > > > > > Does anyone have any pointers? > > > > > > Thanks > > > Kevin > > > -- > > > > > > *Kevin O'Connor* > > > Principal DevOps Engineer > > > M: 617-834-1291 > > > > > > email-footer-logos.jpg (1000×120) > > > > > > STATEMENT OF CONFIDENTIALITY: The information contained in this > message and any attachments are intended solely for the addressee(s) and > may contain confidential or privileged information. If you are not the > intended recipient, or responsible for delivering the e-mail to the > intended recipient, you have received this message in error. Any use, > dissemination, forwarding, printing, or copying is strictly prohibited. > Please notify Ampion immediately at secur...@ampion.net and destroy all > copies of this message and any attachments. > > > > > > > > > _______________________________________________ > > > > > > Manage your clamav-users mailing list subscription / unsubscribe: > > > https://lists.clamav.net/mailman/listinfo/clamav-users > <https://lists.clamav.net/mailman/listinfo/clamav-users> > > > > > > > > > Help us build a comprehensive ClamAV guide: > > > https://github.com/Cisco-Talos/clamav-documentation > <https://github.com/Cisco-Talos/clamav-documentation> > > > > > > https://docs.clamav.net/#mailing-lists-and-chat > <https://docs.clamav.net/#mailing-lists-and-chat> > > > > _______________________________________________ > > > > Manage your clamav-users mailing list subscription / unsubscribe: > > https://lists.clamav.net/mailman/listinfo/clamav-users > <https://lists.clamav.net/mailman/listinfo/clamav-users> > > > > > > Help us build a comprehensive ClamAV guide: > > https://github.com/Cisco-Talos/clamav-documentation > <https://github.com/Cisco-Talos/clamav-documentation> > > > > https://docs.clamav.net/#mailing-lists-and-chat > <https://docs.clamav.net/#mailing-lists-and-chat> > > > > _______________________________________________ > > Manage your clamav-users mailing list subscription / unsubscribe: > https://lists.clamav.net/mailman/listinfo/clamav-users > <https://lists.clamav.net/mailman/listinfo/clamav-users> > > > Help us build a comprehensive ClamAV guide: > https://github.com/Cisco-Talos/clamav-documentation > <https://github.com/Cisco-Talos/clamav-documentation> > > https://docs.clamav.net/#mailing-lists-and-chat > <https://docs.clamav.net/#mailing-lists-and-chat> > >
_______________________________________________ Manage your clamav-users mailing list subscription / unsubscribe: https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/Cisco-Talos/clamav-documentation https://docs.clamav.net/#mailing-lists-and-chat
