Re: [clamav-users] Question on ClamAV memory usage with respect to the signature database

2024-06-24 Thread Mikhail Soumar via clamav-users 
Hi Micah,

Thank you for your response. I have been actually trying what you suggested 
with the sigtool command, and when removing Windows signatures from both 
daily.cvd and main.cvd, we saw a memory savings of about 1 GB during the scan, 
from 1.5 GB to 500-600 MB. However, I still haven’t figured out a good way to 
rebuild the databases so that we can update them using cdiffs or some variation 
thereof (for example, getting the diff, removing any lines containing “Win.”, 
and then proceeding with the new diff).

In particular, when using `sigtool --build` I wasn’t able to find anything in 
the documentation/code about what’s expected for the mandatory --server 
parameter, or how to write the cdiffs in freshclam (or sigtool) to a file to do 
the aforementioned modifications to remove any references to Windows 
signatures. Is there anything you suggest we do to maintain a current database 
in this manner?

Otherwise, since the majority of Windows signatures are in main.cvd, we can 
probably get most of the memory savings by just dropping main.cvd as you 
suggested, and either omitting it from the scan or just including the 
non-Windows signatures, and then updating the daily and bytecode databases as 
normal.

Thank you,
Mikhail


From: Micah Snyder (micasnyd) 
Sent: Monday, June 24, 2024 12:35 PM
To: clamav-users@lists.clamav.net
Cc: Mikhail Soumar 
Subject: [EXTERNAL] Re: Question on ClamAV memory usage with respect to the 
signature database

You don't often get email from micas...@cisco.com<mailto:micas...@cisco.com>. 
Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Hi Mikhail,

As you probably know, the clamav signature database is comprised of daily.cvd, 
main.cvd, and bytecode.cvd.  Note: I say "cvd" but the file will have a "cld" 
extension if freshclam has updated it from an older version using our cdiff 
patching update mechanism.

Daily.cvd is updated daily and contains the most recent threats. We 
infrequently migrate the most stable signatures to main.cvd.  Bytecode.cvd 
doesn't change often but contains some more complex logic for detecting malware.

One option is to drop main.cvd (or main.cld) and only scan with daily.cvd + 
bytecode.cvd.  That won't detect as many older threats but should still detect 
recent threats and will reduce your memory footprint.

Another idea to meet your specific request (no windows sigs) is to use "sigtool 
--unpack​ CVDFILE" to extract the signature files into the current directory 
from a CVD archive.  You could then strip out any lines containing "Win.".  I 
haven't actually tried this, so I don't know how much memory savings you'll 
realize. I'd be curious what you find if you do try it.

If you look at the clamav-users mailing list archives, you may notice some 
folks discovered we have a linux.cvd​.  I DON'T recommend using it.  The Linux 
CVD is made to be supplemental to the Secure Endpoint for Linux client, which 
has other detection mechanisms.  Linux.cvd has barely changed in the last 
couple years. But it is very lightweight.

I don't have any other ideas at this time. I've been pushing for quite a while 
now on an internal project to identify lower-value signatures so we can archive 
them to reduce the RAM requirements, improve load time, and improve scan time. 
That's been very slow going and I really don't know what else to say about it. 爛

Regards,
Micah


Micah Snyder (they/them)
ClamAV Development
Talos
Cisco Systems, Inc.

From: clamav-users 
mailto:clamav-users-boun...@lists.clamav.net>>
 on behalf of Mikhail Soumar via clamav-users 
mailto:clamav-users@lists.clamav.net>>
Sent: Monday, June 17, 2024 9:06 PM
To: clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net> 
mailto:clamav-users@lists.clamav.net>>
Cc: Mikhail Soumar mailto:msou...@microsoft.com>>
Subject: [clamav-users] Question on ClamAV memory usage with respect to the 
signature database


Hello,



We are a team from Microsoft Azure running ClamAV on small Linux VMs, and due 
to business and cost reasons we cannot use larger VMs. Peak memory usage of 
ClamAV is between 1.2GB and 1.5GB, which is unsustainable on our VMs, and we 
are looking for ways to reduce this. There are some tips to reduce memory usage 
in the Docker section of the documentation (Docker - ClamAV 
Documentation<https://docs.clamav.net/manual/Installing/Docker.html#memory-ram-requirements>)
 although if I understand correctly the 1.2GB load is unavoidable even with the 
suggestions listed on this page.



We have been told that one possibility is to remove all virus signatures that 
are Windows-specific, which would reduce the memory footprint to about 300 MB. 
Elsewhere on the ClamAV FAQ I see a few different ways to add signatures to the 
database but none about taking a subset. Would this be something you support or 
 recommend for our use case?

Re: [clamav-users] Question on ClamAV memory usage with respect to the signature database

2024-06-24 Thread Micah Snyder (micasnyd) via clamav-users 
Hi Mikhail,

As you probably know, the clamav signature database is comprised of daily.cvd, 
main.cvd, and bytecode.cvd.  Note: I say "cvd" but the file will have a "cld" 
extension if freshclam has updated it from an older version using our cdiff 
patching update mechanism.

Daily.cvd is updated daily and contains the most recent threats. We 
infrequently migrate the most stable signatures to main.cvd.  Bytecode.cvd 
doesn't change often but contains some more complex logic for detecting malware.

One option is to drop main.cvd (or main.cld) and only scan with daily.cvd + 
bytecode.cvd.  That won't detect as many older threats but should still detect 
recent threats and will reduce your memory footprint.

Another idea to meet your specific request (no windows sigs) is to use "sigtool 
--unpack​ CVDFILE" to extract the signature files into the current directory 
from a CVD archive.  You could then strip out any lines containing "Win.".  I 
haven't actually tried this, so I don't know how much memory savings you'll 
realize. I'd be curious what you find if you do try it.

If you look at the clamav-users mailing list archives, you may notice some 
folks discovered we have a linux.cvd​.  I DON'T recommend using it.  The Linux 
CVD is made to be supplemental to the Secure Endpoint for Linux client, which 
has other detection mechanisms.  Linux.cvd has barely changed in the last 
couple years. But it is very lightweight.

I don't have any other ideas at this time. I've been pushing for quite a while 
now on an internal project to identify lower-value signatures so we can archive 
them to reduce the RAM requirements, improve load time, and improve scan time. 
That's been very slow going and I really don't know what else to say about it. 爛

Regards,
Micah


Micah Snyder (they/them)
ClamAV Development
Talos
Cisco Systems, Inc.

From: clamav-users  on behalf of Mikhail 
Soumar via clamav-users 
Sent: Monday, June 17, 2024 9:06 PM
To: clamav-users@lists.clamav.net 
Cc: Mikhail Soumar 
Subject: [clamav-users] Question on ClamAV memory usage with respect to the 
signature database


Hello,



We are a team from Microsoft Azure running ClamAV on small Linux VMs, and due 
to business and cost reasons we cannot use larger VMs. Peak memory usage of 
ClamAV is between 1.2GB and 1.5GB, which is unsustainable on our VMs, and we 
are looking for ways to reduce this. There are some tips to reduce memory usage 
in the Docker section of the documentation (Docker - ClamAV 
Documentation<https://docs.clamav.net/manual/Installing/Docker.html#memory-ram-requirements>)
 although if I understand correctly the 1.2GB load is unavoidable even with the 
suggestions listed on this page.



We have been told that one possibility is to remove all virus signatures that 
are Windows-specific, which would reduce the memory footprint to about 300 MB. 
Elsewhere on the ClamAV FAQ I see a few different ways to add signatures to the 
database but none about taking a subset. Would this be something you support or 
 recommend for our use case? If not, are there alternatives we can consider to 
reduce the memory footprint of ClamAV well below 1.2GB?



Thank you,

Mikhail
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

Re: [clamav-users] Question on ClamAV memory usage with respect to the signature database

2024-06-18 Thread Andrew C Aitchison via clamav-users 

On Tue, 18 Jun 2024, Mikhail Soumar via clamav-users wrote:


We are a team from Microsoft Azure running ClamAV on small Linux
VMs, and due to business and cost reasons we cannot use larger
VMs. Peak memory usage of ClamAV is between 1.2GB and 1.5GB, which
is unsustainable on our VMs, and we are looking for ways to reduce
this. There are some tips to reduce memory usage in the Docker
section of the documentation (Docker - ClamAV
Documentation)
although if I understand correctly the 1.2GB load is unavoidable
even with the suggestions listed on this page.

We have been told that one possibility is to remove all virus
signatures that are Windows-specific, which would reduce the memory
footprint to about 300 MB. Elsewhere on the ClamAV FAQ I see a few
different ways to add signatures to the database but none about
taking a subset. Would this be something you support or recommend
for our use case? If not, are there alternatives we can consider to
reduce the memory footprint of ClamAV well below 1.2GB?


ClamAV has never caught a Linux virus for me, so I don't know whether
it makes sense to run ClamAV without the Windows data.
Do you have the resources to curate a custom database,
bearing in mind that the standard dbs are updated daily ?
(freshclam and cvdupdate do work with the cdiff incremental updates,
so at least you would not have to remove the same signatures from the
database every day.)

I don't know how viable this is, but you do not have to run the ClamAV
daemon on every VM; you can use a remote daemon and pass files to be
scanned with clamdscan. This would also save you more than 10 seconds
at startup.

How much memory does Microsoft Defender use on Linux ?

--
Andrew C. Aitchison  Kendal, UK
   and...@aitchison.me.uk
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

[clamav-users] Question on ClamAV memory usage with respect to the signature database

2024-06-17 Thread Mikhail Soumar via clamav-users 
Hello,

We are a team from Microsoft Azure running ClamAV on small Linux VMs, and due 
to business and cost reasons we cannot use larger VMs. Peak memory usage of 
ClamAV is between 1.2GB and 1.5GB, which is unsustainable on our VMs, and we 
are looking for ways to reduce this. There are some tips to reduce memory usage 
in the Docker section of the documentation (Docker - ClamAV 
Documentation)
 although if I understand correctly the 1.2GB load is unavoidable even with the 
suggestions listed on this page.

We have been told that one possibility is to remove all virus signatures that 
are Windows-specific, which would reduce the memory footprint to about 300 MB. 
Elsewhere on the ClamAV FAQ I see a few different ways to add signatures to the 
database but none about taking a subset. Would this be something you support or 
 recommend for our use case? If not, are there alternatives we can consider to 
reduce the memory footprint of ClamAV well below 1.2GB?

Thank you,
Mikhail
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat