Re: [clamav-users] [ext] Re: parallel processes fail at startup when clamd is running

[Ralf Hildebrandt via clamav-users]

* JOHN URBAN via clamav-users :

> Doing a scan of the entire locally attached storage on Linux nodes,
> including /tmp and /var; and the problem is basically that MPI
> programs trying to launch while that full scan is running fail to
> start up. Once the programs start they do not commonly fail; but a
> very high number of jobs trying to start up when the scan is progress
> fail to start properly. Memory is not a problem; all nodes have >128GB
> of memory.

Since it's so easy to reproduce, why not start those programs using
strace to see which syscalls are failing:

strace --failed-only $program

-- 
Ralf Hildebrandt
Charité - Universitätsmedizin Berlin
Geschäftsbereich IT | Abteilung Netzwerk

Campus Benjamin Franklin (CBF)
Haus I | 1. OG | Raum 105
Hindenburgdamm 30 | D-12203 Berlin

Tel. +49 30 450 570 155
ralf.hildebra...@charite.de
https://www.charite.de
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

Re: [clamav-users] parallel processes fail at startup when clamd is running

[JOHN URBAN via clamav-users]

Doing a scan of the entire locally attached storage on Linux nodes, including 
/tmp and /var; and the problem is basically that MPI programs trying to launch 
while that full scan is running fail to start up. Once the programs start they 
do not commonly fail; but a very high number of jobs trying to start up when 
the scan is progress fail to start properly. Memory is not a problem; all nodes 
have >128GB of memory.

> On 11/28/2022 9:53 AM G.W. Haywood via clamav-users 
>  wrote:
> 
>  
> Hi there,
> 
> On Mon, 28 Nov 2022, JOHN URBAN via clamav-users wrote:
> 
> > We are experiencing a large number of MPI jobs failing indicating
> > the fabric is unavailable when the scans are running. Early in the
> > investigation so not sure if locking, timing, response time or other
> > factors are involved, but I wanted to ask a quick gernal question to
> > see if this is a known issue with easy answers. If not, we will post
> > more detailed information as it is determined.
> 
> More information would probably help.  Please could you clarify why in
> your subject you write "when clamd is running", yet in the message you
> write "when the scans are running"?  Even if it's running, clamd might
> not be scanning anything but if it's loaded the official signatures it
> will still probably be using a gigabyte or so of RAM, while it's doing
> nothing but wait for a client connection.
> 
> MPI doesn't figure large in the ClamAV mailing list archives, and MPI
> together with ClamAV was equally unrewarding.  The old ClamAV Bugzilla
> seems to be broken (at least for searches) and the Cisco/Talos ClamAV
> Github issues
> 
> https://github.com/Cisco-Talos/clamav/search?q=MPI=
> 
> gave me no results.  The closest I could get in my searching was [*]:
> 
> https://marc.info/?l=clamav-users=128309131408757=2
> 
> I found it by grepping my local mail archive directory, then perusing
> my favourite mail archiver.  It's a very old post but even so it might
> be helpful.
> 
> What are you actually doing with ClamAV?
> 
> [*] Sorry for those who don't care for the MARC archive, but it seems
> that Pipermail goes back only as far as February 2014. :/
> 
> -- 
> 
> 73,
> Ged.
> ___
> 
> Manage your clamav-users mailing list subscription / unsubscribe:
> https://lists.clamav.net/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/Cisco-Talos/clamav-documentation
> 
> https://docs.clamav.net/#mailing-lists-and-chat
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

Re: [clamav-users] parallel processes fail at startup when clamd is running

[JOHN URBAN via clamav-users]

> On 11/28/2022 11:23 AM Andrew C Aitchison via clamav-users 
>  wrote:
> 
>  
> On Mon, 28 Nov 2022, JOHN URBAN via clamav-users wrote:
> 
> > We are experiencing a large number of MPI jobs failing indicating
> > the fabric is unavailable when the scans are running. Early in the
> > investigation so not sure if locking, timing, response time or other
> > factors are involved, but I wanted to ask a quick gernal question to
> > see if this is a known issue with easy answers. If not, we will post
> > more detailed information as it is determined.
> 
> Not an issue that I am familiar with.
> 
> Are the MPI jobs related to clamav, or just running
> on a system with clamav ?
> 
> Is clamav doing on-access analysis ?  If so I wonder whether it is
> attempting to access the same file, or worse same file-handle, for
> each mpi thread, simultaneously.
> 
> If I remember correctly "fabric" can be a technical term to do with
> message passing, parallelism and networking.
> Is that how you are using it ?
> 
> -- 
> Andrew C. Aitchison  Kendal, UK
> and...@aitchison.me.uk
> ___
> 
> Manage your clamav-users mailing list subscription / unsubscribe:
> https://lists.clamav.net/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/Cisco-Talos/clamav-documentation
> 
> https://docs.clamav.net/#mailing-lists-and-chat


Yes; it covers all the parts to create a network connection, simplest 
definition is probably "The communications network MPI constructs either by 
itself or using a daemon". So it covers if you are using IB, ethernet, and so 
on in particular. In this case they are Infiniband connections using an OFA 
layer.  A connection can include the authentication method and process, the 
hardware used to pass messages, the protocol used, which libraries and even 
which compiler was used in the most general usage.
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

[clamav-users] ClamAV 1.0.0 LTS released

[Micah Snyder (micasnyd) via clamav-users]

Read this online at https://blog.clamav.net/2022/11/clamav-100-lts-released.html


ClamAV 1.0.0 LTS released

The ClamAV 1.0.0 feature release is now stable and available for download on 
ClamAV.net or through Docker 
Hub.

ClamAV 1.0.0 includes the following improvements and changes.

Major changes

  *   Support for decrypting read-only OLE2-based XLS files that are encrypted 
with the default password. Use of the default password will now appear in the 
metadata JSON.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/700

  *   Overhauled the implementation of the all-match feature. The newer code is 
more reliable and easier to maintain.

 *   This project fixed several known issues with signature detection in 
all- match mode:

*   Enabled embedded file-type recognition signatures to match when a 
malware signature also matched in a scan of the same layer.

*   Enabled bytecode signatures to run in all-match mode after a match 
has occurred.

*   Fixed an assortment of all-match edge case issues.

 *   Added multiple test cases to verify correct all-match behavior.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/687

  *   Added a new callback to the public API for inspecting file content during 
a scan at each layer of archive extraction.

 *   The new callback function type is clcb_file_inspection defined in 
clamav.h.

 *   The function cl_engine_set_clcb_file_inspection() may be used to 
enable the callback prior to performing a scan.

 *   This new callback is to be considered unstable for the 1.0 release. We 
may alter this function in a subsequent feature version.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/170

  *   Added a new function to the public API for unpacking CVD signature 
archives.

 *   The new function is cl_cvdunpack(). The last parameter for the 
function may be set to verify if a CVD's signature is valid before unpacking 
the CVD content to the destination directory.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/690

  *   The option to build with an external TomsFastMath library has been 
removed. ClamAV requires non-default build options for TomsFastMath to support 
bigger floating point numbers. Without this change, database and Windows 
EXE/DLL authenticode certificate validation may fail. The 
ENABLE_EXTERNAL_TOMSFASTMATH build is now ignored.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/742

  *   Moved the Dockerfile and supporting scripts from the main ClamAV 
repository over to a new repository: 
https://github.com/Cisco-Talos/clamav-docker

The separate repository will make it easier to update the images and fix issues 
with images for released ClamAV versions.

Any users building the ClamAV Docker image rather than pulling them from Docker 
Hub will have to get the latest Docker files from the new location.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/764

  *   Increased the SONAME major version for libclamav because of ABI changes 
between the 0.103 LTS release and the 1.0 LTS release.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/778

Other improvements

  *   Add checks to limit PDF object extraction recursion.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/629

  *   Increased the limit for memory allocations based on untrusted input and 
altered the warning message when the limit is exceeded so that it is more 
helpful and less dramatic.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/723

  *   Dramatically improved the build time of libclamav-Rust unit tests. The 
unit test build is included in the time limit for the test itself and was 
timing out on slower systems. The ClamAV Rust code modules now share the same 
build directory, which also reduces the amount of disk space used for the build.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/694

  *   For Windows: The debugging symbol (PDB) files are now installed alongside 
the DLL and LIB library files when built in "RelWithDebInfo" or "Debug" mode.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/669

  *   Relaxed the constraints on the check for overlapping ZIP file entries so 
as not to alert on slightly malformed, but non-malicious, Java (JAR) archives.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/561

  *   Increased the time limit in FreshClam before warning if the DNS entry is 
stale. In combination with changes to update the DNS entry more frequently, 
this should prevent false alarms of failures in the database publication system.

 *   GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/597

  *   Docker: The C library header files are 

Re: [clamav-users] parallel processes fail at startup when clamd is running

[Andrew C Aitchison via clamav-users]

On Mon, 28 Nov 2022, JOHN URBAN via clamav-users wrote:


We are experiencing a large number of MPI jobs failing indicating
the fabric is unavailable when the scans are running. Early in the
investigation so not sure if locking, timing, response time or other
factors are involved, but I wanted to ask a quick gernal question to
see if this is a known issue with easy answers. If not, we will post
more detailed information as it is determined.


Not an issue that I am familiar with.

Are the MPI jobs related to clamav, or just running
on a system with clamav ?

Is clamav doing on-access analysis ?  If so I wonder whether it is
attempting to access the same file, or worse same file-handle, for
each mpi thread, simultaneously.

If I remember correctly "fabric" can be a technical term to do with
message passing, parallelism and networking.
Is that how you are using it ?

--
Andrew C. Aitchison  Kendal, UK
   and...@aitchison.me.uk
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

Re: [clamav-users] parallel processes fail at startup when clamd is running

[G.W. Haywood via clamav-users]

Hi there,

On Mon, 28 Nov 2022, JOHN URBAN via clamav-users wrote:


We are experiencing a large number of MPI jobs failing indicating
the fabric is unavailable when the scans are running. Early in the
investigation so not sure if locking, timing, response time or other
factors are involved, but I wanted to ask a quick gernal question to
see if this is a known issue with easy answers. If not, we will post
more detailed information as it is determined.


More information would probably help.  Please could you clarify why in
your subject you write "when clamd is running", yet in the message you
write "when the scans are running"?  Even if it's running, clamd might
not be scanning anything but if it's loaded the official signatures it
will still probably be using a gigabyte or so of RAM, while it's doing
nothing but wait for a client connection.

MPI doesn't figure large in the ClamAV mailing list archives, and MPI
together with ClamAV was equally unrewarding.  The old ClamAV Bugzilla
seems to be broken (at least for searches) and the Cisco/Talos ClamAV
Github issues

https://github.com/Cisco-Talos/clamav/search?q=MPI=

gave me no results.  The closest I could get in my searching was [*]:

https://marc.info/?l=clamav-users=128309131408757=2

I found it by grepping my local mail archive directory, then perusing
my favourite mail archiver.  It's a very old post but even so it might
be helpful.

What are you actually doing with ClamAV?

[*] Sorry for those who don't care for the MARC archive, but it seems
that Pipermail goes back only as far as February 2014. :/

--

73,
Ged.
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

[clamav-users] parallel processes fail at startup when clamd is running

[JOHN URBAN via clamav-users]

We are experiencing a large number of MPI jobs failing indicating the fabric is 
unavailable when the scans are running. Early in the investigation so not sure 
if locking, timing, response time or other factors are involved, but I wanted 
to ask a quick gernal question to see if this is a known issue with easy 
answers. If not, we will post more detailed information as it is determined.___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

Re: [clamav-users] Scanned files count

[G.W. Haywood via clamav-users]

Hi there,

On Sun, 27 Nov 2022, Jorge Elissalde via clamav-users wrote:


Is there a way to get the count of scanned files for a "SCAN folder"
command?


The question lacks context, but maybe something like this instead?

find /path/ -type f | xargs -I'{}' clamdscan '{}'

--

73,
Ged.
___

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat