Re: [clamav-users] Multiple Streams embedded as base64 inside xml

[G.W. Haywood via clamav-users]

Hi there,

On Fri, 24 Apr 2020, Kris Deugau wrote:


G.W. Haywood via clamav-users wrote:

 It's quite possible that a scan could catch some
known problem in *any* file, no matter how compressed, containerized
and obfuscated, if there's already a signature which matches something
in the raw file (that is, before any extraction and/or decoding takes
place);


That's not entirely true, although I'd be happy to be proven wrong.

I've tried a couple of times to create signatures for Javascript malware (and 
asked for pointers on this list a couple of times), based on an obfuscation 
pattern in a series of raw files.  I have yet to find a way to actually match 
on the actual raw file in those cases.


I see some posts from you in 2016 which seemed to be basically about
normalization.  Normalization was causing signatures for those things
to fail to match, but switching normalization off would have the same
effect on signatures which needed to work on normalized text.  Absent
a signature type which calls for non-normalized text, I think the way
I'd approach that would be to run two instances of clamd - one for the
bulk of the signatures, and one for the (few?) custom signatures which
need to work on the raw files.  In 2015 you said that you had trouble
getting signatures of the form

AB??CD??EF??...

to work.  I don't know if that's still a problem, but if I were going
to look for such things I'd find it much quicker and easier to add a
Perl regex to my milter configuration than to write ClamAV signatures.
4-5 years ago I was heavily overworked with a new milter, otherwise I
might have piped up at the time.  For the omissions I apologize.

I've remarked before that the bodies of mail which you and I seem to
see are very different.  I don't recall ever seeing any of the kind of
obfuscation which has bothered you, but then I probably drop the mails
before they get as far as body scanning.  That's a luxury I can afford
which perhaps you can't, but anything from a Yahoo server which claims
a gmail sender address is, in my view, fair game...

--

73,
Ged.

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Multiple Streams embedded as base64 inside xml

[Kris Deugau]

G.W. Haywood via clamav-users wrote:

 It's quite possible that a scan could catch some
known problem in *any* file, no matter how compressed, containerized
and obfuscated, if there's already a signature which matches something
in the raw file (that is, before any extraction and/or decoding takes
place);


That's not entirely true, although I'd be happy to be proven wrong.

I've tried a couple of times to create signatures for Javascript malware 
(and asked for pointers on this list a couple of times), based on an 
obfuscation pattern in a series of raw files.  I have yet to find a way 
to actually match on the actual raw file in those cases.


-kgd

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Multiple Streams embedded as base64 inside xml

[G.W. Haywood via clamav-users]

Hi there,

On Thu, 23 Apr 2020, Görkem ÇINAR via clamav-users wrote:


I have an xml file which has list of pdf files embedded as base64.

When I scan that xml file, does it also scans those base64 content inside
that xml or do i need to convert those base64 contents into different
streams and scan them individually?


If ClamAV recognizes that there's base64 encoded text to be scanned it
will try to scan it, but it's not as simple as that.  See for example

https://blog.talosintelligence.com/2013/01/the-0-day-that-wasnt-dissecting-highly.html

To get an answer in one particular case - but perhaps _only_ in that
particular case, see

http://www.clamav.net/documents/creating-signatures-for-clamav

especially the part about half way down the page which talks about

clamscan --debug

and saving temporary files to show how ClamAV has processed the file.

A signature is just something which matches a string of bytes in the
data being scanned.  It's quite possible that a scan could catch some
known problem in *any* file, no matter how compressed, containerized
and obfuscated, if there's already a signature which matches something
in the raw file (that is, before any extraction and/or decoding takes
place); so it might not be necessary for ClamAV to do any processing
on the file before scanning.  Some signatures look specifically for
strings which have been obfuscated; try for example

sigtool -l | grep Obfuscated

for what's in your ClamAV database.

While ClamAV is of course capable of decoding base64 text, there are
caveats.  There's a tradeoff between scan times and the probability
that something detectable might be present in what's being scanned,
and the signatures themselves contain a field which determines their
applicability so that ClamAV doesn't waste its time scanning for some
threat which cannot be present in the scanned data.  If a signature is
restricted to a certain kind of data (it doesn't have to be, but many
are), then no matter whether or not it would match anything in the
scanned data, it won't be used in the scan if ClamAV believes that it
is not scanning that kind of data.  One of the things many malicious
authors try (sometimes quite hard, as you've seen) to do is hide the
real intent of their creation.  Sometimes they're successful, so even
if the answer to your question was a simple "yes", you couldn't really
rely on it.

Not only are you to some extent at the mercy of the malware authors,
you also to some extent depend on the whims of the signature writers.

--

73,
Ged.

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] IP Blacklisted by Mirror

[Kamil Wójcik via clamav-users]

Thanks all for replies. I manage to find root cause of the issue. 

My infrastructure is hosted on EC2 with EC2 instance having public IP assigned. 
I’m using dockerized version of clamav. Due to OOM, container got into kind of 
CrashLoopBackOff situation. It was crashing during signature update and 
starting again. This caused frequent requests for signatures to mirror which at 
some point caused IP blacklisting. 

 In my situation upgrade of ec2 to bigger will cause allocation of new IP 
addressed. Solving issue.

This would have been avoided with proper clamav monitoring. 

Regards Kamil

> On 23 Apr 2020, at 17:28, Joel Esler (jesler) via clamav-users 
>  wrote:
> 
> Team —
> 
> I control Cloudflare.  Who is blocked and who is not is literally up to me.  
> If you are being blocked, feel free to write me 1:1, share your IP with me, 
> and I’ll tell you why you’re blocked.  A ticket can also be filed on 
> bugzilla.clamav.net  under “mirrors”
> 
> 
> 
>> On Apr 23, 2020, at 10:46 AM, web...@manfbraun.de 
>>  wrote:
>> 
>> Hello!
>> 
>> I reported that exakt problem years ago.
>> 
>> There is NO direct solution.
>> 
>> It's that, that cloudflaire is not interested
>> in requests of users - it' "a engine".
>> They are even blocking the tor network and
>> redirect to google, to solve a pscholical
>> fingerprint to identify you, it's just nothing
>> else then IT-Oligarchism.
>> 
>> I was never able to contact them but came
>> to the lucky situation to have another box,
>> a VPS. So I installed a proxy there and configured
>> clam to use it. Problem fixed.
>> 
>> Has, at least, nothing to do with the amount
>> of request, which are 24/day for my case.
>> 
>> Best regards,
>> Manfred
>> 
>> 
>>> -Original Message-
>>> From: clamav-users [mailto:clamav-users-boun...@lists.clamav.net 
>>> ] On
>>> Behalf Of Matus UHLAR - fantomas
>>> Sent: Thursday, April 23, 2020 4:09 PM
>>> To: clamav-users@lists.clamav.net 
>>> Subject: Re: [clamav-users] IP Blacklisted by Mirror
>>> 
>>> On 23.04.20 15:24, Kamil Wójcik via clamav-users wrote:
 I have clamav configured with freshclam for signature updates.
 I found that freshclam is failing to update signatures with error "Wed
>>> Apr 22 10:12:34 2020 -> ^getpatch: Can't download daily-25717.cdiff from
>>> db.local.clamav.net  
>>> >”
 
 I checked with curl and I’m getting HTTP 403 with response "error code:
>>> 1006”. I believe this is cloudflare error indicating that  ip was
>>> blacklisted. Is there any other public mirror that I could use instead ?
 What should I do to get ip removed from blacklist ?
>>> 
>>> how often do you mirror? 50 times a day.
>>> Aren't you by any chance behind NAT, maybe with more machines?
>>> --
>>> Matus UHLAR - fantomas, uh...@fantomas.sk  ; 
>>> http://www.fantomas.sk/ 
>>> Warning: I wish NOT to receive e-mail advertising to this address.
>>> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
>>> Windows 2000: 640 MB ought to be enough for anybody
>>> 
>>> ___
>>> 
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net 
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>> 
>>> 
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>> 
>>> http://www.clamav.net/contact.html#ml
>> 
>> 
>> ___
>> 
>> clamav-users mailing list
>> clamav-users@lists.clamav.net 
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>> 
>> 
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>> 
>> http://www.clamav.net/contact.html#ml
> 
> 
> ___
> 
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml


___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] about clamd boot sequence on Linux system.

[Tsutomu Oyamada]

Hi,

Thank you for your reply.
I'm sorry for the slow reply.

I understood that there were two processes when clamd was started. In
addition, since the DB load time of clamd varies depending on the amount
of memory installed in the system, the existence time of the two
processes varies depending on the system.

If the clamd process is using an official CVD file, it will require 2GB
or more of the system's memory.

Thank you so much.

Betregard,
T.O.

On Mon, 20 Apr 2020 14:21:00 +0100 (BST)
"G.W. Haywood via clamav-users"  wrote:

> Hi there,
> 
> On Mon, 20 Apr 2020, Tsutomu Oyamada wrote:
> 
> > There are two processes temporarily at clamd startup, is this a 
> > specification?
> 
> If I understand your English, yes.  There will be two processes (or
> threads) running every time the database is being reloaded.  Each will
> use about the same maximum amount of memory, although one will exit
> after the reload is completed and its memory will then be released.
> 
> Please be aware of the distinction between a database update (which is
> performed by freshclam) and a database reload (which is performed by
> clamd itself).  A reload may take place immediately after an update if
> freshclam signals clamd to reload it; if freshclam does not do so, and
> that is configurable, it will take place when clamd next notices that
> the database has changed (usually when it is next called upon to scan
> something).
> 
> Please also be aware that if you run 'clamscan' then it will load its
> own copy of the databases too, but 'clamdscan' will not - it will use
> the clamd daemon to do the scanning.
> 
> > Is this going to be three or more?
> 
> Not normally, but you are at liberty to run more than one clamd
> process (if you configure them correctly) and I frequently do that.
> In such a case you are expected to know exactly what you are doing,
> and why you are doing it, and to have enough memory.
> 
> > On my system, after booting, it is in a state of following a few seconds.
> >
> > ps -aux
> > root  75687  100 44.2 944120 899844 ?   RN   00:00   0:27 
> > /usr/lib/clamav/clamd --config-file=/etc/clamav/clamd.conf
> > root  75856  0.0 44.0 1017852 895532 ?  SNsl 00:00   0:00 
> > /usr/lib/clamav/clamd --config-file=/etc/clamav/clamd.conf
> 
> The command which you gave above did not produce the output which you
> claim was produced.  It would be more helpful to give a command such as
> 
> ps -aux | grep clam
> 
> So that we can see exactly what is happening.
> 
> > This was not the case on systems with a lot of memory.
> 
> You have not said how much memory is present on the system!  But for a
> system running clamd you should normally expect to need more than two
> GBytes because during a database update clamd will have two copies of
> the databases loaded (and just a single copy of the official databases
> uses about one GByte of RAM) - and of course the rest of the system
> needs memory too.  You _can_ get away with using swap, but it will
> slow things down dramatically.  Even if it does not need to use swap,
> for just the official databases, depending on the performance of your
> systems you can expect a database reload to take anywhere between some
> seconds and some minutes.  In addition to the 'official' databases
> from Cisco/Talos I will typically use 30 - 40 'unofficial' databases;
> most of them aim to recognize spam rather than malware, but there is a
> lot of overlap.
> 
> -- 
> 73,
> Ged.
> 
> ___
> 
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml
> 



___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml