[Rspamd-Users] ClamAV and rspamd : log question

[G.W. Haywood]

Hi there,

On Thu, 16 Feb 2023, Mickaël Dequidt wrote:

> Would anyone be able to explain to me why I can't see a log line
>
>> lua; clamav.lua:131: clamav: message or mime_part is clean
>
> for each clean message but only for ~ 1/3 of them ?

Without more information I can't answer your question directly, but I
think that I can offer some advice.  For anyone else reading, this is
all assuming that ClamAV is being used to scan mail, so if that's not
what you're doing then the advice probably isn't useful to you.

You need to put a lot of thought into your configuration for the
ClamAV scanner.  The scanner has its own configuration, which is quite
separate from the rspamd configuration.  Setting it up needs care, but
I won't go into that in detail here - just a few tips.

Before anything else make sure that ClamAV itself is logging useful
stuff by checking the configuration file, probably 'clamd.conf' (or if
you're using one of the packaged versions from one of the stir-crazy
distributions like RedHat, which renames many of the configuration
files which it distributes, maybe 'scan.conf').  At least to begin
with make the logging verbose.  Check that you can find the log and
that you can search it, e.g. tail it with a pager like 'less' - you
might find that you have what you need in the ClamAV logs already.

When you've checked the configuration, and restarted the clamd daemon
if necessary after any configuration changes, look in the ClamAV logs
to see if they support what the rspamd logs seem to be telling you.

> here is my local.d/antivirus.conf ::
>
>> clamav {
>>     scan_mime_parts = false;
>>     scan_text_mime = false;
>>     scan_image_mime = false;
>>     max_size = 20000000000;
>>     symbol = "CLAM_VIRUS";
>>     type = "clamav";
>>     log_clean = true;
>>     servers = "127.0.0.1:3310";
>>     patterns_fail {
>>         CLAM_LIMITS_EXCEEDED = '^Heuristics\.Limits\.Exceeded$';
>>     }
>>     timeout = 30.0;
>>     whitelist = "/etc/rspamd/antivirus.wl";
>> }

Looking now at your rspamd configuration:

1. Your first three configuration lines are "scan_something = false"
which might not be what you really want.  You do not say if you are
using third-party ClamAV databases.  If not, I recommend that you do
at least grab the Sanesecurity databases because (in my experience of
more than 20 years using ClamAV) if you use only the ClamAV 'official'
databases you'll get much poorer results.  Some of the 'unofficial'
databases might perform better if e.g. you enable text part scans at
the very least, and there are ClamAV signatures for malicious images
so you might want to enable everything.

2. Your line "max_size = 20000000000" makes no sense.  I'm sure you
would never want to scan a 20 gigabyte email, even if it could be done
in reasonable time without causing problems for ClamAV.  The maximum
possible scan size configurable for ClamAV is ten times smaller than
your number anyway.  ClamAV has its own maxima, and the default values
might be smaller than you'd expect.  There's an element of protection
from Denial Of Service attacks in the configuration.  Please read the
ClamAV documentation for more information, especially the 'man' page
for clamd.conf which contains useful calculation.

3, You have "servers" set for a TCP connection.  You might want to use
a Unix socket instead, which is more secure and probably faster.  This
assumes that the ClamAV scanner is on the host running rspamd, if it's
on a separate host then you have little choice but to use TCP.

4, I admit I've never used these patterns with rspamd, but I wonder if
your "patterns_fail" regex will ever match.  I should expect that the
'$' symbol to anchor the word "Exceeded" to the end of a line but when
ClamAV gives such a warning, it gives something like these:

Heuristics.Limits.Exceeded.MaxFileSize
Heuristics.Limits.Exceeded.MaxScanTime

Maybe you'll just need to remove the '$' symbol.

5. Your timeout setting is only 30 seconds.  ClamaV is not especially
fast, and - depending on many and very variable factors, like system
specifications and load - for some emails with large attachments this
might not be enough.  On modest hardware, I've seen relatively small
PDFs for example take some tens of seconds to scan.  Having said that
I doubt that many mail systems handle mail in which two-thirds of the
messages take longer than 30 seconds to scan, so I doubt this could be
the explanation for your observations.  ClamAV has its own timeouts in
the file clamd.conf (or whatever it's called on your system).

> It makes me fear that clamav doesn't scan everything and that some
> emails are just passing through...

Even if you *are* scanning all emails, my estimate of ClamAV's success
rate in finding malicious content is of the order of a few percent.
Sometimes I've published numbers on the ClamAV users mailing list, see
for example

https://marc.info/?l=clamav-users&m=162379914711853&w=2

but it still seems to me most people think that if they've scanned
something with *anything* then it must be safe.  You can't rely on
*any* virus scanner to give 100% protection.  You can always be sure
that some threats will be missed.  The proportion of threats *missed*
ranges between about one in six for the best scanner on its best day
to around 19 out of 20, even on an average day, for any of the worst.

In the systems here, a few dozen Yara rules catch a great deal more
than any virus scanner, but it demands a lot of skilled attention to
ensure that false positives are handled appropriately.  There's no
such thing as a 'fire-and-forget' anti-virus system I'm afraid.

-- 

73,
Ged.