[Rspamd-Users] ClamAV and rspamd : log question

Fri Feb 17 07:42:35 UTC 2023

Well, thank you for this very thorough answer, I guess I got more than I 
bargained for !

I wasn't very precise in my original email, sorry about that. Let me 
clarify a little : I am migrating a MX from an old antispam solution to 
rspamd, and clamav is following me with quite a bit of history, 
including many unofficial signatures, sanesecurity, yara rules, and 
subscription to other sources. Our conf has so far stood the test of 
time, but I do admit I've only ever seen clamav log infected mails, not 
clean ones. I'll look into that to see if there is a way for it to log 
everything is scans. And of course, I don't expect clamav to be an 
absolute bulwark against any and all possible dangers : 0-day malware 
will pass through, issues will arise, nothing is fullproof. But I do 
expect clamav to check every email to detect what it can detect every 
time it appears.

Going back to my conf file : there was a typo in the max_size, it was 
supposed to be 20Mo (which is the max size accepted by the local postfix 
anyway), that will be corrected. I wasn't sure about the timeout value, 
I will up it.

Now, the options

> scan_mime_parts
> scan_text_mime
> scan_image_mime

as I understood them, were supposed to be set to true to pass every mime 
parts of an email separately to clamav, as opposed to the entire message 
sent as a whole. I conducted tests that suggested that some signatures 
weren't matched if the message wasn't scanned as a whole, hence my 
setting to "false". But I would welcome with enthusiasm any explanation 
as to the advantages and drawbacks of these options, because the online 
documentation is rather scarce.

Now, for my original question : I know that clamav scans some emails, as 
I have seen it in the logs. Bu my understanding of the "log_clean = 
true;" setting was that a line of log would be added for every clean 
clamav scan. And what I see in my logs is that the processing of some 
messages creates such a log line :

> lua; clamav.lua:131: clamav: message or mime_part is clean

some, but not all. Which is why I wonder : is rspamd only logging some 
clean scan, or is rspamd only scanning some messages ?

I hope the situation is clearer.

Le 16/02/2023 à 17:20, G.W. Haywood via Users a écrit :
> Hi there,
> 
> On Thu, 16 Feb 2023, Mickaël Dequidt wrote:
> 
>> Would anyone be able to explain to me why I can't see a log line
>>
>>> lua; clamav.lua:131: clamav: message or mime_part is clean
>>
>> for each clean message but only for ~ 1/3 of them ?
> 
> Without more information I can't answer your question directly, but I
> think that I can offer some advice.  For anyone else reading, this is
> all assuming that ClamAV is being used to scan mail, so if that's not
> what you're doing then the advice probably isn't useful to you.
> 
> You need to put a lot of thought into your configuration for the
> ClamAV scanner.  The scanner has its own configuration, which is quite
> separate from the rspamd configuration.  Setting it up needs care, but
> I won't go into that in detail here - just a few tips.
> 
> Before anything else make sure that ClamAV itself is logging useful
> stuff by checking the configuration file, probably 'clamd.conf' (or if
> you're using one of the packaged versions from one of the stir-crazy
> distributions like RedHat, which renames many of the configuration
> files which it distributes, maybe 'scan.conf').  At least to begin
> with make the logging verbose.  Check that you can find the log and
> that you can search it, e.g. tail it with a pager like 'less' - you
> might find that you have what you need in the ClamAV logs already.
> 
> When you've checked the configuration, and restarted the clamd daemon
> if necessary after any configuration changes, look in the ClamAV logs
> to see if they support what the rspamd logs seem to be telling you.
> 
>> here is my local.d/antivirus.conf ::
>>
>>> clamav {
>>>     scan_mime_parts = false;
>>>     scan_text_mime = false;
>>>     scan_image_mime = false;
>>>     max_size = 20000000000;
>>>     symbol = "CLAM_VIRUS";
>>>     type = "clamav";
>>>     log_clean = true;
>>>     servers = "127.0.0.1:3310";
>>>     patterns_fail {
>>>         CLAM_LIMITS_EXCEEDED = '^Heuristics\.Limits\.Exceeded$';
>>>     }
>>>     timeout = 30.0;
>>>     whitelist = "/etc/rspamd/antivirus.wl";
>>> }
> 
> Looking now at your rspamd configuration:
> 
> 1. Your first three configuration lines are "scan_something = false"
> which might not be what you really want.  You do not say if you are
> using third-party ClamAV databases.  If not, I recommend that you do
> at least grab the Sanesecurity databases because (in my experience of
> more than 20 years using ClamAV) if you use only the ClamAV 'official'
> databases you'll get much poorer results.  Some of the 'unofficial'
> databases might perform better if e.g. you enable text part scans at
> the very least, and there are ClamAV signatures for malicious images
> so you might want to enable everything.
> 
> 2. Your line "max_size = 20000000000" makes no sense.  I'm sure you
> would never want to scan a 20 gigabyte email, even if it could be done
> in reasonable time without causing problems for ClamAV.  The maximum
> possible scan size configurable for ClamAV is ten times smaller than
> your number anyway.  ClamAV has its own maxima, and the default values
> might be smaller than you'd expect.  There's an element of protection
> from Denial Of Service attacks in the configuration.  Please read the
> ClamAV documentation for more information, especially the 'man' page
> for clamd.conf which contains useful calculation.
> 
> 3, You have "servers" set for a TCP connection.  You might want to use
> a Unix socket instead, which is more secure and probably faster.  This
> assumes that the ClamAV scanner is on the host running rspamd, if it's
> on a separate host then you have little choice but to use TCP.
> 
> 4, I admit I've never used these patterns with rspamd, but I wonder if
> your "patterns_fail" regex will ever match.  I should expect that the
> '$' symbol to anchor the word "Exceeded" to the end of a line but when
> ClamAV gives such a warning, it gives something like these:
> 
> Heuristics.Limits.Exceeded.MaxFileSize
> Heuristics.Limits.Exceeded.MaxScanTime
> 
> Maybe you'll just need to remove the '$' symbol.
> 
> 5. Your timeout setting is only 30 seconds.  ClamaV is not especially
> fast, and - depending on many and very variable factors, like system
> specifications and load - for some emails with large attachments this
> might not be enough.  On modest hardware, I've seen relatively small
> PDFs for example take some tens of seconds to scan.  Having said that
> I doubt that many mail systems handle mail in which two-thirds of the
> messages take longer than 30 seconds to scan, so I doubt this could be
> the explanation for your observations.  ClamAV has its own timeouts in
> the file clamd.conf (or whatever it's called on your system).
> 
>> It makes me fear that clamav doesn't scan everything and that some
>> emails are just passing through...
> 
> Even if you *are* scanning all emails, my estimate of ClamAV's success
> rate in finding malicious content is of the order of a few percent.
> Sometimes I've published numbers on the ClamAV users mailing list, see
> for example
> 
> https://marc.info/?l=clamav-users&m=162379914711853&w=2
> 
> but it still seems to me most people think that if they've scanned
> something with *anything* then it must be safe.  You can't rely on
> *any* virus scanner to give 100% protection.  You can always be sure
> that some threats will be missed.  The proportion of threats *missed*
> ranges between about one in six for the best scanner on its best day
> to around 19 out of 20, even on an average day, for any of the worst.
> 
> In the systems here, a few dozen Yara rules catch a great deal more
> than any virus scanner, but it demands a lot of skilled attention to
> ensure that false positives are handled appropriately.  There's no
> such thing as a 'fire-and-forget' anti-virus system I'm afraid.
> 

-- 
Mickaël DEQUIDT
IFREMER - Service IRSI/RIC
Centre Ifremer Bretagne - ZI de la pointe du diable
CS 10070 - 29280 Plouzané
Tel : +33 (0)2 98 22 46 04 - Fax : +33 (0)2 98 22 46 47