[Rspamd-Users] ClamAV and rspamd : log question

[G.W. Haywood]

Hi there,

On Fri, 17 Feb 2023, Mickaël Dequidt wrote:

> Well, thank you for this very thorough answer, I guess I got more
> than I bargained for !

Maybe, but you aren't the only one reading this. :)

> I wasn't very precise ...

We've all been guilty of that. :)

> ... I am migrating a MX from an old antispam solution to rspamd, and
> clamav is following me with quite a bit of history, including many
> unofficial signatures, sanesecurity, yara rules, and ...

I'm totally sold on Yara rules, but the ClamAV implementation of Yara
is appalling - it's a decade old (roughly Yara version 2) and riddled
with crashing faults.  After trying (and failing) to get the ClamAV
team to do something about that I abandoned the effort.  My milters
now use genuine Yara which is vastly better.  At the moment my Yara
scans need to start the Yara executable for every scan but I'm working
on a Yara daemon which will be a lot more efficient.  I looked for one
but to my surprise I failed to find anything.  Incidentally the rules
from Sanesecurity are the only ones I've found especially effective,
and if it were not for those I wouldn't bother with ClamAV at all.  It
hogs a huge amount of resources for precious little reward.  Here's a
trawl through the last four months of ClamAV logs here:

$ ls -l /.../clamav/databases/*db | wc -l
41
$ grep FOUND /... |cut -d':' -f4|cut -d'.' -f1|sort|uniq -c|sort -n
       1  Heuristics
       1  Java
       2  Win
       2  winnow
       2  Xls
      14  MiscreantPunch
      28  Porcupine
     529  Sanesecurity

Note that much of this will be spam, not malware.  I'm not especially
interested in distinguishing between the two, although I do (and send
automated reports to the ClamAV team for malware it doesn't find).

> ... I've only ever seen clamav log infected mails, not clean
> ones. I'll look into that to see if there is a way for it to log
> everything is scans.

ClamAV can certainly do that, it does it here routinely.  Perhaps
there have been changes in the API which trashed the rspamd/ClamAV
interface?  I haven't checked, I don't use it - the interface here
is directly between ClamAV and my milters.

> ... of course, I don't expect clamav to be an absolute bulwark ...

:)

> But I do expect clamav to check every email to detect what it can detect 
> every time it appears.

You just have to tell it. :)

> Now, the options
>
>> scan_mime_parts
>> scan_text_mime
>> scan_image_mime
>
> as I understood them, were supposed to be set to true to pass every mime 
> parts of an email separately to clamav, as opposed to the entire message sent 
> as a whole. I conducted tests that suggested that some signatures weren't 
> matched if the message wasn't scanned as a whole ...

It depends on the signatures of course.  Be aware that because ClamAV
is designed to scan literally anything, for many signatures it needs
to decide what it's scanning - in your case (and mine) for many sigs
it probably needs to know either that it's scanning a mail message or
that it's scanning for example a base64-encoded MIME part or whatever.
You might experiment to see what happens if you feed different chunks
directly to its socket, that's what I do if I'm testing ideas.  My
milters break a message down into MIME pieces for the Yara scans, but
send the full message both to ClamAV and to a (separate) Yara scanner.

> ... the online documentation is rather scarce.

Hmmm.

> Now, for my original question : I know that clamav scans some emails, as I 
> have seen it in the logs. But my understanding of the "log_clean = true;" 
> setting was that a line of log would be added for every clean clamav scan. 
> And what I see in my logs is that the processing of some messages creates 
> such a log line :
>
>> lua; clamav.lua:131: clamav: message or mime_part is clean
>
> some, but not all. Which is why I wonder : is rspamd only logging some clean 
> scan, or is rspamd only scanning some messages ?

As I said to begin with, I can't answer that question directly without
more information but you can certainly get ClamAV to log clean
messages in its own logs, which should readily give you your answer. :)

> I hope the situation is clearer.

Thanks for the clarification.  I've now no doubt you're on top of it. :)

-- 

73,
Ged.