[Rspamd-Users] ClamAV and rspamd : log question

[Mickaël Dequidt]

Hello, me again /o\

Things are moving forward : clamav logging allowed me to see that NOT 
all successful smtp transactions were scanned for viruses. But I also 
realized that all greylisted emails are scanned, and so far every 
message I saw that was accepted by rspamd without a direct clamav scan, 
was a previously greylisted message.

So, from a purely rspamd point of view, is rspamd keeping its scan 
results cached (in redis, I would guess ?) when it greylists things, so 
as not to re-scan them if they come back (since it clearly detects them 
as "coming back") ? Would that be a thing ?

Thanks,


Le 17/02/2023 à 12:38, G.W. Haywood via Users a écrit :
> Hi there,
> 
> On Fri, 17 Feb 2023, Mickaël Dequidt wrote:
> 
>> Well, thank you for this very thorough answer, I guess I got more
>> than I bargained for !
> 
> Maybe, but you aren't the only one reading this. :)
> 
>> I wasn't very precise ...
> 
> We've all been guilty of that. :)
> 
>> ... I am migrating a MX from an old antispam solution to rspamd, and
>> clamav is following me with quite a bit of history, including many
>> unofficial signatures, sanesecurity, yara rules, and ...
> 
> I'm totally sold on Yara rules, but the ClamAV implementation of Yara
> is appalling - it's a decade old (roughly Yara version 2) and riddled
> with crashing faults.  After trying (and failing) to get the ClamAV
> team to do something about that I abandoned the effort.  My milters
> now use genuine Yara which is vastly better.  At the moment my Yara
> scans need to start the Yara executable for every scan but I'm working
> on a Yara daemon which will be a lot more efficient.  I looked for one
> but to my surprise I failed to find anything.  Incidentally the rules
> from Sanesecurity are the only ones I've found especially effective,
> and if it were not for those I wouldn't bother with ClamAV at all.  It
> hogs a huge amount of resources for precious little reward.  Here's a
> trawl through the last four months of ClamAV logs here:
> 
> $ ls -l /.../clamav/databases/*db | wc -l
> 41
> $ grep FOUND /... |cut -d':' -f4|cut -d'.' -f1|sort|uniq -c|sort -n
>        1  Heuristics
>        1  Java
>        2  Win
>        2  winnow
>        2  Xls
>       14  MiscreantPunch
>       28  Porcupine
>      529  Sanesecurity
> 
> Note that much of this will be spam, not malware.  I'm not especially
> interested in distinguishing between the two, although I do (and send
> automated reports to the ClamAV team for malware it doesn't find).
> 
>> ... I've only ever seen clamav log infected mails, not clean
>> ones. I'll look into that to see if there is a way for it to log
>> everything is scans.
> 
> ClamAV can certainly do that, it does it here routinely.  Perhaps
> there have been changes in the API which trashed the rspamd/ClamAV
> interface?  I haven't checked, I don't use it - the interface here
> is directly between ClamAV and my milters.
> 
>> ... of course, I don't expect clamav to be an absolute bulwark ...
> 
> :)
> 
>> But I do expect clamav to check every email to detect what it can 
>> detect every time it appears.
> 
> You just have to tell it. :)
> 
>> Now, the options
>>
>>> scan_mime_parts
>>> scan_text_mime
>>> scan_image_mime
>>
>> as I understood them, were supposed to be set to true to pass every 
>> mime parts of an email separately to clamav, as opposed to the entire 
>> message sent as a whole. I conducted tests that suggested that some 
>> signatures weren't matched if the message wasn't scanned as a whole ...
> 
> It depends on the signatures of course.  Be aware that because ClamAV
> is designed to scan literally anything, for many signatures it needs
> to decide what it's scanning - in your case (and mine) for many sigs
> it probably needs to know either that it's scanning a mail message or
> that it's scanning for example a base64-encoded MIME part or whatever.
> You might experiment to see what happens if you feed different chunks
> directly to its socket, that's what I do if I'm testing ideas.  My
> milters break a message down into MIME pieces for the Yara scans, but
> send the full message both to ClamAV and to a (separate) Yara scanner.
> 
>> ... the online documentation is rather scarce.
> 
> Hmmm.
> 
>> Now, for my original question : I know that clamav scans some emails, 
>> as I have seen it in the logs. But my understanding of the "log_clean 
>> = true;" setting was that a line of log would be added for every clean 
>> clamav scan. And what I see in my logs is that the processing of some 
>> messages creates such a log line :
>>
>>> lua; clamav.lua:131: clamav: message or mime_part is clean
>>
>> some, but not all. Which is why I wonder : is rspamd only logging some 
>> clean scan, or is rspamd only scanning some messages ?
> 
> As I said to begin with, I can't answer that question directly without
> more information but you can certainly get ClamAV to log clean
> messages in its own logs, which should readily give you your answer. :)
> 
>> I hope the situation is clearer.
> 
> Thanks for the clarification.  I've now no doubt you're on top of it. :)
> 

-- 
Mickaël DEQUIDT
IFREMER - Service IRSI/RIC
Centre Ifremer Bretagne - ZI de la pointe du diable
CS 10070 - 29280 Plouzané
Tel : +33 (0)2 98 22 46 04 - Fax : +33 (0)2 98 22 46 47