[Rspamd-Users] Things I don't understand yet

Thu Feb 15 11:16:50 UTC 2024

Hello G.W. Haywood,
Thank you for your detailed help. That gave me some food for thought.

During the night I completely removed Rspamd from my system and 
reinstalled it and also emptied redis. I have been using the Rspamd 
repository from the beginning as you described. So I'm up to date.
After reinstalling I have no errors through rspamadm configtest. Then I 
only set up the bare essentials. User account for the web frontend, 
Redis, ClamAV connection. And then I watched what happened for a day. 
Around 25,000 emails went through Rspamd and around 20 were “false 
positives”. Which isn't actually that bad. It was just the usual 
suspects who always manage to get through.

Then I first attached my whitelist via multimap, as some important 
channels had values of +1-3 and were close to the spam limit.
Keep watching...
Then the blacklist. I was now able to pull out the usual suspects.
After 2-3 days now I notice that sometimes the multimap is not taken 
into account at all. So spam arrives which is then rated as 2 and gets 
through, but the header doesn't show me a test for my multimap blacklist 
that would have prevented it. I can't yet see a pattern when the 
multimap is ignored. Could it be that I first check a blacklist and then 
the whitelist? Is there even an order?
What I observe is that the number of checks that I display in the header 
of the email always varies and has no logic. Sometimes RBL's work and 
then only my blacklist and the RBL's don't. I don't see any connection yet.
But I will continue to monitor this.
Trie and error

Thanks
Christian

German Version

Hallo G.W. Haywood,
Danke für Deine ausführliche Hilfe. Das hat mir einige Denkanstöße gegeben.

In der Nacht habe ich Rspamd komplett von meinem System entfernt und neu 
installiert und auch redis gelleert. Ich habe schon von Anfang an die 
Repository von Rspamd verwendet wie Du beschrieben hast. Ich bin also 
auf dem neusten Stand.
Nach der Neuinstallation habe ich durch rspamadm configtest keine 
Fehler. Dann habe ich nur das notwendigste eingerichtet. Benutzeraccount 
für das Webfrontend, Redis,ClamAV Verbindung. Und dann beobachtete ich 
einen Tag was passiert. Ca. 25000 Emails gingen durch Rspamd und ca. 20 
waren "false positive". Was eigentlich nicht so schlecht ist. Es waren 
nur die Üblichen verdächtigen, die es immer schaffen durchzukommen.

Dann habe ich erst mal meine Whiteliste über multimap eingehängt, da 
einige wichtige sender werte von +1-3 hatten und knapp an der Spam 
Grenze waren.
Weiter beobachten ..
Dann die Blacklist. Die üblichen verdächtigen konnte ich jetzt auch 
rausziehen.
Nach jetzt 2-3 Tagen stelle ich doch fest, das manchmal die multimap gar 
nicht beachtet wird. Es kommt also Spam an die dann aber mit 2 bewertet 
wird und durchkommt, aber im Header wird mir kein Test auf meine 
multimap Blacklist angezeigt die es verhindert hätte. Ich kann noch kein 
Muster erkennen, wann die multimap ignoriert wird. Kann es dran liegen 
das ich zuerst eine blacklist prüfe und dann die whiteliste. Gibt es da 
überhaupt eine Reihenfolge.
Was ich beobachte ist das die Anzahl der Prüfungen, die ich mir im 
header der Mail anzeigen lasse, immer variiert und keine Logig hat. 
Manchmal greifen RBL´s und dann aber nur meine blacklist und die RBL´s 
nicht. Da sehe ich noch keinen Zusammenhang.
Ich werde das aber weiter beobachten.
Trie and Error

Danke
Christian

Am 13.02.2024 um 13:19 schrieb G.W. Haywood:
> Hello again,
> 
> On Mon, 12 Feb 2024, christian via Users wrote:
>> Am 12.02.2024 um 14:18 schrieb G.W. Haywood:
>>> On Mon, 12 Feb 2024, christian via Users wrote:
>>>
>>>> ... no evaluation is made based on my multimap ...
>>>
>>> Here's something I found useful to try to make sense of things:
>>>
>>> https://www.0xf8.org/2018/05/an-alternative-introduction-to-rspamd-configuration-modules/
>>>
>>> Just a guess, it might be an entry triggered in a whitelist or for
>>> example mid, both of which you probably have enabled.
>>>
>>> ... try ... rspamadm configtest
>>
>> My rspamadm configtest gives the following:
>>
>> rspamadm configtest
>> duplicate symbol: SUBJ_ALL_CAPS, skip registering
>> unknown type of attribute enabled for regexp module
>> cannot find dependency on symbol IS_IN_WHITELIST for symbol 
>> FORCE_ACTION_MY_WHITELIST
>> cannot find dependency on symbol FPROT_VIRUS for symbol 
>> FORCE_ACTION_MY_WHITELIST
>> syntax OK
>>
>> But unfortunately I can't do anything with the information.
>>
>> What's more, I haven't changed anything in the symbols displayed.
> 
> With a configuration as complex as that of rspamd, in my view it's
> absolutely essential to be very methodical when making any changes.
> 
> Do you have a record of all the changes that you made?  Do you have a
> copy of your original configuration before you made any changes?  One
> of the tools I use a great deal for this sort of work is 'diff'.  It
> is, I suppose, really intended for people writing code, but it makes
> it easy to compare huge sets of configuration options with changes in
> files which hap-hazardly distributed within a large directory tree.
> 
> To test the latest installation instructions for rspamd which are
> found on the rspamd Website, I followed them.  On a box which runs
> Debian 'Bullseye', I ran the script below:
> 
> 8<----------------------------------------------------------------------
> #!/bin/bash
> apt-get install -y lsb-release wget gpg
> CODENAME=`lsb_release -c -s`
> mkdir -p /etc/apt/keyrings
> wget -O- https://rspamd.com/apt-stable/gpg.key | gpg --dearmor | tee 
> /etc/apt/keyrings/rspamd.gpg > /dev/null
> echo "deb [signed-by=/etc/apt/keyrings/rspamd.gpg] 
> http://rspamd.com/apt-stable/ $CODENAME main" | tee 
> /etc/apt/sources.list.d/rspamd.list
> echo "deb-src [signed-by=/etc/apt/keyrings/rspamd.gpg] 
> http://rspamd.com/apt-stable/ $CODENAME main"  | tee -a 
> /etc/apt/sources.list.d/rspamd.list
> apt-get update
> apt-get --no-install-recommends install rspamd
> 8<----------------------------------------------------------------------
> 
> It only took a minute or so, and ran to completion with no issues.
> 
> The installation script is more or less according to the instructions at
> 
> https://rspamd.com/downloads.html
> 
> but I removed 'sudo' because I ran the script as root.
> 
> Below is what happened with rspamd newly installed using the script:
> 
> 8<----------------------------------------------------------------------
> # rspamadm --version
> Rspamadm 3.8.1
> # rspamadm configtest
> syntax OK
> # ls -l /usr/share/rspamd/
> total 264
> -rw-r--r--  1 root root 242327 Jan 25 19:10 effective_tld_names.dat
> drwxr-xr-x  2 root root   4096 Feb 13 10:45 elastic
> drwxr-xr-x  2 root root   4096 Feb 13 10:45 languages
> drwxr-xr-x 10 root root   4096 Feb 13 10:45 lualib
> drwxr-xr-x  2 root root   4096 Feb 13 10:45 plugins
> drwxr-xr-x  4 root root   4096 Feb 13 10:45 rules
> drwxr-xr-x  6 root root   4096 Feb 13 10:45 www
> # ls -l /etc/rspamd/
> total 96
> -rw-r--r-- 1 root root 1213 Jan 25 19:10 actions.conf
> -rw-r--r-- 1 root root  365 Jan 25 19:10 cgp.inc
> -rw-r--r-- 1 root root 1318 Jan 25 19:10 common.conf
> -rw-r--r-- 1 root root 7614 Jan 25 19:10 composites.conf
> -rw-r--r-- 1 root root 5154 Jan 25 19:10 groups.conf
> -rw-r--r-- 1 root root  874 Jan 25 19:10 lang_detection.inc
> drwxr-xr-x 2 root root 4096 Jan 25 19:31 local.d
> -rw-r--r-- 1 root root 1186 Jan 25 19:10 logging.inc
> drwxr-xr-x 2 root root 4096 Feb 13 10:45 maps.d
> -rw-r--r-- 1 root root  921 Jan 25 19:10 metrics.conf
> -rw-r--r-- 1 root root  703 Jan 25 19:10 modules.conf
> drwxr-xr-x 2 root root 4096 Feb 13 10:45 modules.d
> -rw-r--r-- 1 root root 2020 Jan 25 19:10 options.inc
> drwxr-xr-x 2 root root 4096 Jan 25 19:31 override.d
> -rw-r--r-- 1 root root 2787 Jan 25 19:10 rspamd.conf
> drwxr-xr-x 2 root root 4096 Feb 13 10:45 scores.d
> -rw-r--r-- 1 root root 1799 Jan 25 19:10 settings.conf
> -rw-r--r-- 1 root root 2169 Jan 25 19:10 statistic.conf
> -rw-r--r-- 1 root root  618 Jan 25 19:10 worker-controller.inc
> -rw-r--r-- 1 root root  654 Jan 25 19:10 worker-fuzzy.inc
> -rw-r--r-- 1 root root  525 Jan 25 19:10 worker-normal.inc
> -rw-r--r-- 1 root root 1363 Jan 25 19:10 worker-proxy.inc
> 8<----------------------------------------------------------------------
> 
> As you can see the files all have a relatively recent timestamp, which
> I find comforting when I do a new installtaion.  Directory timestamps
> in this case are when they were created, that's not important here but
> if I have trouble finding changes in more or less anything I'll often
> look for files/directories in a directory tree with recent timestamps.
> 
> It's clear that you've done things with which the rspamd configuration
> test is a little unhappy, but it _does_ say it's 'OK' so I think it
> will probably be working as designed.  Whether or not it's working as
> you intend is another matter.  You could always try the stricter test
> according to the 'man' page - this is the result on my new config:
> 
> # rspamadm --var=DBDIR=/tmp configtest -c /etc/rspamd/rspamd.conf -s
> syntax OK
> 
> If it were my system I think I'd want to be sure that all the warnings
> were gone before I'd be happy.  Perhaps you could back out the changes
> which you've made one-by-one until the output is silenced.  If you do
> not have a record of the changes you could start with a fresh config.
> Document each and every change you make, but also (1) test it and (2)
> dump it to a file each time you change it, with something like
> 
> # rspamadm configdump > ~/rspamd.configdump.$(date -Iseconds)
> 
> so you always have a record of a sane configuration if you break it.
> Unfortunately comparing the outputs of 'configdump' taken at different
> times might not be very helpful; the order in which the various parts
> are output seems to be non-deterministic.
> 
> To address the issue of some mails being passed through without being
> processed by some of the modules I think maybe you should investigate
> the parts of the configuration which implement whitelisting, but *not*
> before you're happy that you know exactly what changes you have made
> to the configuration and exactly what you expect these changes to do.
> 
> If you still can't figure out why you aren't getting the results you
> expect, you can post to this list the changes which you have made (or
> as I said earlier the full configuration dump).
> 
> If you have an example mail which was blocked by your configuration,
> (and should have been) and one which you feel is similar but was not
> (and should have been) perhaps you can put them somewhere where we
> can see them and try to do some guesswork.
> 
>> If I understand correctly, Rspamd's basic settings are fetched from
>> /usr/share/rspamd. I can then adjust this under /etc/rspamd/local.d.
>> Some like multimap have no basic settings and are created themselves
>> and read by rspamd.
> 
> Depending on how you look at things, fortunately or unfortunately the
> paths for the configuration directories are themselves configurable.
> That means the exact locations can be different in different systems
> and the single biggest factors which determine the locations are the
> way you installed rspamd and from where you downloaded the package.
> 
> If you installed as per the instruction on the rspamd Website your
> understanding is correct.  I do not know what might happen if you
> installed using packages from your Linux distribution, but if for
> example you used the Debian package with just
> 
> apt-get install rspamd
> 
> there's a good chance that things will be four or five years out of
> date and horribly broken.
> 
>> As a non-English speaker, I find it quite difficult to use the
>> docs. Excuse me.
> 
> Your English is good but the documentation on the rspamd Website is I
> fear not ideal for a new user.  That's why I pointed to an alternative
> which, although it is somewhat out of date, tries a bit harder to make
> the configuration make sense to someone new.  The array of options in
> the rspamd configuration is immense and it's too easy to find yourself
> lost in the woods.
> 