[Rspamd-Users] Simple rule trouble.

[Michelle Sullivan]

> On 11 Sep 2025, at 02:04, G.W. Haywood <rspamd at jubileegroup.co.uk> wrote:
> 
> Hi there,
> 
> On Thu, 11 Sep 2025, Michelle Sullivan wrote:
> 
>> ... still runs into the issue of I have to deploy them which is
>> change control, peer-review, approval, scheduling then finally 4
>> hours of someone deploying them
> 
> Four hours seems onerous.  Do you use anything like Puppet?

I used to but not here - the infrastructure is not small.

> 
>> ... impractical for something that needs a dynamic response.
> 
> I run milters which use Yara rules to flag e.g. IP addresses as
> 'unwanted' or something to that effect.  The flags (plus a load of
> other stuff) are written to our Postgres 'connections' table which a
> cron job queries every few minutes.  When this script finds e.g. a
> flagged IP it adds an ipset DROP rule.  There are currently just over
> nine hundred rules, which can be anything from a /32 to a /8.

Not something I can use here.

> 
> Almost everything is dynamic, the only manual steps are (1) modifying
> (or possibly adding) Yara rules when nothing has flagged the unwanted
> connectioon/message before it falls through to our greylisting, and
> (2) consolidating ipset rules every now and then to reduce numbers of
> rules, for example 192.0.2.0/32,192.0.2.1/32 => 192.0.2.0/31.
> 
> Activity (2) could easily and fairly trivially be automated too.
> 
> Would something like that reduce the time you spend hidebound?

Not really, to give you an idea, and wander off-topic briefly, for everyone here…

One of four ‘regions’ a 10 minute window of the MX servers across 9am today:

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-3.png
Type: image/png
Size: 47381 bytes
Desc: not available
URL: <https://lists.rspamd.com/pipermail/users/attachments/20250911/14069559/attachment.png>
-------------- next part --------------


This sort of system needs careful control, and has (what I think is a little too onerous) overhead to deploy new configurations… auto scaling of servers to cope with traffic spikes etc.. the following graph is what made it through everything to the end of the data command in the same time window:

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-4.png
Type: image/png
Size: 48976 bytes
Desc: not available
URL: <https://lists.rspamd.com/pipermail/users/attachments/20250911/14069559/attachment-0001.png>
-------------- next part --------------

Which many here are likely to find enlightening (Andrew L won’t :) *waves*) but for those not in the million+ user levels the top graph is quite surprising when one expects to see the bottom graph.

Years ago across all the systems I was involved with you would see the bottom graph for weekdays and @SORBS I had similar DNS/RBL query graphs for years..  However, nowadays the top graph is not uncommon to see now and you’ll notice the constant noise before 9am which drops off after the 9am peak and scaling event.. it recovers a few hours later I have no idea why the spammers back off, its my team's busiest time of the day with meetings so it would be the ideal time to attack.. least eyeballs etc.  

So for those commenting that don’t have experience with million user systems and larger please be aware that there is a scaling point where you need a team to do deployments and change control and deployment of configuration changes require approvals, change control and some suggestions can look a little silly, and I apologize if that might seem condescending, I don’t intend on it being so, but there is a massive difference in how you can deploy to a 10000 user system to what you need to do for a million user system.

.. anyhow back to the issue ..

I am built in AWS, and lots of change control and teams involved so changing /etc/rspamd is something not to take lightly and as such have to have a way to wrote multi-level rules in remote maps.. currently I have multimap.conf looking at some S3 buckets for maps, but each new one requires a new deployment, whereas updating an existing (already configured one) is trivial (eg for adding a new regexp pattern for something already checked against like the subject)..  What appears not to be trivial is a multi level map which Vsevolod has already thought might be something interesting in another part of this thread and is something I really need where I can say (another real(ish) pseudo example)

If Microsoft IPs and Envelope-Domain is not hotmail.com or outlook.com and matches this regexp then score it with a X.Y score.

But I have to have a way I can add that rule a file/map that is remote hosted in an S3 bucket (https:// mappable) for example so I could (in theory) have it in production within a minute.  Currently I would need to add maps, update config (maybe an existing map or 2) but then update the composite rule which requires a 4 hour (minimum) deployment window… which of course it often too late in the party… especially if I screw it up - which I will sooner or later - and it needs to be rolled back.

Anyhow there’s where I am going with this and the actual detail that puts things into perspective where it wouldn’t have been before.

Regards,

Michelle