[Rspamd-Users] DKIM keys in Redis

[Gerald Galster]

> I run rspamd (3.4 on Debian stable, amd64) alongside opensmtpd for a

Keep this in mind if problems occur:
https://rspamd.com/downloads.html#debian-standard-repos-notes

> couple of domains. The setup is pretty vanilla, nothing crazy.
> Recently an idea started to grow on me for which I want a 'programmatic' 
> way to add DKIM keys to the rspamd configuration. Of course I could 
> append a stanza to local.d/dkim_signing.conf and bounce the daemon, but 
> [1] speaks about 'DKIM keys in Redis', which seams more appropriate.

This is overkill for a couple of domains.

https://rspamd.com/doc/modules/dkim_signing.html#principles-of-operation

"The default global configuration (fallback mode) searches for keys at
 the defined path. This path is constructed using the eSLD normalized
 domain name of the header from and the default selector defined with
 selector (dkim). For example, the search path for user at test.example.com
 would be /var/lib/rspamd/dkim/example.com.dkim.key. If a key is found,
 the message will be signed."

https://rspamd.com/doc/modules/dkim_signing.html#configuration

  # Default path to key, can include '$domain' and '$selector' variables
  path = "/var/lib/rspamd/dkim/$domain.$selector.key";
  
  # Default selector to use
  selector = "dkim";

Rspamd will look for a key in /var/lib/rspamd/dkim/$domain.dkim.key
by default.

Just create a key like:

  /usr/bin/rspamadm dkim_keygen -s dkim -b 2048 -d yourdomain.com
    -k /var/lib/rspamd/dkim/yourdomain.com.dkim.key

  /bin/chown _rspamd:_rspamd /var/lib/rspamd/dkim/yourdomain.com.dkim.key

This will save the private key in /var/lib/rspamd/dkim/...
and print the dkim._domainkey data that has to be published via DNS.

The key is a static file then, that is very easy to backup (cp, tar, rsync).

Also have a look at rspamd.log (or journalctl): when a sasl authenticated
user sends an email rspamd logs its search for a dkim key.
  

> My issue is that I do not have a good mental model of what Redis 
> _really_ is or how rspamd uses it. Yes, Redis is a KV store with a 
> network interface, sure. But when it comes to how Redis persists data or 
> should be backed-up, it gets very fuzzy for me. And, before I put data 

Redis is an in-memory key value store that periodically commits changes
to disk (rdb files). It's up to the redis configuration when that happens
and its size is also limited by configuration.

For backup you usually just copy or rsync the .rdb file, given you
don't use AOF.

You have already mentioned this:
https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/


> somewhere that I do not want to lose (like DKIM keys), I want to 
> understand the properties of the software sufficiently. This train of 
> thought also made me understand that I should perhaps have thought about 
> this earlier, since Redis is a dependency of rspamd anyways, and I 
> frankly do not actually know why.

Due to redis license changes some projects recommend valkey, a redis fork:
https://valkey.io/

Redis is a key/value store that is faster for this type of operation
than a relational database. That's why it's used in rspamd.

Think of it as an external hashmap or dict with persistence. You ask for a
key and get a value. A relational database on the other hand may have multiple
columns, foreign keys, a query planner and so on.


> It seems to me like rspamd uses Redis like a traditional database, at 
> least for some use cases, but isn't Redis way more volatile than a 
> typical relational database?  Like... Something is being written into 
> Redis and power goes away. Can I be confident data won't get lost?

This depends on the configuration, as with traditional databases.
It is possible to loose data in a powerloss situation but in most
cases this does not matter. Consider for example greylisting: you
might loose the information that a certain ip address has been
verified. All that happens is that ip needs a fresh start.
Same with reputation or Bayes, generally loosing such small amounts
of data does not matter.

https://rspamd.com/doc/modules/bayes_expiry.html#limiting-memory-usage-to-a-fixed-amount

# redis.conf
...
maxmemory 500MB
maxmemory-policy volatile-ttl

With this setting redis will evict old keys when there is no space
left. This can be considered as a desired form of "data loss".


> [2] makes me believe that at least some of the data that rspamd stores 
> in Redis should be treated with diligence.
> 
> * Should the Redis instance that runs alongside a typical rspamd 
>  instance be backed up?

It is always a good idea to have a backup in case of a hardware crash
or when you delete something by mistake.

> I tried reading about the topic, primarily [3], but that honestly 
> sparked more questions than it answered. I have no clue what the right 
> course of action is in regards to
> - Making sure that data being written into Redis is being persisted to 
>  disk
> - Properly backing up data that lives inside Redis
> 
> My current mental model of those aspects of Redis can only be described 
> as FUD.
> 
> * Is there a recommendation from the rspamd project regarding Redis 
>  settings that influence data persistence? (RDB, AOF).

See RDB advantages from the redis link you referenced:

  RDB is a very compact single-file point-in-time representation of
  your Redis data. RDB files are perfect for backups. For instance
  you may want to archive your RDB files every hour for the latest
  24 hours, and to save an RDB snapshot every day for 30 days.
  This allows you to easily restore different versions of the data
  set in case of disasters.

I don't know if there is an official recommendation, but in my
opinion RDB is a good start as it is very easy to backup. In case
your installation is very complex and performance critical, you
should read up on redis/valkey and key value stores in general.

Otherwise it seems sufficient to backup once a day or even once
a week.


> Due to my lack of sufficiently deep understanding I am considering to 
> use something that I know how to operate reliably (e.g. sqlite) as 
> source of truth for DKIM keys and inject data into Redis when the daemon 
> is bounced, if required, due to fear that Redis persistence might not 
> work as expected.

You don't need redis for DKIM keys. The difference with sqlite is
that you can copy/backup .rdb files without corruption as periodic
background commits create a new .rdb file.


> My hope is that I can get some recommendations on this mailing list so 
> that I can avoid the (probably) unnecessary additional step of classic 
> SQL database.

Rspamd does not use classic sql databases like mysql or postgres.
You can use clickhouse for metadata export, but this is a columnar
data store unlike sqlite/mysql/...


> Apologies if (at least some of) those questions should actually go to 
> the Redis people. IMHO the questions mainly relate to how Redis is being 
> used by rspamd.

Redis is mainly used as a cache and small data loss due to uncommitted
changes is usually not a big problem.

Best regards,
Gerald