[Rspamd-Announce] Rspamd 1.9.1 has been released

[Vsevolod Stakhov]

We have released Rspamd 1.9.1 today.

This release includes one potentially dangerous change: all
configuration files are now preprocessed using Jinja templates.

Hence, if you have sequences like {=/=}, or {%/%}, or {#/#}
ANYWHERE in the configuration files including even comments then you
need to take extra care when moving these configuration to the new
version! There are workarounds described above to do that. {% endraw %}

Here is the list of the most important changes in this version.


Jinja templates in the configuration

From version 1.9.1, Rspamd supports Jinja2 templates provided by Lupa
Lua library. You can read the basic syntax documnentation and the
abilities provided by these templating engines using the links above.
Rspamd itself uses a specific syntax for variable tags: {= and =}
instead of the traditional {{ and }} as these tags could mean, e.g. a
table in table in Lua.

Templating might be useful to hide some secrets from config files and
places them in environment. Rspamd automatically reads environment
variables that start from RSPAMD_ prefix and pushes it to the env
variable, e.g. RSPAMD_foo=bar comes to env.foo="bar" in templates.


New template subcommand in Rspamadm

Rspamadm has now template subcommand to apply templates engine to the
input file or files:

Options supported:

|– |-n, –no-vars | Don’t add Rspamd internal variables | | -e , –env |
Load additional environment var from specific file (name=value) | | -l ,
–lua-env | Load additional environment vars from specific file (lua
source) | | -s , –suffix | Store files with the new suffix | | -i,
–inplace | Replace input file(s) |


Changes in URLs extraction for HTML parts

Rspamd now tries to extract URLs from plain text of HTML parts.
Unfortunately, despite of being contraversal, some Email clients do that
as well. One of the notable example is Outlook. Hence, from this release
Rspamd also looks for URLs in plain HTML text.


Per user settings for mime_types plugin

Mime types plugin now supports per user settings to allow individual
black and white lists of extensions. Here is an example to increase
score for exe extensions for some specific user:

    test {
      from = "user at example.com";

      apply {
        plugins {
          mime_types = {
            bad_extensions = {
              exe = 100500,
            }
          }
        }
      }
    }

Mime types plugin now also supports reverse mapping of content type to
extension to allow processing of attachments where an exact file name is
not specified.


Better greylisting conditioning

It is now possible to disable or enable greylisting in Rspamd based on
the presence of some specific symbols. This feature allows more fine
grained greylisting control.


Bitcoin addresses validation

It is not a secret that the wave of spam and scam related to crypto
currencies has been flooding the email flows in the recent time. Rspamd
has a special rule called LEAKED_PASSWORD_SPAM to block some of the scam
types. In this version, Rspamd also checks bitcoin wallets to
distinguish them from random long strings to reduce false positives rate
significantly. It also allows to build a database of wallets used for
scam and spam.


Replies plugin validation

Replies plugin now stores the from/reply-to address when tracking
outbound messages and whitelists merely replies that come that address.
It helps to avoid replies abusing where spammers were able to catch some
legit message ids somewhere in public lists and used them in In-Reply-To
headers to dodge spam filtering in Rspamd.


List of major bug fixes

This version includes some important fixes:

-   Add crash safety for HTTP async routines
-   Clickhouse: Fix table schema upload
-   Core: Fix squeezed dependencies handling for virtual symbols
-   Finally fix default parameters parsing in actions section
-   Fix ES sending logic (restore from coroutines mess)
-   Fix finishing script for Clickhouse collection
-   Fix priority for regexp symbols registration
-   Neural: Fix training
-   Rework cached Redis logic to avoid sentinels breaking
-   SURBL: Fix regression in surbl module
-   Fix double signing in the milter


Full list of the meaningful changes

-   [Conf] Add vendor groups for symbols
-   [Feature] Add rspamadm template command
-   [Feature] Allow to add messages from settings
-   [Feature] Allow unconnected DNS servers operations
-   [Feature] Check limits after being set, migrate to uint64
-   [Feature] Greylist: Allow to disable greylisting depending on
    symbols
-   [Feature] Improve lua binary strings output
-   [Feature] Mime_types: Implement user configurable extension filters
-   [Feature] Mime_types: When no extension defined, detect it by
    content
-   [Feature] Preprocess config files using jinja templates
-   [Feature] Replies: Filter replies sender to limit whitelisting to
    direct messages
-   [Feature] Treat all tags with HREF as a potential hyperlinks
-   [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM
-   [Fix] Add crash safety for HTTP async routines
-   [Fix] Another fix for Redis sentinel
-   [Fix] Clickhouse: Fix table schema upload
-   [Fix] Core: Fix squeezed dependencies handling for virtual symbols
-   [Fix] Finally fix default parameters parsing in actions section
-   [Fix] Fix ES sending logic (restore from coroutines mess)
-   [Fix] Fix finishing script for clickhouse collection
-   [Fix] Fix priority for regexp symbols registriation
-   [Fix] Fix various issues found by PVS Studio
-   [Fix] Initialize lua debugging earlier
-   [Fix] Neural: Fix training
-   [Fix] Rework cached Redis logic to avoid sentinels breaking
-   [Fix] SURBL: Fix regression in surbl module
-   [Fix] Fix double signing in the milter
-   [Project] Add support of HTTP proxy in requests
-   [Rework] Change lua global variables registration
-   [Rework] Rework HTML content urls extraction
-   [Rework] Start rework of aliasing in Rspamd
-   [WebUI] Combine Scan and Learning into one tab
-   [WebUI] Fix symbol score input type
-   [WebUI] Show grayed out pie
-   [WebUI] Update Throughput summary values dynamically