[Rspamd-Users] Rspamd 1.9.0 has been released

[Vsevolod Stakhov]

We have released Rspamd 1.9.0 today.

There are various important features in this release. The vast majority
of those should not have any impact on the existing systems. However,
you are recommended to read the Upgrade Notes.

This release contains lots of improvements, reworks and bugs being
fixed. Here is a list of the most important changes in this release:


External services

Rspamd is now shipped with the external services module contributed by
Carsten Rosenberg. This module provides a generic integration with the
following services:

-   Generic ICAP protocol:
    -   ClamAV (using c-icap server and squidclamav)
    -   Sophos (via SAVDI)
    -   Symantec Protection Engine for Cloud Services
    -   Kaspersky Web Traffic Security 6.0
-   OleTools
-   DCC
-   VadeSecure

This plugin is a part of a more generic lua_scanners framework that
allows more flexible integration with different Antivirus and AntiSpam
OEM services.


New mime modify tool in Rspamadm

Rspamadm mime subcommand now allows to modify messages. This tool allows
to add or remove headers in a message, add footers in HTML and text
parts and do some subject rewriting. For example, to add a footer and
rewrite subject in a message, one can use the following command:
rspamadm mime modify --text-footer=footer.txt --html-footer=footer.html
--rewrite-header="Subject=TEST: %s".

This tool has full MIME support (including multipart messages), supports
various messages encoding and convert those to UTF8, allows to modify
both plain old messages and multipart messages with attachments. It also
properly detects and excludes GPG signed/encrypted messages.


Offline DKIM signing tool in Rspamadm

Another tool that has been added to rspamadm is sign subcommand. This
command allows to perform DKIM signing using a specific private key for
some message or messages. It can either return isolated header or modify
the message itself.

In conjunction with the previous tooling, it could be used to modify and
sign messages produced by mailing lists or some local forwarding
scripts.

Please bear in mind that this tool is available when using LuaJIT only
as it requires FFI interfaces.


HTTP Keep-Alive support

>From this version, Rspamd supports keep-alive in its HTTP Lua client.
This could be used to implement high frequent requests to some external
services and to reduce load by keeping the pool of HTTP connections
instead of opening a connection per each request.


New Lua UDP client

Rspamd now support sending of generic UDP requests as well as TCP ones.
There are various modes supported, both one-way and two-ways with
optional retransmits and timeouts. This module comes with the
documentation available.


Better unicode normalisation

This version comes with further improvements towards unicode
normalisation and detecting anomalies. Words now are sanitized from any
combining characters after translating to NFKC form.


Configuration graph utility

You can now visualise your configuration by building the include graph.
There is a new tool called rspamadm configgraph that takes configuration
and convert it to the graphviz DOT graph.


Flexible actions support

Rspamd now support any actions definitions in addition to normal ones.
You can set custom actions with thresholds or without (e.g. to set
action equal to phishing or to social). All actions should be explicitly
defined in the configuration.


New Received headers parser

We have migrated from a strict RFC compatible state machine to a custom
parser for the Received headers. This change allows to extract more data
from non-conforming Received headers used by some MTA (Exim is one of
the notable examples).


Telephone URLs support

Rspamd now parses and processes telephone URLs. That allows to build
blacklist for spam/scam/phishing phones as well as plain URLs.


Support for ED25519 signatures

Rspamd now supports dual signing and ed25519 DKIM signatures. New
ed25519 keys could be generated using rspamadm dkim_keygen tool:

    rspamadm dkim_keygen -t ed25519

CuHc4MOZYXEVH0M4WFQHL5UC2NbVJO8aq2CjGNznxm36mJPlu9GVMfq0lQI1dkeoHByqfsJgMgnCX0vFeMkjoA==
    selector._domainkey IN TXT ( "v=DKIM1; k=ed25519; "
        "p=+piT5bvRlTH6tJUCNXZHqBwcqn7CYDIJwl9LxXjJI6A=" ) ;

Ed25519 keys are much shorter than RSA providing RSA2048 security margin
in just 32 bytes for public keys. Unfortunately, these signatures are
not widely supported so dual signing is still required. We believe that
support of the modern algorithms in Rspamd would those algorithms to
spread.


Custom functions in Regexp module

Regexp module is now extended with custom Lua functions support. This
feature allows to mix fast regexp rules and custom logic of Lua rules
without explicit composite rules.


Added support of gzip archives

Rspamd now support reading filenames from GZip archives that are
surprisingly often used by some spam senders. With this feature, Rspamd
can filter some tricky scam emails that are targeting to install
backdoors or malware on users’ machines (e.g. cryptolockers).


Additional detection of types for attachments

To filter malware and bad attachments that are somehow hidden by
malicious Content-Type header, Rspamd also performs libmagic scan on
attachments to detect the real type by its content. This is useful to
detect and filter some tricky malware that utilizes bugs in popular
email clients.


Lots of major fixes

This version includes many major fixes that required massive rework to
improve stability and performance:

-   Race conditions in the maps reading code
-   Support RFC2231 encoding in headers
-   Better zero characters handling
-   Better HTML parsing and handling of the URLs
-   Coroutines are now explicitly separated from the async code to
    prevent tricky race conditions that caused crashes on certain load
-   Allow to disable/enable composite symbols
-   Lots of fixes in 7z processor
-   Detect encrypted rarv5 archives
-   Fix ETags support
-   Fix processing of NDNs of certain type
-   Improved Content-Type parsing
-   Fix deletion of the duplicate headers
-   Fix parsing of mime parts without closing boundary

We recommend to update Rspamd to this version to apply all these
features and fixes.


Full list of the meaningful changes

-   [Conf] Add missing includes
-   [Conf] Move to options
-   [Conf] Rbl: DWL is actually special whitelist
-   [Conf] Relax some uribl rules
-   [Conf] Remove abuse.ch
-   [CritFix] Html: Entities are not valid within tag params values
-   [Feature] Add rspamadm mime sign tool
-   [Feature] Add configgraph utility
-   [Feature] Add dedicated ZW spaces detection for URLs
-   [Feature] Add flag to url object when visible part is url_like
-   [Feature] Add method task:lookup_words
-   [Feature] Add pyzor support (by crosenberg)
-   [Feature] Allow to add upstream watchers to Lua API
-   [Feature] Allow to set rewrite subject pattern from settings
-   [Feature] Better escaping of unicode
-   [Feature] Clickhouse: Allow to store subject in Clickhouse
-   [Feature] Core: Add QP encoding utility
-   [Feature] Core: Add libmagic detection for all parts
-   [Feature] Core: Add support for gzip archives
-   [Feature] Core: Allow to construct scan tasks from raw data
-   [Feature] Core: Detect charset in archived files
-   [Feature] Core: Ignore and mark invisible spaces
-   [Feature] Core: Normalise zero-width spaces in urls
-   [Feature] Core: Process data urls for images
-   [Feature] Core: Relax quoted-printable encoding
-   [Feature] Core: Support RFC2231 encoding in headers
-   [Feature] Core: Support telephone URLs
-   [Feature] Core: allow to emit soft reject on task timeout
-   [Feature] DCC: Add bulkness and reputation checks to dcc
-   [Feature] Elastic: Modernize plugin
-   [Feature] Export visible part of url to lua
-   [Feature] Fuzzy_storage: add preliminary support of rate limits
-   [Feature] HTML: Specially treat data urls in HTML
-   [Feature] Implement event watchers for upstreams
-   [Feature] Implement includes tracing in Lua
-   [Feature] Improve dkim part in configwizard
-   [Feature] Lua_scanners: Add VadeSecure engine support
-   [Feature] Lua_task: Add flexible method to get specific urls
-   [Feature] Mime_types: Add MIME_BAD_UNICODE rule
-   [Feature] Mime_types: Use detected content type as well
-   [Feature] Plugins: Add preliminary version of the external services
    plugin
-   [Feature] Query sentinel on master errors
-   [Feature] Regexp: Allow local lua functions in Rspamd regexp module
-   [Feature] Rspamadm: Allow to append footers to plain messages
-   [Feature] Rspamadm: Allow to rewrite headers in messages
-   [Feature] Selectors: Add ipmask processor
-   [Feature] Settings: Allow hostname match
-   [Feature] Settings: Allow local when selecting settings
-   [Feature] Settings: Allow multiple selectors
-   [Feature] Settings: Allow to inverse conditions
-   [Feature] Support User-Agent in HTTP requests
-   [Feature] Support ed25519 dkim keys generation
-   [Feature] Try to filter bad unicode types during normalisation
-   [Feature] external_services - oletools (olefy) support
-   [Feature] lua_scanners - icap protocol support
-   [Feature] lua_scanners - spamassassin spam scanner
-   [Fix] Add filter for absurdic URLs
-   [Fix] Add some more cases for Received header
-   [Fix] Allow to disable/enable composite symbols
-   [Fix] Arc: Use a separated list of headers for arc signing
-   [Fix] Archive: Final fixes for 7z archives
-   [Fix] Clickhouse: Fix database usage
-   [Fix] Controller: Make save stats timer persistent
-   [Fix] Core: Detect encrypted rarv5 archives
-   [Fix] Core: Don’t detect language twice
-   [Fix] Core: Fix address rotation bug
-   [Fix] Core: Fix content calculations for message parts
-   [Fix] Core: Fix emails comments parsing and other issues
-   [Fix] Core: Fix etags support
-   [Fix] Core: Fix headers folding on the last token
-   [Fix] Core: Fix iso-8859-16 encoding
-   [Fix] Core: Fix log_urls flag (and encrypted logging)
-   [Fix] Core: Fix part length when dealing with boundaries
-   [Fix] Core: Fix parts distance calculations
-   [Fix] Core: Fix processing of NDNs of certain type
-   [Fix] Core: Implement logic to find some bad characters in URLs
-   [Fix] Core: treat nodes with ttl properly in lru cache
-   [Fix] Fix Content-Type parsing
-   [Fix] Fix HTTP headers signing case
-   [Fix] Fix control interface
-   [Fix] Fix deletion of the duplicate headers
-   [Fix] Fix emails filtering in emails module
-   [Fix] Fix greylisting log message and logic
-   [Fix] Fix issues with storing of the accepted addr in rspamd control
-   [Fix] Fix maps object update race condition
-   [Fix] Fix memor leaks and whitespace processing
-   [Fix] Fix processing of null bytes in headers
-   [Fix] Fix rcpt_mime and from_mime in user settings
-   [Fix] Fix rfc2047 decoding for CD headers
-   [Fix] Fix rfc2231 for Content-Disposition header
-   [Fix] Fix setting of the subject pattern in config
-   [Fix] Greylist: fix records checking
-   [Fix] HTML: Another HTML comments exception fix
-   [Fix] HTML: Another entities decoding logic fix
-   [Fix] HTML: Fix HTML comments with many dashes
-   [Fix] HTML: Fix entities in HTML attributes
-   [Fix] HTML: Fix some more SGML tags issues
-   [Fix] Ignore whitespaces at the end of value in DKIM records
-   [Fix] MID module: Fix DKIM domain matching
-   [Fix] Milter_headers: Fix remove_upstream_spam_flag and modernise
    config
-   [Fix] Mime_parser: Fix issue with parsing of the trailing garbadge
-   [Fix] Mime_parser: Fix parsing of mime parts without closing
    boundary
-   [Fix] Multimap: Fix operating with userdata
-   [Fix] Process orphaned symbols section
-   [Fix] Rdns: Fix multiple replies in fake replies
-   [Fix] Rework groups scores definitions
-   [Fix] Set proper element when reading data from Sentinel
-   [Fix] Set rspamd user to initialise supplementary groups on reload
-   [Fix] Settings: Fix selectors usage
-   [Fix] Sort data received from Sentinel to avoid constant replacing
-   [Fix] groups.conf - filename typo
-   [Fix] lua_scanner - oletools typos, logging
-   [Fix] lua_scanners - actions and symbol_fail
-   [Fix] lua_scanners - fix luacheck
-   [Fix] lua_scanners - kaspersky - response with fname
-   [Fix] lua_scanners - savapi redis prefix
-   [Fix] tests - antivirus - fprot symbols
-   [Project] Add concept of flexible actions
-   [Project] Add heuristical from parser to received parser
-   [Project] Add new flags to clickhouse, redis and elastic exporters
-   [Project] Attach new received parser
-   [Project] Fallback to callbacks from coroutines
-   [Project] Implement keep-alive support in lua_http
-   [Project] Lua_udp: Implement fully functional client
-   [Project] Plug keepalive knobs into http connection handling
-   [Project] Rspamadm: Add modify tool
-   [Rework] Convert rspamd-server to a shared library
-   [Rework] Dcc: Rework DCC plugin
-   [Rework] Enable explicit coroutines symbols
-   [Rework] Rework telephone urls parsing logic
-   [Rework] Rewrite RBL module
-   [Rework] Settings: Rework settings check
-   [Rework] Slashing: Distinguish lualibdir, pluginsdir and sharedir
-   [Rework] Unify task_timeout
-   [Rework] Use VEX instructions in assembly, relocate
-   [WebUI] Notify user if uploaded data was not learned
-   [WebUI] Remove redundant condition