[Rspamd-Announce] Rspamd 1.9.3 has been released

[Vsevolod Stakhov]

We have released Rspamd 1.9.3 today.

This release contains some new features and many bug fixes. There are no
incompatible changes introduced with this release to our best knowledge.

This release includes the following features and important changes.


Hashicorp Vault support

From version 1.9.3, Rspamd can use Hashicorp Vault to store and manage
DKIM keys. Vault usage provides secure and flexible storage of the
private keys that can scale and use various backends to store sensible
data (secrets).

There is a new subcommand for rspamadm utility called vault that is
intended to create/remove and securely rotate DKIM private keys using
vault.

You can read more about it using the following link:
https://rspamd.com/doc/modules/dkim_signing.html#dkim-signing-using-vault


Added least passthrough result

Some modules should set metric result as LEAST POSSIBLE result. For
example, DMARC policy failure should at least mark failed messages as
spam but it should not prevent messages from being rejected. From this
release, such modules use least policy to set actions allowing to apply
a more strict policy if needed.


Tunable memory management

From this version, Rspamd allows to manage memory policies for Lua
garbage collection allowing to fit memory/cpu constraints more flexible.
When Rspamd is built with jemalloc (e.g. in the default packages
provided by the project), it can also print detailed memory statistics
on full gc loops.

Here is an example for tuning GC in Rspamd when there are lots of free
memory available (around 1Gb per scanner process):

    # local.d/options.inc
    # http://pgl.yoyo.org/luai/i/2.10+Garbage+Collection
    lua_gc_step = 100;
    lua_gc_pause = 400;
    # number of scanned messages to perform full GC iteration
    full_gc_iters = 10000;


Improved oversigning logic

It is now possible to oversign existing only headers and ignore it if a
header is missing. It is done by changing (o) to (x).

The default list of headers signed is changed accordingly:

  Header                      Sign type
  --------------------------- ------------------------
  From                        Strictly oversign
  Sender                      Conditionally oversign
  Reply-To                    Strictly oversign
  Subject                     Strictly oversign
  Date                        Conditionally oversign
  Message-Id                  Conditionally oversign
  To                          Strictly oversign
  Cc                          Strictly oversign
  Mime-Version                Conditionally oversign
  Content-Type                Conditionally oversign
  Content-Transfer-Encoding   Conditionally oversign
  Resent-To                   Do not oversign
  Resent-Cc                   Do not oversign
  Resent-From                 Do not oversign
  Resent-Sender               Do not oversign
  Resent-Message-Id           Do not oversign
  In-Reply-To                 Conditionally oversign
  References                  Conditionally oversign
  List-Id                     Do not oversign
  List-Help                   Do not oversign
  List-Owner                  Do not oversign
  List-Unsubscribe            Do not oversign
  List-Subscribe              Do not oversign
  List-Post                   Do not oversign
  Openpgp                     Conditionally oversign
  Autocrypt                   Conditionally oversign


Important bugs fixes

Here is the list of the most important bugs fixes:

-   HTML: Fix size attribute processing - this issue caused rule
    MANY_INVISIBLE_PARTS to be improperly triggered on many HTML
    messages
-   Do not blacklist mail by SPF/DMARC for local/authed users
-   Lots of Clickhouse plugin fixes
-   Fix buffer overflow when printing small floats - this issue caused
    random crashes in WebUI reported by many users
-   Fix DoS caused by bug in glib - details in
    https://gitlab.gnome.org/GNOME/glib/issues/1775


Full list of the meaningful changes

-   [Conf] Add IP_SCORE_FREEMAIL composite rule
-   [Feature] Add cryptobox method to generate dkim keypairs
-   [Feature] Add fast hashes to lua cryptobox hash
-   [Feature] Add least passthrough results
-   [Feature] Allow oversign if exists mode
-   [Feature] Clickhouse: Modernise table initial schema
-   [Feature] Implement IUF interface for specific fast hashes
-   [Feature] Lua_util: Allow to obfuscate different fields
-   [Feature] Tune memory management in Rspamd and Lua
-   [Fix] Avoid buffer overflow when printing long lua strings
-   [Fix] Change the default oversigning headers to a more sane list
-   [Fix] Clickhouse: Do not store digest as it is not needed now
-   [Fix] Clickhouse: Fix lots of storage issues
-   [Fix] Clickhouse: Support custom actions
-   [Fix] Deny URLs where hostname is bogus
-   [Fix] Do not blacklist mail by SPF/DMARC for local/authed users
-   [Fix] Fix DoS caused by bug in glib
-   [Fix] Fix UCL parsing of the multiline strings
-   [Fix] Fix buffer overflow when printing small floats
-   [Fix] Fix init code for servers keypairs cache
-   [Fix] Fix issue with urls with no tld (e.g. IP)
-   [Fix] Fix memory in arc signing logic
-   [Fix] Fix memory leak in language detector during reloads
-   [Fix] Fix mixed case content type processing
-   [Fix] Fix processing of the ip urls in file
-   [Fix] Fix use after free
-   [Fix] HTML: Fix size attribute processing
-   [Fix] Hum, it seems that 99ff1c8 was not correct
-   [Fix] Lua_task: Fix task:get_from method
-   [Fix] Preserve fd when mapping file to scan
-   [Fix] Re-use milter_headers settings when doing arc signing
-   [Fix] Set dmarc force action as least action
-   [Fix] Switch to GMT
-   [Fix] allow PKCS7 signatures to be text/plain, too
-   [Project] Add initial version of the vault management tool
-   [Project] Add vault support for DKIM and ARC signing
-   [Project] Implement keys rotation in the vault
-   [Project] Improve dkim keys generation for vault
-   [Project] Improve keys creation in rspamadm vault
-   [Rework] Move lua_worker to a dedicated unit
-   [WebUI] Add URL fragments (#) support
-   [WebUI] Fix AJAX request URL