[Rspamd-Announce] Rspamd 2.3 has been released

[Vsevolod Stakhov]

We have released Rspamd 2.3 today.

This release has various improvements, pdf parsing support and new SPF
plugin. Numerous of the bug fixes, including some critical ones have
also been applied during this release cycle.

Here is the list of the most important changes.


Lua content library

This library is a part of content scanning project that enables Rspamd
to process content formats that are commonly met in email. In this
release, we have added some preliminary PDF files support.

Rspamd supports the following features so far:

-   URLs extraction from PDF files
-   JavaScript extraction from actions
-   Using of fuzzy hashes storage to store suspicious js extracted from
    PDF files
-   Couple of rules related to bad PDF files (e.g. PDF_ENCRYPTED)

This library is written in pure Lua, using LPEG and Hyperscan under the
hood to provide memory safety and high processing speed.


SPF plugin revamped

SPF plugin has been one of the oldest plugins in Rspamd. Unfortunately
due to the fact that it has been written in C, it was very hard to add
new functions to this plugin nor maintain it in the consistent state. In
this release, SPF plugin has been rewritten in Lua providing
configuration compatible mode with some of new features, such as support
of the external relay handling


Slow rules protection

Rspamd will now try to check async timers and events when it notices
that some rule takes too much time. This will allow to break dead rules
to hold processing for more than task timeout time. In future, we plan
to extend task timeout relation for all internal rule timers.


Other features

Here is a list of other important but not categorised features:

-   Allow milter code to deal with multiple headers
-   Antivirus : Add Avast support
-   Dkim_signing : Allow to sign via milter_headers
-   Send quit command to Redis
-   Speed up is_ascii function
-   Improve memory pool allocation routines and add memory debugging


Important bug fixes

-   Critical fix: fix html entities decoding. This bug could cause
    incorrect URLs being extracted from messages.
-   Critical fix: fix re cache when mix of pcre and hyperscan is used.
    This bug could cause random failures when scanning regular
    expressions that could not be evaluated nor approximated to
    hyperscan.
-   Fix arc seal validation
-   Fix base tag processing according to stupid HTML renderer behaviour
-   Fix dealing with \0 in ucl strings and JSON
-   Fix gpg parts misdetection
-   Fix ignored symbols exporting (e.g. to ClickHouse)
-   Fix processing of numeric url’s
-   Fix processing of the closed tcp connections
-   Fix mixed charset rule for some languages (e.g. Czech)
-   Fix mixing of the IPv4 and IPv6 addresses in Radix maps
-   Fix soft hypen processing
-   Fix O(N^2) algorithm when comparing recipients
-   Various stability improvements when dealing with large or specially
    crafted messages


Full list of the meaningful changes

-   [Conf] SPF is no longer a C module
-   [Conf] Update spamtrap map path example
-   [CritFix] Fix html entities decoding
-   [CritFix] Fix re cache when mix of pcre and hyperscan is used
-   [Feature] Allow milter code to deal with multiple headers
-   [Feature] Antivirus: Add avast support
-   [Feature] Dkim_signing: Allow to sign via milter_headers
-   [Feature] Implement content hashes
-   [Feature] Lua_text: Add regexp split iterator method
-   [Feature] Lua_text: Implement flattening of the input tables
-   [Feature] Send quit command to Redis
-   [Feature] Speed up is_ascii function
-   [Feature] Spf: Add external_relay option
-   [Fix] Avoid double escaping
-   [Fix] Fix O(N^2) algorithm
-   [Fix] Fix arc seal validation
-   [Fix] Fix base tag processing according to stupid HTML renderer
    behaviour
-   [Fix] Fix dealing with \0 in ucl strings and JSON
-   [Fix] Fix gpg parts misdetection
-   [Fix] Fix ignored symbols exporting
-   [Fix] Fix processing of numeric url’s
-   [Fix] Fix processing of the closed tcp connections
-   [Fix] Fix regexp type check for pcre2
-   [Fix] Fix urls encode function
-   [Fix] Fix urls shifting when doing decode to include separators
-   [Fix] Fix white on white rule and add is_leaf flag
-   [Fix] Further fixes in charset detection
-   [Fix] Ignore diacritics in chartable module for specific languages
-   [Fix] Limit size of symbols options by max_opts_len option
-   [Fix] More fixes in html tag content calculations
-   [Fix] Plug memory leak in fuzzy storage
-   [Fix] Process high priority settings even if settings/id has been
    specified
-   [Fix] Select a different upstream on last retransmit
-   [Fix] Treat soft hyphen as zero width space
-   [Fix] Try harder to watch the lifetime of the key_stat
-   [Fix] Use ipv6-mapped-ipv4 addresses in radix trie
-   [Project] Add logic to break execution when processing symbols*
-   [Project] Add methods to set specific content for mime parts from
    Lua
-   [Project] Lua_content: support PDF files
-   [Project] Move dns_tool to using of the rspamd_spf from FFI module
-   [Project] Preliminary SPF plugin in Lua
-   [Project] Show debug stat for memory pool
-   [Project] Some rework about specific data that is now tagged
-   [Project] Start reworking of the mempool structure
-   [Rework] Allow to add userdata as symbols options
-   [Rework] Change mime part specifics handling
-   [Rework] Move LRU SPF cache from spf plugin
-   [Rework] Rework HTML tags content attachment
-   [Rework] Rework options hash structure
-   [Rework] Start lua_content library
-   [Rework] Stop using of uthash for http headers
-   [Rework] Use faster hashing approach for memory pools variables
-   [Rules] Add PDF related rules