[Rspamd-Users] Rspamd 2.0 has been released

Wed Oct 16 06:20:57 UTC 2019

Hi Vsevolod,

thank you very much for your work!

In the message below, you mentioned that users or ratelimit module  
should read the upgrading instructions (is this it?  
https://rspamd.com/doc/migration.html#migration-to-rspamd-20 ). I read  
it and didn't find anything related to ratelimit module. Is something  
missing there?

Thank you.

azur

Citát Vsevolod Stakhov <vsevolod at rspamd.com>:

> We have released Rspamd 2.0 today!
>
> This version encompasses new versioning schema that will be used in
> future Rspamd releases: specifically, instead of the
> <major>.<minor>.<patch>, Rspamd will use just <major>.<minor> versioning
> schema. This happens because the <major> number has never been increased
> for many years and <minor> number has been used as a real version indicator.
>
> Upgrade notes
>
> There are various important features in this release. The vast majority
> of those should not have any visible impact on the existing systems.
> However, you are recommended to read the Upgrade Notes.
>
> The main potential source of incompatibilities is the deprecation of the
> surbl and emails modules that have been replaced with rbl module. The
> default Bayes backend is also changed to Redis now while the Sqlite
> backend is now marked as deprecated and is not recommended for use.
> ip_score, neural and ratelimit modules users are strongly advised to
> read the upgrading notes!
>
>
> Packages support
>
> In this version of Rspamd, we have stopped support of the following OS
> variants:
>
> -   Ubuntu trusty (reached EOL)
> -   Centos 6 (almost reached EOL)
>
> We have added Centos 8 packages instead.
>
> As usually, Rspamd project strongly recommends NOT TO USE the packages
> that are provided by 3rd parties, including your own LINUX DISTRIBUTION.
> These packages are usually out-of-date, built incorrectly and
> accordingly, they are not supported by Rspamd project. Please use the
> official packages only. FreeBSD ports are considered official packages
> as they are supported by Rspamd project directly (well, strictly
> speaking by myself).
>
> Here is the list of the most important changes in this release.
>
>
> Libevent has been replaced with bundled libev
>
> After many years of using the libevent library Rspamd switched to libev
> library. The main reason was performance and control: there were many
> libevent versions shipped with various supported platforms and many of
> those lacked important features, such as inotify support for Linux.
> Switching to libev allowed us to simplify the code, improve signals
> handling, improve timeouts handling and deal with file maps changes
> instantly due to inotify.
>
>
> Torch has been dropped from Rspamd
>
> Lua torch has served as a powerful engine for ML and neural networks in
> Rspamd for quite a long time. However, it is no longer maintained or
> updated and its support has proven to be a nightmare. There were also
> important bugs that could not be fixed due to the code complexity. From
> version 2.0, Rspamd adopted kann library that is much more friendly for
> embedding and provides very convenient interfaces that are now exported
> via Lua.
>
>
> RBL module improvements and replacement of the SURBL and Emails module
>
> RBL module has replaced both emails and surbl modules unifying all
> Runtime Black Lists checks in a single place. It has added new RBL
> types, such as selectors, and the simplified extending of the existing
> rules to more powerful ones.
>
> Emails rules with maps instead of DNS RBLs are NO LONGER SUPPORTED.
> Please use multimap with selectors instead.
>
>
> New Lua Magic library
>
> For file types detection, Rspamd now uses an own implementation of
> detection library based on Lua and Hyperscan (where possible) instead of
> libmagic. There are 4 major concerns for that:
>
> -   Libmagic is a generic library that can easily detect pdp11 a.out
>     format but can fail in docx detection surprisingly often
> -   We need performance and libmagic is not about performance at all
> -   We want to add new detection heuristics instead of relying on 3rd
>     party strict rules
> -   Libmagic API is not very suitable for us
>
> With the new library, Rspamd can detect part types in just a couple of
> microseconds and find the vast majority of the _interesting_ things,
> such as executables, archives, images, html and so on and so forth.
>
>
> Neural module rework
>
> Neural networks module has been almost totally rewritten to support KANN
> library and symbols profiles. Now, Rspamd will not reset neural network
> on each individual symbol change - it will try to use the most
> appropriate network instead. Many issues with neural learning dead locks
> have also been addressed.
>
>
> Clickhouse module improvement
>
> -   Added LowCardinality fields to improve storage requirements
> -   Fixed retention code
> -   Significantly optimized memory usage by using userdata instead of
>     interned strings
>
>
> Multimap module
>
> Various new features, including maps combinations and
> dependent maps(https://rspamd.com/doc/modules/multimap.html#dependent-maps).
>
>
> Maillist module
>
> Improved mailing lists detection and reworked detection heuristic.
>
>
> Heartbeats support
>
> Rspamd workers now send heartbeat events to the main process. In turn,
> the main process can now kill hanged workers if a reasonable amount of
> heartbeats have been lost. This feature is not enabled by default for
> now.
>
>
> Lua scanners improvements
>
> There are lots of additions in lua scanners. Many of those have been
> contributed by Carsten Rosenberg from HeinleinSupport.
>
> New antivirus engines support:
>
> -   Kaspersky ScanEngine: https://www.kaspersky.com/scan-engine
> -   Trend Micro IWSVA support via icap (by @c-rosenberg)
> -   F-Secure Internet Gatekeeper via icap (by @c-rosenberg)
>
> New external scanners:
>
> -   Razor support (by @c-rosenberg)
> -   Better oletools support (by @c-rosenberg)
> -   P0F support as a separate module (by Denis Paavilainen -
>     @denpamusic)
>
>
> Mime modifications
>
> From version 2.0, Rspamd allows modifying messages via Lua API methods.
> This support required massive rework of the internal structures and have
> been tested by Migadu. These functions are implemented in the lua_mime
> library.
>
>
> Users settings improvements
>
> Rspamd now treats settings differently if they are set via Settings-Id:
> there are certain performance benefits and better logging in all
> modules. It is also possible to bind rules explicitly to certain
> settings id allowing to separate mail processing flows more efficiently.
>
>
> Upstreams library improvements
>
> -   Added lazy resolving of the upstreams
> -   Added SRV upstreams to resolve SRV records for both names, ports,
>     and priorities (e.g. by using Hashicorp Konsul DNS)
> -   Use random strings for monitoring sanity
>
>
> Performance improvements
>
> -   Improved base64 decoding for typical outputs
> -   Langdet: Limit number of stop words to be checked
> -   Added sanity limit for task:get_urls() method to avoid Lua memory
>     blow
> -   Maps: Allow caching for complex maps
> -   Settings fast path have been added
> -   Lua core: use lightuserdata to index classes to avoid strings
>     interning
> -   HTTP(s) keep-alive support has been added
>
>
> Rules and other improvements
>
> -   Added BITCOIN_ADDR symbol to allow custom composite rules creation
>     to block scam campaigns
> -   Support Litecoin addresses
> -   Implement syntax highlighting for Lua
> -   Allow execution of async events when hs compiles regexps
> -   Bayes expiry: eliminate default expiration mode (use lazy mode all
>     the time)
> -   Eliminate lua_squeeze as it has shown no improvements
> -   Drop url tags
> -   Eliminate virtual scan time as it is useless
> -   Use replxx instead of linenoise
> -   Added SSL/STARTTLS support to lua_tcp library
> -   Implemented SSL graceful closing
>
> This version of Rspamd contains a number of other minor and major
> improvements and fixes compared to the 1.9 branch. This includes some
> bugs that were fixed in 2.0 and that could cause certain issues, hangs
> or crashes with certain emails.
> --
> Users mailing list
> Users at lists.rspamd.com
> https://lists.rspamd.com/mailman/listinfo/users