From vsevolod at rspamd.com Tue Mar 12 16:02:24 2019 From: vsevolod at rspamd.com (Vsevolod Stakhov) Date: Tue, 12 Mar 2019 16:02:24 +0000 Subject: [Rspamd-Announce] Rspamd 1.9.0 has been released Message-ID: <61776126-540b-de39-39ae-44a43f25340c@rspamd.com> We have released Rspamd 1.9.0 today. There are various important features in this release. The vast majority of those should not have any impact on the existing systems. However, you are recommended to read the Upgrade Notes. This release contains lots of improvements, reworks and bugs being fixed. Here is a list of the most important changes in this release: External services Rspamd is now shipped with the external services module contributed by Carsten Rosenberg. This module provides a generic integration with the following services: - Generic ICAP protocol: - ClamAV (using c-icap server and squidclamav) - Sophos (via SAVDI) - Symantec Protection Engine for Cloud Services - Kaspersky Web Traffic Security 6.0 - OleTools - DCC - VadeSecure This plugin is a part of a more generic lua_scanners framework that allows more flexible integration with different Antivirus and AntiSpam OEM services. New mime modify tool in Rspamadm Rspamadm mime subcommand now allows to modify messages. This tool allows to add or remove headers in a message, add footers in HTML and text parts and do some subject rewriting. For example, to add a footer and rewrite subject in a message, one can use the following command: rspamadm mime modify --text-footer=footer.txt --html-footer=footer.html --rewrite-header="Subject=TEST: %s". This tool has full MIME support (including multipart messages), supports various messages encoding and convert those to UTF8, allows to modify both plain old messages and multipart messages with attachments. It also properly detects and excludes GPG signed/encrypted messages. Offline DKIM signing tool in Rspamadm Another tool that has been added to rspamadm is sign subcommand. This command allows to perform DKIM signing using a specific private key for some message or messages. It can either return isolated header or modify the message itself. In conjunction with the previous tooling, it could be used to modify and sign messages produced by mailing lists or some local forwarding scripts. Please bear in mind that this tool is available when using LuaJIT only as it requires FFI interfaces. HTTP Keep-Alive support From this version, Rspamd supports keep-alive in its HTTP Lua client. This could be used to implement high frequent requests to some external services and to reduce load by keeping the pool of HTTP connections instead of opening a connection per each request. New Lua UDP client Rspamd now support sending of generic UDP requests as well as TCP ones. There are various modes supported, both one-way and two-ways with optional retransmits and timeouts. This module comes with the documentation available. Better unicode normalisation This version comes with further improvements towards unicode normalisation and detecting anomalies. Words now are sanitized from any combining characters after translating to NFKC form. Configuration graph utility You can now visualise your configuration by building the include graph. There is a new tool called rspamadm configgraph that takes configuration and convert it to the graphviz DOT graph. Flexible actions support Rspamd now support any actions definitions in addition to normal ones. You can set custom actions with thresholds or without (e.g.?to set action equal to phishing or to social). All actions should be explicitly defined in the configuration. New Received headers parser We have migrated from a strict RFC compatible state machine to a custom parser for the Received headers. This change allows to extract more data from non-conforming Received headers used by some MTA (Exim is one of the notable examples). Telephone URLs support Rspamd now parses and processes telephone URLs. That allows to build blacklist for spam/scam/phishing phones as well as plain URLs. Support for ED25519 signatures Rspamd now supports dual signing and ed25519 DKIM signatures. New ed25519 keys could be generated using rspamadm dkim_keygen tool: rspamadm dkim_keygen -t ed25519 CuHc4MOZYXEVH0M4WFQHL5UC2NbVJO8aq2CjGNznxm36mJPlu9GVMfq0lQI1dkeoHByqfsJgMgnCX0vFeMkjoA== selector._domainkey IN TXT ( "v=DKIM1; k=ed25519; " "p=+piT5bvRlTH6tJUCNXZHqBwcqn7CYDIJwl9LxXjJI6A=" ) ; Ed25519 keys are much shorter than RSA providing RSA2048 security margin in just 32 bytes for public keys. Unfortunately, these signatures are not widely supported so dual signing is still required. We believe that support of the modern algorithms in Rspamd would those algorithms to spread. Custom functions in Regexp module Regexp module is now extended with custom Lua functions support. This feature allows to mix fast regexp rules and custom logic of Lua rules without explicit composite rules. Added support of gzip archives Rspamd now support reading filenames from GZip archives that are surprisingly often used by some spam senders. With this feature, Rspamd can filter some tricky scam emails that are targeting to install backdoors or malware on users? machines (e.g.?cryptolockers). Additional detection of types for attachments To filter malware and bad attachments that are somehow hidden by malicious Content-Type header, Rspamd also performs libmagic scan on attachments to detect the real type by its content. This is useful to detect and filter some tricky malware that utilizes bugs in popular email clients. Lots of major fixes This version includes many major fixes that required massive rework to improve stability and performance: - Race conditions in the maps reading code - Support RFC2231 encoding in headers - Better zero characters handling - Better HTML parsing and handling of the URLs - Coroutines are now explicitly separated from the async code to prevent tricky race conditions that caused crashes on certain load - Allow to disable/enable composite symbols - Lots of fixes in 7z processor - Detect encrypted rarv5 archives - Fix ETags support - Fix processing of NDNs of certain type - Improved Content-Type parsing - Fix deletion of the duplicate headers - Fix parsing of mime parts without closing boundary We recommend to update Rspamd to this version to apply all these features and fixes. Full list of the meaningful changes - [Conf] Add missing includes - [Conf] Move to options - [Conf] Rbl: DWL is actually special whitelist - [Conf] Relax some uribl rules - [Conf] Remove abuse.ch - [CritFix] Html: Entities are not valid within tag params values - [Feature] Add rspamadm mime sign tool - [Feature] Add configgraph utility - [Feature] Add dedicated ZW spaces detection for URLs - [Feature] Add flag to url object when visible part is url_like - [Feature] Add method task:lookup_words - [Feature] Add pyzor support (by crosenberg) - [Feature] Allow to add upstream watchers to Lua API - [Feature] Allow to set rewrite subject pattern from settings - [Feature] Better escaping of unicode - [Feature] Clickhouse: Allow to store subject in Clickhouse - [Feature] Core: Add QP encoding utility - [Feature] Core: Add libmagic detection for all parts - [Feature] Core: Add support for gzip archives - [Feature] Core: Allow to construct scan tasks from raw data - [Feature] Core: Detect charset in archived files - [Feature] Core: Ignore and mark invisible spaces - [Feature] Core: Normalise zero-width spaces in urls - [Feature] Core: Process data urls for images - [Feature] Core: Relax quoted-printable encoding - [Feature] Core: Support RFC2231 encoding in headers - [Feature] Core: Support telephone URLs - [Feature] Core: allow to emit soft reject on task timeout - [Feature] DCC: Add bulkness and reputation checks to dcc - [Feature] Elastic: Modernize plugin - [Feature] Export visible part of url to lua - [Feature] Fuzzy_storage: add preliminary support of rate limits - [Feature] HTML: Specially treat data urls in HTML - [Feature] Implement event watchers for upstreams - [Feature] Implement includes tracing in Lua - [Feature] Improve dkim part in configwizard - [Feature] Lua_scanners: Add VadeSecure engine support - [Feature] Lua_task: Add flexible method to get specific urls - [Feature] Mime_types: Add MIME_BAD_UNICODE rule - [Feature] Mime_types: Use detected content type as well - [Feature] Plugins: Add preliminary version of the external services plugin - [Feature] Query sentinel on master errors - [Feature] Regexp: Allow local lua functions in Rspamd regexp module - [Feature] Rspamadm: Allow to append footers to plain messages - [Feature] Rspamadm: Allow to rewrite headers in messages - [Feature] Selectors: Add ipmask processor - [Feature] Settings: Allow hostname match - [Feature] Settings: Allow local when selecting settings - [Feature] Settings: Allow multiple selectors - [Feature] Settings: Allow to inverse conditions - [Feature] Support User-Agent in HTTP requests - [Feature] Support ed25519 dkim keys generation - [Feature] Try to filter bad unicode types during normalisation - [Feature] external_services - oletools (olefy) support - [Feature] lua_scanners - icap protocol support - [Feature] lua_scanners - spamassassin spam scanner - [Fix] Add filter for absurdic URLs - [Fix] Add some more cases for Received header - [Fix] Allow to disable/enable composite symbols - [Fix] Arc: Use a separated list of headers for arc signing - [Fix] Archive: Final fixes for 7z archives - [Fix] Clickhouse: Fix database usage - [Fix] Controller: Make save stats timer persistent - [Fix] Core: Detect encrypted rarv5 archives - [Fix] Core: Don?t detect language twice - [Fix] Core: Fix address rotation bug - [Fix] Core: Fix content calculations for message parts - [Fix] Core: Fix emails comments parsing and other issues - [Fix] Core: Fix etags support - [Fix] Core: Fix headers folding on the last token - [Fix] Core: Fix iso-8859-16 encoding - [Fix] Core: Fix log_urls flag (and encrypted logging) - [Fix] Core: Fix part length when dealing with boundaries - [Fix] Core: Fix parts distance calculations - [Fix] Core: Fix processing of NDNs of certain type - [Fix] Core: Implement logic to find some bad characters in URLs - [Fix] Core: treat nodes with ttl properly in lru cache - [Fix] Fix Content-Type parsing - [Fix] Fix HTTP headers signing case - [Fix] Fix control interface - [Fix] Fix deletion of the duplicate headers - [Fix] Fix emails filtering in emails module - [Fix] Fix greylisting log message and logic - [Fix] Fix issues with storing of the accepted addr in rspamd control - [Fix] Fix maps object update race condition - [Fix] Fix memor leaks and whitespace processing - [Fix] Fix processing of null bytes in headers - [Fix] Fix rcpt_mime and from_mime in user settings - [Fix] Fix rfc2047 decoding for CD headers - [Fix] Fix rfc2231 for Content-Disposition header - [Fix] Fix setting of the subject pattern in config - [Fix] Greylist: fix records checking - [Fix] HTML: Another HTML comments exception fix - [Fix] HTML: Another entities decoding logic fix - [Fix] HTML: Fix HTML comments with many dashes - [Fix] HTML: Fix entities in HTML attributes - [Fix] HTML: Fix some more SGML tags issues - [Fix] Ignore whitespaces at the end of value in DKIM records - [Fix] MID module: Fix DKIM domain matching - [Fix] Milter_headers: Fix remove_upstream_spam_flag and modernise config - [Fix] Mime_parser: Fix issue with parsing of the trailing garbadge - [Fix] Mime_parser: Fix parsing of mime parts without closing boundary - [Fix] Multimap: Fix operating with userdata - [Fix] Process orphaned symbols section - [Fix] Rdns: Fix multiple replies in fake replies - [Fix] Rework groups scores definitions - [Fix] Set proper element when reading data from Sentinel - [Fix] Set rspamd user to initialise supplementary groups on reload - [Fix] Settings: Fix selectors usage - [Fix] Sort data received from Sentinel to avoid constant replacing - [Fix] groups.conf - filename typo - [Fix] lua_scanner - oletools typos, logging - [Fix] lua_scanners - actions and symbol_fail - [Fix] lua_scanners - fix luacheck - [Fix] lua_scanners - kaspersky - response with fname - [Fix] lua_scanners - savapi redis prefix - [Fix] tests - antivirus - fprot symbols - [Project] Add concept of flexible actions - [Project] Add heuristical from parser to received parser - [Project] Add new flags to clickhouse, redis and elastic exporters - [Project] Attach new received parser - [Project] Fallback to callbacks from coroutines - [Project] Implement keep-alive support in lua_http - [Project] Lua_udp: Implement fully functional client - [Project] Plug keepalive knobs into http connection handling - [Project] Rspamadm: Add modify tool - [Rework] Convert rspamd-server to a shared library - [Rework] Dcc: Rework DCC plugin - [Rework] Enable explicit coroutines symbols - [Rework] Rework telephone urls parsing logic - [Rework] Rewrite RBL module - [Rework] Settings: Rework settings check - [Rework] Slashing: Distinguish lualibdir, pluginsdir and sharedir - [Rework] Unify task_timeout - [Rework] Use VEX instructions in assembly, relocate - [WebUI] Notify user if uploaded data was not learned - [WebUI] Remove redundant condition From vsevolod at rspamd.com Fri Apr 5 10:36:45 2019 From: vsevolod at rspamd.com (Vsevolod Stakhov) Date: Fri, 5 Apr 2019 11:36:45 +0100 Subject: [Rspamd-Announce] Rspamd 1.9.1 has been released Message-ID: We have released Rspamd 1.9.1 today. This release includes one potentially dangerous change: all configuration files are now preprocessed using Jinja templates. Hence, if you have sequences like {=/=}, or {%/%}, or {#/#} ANYWHERE in the configuration files including even comments then you need to take extra care when moving these configuration to the new version! There are workarounds described above to do that. {% endraw %} Here is the list of the most important changes in this version. Jinja templates in the configuration From version 1.9.1, Rspamd supports Jinja2 templates provided by Lupa Lua library. You can read the basic syntax documnentation and the abilities provided by these templating engines using the links above. Rspamd itself uses a specific syntax for variable tags: {= and =} instead of the traditional {{ and }} as these tags could mean, e.g.?a table in table in Lua. Templating might be useful to hide some secrets from config files and places them in environment. Rspamd automatically reads environment variables that start from RSPAMD_ prefix and pushes it to the env variable, e.g. RSPAMD_foo=bar comes to env.foo="bar" in templates. New template subcommand in Rspamadm Rspamadm has now template subcommand to apply templates engine to the input file or files: Options supported: |? |-n, ?no-vars | Don?t add Rspamd internal variables | | -e , ?env | Load additional environment var from specific file (name=value) | | -l , ?lua-env | Load additional environment vars from specific file (lua source) | | -s , ?suffix | Store files with the new suffix | | -i, ?inplace | Replace input file(s) | Changes in URLs extraction for HTML parts Rspamd now tries to extract URLs from plain text of HTML parts. Unfortunately, despite of being contraversal, some Email clients do that as well. One of the notable example is Outlook. Hence, from this release Rspamd also looks for URLs in plain HTML text. Per user settings for mime_types plugin Mime types plugin now supports per user settings to allow individual black and white lists of extensions. Here is an example to increase score for exe extensions for some specific user: test { from = "user at example.com"; apply { plugins { mime_types = { bad_extensions = { exe = 100500, } } } } } Mime types plugin now also supports reverse mapping of content type to extension to allow processing of attachments where an exact file name is not specified. Better greylisting conditioning It is now possible to disable or enable greylisting in Rspamd based on the presence of some specific symbols. This feature allows more fine grained greylisting control. Bitcoin addresses validation It is not a secret that the wave of spam and scam related to crypto currencies has been flooding the email flows in the recent time. Rspamd has a special rule called LEAKED_PASSWORD_SPAM to block some of the scam types. In this version, Rspamd also checks bitcoin wallets to distinguish them from random long strings to reduce false positives rate significantly. It also allows to build a database of wallets used for scam and spam. Replies plugin validation Replies plugin now stores the from/reply-to address when tracking outbound messages and whitelists merely replies that come that address. It helps to avoid replies abusing where spammers were able to catch some legit message ids somewhere in public lists and used them in In-Reply-To headers to dodge spam filtering in Rspamd. List of major bug fixes This version includes some important fixes: - Add crash safety for HTTP async routines - Clickhouse: Fix table schema upload - Core: Fix squeezed dependencies handling for virtual symbols - Finally fix default parameters parsing in actions section - Fix ES sending logic (restore from coroutines mess) - Fix finishing script for Clickhouse collection - Fix priority for regexp symbols registration - Neural: Fix training - Rework cached Redis logic to avoid sentinels breaking - SURBL: Fix regression in surbl module - Fix double signing in the milter Full list of the meaningful changes - [Conf] Add vendor groups for symbols - [Feature] Add rspamadm template command - [Feature] Allow to add messages from settings - [Feature] Allow unconnected DNS servers operations - [Feature] Check limits after being set, migrate to uint64 - [Feature] Greylist: Allow to disable greylisting depending on symbols - [Feature] Improve lua binary strings output - [Feature] Mime_types: Implement user configurable extension filters - [Feature] Mime_types: When no extension defined, detect it by content - [Feature] Preprocess config files using jinja templates - [Feature] Replies: Filter replies sender to limit whitelisting to direct messages - [Feature] Treat all tags with HREF as a potential hyperlinks - [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM - [Fix] Add crash safety for HTTP async routines - [Fix] Another fix for Redis sentinel - [Fix] Clickhouse: Fix table schema upload - [Fix] Core: Fix squeezed dependencies handling for virtual symbols - [Fix] Finally fix default parameters parsing in actions section - [Fix] Fix ES sending logic (restore from coroutines mess) - [Fix] Fix finishing script for clickhouse collection - [Fix] Fix priority for regexp symbols registriation - [Fix] Fix various issues found by PVS Studio - [Fix] Initialize lua debugging earlier - [Fix] Neural: Fix training - [Fix] Rework cached Redis logic to avoid sentinels breaking - [Fix] SURBL: Fix regression in surbl module - [Fix] Fix double signing in the milter - [Project] Add support of HTTP proxy in requests - [Rework] Change lua global variables registration - [Rework] Rework HTML content urls extraction - [Rework] Start rework of aliasing in Rspamd - [WebUI] Combine Scan and Learning into one tab - [WebUI] Fix symbol score input type - [WebUI] Show grayed out pie - [WebUI] Update Throughput summary values dynamically From vsevolod at rspamd.com Tue Apr 16 16:42:44 2019 From: vsevolod at rspamd.com (Vsevolod Stakhov) Date: Tue, 16 Apr 2019 17:42:44 +0100 Subject: [Rspamd-Announce] Rspamd 1.9.2 has been released Message-ID: <8ca8ba87-2484-e859-7286-800b58457d08@rspamd.com> We have released Rspamd 1.9.2 today. This release contains some new features and bug fixes. The only potentially slashing changes are the changes in Clickhouse module: - Times are now stored in GMT timezone so you can use Clickhouse for analytics that crosses time zones. The potential drawback is the mess with the currently stored data. This should be resolved automatically once new data arrives. - Clickhouse schema has been updated to the version 4 with new fields and some minor changes. The existing database should be converted automatically and there are no incompatible changes in columns. This release includes the following features. Improvements in Clickhouse plugin Rspamd now stores more data in Clickhouse: - Mime recipients - Message IDs - Scan time for a message, both normal and virtual - SPF checks results - Some new calculated columns, such as MIMERcpt, MIMEFrom, SMTPFrom and SMTPRcpt These columns are intended to improve analytical capabilities of Clickhouse plugin. OpenDKIM compatible DKIM signing setup This version now includes a simplified DKIM signing setup option inspired with OpenDKIM. You can read more about it here: https://rspamd.com/doc/modules/dkim_signing.html#use-of-signing_table This mode is intended to simplify migration from the existing setups based on OpenDKIM to Rspamd. Better encrypted archives support Rspamd can now properly detect encryption in ZIP archives. Mime types plugin now also tries to resolve hex encoding hack used by some spammers to send malware to users (see PR 2582). Calendar files parser From the version 1.9.2, Rspamd can extract meaningful data from Calendar files in iCal format (.ics files). These files are sometimes used by spammers so Rspamd can now extract hyperlinks and emails from calendar attachments to improve filtering quality. New rspamadm dns_tool utility It is now possible to do some DNS checks with the new tool. For example, it is now possible to verify SPF records as they are observed by Rspamd, including elements extraction, for example a or mx and verification of the IP addresses. Here is how it looks like: Better bitcoin addresses detection We have improved bitcoin addresses detection by fixing some issues in the BTC wallet validation code. It now allows to catch Pay-To-Script addresses. Full list of the meaningful changes - [Conf] Allow to load users plugins from plugins.d - [Conf] oversign openpgp and autocrypt headers - [Feature] Add SPF FFI library for Lua - [Feature] Add more verbosity for SPF caching - [Feature] Antivirus: Handle encrypted files specially - [Feature] Clickhouse: Slashing - add new fields to CH - [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table - [Feature] Dkim_signing: Allow to use new options as maps - [Feature] Import fpconv library - [Feature] Lua_maps: Allow static regexp and glob maps - [Feature] Parse ical files - [Feature] Rspamadm: Add dns_tool utility - [Feature] Store SPF records digests - [Feature] Use fpconv girsu2 implementation for printing floats - [Fix] Clickhouse: Use integer seconds when inserting rows - [Fix] Fix floating point printing - [Fix] Fix processing of embedded urls - [Fix] Lua_clickhouse: Fix CH errors processing - [Fix] Make spf digest stable - [Fix] Properly detect encrypted files in zip archives - [Fix] Slashing: Store times in GMT timezone in ClickHouse - [Rules] Add additional conditions to perform BTC checks - [Rules] Fix pay-to-hash addresses validation From vsevolod at rspamd.com Mon May 13 13:28:32 2019 From: vsevolod at rspamd.com (Vsevolod Stakhov) Date: Mon, 13 May 2019 14:28:32 +0100 Subject: [Rspamd-Announce] Rspamd 1.9.3 has been released Message-ID: <216bbae0-74b6-71e3-991f-c210606d26c3@rspamd.com> We have released Rspamd 1.9.3 today. This release contains some new features and many bug fixes. There are no incompatible changes introduced with this release to our best knowledge. This release includes the following features and important changes. Hashicorp Vault support From version 1.9.3, Rspamd can use Hashicorp Vault to store and manage DKIM keys. Vault usage provides secure and flexible storage of the private keys that can scale and use various backends to store sensible data (secrets). There is a new subcommand for rspamadm utility called vault that is intended to create/remove and securely rotate DKIM private keys using vault. You can read more about it using the following link: https://rspamd.com/doc/modules/dkim_signing.html#dkim-signing-using-vault Added least passthrough result Some modules should set metric result as LEAST POSSIBLE result. For example, DMARC policy failure should at least mark failed messages as spam but it should not prevent messages from being rejected. From this release, such modules use least policy to set actions allowing to apply a more strict policy if needed. Tunable memory management From this version, Rspamd allows to manage memory policies for Lua garbage collection allowing to fit memory/cpu constraints more flexible. When Rspamd is built with jemalloc (e.g.?in the default packages provided by the project), it can also print detailed memory statistics on full gc loops. Here is an example for tuning GC in Rspamd when there are lots of free memory available (around 1Gb per scanner process): # local.d/options.inc # http://pgl.yoyo.org/luai/i/2.10+Garbage+Collection lua_gc_step = 100; lua_gc_pause = 400; # number of scanned messages to perform full GC iteration full_gc_iters = 10000; Improved oversigning logic It is now possible to oversign existing only headers and ignore it if a header is missing. It is done by changing (o) to (x). The default list of headers signed is changed accordingly: Header Sign type --------------------------- ------------------------ From Strictly oversign Sender Conditionally oversign Reply-To Strictly oversign Subject Strictly oversign Date Conditionally oversign Message-Id Conditionally oversign To Strictly oversign Cc Strictly oversign Mime-Version Conditionally oversign Content-Type Conditionally oversign Content-Transfer-Encoding Conditionally oversign Resent-To Do not oversign Resent-Cc Do not oversign Resent-From Do not oversign Resent-Sender Do not oversign Resent-Message-Id Do not oversign In-Reply-To Conditionally oversign References Conditionally oversign List-Id Do not oversign List-Help Do not oversign List-Owner Do not oversign List-Unsubscribe Do not oversign List-Subscribe Do not oversign List-Post Do not oversign Openpgp Conditionally oversign Autocrypt Conditionally oversign Important bugs fixes Here is the list of the most important bugs fixes: - HTML: Fix size attribute processing - this issue caused rule MANY_INVISIBLE_PARTS to be improperly triggered on many HTML messages - Do not blacklist mail by SPF/DMARC for local/authed users - Lots of Clickhouse plugin fixes - Fix buffer overflow when printing small floats - this issue caused random crashes in WebUI reported by many users - Fix DoS caused by bug in glib - details in https://gitlab.gnome.org/GNOME/glib/issues/1775 Full list of the meaningful changes - [Conf] Add IP_SCORE_FREEMAIL composite rule - [Feature] Add cryptobox method to generate dkim keypairs - [Feature] Add fast hashes to lua cryptobox hash - [Feature] Add least passthrough results - [Feature] Allow oversign if exists mode - [Feature] Clickhouse: Modernise table initial schema - [Feature] Implement IUF interface for specific fast hashes - [Feature] Lua_util: Allow to obfuscate different fields - [Feature] Tune memory management in Rspamd and Lua - [Fix] Avoid buffer overflow when printing long lua strings - [Fix] Change the default oversigning headers to a more sane list - [Fix] Clickhouse: Do not store digest as it is not needed now - [Fix] Clickhouse: Fix lots of storage issues - [Fix] Clickhouse: Support custom actions - [Fix] Deny URLs where hostname is bogus - [Fix] Do not blacklist mail by SPF/DMARC for local/authed users - [Fix] Fix DoS caused by bug in glib - [Fix] Fix UCL parsing of the multiline strings - [Fix] Fix buffer overflow when printing small floats - [Fix] Fix init code for servers keypairs cache - [Fix] Fix issue with urls with no tld (e.g.?IP) - [Fix] Fix memory in arc signing logic - [Fix] Fix memory leak in language detector during reloads - [Fix] Fix mixed case content type processing - [Fix] Fix processing of the ip urls in file - [Fix] Fix use after free - [Fix] HTML: Fix size attribute processing - [Fix] Hum, it seems that 99ff1c8 was not correct - [Fix] Lua_task: Fix task:get_from method - [Fix] Preserve fd when mapping file to scan - [Fix] Re-use milter_headers settings when doing arc signing - [Fix] Set dmarc force action as least action - [Fix] Switch to GMT - [Fix] allow PKCS7 signatures to be text/plain, too - [Project] Add initial version of the vault management tool - [Project] Add vault support for DKIM and ARC signing - [Project] Implement keys rotation in the vault - [Project] Improve dkim keys generation for vault - [Project] Improve keys creation in rspamadm vault - [Rework] Move lua_worker to a dedicated unit - [WebUI] Add URL fragments (#) support - [WebUI] Fix AJAX request URL From vsevolod at rspamd.com Fri Oct 11 17:00:30 2019 From: vsevolod at rspamd.com (Vsevolod Stakhov) Date: Fri, 11 Oct 2019 18:00:30 +0100 Subject: [Rspamd-Announce] Rspamd 2.0 has been released Message-ID: <9f3c9a37-5d34-0425-3dd4-05cb7cd3c1d8@rspamd.com> We have released Rspamd 2.0 today! This version encompasses new versioning schema that will be used in future Rspamd releases: specifically, instead of the .., Rspamd will use just . versioning schema. This happens because the number has never been increased for many years and number has been used as a real version indicator. Upgrade notes There are various important features in this release. The vast majority of those should not have any visible impact on the existing systems. However, you are recommended to read the Upgrade Notes. The main potential source of incompatibilities is the deprecation of the surbl and emails modules that have been replaced with rbl module. The default Bayes backend is also changed to Redis now while the Sqlite backend is now marked as deprecated and is not recommended for use. ip_score, neural and ratelimit modules users are strongly advised to read the upgrading notes! Packages support In this version of Rspamd, we have stopped support of the following OS variants: - Ubuntu trusty (reached EOL) - Centos 6 (almost reached EOL) We have added Centos 8 packages instead. As usually, Rspamd project strongly recommends NOT TO USE the packages that are provided by 3rd parties, including your own LINUX DISTRIBUTION. These packages are usually out-of-date, built incorrectly and accordingly, they are not supported by Rspamd project. Please use the official packages only. FreeBSD ports are considered official packages as they are supported by Rspamd project directly (well, strictly speaking by myself). Here is the list of the most important changes in this release. Libevent has been replaced with bundled libev After many years of using the libevent library Rspamd switched to libev library. The main reason was performance and control: there were many libevent versions shipped with various supported platforms and many of those lacked important features, such as inotify support for Linux. Switching to libev allowed us to simplify the code, improve signals handling, improve timeouts handling and deal with file maps changes instantly due to inotify. Torch has been dropped from Rspamd Lua torch has served as a powerful engine for ML and neural networks in Rspamd for quite a long time. However, it is no longer maintained or updated and its support has proven to be a nightmare. There were also important bugs that could not be fixed due to the code complexity. From version 2.0, Rspamd adopted kann library that is much more friendly for embedding and provides very convenient interfaces that are now exported via Lua. RBL module improvements and replacement of the SURBL and Emails module RBL module has replaced both emails and surbl modules unifying all Runtime Black Lists checks in a single place. It has added new RBL types, such as selectors, and the simplified extending of the existing rules to more powerful ones. Emails rules with maps instead of DNS RBLs are NO LONGER SUPPORTED. Please use multimap with selectors instead. New Lua Magic library For file types detection, Rspamd now uses an own implementation of detection library based on Lua and Hyperscan (where possible) instead of libmagic. There are 4 major concerns for that: - Libmagic is a generic library that can easily detect pdp11 a.out format but can fail in docx detection surprisingly often - We need performance and libmagic is not about performance at all - We want to add new detection heuristics instead of relying on 3rd party strict rules - Libmagic API is not very suitable for us With the new library, Rspamd can detect part types in just a couple of microseconds and find the vast majority of the _interesting_ things, such as executables, archives, images, html and so on and so forth. Neural module rework Neural networks module has been almost totally rewritten to support KANN library and symbols profiles. Now, Rspamd will not reset neural network on each individual symbol change - it will try to use the most appropriate network instead. Many issues with neural learning dead locks have also been addressed. Clickhouse module improvement - Added LowCardinality fields to improve storage requirements - Fixed retention code - Significantly optimized memory usage by using userdata instead of interned strings Multimap module Various new features, including maps combinations and dependent maps(https://rspamd.com/doc/modules/multimap.html#dependent-maps). Maillist module Improved mailing lists detection and reworked detection heuristic. Heartbeats support Rspamd workers now send heartbeat events to the main process. In turn, the main process can now kill hanged workers if a reasonable amount of heartbeats have been lost. This feature is not enabled by default for now. Lua scanners improvements There are lots of additions in lua scanners. Many of those have been contributed by Carsten Rosenberg from HeinleinSupport. New antivirus engines support: - Kaspersky ScanEngine: https://www.kaspersky.com/scan-engine - Trend Micro IWSVA support via icap (by @c-rosenberg) - F-Secure Internet Gatekeeper via icap (by @c-rosenberg) New external scanners: - Razor support (by @c-rosenberg) - Better oletools support (by @c-rosenberg) - P0F support as a separate module (by Denis Paavilainen - @denpamusic) Mime modifications From version 2.0, Rspamd allows modifying messages via Lua API methods. This support required massive rework of the internal structures and have been tested by Migadu. These functions are implemented in the lua_mime library. Users settings improvements Rspamd now treats settings differently if they are set via Settings-Id: there are certain performance benefits and better logging in all modules. It is also possible to bind rules explicitly to certain settings id allowing to separate mail processing flows more efficiently. Upstreams library improvements - Added lazy resolving of the upstreams - Added SRV upstreams to resolve SRV records for both names, ports, and priorities (e.g.?by using Hashicorp Konsul DNS) - Use random strings for monitoring sanity Performance improvements - Improved base64 decoding for typical outputs - Langdet: Limit number of stop words to be checked - Added sanity limit for task:get_urls() method to avoid Lua memory blow - Maps: Allow caching for complex maps - Settings fast path have been added - Lua core: use lightuserdata to index classes to avoid strings interning - HTTP(s) keep-alive support has been added Rules and other improvements - Added BITCOIN_ADDR symbol to allow custom composite rules creation to block scam campaigns - Support Litecoin addresses - Implement syntax highlighting for Lua - Allow execution of async events when hs compiles regexps - Bayes expiry: eliminate default expiration mode (use lazy mode all the time) - Eliminate lua_squeeze as it has shown no improvements - Drop url tags - Eliminate virtual scan time as it is useless - Use replxx instead of linenoise - Added SSL/STARTTLS support to lua_tcp library - Implemented SSL graceful closing This version of Rspamd contains a number of other minor and major improvements and fixes compared to the 1.9 branch. This includes some bugs that were fixed in 2.0 and that could cause certain issues, hangs or crashes with certain emails. From vsevolod at rspamd.com Mon Oct 28 17:36:00 2019 From: vsevolod at rspamd.com (Vsevolod Stakhov) Date: Mon, 28 Oct 2019 17:36:00 +0000 Subject: [Rspamd-Announce] Rspamd 2.1 has been released Message-ID: We have released Rspamd 2.1 today. This release contains some new features and many bug fixes. There are no incompatible changes introduced with this release to our best knowledge. This release includes the following features and important fixes. Add uuencode support Despite of being very old standard, UUenconde parts are still quite common in the email traffic observed. From this version, Rspamd supports uuencoded parts (both normal and base64 version). Critical issue found in dkim verification There was a critical regression in 2.0 DKIM verification code caused verification failures for some of the valid DKIM signatures. More details are in the GitHub issue. Improved neural training There are number of fixes and improvements in the Neural module. Now all training samples are balanced using random sampling allowing a smoother training vectors selection. Some number of bugs has been fixed, as well as scores are NO LONGER RECOMMENDED to select training vectors - Rspamd automatically applies heuristic to select messages for learning. Also some issues around infinities and learning threads count have been addressed. Maps fixes There are number of fixes and improvements around maps handling logic. This include fixes for both HTTP and file maps, as well as better timeout and caches handling. Event loop fixes Rspamd could previously select an inefficient backend on some OSes, notably, on BSD and OSx. This version should fix it. The ability to configure the events backend manually via the configuration file has also been added to Rspamd. Full list of the meaningful changes - [Conf] Update neural.conf - [CritFix] Fix dkim verification for multiple headers listed - [Feature] Add support of uudecode - [Feature] Allow to explicitly set events backend - [Feature] Implement configurable limits for SPF lookups - [Feature] Lua_scanners: Use lua magic for inclusion/exclusion logic - [Feature] Multimap: Do not check files in office archives - [Feature] Neural: Add sampling when storing training vectors - [Feature] SPF: Allow to disable AAAA checks in configuration - [Feature] Spf: Add limits configuration support - [Feature] Store etag in cached HTTP maps + better logging - [Feature] Support segwit BTC addresses, fix LTC verification - [Feature] Support uuencoding - [Fix] Add configurable number of threads for OpenBLAS - [Fix] Add workaround for ragel 7 in hyperscan related maps code - [Fix] Another fix for numeric urls parsing - [Fix] Correct EMA time calculations - [Fix] Do not treat archives as text - [Fix] Do not use strdup on data extracted from lua - [Fix] Fix a failure calcuating URL reputation. - [Fix] Fix crash due to constructors init order - [Fix] Fix crash on parts with no cd - [Fix] Fix empty prefilters that require mime structures - [Fix] Fix event loop creation - [Fix] Fix issues sending DMARC reports. - [Fix] Fix misprint - [Fix] Fix saving of the file maps - [Fix] Fix size calculations when converting from utf16 - [Fix] Fix support of disable_monitoring in rbl - [Fix] Fix use-after-free - [Fix] Fix zip files check to relax requirements - [Fix] Important hiredis fixes - [Fix] Lot?s of fixes in maps check logic - [Fix] Lua_tcp: Deal with temporary fails on write - [Fix] Lua_tcp: Make write errors fatal and rework error handlers - [Fix] Meta: Filter some more values - [Fix] Neural: Add protection agains infinities - [Fix] Oops, fix math.huge invocation - [Fix] Plug memory leak - [Fix] Sigh, another email to string fix - [Fix] Try to fix another ownership race in ssl connection - [Fix] Uuencode: Fix parsing of corrupted uuencode - [Fix] lua_scanners - razor rename need_check function - [Rework] Require CMake 3.9 to work, remove manual lto crap From vsevolod at rspamd.com Tue Nov 19 17:43:51 2019 From: vsevolod at rspamd.com (Vsevolod Stakhov) Date: Tue, 19 Nov 2019 17:43:51 +0000 Subject: [Rspamd-Announce] Rspamd 2.2 has been released Message-ID: <6ed0d46a-e7a6-39c5-9996-259ad636b817@rspamd.com> We have released Rspamd 2.2 today. This release contains some new features and many bug fixes. There are no incompatible changes introduced with this release to our best knowledge. This release includes the following features and important fixes. Added virustotal support Rspamd now supports Virustotal as an Antivirus plugin. You need to obtain API key to use this service. All normal antivirus module operations are applicable to this plugin. Clickhouse collection rework Rspamd now does Clickhouse data collection in a separate perioric event. It allows to do collections based on time, number of rows (as previously) or on amount of memory used. More details are in the GitHub issue. ASAN builds Rspamd packages have now ASAN branches to help debugging issues with Rspamd and provide better feedback for the developers. The details about ASAN builds are covered in this FAQ section. Faster base64 decoding We have applied number of optimizations to improve the performance of base64 decoding on the modern hardware (especially with AVX2 and/or SSE4.2 support). Fast unicode validation library Rspamd now uses number of techniques to improve utf8 validation by utilising modern CPU instructions, such as AVX2 and SSE4. This code is based on the work from Yibo Cai and achieves around 0.5 CPU cycles per byte speed when using AVX2 codec. Upstreams fixes There are number of significant improvements in the upstreams library of Rspamd. Specifically, that includes better consistent hashing, better upstreams marking logic and improved logging. Build system rework The CMake based build system has been reworked to use more modern design practices provided by newer CMake versions (Rspamd now requires CMake 3.9 as minimum). New build system should improve multiple configurations support and simplify CMake build files. Full list of the meaningful changes - [Conf] Antivirus: Fix the default config - [Feature] Add verdict library in lua - [Feature] Allow exception when choosing upstream - [Feature] Allow to disable symbols from the metric config - [Feature] Allow to limit maps per specific worker - [Feature] Always validate Rspamd protocol output - [Feature] Antivirus: Add preliminary virustotal support - [Feature] Clickhouse: Rework Clickhouse collection logic - [Feature] Improve base64 usage - [Feature] Shutdown timeout is now associated with task timeout - [Fix] #3129 Multiple classifiers on redis working incorrectly - [Fix] Allow real upstreams configuration - [Fix] Another try to fix slow callbacks and timers - [Fix] Check results of write message as SSL can bork them - [Fix] Clickhouse: Avoid potential races in collection - [Fix] Clickhouse: Fix periodic script - [Fix] Fail DNS upstream on each retransmit attempt - [Fix] Fix consistent hashing when upstreams are marked inactive - [Fix] Fix issues found - [Fix] Fix off-by-one in retries for the proxy - [Fix] Fix termination - [Fix] Fix upstreams exclusion logic - [Fix] Fix utf8 validation for symbols options and empty strings - [Fix] Oops, fix maps reload - [Fix] Rbl: Allow utf8 lookups for IDN domains - [Fix] Sigh, another try to fix brain-damaged openssl - [Project] Add fast utf8 validation library - [Project] Use own utf8 validation instead of glib - [Rework] Another phase of finish actions rework - [Rework] Further cmake system rework - [Rework] Further isolation of the controller?s functions - [Rework] Make cmake structure more modular - [Rework] Move cmake modules to a dedicated path - [Rework] Replace controller functions by any scanner worker if needed - [Rework] Rework final scripts logic - [Rework] Rewrite rspamd_str_make_utf_valid function