[Rspamd-Users] Rspamd 1.8.0 has been released

[B. Reino]

On Mon, 24 Sep 2018, Sophie Loewenthal wrote:

> Hi,
>
> Thanks for the new release. I have a question:-
> "... please check the migration notes" which are where?  Did I miss them?

Here: https://rspamd.com/doc/migration.html :)

>
> Thanks , Sophie
> Sent from phone, thus brief
>
> On September 24, 2018 4:09:29 PM CEST, Vsevolod Stakhov <vsevolod at rspamd.com> wrote:
>> We have released Rspamd 1.8.0 today.
>>
>> There are couple of the incompatibilities introduced, hence, please
>> check the migration notes, especially if you use Clickhouse module or
>> users settings.
>>
>>
>> The most important features and fixes
>>
>> -   New SELECTORS FRAMEWORK
>>
>>    This framework allows to combine and process different data
>>    extracted from messages and use that in different plugins, such as
>>    multimap, reputation or ratelimits. It is also possible to use data
>>    extracted in Rspamd regular expressions.
>>
>> -   COROUTINES API support in Lua
>>
>>    Now you can write code in a usual imperative manner but you still
>>    will not block any other tasks. Each potentially blocking operation
>>    creates a yielding-point. In turn, this means the code is suspended
>>    until the operation is done (just like blocking) and resumes only
>>    when there is some result. Meanwhile, other tasks are processed as
>>    usual.
>>
>> -   Clickhouse optimization
>>
>>   Rspamd now uses a flat table to optimize ClickHouse SQL requests. In
>>    fact, joins are not recommended by the ClickHouse developers as
>>    multiple joins have proven to be slow. Hence, Rspamd has moved all
>>    data to a single table. Schema migration is done automatically,
>>    however, please read the migration notes in case of any doubts. Old
>>    data is not migrated nor deleted automatically.
>>
>>    There is now optional data retention support in the ClickHouse
>>    module. You can set retention policies for the data stored in
>>    Clickhouse to conform different regulations (e.g. GDPR).
>>
>> -   Unicode processing improvements
>>
>>    Rspamd now normalizes all unicode data using NFKC schema prior to
>>    processing. This helps to prevent “glyph” attacks used by some
>>    spammers nowadays. Unicode conversion has also been improved to
>>    continue on bad symbols instead of giving up and working with raw
>>    data.
>>
>> -   Language detection improvements
>>
>>   We have reworked the language detector to use stop-words and rely on
>>    unicode glyphs more extensively. As the result of this work, the
>>    speed of language detection has been increased significantly (by 10
>>   times in some cases). The preciseness of the detection has also been
>>    improved.
>>
>> -   Fixed various bugs in sesssions handling
>>
>>    We have located and fixed various hidden issues caused by async
>>    rules chaining. It might cause inconsistencies in the dependencies
>>    processing, crashes in rare cases and other “bad things”.
>>
>> -   Various Web Interface improvements and fixes
>>
>>    There are multiple improvements and fixes in the Web Interface. In
>>    particular, the issues with cluster support and aggregation have
>>    been addressed.
>>
>> -   New mailing lists for the project
>>    -   Rspamd-Users: general purpose mailing list that will replace
>>        this group
>>    -   Rspamd-Announce: read only list with low traffic that is
>>        intended for project announcements only, for example, new
>>        versions, or vulnerabilities disclosures
>>
>>    Unfortunately, there is no automatic conversion from the Google
>>    groups to the new mailing lists, hence, you need to subscribe to
>>    those lists manually. We are sorry about the potential
>>    inconveniences caused by this transition.
>>
>>    To subscribe to a list, click the list name at
>>    https://lists.rspamd.com/mailman/listinfo . The page that is
>>    displayed should contain all of the necessary subscription
>>   instructions for that list. You can always find all possible support
>>    channels here.
>>
>>
>> Full list of the meaningful changes
>>
>> -   [Feature] Add arguments schemas to processors and extractors
>> -   [Feature] Add functional selectors library
>> -   [Feature] Add generic selector to reputation module
>> -   [Feature] Add more ratelimits: by digest, by attachments data, by
>>    filenames
>> -   [Feature] Add preliminary stop words detection support
>> -   [Feature] Add pure Lua debugm function
>> -   [Feature] Add schema validation for Redis settings
>> -   [Feature] Add selectors combine function
>> -   [Feature] Add some recursion protection to lua logger
>> -   [Feature] Add support for Lua API tracing
>> -   [Feature] Allow to apply schema to arguments
>> -   [Feature] Allow to get dkim signing data directly from HTTP headers
>> -   [Feature] Allow to reuse existing authentication results
>> -   [Feature] Cache selectors results in re runtime
>> -   [Feature] Implement new text tokenizer based on libicu
>> -   [Feature] Integrate selectors framework to multimap
>> -   [Feature] Relax FORGED_RECIPIENTS
>> -   [Feature] Support (almost) all html entities
>> -   [Feature] Support adding and deletion of recipients in the milter
>>    block
>> -   [Feature] Support gathering HTTP body from fragments in lua_http
>> -   [Feature] Support multi flag in regexp and glob maps
>> -   [Feature] Support selectors in ratelimit module
>> -   [Feature] Support selectors in settings
>> -   [Feature] Use khash in HTML parser
>> -   [Feature] Use pure Lua debugm function
>> -   [Fix] Add fail-safety for destroying sessions
>> -   [Fix] Allow to add result-less fake DNS records
>> -   [Fix] Another try to fix race conditions on config unload
>> -   [Fix] Call Lua callback on DNS timeouts
>> -   [Fix] Deprecate task:inc_dns_req as it is redundant
>> -   [Fix] Do not allow events deletions on cleanup
>> -   [Fix] Do not try to process skipped messages
>> -   [Fix] Fix HTTP requests with no body
>> -   [Fix] Fix another cleanup race condition
>> -   [Fix] Fix bug in processing of pcre regexps
>> -   [Fix] Fix byte array allocation in the pool
>> -   [Fix] Fix crashes on task cleanup
>> -   [Fix] Fix dynamic buckets in ratelimits
>> -   [Fix] Fix endless loop when waiting for Rspamd to stop
>> -   [Fix] Fix lua_util.str_split in case of delimiters set
>> -   [Fix] Fix more issues with watching of async events
>> -   [Fix] Fix stop words detection and loading logic
>> -   [Fix] Fix various corner cases for language detection
>> -   [Fix] Fix watchers in lua_tcp
>> -   [Fix] Fix words decay algorithm
>> -   [Fix] Implement watchers replacement to handle nested calls
>> -   [Fix] Save faked code into fake dns record
>> -   [Fix] Show the proper frame when using lua_util.debugm
>> -   [Fix] Use fake dns records in tests
>> -   [Fix] Use unicode replacements for HTML entities
>> -   [Fix] fixed “cannot find dependency on symbol 1” issue when using
>>    replaced symbols in spamassassin rules
>> -   [Fix] partition_id is not available in old versions of CH
>> -   [Project] Add implicit conversion logic to selectors
>> -   [Project] Add initial support for selectors in regexps
>> -   [Project] Add method concept
>> -   [Project] Further changes in unicode operations
>> -   [Project] Implement Clickhouse migrations
>> -   [Project] Implement implicit conversions to userdata
>> -   [Project] Implement insert method
>> -   [Project] Implement selectors registration for regular expressions
>> -   [Project] Implement selectors support in re_cache
>> -   [Project] Improve language detector: cleanup unused
>> files,categorize
>> -   [Project] Migrate CH data to a fat table
>> -   [Project] Rework selectors logic
>> -   [Project] Start Clickhouse utilities library
>> -   [Project] Start unicode rework
>> -   [Project] coroutine threaded model for API calls: thread pool
>> -   [Rework] Move phishtank to a DNS based service
>> -   [Rework] Rework Clickhouse plugin to use the new API
>> -   [Rework] Rework language detector
>> -   [Rework] Rework utf content processing in text parts
>> -   [WebUI] Add progress bar for AJAX requests
>> -   [WebUI] Avoid errors table reinitialization
>> -   [WebUI] Avoid history table reinitialization
>> -   [WebUI] Avoid throughput summary table reinitialization
>> -   [WebUI] Destroy summary table on disconnect
>> -   [WebUI] Fix “auth” request URL
>> -   [WebUI] Fix disabling and hiding controls on page reload
>> -   [WebUI] Fix maps loading from neighbours
>> -   [WebUI] Fix symbols sorting by score
>> -   [WebUI] Fix tables destroying
>> -   [WebUI] Fix throughput data consolidation
>> -   [WebUI] Fix upload buttons disabling
>> -   [WebUI] Notify user on module loading failure
>> -   [WebUI] Update FooTable 3.1.4 -> 3.1.6
>> --
>> Users mailing list
>> Users at lists.rspamd.com
>> https://lists.rspamd.com/mailman/listinfo/users
>