[Rspamd-Users] Rspamd 1.8.0 has been released

[Sophie Loewenthal]

Hi,

Thanks for the new release. I have a question:-
"... please check the migration notes" which are where?  Did I miss them?

Thanks , Sophie
Sent from phone, thus brief

On September 24, 2018 4:09:29 PM CEST, Vsevolod Stakhov <vsevolod at rspamd.com> wrote:
>We have released Rspamd 1.8.0 today.
>
>There are couple of the incompatibilities introduced, hence, please
>check the migration notes, especially if you use Clickhouse module or
>users settings.
>
>
>The most important features and fixes
>
>-   New SELECTORS FRAMEWORK
>
>    This framework allows to combine and process different data
>    extracted from messages and use that in different plugins, such as
>    multimap, reputation or ratelimits. It is also possible to use data
>    extracted in Rspamd regular expressions.
>
>-   COROUTINES API support in Lua
>
>    Now you can write code in a usual imperative manner but you still
>    will not block any other tasks. Each potentially blocking operation
>    creates a yielding-point. In turn, this means the code is suspended
>    until the operation is done (just like blocking) and resumes only
>    when there is some result. Meanwhile, other tasks are processed as
>    usual.
>
>-   Clickhouse optimization
>
>   Rspamd now uses a flat table to optimize ClickHouse SQL requests. In
>    fact, joins are not recommended by the ClickHouse developers as
>    multiple joins have proven to be slow. Hence, Rspamd has moved all
>    data to a single table. Schema migration is done automatically,
>    however, please read the migration notes in case of any doubts. Old
>    data is not migrated nor deleted automatically.
>
>    There is now optional data retention support in the ClickHouse
>    module. You can set retention policies for the data stored in
>    Clickhouse to conform different regulations (e.g. GDPR).
>
>-   Unicode processing improvements
>
>    Rspamd now normalizes all unicode data using NFKC schema prior to
>    processing. This helps to prevent “glyph” attacks used by some
>    spammers nowadays. Unicode conversion has also been improved to
>    continue on bad symbols instead of giving up and working with raw
>    data.
>
>-   Language detection improvements
>
>   We have reworked the language detector to use stop-words and rely on
>    unicode glyphs more extensively. As the result of this work, the
>    speed of language detection has been increased significantly (by 10
>   times in some cases). The preciseness of the detection has also been
>    improved.
>
>-   Fixed various bugs in sesssions handling
>
>    We have located and fixed various hidden issues caused by async
>    rules chaining. It might cause inconsistencies in the dependencies
>    processing, crashes in rare cases and other “bad things”.
>
>-   Various Web Interface improvements and fixes
>
>    There are multiple improvements and fixes in the Web Interface. In
>    particular, the issues with cluster support and aggregation have
>    been addressed.
>
>-   New mailing lists for the project
>    -   Rspamd-Users: general purpose mailing list that will replace
>        this group
>    -   Rspamd-Announce: read only list with low traffic that is
>        intended for project announcements only, for example, new
>        versions, or vulnerabilities disclosures
>
>    Unfortunately, there is no automatic conversion from the Google
>    groups to the new mailing lists, hence, you need to subscribe to
>    those lists manually. We are sorry about the potential
>    inconveniences caused by this transition.
>
>    To subscribe to a list, click the list name at
>    https://lists.rspamd.com/mailman/listinfo . The page that is
>    displayed should contain all of the necessary subscription
>   instructions for that list. You can always find all possible support
>    channels here.
>
>
>Full list of the meaningful changes
>
>-   [Feature] Add arguments schemas to processors and extractors
>-   [Feature] Add functional selectors library
>-   [Feature] Add generic selector to reputation module
>-   [Feature] Add more ratelimits: by digest, by attachments data, by
>    filenames
>-   [Feature] Add preliminary stop words detection support
>-   [Feature] Add pure Lua debugm function
>-   [Feature] Add schema validation for Redis settings
>-   [Feature] Add selectors combine function
>-   [Feature] Add some recursion protection to lua logger
>-   [Feature] Add support for Lua API tracing
>-   [Feature] Allow to apply schema to arguments
>-   [Feature] Allow to get dkim signing data directly from HTTP headers
>-   [Feature] Allow to reuse existing authentication results
>-   [Feature] Cache selectors results in re runtime
>-   [Feature] Implement new text tokenizer based on libicu
>-   [Feature] Integrate selectors framework to multimap
>-   [Feature] Relax FORGED_RECIPIENTS
>-   [Feature] Support (almost) all html entities
>-   [Feature] Support adding and deletion of recipients in the milter
>    block
>-   [Feature] Support gathering HTTP body from fragments in lua_http
>-   [Feature] Support multi flag in regexp and glob maps
>-   [Feature] Support selectors in ratelimit module
>-   [Feature] Support selectors in settings
>-   [Feature] Use khash in HTML parser
>-   [Feature] Use pure Lua debugm function
>-   [Fix] Add fail-safety for destroying sessions
>-   [Fix] Allow to add result-less fake DNS records
>-   [Fix] Another try to fix race conditions on config unload
>-   [Fix] Call Lua callback on DNS timeouts
>-   [Fix] Deprecate task:inc_dns_req as it is redundant
>-   [Fix] Do not allow events deletions on cleanup
>-   [Fix] Do not try to process skipped messages
>-   [Fix] Fix HTTP requests with no body
>-   [Fix] Fix another cleanup race condition
>-   [Fix] Fix bug in processing of pcre regexps
>-   [Fix] Fix byte array allocation in the pool
>-   [Fix] Fix crashes on task cleanup
>-   [Fix] Fix dynamic buckets in ratelimits
>-   [Fix] Fix endless loop when waiting for Rspamd to stop
>-   [Fix] Fix lua_util.str_split in case of delimiters set
>-   [Fix] Fix more issues with watching of async events
>-   [Fix] Fix stop words detection and loading logic
>-   [Fix] Fix various corner cases for language detection
>-   [Fix] Fix watchers in lua_tcp
>-   [Fix] Fix words decay algorithm
>-   [Fix] Implement watchers replacement to handle nested calls
>-   [Fix] Save faked code into fake dns record
>-   [Fix] Show the proper frame when using lua_util.debugm
>-   [Fix] Use fake dns records in tests
>-   [Fix] Use unicode replacements for HTML entities
>-   [Fix] fixed “cannot find dependency on symbol 1” issue when using
>    replaced symbols in spamassassin rules
>-   [Fix] partition_id is not available in old versions of CH
>-   [Project] Add implicit conversion logic to selectors
>-   [Project] Add initial support for selectors in regexps
>-   [Project] Add method concept
>-   [Project] Further changes in unicode operations
>-   [Project] Implement Clickhouse migrations
>-   [Project] Implement implicit conversions to userdata
>-   [Project] Implement insert method
>-   [Project] Implement selectors registration for regular expressions
>-   [Project] Implement selectors support in re_cache
>-   [Project] Improve language detector: cleanup unused
>files,categorize
>-   [Project] Migrate CH data to a fat table
>-   [Project] Rework selectors logic
>-   [Project] Start Clickhouse utilities library
>-   [Project] Start unicode rework
>-   [Project] coroutine threaded model for API calls: thread pool
>-   [Rework] Move phishtank to a DNS based service
>-   [Rework] Rework Clickhouse plugin to use the new API
>-   [Rework] Rework language detector
>-   [Rework] Rework utf content processing in text parts
>-   [WebUI] Add progress bar for AJAX requests
>-   [WebUI] Avoid errors table reinitialization
>-   [WebUI] Avoid history table reinitialization
>-   [WebUI] Avoid throughput summary table reinitialization
>-   [WebUI] Destroy summary table on disconnect
>-   [WebUI] Fix “auth” request URL
>-   [WebUI] Fix disabling and hiding controls on page reload
>-   [WebUI] Fix maps loading from neighbours
>-   [WebUI] Fix symbols sorting by score
>-   [WebUI] Fix tables destroying
>-   [WebUI] Fix throughput data consolidation
>-   [WebUI] Fix upload buttons disabling
>-   [WebUI] Notify user on module loading failure
>-   [WebUI] Update FooTable 3.1.4 -> 3.1.6
>-- 
>Users mailing list
>Users at lists.rspamd.com
>https://lists.rspamd.com/mailman/listinfo/users