From aste at sveiks.lv Mon Sep 13 14:05:48 2021 From: aste at sveiks.lv (Aste) Date: Mon, 13 Sep 2021 17:05:48 +0300 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> Message-ID: <5710476131.20210913170548@sveiks.lv> Hello Faisal, Tuesday, August 31, 2021, 1:45:56 AM, you wrote: FM> Hi there, FM> If you remove reporting = true; it should work. You need to replace it by /enabled = true;/ inside the /reporting/ array which you already have. FM> |# local.d/dmarc.conf reporting { # Required attributes enabled = true; # Enable reports in general| Thank you it's solve initial problem, bet reporting still doesn't work, now I getting error Executing "rspamadm dmarc_report" results with error. call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - [Lua]}; sometimes I getting errors for misconfigured domains like cannot resolve ****** Cannot process reports for domain ***** So some parts of script is working, but no emails was sent, and always getting that error "call to rspamadm lua script failed" at the end. -- Best regards, Aste mailto:aste at sveiks.lv FM> For more info on the new parameters, check out the docs page at https://rspamd.com/doc/modules/dmarc.html#reporting FM> On 8/23/21 8:33 AM, Aste wrote: >> Hi, >> >> Seams that new (3.0 rspamd) DMARC report code doesnt't load config >> >> running "rspamadm dmarc_report" returns >> "dmarc reporting is not enabled, exiting" >> >> dmarc_settings = rspamd_config:get_all_opt('dmarc') returns nill >> located in rspamd/lualib/rspamadm/dmarc_report.lua >> >> configdump founds dmarc section without problems >> >> rspamadm configdump dmarc >> *** Section dmarc *** >> servers = "192.168.x.x:6379"; >> actions { >> quarantine = "add_header"; >> reject = "reject"; >> } >> reporting = true; >> reporting { >> report_local_controller = false; >> msgid_from = "rspamd"; >> enabled = true; >> keys_expire = 172800; >> domain = "*******"; >> max_entries = 1000; >> from_name = "DMARC reporting"; >> smtp = "127.0.0.1"; >> smtp_port = 25; >> email = "postmaster@**********"; >> helo = "rspamd.localhost"; >> org_name = "*********"; >> } >> >> > From timc at slowb.ro Fri Sep 17 07:50:18 2021 From: timc at slowb.ro (Tim C) Date: Fri, 17 Sep 2021 17:50:18 +1000 Subject: [Rspamd-Users] DMARC policies on emails that forward from mailing lists go to spam Message-ID: Hi Everyone, Has anyone figured out a way in which emails from mailing lists can be allowed when they fail the DMARC policy, as the SPF doesn't align? It seems to be a hit and miss for emails, and most likely it is due to the DMARC policy for the respective users. Is this something I can setup in rspamd? As ideally I'd like to keep rspamd anti-spam/anti-virus and not just sieve filter all mailing lists before they anti-spam has a chance to flag it. Cheers, Tim From thomas at plant.systems Fri Sep 17 08:44:52 2021 From: thomas at plant.systems (Thomas) Date: Fri, 17 Sep 2021 10:44:52 +0200 Subject: [Rspamd-Users] Disable symbol on certain filetype Message-ID: Hello. We have the problem as described in this github issue: https://github.com/rspamd/rspamd/issues/3837 When a file has a .p7m extension rspamd inserts BOGUS_ENCRYPTED_AND_TEXT and HEADER_BROKEN symbols and identifies the mail as spam. Now I have made a simple multipmap rule to add a negative score if a p7m file is detected: WHITELIST_FILE_EXTENSION { ??????? type = "filename"; ??????? filter = "extension"; ??????? map = "${LOCAL_CONFDIR}/maps/filename_extension_wl.map"; ??????? score = -30; ??????? symbol = "WHITELIST_FILE_EXTENSION"; } Is there a way to simply disable the two symbols BOGUS_ENCRYPTED_AND_TEXT/HEADER_BROKEN if we encounter a p7m file? Or is the above the better solution? Regards, Thomas From philip.colmer at linaro.org Mon Sep 20 07:38:09 2021 From: philip.colmer at linaro.org (Philip Colmer) Date: Mon, 20 Sep 2021 08:38:09 +0100 Subject: [Rspamd-Users] Help needed with DKIM and ARC (mis)configuration please Message-ID: I'm trying to configure rspamd so that it provides all of the DKIM and ARC handling when used in conjunction with Mailman 3. I'm using Postfix as the MTA. DKIM itself seems to be working, in that if Mailman 3 sends (say) a password reset email, the receiving mail system says that the DKIM headers are OK: Delivered-To: philip.colmer at codelinaro.org Received-SPF: pass (zohomail.com: domain of mm3.lavasoftware.org designates 3.230.84.86 as permitted sender) client-ip=3.230.84.86; envelope-from=postorius at mm3.lavasoftware.org; helo=mm3.lavasoftware.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of mm3.lavasoftware.org designates 3.230.84.86 as permitted sender) smtp.mailfrom=postorius at mm3.lavasoftware.org ARC-Seal: i=1; a=rsa-sha256; t=1632122989; cv=none; d=zohomail.com; s=zohoarc; b=ANrhqpZBbcyb5vTHyW8/oZGzKVK+fqUVpAoqRToZ0ybk6nplq5gcj2MWN12osnpIT8XQzKx1W1VZ7yTnmsxZgMczbgLgGgu5DrQU8sz7kjGOoiUEMicJCga82GGHK6X6cFNF2Am83S3Hs2PRoIuriaPggASbTqLZ4gzNHuLlavw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1632122989; h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To; bh=Na5VyfB/MJFBjvpUi2TPUqikg420+PNUB2IXz4sZ3tw=; b=Ftpt4gX7nhNfDsuwEKH3X2gmu0Fo446u9IRLut6Y93QCJL0dP2/5funAQhcT5RwBglyeH/LMMCs6tu+0sFnSDteNLPbkmbLjK9uZtCnk42+uBRUqEdbPy8DSqgxUclA9KwKvIkd3asld0c8yMxo7u9jMdnkBs6TxeCphReF3vho= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of mm3.lavasoftware.org designates 3.230.84.86 as permitted sender) smtp.mailfrom=postorius at mm3.lavasoftware.org Return-Path: Received: from mm3.lavasoftware.org (mm3.lavasoftware.org [3.230.84.86]) by mx.zohomail.com with SMTPS id 1632122989893975.8082437998135; Mon, 20 Sep 2021 00:29:49 -0700 (PDT) Received: from ip-172-31-2-177.ec2.internal (localhost [127.0.0.1]) by mm3.lavasoftware.org (Postfix) with ESMTP id 4B440BE547 for ; Mon, 20 Sep 2021 07:29:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mm3.lavasoftware.org; s=dkim; t=1632122988; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Na5VyfB/MJFBjvpUi2TPUqikg420+PNUB2IXz4sZ3tw=; b=k5LcpvNvZ3l0tCjoCy+T25HotZtilhzOtLmBUnC+nsja6gB/F/rrqTAYU3XcXKMvNATf2T gLrmn+kLVKv3dbCJyfaJyZqYIcK37klx/Cb88kv6m2vq6FSTtFiQZpzqGv9AvhPYK2BiZx +lHq0bmlJ6fp2jRbI5T9WbwtzzrJNUs= Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [lavasoftware.org] Password Reset E-mail From: postorius at mm3.lavasoftware.org To: philip.colmer at codelinaro.org Date: Mon, 20 Sep 2021 07:29:48 -0000 Message-ID: <163212298829.38042.11881199049009320543 at ip-172-31-2-177.ec2.internal> X-ZohoMail-DKIM: pass (identity @mm3.lavasoftware.org) X-ZohoMail-Owner: <163212298829.38042.11881199049009320543 at ip-172-31-2-177.ec2.internal>+zmo_0_postorius at mm3.lavasoftware.org However, if I post to a mailing list on the server, the received email is not considered to be OK: Delivered-To: philip.colmer at codelinaro.org Received-SPF: pass (zohomail.com: domain of mm3.lavasoftware.org designates 3.230.84.86 as permitted sender) client-ip=3.230.84.86; envelope-from=test-bounces+philip.colmer=codelinaro.org at mm3.lavasoftware.org; helo=mm3.lavasoftware.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of mm3.lavasoftware.org designates 3.230.84.86 as permitted sender) smtp.mailfrom=test-bounces+philip.colmer=codelinaro.org at mm3.lavasoftware.org; arc=fail (BodyHash is different from the expected one) Return-Path: Received: from mm3.lavasoftware.org (mm3.lavasoftware.org [3.230.84.86]) by mx.zohomail.com with SMTPS id 1632121826593429.3377835910852; Mon, 20 Sep 2021 00:10:26 -0700 (PDT) Received: from ip-172-31-2-177.ec2.internal (localhost [127.0.0.1]) by mm3.lavasoftware.org (Postfix) with ESMTP id ABFAFBEA8D for ; Mon, 20 Sep 2021 07:10:25 +0000 (UTC) Received: from sender4-op-o14.zoho.com (sender4-op-o14.zoho.com [136.143.188.14]) by mm3.lavasoftware.org (Postfix) with ESMTPS id B5FC1BE61C for ; Mon, 20 Sep 2021 07:10:22 +0000 (UTC) Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1632121819028942.4423330943655; Mon, 20 Sep 2021 00:10:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mm3.lavasoftware.org; s=dkim; t=1632121825; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-owner: list-unsubscribe:list-subscribe:list-post; bh=zJ40PoiQolFhFDNtGN6hEUNOJtDCnSu+T+9+9qkGBJ4=; b=WlzzQQsEEiRUF2Bw3fKEHZ/v7CC0euY/qUEwLjhOy0jPNoKhCE1YuO+O2+AcaUAxneufqN RzZEPtbIYZm29w1jG5ZYoNmiDA42+v8ZasIagWc50+ryWDnX4YSSYk4RMBCXwznu5HBMjL DrRDPq4Q92WdPKsYho3ABb97Hzl4mlI= ARC-Seal: i=1; a=rsa-sha256; t=1632121820; cv=none; d=zohomail.com; s=zohoarc; b=I7Nm7s4ikypN7iEcMzmWleseGtYzQUx2e72QEwV2hXKXxVQs0yrbx2744dcbNFAJgOMJl7C0PMFWVEMTm5uiFxN+zLwpvNJ5i9Sw5XQXNzkP4yuVBGC8R2+ZJ+I2W8e3dxsdOp1SZL7ROMEnbISXaIKLWAOmB8H0Cu8etwTuCHA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1632121820; h=Content-Type:Date:From:MIME-Version:Message-ID:Subject:To; bh=UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=; b=DzYG0hVPiGOGagfchCVHJFkl1R9H2EvvB338GBsKGauzUtwLSIs3clIyoyr3vLTG2uBZYHZGSZk1JeODyAg/Y9x8UluqA1V4kUU5TITVyaX3dUslHhaXW5FP5ds1hK4+08lLan9SLKXtRrfNPonGG8B9umRieCpno8AcodDo+Ak= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=codelinaro.org; spf=pass smtp.mailfrom=philip.colmer at codelinaro.org; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1632121820; s=zoho; d=codelinaro.org; i=philip.colmer at codelinaro.org; h=Date:From:To:Message-Id:Subject:MIME-Version:Content-Type; bh=UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=; b=Ay3K1SsWxWR+ehw1xplPkf4c61D/hsK3rNDtV/gl1SpH5odBdVQtn+tD4X/jmSzo v4dlnV5v/SEu5bEKcx1+1t+jhOlqKoxpip2/O3i75KRbQutUoGKmS46c9Pd6l8q4/ww k7z05XUAHJIwOE0GcLJ+75UMvjPoOZTPNX2Xuk0k= Date: Mon, 20 Sep 2021 08:10:19 +0100 To: "test" Message-Id: <17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org> MIME-Version: 1.0 Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=mm3.lavasoftware.org; s=dkim; t=1632121823; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: dkim-signature; bh=UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=; b=kd2l8DuYzeIwb+IV2iAe4QPqaTbpc11il+5GRnG6n38/kj2o7wbsbxSz/Xc9K2GndFHMcC zq8VXuca+inWtDsbiPPPUOP4T8HhfbhXOpCFGFrkb5kdCxju8dJ+HZeExVfgAQkPONBQIU NAUuuls+jnysLBLmeg8yy1lFOGXHKq8= ARC-Authentication-Results: i=2; mm3.lavasoftware.org; dkim=pass header.d=codelinaro.org header.s=zoho header.b=Ay3K1SsW; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1"); spf=pass (mm3.lavasoftware.org: domain of philip.colmer at codelinaro.org designates 136.143.188.14 as permitted sender) smtp.mailfrom=philip.colmer at codelinaro.org ARC-Seal: i=2; s=dkim; d=mm3.lavasoftware.org; t=1632121823; a=rsa-sha256; cv=pass; b=sL4/9K5tcg3NqHS1Cn5HMUFqO02DB43na23WyE3ke6AB1igpY/1o7xDYvt6Vqu9OMc8X/7 Tn75o1hi2cUrXD2Ju40FxirvG3Rwa47CzfvrHRXv6KhaYAe01Qbb1zmKe6ECV1MZxRy/OA 52z4ccr1QBVCAM/j6BSSgNcO3uk13A4= Message-ID-Hash: DJXYU37XDC54P2YNVHOUQOM5ILEFDLGG X-Message-ID-Hash: DJXYU37XDC54P2YNVHOUQOM5ILEFDLGG X-MailFrom: philip.colmer at codelinaro.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.4 Precedence: list Subject: [Test] Another ARC test email List-Id: Archived-At: <> List-Archive: <> List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Philip Colmer via Test Reply-To: Philip Colmer Content-Type: multipart/mixed; boundary="===============4180264220719142576==" X-ZohoMail-DKIM: pass (identity @mm3.lavasoftware.org) local.d/arc.conf: allow_hdrfrom_mismatch = true; allow_username_mismatch = true; use_esld = false; domain { mm3.lavasoftware.org { selector = "dkim"; path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key"; } } local.d/dkim_signing.conf: allow_hdrfrom_mismatch_sign_networks = true; allow_username_mismatch = true; domain { mm3.lavasoftware.org { selector = "dkim"; path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key"; } } use_esld = false; sign_authenticated = false; use_domain = "header"; allow_hdrfrom_mismatch = true; local.d/worker-proxy.inc: milter = yes; # Enable milter mode timeout = 120s; # Needed for Milter usually upstream "local" { default = yes; # Self-scan upstreams are always default self_scan = yes; # Enable self-scan } count = 4; # Spawn more processes in self-scan mode max_retries = 5; # How many times master is queried in case of failure discard_on_reject = false; # Discard message instead of rejection quarantine_on_reject = false; # Tell MTA to quarantine rejected messages spam_header = "X-Spam"; # Use the specific spam header reject_message = "Spam message rejected"; # Use custom rejection message local.d/worker-normal.inc: enabled = false; Postfix configuration extract for the milter: #smtpd_milters = unix:/var/lib/rspamd/milter.sock # or for TCP socket smtpd_milters = inet:localhost:11332 non_smtpd_milters = inet:localhost:11332 # skip mail without checks if something goes wrong milter_default_action = accept # 6 is the default milter protocol version; # prior to Postfix 2.6 the default protocol was 2. # milter_protocol = 6 When the email originally comes in to be submitted to the mailing list, I think these are the relevant lines from the rspamd log: 2021-09-20 07:10:22 #37324(rspamd_proxy) <8e1c5c>; proxy; proxy_accept_socket: accepted milter connection from 127.0.0.1 port 36088 2021-09-20 07:10:22 #37324(rspamd_proxy) <8e1c5c>; milter; rspamd_milter_process_command: got connection from 136.143.188.14:17451 2021-09-20 07:10:22 #37324(rspamd_proxy) <8e1c5c>; proxy; rspamd_message_parse: loaded message; id: <17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>; queue-id: ; size: 2400; checksum: <9f4bf024cb485cbef8e1b1e030a48b9b> 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy; rspamd_spf_maybe_return: stored record for codelinaro.org (0x4d20af7734be3ae1) in LRU cache for 300 seconds, 2/2000 elements in the cache 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy; dkim_module_key_handler: stored DKIM key for zoho._domainkey.codelinaro.org in LRU cache for 300 seconds, 2/2000 elements in the cache 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; lua; greylist.lua:318: Score too low - skip greylisting 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; lua; neural.lua:315: skip ham sample to keep spam/ham balance; probability 1; 0 spam and 1 ham vectors stored 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy; rspamd_task_write_log: id: <17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>, qid: , ip: 136.143.188.14, from: , (default: F (no action): [-1.49/15.00] [ARC_ALLOW(-1.00){zohomail.com:s=zohoarc:i=1;},R_DKIM_ALLOW(-0.20){codelinaro.org:s=zoho;},R_SPF_ALLOW(-0.20){+ip4:136.143.188.0/24;},MIME_GOOD(-0.10){multipart/alternative;text/plain;},XM_UA_NO_VERSION(0.01){},ARC_SIGNED(0.00){mm3.lavasoftware.org:s=dkim:i=2;},ASN(0.00){asn:2639, ipnet:136.143.188.0/23, country:US;},DKIM_TRACE(0.00){codelinaro.org:+;},DMARC_NA(0.00){codelinaro.org;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:~;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_TWO(0.00){2;},RCVD_IN_DNSWL_NONE(0.00){136.143.188.14:from;},RCVD_TLS_LAST(0.00){},RWL_MAILSPIKE_POSSIBLE(0.00){136.143.188.14:from;},TO_DN_ALL(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 2400, time: 294.179ms, dns req: 27, digest: <9f4bf024cb485cbef8e1b1e030a48b9b>, rcpts: , mime_rcpts: 2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned using pcre, 1.00KiB scanned total 2021-09-20 07:10:23 #37324(rspamd_proxy) <378793>; proxy; proxy_milter_finish_handler: finished milter connection Then, when Mailman is sending the modified email back out, these are the relevant lines: 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; proxy_accept_socket: accepted milter connection from 127.0.0.1 port 36142 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; milter; rspamd_milter_process_command: got connection from 127.0.0.1:54506 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; rspamd_message_parse: loaded message; id: <17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>; queue-id: <5C73ABE61C>; size: 5079; checksum: 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; rspamd_mime_part_detect_language: detected part language: en 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua; greylist.lua:204: skip greylisting for local networks and/or authorized users 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; dkim_symbol_callback: skip DKIM checks for local networks and authorized users 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua; spf.lua:186: skip SPF checks for local networks and authorized users 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua; dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not checked 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua; once_received.lua:99: Skipping once_received for authenticated user or local network 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; dkim; rspamd_dkim_check: arc_sig: bh value mismatch: got lBu5g1QPVcNe04bBZ1chCV35tI3E+GY1OFtuw7mUsAY=, expected UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=; body length 1081->1079; d=mm3.lavasoftware.org; s=dkim 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua; greylist.lua:318: Score too low - skip greylisting 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; rspamd_task_write_log: id: <17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>, qid: <5C73ABE61C>, ip: 127.0.0.1, from: , (default: F (no action): [0.80/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] = sig:mm3.lavasoftware.org:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){test at mm3.lavasoftware.org;test-bounces at mm3.lavasoftware.org;},HAS_REPLYTO(0.00){philip.colmer at codelinaro.org;},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},REPLYTO_DOM_NEQ_FROM_DOM(0.00){},TAGGED_FROM(0.00){philip.colmer=linaro.org;},TO_DN_ALL(0.00){},TO_EQ_FROM(0.00){}]), len: 5079, time: 236.610ms, dns req: 13, digest: , rcpts: , mime_rcpts: 2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 4 regexps matched, 176 regexps total, 86 regexps cached, 0B scanned using pcre, 2.01KiB scanned total 2021-09-20 07:10:25 #37321(rspamd_proxy) ; proxy; rspamd_message_parse: loaded message; id: <17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>; queue-id: ; size: 5079; checksum: 2021-09-20 07:10:25 #37321(rspamd_proxy) ; proxy; rspamd_mime_part_detect_language: detected part language: en 2021-09-20 07:10:25 #37321(rspamd_proxy) ; lua; greylist.lua:204: skip greylisting for local networks and/or authorized users 2021-09-20 07:10:25 #37321(rspamd_proxy) ; proxy; dkim_symbol_callback: skip DKIM checks for local networks and authorized users 2021-09-20 07:10:25 #37321(rspamd_proxy) ; lua; spf.lua:186: skip SPF checks for local networks and authorized users 2021-09-20 07:10:25 #37321(rspamd_proxy) ; dkim; rspamd_dkim_check: arc_sig: bh value mismatch: got zJ40PoiQolFhFDNtGN6hEUNOJtDCnSu+T+9+9qkGBJ4=, expected UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=; body length 1081->1079; d=mm3.lavasoftware.org; s=dkim 2021-09-20 07:10:25 #37321(rspamd_proxy) ; lua; dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not checked 2021-09-20 07:10:25 #37321(rspamd_proxy) ; lua; once_received.lua:99: Skipping once_received for authenticated user or local network 2021-09-20 07:10:25 #37321(rspamd_proxy) ; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required 2021-09-20 07:10:25 #37321(rspamd_proxy) ; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required 2021-09-20 07:10:25 #37321(rspamd_proxy) ; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing 2021-09-20 07:10:25 #37321(rspamd_proxy) ; lua; greylist.lua:318: Score too low - skip greylisting 2021-09-20 07:10:25 #37321(rspamd_proxy) ; proxy; rspamd_task_write_log: id: <17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>, qid: , ip: 127.0.0.1, from: , (default: F (no action): [0.79/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] = sig:mm3.lavasoftware.org:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){test at mm3.lavasoftware.org;test-bounces at mm3.lavasoftware.org;},HAS_REPLYTO(0.00){philip.colmer at codelinaro.org;},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},REPLYTO_DOM_NEQ_FROM_DOM(0.00){},TAGGED_FROM(0.00){philip.colmer=codelinaro.org;},TO_DN_ALL(0.00){},TO_EQ_FROM(0.00){}]), len: 5079, time: 11.408ms, dns req: 12, digest: , rcpts: , mime_rcpts: 2021-09-20 07:10:25 #37321(rspamd_proxy) ; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 4 regexps matched, 176 regexps total, 86 regexps cached, 0B scanned using pcre, 2.01KiB scanned total 2021-09-20 07:10:25 #37321(rspamd_proxy) <631f9b>; proxy; proxy_milter_finish_handler: finished milter connection 2021-09-20 07:16:14 #37324(rspamd_proxy) <52c995>; proxy; proxy_accept_socket: accepted milter connection from 127.0.0.1 port 39796 2021-09-20 07:16:14 #37324(rspamd_proxy) <52c995>; milter; rspamd_milter_process_command: got connection from 35.179.93.71:47834 2021-09-20 07:16:14 #37324(rspamd_proxy) <52c995>; proxy; proxy_milter_finish_handler: finished milter connection 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; proxy_accept_socket: accepted milter connection from 127.0.0.1 port 48300 2021-09-20 07:29:48 #37324(rspamd_proxy) ; milter; rspamd_milter_process_command: got connection from 127.0.0.1:38432 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; rspamd_message_parse: loaded message; id: <163212298829.38042.11881199049009320543 at ip-172-31-2-177.ec2.internal>; queue-id: <4B440BE547>; size: 816; checksum: 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; rspamd_mime_part_detect_language: detected part language: en 2021-09-20 07:29:48 #37324(rspamd_proxy) ; lua; greylist.lua:204: skip greylisting for local networks and/or authorized users 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; dkim_symbol_callback: skip DKIM checks for local networks and authorized users 2021-09-20 07:29:48 #37324(rspamd_proxy) ; lua; spf.lua:186: skip SPF checks for local networks and authorized users 2021-09-20 07:29:48 #37324(rspamd_proxy) ; lua; dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not checked 2021-09-20 07:29:48 #37324(rspamd_proxy) ; lua; once_received.lua:99: Skipping once_received for authenticated user or local network 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing 2021-09-20 07:29:48 #37324(rspamd_proxy) ; lua; greylist.lua:318: Score too low - skip greylisting 2021-09-20 07:29:48 #37324(rspamd_proxy) ; lua; neural.lua:315: skip ham sample to keep spam/ham balance; probability 1; 0 spam and 1 ham vectors stored 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; rspamd_task_write_log: id: <163212298829.38042.11881199049009320543 at ip-172-31-2-177.ec2.internal>, qid: <4B440BE547>, ip: 127.0.0.1, from: , (default: F (no action): [-0.10/15.00] [MIME_GOOD(-0.10){text/plain;},ARC_NA(0.00){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FROM_EQ_ENVFROM(0.00){},FROM_NO_DN(0.00){},MIME_TRACE(0.00){0:+;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_ZERO(0.00){0;},TO_DN_NONE(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 816, time: 290.349ms, dns req: 15, digest: , rcpts: , mime_rcpts: 2021-09-20 07:29:48 #37324(rspamd_proxy) ; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 2 regexps matched, 176 regexps total, 47 regexps cached, 0B scanned using pcre, 1.89KiB scanned total 2021-09-20 07:29:48 #37324(rspamd_proxy) <2f90dc>; proxy; proxy_milter_finish_handler: finished milter connection 2021-09-20 07:36:16 #37323(rspamd_proxy) <06ee59>; proxy; proxy_accept_socket: accepted milter connection from 127.0.0.1 port 52314 2021-09-20 07:36:16 #37323(rspamd_proxy) <06ee59>; milter; rspamd_milter_process_command: got connection from 18.170.226.166:21345 2021-09-20 07:36:16 #37323(rspamd_proxy) <06ee59>; proxy; proxy_milter_finish_handler: finished milter connection I realise there is a lot to digest there but does anyone have any suggestions on what I've misconfigured, please? Thanks. Regards Philip From chris at cretaforce.gr Mon Sep 20 14:52:13 2021 From: chris at cretaforce.gr (Christos Chatzaras) Date: Mon, 20 Sep 2021 17:52:13 +0300 Subject: [Rspamd-Users] rate limit and mail delivery reports Message-ID: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr> Hello, I use two postfix relays to send e-mails from other servers. As I can't use selector 'user.lower' because the relays don't have users, my ratelimit.conf contains: rates { "12000" = { selector = 'from.lower'; bucket = { burst = 12000; rate = "12000 / 1d"; } } } The problem is that if many senders ask for "delivery reports" then this rate limit is hit because all these reports use "From: MAILER-DAEMON at smtp1.example.com" or "From: MAILER-DAEMON at smtp2.example.com". Is any way to exclude MAILER-DAEMON at smtp1.example.com and MAILER-DAEMON at smtp2.example.com from the rate limit? From kyle at cci1986.com Mon Sep 20 17:05:25 2021 From: kyle at cci1986.com (Kyle A.) Date: Mon, 20 Sep 2021 13:05:25 -0400 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: <5710476131.20210913170548@sveiks.lv> References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> Message-ID: Aste, Did you get it resolved? Today I have upgraded V2.7 -> V3.0, updated the dmarc.conf, and I have almost the exact same error you reported. Mine says this: # rspamadm dmarc_report call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - [Lua]}; Upon further review, I think my error is character for character the same as your error. I do not understand and there is no help for this. -Kyle On 9/13/2021 10:05 AM, Aste wrote: > Thank you it's solve initial problem, bet reporting still doesn't work, now I getting error > Executing "rspamadm dmarc_report" results with error. > > call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - [Lua]}; > From m.kliewe at team.mail.de Mon Sep 20 20:39:41 2021 From: m.kliewe at team.mail.de (Michael Kliewe) Date: Mon, 20 Sep 2021 22:39:41 +0200 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> Message-ID: Hi, Am 20.09.2021 um 19:05 schrieb Kyle A.: > Aste, > > Did you get it resolved? > > Today I have upgraded V2.7 -> V3.0, updated the dmarc.conf, and I have > almost the exact same error you reported.? Mine says this: > > # rspamadm dmarc_report > call to rspamadm lua script failed (2): > /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid > argument: nil; trace: > [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - > process_report_entry [Lua]}; > [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - > prepare_report [Lua]}; > [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - > process_report_date [Lua]}; > [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - > [Lua]}; > > Upon further review, I think my error is character for character the > same as your error.? I do not understand and there is no help for this. I also got this error when I first manually ran sudo rspamadm dmarc_report after upgrading from 2.7 to 3.0.2 on Ubuntu. But: One day later I ran it again manually: sudo rspamadm dmarc_report -v (with verbose mode on) and it was working fine, it sent 10 DMARC reports. Maybe there was some old/invalid data in the redis storage, which is why the first invocation failed with "invalid argument: nil"? Please do us all a favor and don't run the cronjob at exactly midnight. Choose a random time in a day, then rate-limits at recipient mailboxes will not be a problem, and server load is distributed. If you receive bounces because the recipient mailbox is over quota or does not exist, or does not accept attachments, try to contact the recipient somehow and tell them about the problem. We need to get those bounce-problems fixed together. Michael > > On 9/13/2021 10:05 AM, Aste wrote: >> Thank you it's solve? initial problem, bet reporting still doesn't >> work, now I getting error >> Executing "rspamadm dmarc_report" results with error. >> >> call to rspamadm lua script failed (2): >> /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid >> argument: nil; trace: >> [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - >> process_report_entry [Lua]}; >> [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - >> prepare_report [Lua]}; >> [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - >> process_report_date [Lua]}; >> [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - >> [Lua]}; From bruns at 2mbit.com Mon Sep 20 20:30:13 2021 From: bruns at 2mbit.com (Brielle) Date: Mon, 20 Sep 2021 14:30:13 -0600 Subject: [Rspamd-Users] rspamd not handling pause / multi-line 220- in SMTP exchange Message-ID: <9830de2e-0916-2238-ec5f-22ed7f742bf1@2mbit.com> Hello, Discovered a slight issue with rspamd's dmarc_report feature, getting following errors: Couldn't send mail for inmoment.com: bad smtp responce on stage rcpt: "220-" when "3" expected When checking exim's logs... 2021-09-20 14:19:05 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "HELO mail.xxxxxx.org" H=localhost [127.0.0.1] U=root next input="MAIL FROM: \r\nRCPT TO: \r\nDATA\r\n" My SMTP server adds a momentary delay before spitting out a multi line 220 banner (using 220-) which is designed to mess with spambots that don't bother to implement proper RFC compliance. Looks like the report feature either doesn't wait for the final 220 before sending the HELO, or doesn't recognize that 220- means multi line (see RFC 2821 sec 3.1) and just starts trying to send commands right away. So, not sure if its the initial delay that's causing the problem or the 220 handling. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org From chris at cretaforce.gr Mon Sep 20 20:56:31 2021 From: chris at cretaforce.gr (Christos Chatzaras) Date: Mon, 20 Sep 2021 23:56:31 +0300 Subject: [Rspamd-Users] rate limit and mail delivery reports In-Reply-To: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr> References: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr> Message-ID: <8F97ABAD-AC79-4C6A-B92F-EA748E260E7B@cretaforce.gr> > On 20 Sep 2021, at 17:52, Christos Chatzaras wrote: > > Hello, > > I use two postfix relays to send e-mails from other servers. > > As I can't use selector 'user.lower' because the relays don't have users, my ratelimit.conf contains: > > rates { > "12000" = { > selector = 'from.lower'; > bucket = { > burst = 12000; > rate = "12000 / 1d"; > } > } > } > > The problem is that if many senders ask for "delivery reports" then this rate limit is hit because all these reports use "From: MAILER-DAEMON at smtp1.example.com" or "From: MAILER-DAEMON at smtp2.example.com". Is any way to exclude MAILER-DAEMON at smtp1.example.com and MAILER-DAEMON at smtp2.example.com from the rate limit? > Finally I made some changes to my setup to be able to use 'user.lower' selector. From helge.wiethoff at thga.de Tue Sep 21 07:13:39 2021 From: helge.wiethoff at thga.de (Wiethoff, Helge) Date: Tue, 21 Sep 2021 07:13:39 +0000 Subject: [Rspamd-Users] rate limit and mail delivery reports In-Reply-To: <8F97ABAD-AC79-4C6A-B92F-EA748E260E7B@cretaforce.gr> References: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr> <8F97ABAD-AC79-4C6A-B92F-EA748E260E7B@cretaforce.gr> Message-ID: Hi Christos, Am Montag, dem 20.09.2021 um 23:56 +0300 schrieb Christos Chatzaras: > Finally I made some changes to my setup to be able to use 'user.lower' selector. Could you be so kind and post your changes? :-) Best Helge From chris at cretaforce.gr Tue Sep 21 07:42:45 2021 From: chris at cretaforce.gr (Christos Chatzaras) Date: Tue, 21 Sep 2021 10:42:45 +0300 Subject: [Rspamd-Users] rate limit and mail delivery reports In-Reply-To: References: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr> <8F97ABAD-AC79-4C6A-B92F-EA748E260E7B@cretaforce.gr> Message-ID: <247A76EC-B946-4403-B91B-92DA6C456E39@cretaforce.gr> > On 21 Sep 2021, at 10:13, Wiethoff, Helge wrote: > > Hi Christos, > > Am Montag, dem 20.09.2021 um 23:56 +0300 schrieb Christos Chatzaras: >> Finally I made some changes to my setup to be able to use 'user.lower' selector. > > Could you be so kind and post your changes? :-) > > Best > Helge My setup was: 1) For my servers I was using rspamd only for smtp and not for submission. So I was scanning incoming messages on the servers and not outgoing. 2) Outgoing e-mails were forwarded to two relays that do the final delivery. In these relays I was using rspamd to check outgoing e-mails. My new setup is: 1) My servers do rspamd checks for both smtp and for submission. So I scan both incoming and outgoing messages on the server level. 2) Outgoing e-mails are forwarded to two relays but as e-mails already scanned on the server level I remove scanning at the relay level. With my old setup and "from.lower" selector rate limiting was enabled in both incoming and outgoing e-mails which caused me this issue with mail delivery reports when this limit hit for incoming e-mails. With these changes all scanning is done in the server level and by using "user.lower" instead of "from.lower" I have rate limiting enabled only for outgoing e-mails and not for incoming e-mails. From aste at sveiks.lv Tue Sep 21 07:59:51 2021 From: aste at sveiks.lv (Aste) Date: Tue, 21 Sep 2021 10:59:51 +0300 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> Message-ID: <1994181029.20210921105951@latnet.lv> Hi! Monday, September 20, 2021, 8:05:25 PM, you wrote: KA> Aste, KA> Did you get it resolved? Didn't find solution, revertet back to 2.7 with some bugs in DMARC reporting, but at least this version can send reports. -- Best regards Aste KA> Today I have upgraded V2.7 -> V3.0, updated the dmarc.conf, and I have almost the exact same error you reported. Mine says this: KA> # rspamadm dmarc_report KA> call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - [Lua]}; KA> Upon further review, I think my error is character for character the same as your error. I do not understand and there is no help for this. KA> -Kyle KA> On 9/13/2021 10:05 AM, Aste wrote: >> Thank you it's solve initial problem, bet reporting still doesn't work, now I getting error >> Executing "rspamadm dmarc_report" results with error. >> > call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - [Lua]}; >> -- KA> Users mailing list KA> Users at lists.rspamd.com KA> https://lists.rspamd.com/mailman/listinfo/users From aste at sveiks.lv Tue Sep 21 08:21:21 2021 From: aste at sveiks.lv (Aste) Date: Tue, 21 Sep 2021 11:21:21 +0300 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> Message-ID: <744144092.20210921112121@latnet.lv> Hi! Monday, September 20, 2021, 11:39:41 PM, you wrote: MK> I also got this error when I first manually ran MK> sudo rspamadm dmarc_report MK> after upgrading from 2.7 to 3.0.2 on Ubuntu. MK> But: MK> One day later I ran it again manually: MK> sudo rspamadm dmarc_report -v MK> (with verbose mode on) and it was working fine, it sent 10 DMARC reports. MK> Maybe there was some old/invalid data in the redis storage, which is why the first invocation failed with "invalid argument: nil"? I was changed redis prefix to avoid usage of old data, but it doesn't help. Also after 2 days the same error. Some parts of script was working, because I get errors about misconfigured domains, but at the end always the same script erros and no reports was sent. -- Best regards Aste From a.wass at glas-gasperlmair.at Tue Sep 21 09:53:20 2021 From: a.wass at glas-gasperlmair.at (Andreas Wass - Glas Gasperlmair) Date: Tue, 21 Sep 2021 11:53:20 +0200 Subject: [Rspamd-Users] multimap - content filter not working Message-ID: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at> Hi, i'm trying to use rspamd as soon as possible for our company mailserver, and testing content filtering with multimap as shown in https://rspamd.com/doc/modules/multimap.html *Configured the following.*.. Multiple symbol maps From the version 1.3.1, it is possible to define multiple symbols and scores using multimap module. To do that, you should define all possible symbols using|symbols|option in multimap: |# local.d/multimap.conf CONTENT_BLACKLISTED { type = "content"; filter = "body"; # can be headers, full, oneline, text, rawtext map = "${LOCAL_CONFDIR}/local.d/local_content.map"; symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; regexp = true; } | In this example, you can use 3 symbols: * CONTENT_BLACKLISTED * CONTENT_BLACKLISTED1 * CONTENT_BLACKLISTED2 the map: |# Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: CONTENT_BLACKLISTED /re3/ | Symbols that are not defined in the|symbols|attribute but used in the map are ignored and replaced by the default map symbol. If the value of a key-value pair is missing, then Rspamd just inserts the default symbol with dynamic weight equal to|1.0|(which is multiplied by metric score afterwards) *...matches the right symbol (CONTENT_BLACKLISTED1) in the header :* X-Spamd-Result: default: False [0.80 / 15.00]; ... CONTENT_BLACKLISTED1(0.00)[]; ... *...but why is score 0.00 and not 10 like it is defined in local_content.map?* best regards, Andy From a.wass at glas-gasperlmair.at Tue Sep 21 10:43:13 2021 From: a.wass at glas-gasperlmair.at (Andreas Wass - Glas Gasperlmair) Date: Tue, 21 Sep 2021 12:43:13 +0200 Subject: [Rspamd-Users] multimap - content filter not working In-Reply-To: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at> References: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at> Message-ID: sorry, something went wrong with formating in my first post Hi, i'm trying to use rspamd as soon as possible for our company mailserver, and testing content filtering with multimap as shown in https://rspamd.com/doc/modules/multimap.html Configured the following... Multiple symbol maps From the version 1.3.1, it is possible to define multiple symbols and scores using multimap module. To do that, you should define all possible symbols using symbols option in multimap: # local.d/multimap.conf CONTENT_BLACKLISTED { ? type = "content"; ? filter = "body"; # can be headers, full, oneline, text, rawtext ? map = "${LOCAL_CONFDIR}/local.d/local_content.map"; ? symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; ? regexp = true; } In this example, you can use 3 symbols: ??? CONTENT_BLACKLISTED ??? CONTENT_BLACKLISTED1 ??? CONTENT_BLACKLISTED2 the map: # Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: CONTENT_BLACKLISTED /re3/ Symbols that are not defined in the symbols attribute but used in the map are ignored and replaced by the default map symbol. If the value of a key-value pair is missing, then Rspamd just inserts the default symbol with dynamic weight equal to 1.0 (which is multiplied by metric score afterwards) ...matches the right symbol (CONTENT_BLACKLISTED1) in the header : X-Spamd-Result: default: False [0.80 / 15.00]; ??? ... ??? CONTENT_BLACKLISTED1(0.00)[]; ??? ... ...but why is score 0.00 and not 10 like it is defined in local_content.map? best regards, Andy Am 21.09.2021 um 11:53 schrieb Andreas Wass - Glas Gasperlmair: > Hi, > > i'm trying to use rspamd as soon as possible for our company > mailserver, and testing content filtering with multimap as shown in > > https://rspamd.com/doc/modules/multimap.html > > *Configured the following.*.. > > > ?? Multiple symbol maps > > From the version 1.3.1, it is possible to define multiple symbols and > scores using multimap module. To do that, you should define all > possible symbols using|symbols|option in multimap: > > |# local.d/multimap.conf CONTENT_BLACKLISTED { type = "content"; > filter = "body"; # can be headers, full, oneline, text, rawtext map = > "${LOCAL_CONFDIR}/local.d/local_content.map"; symbols = > ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; regexp = true; } | > > In this example, you can use 3 symbols: > > ?* CONTENT_BLACKLISTED > ?* CONTENT_BLACKLISTED1 > ?* CONTENT_BLACKLISTED2 > > the map: > > |# Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default > score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: > CONTENT_BLACKLISTED /re3/ | > > Symbols that are not defined in the|symbols|attribute but used in the > map are ignored and replaced by the default map symbol. If the value > of a key-value pair is missing, then Rspamd just inserts the default > symbol with dynamic weight equal to|1.0|(which is multiplied by metric > score afterwards) > > > *...matches the right symbol (CONTENT_BLACKLISTED1) in the header :* > > X-Spamd-Result: default: False [0.80 / 15.00]; > ????... > ????CONTENT_BLACKLISTED1(0.00)[]; > ????... > > *...but why is score 0.00 and not 10 like it is defined in > local_content.map?* > > > best regards, Andy > From aste at sveiks.lv Tue Sep 21 11:12:20 2021 From: aste at sveiks.lv (Aste) Date: Tue, 21 Sep 2021 14:12:20 +0300 Subject: [Rspamd-Users] multimap - content filter not working In-Reply-To: References: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at> Message-ID: <1772880288.20210921141220@latnet.lv> Hi! Tuesday, September 21, 2021, 1:43:13 PM, you wrote: AWGG> # local.d/multimap.conf AWGG> CONTENT_BLACKLISTED { AWGG> ? type = "content"; AWGG> ? filter = "body"; # can be headers, full, oneline, text, rawtext AWGG> ? map = "${LOCAL_CONFDIR}/local.d/local_content.map"; According to documentation you need to specify map type as multimap map = regexp_multi; ${LOCAL_CONFDIR}/local.d/local_content.map; And you always can add global score there score = 10; AWGG> ? symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; AWGG> ? regexp = true; AWGG> } AWGG> In this example, you can use 3 symbols: AWGG> ??? CONTENT_BLACKLISTED AWGG> ??? CONTENT_BLACKLISTED1 AWGG> ??? CONTENT_BLACKLISTED2 AWGG> the map: AWGG> # Symbol + score AWGG> /re1/ CONTENT_BLACKLISTED1:10 AWGG> # Symbol with default score AWGG> /re2/ CONTENT_BLACKLISTED2 AWGG> # Just a default symbol: CONTENT_BLACKLISTED AWGG> /re3/ AWGG> Symbols that are not defined in the symbols attribute but used in the map are ignored and replaced by the default map symbol. If the value of a key-value pair is missing, then Rspamd just inserts the default symbol with dynamic weight equal to 1.0 (which is multiplied by metric score afterwards) AWGG> ...matches the right symbol (CONTENT_BLACKLISTED1) in the header : AWGG> X-Spamd-Result: default: False [0.80 / 15.00]; AWGG> ??? ... AWGG> ??? CONTENT_BLACKLISTED1(0.00)[]; AWGG> ??? ... AWGG> ...but why is score 0.00 and not 10 like it is defined in local_content.map? AWGG> best regards, Andy AWGG> Am 21.09.2021 um 11:53 schrieb Andreas Wass - Glas Gasperlmair: >> Hi, >> >> i'm trying to use rspamd as soon as possible for our company > mailserver, and testing content filtering with multimap as shown in >> >> https://rspamd.com/doc/modules/multimap.html >> >> *Configured the following.*.. >> >> >> ?? Multiple symbol maps >> >> From the version 1.3.1, it is possible to define multiple symbols and > scores using multimap module. To do that, you should define all > possible symbols using|symbols|option in multimap: >> >> |# local.d/multimap.conf CONTENT_BLACKLISTED { type = "content"; > filter = "body"; # can be headers, full, oneline, text, rawtext map = > "${LOCAL_CONFDIR}/local.d/local_content.map"; symbols = > ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; regexp = true; } | >> >> In this example, you can use 3 symbols: >> >> ?* CONTENT_BLACKLISTED >> ?* CONTENT_BLACKLISTED1 >> ?* CONTENT_BLACKLISTED2 >> >> the map: >> >> |# Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default > score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: > CONTENT_BLACKLISTED /re3/ | >> >> Symbols that are not defined in the|symbols|attribute but used in the > map are ignored and replaced by the default map symbol. If the value > of a key-value pair is missing, then Rspamd just inserts the default > symbol with dynamic weight equal to|1.0|(which is multiplied by metric > score afterwards) >> >> >> *...matches the right symbol (CONTENT_BLACKLISTED1) in the header :* >> >> X-Spamd-Result: default: False [0.80 / 15.00]; >> ????... >> ????CONTENT_BLACKLISTED1(0.00)[]; >> ????... >> >> *...but why is score 0.00 and not 10 like it is defined in > local_content.map?* >> >> >> best regards, Andy >> -- Best regards Aste From a.wass at glas-gasperlmair.at Tue Sep 21 12:49:09 2021 From: a.wass at glas-gasperlmair.at (Andreas Wass - Glas Gasperlmair) Date: Tue, 21 Sep 2021 14:49:09 +0200 Subject: [Rspamd-Users] multimap - content filter not working In-Reply-To: <1772880288.20210921141220@latnet.lv> References: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at> <1772880288.20210921141220@latnet.lv> Message-ID: <46c0cd9b-77ac-a289-1416-0d2a186e0330@glas-gasperlmair.at> found a solution and did it this way: 1. step vi /etc/rspamd/local.d/multimap.conf CONTENT_BLACKLISTED { ? type = "content"; ? filter = "body"; # can be headers, full, oneline, text, rawtext ? map = "${LOCAL_CONFDIR}/local.d/local_content.map"; ? symbols = ["CONTENT_BLACKLISTED1"]; ? regexp = true; } 2. step vi /etc/rspamd/local.d/groups.conf group "MyGroup" { ? symbols =? { ??? "CONTENT_BLACKLISTED1" { ????? # score = Multiplikator und wird mit Wert in local_content.map (steht hinter dem Doppelpunkt z.B. /Spende/ CONTENT_BLACKLISTED1:10) multipliziert ????? score = 1.0; ??? } ? } } 3. step vi /etc/rspamd/local.d/local_content.map # searchtext (case sensitiv) - Symbol - Score # Score wird mit Multiplikator in /etc/rspamd/local.d/groups.conf multipliziert # Score >= 15 rejecten /Wir haben eine Spende f?r Sie/ CONTENT_BLACKLISTED1:30 # ab Score >= 6 soft reject (greylist) /Die Spende/ CONTENT_BLACKLISTED1:7 best regards, Andy Am 21.09.2021 um 13:12 schrieb Aste: > Hi! > > Tuesday, September 21, 2021, 1:43:13 PM, you wrote: > AWGG> # local.d/multimap.conf > AWGG> CONTENT_BLACKLISTED { > AWGG> ? type = "content"; > AWGG> ? filter = "body"; # can be headers, full, oneline, text, rawtext > AWGG> ? map = "${LOCAL_CONFDIR}/local.d/local_content.map"; > According to documentation you need to specify map type as multimap > map = regexp_multi; ${LOCAL_CONFDIR}/local.d/local_content.map; > > And you always can add global score there > score = 10; > > AWGG> ? symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; > AWGG> ? regexp = true; > AWGG> } > > AWGG> In this example, you can use 3 symbols: > > AWGG> ??? CONTENT_BLACKLISTED > AWGG> ??? CONTENT_BLACKLISTED1 > AWGG> ??? CONTENT_BLACKLISTED2 > > AWGG> the map: > > AWGG> # Symbol + score > AWGG> /re1/ CONTENT_BLACKLISTED1:10 > AWGG> # Symbol with default score > AWGG> /re2/ CONTENT_BLACKLISTED2 > AWGG> # Just a default symbol: CONTENT_BLACKLISTED > AWGG> /re3/ > > AWGG> Symbols that are not defined in the symbols attribute but used in the map are ignored and replaced by the default map symbol. If the value of a key-value pair is missing, then Rspamd just inserts the default symbol with dynamic weight equal to 1.0 (which is multiplied by metric score afterwards) > > > AWGG> ...matches the right symbol (CONTENT_BLACKLISTED1) in the header : > > AWGG> X-Spamd-Result: default: False [0.80 / 15.00]; > AWGG> ??? ... > AWGG> ??? CONTENT_BLACKLISTED1(0.00)[]; > AWGG> ??? ... > > AWGG> ...but why is score 0.00 and not 10 like it is defined in local_content.map? > > > AWGG> best regards, Andy > > AWGG> Am 21.09.2021 um 11:53 schrieb Andreas Wass - Glas Gasperlmair: >>> Hi, >>> >>> i'm trying to use rspamd as soon as possible for our company > mailserver, and testing content filtering with multimap as shown in >>> >>> https://rspamd.com/doc/modules/multimap.html >>> >>> *Configured the following.*.. >>> >>> >>> ?? Multiple symbol maps >>> >>> From the version 1.3.1, it is possible to define multiple symbols and > scores using multimap module. To do that, you should define all > possible symbols using|symbols|option in multimap: >>> >>> |# local.d/multimap.conf CONTENT_BLACKLISTED { type = "content"; > filter = "body"; # can be headers, full, oneline, text, rawtext map = > "${LOCAL_CONFDIR}/local.d/local_content.map"; symbols = > ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; regexp = true; } | >>> >>> In this example, you can use 3 symbols: >>> >>> ?* CONTENT_BLACKLISTED >>> ?* CONTENT_BLACKLISTED1 >>> ?* CONTENT_BLACKLISTED2 >>> >>> the map: >>> >>> |# Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default > score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: > CONTENT_BLACKLISTED /re3/ | >>> >>> Symbols that are not defined in the|symbols|attribute but used in the > map are ignored and replaced by the default map symbol. If the value > of a key-value pair is missing, then Rspamd just inserts the default > symbol with dynamic weight equal to|1.0|(which is multiplied by metric > score afterwards) >>> >>> >>> *...matches the right symbol (CONTENT_BLACKLISTED1) in the header :* >>> >>> X-Spamd-Result: default: False [0.80 / 15.00]; >>> ????... >>> ????CONTENT_BLACKLISTED1(0.00)[]; >>> ????... >>> >>> *...but why is score 0.00 and not 10 like it is defined in > local_content.map?* >>> >>> >>> best regards, Andy >>> > > > From philip.colmer at linaro.org Tue Sep 21 12:55:15 2021 From: philip.colmer at linaro.org (Philip Colmer) Date: Tue, 21 Sep 2021 13:55:15 +0100 Subject: [Rspamd-Users] When should ARC headers be added? Message-ID: I'm trying to understand when an ARC set (the three ARC headers) should be added - when an email is received by a MTA, when a MTA is sending an email out or both? I've read RFC8617 and it doesn't seem to be entirely clear but the behaviour I'm seeing from other systems seems to suggest that it can be both. If that is the case, I'm struggling with configuring the ARC module appropriately and would welcome some insight/assistance. My test scenario is sending an email from domain A to domain B, which is running Mailman 3. Mailman 3 adds a signature to the email and changes the subject line before sending the email back to domain A. When domain A sends the email to domain B, it adds a DKIM header and an ARC header. When the email reaches domain B, Rspamd's ARC module tries to create an ARC set for domain A, which it clearly can't do. After Mailman 3 has done its thing, it sends the email to Postfix which then sends it to Rspamd. It looks like the ARC module gets called before the dkim_signing module, which causes a problem because the last (only?) DKIM header that the ARC module can see is one that was created by domain A, which no longer matches because Mailman 3 has altered the email. A DKIM header does get added by Rspamd but there isn't an ARC Set getting added by Rspamd. I'm not sure if this is because I've misconfigured the ARC module or because of the rspamd_dkim_check errors I'm getting: dkim; rspamd_dkim_check: arc_sig: bh value mismatch: got T3JEKsVRCAWS09CDUv/Nc9LZJQ387FwCcC1OB2ceZ5g=, expected 57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075; d=zohomail.com; s=zohoarc Those seem to come from the dkim module but I'm not sure why it would be checking against the DKIM header generated by domain A rather than just generating a DKIM header itself. I've included the full log output for when Postfix calls Rspamd on message receipt and message sending below, just in case I'm overlooking something. Thanks in advance for any advice/insight offered. Regards Philip dkim_signing config: sign_networks [ "127.2.4.7", ] try_fallback = false; allow_envfrom_empty = true; allow_hdrfrom_multiple = false; use_esld = false; sign_local = true; key_prefix = "DKIM_KEYS"; sign_authenticated = false; selector = "dkim"; use_redis = false; use_domain = "envelope"; allow_hdrfrom_mismatch = true; allow_hdrfrom_mismatch_sign_networks = true; allow_username_mismatch = true; domain { mm3.lavasoftware.org { selector = "dkim"; path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key"; } } symbol = "DKIM_SIGNED"; arc config: selector = "arc"; domain { mm3.lavasoftware.org { selector = "dkim"; path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key"; } } use_domain = "header"; allow_hdrfrom_mismatch = true; allow_envfrom_empty = true; allow_hdrfrom_multiple = false; sign_inbound = true; sign_local = false; sign_authenticated = false; use_esld = false; try_fallback = true; use_redis = false; key_prefix = "ARC_KEYS"; sign_networks [ "127.2.4.7", ] symbol_sign = "ARC_SIGNED"; allow_username_mismatch = true; When Postfix first receives the email from domain A: 2021-09-21 12:50:21 #563873(rspamd_proxy) <2b2ef0>; proxy; proxy_accept_socket: accepted milter connection from 127.0.0.1 port 44648 2021-09-21 12:50:22 #563873(rspamd_proxy) <2b2ef0>; milter; rspamd_milter_process_command: got connection from 136.143.188.14:17412 2021-09-21 12:50:22 #563873(rspamd_proxy) <2b2ef0>; proxy; rspamd_message_parse: loaded message; id: <17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>; queue-id: ; size: 2391; checksum: <494b8362da04adff9b1bd4b5bd01466d> 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy; dkim_module_key_handler: stored DKIM key for zoho._domainkey.codelinaro.org in LRU cache for 300 seconds, 1/2000 elements in the cache 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; dkim_signing; lua_dkim_tools.lua:170: mail is ineligible for signing 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy; rspamd_spf_maybe_return: stored record for codelinaro.org (0x4d20af7734be3ae1) in LRU cache for 300 seconds, 1/2000 elements in the cache 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:205: got 1 arc sections 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:342: processed arc signature zohomail.com[1]: true(nil), 0 processed 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:260: checked arc signature zohomail.com: true(nil), 0 processed 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:226: checked arc seal: true(nil), 1 processed 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; lua_dkim_tools.lua:168: mail was sent to us 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; lua_dkim_tools.lua:382: use domain(header) for signature: codelinaro.org 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; lua_dkim_tools.lua:402: final DKIM domain: codelinaro.org 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; lua_dkim_tools.lua:46: add key "/var/lib/rspamd/arc/$domain.$selector.key" using default path 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; lua_dkim_tools.lua:51: set selector to "arc" using default selector 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; lua_dkim_tools.lua:51: set domain to "codelinaro.org" using dkim_domain 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:682: cannot read key from /var/lib/rspamd/arc/codelinaro.org.arc.key: No such file or directory 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua; greylist.lua:318: Score too low - skip greylisting 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua; neural.lua:315: skip ham sample to keep spam/ham balance; probability 1; 0 spam and 1 ham vectors stored 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua; neural.lua:69: created new ANN profile for default:default, data stored at prefix rn_default_default_cjf5d845_0 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy; rspamd_task_write_log: id: <17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid: , ip: 136.143.188.14, from: , (default: F (no action): [-1.49/15.00] [ARC_ALLOW(-1.00){zohomail.com:s=zohoarc:i=1;},R_DKIM_ALLOW(-0.20){codelinaro.org:s=zoho;},R_SPF_ALLOW(-0.20){+ip4:136.143.188.0/24;},MIME_GOOD(-0.10){multipart/alternative;text/plain;},XM_UA_NO_VERSION(0.01){},ASN(0.00){asn:2639, ipnet:136.143.188.0/23, country:US;},DKIM_TRACE(0.00){codelinaro.org:+;},DMARC_NA(0.00){codelinaro.org;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:~;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_TWO(0.00){2;},RCVD_IN_DNSWL_NONE(0.00){136.143.188.14:from;},RCVD_TLS_LAST(0.00){},RWL_MAILSPIKE_POSSIBLE(0.00){136.143.188.14:from;},TO_DN_ALL(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 2391, time: 672.016ms, dns req: 27, digest: <494b8362da04adff9b1bd4b5bd01466d>, rcpts: , mime_rcpts: 2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned using pcre, 1014B scanned total 2021-09-21 12:50:23 #563873(rspamd_proxy) ; proxy; proxy_milter_finish_handler: finished milter connection When Mailman 3 sends the email to Postfix for sending out: 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; proxy_accept_socket: accepted milter connection from 127.0.0.1 port 44746 2021-09-21 12:50:26 #563874(rspamd_proxy) ; milter; rspamd_milter_process_command: got connection from 127.0.0.1:34878 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; rspamd_message_parse: loaded message; id: <17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>; queue-id: <51B0ABEA9C>; size: 3909; checksum: <1b29c03e5f5475bcabec1e787f28d4ef> 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; rspamd_mime_part_detect_language: detected part language: en 2021-09-21 12:50:26 #563874(rspamd_proxy) ; lua; greylist.lua:204: skip greylisting for local networks and/or authorized users 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; dkim_symbol_callback: skip DKIM checks for local networks and authorized users 2021-09-21 12:50:26 #563874(rspamd_proxy) ; lua; spf.lua:186: skip SPF checks for local networks and authorized users 2021-09-21 12:50:26 #563874(rspamd_proxy) ; arc; arc.lua:205: got 1 arc sections 2021-09-21 12:50:26 #563874(rspamd_proxy) ; arc; arc.lua:342: processed arc signature zohomail.com[1]: true(nil), 0 processed 2021-09-21 12:50:26 #563874(rspamd_proxy) ; lua; dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not checked 2021-09-21 12:50:26 #563874(rspamd_proxy) ; dkim_signing; lua_dkim_tools.lua:166: mail is from local address 2021-09-21 12:50:26 #563874(rspamd_proxy) ; dkim_signing; lua_dkim_tools.lua:382: use domain(envelope) for signature: mm3.lavasoftware.org 2021-09-21 12:50:26 #563874(rspamd_proxy) ; dkim_signing; lua_dkim_tools.lua:402: final DKIM domain: mm3.lavasoftware.org 2021-09-21 12:50:26 #563874(rspamd_proxy) ; dkim_signing; lua_dkim_tools.lua:51: set domain to "mm3.lavasoftware.org" using dkim_domain 2021-09-21 12:50:26 #563874(rspamd_proxy) ; dkim_signing; dkim_signing.lua:129: using key "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key", use selector "dkim" for domain "mm3.lavasoftware.org" 2021-09-21 12:50:26 #563874(rspamd_proxy) ; lua; once_received.lua:99: Skipping once_received for authenticated user or local network 2021-09-21 12:50:26 #563874(rspamd_proxy) ; dkim; rspamd_dkim_check: arc_sig: bh value mismatch: got i2Ji19OooPA0mAVbxb/Wh0tqFcGTSvsGoLDHuoc4EIs=, expected 57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075; d=zohomail.com; s=zohoarc 2021-09-21 12:50:26 #563874(rspamd_proxy) ; arc; arc.lua:260: checked arc signature zohomail.com: false(reject), 0 processed 2021-09-21 12:50:26 #563874(rspamd_proxy) ; arc; lua_dkim_tools.lua:170: mail is ineligible for signing 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing 2021-09-21 12:50:26 #563874(rspamd_proxy) ; lua; greylist.lua:318: Score too low - skip greylisting 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; rspamd_task_write_log: id: <17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid: <51B0ABEA9C>, ip: 127.0.0.1, from: , (default: F (no action): [0.80/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] = sig:zohomail.com:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){philip.colmer at codelinaro.org;test-bounces at mm3.lavasoftware.org;},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},TAGGED_FROM(0.00){philip.colmer=linaro.org;},TO_DN_ALL(0.00){}]), len: 3909, time: 243.345ms, dns req: 11, digest: <1b29c03e5f5475bcabec1e787f28d4ef>, rcpts: , mime_rcpts: 2021-09-21 12:50:26 #563874(rspamd_proxy) ; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned using pcre, 1.90KiB scanned total 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy; rspamd_message_parse: loaded message; id: <17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>; queue-id: <9D95CBEA9F>; size: 3909; checksum: <1b29c03e5f5475bcabec1e787f28d4ef> 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy; rspamd_mime_part_detect_language: detected part language: en 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua; greylist.lua:204: skip greylisting for local networks and/or authorized users 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy; dkim_symbol_callback: skip DKIM checks for local networks and authorized users 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua; spf.lua:186: skip SPF checks for local networks and authorized users 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:205: got 1 arc sections 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim; rspamd_dkim_check: arc_sig: bh value mismatch: got T3JEKsVRCAWS09CDUv/Nc9LZJQ387FwCcC1OB2ceZ5g=, expected 57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075; d=zohomail.com; s=zohoarc 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:260: checked arc signature zohomail.com: false(reject), 0 processed 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:342: processed arc signature zohomail.com[1]: true(nil), 0 processed 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua; dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not checked 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; lua_dkim_tools.lua:170: mail is ineligible for signing 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing; lua_dkim_tools.lua:166: mail is from local address 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing; lua_dkim_tools.lua:382: use domain(envelope) for signature: mm3.lavasoftware.org 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing; lua_dkim_tools.lua:402: final DKIM domain: mm3.lavasoftware.org 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing; lua_dkim_tools.lua:51: set domain to "mm3.lavasoftware.org" using dkim_domain 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing; dkim_signing.lua:129: using key "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key", use selector "dkim" for domain "mm3.lavasoftware.org" 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua; once_received.lua:99: Skipping once_received for authenticated user or local network 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua; greylist.lua:318: Score too low - skip greylisting 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy; rspamd_task_write_log: id: <17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid: <9D95CBEA9F>, ip: 127.0.0.1, from: , (default: F (no action): [0.79/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] = sig:zohomail.com:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){philip.colmer at codelinaro.org;test-bounces at mm3.lavasoftware.org;},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},TAGGED_FROM(0.00){philip.colmer=codelinaro.org;},TO_DN_ALL(0.00){}]), len: 3909, time: 9.960ms, dns req: 10, digest: <1b29c03e5f5475bcabec1e787f28d4ef>, rcpts: , mime_rcpts: 2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned using pcre, 1.90KiB scanned total 2021-09-21 12:50:26 #563874(rspamd_proxy) <6370ae>; proxy; proxy_milter_finish_handler: finished milter connection From kyle at cci1986.com Tue Sep 21 13:18:58 2021 From: kyle at cci1986.com (Kyle A.) Date: Tue, 21 Sep 2021 09:18:58 -0400 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> Message-ID: Thanks for the suggestion, Michael. I ran it again today and did not get the same error. I got this instead: # rspamadm dmarc_report No reports for 20210921 No reports for 20210920 Segmentation fault (core dumped) # I'll try again tomorrow and see if it seg faults with fresh V3.0 data only. This is a fully patched up, dedicated CentOS 8 system. I put the log messages down at the bottom for anyone who wants to scroll all the way down there to see them. -Kyle On 9/20/2021 4:39 PM, Michael Kliewe wrote: > I also got this error when I first manually ran > sudo rspamadm dmarc_report > after upgrading from 2.7 to 3.0.2 on Ubuntu. > > But: > One day later I ran it again manually: > sudo rspamadm dmarc_report -v > (with verbose mode on) and it was working fine, it sent 10 DMARC reports. > > Maybe there was some old/invalid data in the redis storage, which is why > the first invocation failed with "invalid argument: nil"? > > Michael Sep 21 09:08:56 mailserver kernel: rspamadm[8258]: segfault at 30 ip 00007fda52d1868c sp 00007ffdd54906b0 error 6 in librspamd-server.s o[7fda522a3000+c70000] Sep 21 09:08:56 mailserver kernel: Code: 49 89 40 38 4d 89 ca 4c 89 c1 e9 73 ff ff ff 0f 1f 80 00 00 00 00 4d 89 58 28 4d 89 43 30 e9 7 6 fc ff ff 0f 1f 00 48 8b 4e 30 <49> 89 4b 30 4c 8b 66 30 4d 85 e4 0f 85 6e fe ff ff e9 6a fc ff ff Sep 21 09:08:56 mailserver systemd[1]: Created slice system-systemd\x2dcoredump.slice. Sep 21 09:08:56 mailserver systemd[1]: Started Process Core Dump (PID 8259/UID 0). Sep 21 09:08:58 mailserver systemd-coredump[8260]: Process 8258 (rspamadm) of user 0 dumped core.#012#012Stack trace of thread 8258:#01 2#0 0x00007fda52d1868c je_extent_heap_remove (librspamd-server.so)#012#1 0x00007fda52ce80e3 je_arena_dalloc_bin_junked_locked (l ibrspamd-server.so)#012#2 0x00007fda52d3e862 je_tcache_bin_flush_small (librspamd-server.so)#012#3 0x00007fda52cdf81a je_free_de fault (librspamd-server.so)#012#4 0x00007fda523ef532 ucl_hash_destroy (librspamd-server.so)#012#5 0x00007fda523fbbe4 ucl_object_ dtor_unref (librspamd-server.so)#012#6 0x00007fda523ef532 ucl_hash_destroy (librspamd-server.so)#012#7 0x00007fda523fbbe4 ucl_ob ject_dtor_unref (librspamd-server.so)#012#8 0x00007fda523ef532 ucl_hash_destroy (librspamd-server.so)#012#9 0x00007fda523fbcfe u cl_object_unref (librspamd-server.so)#012#10 0x00007fda52630e7e rspamd_config_free (librspamd-server.so)#012#11 0x000000000040c781 main (rspamadm)#012#12 0x00007fda4cbc0493 __libc_start_main (libc.so.6)#012#13 0x000000000040cb2e _start (rspamadm) Sep 21 09:08:59 mailserver systemd[1]: systemd-coredump at 0-8259-0.service: Succeeded. From m.kliewe at team.mail.de Tue Sep 21 15:15:41 2021 From: m.kliewe at team.mail.de (Michael Kliewe) Date: Tue, 21 Sep 2021 17:15:41 +0200 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> Message-ID: <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de> Hi Kyle, try the verbose mode ("rspamadm dmarc_report -v"), you should get more information then. Maybe it helps debugging the problem, and maybe you see where it seg faults. Michael Am 21.09.2021 um 15:18 schrieb Kyle A.: > Thanks for the suggestion, Michael.? I ran it again today and did not > get the same error.? I got this instead: > > # rspamadm dmarc_report > No reports for 20210921 > No reports for 20210920 > Segmentation fault (core dumped) > # > > I'll try again tomorrow and see if it seg faults with fresh V3.0 data > only.? This is a fully patched up, dedicated CentOS 8 system.? I put > the log messages down at the bottom for anyone who wants to scroll all > the way down there to see them. > > -Kyle > > On 9/20/2021 4:39 PM, Michael Kliewe wrote: >> I also got this error when I first manually ran >> sudo rspamadm dmarc_report >> after upgrading from 2.7 to 3.0.2 on Ubuntu. >> >> But: >> One day later I ran it again manually: >> sudo rspamadm dmarc_report -v >> (with verbose mode on) and it was working fine, it sent 10 DMARC >> reports. >> >> Maybe there was some old/invalid data in the redis storage, which is >> why the first invocation failed with "invalid argument: nil"? >> >> Michael > > > > > Sep 21 09:08:56 mailserver kernel: rspamadm[8258]: segfault at 30 ip > 00007fda52d1868c sp 00007ffdd54906b0 error 6 in librspamd-server.s > ??????????????? o[7fda522a3000+c70000] > Sep 21 09:08:56 mailserver kernel: Code: 49 89 40 38 4d 89 ca 4c 89 c1 > e9 73 ff ff ff 0f 1f 80 00 00 00 00 4d 89 58 28 4d 89 43 30 e9 7 > ????????????? 6 fc ff ff 0f 1f 00 48 8b 4e 30 <49> 89 4b 30 4c 8b 66 > 30 4d 85 e4 0f 85 6e fe ff ff e9 6a fc ff ff > Sep 21 09:08:56 mailserver systemd[1]: Created slice > system-systemd\x2dcoredump.slice. > Sep 21 09:08:56 mailserver systemd[1]: Started Process Core Dump (PID > 8259/UID 0). > Sep 21 09:08:58 mailserver systemd-coredump[8260]: Process 8258 > (rspamadm) of user 0 dumped core.#012#012Stack trace of thread 8258:#01 > ???????????????????? 2#0? 0x00007fda52d1868c je_extent_heap_remove > (librspamd-server.so)#012#1? 0x00007fda52ce80e3 > je_arena_dalloc_bin_junked_locked (l > ibrspamd-server.so)#012#2? 0x00007fda52d3e862 > je_tcache_bin_flush_small (librspamd-server.so)#012#3 > 0x00007fda52cdf81a je_free_de > ??????? fault (librspamd-server.so)#012#4? 0x00007fda523ef532 > ucl_hash_destroy (librspamd-server.so)#012#5? 0x00007fda523fbbe4 > ucl_object_ ???????????????????????????????? dtor_unref > (librspamd-server.so)#012#6 ?0x00007fda523ef532 ucl_hash_destroy > (librspamd-server.so)#012#7 0x00007fda523fbbe4 ucl_ob > ?????????????????????????????????????????????? ject_dtor_unref > (librspamd-server.so)#012#8? 0x00007fda523ef532 ucl_hash_destroy > (librspamd-server.so)#012#9? 0x00007fda523fbcfe u > cl_object_unref (librspamd-server.so)#012#10 0x00007fda52630e7e > rspamd_config_free (librspamd-server.so)#012#11 0x000000000040c781 > ???????????????? main (rspamadm)#012#12 0x00007fda4cbc0493 > __libc_start_main (libc.so.6)#012#13 0x000000000040cb2e _start (rspamadm) > Sep 21 09:08:59 mailserver systemd[1]: > systemd-coredump at 0-8259-0.service: Succeeded. From kyle at cci1986.com Tue Sep 21 19:05:31 2021 From: kyle at cci1986.com (Kyle A.) Date: Tue, 21 Sep 2021 15:05:31 -0400 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de> References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de> Message-ID: <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com> Michael, Thanks for the suggestion. This is interesting: [mailserver:~ root]# rspamadm dmarc_report -v previous last report date is 1632160659 Process date 20210921 No reports for 20210921 Process date 20210920 No reports for 20210920 send data for 0 domains (from 1 to 0) [mailserver:~ root]# rspamadm dmarc_report No reports for 20210921 No reports for 20210920 [mailserver:~ root]# The GOOD news is it did not seg fault. The bad news it is not showing any data to report. This server has handled about 1.2k messages so far today, so there should be data to report. Thanks, Kyle On 9/21/2021 11:15 AM, Michael Kliewe wrote: > Hi Kyle, > > try the verbose mode ("rspamadm dmarc_report -v"), you should get more > information then. Maybe it helps debugging the problem, and maybe you > see where it seg faults. > > Michael From emawata at gmail.com Wed Sep 22 17:38:40 2021 From: emawata at gmail.com (SysAdmin EM) Date: Wed, 22 Sep 2021 14:38:40 -0300 Subject: [Rspamd-Users] Neural Network manually training Message-ID: Hello, I am trying to manually train the neural network since I see that many emails that are SPAM are being cataloged as NEURAL_HAM. This is my config: servers = "127.0.0.1:6379"; train { max_trains = 1k; # Number ham/spam samples needed to start train max_usages = 20; # Number of learn iterations while ANN data is valid learning_rate = 0.01; # Rate of learning max_iterations = 25; # Maximum iterations of learning (better preciseness but also lower speed of learning) } ann_expire = 2d; # For how long ANN should be preserved in Redis The documentation (https://rspamd.com/doc/modules/neural.html) indicates that the files should be placed in /plugins/neural/learn. What would be the exact path? in /etc/rspamd/local.d? or other path? once the neural network has been trained, can automatic training be deactivated? or do you always have to train? Regards, From kyle at cci1986.com Wed Sep 22 17:51:41 2021 From: kyle at cci1986.com (Kyle A.) Date: Wed, 22 Sep 2021 13:51:41 -0400 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com> References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de> <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com> Message-ID: No seg fault anymore, but this module seems broken: [mailserver:~ root]# rspamadm dmarc_report No reports for 20210922 No reports for 20210921 [mailserver:~ root]# At least 1k more messages have been handled since my last post. DMARC reporting worked for us in all the 2.x versions. Thanks, Kyle On 9/21/2021 3:05 PM, Kyle A. wrote: > Michael, > > Thanks for the suggestion.? This is interesting: > > [mailserver:~ root]# rspamadm dmarc_report -v > previous last report date is 1632160659 > Process date 20210921 > No reports for 20210921 > Process date 20210920 > No reports for 20210920 > send data for 0 domains (from 1 to 0) > [mailserver:~ root]# rspamadm dmarc_report > No reports for 20210921 > No reports for 20210920 > [mailserver:~ root]# > > The GOOD news is it did not seg fault.? The bad news it is not showing > any data to report.? This server has handled about 1.2k messages so far > today, so there should be data to report. > > Thanks, > Kyle From rspamd at vlh.dk Wed Sep 22 18:32:03 2021 From: rspamd at vlh.dk (rspamd at vlh.dk) Date: Wed, 22 Sep 2021 20:32:03 +0200 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de> <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com> Message-ID: <003d01d7afe0$27cb02b0$77610810$@vlh.dk> Sounds strange, works for me with a very basic local.d/dmarc.conf actions = { quarantine = "add_header"; reject = "reject"; } reporting { enabled = true org_name = "vlh.dk"; domain = "vlh.dk"; email = "dmarc at vlh.dk"; smtp = "mail.vlh.dk"; smtp_port = 25; helo = "mail.vlh.dk"; # Number of retries on temporary errors (2 if unset) # retries = 2; } mail ~ # rspamadm dmarc_report No reports for 20210921 Reporting collection has finished 1 dates processed, 1 reports: 1 completed, 0 failed Very low volume private mailserver ? Kind regards, Kim Sindalsen > -----Original Message----- > From: Users On Behalf Of Kyle A. > Sent: 22. september 2021 19:52 > To: users at lists.rspamd.com > Subject: Re: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load > config > > No seg fault anymore, but this module seems broken: > > [mailserver:~ root]# rspamadm dmarc_report No reports for 20210922 No > reports for 20210921 [mailserver:~ root]# > > At least 1k more messages have been handled since my last post. DMARC > reporting worked for us in all the 2.x versions. > > Thanks, > Kyle From Ralf.Hildebrandt at charite.de Thu Sep 23 07:52:50 2021 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 23 Sep 2021 09:52:50 +0200 Subject: [Rspamd-Users] [ext] Neural Network manually training In-Reply-To: References: Message-ID: * SysAdmin EM : > The documentation (https://rspamd.com/doc/modules/neural.html) indicates > that the files should be placed in /plugins/neural/learn. No, that's not what it says! "The controller endpoint /plugins/neural/learn facilitates manual training of neural networks & accepts a JSON POST" So basically it's a web service (on the rspamd Server) that accepts a POST request. I must admit the whole paragraph lacks an example (especially for "spam_vec and ham_vec" and "rule") Ralf Hildebrandt Charit? - Universit?tsmedizin Berlin Gesch?ftsbereich IT | Abteilung Netzwerk Campus Benjamin Franklin (CBF) Haus I | 1. OG | Raum 105 Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 ralf.hildebrandt at charite.de https://www.charite.de From emawata at gmail.com Thu Sep 23 10:42:42 2021 From: emawata at gmail.com (SysAdmin EM) Date: Thu, 23 Sep 2021 07:42:42 -0300 Subject: [Rspamd-Users] [ext] Neural Network manually training In-Reply-To: References: Message-ID: Thanks for the reply. Would you tell me how I do the manual training? I can't understand how to do it. Regards, On Thu, Sep 23, 2021 at 4:54 AM Ralf Hildebrandt < Ralf.Hildebrandt at charite.de> wrote: > * SysAdmin EM : > > > The documentation (https://rspamd.com/doc/modules/neural.html) indicates > > that the files should be placed in /plugins/neural/learn. > > No, that's not what it says! > > "The controller endpoint /plugins/neural/learn facilitates manual > training of neural networks & accepts a JSON POST" > > So basically it's a web service (on the rspamd Server) that accepts a > POST request. > > I must admit the whole paragraph lacks an example (especially for > "spam_vec and ham_vec" and "rule") > > Ralf Hildebrandt > Charit? - Universit?tsmedizin Berlin > Gesch?ftsbereich IT | Abteilung Netzwerk > > Campus Benjamin Franklin (CBF) > Haus I | 1. OG | Raum 105 > Hindenburgdamm 30 | D-12203 Berlin > > Tel. +49 30 450 570 155 > ralf.hildebrandt at charite.de > https://www.charite.de > -- > Users mailing list > Users at lists.rspamd.com > https://lists.rspamd.com/mailman/listinfo/users > From Ralf.Hildebrandt at charite.de Thu Sep 23 10:51:25 2021 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 23 Sep 2021 12:51:25 +0200 Subject: [Rspamd-Users] [ext] Neural Network manually training In-Reply-To: References: Message-ID: * SysAdmin EM : > Thanks for the reply. > > Would you tell me how I do the manual training? I can't understand how to > do it. I can't since I also don't understand what teh webservice actually expects :) Ralf Hildebrandt Charit? - Universit?tsmedizin Berlin Gesch?ftsbereich IT | Abteilung Netzwerk Campus Benjamin Franklin (CBF) Haus I | 1. OG | Raum 105 Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 ralf.hildebrandt at charite.de https://www.charite.de From emawata at gmail.com Mon Sep 27 13:20:19 2021 From: emawata at gmail.com (SysAdmin EM) Date: Mon, 27 Sep 2021 10:20:19 -0300 Subject: [Rspamd-Users] [ext] Neural Network manually training In-Reply-To: References: Message-ID: Any helps for this?? On Thu, Sep 23, 2021 at 7:52 AM Ralf Hildebrandt < Ralf.Hildebrandt at charite.de> wrote: > * SysAdmin EM : > > Thanks for the reply. > > > > Would you tell me how I do the manual training? I can't understand how to > > do it. > > I can't since I also don't understand what teh webservice actually > expects :) > > Ralf Hildebrandt > Charit? - Universit?tsmedizin Berlin > Gesch?ftsbereich IT | Abteilung Netzwerk > > Campus Benjamin Franklin (CBF) > Haus I | 1. OG | Raum 105 > Hindenburgdamm 30 | D-12203 Berlin > > Tel. +49 30 450 570 155 > ralf.hildebrandt at charite.de > https://www.charite.de > -- > Users mailing list > Users at lists.rspamd.com > https://lists.rspamd.com/mailman/listinfo/users > From azurit at pobox.sk Tue Sep 28 12:47:45 2021 From: azurit at pobox.sk (azurit at pobox.sk) Date: Tue, 28 Sep 2021 14:47:45 +0200 Subject: [Rspamd-Users] Bayes not scanning some e-mails Message-ID: <20210928144745.Horde.j0vvuTNfYx7ZYbM__iJQhXD@webmail.inetadmin.eu> Hi all, recently, i noticed that not all e-mails contains symbols from bayes/statistic module - none of BAYES_HAM and BAYES_SPAM. I was able to catch one of such messages with debugging info enabled (see below). Can anyone help me to understand why is this happening? In this particular case, the message was spam and was delivered into INBOX because of missing score from bayes. Thanks. Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 253878091069365434 <#F:Vladislav:#h:C-T:text/html; charset="UTF-8"> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 1319675597245360303 <#F:Vladislav:#m:#ho> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 5654384397166480846 <#F:Vladislav:#cs:UTF-8> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 11470203402223934998 <#F:Vladislav:#lang:en> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 16760811356819922963 <#u:yetisirmi.com:#F:Vladislav> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 243482661706484953 <#u:yetisirmi.com:#h:C-T:text/html; charset="UTF-8"> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 8166284013702598116 <#u:yetisirmi.com:#m:#ho> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 6749161068152776729 <#u:yetisirmi.com:#cs:UTF-8> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 7533617803319042616 <#dt:4:01:#u:yetisirmi.com> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 8072560273546414927 <#dt:4:01:#F:Vladislav> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 14436082610651627817 <#dt:4:01:#h:C-T:text/html; charset="UTF-8"> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 1430216995712494580 <#dt:4:01:#m:#ho> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 8084564344892568728 <#aur:s=pass,d=pass:chatwasap.com:#dt:4:01> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 12330708087224283914 <#aur:s=pass,d=pass:chatwasap.com:#u:yetisirmi.com> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 4462463662119808283 <#aur:s=pass,d=pass:chatwasap.com:#F:Vladislav> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 12434003267573628141 <#aur:s=pass,d=pass:chatwasap.com:#h:C-T:text/html; charset="UTF-8"> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 6863361389720196180 <#rcv::lmtp:#aur:s=pass,d=pass:chatwasap.com> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 7539326144642557518 <#rcv::lmtp:#dt:4:01> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 11824198664993959380 <#rcv::lmtp:#u:yetisirmi.com> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 12294035829013819971 <#rcv::lmtp:#F:Vladislav> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 16686934200068670817 <#rcv:103.72.162.0:esmtps:#rcv::lmtp> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 16045033416894290287 <#rcv:103.72.162.0:esmtps:#aur:s=pass,d=pass:chatwasap.com> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 8743901820467606605 <#rcv:103.72.162.0:esmtps:#dt:4:01> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 14815350519403958349 <#rcv:103.72.162.0:esmtps:#u:yetisirmi.com> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 15416532422982920442 <#rcv:194.146.36.0:esmtpsa:#rcv:103.72.162.0:esmtps> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 1598239120901030481 <#rcv:194.146.36.0:esmtpsa:#rcv::lmtp> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 4328058460560150171 <#rcv:194.146.36.0:esmtpsa:#aur:s=pass,d=pass:chatwasap.com> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; bayes; bayes_classify_token: token(meta) 15116481825142164251 <#rcv:194.146.36.0:esmtpsa:#dt:4:01> probabilistically skipped Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; csession; rspamd_task_write_log: id: , ip: , from: >, (default: F (no action): [4.10/8.00] [HFILTER_HOSTNAME_UNKNOWN(2.50){},FROM_EXCESS_QP(1.20){},MISSING_DATE(1.00){},SPF_REPUTATION_HAM(-0.48){-0.48426960253533;},MIME_HTML_ONLY(0.20){},R_DKIM_ALLOW(-0.20){chatwasap.com:s=default;},R_SPF_ALLOW(-0.20){+a;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},ABUSE_SURBL(0.00){yetisirmi.com:url;},ASN(0.00){asn:29405, ipnet:/20, country:SK;},DBL_SPAM(0.00){yetisirmi.com:url;},DKIM_TRACE(0.00){chatwasap.com:+;},DMARC_NA(0.00){chatwasap.com;},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){sender at chatwasap.com;SRS0=0MnC=OR=chatwasap.com=sender at inetadmin.eu;},HAS_X_ANTIABUSE(0.00){},HAS_X_AS(0.00){sender at chatwasap.com;},HAS_X_GMSV(0.00){sender at chatwasap.com;},HAS_X_SOURCE(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:~;},PREVIOUSLY_DELIVERED(0.00){;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){4;},RCVD_VIA_SMTP_AUTH(0.00){},TO_DN_NONE(0.00){},URIBL_BLACK(0.00){yetisirmi.com:url;}]), len: 9455, time: 354.826ms, dns req: 40, digest: <01ffc08c6270b0d08cf240c866f7d3e0>, mime_rcpts: Sep 28 14:26:05 server00 rspamd[30529]: <80a5d6>; csession; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 5 regexps matched, 175 regexps total, 66 regexps cached, 0B scanned using pcre, 7.17KiB scanned total From kyle at cci1986.com Wed Sep 29 16:09:36 2021 From: kyle at cci1986.com (Kyle A.) Date: Wed, 29 Sep 2021 12:09:36 -0400 Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load config In-Reply-To: <003d01d7afe0$27cb02b0$77610810$@vlh.dk> References: <1907311023.20210823163332@latnet.lv> <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com> <5710476131.20210913170548@sveiks.lv> <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de> <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com> <003d01d7afe0$27cb02b0$77610810$@vlh.dk> Message-ID: <85674a73-935f-f6ce-a81c-4962dbebff41@cci1986.com> Might be fixed(?). This morning I was *still getting seg faults* from the V3.0 dmarc module. I got this earlier: [mailserver:/etc/rspamd/local.d root]# rspamadm dmarc_report -v previous last report date is 1632842756 Process date 20210929 No reports for 20210929 Process date 20210928 No reports for 20210928 send data for 0 domains (from 1 to 0) Segmentation fault (core dumped) <<<--------- oops I dumped my very generic config to paste here using "rspamadm configdump dmarc". In various windows I was monitoring my redis keys using redis-cli --stat redis-cli monitor I use the "key_prefix" parameter in my dmarc.conf, so I used that to my advantage: redis-cli --scan --pattern 'dmarc_*' dmarc_rpt;shelf-awareness.com;mailto:neil at shelf-awareness.com;20210929 dmarc_idx;20210929 What? I was surprised there was a dmarc record!!!! the first record since installing V3.0!!!!! So I ran the report again and it worked for the first time without a seg fault since I upgraded to V3.0. Is it fixed? I don't know. *I didn't change anything.* I was just gathering information for this e-mail reply when it (magically) started working. I'll try to remember to report back if it keeps working or if it fails again and I'm still having problems. Thanks for the help so far. Thanks, Kyle On 9/22/2021 2:32 PM, Kim Sindalsen via Users wrote: > Sounds strange, works for me with a very basic local.d/dmarc.conf > > actions = { > quarantine = "add_header"; > reject = "reject"; > } > reporting { > enabled = true > org_name = "vlh.dk"; > domain = "vlh.dk"; > email = "dmarc at vlh.dk"; > smtp = "mail.vlh.dk"; > smtp_port = 25; > helo = "mail.vlh.dk"; > # Number of retries on temporary errors (2 if unset) > # retries = 2; > } > > mail ~ # rspamadm dmarc_report > No reports for 20210921 > Reporting collection has finished 1 dates processed, 1 reports: 1 completed, 0 failed > > Very low volume private mailserver ? > > > Kind regards, > Kim Sindalsen > >> -----Original Message----- >> From: Users On Behalf Of Kyle A. >> Sent: 22. september 2021 19:52 >> To: users at lists.rspamd.com >> Subject: Re: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load >> config >> >> No seg fault anymore, but this module seems broken: >> >> [mailserver:~ root]# rspamadm dmarc_report No reports for 20210922 No >> reports for 20210921 [mailserver:~ root]# >> >> At least 1k more messages have been handled since my last post. DMARC >> reporting worked for us in all the 2.x versions. >> >> Thanks, >> Kyle >