From aste at sveiks.lv  Mon Sep 13 14:05:48 2021
From: aste at sveiks.lv (Aste)
Date: Mon, 13 Sep 2021 17:05:48 +0300
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
References: <1907311023.20210823163332@latnet.lv>
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
Message-ID: <5710476131.20210913170548@sveiks.lv>

Hello Faisal,

Tuesday, August 31, 2021, 1:45:56 AM, you wrote:

FM> Hi there,

FM> If you remove reporting = true; it should work. You need to replace it by /enabled = true;/ inside the /reporting/ array which you already have.
FM> |# local.d/dmarc.conf reporting { # Required attributes enabled = true; # Enable reports in general|

Thank you it's solve  initial problem, bet reporting still doesn't work, now I getting error
Executing "rspamadm dmarc_report" results with error.

call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - <unknown> [Lua]};

sometimes  I getting errors for misconfigured domains like

cannot resolve ******
Cannot process reports for domain *****

So some parts of script is working, but no emails was sent, and  always getting  that error "call to rspamadm lua script failed" at the end.

-- 
Best regards,
 Aste                            mailto:aste at sveiks.lv

FM> For more info on the new parameters, check out the docs page at https://rspamd.com/doc/modules/dmarc.html#reporting

FM> On 8/23/21 8:33 AM, Aste wrote:
>> Hi,
>>
>> Seams that new (3.0 rspamd) DMARC report code doesnt't load config
>>
>> running  "rspamadm dmarc_report" returns
>> "dmarc reporting is not enabled, exiting"
>>
>> dmarc_settings = rspamd_config:get_all_opt('dmarc')  returns nill
>> located in  rspamd/lualib/rspamadm/dmarc_report.lua
>>
>> configdump founds dmarc section without problems
>>
>> rspamadm configdump  dmarc
>> *** Section dmarc ***
>> servers = "192.168.x.x:6379";
>> actions {
>>      quarantine = "add_header";
>>      reject = "reject";
>> }
>> reporting = true;
>> reporting {
>>      report_local_controller = false;
>>      msgid_from = "rspamd";
>>      enabled = true;
>>      keys_expire = 172800;
>>      domain = "*******";
>>      max_entries = 1000;
>>      from_name = "DMARC reporting";
>>      smtp = "127.0.0.1";
>>      smtp_port = 25;
>>      email = "postmaster@**********";
>>      helo = "rspamd.localhost";
>>      org_name = "*********";
>> }
>>
>>    >



From timc at slowb.ro  Fri Sep 17 07:50:18 2021
From: timc at slowb.ro (Tim C)
Date: Fri, 17 Sep 2021 17:50:18 +1000
Subject: [Rspamd-Users] DMARC policies on emails that forward from mailing
 lists go to spam
Message-ID: <e1b3c9c5-ff6e-ede1-3410-5f7c8f2f69b6@slowb.ro>

Hi Everyone,

Has anyone figured out a way in which emails from mailing lists can be 
allowed when they fail the DMARC policy, as the SPF doesn't align?

It seems to be a hit and miss for emails, and most likely it is due to 
the DMARC policy for the respective users.

Is this something I can setup in rspamd? As ideally I'd like to keep 
rspamd anti-spam/anti-virus and not just sieve filter all mailing lists 
before they anti-spam has a chance to flag it.

Cheers,

Tim

From thomas at plant.systems  Fri Sep 17 08:44:52 2021
From: thomas at plant.systems (Thomas)
Date: Fri, 17 Sep 2021 10:44:52 +0200
Subject: [Rspamd-Users] Disable symbol on certain filetype
Message-ID: <bf44c52f-2e8f-74ed-900f-caf6ad3e8b08@plant.systems>

Hello.

We have the problem as described in this github issue: 
https://github.com/rspamd/rspamd/issues/3837

When a file has a .p7m extension rspamd inserts BOGUS_ENCRYPTED_AND_TEXT 
and HEADER_BROKEN symbols and identifies the mail as spam.
Now I have made a simple multipmap rule to add a negative score if a p7m 
file is detected:

WHITELIST_FILE_EXTENSION {
 ??????? type = "filename";
 ??????? filter = "extension";
 ??????? map = "${LOCAL_CONFDIR}/maps/filename_extension_wl.map";
 ??????? score = -30;
 ??????? symbol = "WHITELIST_FILE_EXTENSION";
}


Is there a way to simply disable the two symbols 
BOGUS_ENCRYPTED_AND_TEXT/HEADER_BROKEN if we encounter a p7m file? Or is 
the above the better solution?

Regards,
Thomas

From philip.colmer at linaro.org  Mon Sep 20 07:38:09 2021
From: philip.colmer at linaro.org (Philip Colmer)
Date: Mon, 20 Sep 2021 08:38:09 +0100
Subject: [Rspamd-Users] Help needed with DKIM and ARC (mis)configuration
 please
Message-ID: <CAKTSSTgGEZMGdeX13xEZX=RjUrAUDJ40b0rKUtHpt9vNJLEJ-A@mail.gmail.com>

I'm trying to configure rspamd so that it provides all of the DKIM and
ARC handling when used in conjunction with Mailman 3. I'm using
Postfix as the MTA.

DKIM itself seems to be working, in that if Mailman 3 sends (say) a
password reset email, the receiving mail system says that the DKIM
headers are OK:

Delivered-To: philip.colmer at codelinaro.org
Received-SPF: pass (zohomail.com: domain of mm3.lavasoftware.org
designates 3.230.84.86 as permitted sender) client-ip=3.230.84.86;
envelope-from=postorius at mm3.lavasoftware.org;
helo=mm3.lavasoftware.org;
Authentication-Results: mx.zohomail.com;
    dkim=pass;
    spf=pass (zohomail.com: domain of mm3.lavasoftware.org designates
3.230.84.86 as permitted sender)
smtp.mailfrom=postorius at mm3.lavasoftware.org
ARC-Seal: i=1; a=rsa-sha256; t=1632122989; cv=none;
    d=zohomail.com; s=zohoarc;
    b=ANrhqpZBbcyb5vTHyW8/oZGzKVK+fqUVpAoqRToZ0ybk6nplq5gcj2MWN12osnpIT8XQzKx1W1VZ7yTnmsxZgMczbgLgGgu5DrQU8sz7kjGOoiUEMicJCga82GGHK6X6cFNF2Am83S3Hs2PRoIuriaPggASbTqLZ4gzNHuLlavw=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed;
d=zohomail.com; s=zohoarc;
    t=1632122989;
h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To;
    bh=Na5VyfB/MJFBjvpUi2TPUqikg420+PNUB2IXz4sZ3tw=;
    b=Ftpt4gX7nhNfDsuwEKH3X2gmu0Fo446u9IRLut6Y93QCJL0dP2/5funAQhcT5RwBglyeH/LMMCs6tu+0sFnSDteNLPbkmbLjK9uZtCnk42+uBRUqEdbPy8DSqgxUclA9KwKvIkd3asld0c8yMxo7u9jMdnkBs6TxeCphReF3vho=
ARC-Authentication-Results: i=1; mx.zohomail.com;
    dkim=pass;
    spf=pass (zohomail.com: domain of mm3.lavasoftware.org designates
3.230.84.86 as permitted sender)
smtp.mailfrom=postorius at mm3.lavasoftware.org
Return-Path: <postorius at mm3.lavasoftware.org>
Received: from mm3.lavasoftware.org (mm3.lavasoftware.org
[3.230.84.86]) by mx.zohomail.com
    with SMTPS id 1632122989893975.8082437998135; Mon, 20 Sep 2021
00:29:49 -0700 (PDT)
Received: from ip-172-31-2-177.ec2.internal (localhost [127.0.0.1])
    by mm3.lavasoftware.org (Postfix) with ESMTP id 4B440BE547
    for <philip.colmer at codelinaro.org>; Mon, 20 Sep 2021 07:29:48 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mm3.lavasoftware.org;
    s=dkim; t=1632122988;
    h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
     to:to:cc:mime-version:mime-version:content-type:content-type:
     content-transfer-encoding:content-transfer-encoding;
    bh=Na5VyfB/MJFBjvpUi2TPUqikg420+PNUB2IXz4sZ3tw=;
    b=k5LcpvNvZ3l0tCjoCy+T25HotZtilhzOtLmBUnC+nsja6gB/F/rrqTAYU3XcXKMvNATf2T
    gLrmn+kLVKv3dbCJyfaJyZqYIcK37klx/Cb88kv6m2vq6FSTtFiQZpzqGv9AvhPYK2BiZx
    +lHq0bmlJ6fp2jRbI5T9WbwtzzrJNUs=
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [lavasoftware.org] Password Reset E-mail
From: postorius at mm3.lavasoftware.org
To: philip.colmer at codelinaro.org
Date: Mon, 20 Sep 2021 07:29:48 -0000
Message-ID: <163212298829.38042.11881199049009320543 at ip-172-31-2-177.ec2.internal>
X-ZohoMail-DKIM: pass (identity @mm3.lavasoftware.org)
X-ZohoMail-Owner:
<163212298829.38042.11881199049009320543 at ip-172-31-2-177.ec2.internal>+zmo_0_postorius at mm3.lavasoftware.org

However, if I post to a mailing list on the server, the received email
is not considered to be OK:

Delivered-To: philip.colmer at codelinaro.org
Received-SPF: pass (zohomail.com: domain of mm3.lavasoftware.org
designates 3.230.84.86 as permitted sender) client-ip=3.230.84.86;
envelope-from=test-bounces+philip.colmer=codelinaro.org at mm3.lavasoftware.org;
helo=mm3.lavasoftware.org;
Authentication-Results: mx.zohomail.com;
    dkim=pass;
    spf=pass (zohomail.com: domain of mm3.lavasoftware.org designates
3.230.84.86 as permitted sender)
smtp.mailfrom=test-bounces+philip.colmer=codelinaro.org at mm3.lavasoftware.org;
    arc=fail (BodyHash is different from the expected one)
Return-Path: <test-bounces+philip.colmer=codelinaro.org at mm3.lavasoftware.org>
Received: from mm3.lavasoftware.org (mm3.lavasoftware.org
[3.230.84.86]) by mx.zohomail.com
    with SMTPS id 1632121826593429.3377835910852; Mon, 20 Sep 2021
00:10:26 -0700 (PDT)
Received: from ip-172-31-2-177.ec2.internal (localhost [127.0.0.1])
    by mm3.lavasoftware.org (Postfix) with ESMTP id ABFAFBEA8D
    for <philip.colmer at codelinaro.org>; Mon, 20 Sep 2021 07:10:25 +0000 (UTC)
Received: from sender4-op-o14.zoho.com (sender4-op-o14.zoho.com
[136.143.188.14])
    by mm3.lavasoftware.org (Postfix) with ESMTPS id B5FC1BE61C
    for <test at mm3.lavasoftware.org>; Mon, 20 Sep 2021 07:10:22 +0000 (UTC)
Received: from mail.zoho.com by mx.zohomail.com
    with SMTP id 1632121819028942.4423330943655; Mon, 20 Sep 2021
00:10:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mm3.lavasoftware.org;
    s=dkim; t=1632121825;
h=from:from:reply-to:reply-to:subject:subject:date:date:
     message-id:message-id:to:to:cc:mime-version:mime-version:
     content-type:content-type:list-id:list-help:list-owner:
     list-unsubscribe:list-subscribe:list-post;
    bh=zJ40PoiQolFhFDNtGN6hEUNOJtDCnSu+T+9+9qkGBJ4=;
    b=WlzzQQsEEiRUF2Bw3fKEHZ/v7CC0euY/qUEwLjhOy0jPNoKhCE1YuO+O2+AcaUAxneufqN
    RzZEPtbIYZm29w1jG5ZYoNmiDA42+v8ZasIagWc50+ryWDnX4YSSYk4RMBCXwznu5HBMjL
    DrRDPq4Q92WdPKsYho3ABb97Hzl4mlI=
ARC-Seal: i=1; a=rsa-sha256; t=1632121820; cv=none;
    d=zohomail.com; s=zohoarc;
    b=I7Nm7s4ikypN7iEcMzmWleseGtYzQUx2e72QEwV2hXKXxVQs0yrbx2744dcbNFAJgOMJl7C0PMFWVEMTm5uiFxN+zLwpvNJ5i9Sw5XQXNzkP4yuVBGC8R2+ZJ+I2W8e3dxsdOp1SZL7ROMEnbISXaIKLWAOmB8H0Cu8etwTuCHA=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed;
d=zohomail.com; s=zohoarc;
    t=1632121820; h=Content-Type:Date:From:MIME-Version:Message-ID:Subject:To;
    bh=UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=;
    b=DzYG0hVPiGOGagfchCVHJFkl1R9H2EvvB338GBsKGauzUtwLSIs3clIyoyr3vLTG2uBZYHZGSZk1JeODyAg/Y9x8UluqA1V4kUU5TITVyaX3dUslHhaXW5FP5ds1hK4+08lLan9SLKXtRrfNPonGG8B9umRieCpno8AcodDo+Ak=
ARC-Authentication-Results: i=1; mx.zohomail.com;
    dkim=pass header.i=codelinaro.org;
    spf=pass smtp.mailfrom=philip.colmer at codelinaro.org;
    dmarc=pass header.from=<philip.colmer at codelinaro.org>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1632121820;
    s=zoho; d=codelinaro.org; i=philip.colmer at codelinaro.org;
    h=Date:From:To:Message-Id:Subject:MIME-Version:Content-Type;
    bh=UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=;
    b=Ay3K1SsWxWR+ehw1xplPkf4c61D/hsK3rNDtV/gl1SpH5odBdVQtn+tD4X/jmSzo
    v4dlnV5v/SEu5bEKcx1+1t+jhOlqKoxpip2/O3i75KRbQutUoGKmS46c9Pd6l8q4/ww
    k7z05XUAHJIwOE0GcLJ+75UMvjPoOZTPNX2Xuk0k=
Date: Mon, 20 Sep 2021 08:10:19 +0100
To: "test" <test at mm3.lavasoftware.org>
Message-Id: <17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>
MIME-Version: 1.0
Importance: Medium
User-Agent: Zoho Mail
X-Mailer: Zoho Mail
ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed;
    d=mm3.lavasoftware.org; s=dkim; t=1632121823;
    h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
     to:to:cc:mime-version:mime-version:content-type:content-type:
     dkim-signature; bh=UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=;
    b=kd2l8DuYzeIwb+IV2iAe4QPqaTbpc11il+5GRnG6n38/kj2o7wbsbxSz/Xc9K2GndFHMcC
    zq8VXuca+inWtDsbiPPPUOP4T8HhfbhXOpCFGFrkb5kdCxju8dJ+HZeExVfgAQkPONBQIU
    NAUuuls+jnysLBLmeg8yy1lFOGXHKq8=
ARC-Authentication-Results: i=2;
    mm3.lavasoftware.org;
    dkim=pass header.d=codelinaro.org header.s=zoho header.b=Ay3K1SsW;
    dmarc=none;
    arc=pass ("zohomail.com:s=zohoarc:i=1");
    spf=pass (mm3.lavasoftware.org: domain of
philip.colmer at codelinaro.org designates 136.143.188.14 as permitted
sender) smtp.mailfrom=philip.colmer at codelinaro.org
ARC-Seal: i=2; s=dkim; d=mm3.lavasoftware.org; t=1632121823; a=rsa-sha256;
    cv=pass;
    b=sL4/9K5tcg3NqHS1Cn5HMUFqO02DB43na23WyE3ke6AB1igpY/1o7xDYvt6Vqu9OMc8X/7
    Tn75o1hi2cUrXD2Ju40FxirvG3Rwa47CzfvrHRXv6KhaYAe01Qbb1zmKe6ECV1MZxRy/OA
    52z4ccr1QBVCAM/j6BSSgNcO3uk13A4=
Message-ID-Hash: DJXYU37XDC54P2YNVHOUQOM5ILEFDLGG
X-Message-ID-Hash: DJXYU37XDC54P2YNVHOUQOM5ILEFDLGG
X-MailFrom: philip.colmer at codelinaro.org
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved;
emergency; loop; banned-address; member-moderation;
nonmember-moderation; administrivia; implicit-dest; max-recipients;
max-size; news-moderation; no-subject; digests; suspicious-header
X-Mailman-Version: 3.3.4
Precedence: list
Subject: [Test] Another ARC test email
List-Id: <test.mm3.lavasoftware.org>
Archived-At: <>
List-Archive: <>
List-Help: <mailto:test-request at mm3.lavasoftware.org?subject=help>
List-Owner: <mailto:test-owner at mm3.lavasoftware.org>
List-Post: <mailto:test at mm3.lavasoftware.org>
List-Subscribe: <mailto:test-join at mm3.lavasoftware.org>
List-Unsubscribe: <mailto:test-leave at mm3.lavasoftware.org>
From: Philip Colmer via Test <test at mm3.lavasoftware.org>
Reply-To: Philip Colmer <philip.colmer at codelinaro.org>
Content-Type: multipart/mixed; boundary="===============4180264220719142576=="
X-ZohoMail-DKIM: pass (identity @mm3.lavasoftware.org)

local.d/arc.conf:
allow_hdrfrom_mismatch = true;
allow_username_mismatch = true;
use_esld = false;
domain {
    mm3.lavasoftware.org {
        selector = "dkim";
        path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key";
    }
}

local.d/dkim_signing.conf:
allow_hdrfrom_mismatch_sign_networks = true;
allow_username_mismatch = true;
domain {
    mm3.lavasoftware.org {
        selector = "dkim";
        path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key";
    }
}
use_esld = false;
sign_authenticated = false;
use_domain = "header";
allow_hdrfrom_mismatch = true;

local.d/worker-proxy.inc:
milter = yes; # Enable milter mode
timeout = 120s; # Needed for Milter usually
upstream "local" {
  default = yes; # Self-scan upstreams are always default
  self_scan = yes; # Enable self-scan
}
count = 4; # Spawn more processes in self-scan mode
max_retries = 5; # How many times master is queried in case of failure
discard_on_reject = false; # Discard message instead of rejection
quarantine_on_reject = false; # Tell MTA to quarantine rejected messages
spam_header = "X-Spam"; # Use the specific spam header
reject_message = "Spam message rejected"; # Use custom rejection message

local.d/worker-normal.inc:
enabled = false;

Postfix configuration extract for the milter:

#smtpd_milters = unix:/var/lib/rspamd/milter.sock
# or for TCP socket
smtpd_milters = inet:localhost:11332
non_smtpd_milters = inet:localhost:11332

# skip mail without checks if something goes wrong
milter_default_action = accept

# 6 is the default milter protocol version;
# prior to Postfix 2.6 the default protocol was 2.
# milter_protocol = 6

When the email originally comes in to be submitted to the mailing
list, I think these are the relevant lines from the rspamd log:

2021-09-20 07:10:22 #37324(rspamd_proxy) <8e1c5c>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
36088
2021-09-20 07:10:22 #37324(rspamd_proxy) <8e1c5c>; milter;
rspamd_milter_process_command: got connection from
136.143.188.14:17451
2021-09-20 07:10:22 #37324(rspamd_proxy) <8e1c5c>; proxy;
rspamd_message_parse: loaded message; id:
<17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>;
queue-id: <B5FC1BE61C>; size: 2400; checksum:
<9f4bf024cb485cbef8e1b1e030a48b9b>
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy;
rspamd_spf_maybe_return: stored record for codelinaro.org
(0x4d20af7734be3ae1) in LRU cache for 300 seconds, 2/2000 elements in
the cache
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy;
dkim_module_key_handler: stored DKIM key for
zoho._domainkey.codelinaro.org in LRU cache for 300 seconds, 2/2000
elements in the cache
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; lua;
neural.lua:315: skip ham sample to keep spam/ham balance; probability
1; 0 spam and 1 ham vectors stored
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy;
rspamd_task_write_log: id:
<17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>, qid:
<B5FC1BE61C>, ip: 136.143.188.14, from:
<philip.colmer at codelinaro.org>, (default: F (no action): [-1.49/15.00]
[ARC_ALLOW(-1.00){zohomail.com:s=zohoarc:i=1;},R_DKIM_ALLOW(-0.20){codelinaro.org:s=zoho;},R_SPF_ALLOW(-0.20){+ip4:136.143.188.0/24;},MIME_GOOD(-0.10){multipart/alternative;text/plain;},XM_UA_NO_VERSION(0.01){},ARC_SIGNED(0.00){mm3.lavasoftware.org:s=dkim:i=2;},ASN(0.00){asn:2639,
ipnet:136.143.188.0/23,
country:US;},DKIM_TRACE(0.00){codelinaro.org:+;},DMARC_NA(0.00){codelinaro.org;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:~;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_TWO(0.00){2;},RCVD_IN_DNSWL_NONE(0.00){136.143.188.14:from;},RCVD_TLS_LAST(0.00){},RWL_MAILSPIKE_POSSIBLE(0.00){136.143.188.14:from;},TO_DN_ALL(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]),
len: 2400, time: 294.179ms, dns req: 27, digest:
<9f4bf024cb485cbef8e1b1e030a48b9b>, rcpts:
<test at mm3.lavasoftware.org>, mime_rcpts: <test at mm3.lavasoftware.org>
2021-09-20 07:10:23 #37324(rspamd_proxy) <8e1c5c>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned
using pcre, 1.00KiB scanned total
2021-09-20 07:10:23 #37324(rspamd_proxy) <378793>; proxy;
proxy_milter_finish_handler: finished milter connection

Then, when Mailman is sending the modified email back out, these are
the relevant lines:

2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
36142
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; milter;
rspamd_milter_process_command: got connection from 127.0.0.1:54506
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
rspamd_message_parse: loaded message; id:
<17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>;
queue-id: <5C73ABE61C>; size: 5079; checksum:
<dc5a7d4f5acc7e072e651fb61d0d43d5>
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
rspamd_mime_part_detect_language: detected part language: en
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua;
greylist.lua:204: skip greylisting for local networks and/or
authorized users
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
dkim_symbol_callback: skip DKIM checks for local networks and
authorized users
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua; spf.lua:186:
skip SPF checks for local networks and authorized users
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua; dmarc.lua:349:
skip DMARC checks as either SPF or DKIM were not checked
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua;
once_received.lua:99: Skipping once_received for authenticated user or
local network
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; dkim;
rspamd_dkim_check: arc_sig: bh value mismatch: got
lBu5g1QPVcNe04bBZ1chCV35tI3E+GY1OFtuw7mUsAY=, expected
UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=; body length 1081->1079;
d=mm3.lavasoftware.org; s=dkim
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
rspamd_task_write_log: id:
<17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>, qid:
<5C73ABE61C>, ip: 127.0.0.1, from:
<test-bounces at mm3.lavasoftware.org>, (default: F (no action):
[0.80/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] =
sig:mm3.lavasoftware.org:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){test at mm3.lavasoftware.org;test-bounces at mm3.lavasoftware.org;},HAS_REPLYTO(0.00){philip.colmer at codelinaro.org;},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},REPLYTO_DOM_NEQ_FROM_DOM(0.00){},TAGGED_FROM(0.00){philip.colmer=linaro.org;},TO_DN_ALL(0.00){},TO_EQ_FROM(0.00){}]),
len: 5079, time: 236.610ms, dns req: 13, digest:
<dc5a7d4f5acc7e072e651fb61d0d43d5>, rcpts: <philip.colmer at linaro.org>,
mime_rcpts: <test at mm3.lavasoftware.org>
2021-09-20 07:10:25 #37321(rspamd_proxy) <558bb4>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 86 regexps cached, 0B scanned
using pcre, 2.01KiB scanned total
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; proxy;
rspamd_message_parse: loaded message; id:
<17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>;
queue-id: <ABFAFBEA8D>; size: 5079; checksum:
<dc5a7d4f5acc7e072e651fb61d0d43d5>
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; proxy;
rspamd_mime_part_detect_language: detected part language: en
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; lua;
greylist.lua:204: skip greylisting for local networks and/or
authorized users
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; proxy;
dkim_symbol_callback: skip DKIM checks for local networks and
authorized users
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; lua; spf.lua:186:
skip SPF checks for local networks and authorized users
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; dkim;
rspamd_dkim_check: arc_sig: bh value mismatch: got
zJ40PoiQolFhFDNtGN6hEUNOJtDCnSu+T+9+9qkGBJ4=, expected
UafOni038olf8S2QOrAIdd1uG+CPy6lUXSacJpOh40o=; body length 1081->1079;
d=mm3.lavasoftware.org; s=dkim
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; lua; dmarc.lua:349:
skip DMARC checks as either SPF or DKIM were not checked
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; lua;
once_received.lua:99: Skipping once_received for authenticated user or
local network
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; proxy;
rspamd_task_write_log: id:
<17c020a8f7a.adeaa2cf45941.5022929780341533192 at codelinaro.org>, qid:
<ABFAFBEA8D>, ip: 127.0.0.1, from:
<test-bounces at mm3.lavasoftware.org>, (default: F (no action):
[0.79/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] =
sig:mm3.lavasoftware.org:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){test at mm3.lavasoftware.org;test-bounces at mm3.lavasoftware.org;},HAS_REPLYTO(0.00){philip.colmer at codelinaro.org;},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},REPLYTO_DOM_NEQ_FROM_DOM(0.00){},TAGGED_FROM(0.00){philip.colmer=codelinaro.org;},TO_DN_ALL(0.00){},TO_EQ_FROM(0.00){}]),
len: 5079, time: 11.408ms, dns req: 12, digest:
<dc5a7d4f5acc7e072e651fb61d0d43d5>, rcpts:
<philip.colmer at codelinaro.org>, mime_rcpts:
<test at mm3.lavasoftware.org>
2021-09-20 07:10:25 #37321(rspamd_proxy) <f51081>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 86 regexps cached, 0B scanned
using pcre, 2.01KiB scanned total
2021-09-20 07:10:25 #37321(rspamd_proxy) <631f9b>; proxy;
proxy_milter_finish_handler: finished milter connection
2021-09-20 07:16:14 #37324(rspamd_proxy) <52c995>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
39796
2021-09-20 07:16:14 #37324(rspamd_proxy) <52c995>; milter;
rspamd_milter_process_command: got connection from 35.179.93.71:47834
2021-09-20 07:16:14 #37324(rspamd_proxy) <52c995>; proxy;
proxy_milter_finish_handler: finished milter connection
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
48300
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; milter;
rspamd_milter_process_command: got connection from 127.0.0.1:38432
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
rspamd_message_parse: loaded message; id:
<163212298829.38042.11881199049009320543 at ip-172-31-2-177.ec2.internal>;
queue-id: <4B440BE547>; size: 816; checksum:
<cb2b529764ef7f1d154abc4b82e1623c>
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
rspamd_mime_part_detect_language: detected part language: en
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; lua;
greylist.lua:204: skip greylisting for local networks and/or
authorized users
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
dkim_symbol_callback: skip DKIM checks for local networks and
authorized users
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; lua; spf.lua:186:
skip SPF checks for local networks and authorized users
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; lua; dmarc.lua:349:
skip DMARC checks as either SPF or DKIM were not checked
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; lua;
once_received.lua:99: Skipping once_received for authenticated user or
local network
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; lua;
neural.lua:315: skip ham sample to keep spam/ham balance; probability
1; 0 spam and 1 ham vectors stored
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
rspamd_task_write_log: id:
<163212298829.38042.11881199049009320543 at ip-172-31-2-177.ec2.internal>,
qid: <4B440BE547>, ip: 127.0.0.1, from:
<postorius at mm3.lavasoftware.org>, (default: F (no action):
[-0.10/15.00] [MIME_GOOD(-0.10){text/plain;},ARC_NA(0.00){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FROM_EQ_ENVFROM(0.00){},FROM_NO_DN(0.00){},MIME_TRACE(0.00){0:+;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_ZERO(0.00){0;},TO_DN_NONE(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]),
len: 816, time: 290.349ms, dns req: 15, digest:
<cb2b529764ef7f1d154abc4b82e1623c>, rcpts:
<philip.colmer at codelinaro.org>, mime_rcpts:
<philip.colmer at codelinaro.org>
2021-09-20 07:29:48 #37324(rspamd_proxy) <aedc64>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
2 regexps matched, 176 regexps total, 47 regexps cached, 0B scanned
using pcre, 1.89KiB scanned total
2021-09-20 07:29:48 #37324(rspamd_proxy) <2f90dc>; proxy;
proxy_milter_finish_handler: finished milter connection
2021-09-20 07:36:16 #37323(rspamd_proxy) <06ee59>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
52314
2021-09-20 07:36:16 #37323(rspamd_proxy) <06ee59>; milter;
rspamd_milter_process_command: got connection from
18.170.226.166:21345
2021-09-20 07:36:16 #37323(rspamd_proxy) <06ee59>; proxy;
proxy_milter_finish_handler: finished milter connection

I realise there is a lot to digest there but does anyone have any
suggestions on what I've misconfigured, please?

Thanks.

Regards

Philip

From chris at cretaforce.gr  Mon Sep 20 14:52:13 2021
From: chris at cretaforce.gr (Christos Chatzaras)
Date: Mon, 20 Sep 2021 17:52:13 +0300
Subject: [Rspamd-Users] rate limit and mail delivery reports
Message-ID: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr>

Hello,

I use two postfix relays to send e-mails from other servers.

As I can't use selector 'user.lower' because the relays don't have users, my ratelimit.conf contains:

rates {
    "12000" = {
      selector = 'from.lower';
      bucket = {
        burst = 12000;
        rate = "12000 / 1d";
      }
    }
}

The problem is that if many senders ask for "delivery reports" then this rate limit is hit because all these reports use "From: MAILER-DAEMON at smtp1.example.com" or "From: MAILER-DAEMON at smtp2.example.com". Is any way to exclude MAILER-DAEMON at smtp1.example.com and MAILER-DAEMON at smtp2.example.com from the rate limit?


From kyle at cci1986.com  Mon Sep 20 17:05:25 2021
From: kyle at cci1986.com (Kyle A.)
Date: Mon, 20 Sep 2021 13:05:25 -0400
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <5710476131.20210913170548@sveiks.lv>
References: <1907311023.20210823163332@latnet.lv>
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
Message-ID: <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>

Aste,

Did you get it resolved?

Today I have upgraded V2.7 -> V3.0, updated the dmarc.conf, and I have 
almost the exact same error you reported.  Mine says this:

# rspamadm dmarc_report
call to rspamadm lua script failed (2): 
/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid 
argument: nil; trace: 
[1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - 
process_report_entry [Lua]}; 
[2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - 
prepare_report [Lua]}; 
[3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - 
process_report_date [Lua]}; 
[4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - <unknown> 
[Lua]};

Upon further review, I think my error is character for character the 
same as your error.  I do not understand and there is no help for this.

-Kyle

On 9/13/2021 10:05 AM, Aste wrote:
> Thank you it's solve  initial problem, bet reporting still doesn't work, now I getting error
> Executing "rspamadm dmarc_report" results with error.
> 
> call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - <unknown> [Lua]};
> 

From m.kliewe at team.mail.de  Mon Sep 20 20:39:41 2021
From: m.kliewe at team.mail.de (Michael Kliewe)
Date: Mon, 20 Sep 2021 22:39:41 +0200
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
References: <1907311023.20210823163332@latnet.lv>
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
 <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
Message-ID: <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>

Hi,

Am 20.09.2021 um 19:05 schrieb Kyle A.:
> Aste,
>
> Did you get it resolved?
>
> Today I have upgraded V2.7 -> V3.0, updated the dmarc.conf, and I have 
> almost the exact same error you reported.? Mine says this:
>
> # rspamadm dmarc_report
> call to rspamadm lua script failed (2): 
> /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid 
> argument: nil; trace: 
> [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - 
> process_report_entry [Lua]}; 
> [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - 
> prepare_report [Lua]}; 
> [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - 
> process_report_date [Lua]}; 
> [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - 
> <unknown> [Lua]};
>
> Upon further review, I think my error is character for character the 
> same as your error.? I do not understand and there is no help for this.

I also got this error when I first manually ran
sudo rspamadm dmarc_report
after upgrading from 2.7 to 3.0.2 on Ubuntu.

But:
One day later I ran it again manually:
sudo rspamadm dmarc_report -v
(with verbose mode on) and it was working fine, it sent 10 DMARC reports.

Maybe there was some old/invalid data in the redis storage, which is why 
the first invocation failed with "invalid argument: nil"?

Please do us all a favor and don't run the cronjob at exactly midnight. 
Choose a random time in a day, then rate-limits at recipient mailboxes 
will not be a problem, and server load is distributed. If you receive 
bounces because the recipient mailbox is over quota or does not exist, 
or does not accept attachments, try to contact the recipient somehow and 
tell them about the problem. We need to get those bounce-problems fixed 
together.

Michael


>
> On 9/13/2021 10:05 AM, Aste wrote:
>> Thank you it's solve? initial problem, bet reporting still doesn't 
>> work, now I getting error
>> Executing "rspamadm dmarc_report" results with error.
>>
>> call to rspamadm lua script failed (2): 
>> /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid 
>> argument: nil; trace: 
>> [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - 
>> process_report_entry [Lua]}; 
>> [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - 
>> prepare_report [Lua]}; 
>> [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - 
>> process_report_date [Lua]}; 
>> [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - 
>> <unknown> [Lua]}; 


From bruns at 2mbit.com  Mon Sep 20 20:30:13 2021
From: bruns at 2mbit.com (Brielle)
Date: Mon, 20 Sep 2021 14:30:13 -0600
Subject: [Rspamd-Users] rspamd not handling pause / multi-line 220- in SMTP
 exchange
Message-ID: <9830de2e-0916-2238-ec5f-22ed7f742bf1@2mbit.com>

Hello,

Discovered a slight issue with rspamd's dmarc_report feature, getting 
following errors:

Couldn't send mail for inmoment.com: bad smtp responce on stage rcpt: 
"220-" when "3" expected


When checking exim's logs...

2021-09-20 14:19:05 SMTP protocol synchronization error (next input sent 
too soon: pipelining was not advertised): rejected "HELO 
mail.xxxxxx.org" H=localhost [127.0.0.1] U=root next input="MAIL FROM: 
<dmarc_reports at xxxxxxxxxx>\r\nRCPT TO: 
<dmarc-reports at xxxxxxxxxxxx\r\nRCPT TO: 
<xxxxxxxxxxxxx at xxxxxxxxxxxxxx>\r\nDATA\r\n"


My SMTP server adds a momentary delay before spitting out a multi line 
220 banner (using 220-) which is designed to mess with spambots that 
don't bother to implement proper RFC compliance.

Looks like the report feature either doesn't wait for the final 220 
before sending the HELO, or doesn't recognize that 220- means multi line 
(see RFC 2821 sec 3.1) and just starts trying to send commands right away.

So, not sure if its the initial delay that's causing the problem or the 
220 handling.

-- 
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org    /     http://www.ahbl.org

From chris at cretaforce.gr  Mon Sep 20 20:56:31 2021
From: chris at cretaforce.gr (Christos Chatzaras)
Date: Mon, 20 Sep 2021 23:56:31 +0300
Subject: [Rspamd-Users] rate limit and mail delivery reports
In-Reply-To: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr>
References: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr>
Message-ID: <8F97ABAD-AC79-4C6A-B92F-EA748E260E7B@cretaforce.gr>


> On 20 Sep 2021, at 17:52, Christos Chatzaras <chris at cretaforce.gr> wrote:
> 
> Hello,
> 
> I use two postfix relays to send e-mails from other servers.
> 
> As I can't use selector 'user.lower' because the relays don't have users, my ratelimit.conf contains:
> 
> rates {
>    "12000" = {
>      selector = 'from.lower';
>      bucket = {
>        burst = 12000;
>        rate = "12000 / 1d";
>      }
>    }
> }
> 
> The problem is that if many senders ask for "delivery reports" then this rate limit is hit because all these reports use "From: MAILER-DAEMON at smtp1.example.com" or "From: MAILER-DAEMON at smtp2.example.com". Is any way to exclude MAILER-DAEMON at smtp1.example.com and MAILER-DAEMON at smtp2.example.com from the rate limit?
> 

Finally I made some changes to my setup to be able to use 'user.lower' selector.

From helge.wiethoff at thga.de  Tue Sep 21 07:13:39 2021
From: helge.wiethoff at thga.de (Wiethoff, Helge)
Date: Tue, 21 Sep 2021 07:13:39 +0000
Subject: [Rspamd-Users] rate limit and mail delivery reports
In-Reply-To: <8F97ABAD-AC79-4C6A-B92F-EA748E260E7B@cretaforce.gr>
References: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr>
 <8F97ABAD-AC79-4C6A-B92F-EA748E260E7B@cretaforce.gr>
Message-ID: <da70fc87bd32eab58db0ba89ad4a6125b278f951.camel@thga.de>

Hi Christos,

Am Montag, dem 20.09.2021 um 23:56 +0300 schrieb Christos Chatzaras:
> Finally I made some changes to my setup to be able to use 'user.lower' selector.

Could you be so kind and post your changes? :-)

Best
Helge

From chris at cretaforce.gr  Tue Sep 21 07:42:45 2021
From: chris at cretaforce.gr (Christos Chatzaras)
Date: Tue, 21 Sep 2021 10:42:45 +0300
Subject: [Rspamd-Users] rate limit and mail delivery reports
In-Reply-To: <da70fc87bd32eab58db0ba89ad4a6125b278f951.camel@thga.de>
References: <671B0BA0-27C6-48F0-B77B-3475F30774F0@cretaforce.gr>
 <8F97ABAD-AC79-4C6A-B92F-EA748E260E7B@cretaforce.gr>
 <da70fc87bd32eab58db0ba89ad4a6125b278f951.camel@thga.de>
Message-ID: <247A76EC-B946-4403-B91B-92DA6C456E39@cretaforce.gr>



> On 21 Sep 2021, at 10:13, Wiethoff, Helge <helge.wiethoff at thga.de> wrote:
> 
> Hi Christos,
> 
> Am Montag, dem 20.09.2021 um 23:56 +0300 schrieb Christos Chatzaras:
>> Finally I made some changes to my setup to be able to use 'user.lower' selector.
> 
> Could you be so kind and post your changes? :-)
> 
> Best
> Helge

My setup was:

1) For my servers I was using rspamd only for smtp and not for submission. So I was scanning incoming messages on the servers and not outgoing.

2) Outgoing e-mails were forwarded to two relays that do the final delivery. In these relays I was using rspamd to check outgoing e-mails.

My new setup is:

1) My servers do rspamd checks for both smtp and for submission. So I scan both incoming and outgoing messages on the server level.

2) Outgoing e-mails are forwarded to two relays but as e-mails already scanned on the server level I remove scanning at the relay level.

With my old setup and "from.lower" selector rate limiting was enabled in both incoming and outgoing e-mails which caused me this issue with mail delivery reports when this limit hit for incoming e-mails.

With these changes all scanning is done in the server level and by using "user.lower" instead of "from.lower" I have rate limiting enabled only for outgoing e-mails and not for incoming e-mails.

From aste at sveiks.lv  Tue Sep 21 07:59:51 2021
From: aste at sveiks.lv (Aste)
Date: Tue, 21 Sep 2021 10:59:51 +0300
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
References: <1907311023.20210823163332@latnet.lv> 
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
 <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
Message-ID: <1994181029.20210921105951@latnet.lv>

Hi!

Monday, September 20, 2021, 8:05:25 PM, you wrote:
KA> Aste,
KA> Did you get it resolved?
Didn't find solution, revertet back to 2.7 with  some bugs in DMARC reporting, but at least this version can send reports.

-- 
Best regards
Aste


KA> Today I have upgraded V2.7 -> V3.0, updated the dmarc.conf, and I have almost the exact same error you reported.  Mine says this:

KA> # rspamadm dmarc_report
KA> call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - <unknown> [Lua]};

KA> Upon further review, I think my error is character for character the same as your error.  I do not understand and there is no help for this.

KA> -Kyle

KA> On 9/13/2021 10:05 AM, Aste wrote:
>> Thank you it's solve  initial problem, bet reporting still doesn't work, now I getting error
>> Executing "rspamadm dmarc_report" results with error.
>> > call to rspamadm lua script failed (2): /usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258: invalid argument: nil; trace: [1]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:258 - process_report_entry [Lua]}; [2]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:507 - prepare_report [Lua]}; [3]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:602 - process_report_date [Lua]}; [4]:{/usr/share/rspamd/lualib/rspamadm//dmarc_report.lua:679 - <unknown> [Lua]};
>> -- 
KA> Users mailing list
KA> Users at lists.rspamd.com
KA> https://lists.rspamd.com/mailman/listinfo/users




From aste at sveiks.lv  Tue Sep 21 08:21:21 2021
From: aste at sveiks.lv (Aste)
Date: Tue, 21 Sep 2021 11:21:21 +0300
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>
References: <1907311023.20210823163332@latnet.lv> 
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
 <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
 <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>
Message-ID: <744144092.20210921112121@latnet.lv>

Hi!

Monday, September 20, 2021, 11:39:41 PM, you wrote:
MK> I also got this error when I first manually ran
MK> sudo rspamadm dmarc_report
MK> after upgrading from 2.7 to 3.0.2 on Ubuntu.

MK> But:
MK> One day later I ran it again manually:
MK> sudo rspamadm dmarc_report -v
MK> (with verbose mode on) and it was working fine, it sent 10 DMARC reports.

MK> Maybe there was some old/invalid data in the redis storage, which is why the first invocation failed with "invalid argument: nil"?
I was  changed redis prefix to avoid usage of old data, but it doesn't  help.  Also after 2 days the same error. 
Some parts of script was working, because I get errors about misconfigured domains, but at the end always the same script erros and no reports was sent.

-- 
Best regards
Aste


From a.wass at glas-gasperlmair.at  Tue Sep 21 09:53:20 2021
From: a.wass at glas-gasperlmair.at (Andreas Wass - Glas Gasperlmair)
Date: Tue, 21 Sep 2021 11:53:20 +0200
Subject: [Rspamd-Users] multimap - content filter not working
Message-ID: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at>

Hi,

i'm trying to use rspamd as soon as possible for our company mailserver, 
and testing content filtering with multimap as shown in

https://rspamd.com/doc/modules/multimap.html

*Configured the following.*..


    Multiple symbol maps

 From the version 1.3.1, it is possible to define multiple symbols and 
scores using multimap module. To do that, you should define all possible 
symbols using|symbols|option in multimap:

|# local.d/multimap.conf CONTENT_BLACKLISTED { type = "content"; filter 
= "body"; # can be headers, full, oneline, text, rawtext map = 
"${LOCAL_CONFDIR}/local.d/local_content.map"; symbols = 
["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; regexp = true; } |

In this example, you can use 3 symbols:

  * CONTENT_BLACKLISTED
  * CONTENT_BLACKLISTED1
  * CONTENT_BLACKLISTED2

the map:

|# Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default 
score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: 
CONTENT_BLACKLISTED /re3/ |

Symbols that are not defined in the|symbols|attribute but used in the 
map are ignored and replaced by the default map symbol. If the value of 
a key-value pair is missing, then Rspamd just inserts the default symbol 
with dynamic weight equal to|1.0|(which is multiplied by metric score 
afterwards)


*...matches the right symbol (CONTENT_BLACKLISTED1) in the header :*

X-Spamd-Result: default: False [0.80 / 15.00];
	...
	CONTENT_BLACKLISTED1(0.00)[];
	...

*...but why is score 0.00 and not 10 like it is defined in 
local_content.map?*


best regards, Andy


From a.wass at glas-gasperlmair.at  Tue Sep 21 10:43:13 2021
From: a.wass at glas-gasperlmair.at (Andreas Wass - Glas Gasperlmair)
Date: Tue, 21 Sep 2021 12:43:13 +0200
Subject: [Rspamd-Users] multimap - content filter not working
In-Reply-To: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at>
References: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at>
Message-ID: <bc1c3c7b-3acf-822f-f95d-58fd7c281df1@glas-gasperlmair.at>

sorry, something went wrong with formating in my first post

Hi,

i'm trying to use rspamd as soon as possible for our company mailserver, 
and testing content filtering with multimap as shown in

https://rspamd.com/doc/modules/multimap.html

Configured the following...
Multiple symbol maps

 From the version 1.3.1, it is possible to define multiple symbols and 
scores using multimap module. To do that, you should define all possible 
symbols using symbols option in multimap:

# local.d/multimap.conf
CONTENT_BLACKLISTED {
 ? type = "content";
 ? filter = "body"; # can be headers, full, oneline, text, rawtext
 ? map = "${LOCAL_CONFDIR}/local.d/local_content.map";
 ? symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"];
 ? regexp = true;
}

In this example, you can use 3 symbols:

 ??? CONTENT_BLACKLISTED
 ??? CONTENT_BLACKLISTED1
 ??? CONTENT_BLACKLISTED2

the map:

# Symbol + score
/re1/ CONTENT_BLACKLISTED1:10
# Symbol with default score
/re2/ CONTENT_BLACKLISTED2
# Just a default symbol: CONTENT_BLACKLISTED
/re3/

Symbols that are not defined in the symbols attribute but used in the 
map are ignored and replaced by the default map symbol. If the value of 
a key-value pair is missing, then Rspamd just inserts the default symbol 
with dynamic weight equal to 1.0 (which is multiplied by metric score 
afterwards)


...matches the right symbol (CONTENT_BLACKLISTED1) in the header :

X-Spamd-Result: default: False [0.80 / 15.00];
 ??? ...
 ??? CONTENT_BLACKLISTED1(0.00)[];
 ??? ...

...but why is score 0.00 and not 10 like it is defined in local_content.map?


best regards, Andy

Am 21.09.2021 um 11:53 schrieb Andreas Wass - Glas Gasperlmair:
> Hi,
>
> i'm trying to use rspamd as soon as possible for our company 
> mailserver, and testing content filtering with multimap as shown in
>
> https://rspamd.com/doc/modules/multimap.html
>
> *Configured the following.*..
>
>
> ?? Multiple symbol maps
>
> From the version 1.3.1, it is possible to define multiple symbols and 
> scores using multimap module. To do that, you should define all 
> possible symbols using|symbols|option in multimap:
>
> |# local.d/multimap.conf CONTENT_BLACKLISTED { type = "content"; 
> filter = "body"; # can be headers, full, oneline, text, rawtext map = 
> "${LOCAL_CONFDIR}/local.d/local_content.map"; symbols = 
> ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; regexp = true; } |
>
> In this example, you can use 3 symbols:
>
> ?* CONTENT_BLACKLISTED
> ?* CONTENT_BLACKLISTED1
> ?* CONTENT_BLACKLISTED2
>
> the map:
>
> |# Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default 
> score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: 
> CONTENT_BLACKLISTED /re3/ |
>
> Symbols that are not defined in the|symbols|attribute but used in the 
> map are ignored and replaced by the default map symbol. If the value 
> of a key-value pair is missing, then Rspamd just inserts the default 
> symbol with dynamic weight equal to|1.0|(which is multiplied by metric 
> score afterwards)
>
>
> *...matches the right symbol (CONTENT_BLACKLISTED1) in the header :*
>
> X-Spamd-Result: default: False [0.80 / 15.00];
> ????...
> ????CONTENT_BLACKLISTED1(0.00)[];
> ????...
>
> *...but why is score 0.00 and not 10 like it is defined in 
> local_content.map?*
>
>
> best regards, Andy
>


From aste at sveiks.lv  Tue Sep 21 11:12:20 2021
From: aste at sveiks.lv (Aste)
Date: Tue, 21 Sep 2021 14:12:20 +0300
Subject: [Rspamd-Users] multimap - content filter not working
In-Reply-To: <bc1c3c7b-3acf-822f-f95d-58fd7c281df1@glas-gasperlmair.at>
References: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at> 
 <bc1c3c7b-3acf-822f-f95d-58fd7c281df1@glas-gasperlmair.at>
Message-ID: <1772880288.20210921141220@latnet.lv>

Hi!

Tuesday, September 21, 2021, 1:43:13 PM, you wrote:
AWGG> # local.d/multimap.conf
AWGG> CONTENT_BLACKLISTED {
AWGG>  ? type = "content";
AWGG>  ? filter = "body"; # can be headers, full, oneline, text, rawtext
AWGG>  ? map = "${LOCAL_CONFDIR}/local.d/local_content.map";
According to documentation you need to specify map type as multimap
map  = regexp_multi;  ${LOCAL_CONFDIR}/local.d/local_content.map;

And you always can add global score there
score = 10;

AWGG>  ? symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"];
AWGG>  ? regexp = true;
AWGG> }

AWGG> In this example, you can use 3 symbols:

AWGG>  ??? CONTENT_BLACKLISTED
AWGG>  ??? CONTENT_BLACKLISTED1
AWGG>  ??? CONTENT_BLACKLISTED2

AWGG> the map:

AWGG> # Symbol + score
AWGG> /re1/ CONTENT_BLACKLISTED1:10
AWGG> # Symbol with default score
AWGG> /re2/ CONTENT_BLACKLISTED2
AWGG> # Just a default symbol: CONTENT_BLACKLISTED
AWGG> /re3/

AWGG> Symbols that are not defined in the symbols attribute but used in the map are ignored and replaced by the default map symbol. If the value of a key-value pair is missing, then Rspamd just inserts the default symbol with dynamic weight equal to 1.0 (which is multiplied by metric score afterwards)


AWGG> ...matches the right symbol (CONTENT_BLACKLISTED1) in the header :

AWGG> X-Spamd-Result: default: False [0.80 / 15.00];
AWGG>  ??? ...
AWGG>  ??? CONTENT_BLACKLISTED1(0.00)[];
AWGG>  ??? ...

AWGG> ...but why is score 0.00 and not 10 like it is defined in local_content.map?


AWGG> best regards, Andy

AWGG> Am 21.09.2021 um 11:53 schrieb Andreas Wass - Glas Gasperlmair:
>> Hi,
>>
>> i'm trying to use rspamd as soon as possible for our company > mailserver, and testing content filtering with multimap as shown in
>>
>> https://rspamd.com/doc/modules/multimap.html
>>
>> *Configured the following.*..
>>
>>
>> ?? Multiple symbol maps
>>
>> From the version 1.3.1, it is possible to define multiple symbols and > scores using multimap module. To do that, you should define all > possible symbols using|symbols|option in multimap:
>>
>> |# local.d/multimap.conf CONTENT_BLACKLISTED { type = "content"; > filter = "body"; # can be headers, full, oneline, text, rawtext map = > "${LOCAL_CONFDIR}/local.d/local_content.map"; symbols = > ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; regexp = true; } |
>>
>> In this example, you can use 3 symbols:
>>
>> ?* CONTENT_BLACKLISTED
>> ?* CONTENT_BLACKLISTED1
>> ?* CONTENT_BLACKLISTED2
>>
>> the map:
>>
>> |# Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default > score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: > CONTENT_BLACKLISTED /re3/ |
>>
>> Symbols that are not defined in the|symbols|attribute but used in the > map are ignored and replaced by the default map symbol. If the value > of a key-value pair is missing, then Rspamd just inserts the default > symbol with dynamic weight equal to|1.0|(which is multiplied by metric > score afterwards)
>>
>>
>> *...matches the right symbol (CONTENT_BLACKLISTED1) in the header :*
>>
>> X-Spamd-Result: default: False [0.80 / 15.00];
>> ????...
>> ????CONTENT_BLACKLISTED1(0.00)[];
>> ????...
>>
>> *...but why is score 0.00 and not 10 like it is defined in > local_content.map?*
>>
>>
>> best regards, Andy
>>




-- 
Best regards 
Aste


From a.wass at glas-gasperlmair.at  Tue Sep 21 12:49:09 2021
From: a.wass at glas-gasperlmair.at (Andreas Wass - Glas Gasperlmair)
Date: Tue, 21 Sep 2021 14:49:09 +0200
Subject: [Rspamd-Users] multimap - content filter not working
In-Reply-To: <1772880288.20210921141220@latnet.lv>
References: <46640442-9850-b2e7-c43e-14e65b29de6b@glas-gasperlmair.at>
 <bc1c3c7b-3acf-822f-f95d-58fd7c281df1@glas-gasperlmair.at>
 <1772880288.20210921141220@latnet.lv>
Message-ID: <46c0cd9b-77ac-a289-1416-0d2a186e0330@glas-gasperlmair.at>

found a solution and did it this way:

1. step

vi /etc/rspamd/local.d/multimap.conf

CONTENT_BLACKLISTED {
 ? type = "content";
 ? filter = "body"; # can be headers, full, oneline, text, rawtext
 ? map = "${LOCAL_CONFDIR}/local.d/local_content.map";
 ? symbols = ["CONTENT_BLACKLISTED1"];
 ? regexp = true;
}

2. step
vi /etc/rspamd/local.d/groups.conf

group "MyGroup" {
 ? symbols =? {
 ??? "CONTENT_BLACKLISTED1" {
 ????? # score = Multiplikator und wird mit Wert in local_content.map 
(steht hinter dem Doppelpunkt z.B. /Spende/ CONTENT_BLACKLISTED1:10) 
multipliziert
 ????? score = 1.0;
 ??? }
 ? }
}

3. step
vi /etc/rspamd/local.d/local_content.map

# searchtext (case sensitiv) - Symbol - Score
# Score wird mit Multiplikator in /etc/rspamd/local.d/groups.conf 
multipliziert

# Score >= 15 rejecten
/Wir haben eine Spende f?r Sie/ CONTENT_BLACKLISTED1:30

# ab Score >= 6 soft reject (greylist)
/Die Spende/ CONTENT_BLACKLISTED1:7


best regards, Andy

Am 21.09.2021 um 13:12 schrieb Aste:
> Hi!
>
> Tuesday, September 21, 2021, 1:43:13 PM, you wrote:
> AWGG> # local.d/multimap.conf
> AWGG> CONTENT_BLACKLISTED {
> AWGG>  ? type = "content";
> AWGG>  ? filter = "body"; # can be headers, full, oneline, text, rawtext
> AWGG>  ? map = "${LOCAL_CONFDIR}/local.d/local_content.map";
> According to documentation you need to specify map type as multimap
> map  = regexp_multi;  ${LOCAL_CONFDIR}/local.d/local_content.map;
>
> And you always can add global score there
> score = 10;
>
> AWGG>  ? symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"];
> AWGG>  ? regexp = true;
> AWGG> }
>
> AWGG> In this example, you can use 3 symbols:
>
> AWGG>  ??? CONTENT_BLACKLISTED
> AWGG>  ??? CONTENT_BLACKLISTED1
> AWGG>  ??? CONTENT_BLACKLISTED2
>
> AWGG> the map:
>
> AWGG> # Symbol + score
> AWGG> /re1/ CONTENT_BLACKLISTED1:10
> AWGG> # Symbol with default score
> AWGG> /re2/ CONTENT_BLACKLISTED2
> AWGG> # Just a default symbol: CONTENT_BLACKLISTED
> AWGG> /re3/
>
> AWGG> Symbols that are not defined in the symbols attribute but used in the map are ignored and replaced by the default map symbol. If the value of a key-value pair is missing, then Rspamd just inserts the default symbol with dynamic weight equal to 1.0 (which is multiplied by metric score afterwards)
>
>
> AWGG> ...matches the right symbol (CONTENT_BLACKLISTED1) in the header :
>
> AWGG> X-Spamd-Result: default: False [0.80 / 15.00];
> AWGG>  ??? ...
> AWGG>  ??? CONTENT_BLACKLISTED1(0.00)[];
> AWGG>  ??? ...
>
> AWGG> ...but why is score 0.00 and not 10 like it is defined in local_content.map?
>
>
> AWGG> best regards, Andy
>
> AWGG> Am 21.09.2021 um 11:53 schrieb Andreas Wass - Glas Gasperlmair:
>>> Hi,
>>>
>>> i'm trying to use rspamd as soon as possible for our company > mailserver, and testing content filtering with multimap as shown in
>>>
>>> https://rspamd.com/doc/modules/multimap.html
>>>
>>> *Configured the following.*..
>>>
>>>
>>>  ?? Multiple symbol maps
>>>
>>>  From the version 1.3.1, it is possible to define multiple symbols and > scores using multimap module. To do that, you should define all > possible symbols using|symbols|option in multimap:
>>>
>>> |# local.d/multimap.conf CONTENT_BLACKLISTED { type = "content"; > filter = "body"; # can be headers, full, oneline, text, rawtext map = > "${LOCAL_CONFDIR}/local.d/local_content.map"; symbols = > ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"]; regexp = true; } |
>>>
>>> In this example, you can use 3 symbols:
>>>
>>>  ?* CONTENT_BLACKLISTED
>>>  ?* CONTENT_BLACKLISTED1
>>>  ?* CONTENT_BLACKLISTED2
>>>
>>> the map:
>>>
>>> |# Symbol + score /re1/ CONTENT_BLACKLISTED1:10 # Symbol with default > score /re2/ CONTENT_BLACKLISTED2 # Just a default symbol: > CONTENT_BLACKLISTED /re3/ |
>>>
>>> Symbols that are not defined in the|symbols|attribute but used in the > map are ignored and replaced by the default map symbol. If the value > of a key-value pair is missing, then Rspamd just inserts the default > symbol with dynamic weight equal to|1.0|(which is multiplied by metric > score afterwards)
>>>
>>>
>>> *...matches the right symbol (CONTENT_BLACKLISTED1) in the header :*
>>>
>>> X-Spamd-Result: default: False [0.80 / 15.00];
>>>  ????...
>>>  ????CONTENT_BLACKLISTED1(0.00)[];
>>>  ????...
>>>
>>> *...but why is score 0.00 and not 10 like it is defined in > local_content.map?*
>>>
>>>
>>> best regards, Andy
>>>
>
>
>


From philip.colmer at linaro.org  Tue Sep 21 12:55:15 2021
From: philip.colmer at linaro.org (Philip Colmer)
Date: Tue, 21 Sep 2021 13:55:15 +0100
Subject: [Rspamd-Users] When should ARC headers be added?
Message-ID: <CAKTSSTiVeSw9mAWZdZTHcQPF7ELfT-8n8Y6iNCdh2p29NjbuSQ@mail.gmail.com>

I'm trying to understand when an ARC set (the three ARC headers)
should be added - when an email is received by a MTA, when a MTA is
sending an email out or both? I've read RFC8617 and it doesn't seem to
be entirely clear but the behaviour I'm seeing from other systems
seems to suggest that it can be both.

If that is the case, I'm struggling with configuring the ARC module
appropriately and would welcome some insight/assistance.

My test scenario is sending an email from domain A to domain B, which
is running Mailman 3. Mailman 3 adds a signature to the email and
changes the subject line before sending the email back to domain A.

When domain A sends the email to domain B, it adds a DKIM header and
an ARC header.

When the email reaches domain B, Rspamd's ARC module tries to create
an ARC set for domain A, which it clearly can't do. After Mailman 3
has done its thing, it sends the email to Postfix which then sends it
to Rspamd.

It looks like the ARC module gets called before the dkim_signing
module, which causes a problem because the last (only?) DKIM header
that the ARC module can see is one that was created by domain A, which
no longer matches because Mailman 3 has altered the email.

A DKIM header does get added by Rspamd but there isn't an ARC Set
getting added by Rspamd. I'm not sure if this is because I've
misconfigured the ARC module or because of the rspamd_dkim_check
errors I'm getting:

dkim; rspamd_dkim_check: arc_sig: bh value mismatch: got
T3JEKsVRCAWS09CDUv/Nc9LZJQ387FwCcC1OB2ceZ5g=, expected
57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075;
d=zohomail.com; s=zohoarc

Those seem to come from the dkim module but I'm not sure why it would
be checking against the DKIM header generated by domain A rather than
just generating a DKIM header itself.

I've included the full log output for when Postfix calls Rspamd on
message receipt and message sending below, just in case I'm
overlooking something.

Thanks in advance for any advice/insight offered.

Regards

Philip

dkim_signing config:
sign_networks [
    "127.2.4.7",
]
try_fallback = false;
allow_envfrom_empty = true;
allow_hdrfrom_multiple = false;
use_esld = false;
sign_local = true;
key_prefix = "DKIM_KEYS";
sign_authenticated = false;
selector = "dkim";
use_redis = false;
use_domain = "envelope";
allow_hdrfrom_mismatch = true;
allow_hdrfrom_mismatch_sign_networks = true;
allow_username_mismatch = true;
domain {
    mm3.lavasoftware.org {
        selector = "dkim";
        path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key";
    }
}
symbol = "DKIM_SIGNED";

arc config:
selector = "arc";
domain {
    mm3.lavasoftware.org {
        selector = "dkim";
        path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key";
    }
}
use_domain = "header";
allow_hdrfrom_mismatch = true;
allow_envfrom_empty = true;
allow_hdrfrom_multiple = false;
sign_inbound = true;
sign_local = false;
sign_authenticated = false;
use_esld = false;
try_fallback = true;
use_redis = false;
key_prefix = "ARC_KEYS";
sign_networks [
    "127.2.4.7",
]
symbol_sign = "ARC_SIGNED";
allow_username_mismatch = true;

When Postfix first receives the email from domain A:

2021-09-21 12:50:21 #563873(rspamd_proxy) <2b2ef0>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
44648
2021-09-21 12:50:22 #563873(rspamd_proxy) <2b2ef0>; milter;
rspamd_milter_process_command: got connection from
136.143.188.14:17412
2021-09-21 12:50:22 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_message_parse: loaded message; id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>;
queue-id: <A4315BEA9B>; size: 2391; checksum:
<494b8362da04adff9b1bd4b5bd01466d>
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
dkim_module_key_handler: stored DKIM key for
zoho._domainkey.codelinaro.org in LRU cache for 300 seconds, 1/2000
elements in the cache
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; dkim_signing;
lua_dkim_tools.lua:170: mail is ineligible for signing
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_spf_maybe_return: stored record for codelinaro.org
(0x4d20af7734be3ae1) in LRU cache for 300 seconds, 1/2000 elements in
the cache
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:205:
got 1 arc sections
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:342:
processed arc signature zohomail.com[1]: true(nil), 0 processed
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:260:
checked arc signature zohomail.com: true(nil), 0 processed
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:226:
checked arc seal: true(nil), 1 processed
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:168: mail was sent to us
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:382: use domain(header) for signature:
codelinaro.org
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:402: final DKIM domain: codelinaro.org
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:46: add key
"/var/lib/rspamd/arc/$domain.$selector.key" using default path
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:51: set selector to "arc" using default selector
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:51: set domain to "codelinaro.org" using
dkim_domain
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:682:
cannot read key from /var/lib/rspamd/arc/codelinaro.org.arc.key: No
such file or directory
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua;
neural.lua:315: skip ham sample to keep spam/ham balance; probability
1; 0 spam and 1 ham vectors stored
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua;
neural.lua:69: created new ANN profile for default:default, data
stored at prefix rn_default_default_cjf5d845_0
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_task_write_log: id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid:
<A4315BEA9B>, ip: 136.143.188.14, from:
<philip.colmer at codelinaro.org>, (default: F (no action): [-1.49/15.00]
[ARC_ALLOW(-1.00){zohomail.com:s=zohoarc:i=1;},R_DKIM_ALLOW(-0.20){codelinaro.org:s=zoho;},R_SPF_ALLOW(-0.20){+ip4:136.143.188.0/24;},MIME_GOOD(-0.10){multipart/alternative;text/plain;},XM_UA_NO_VERSION(0.01){},ASN(0.00){asn:2639,
ipnet:136.143.188.0/23,
country:US;},DKIM_TRACE(0.00){codelinaro.org:+;},DMARC_NA(0.00){codelinaro.org;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:~;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_TWO(0.00){2;},RCVD_IN_DNSWL_NONE(0.00){136.143.188.14:from;},RCVD_TLS_LAST(0.00){},RWL_MAILSPIKE_POSSIBLE(0.00){136.143.188.14:from;},TO_DN_ALL(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]),
len: 2391, time: 672.016ms, dns req: 27, digest:
<494b8362da04adff9b1bd4b5bd01466d>, rcpts:
<test at mm3.lavasoftware.org>, mime_rcpts: <test at mm3.lavasoftware.org>
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned
using pcre, 1014B scanned total
2021-09-21 12:50:23 #563873(rspamd_proxy) <ff9edd>; proxy;
proxy_milter_finish_handler: finished milter connection

When Mailman 3 sends the email to Postfix for sending out:

2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
44746
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; milter;
rspamd_milter_process_command: got connection from 127.0.0.1:34878
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_message_parse: loaded message; id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>;
queue-id: <51B0ABEA9C>; size: 3909; checksum:
<1b29c03e5f5475bcabec1e787f28d4ef>
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_mime_part_detect_language: detected part language: en
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua;
greylist.lua:204: skip greylisting for local networks and/or
authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
dkim_symbol_callback: skip DKIM checks for local networks and
authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua; spf.lua:186:
skip SPF checks for local networks and authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; arc; arc.lua:205:
got 1 arc sections
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; arc; arc.lua:342:
processed arc signature zohomail.com[1]: true(nil), 0 processed
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua;
dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not
checked
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
lua_dkim_tools.lua:166: mail is from local address
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
lua_dkim_tools.lua:382: use domain(envelope) for signature:
mm3.lavasoftware.org
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
lua_dkim_tools.lua:402: final DKIM domain: mm3.lavasoftware.org
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
lua_dkim_tools.lua:51: set domain to "mm3.lavasoftware.org" using
dkim_domain
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
dkim_signing.lua:129: using key
"/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key", use selector
"dkim" for domain "mm3.lavasoftware.org"
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua;
once_received.lua:99: Skipping once_received for authenticated user or
local network
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim;
rspamd_dkim_check: arc_sig: bh value mismatch: got
i2Ji19OooPA0mAVbxb/Wh0tqFcGTSvsGoLDHuoc4EIs=, expected
57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075;
d=zohomail.com; s=zohoarc
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; arc; arc.lua:260:
checked arc signature zohomail.com: false(reject), 0 processed
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; arc;
lua_dkim_tools.lua:170: mail is ineligible for signing
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_task_write_log: id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid:
<51B0ABEA9C>, ip: 127.0.0.1, from:
<test-bounces at mm3.lavasoftware.org>, (default: F (no action):
[0.80/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] =
sig:zohomail.com:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){philip.colmer at codelinaro.org;test-bounces at mm3.lavasoftware.org;},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},TAGGED_FROM(0.00){philip.colmer=linaro.org;},TO_DN_ALL(0.00){}]),
len: 3909, time: 243.345ms, dns req: 11, digest:
<1b29c03e5f5475bcabec1e787f28d4ef>, rcpts: <philip.colmer at linaro.org>,
mime_rcpts: <test at mm3.lavasoftware.org>
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned
using pcre, 1.90KiB scanned total
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_message_parse: loaded message; id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>;
queue-id: <9D95CBEA9F>; size: 3909; checksum:
<1b29c03e5f5475bcabec1e787f28d4ef>
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_mime_part_detect_language: detected part language: en
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua;
greylist.lua:204: skip greylisting for local networks and/or
authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
dkim_symbol_callback: skip DKIM checks for local networks and
authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua; spf.lua:186:
skip SPF checks for local networks and authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:205:
got 1 arc sections
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim;
rspamd_dkim_check: arc_sig: bh value mismatch: got
T3JEKsVRCAWS09CDUv/Nc9LZJQ387FwCcC1OB2ceZ5g=, expected
57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075;
d=zohomail.com; s=zohoarc
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:260:
checked arc signature zohomail.com: false(reject), 0 processed
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:342:
processed arc signature zohomail.com[1]: true(nil), 0 processed
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua;
dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not
checked
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc;
lua_dkim_tools.lua:170: mail is ineligible for signing
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
lua_dkim_tools.lua:166: mail is from local address
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
lua_dkim_tools.lua:382: use domain(envelope) for signature:
mm3.lavasoftware.org
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
lua_dkim_tools.lua:402: final DKIM domain: mm3.lavasoftware.org
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
lua_dkim_tools.lua:51: set domain to "mm3.lavasoftware.org" using
dkim_domain
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
dkim_signing.lua:129: using key
"/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key", use selector
"dkim" for domain "mm3.lavasoftware.org"
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua;
once_received.lua:99: Skipping once_received for authenticated user or
local network
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_task_write_log: id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid:
<9D95CBEA9F>, ip: 127.0.0.1, from:
<test-bounces at mm3.lavasoftware.org>, (default: F (no action):
[0.79/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] =
sig:zohomail.com:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){philip.colmer at codelinaro.org;test-bounces at mm3.lavasoftware.org;},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},TAGGED_FROM(0.00){philip.colmer=codelinaro.org;},TO_DN_ALL(0.00){}]),
len: 3909, time: 9.960ms, dns req: 10, digest:
<1b29c03e5f5475bcabec1e787f28d4ef>, rcpts:
<philip.colmer at codelinaro.org>, mime_rcpts:
<test at mm3.lavasoftware.org>
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned
using pcre, 1.90KiB scanned total
2021-09-21 12:50:26 #563874(rspamd_proxy) <6370ae>; proxy;
proxy_milter_finish_handler: finished milter connection

From kyle at cci1986.com  Tue Sep 21 13:18:58 2021
From: kyle at cci1986.com (Kyle A.)
Date: Tue, 21 Sep 2021 09:18:58 -0400
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>
References: <1907311023.20210823163332@latnet.lv>
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
 <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
 <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>
Message-ID: <b198d7c4-bd83-666b-9b5d-716749aab757@cci1986.com>

Thanks for the suggestion, Michael.  I ran it again today and did not 
get the same error.  I got this instead:

# rspamadm dmarc_report
No reports for 20210921
No reports for 20210920
Segmentation fault (core dumped)
#

I'll try again tomorrow and see if it seg faults with fresh V3.0 data 
only.  This is a fully patched up, dedicated CentOS 8 system.  I put the 
log messages down at the bottom for anyone who wants to scroll all the 
way down there to see them.

-Kyle

On 9/20/2021 4:39 PM, Michael Kliewe wrote:
> I also got this error when I first manually ran
> sudo rspamadm dmarc_report
> after upgrading from 2.7 to 3.0.2 on Ubuntu.
> 
> But:
> One day later I ran it again manually:
> sudo rspamadm dmarc_report -v
> (with verbose mode on) and it was working fine, it sent 10 DMARC reports.
> 
> Maybe there was some old/invalid data in the redis storage, which is why 
> the first invocation failed with "invalid argument: nil"?
> 
> Michael




Sep 21 09:08:56 mailserver kernel: rspamadm[8258]: segfault at 30 ip 
00007fda52d1868c sp 00007ffdd54906b0 error 6 in librspamd-server.s 
 
                 o[7fda522a3000+c70000]
Sep 21 09:08:56 mailserver kernel: Code: 49 89 40 38 4d 89 ca 4c 89 c1 
e9 73 ff ff ff 0f 1f 80 00 00 00 00 4d 89 58 28 4d 89 43 30 e9 7 
 
               6 fc ff ff 0f 1f 00 48 8b 4e 30 <49> 89 4b 30 4c 8b 66 30 
4d 85 e4 0f 85 6e fe ff ff e9 6a fc ff ff
Sep 21 09:08:56 mailserver systemd[1]: Created slice 
system-systemd\x2dcoredump.slice.
Sep 21 09:08:56 mailserver systemd[1]: Started Process Core Dump (PID 
8259/UID 0).
Sep 21 09:08:58 mailserver systemd-coredump[8260]: Process 8258 
(rspamadm) of user 0 dumped core.#012#012Stack trace of thread 8258:#01 
 
                      2#0  0x00007fda52d1868c je_extent_heap_remove 
(librspamd-server.so)#012#1  0x00007fda52ce80e3 
je_arena_dalloc_bin_junked_locked (l 
 
ibrspamd-server.so)#012#2  0x00007fda52d3e862 je_tcache_bin_flush_small 
(librspamd-server.so)#012#3  0x00007fda52cdf81a je_free_de 
 
         fault (librspamd-server.so)#012#4  0x00007fda523ef532 
ucl_hash_destroy (librspamd-server.so)#012#5  0x00007fda523fbbe4 
ucl_object_ 
                                  dtor_unref (librspamd-server.so)#012#6 
  0x00007fda523ef532 ucl_hash_destroy (librspamd-server.so)#012#7 
0x00007fda523fbbe4 ucl_ob 
                                                ject_dtor_unref 
(librspamd-server.so)#012#8  0x00007fda523ef532 ucl_hash_destroy 
(librspamd-server.so)#012#9  0x00007fda523fbcfe u 
 
cl_object_unref (librspamd-server.so)#012#10 0x00007fda52630e7e 
rspamd_config_free (librspamd-server.so)#012#11 0x000000000040c781 
 
                  main (rspamadm)#012#12 0x00007fda4cbc0493 
__libc_start_main (libc.so.6)#012#13 0x000000000040cb2e _start (rspamadm)
Sep 21 09:08:59 mailserver systemd[1]: 
systemd-coredump at 0-8259-0.service: Succeeded.

From m.kliewe at team.mail.de  Tue Sep 21 15:15:41 2021
From: m.kliewe at team.mail.de (Michael Kliewe)
Date: Tue, 21 Sep 2021 17:15:41 +0200
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <b198d7c4-bd83-666b-9b5d-716749aab757@cci1986.com>
References: <1907311023.20210823163332@latnet.lv>
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
 <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
 <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>
 <b198d7c4-bd83-666b-9b5d-716749aab757@cci1986.com>
Message-ID: <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de>

Hi Kyle,

try the verbose mode ("rspamadm dmarc_report -v"), you should get more 
information then. Maybe it helps debugging the problem, and maybe you 
see where it seg faults.

Michael


Am 21.09.2021 um 15:18 schrieb Kyle A.:
> Thanks for the suggestion, Michael.? I ran it again today and did not 
> get the same error.? I got this instead:
>
> # rspamadm dmarc_report
> No reports for 20210921
> No reports for 20210920
> Segmentation fault (core dumped)
> #
>
> I'll try again tomorrow and see if it seg faults with fresh V3.0 data 
> only.? This is a fully patched up, dedicated CentOS 8 system.? I put 
> the log messages down at the bottom for anyone who wants to scroll all 
> the way down there to see them.
>
> -Kyle
>
> On 9/20/2021 4:39 PM, Michael Kliewe wrote:
>> I also got this error when I first manually ran
>> sudo rspamadm dmarc_report
>> after upgrading from 2.7 to 3.0.2 on Ubuntu.
>>
>> But:
>> One day later I ran it again manually:
>> sudo rspamadm dmarc_report -v
>> (with verbose mode on) and it was working fine, it sent 10 DMARC 
>> reports.
>>
>> Maybe there was some old/invalid data in the redis storage, which is 
>> why the first invocation failed with "invalid argument: nil"?
>>
>> Michael
>
>
>
>
> Sep 21 09:08:56 mailserver kernel: rspamadm[8258]: segfault at 30 ip 
> 00007fda52d1868c sp 00007ffdd54906b0 error 6 in librspamd-server.s
> ??????????????? o[7fda522a3000+c70000]
> Sep 21 09:08:56 mailserver kernel: Code: 49 89 40 38 4d 89 ca 4c 89 c1 
> e9 73 ff ff ff 0f 1f 80 00 00 00 00 4d 89 58 28 4d 89 43 30 e9 7
> ????????????? 6 fc ff ff 0f 1f 00 48 8b 4e 30 <49> 89 4b 30 4c 8b 66 
> 30 4d 85 e4 0f 85 6e fe ff ff e9 6a fc ff ff
> Sep 21 09:08:56 mailserver systemd[1]: Created slice 
> system-systemd\x2dcoredump.slice.
> Sep 21 09:08:56 mailserver systemd[1]: Started Process Core Dump (PID 
> 8259/UID 0).
> Sep 21 09:08:58 mailserver systemd-coredump[8260]: Process 8258 
> (rspamadm) of user 0 dumped core.#012#012Stack trace of thread 8258:#01
> ???????????????????? 2#0? 0x00007fda52d1868c je_extent_heap_remove 
> (librspamd-server.so)#012#1? 0x00007fda52ce80e3 
> je_arena_dalloc_bin_junked_locked (l
> ibrspamd-server.so)#012#2? 0x00007fda52d3e862 
> je_tcache_bin_flush_small (librspamd-server.so)#012#3 
> 0x00007fda52cdf81a je_free_de
> ??????? fault (librspamd-server.so)#012#4? 0x00007fda523ef532 
> ucl_hash_destroy (librspamd-server.so)#012#5? 0x00007fda523fbbe4 
> ucl_object_ ???????????????????????????????? dtor_unref 
> (librspamd-server.so)#012#6 ?0x00007fda523ef532 ucl_hash_destroy 
> (librspamd-server.so)#012#7 0x00007fda523fbbe4 ucl_ob 
> ?????????????????????????????????????????????? ject_dtor_unref 
> (librspamd-server.so)#012#8? 0x00007fda523ef532 ucl_hash_destroy 
> (librspamd-server.so)#012#9? 0x00007fda523fbcfe u
> cl_object_unref (librspamd-server.so)#012#10 0x00007fda52630e7e 
> rspamd_config_free (librspamd-server.so)#012#11 0x000000000040c781
> ???????????????? main (rspamadm)#012#12 0x00007fda4cbc0493 
> __libc_start_main (libc.so.6)#012#13 0x000000000040cb2e _start (rspamadm)
> Sep 21 09:08:59 mailserver systemd[1]: 
> systemd-coredump at 0-8259-0.service: Succeeded.


From kyle at cci1986.com  Tue Sep 21 19:05:31 2021
From: kyle at cci1986.com (Kyle A.)
Date: Tue, 21 Sep 2021 15:05:31 -0400
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de>
References: <1907311023.20210823163332@latnet.lv>
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
 <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
 <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>
 <b198d7c4-bd83-666b-9b5d-716749aab757@cci1986.com>
 <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de>
Message-ID: <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com>

Michael,

Thanks for the suggestion.  This is interesting:

[mailserver:~ root]# rspamadm dmarc_report -v
previous last report date is 1632160659
Process date 20210921
No reports for 20210921
Process date 20210920
No reports for 20210920
send data for 0 domains (from 1 to 0)
[mailserver:~ root]# rspamadm dmarc_report
No reports for 20210921
No reports for 20210920
[mailserver:~ root]#

The GOOD news is it did not seg fault.  The bad news it is not showing 
any data to report.  This server has handled about 1.2k messages so far 
today, so there should be data to report.

Thanks,
Kyle

On 9/21/2021 11:15 AM, Michael Kliewe wrote:
> Hi Kyle,
> 
> try the verbose mode ("rspamadm dmarc_report -v"), you should get more 
> information then. Maybe it helps debugging the problem, and maybe you 
> see where it seg faults.
> 
> Michael

From emawata at gmail.com  Wed Sep 22 17:38:40 2021
From: emawata at gmail.com (SysAdmin EM)
Date: Wed, 22 Sep 2021 14:38:40 -0300
Subject: [Rspamd-Users] Neural Network manually training
Message-ID: <CAGUDtnn0VODXKX3OGiP5vy6MvTTbac02-Mg065WDU-vg_-QRiw@mail.gmail.com>

Hello, I am trying to manually train the neural network since I see that
many emails that are SPAM are being cataloged as NEURAL_HAM.

This is my config:

servers = "127.0.0.1:6379";

train {
  max_trains = 1k; # Number ham/spam samples needed to start train
  max_usages = 20; # Number of learn iterations while ANN data is valid
  learning_rate = 0.01; # Rate of learning
  max_iterations = 25; # Maximum iterations of learning (better preciseness
but also lower speed of learning)
}

ann_expire = 2d; # For how long ANN should be preserved in Redis

The documentation (https://rspamd.com/doc/modules/neural.html) indicates
that the files should be placed in /plugins/neural/learn. What would be the
exact path? in /etc/rspamd/local.d? or other path?

once the neural network has been trained, can automatic training be
deactivated? or do you always have to train?

Regards,

From kyle at cci1986.com  Wed Sep 22 17:51:41 2021
From: kyle at cci1986.com (Kyle A.)
Date: Wed, 22 Sep 2021 13:51:41 -0400
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com>
References: <1907311023.20210823163332@latnet.lv>
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
 <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
 <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>
 <b198d7c4-bd83-666b-9b5d-716749aab757@cci1986.com>
 <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de>
 <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com>
Message-ID: <e46d5c42-abea-622c-f19c-d7d0845dae3c@cci1986.com>

No seg fault anymore, but this module seems broken:

[mailserver:~ root]# rspamadm dmarc_report
No reports for 20210922
No reports for 20210921
[mailserver:~ root]#

At least 1k more messages have been handled since my last post.  DMARC 
reporting worked for us in all the 2.x versions.

Thanks,
Kyle

On 9/21/2021 3:05 PM, Kyle A. wrote:
> Michael,
> 
> Thanks for the suggestion.? This is interesting:
> 
> [mailserver:~ root]# rspamadm dmarc_report -v
> previous last report date is 1632160659
> Process date 20210921
> No reports for 20210921
> Process date 20210920
> No reports for 20210920
> send data for 0 domains (from 1 to 0)
> [mailserver:~ root]# rspamadm dmarc_report
> No reports for 20210921
> No reports for 20210920
> [mailserver:~ root]#
> 
> The GOOD news is it did not seg fault.? The bad news it is not showing 
> any data to report.? This server has handled about 1.2k messages so far 
> today, so there should be data to report.
> 
> Thanks,
> Kyle

From rspamd at vlh.dk  Wed Sep 22 18:32:03 2021
From: rspamd at vlh.dk (rspamd at vlh.dk)
Date: Wed, 22 Sep 2021 20:32:03 +0200
Subject: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
 config
In-Reply-To: <e46d5c42-abea-622c-f19c-d7d0845dae3c@cci1986.com>
References: <1907311023.20210823163332@latnet.lv>
 <22069df3-6384-db91-1536-5070e14fdd08@fmisle.com>
 <5710476131.20210913170548@sveiks.lv>
 <df3ffd77-7ed5-4a49-b089-e5730a10ce94@cci1986.com>
 <a64dcfa7-a4da-ad02-6ea4-1caa472fd98f@team.mail.de>
 <b198d7c4-bd83-666b-9b5d-716749aab757@cci1986.com>
 <5666e187-0f03-c0bf-8995-9da309602b2d@team.mail.de>
 <14cd03f5-2db6-0098-0ada-71edcec38c45@cci1986.com>
 <e46d5c42-abea-622c-f19c-d7d0845dae3c@cci1986.com>
Message-ID: <003d01d7afe0$27cb02b0$77610810$@vlh.dk>

Sounds strange, works for me with a very basic local.d/dmarc.conf

actions = {
        quarantine = "add_header";
        reject = "reject";
}
reporting {
  enabled = true
  org_name = "vlh.dk";
  domain = "vlh.dk";
  email = "dmarc at vlh.dk";
  smtp = "mail.vlh.dk";
  smtp_port = 25;
  helo = "mail.vlh.dk";
  # Number of retries on temporary errors (2 if unset)
  # retries = 2;
}

mail ~ # rspamadm dmarc_report
No reports for 20210921
Reporting collection has finished 1 dates processed, 1 reports: 1 completed, 0 failed

Very low volume private mailserver ?


Kind regards,
Kim Sindalsen

> -----Original Message-----
> From: Users <users-bounces at lists.rspamd.com> On Behalf Of Kyle A.
> Sent: 22. september 2021 19:52
> To: users at lists.rspamd.com
> Subject: Re: [Rspamd-Users] v3.0 rspamd DMARC report code doesnt't load
> config
> 
> No seg fault anymore, but this module seems broken:
> 
> [mailserver:~ root]# rspamadm dmarc_report No reports for 20210922 No
> reports for 20210921 [mailserver:~ root]#
> 
> At least 1k more messages have been handled since my last post.  DMARC
> reporting worked for us in all the 2.x versions.
> 
> Thanks,
> Kyle


From Ralf.Hildebrandt at charite.de  Thu Sep 23 07:52:50 2021
From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt)
Date: Thu, 23 Sep 2021 09:52:50 +0200
Subject: [Rspamd-Users] [ext]  Neural Network manually training
In-Reply-To: <CAGUDtnn0VODXKX3OGiP5vy6MvTTbac02-Mg065WDU-vg_-QRiw@mail.gmail.com>
References: <CAGUDtnn0VODXKX3OGiP5vy6MvTTbac02-Mg065WDU-vg_-QRiw@mail.gmail.com>
Message-ID: <YUwyUr6dwagPdNbX@charite.de>

* SysAdmin EM <emawata at gmail.com>:

> The documentation (https://rspamd.com/doc/modules/neural.html) indicates
> that the files should be placed in /plugins/neural/learn.

No, that's not what it says!

"The controller endpoint /plugins/neural/learn facilitates manual
training of neural networks & accepts a JSON POST"

So basically it's a web service (on the rspamd Server) that accepts a
POST request.

I must admit the whole paragraph lacks an example (especially for
"spam_vec and ham_vec" and "rule")

Ralf Hildebrandt
Charit? - Universit?tsmedizin Berlin
Gesch?ftsbereich IT | Abteilung Netzwerk

Campus Benjamin Franklin (CBF)
Haus I | 1. OG | Raum 105
Hindenburgdamm 30 | D-12203 Berlin

Tel. +49 30 450 570 155
ralf.hildebrandt at charite.de
https://www.charite.de

From emawata at gmail.com  Thu Sep 23 10:42:42 2021
From: emawata at gmail.com (SysAdmin EM)
Date: Thu, 23 Sep 2021 07:42:42 -0300
Subject: [Rspamd-Users] [ext] Neural Network manually training
In-Reply-To: <YUwyUr6dwagPdNbX@charite.de>
References: <CAGUDtnn0VODXKX3OGiP5vy6MvTTbac02-Mg065WDU-vg_-QRiw@mail.gmail.com>
 <YUwyUr6dwagPdNbX@charite.de>
Message-ID: <CAGUDtn=xQX9M7KRzjvGuNaVhzBBC=jRRYVDDRNM9hGUDqoksTw@mail.gmail.com>

Thanks for the reply.

Would you tell me how I do the manual training? I can't understand how to
do it.


Regards,

On Thu, Sep 23, 2021 at 4:54 AM Ralf Hildebrandt <
Ralf.Hildebrandt at charite.de> wrote:

> * SysAdmin EM <emawata at gmail.com>:
>
> > The documentation (https://rspamd.com/doc/modules/neural.html) indicates
> > that the files should be placed in /plugins/neural/learn.
>
> No, that's not what it says!
>
> "The controller endpoint /plugins/neural/learn facilitates manual
> training of neural networks & accepts a JSON POST"
>
> So basically it's a web service (on the rspamd Server) that accepts a
> POST request.
>
> I must admit the whole paragraph lacks an example (especially for
> "spam_vec and ham_vec" and "rule")
>
> Ralf Hildebrandt
> Charit? - Universit?tsmedizin Berlin
> Gesch?ftsbereich IT | Abteilung Netzwerk
>
> Campus Benjamin Franklin (CBF)
> Haus I | 1. OG | Raum 105
> Hindenburgdamm 30 | D-12203 Berlin
>
> Tel. +49 30 450 570 155
> ralf.hildebrandt at charite.de
> https://www.charite.de
> --
> Users mailing list
> Users at lists.rspamd.com
> https://lists.rspamd.com/mailman/listinfo/users
>

From Ralf.Hildebrandt at charite.de  Thu Sep 23 10:51:25 2021
From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt)
Date: Thu, 23 Sep 2021 12:51:25 +0200
Subject: [Rspamd-Users] [ext] Neural Network manually training
In-Reply-To: <CAGUDtn=xQX9M7KRzjvGuNaVhzBBC=jRRYVDDRNM9hGUDqoksTw@mail.gmail.com>
References: <CAGUDtnn0VODXKX3OGiP5vy6MvTTbac02-Mg065WDU-vg_-QRiw@mail.gmail.com>
 <YUwyUr6dwagPdNbX@charite.de>
 <CAGUDtn=xQX9M7KRzjvGuNaVhzBBC=jRRYVDDRNM9hGUDqoksTw@mail.gmail.com>
Message-ID: <YUxcLWL6bjtvAp/r@charite.de>

* SysAdmin EM <emawata at gmail.com>:
> Thanks for the reply.
> 
> Would you tell me how I do the manual training? I can't understand how to
> do it.

I can't since I also don't understand what teh webservice actually
expects :)

Ralf Hildebrandt
Charit? - Universit?tsmedizin Berlin
Gesch?ftsbereich IT | Abteilung Netzwerk

Campus Benjamin Franklin (CBF)
Haus I | 1. OG | Raum 105
Hindenburgdamm 30 | D-12203 Berlin

Tel. +49 30 450 570 155
ralf.hildebrandt at charite.de
https://www.charite.de