[Rspamd-Users] When should ARC headers be added?

Tue Sep 21 12:55:15 UTC 2021

I'm trying to understand when an ARC set (the three ARC headers)
should be added - when an email is received by a MTA, when a MTA is
sending an email out or both? I've read RFC8617 and it doesn't seem to
be entirely clear but the behaviour I'm seeing from other systems
seems to suggest that it can be both.

If that is the case, I'm struggling with configuring the ARC module
appropriately and would welcome some insight/assistance.

My test scenario is sending an email from domain A to domain B, which
is running Mailman 3. Mailman 3 adds a signature to the email and
changes the subject line before sending the email back to domain A.

When domain A sends the email to domain B, it adds a DKIM header and
an ARC header.

When the email reaches domain B, Rspamd's ARC module tries to create
an ARC set for domain A, which it clearly can't do. After Mailman 3
has done its thing, it sends the email to Postfix which then sends it
to Rspamd.

It looks like the ARC module gets called before the dkim_signing
module, which causes a problem because the last (only?) DKIM header
that the ARC module can see is one that was created by domain A, which
no longer matches because Mailman 3 has altered the email.

A DKIM header does get added by Rspamd but there isn't an ARC Set
getting added by Rspamd. I'm not sure if this is because I've
misconfigured the ARC module or because of the rspamd_dkim_check
errors I'm getting:

dkim; rspamd_dkim_check: arc_sig: bh value mismatch: got
T3JEKsVRCAWS09CDUv/Nc9LZJQ387FwCcC1OB2ceZ5g=, expected
57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075;
d=zohomail.com; s=zohoarc

Those seem to come from the dkim module but I'm not sure why it would
be checking against the DKIM header generated by domain A rather than
just generating a DKIM header itself.

I've included the full log output for when Postfix calls Rspamd on
message receipt and message sending below, just in case I'm
overlooking something.

Thanks in advance for any advice/insight offered.

Regards

Philip

dkim_signing config:
sign_networks [
    "127.2.4.7",
]
try_fallback = false;
allow_envfrom_empty = true;
allow_hdrfrom_multiple = false;
use_esld = false;
sign_local = true;
key_prefix = "DKIM_KEYS";
sign_authenticated = false;
selector = "dkim";
use_redis = false;
use_domain = "envelope";
allow_hdrfrom_mismatch = true;
allow_hdrfrom_mismatch_sign_networks = true;
allow_username_mismatch = true;
domain {
    mm3.lavasoftware.org {
        selector = "dkim";
        path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key";
    }
}
symbol = "DKIM_SIGNED";

arc config:
selector = "arc";
domain {
    mm3.lavasoftware.org {
        selector = "dkim";
        path = "/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key";
    }
}
use_domain = "header";
allow_hdrfrom_mismatch = true;
allow_envfrom_empty = true;
allow_hdrfrom_multiple = false;
sign_inbound = true;
sign_local = false;
sign_authenticated = false;
use_esld = false;
try_fallback = true;
use_redis = false;
key_prefix = "ARC_KEYS";
sign_networks [
    "127.2.4.7",
]
symbol_sign = "ARC_SIGNED";
allow_username_mismatch = true;

When Postfix first receives the email from domain A:

2021-09-21 12:50:21 #563873(rspamd_proxy) <2b2ef0>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
44648
2021-09-21 12:50:22 #563873(rspamd_proxy) <2b2ef0>; milter;
rspamd_milter_process_command: got connection from
136.143.188.14:17412
2021-09-21 12:50:22 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_message_parse: loaded message; id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>;
queue-id: <A4315BEA9B>; size: 2391; checksum:
<494b8362da04adff9b1bd4b5bd01466d>
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
dkim_module_key_handler: stored DKIM key for
zoho._domainkey.codelinaro.org in LRU cache for 300 seconds, 1/2000
elements in the cache
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; dkim_signing;
lua_dkim_tools.lua:170: mail is ineligible for signing
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_spf_maybe_return: stored record for codelinaro.org
(0x4d20af7734be3ae1) in LRU cache for 300 seconds, 1/2000 elements in
the cache
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:205:
got 1 arc sections
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:342:
processed arc signature zohomail.com[1]: true(nil), 0 processed
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:260:
checked arc signature zohomail.com: true(nil), 0 processed
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:226:
checked arc seal: true(nil), 1 processed
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:168: mail was sent to us
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:382: use domain(header) for signature:
codelinaro.org
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:402: final DKIM domain: codelinaro.org
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:46: add key
"/var/lib/rspamd/arc/$domain.$selector.key" using default path
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:51: set selector to "arc" using default selector
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc;
lua_dkim_tools.lua:51: set domain to "codelinaro.org" using
dkim_domain
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; arc; arc.lua:682:
cannot read key from /var/lib/rspamd/arc/codelinaro.org.arc.key: No
such file or directory
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua;
neural.lua:315: skip ham sample to keep spam/ham balance; probability
1; 0 spam and 1 ham vectors stored
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; lua;
neural.lua:69: created new ANN profile for default:default, data
stored at prefix rn_default_default_cjf5d845_0
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_task_write_log: id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid:
<A4315BEA9B>, ip: 136.143.188.14, from:
<philip.colmer at codelinaro.org>, (default: F (no action): [-1.49/15.00]
[ARC_ALLOW(-1.00){zohomail.com:s=zohoarc:i=1;},R_DKIM_ALLOW(-0.20){codelinaro.org:s=zoho;},R_SPF_ALLOW(-0.20){+ip4:136.143.188.0/24;},MIME_GOOD(-0.10){multipart/alternative;text/plain;},XM_UA_NO_VERSION(0.01){},ASN(0.00){asn:2639,
ipnet:136.143.188.0/23,
country:US;},DKIM_TRACE(0.00){codelinaro.org:+;},DMARC_NA(0.00){codelinaro.org;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:~;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_TWO(0.00){2;},RCVD_IN_DNSWL_NONE(0.00){136.143.188.14:from;},RCVD_TLS_LAST(0.00){},RWL_MAILSPIKE_POSSIBLE(0.00){136.143.188.14:from;},TO_DN_ALL(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]),
len: 2391, time: 672.016ms, dns req: 27, digest:
<494b8362da04adff9b1bd4b5bd01466d>, rcpts:
<test at mm3.lavasoftware.org>, mime_rcpts: <test at mm3.lavasoftware.org>
2021-09-21 12:50:23 #563873(rspamd_proxy) <2b2ef0>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned
using pcre, 1014B scanned total
2021-09-21 12:50:23 #563873(rspamd_proxy) <ff9edd>; proxy;
proxy_milter_finish_handler: finished milter connection

When Mailman 3 sends the email to Postfix for sending out:

2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
proxy_accept_socket: accepted milter connection from 127.0.0.1 port
44746
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; milter;
rspamd_milter_process_command: got connection from 127.0.0.1:34878
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_message_parse: loaded message; id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>;
queue-id: <51B0ABEA9C>; size: 3909; checksum:
<1b29c03e5f5475bcabec1e787f28d4ef>
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_mime_part_detect_language: detected part language: en
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua;
greylist.lua:204: skip greylisting for local networks and/or
authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
dkim_symbol_callback: skip DKIM checks for local networks and
authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua; spf.lua:186:
skip SPF checks for local networks and authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; arc; arc.lua:205:
got 1 arc sections
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; arc; arc.lua:342:
processed arc signature zohomail.com[1]: true(nil), 0 processed
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua;
dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not
checked
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
lua_dkim_tools.lua:166: mail is from local address
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
lua_dkim_tools.lua:382: use domain(envelope) for signature:
mm3.lavasoftware.org
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
lua_dkim_tools.lua:402: final DKIM domain: mm3.lavasoftware.org
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
lua_dkim_tools.lua:51: set domain to "mm3.lavasoftware.org" using
dkim_domain
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim_signing;
dkim_signing.lua:129: using key
"/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key", use selector
"dkim" for domain "mm3.lavasoftware.org"
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua;
once_received.lua:99: Skipping once_received for authenticated user or
local network
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; dkim;
rspamd_dkim_check: arc_sig: bh value mismatch: got
i2Ji19OooPA0mAVbxb/Wh0tqFcGTSvsGoLDHuoc4EIs=, expected
57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075;
d=zohomail.com; s=zohoarc
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; arc; arc.lua:260:
checked arc signature zohomail.com: false(reject), 0 processed
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; arc;
lua_dkim_tools.lua:170: mail is ineligible for signing
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_task_write_log: id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid:
<51B0ABEA9C>, ip: 127.0.0.1, from:
<test-bounces at mm3.lavasoftware.org>, (default: F (no action):
[0.80/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] =
sig:zohomail.com:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){philip.colmer at codelinaro.org;test-bounces at mm3.lavasoftware.org;},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},TAGGED_FROM(0.00){philip.colmer=linaro.org;},TO_DN_ALL(0.00){}]),
len: 3909, time: 243.345ms, dns req: 11, digest:
<1b29c03e5f5475bcabec1e787f28d4ef>, rcpts: <philip.colmer at linaro.org>,
mime_rcpts: <test at mm3.lavasoftware.org>
2021-09-21 12:50:26 #563874(rspamd_proxy) <b639e9>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned
using pcre, 1.90KiB scanned total
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_message_parse: loaded message; id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>;
queue-id: <9D95CBEA9F>; size: 3909; checksum:
<1b29c03e5f5475bcabec1e787f28d4ef>
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_mime_part_detect_language: detected part language: en
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua;
greylist.lua:204: skip greylisting for local networks and/or
authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
dkim_symbol_callback: skip DKIM checks for local networks and
authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua; spf.lua:186:
skip SPF checks for local networks and authorized users
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:205:
got 1 arc sections
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim;
rspamd_dkim_check: arc_sig: bh value mismatch: got
T3JEKsVRCAWS09CDUv/Nc9LZJQ387FwCcC1OB2ceZ5g=, expected
57RF2PK8//6brM5Ao6K9khEDfgy6VGY6fHK3uipwhDM=; body length 1077->1075;
d=zohomail.com; s=zohoarc
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:260:
checked arc signature zohomail.com: false(reject), 0 processed
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc; arc.lua:342:
processed arc signature zohomail.com[1]: true(nil), 0 processed
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua;
dmarc.lua:349: skip DMARC checks as either SPF or DKIM were not
checked
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; arc;
lua_dkim_tools.lua:170: mail is ineligible for signing
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
lua_dkim_tools.lua:166: mail is from local address
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
lua_dkim_tools.lua:382: use domain(envelope) for signature:
mm3.lavasoftware.org
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
lua_dkim_tools.lua:402: final DKIM domain: mm3.lavasoftware.org
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
lua_dkim_tools.lua:51: set domain to "mm3.lavasoftware.org" using
dkim_domain
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; dkim_signing;
dkim_signing.lua:129: using key
"/var/lib/rspamd/dkim/mm3.lavasoftware.org.dkim.key", use selector
"dkim" for domain "mm3.lavasoftware.org"
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua;
once_received.lua:99: Skipping once_received for authenticated user or
local network
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of
classifier bayes: not enough learns 0; 200 required
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_stat_classifiers_process: skip statistics as SPAM class is
missing
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; lua;
greylist.lua:318: Score too low - skip greylisting
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_task_write_log: id:
<17c08683040.cea0c1d6254957.4985502140819883283 at codelinaro.org>, qid:
<9D95CBEA9F>, ip: 127.0.0.1, from:
<test-bounces at mm3.lavasoftware.org>, (default: F (no action):
[0.79/15.00] [ARC_REJECT(1.00){signature check failed: fail, {[1] =
sig:zohomail.com:reject};},MAILLIST(-0.20){mailman;},MIME_GOOD(-0.10){multipart/mixed;multipart/alternative;text/plain;},RCVD_NO_TLS_LAST(0.10){},HAS_LIST_UNSUB(-0.01){},XM_UA_NO_VERSION(0.01){},DKIM_SIGNED(0.00){mm3.lavasoftware.org:s=dkim;},FORGED_RECIPIENTS_MAILLIST(0.00){},FORGED_SENDER_MAILLIST(0.00){},FROM_HAS_DN(0.00){},FROM_NEQ_ENVFROM(0.00){philip.colmer at codelinaro.org;test-bounces at mm3.lavasoftware.org;},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:+;3:~;4:+;},PREVIOUSLY_DELIVERED(0.00){test at mm3.lavasoftware.org;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},TAGGED_FROM(0.00){philip.colmer=codelinaro.org;},TO_DN_ALL(0.00){}]),
len: 3909, time: 9.960ms, dns req: 10, digest:
<1b29c03e5f5475bcabec1e787f28d4ef>, rcpts:
<philip.colmer at codelinaro.org>, mime_rcpts:
<test at mm3.lavasoftware.org>
2021-09-21 12:50:26 #563874(rspamd_proxy) <209bc2>; proxy;
rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned,
4 regexps matched, 176 regexps total, 84 regexps cached, 0B scanned
using pcre, 1.90KiB scanned total
2021-09-21 12:50:26 #563874(rspamd_proxy) <6370ae>; proxy;
proxy_milter_finish_handler: finished milter connection