[Rspamd-Users] detailed log in pure JSON

[Patrick Proniewski]

Hello,

I’m new to Rspamd, in the process of evaluating the possibility to migrate from Amavisd-new (+SA +clamav).

I need to get (very) detailed logs for every email passing through in order to feed a splunk server (to get dashboards, alerting, and easy & comprehensive forensic).
Small sample of what I have now: https://www.patpro.net/cafesale/json-amavis-splunk.png (some info are not shown here, like user-agent or attachements’ names, etc.).

I’m digging the documentation for few days now and I think collecting the message info is not so difficult.

On the other hand I’m having a hard time figuring how to create a pure JSON output. Best I’ve done so far is to get a big blob of JSON in the middle of main rspamd.log:

2023-09-24 15:18:18 #53980(rspamd_proxy) <0b9aa0>; proxy; finalize_item: slow rule: SURBL_MULTI(483): 3387.30 ms; enable slow timer delay
2023-09-24 15:18:18 #53980(rspamd_proxy) <0b9aa0>; lua; greylist.lua:331: Score too low - skip greylisting
2023-09-24 15:18:18 #53980(rspamd_proxy) <0b9aa0>; lua; [string "return function (t...:4: METATEST {"user":"unknown","rcpt":[" ... JSON YADA YADA JSON ...}
2023-09-24 15:18:18 #53980(rspamd_proxy) <0b9aa0>; lua; neural.lua:351: skip ham sample to keep spam/ham balance; probability 1; 0 spam and 1 ham vectors stored
2023-09-24 15:18:18 #53980(rspamd_proxy) <0b9aa0>; proxy; rspamd_task_write_log: id: <...>, qid: <46438240FD>, ip: 188.68.34.52, from: <...>, ... REGULAR LOG OUTPUT ...


To get this result I’ve used local.d/metadata_exporter.conf that reads:

# Define custom backend(s)
custom_push {
  mine = <<EOD
return function (task, data, rule)
-- Log payload
local rspamd_logger = require "rspamd_logger"
rspamd_logger.infox(task, 'METATEST %s', data)
end
EOD;
}

rules {
   CUSTOM_EXPORT {
	 selector = "default";
	 formatter = "json";
	 backend = "mine";
   }
}

(inspired from the documentation at https://rspamd.com/doc/modules/metadata_exporter.html)


So, is there a way to have pure JSON output written elsewhere than in the main rspamd.log?
Is there a way to improve this default JSON output by adding missing info?

thanks a lot
Patrick