[Rspamd-Users] ratelimit-module behaves differently in rspamd 3.5

Fri May 26 11:37:07 UTC 2023

Not sure that we have the same problem but it seems that after the 
update from 3.4-1 to 3.5-2 ratelimit started to behave different.
We use custom lua script (not touched for year or so). First we noticed 
that some clients can't send messages. Then in the logs I see difference 
in the logs before the update and after:

before update:
2023-05-16 03:35:56 #3456726(rspamd_proxy) <c60e4b>; lua; 
ratelimit.lua:617: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 2.0334126458334 (0.1985:0.0983 
dyn); redis key: RLfyzdsqnmdx9xn1hkksqnquro
2023-05-16 03:36:00 #3456726(rspamd_proxy) <beec3a>; lua; 
ratelimit.lua:617: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 2.0263361208334 (0.1985:0.0983 
dyn); redis key: RLfyzdsqnmdx9xn1hkksqnquro
2023-05-16 03:36:39 #3456725(rspamd_proxy) <593614>; lua; 
ratelimit.lua:617: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 1.9616813625001 (0.1985:0.0983 
dyn); redis key: RLfyzdsqnmdx9xn1hkksqnquro
2023-05-16 03:36:44 #3456727(rspamd_proxy) <084701>; lua; 
ratelimit.lua:617: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 1.9533791000001 (0.1985:0.0983 
dyn); redis key: RLfyzdsqnmdx9xn1hkksqnquro

after the update:

2023-05-25 03:33:33 #3269(rspamd_proxy) <ef882b>; lua; 
ratelimit.lua:624: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 0 (0:0.1119 dyn); redis key: 
RLgxwu4mpxarpgh1gjwyosk3as
2023-05-25 03:33:44 #3268(rspamd_proxy) <fb5572>; lua; 
ratelimit.lua:624: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 0 (0:0.1119 dyn); redis key: 
RLgxwu4mpxarpgh1gjwyosk3as
2023-05-25 03:33:44 #3268(rspamd_proxy) <a09274>; lua; 
ratelimit.lua:624: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 0 (0:0.1119 dyn); redis key: 
RLgxwu4mpxarpgh1gjwyosk3as
2023-05-25 03:33:44 #3268(rspamd_proxy) <0bff37>; lua; 
ratelimit.lua:624: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 0 (0:0.1119 dyn); redis key: 
RLgxwu4mpxarpgh1gjwyosk3as
2023-05-25 03:33:44 #3268(rspamd_proxy) <32123e>; lua; 
ratelimit.lua:624: ratelimit "customrl(rs_customrl_username at domain.tld)" 
exceeded, (30 / 0.008333333333333333): 0 (0:0.1119 dyn); redis key: 
RLgxwu4mpxarpgh1gjwyosk3as

notice the decreasing number in the first example but in the second it 
is always 0 (zero) and none of the messages pass!

Here is the code we have:

/etc/rspamd/custom-ratelimit.lua

================================
   envelope = raw_envelope[1]["addr"]
   domain = raw_envelope[1]["domain"]
         get_user = task:get_user()
  if not  mta_ip == "127.0.0.1" then
          return
  end
  if domain == "somedomain.net" or domain == "somedomain.com" or domain 
== "otherdomain.com" then
          return
  end

  user = ""
has_header = task:has_header('X-PHP-Originating-Script')

  if  has_header and mta_ip == "127.0.0.1"  then
          user = envelope
         else
         user = get_user
         end
     bucket = {
                 burst = 30;
                 rate = "15 / 30m";
                 symbol = "OUT_RATELIMIT_EXCEEDED";
                 message = "Message rate limit exceeded!! Try again later";
   }
   if not user then
           return
   else
   return "rs_customrl_" .. user, bucket
   end
end

return custom_keywords
=============================

/etc/rspamd/local.d/ratelimit.conf
==============================
# local.d/ratelimit.conf

servers = 127.0.0.1;

info_symbol = OUT_RATELIMIT_EXCEEDED

custom_keywords = "/etc/rspamd/custom-ratelimit.lua";

     whitelisted_rcpts = "postmaster,mailer-daemon";
=============================

Strange thing is that we doesn't have the code mentioned in the patch - 
actually we do not have the file that is changed

rspamd --version
Rspamd daemon version 3.5

On 31-Mar-23 2:03 PM, Simbürger, Andreas wrote:
> It can happen that once limited, you cannot send any messages. We observed
> it this week with one of our gateways. However this can happen without the
> patch as well, if the inbound MTA sends too many recipients in one mail.
> 
> This week, we ran into it because the inbound MTA allowed 2000 recipients in one
> Mail and our burst limit per user was 240 :-), ooooops. This connection between
> max number of recipients and burst always existed, but the invisible factor of
> 2 will make people unhappy for not easily observable reasons.
> 
> Cheers,
> Andreas
> 
> Am Friday, dem 31.03.2023 um 11:40 +0100 schrieb Vsevolod Stakhov:
>> On 31/03/2023 11:17, Simbürger, Andreas wrote:
>>> Hi,
>>>
>>> rspamd 3.5 introduced a new behavior when checking ratelimit prefixes.
>>> Since 3.5 it will consider the messages that will be sent out in the current
>>> task as 'pending' messages. This will trigger the ratelimit earlier.
>>>
>>> In the released 3.5 version exists a small bug in the check lua code that is
>>> fixed in master: https://github.com/rspamd/rspamd/pull/4448
>>> Here rspamd will factor in the number of recipients of the current task twice,
>>> resulting in an even higher burst value for incoming messages.
>>>
>>> As the leak rate has a lower bound between tries (the burst value is always clamped
>>> to 0, if it becomes negative between tasks), you can end up in a situation where
>>> you cannot send any new messages, no matter what rate you specified.
>>>
>>> The fix is pretty easy, if you want to repair it in your deployed installation.
>>>
>>> Cheers,
>>> Andreas
>>>
>>
>> It is probably worth to include this patch in the stable packages to
>> avoid confusion. What do you think?
>>
> 
> 