[Rspamd-Users] Redis learned_ids key has huge size.

[Gabriele Nencioni]

On 1/28/20 8:04 AM, Gabriele Nencioni wrote:
> On 1/27/20 4:21 PM, Alexander Moisseev wrote:
>> As any other major operation, I would try it first in test environment.
>> Needless to say you should have a backup.
> Sure.
> 
>> I don't remember exact numbers. You need about 2 more times RAM for new
>> schema + Rspamd keeps old schema until conversion is not finished.I have performed some test with rspamadm configwizard statistic command
> but it returns:
> 
> # rspamadm configwizard statistic
> CPU doesn't have SSSE3 instructions set required for hyperscan, disable it
> No changes found, the wizard is finished now
> 
> and nothing has been changed on redis.
> 
> # grep flags /proc/cpuinfo | head -n1
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl
> xtopology pni cx16 x2apic hypervisor lahf_lm kaiser
> 
> The SSSE3 instructions missed, is that the problem?
> Could you please help me?



Probably the SSS3E instructions missed is not the problem.


I'm adding my classifier configuration dump:
# rspamadm configdump classifier
CPU doesn't have SSSE3 instructions set required for hyperscan, disable it
*** Section classifier ***
bayes {
    backend = "redis";
    min_tokens = 11;
    timeout = 20;
    expire = 15552000;
    lazy = true;
    servers = "127.0.0.1:6379";
    learn_condition = <<EOD
return function(task, is_spam, is_unlearn)
  local learn_type = task:get_request_header('Learn-Type')

  if not (learn_type and tostring(learn_type) == 'bulk') then
    local prob = task:get_mempool():get_variable('bayes_prob', 'double')

    if prob then
      local in_class = false
      local cl
      if is_spam then
        cl = 'spam'
        in_class = prob >= 0.95
      else
        cl = 'ham'
        in_class = prob <= 0.05
      end

      if in_class then
        return false,string.format('already in class %s; probability
%.2f%%',
          cl, math.abs((prob - 0.5) * 200.0))
      end
    end
  end

  return true
end
EOD;
    new_schema = true;
    statfile {
        spam = false;
        symbol = "BAYES_HAM";
    }
    statfile {
        spam = true;
        symbol = "BAYES_SPAM";
    }
    languages_enabled = true;
    tokenizer {
        name = "osb";
    }
    name = "global";
    min_learns = 200;
}

*** End of section classifier ***


Thank you again.
-- 
Gabriele Nencioni