Re: Trying Bayes / Redis

[Marc Perkel]

This Bayes Redis works GREAT. For years I've been trying to get bayes to 
work and now finally IT WORKS



--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: Trying Bayes / Redis

[Axb]

On 12/15/2015 10:57 PM, Marc Perkel wrote:

This Bayes Redis works GREAT. For years I've been trying to get bayes to
work and now finally IT WORKS


good news for once...
just watch memory usage... :)

Re: Trying Bayes / Redis

[Axb]

On 12/12/2015 12:28 AM, Marc Perkel wrote:


redis_version:2.4.10



You're using an ancient Redis version...
SA makes use of LUA support which was added in 2.6.0

You definitely need to upgrde to 3.x and you'll probably need to nuke 
your DB dump  before the upgrade...

Re: Trying Bayes / Redis

[Reindl Harald]

Am 12.12.2015 um 18:57 schrieb Marc Perkel:

On 12/12/15 09:36, Martin Gregorie wrote:

On Sat, 2015-12-12 at 08:49 -0800, Marc Perkel wrote:


I can put redis in thew yum exception list


Or, you can install your self-built version in /usr/local/bin and
adjust $PATH so it preceeds /usr and /usr/bin. This will protect your
version from yum or dnf updates.

Then keep an eye on subsequent updates and remove your version if/when
yum or dnf delivers a more recent version.



Actually I did rpm -e --justdb to take it out so it doesn't get updated


as long there is nothing with a dependency to it installed and get a 
update later :-)




signature.asc
Description: OpenPGP digital signature

Re: Trying Bayes / Redis

[Marc Perkel]

On 12/12/15 08:47, Reindl Harald wrote:



Am 12.12.2015 um 17:44 schrieb Marc Perkel:

On 12/12/15 02:38, Axb wrote:

On 12/12/2015 12:28 AM, Marc Perkel wrote:


redis_version:2.4.10



You're using an ancient Redis version...
SA makes use of LUA support which was added in 2.6.0

You definitely need to upgrde to 3.x and you'll probably need to nuke
your DB dump  before the upgrade...



I'm running Centos 6. I just downloaded and compiled the latest. Can  I
just copy over the executables and keep my data or do I need to start 
over?


if it works with the old data you can keep it, otherwise as said 
"probably need to nuke  your DB dump"


but don't do make && make install on producton machines

sooner or later your build will get overwritten by a yum-update with 
the old version + patches because rpm don't know anything about your 
override


https://wiki.centos.org/HowTos/SetupRpmBuildEnvironment




Worked with the old data.

--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: Trying Bayes / Redis

[Reindl Harald]

Am 12.12.2015 um 18:24 schrieb Marc Perkel:


On 12/12/15 02:38, Axb wrote:

On 12/12/2015 12:28 AM, Marc Perkel wrote:


redis_version:2.4.10



You're using an ancient Redis version...
SA makes use of LUA support which was added in 2.6.0

You definitely need to upgrde to 3.x and you'll probably need to nuke
your DB dump  before the upgrade...



What does LUA mean?


https://en.wikipedia.org/wiki/Lua_%28programming_language%29

heavily used for many things "embeede" in mod_security, apache 
trafficserver as a lightweight scripting language




signature.asc
Description: OpenPGP digital signature

Re: Trying Bayes / Redis

[Marc Perkel]

On 12/12/15 02:38, Axb wrote:

On 12/12/2015 12:28 AM, Marc Perkel wrote:


redis_version:2.4.10



You're using an ancient Redis version...
SA makes use of LUA support which was added in 2.6.0

You definitely need to upgrde to 3.x and you'll probably need to nuke 
your DB dump  before the upgrade...






What does LUA mean?

--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: Trying Bayes / Redis

[Marc Perkel]

On 12/12/15 02:38, Axb wrote:

On 12/12/2015 12:28 AM, Marc Perkel wrote:


redis_version:2.4.10



You're using an ancient Redis version...
SA makes use of LUA support which was added in 2.6.0

You definitely need to upgrde to 3.x and you'll probably need to nuke 
your DB dump  before the upgrade...





I'm running Centos 6. I just downloaded and compiled the latest. Can  I 
just copy over the executables and keep my data or do I need to start over?


--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: Trying Bayes / Redis

[Martin Gregorie]

On Sat, 2015-12-12 at 08:49 -0800, Marc Perkel wrote:

> I can put redis in thew yum exception list
> 
Or, you can install your self-built version in /usr/local/bin and
adjust $PATH so it preceeds /usr and /usr/bin. This will protect your
version from yum or dnf updates. 

Then keep an eye on subsequent updates and remove your version if/when
yum or dnf delivers a more recent version.


Martin

Re: Trying Bayes / Redis

[Marc Perkel]

On 12/12/15 09:36, Martin Gregorie wrote:

On Sat, 2015-12-12 at 08:49 -0800, Marc Perkel wrote:


I can put redis in thew yum exception list


Or, you can install your self-built version in /usr/local/bin and
adjust $PATH so it preceeds /usr and /usr/bin. This will protect your
version from yum or dnf updates.

Then keep an eye on subsequent updates and remove your version if/when
yum or dnf delivers a more recent version.


Martin





Actually I did rpm -e --justdb to take it out so it doesn't get updated.

BTW - this is working GREAT. I'm thinking about learning redis and doing 
interesting things with it.


--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: Trying Bayes / Redis

[Reindl Harald]

Am 12.12.2015 um 17:44 schrieb Marc Perkel:

On 12/12/15 02:38, Axb wrote:

On 12/12/2015 12:28 AM, Marc Perkel wrote:


redis_version:2.4.10



You're using an ancient Redis version...
SA makes use of LUA support which was added in 2.6.0

You definitely need to upgrde to 3.x and you'll probably need to nuke
your DB dump  before the upgrade...



I'm running Centos 6. I just downloaded and compiled the latest. Can  I
just copy over the executables and keep my data or do I need to start over?


if it works with the old data you can keep it, otherwise as said 
"probably need to nuke  your DB dump"


but don't do make && make install on producton machines

sooner or later your build will get overwritten by a yum-update with the 
old version + patches because rpm don't know anything about your override


https://wiki.centos.org/HowTos/SetupRpmBuildEnvironment




signature.asc
Description: OpenPGP digital signature

Re: Trying Bayes / Redis

[Marc Perkel]

On 12/12/15 08:47, Reindl Harald wrote:



Am 12.12.2015 um 17:44 schrieb Marc Perkel:

On 12/12/15 02:38, Axb wrote:

On 12/12/2015 12:28 AM, Marc Perkel wrote:


redis_version:2.4.10



You're using an ancient Redis version...
SA makes use of LUA support which was added in 2.6.0

You definitely need to upgrde to 3.x and you'll probably need to nuke
your DB dump  before the upgrade...



I'm running Centos 6. I just downloaded and compiled the latest. Can  I
just copy over the executables and keep my data or do I need to start 
over?


if it works with the old data you can keep it, otherwise as said 
"probably need to nuke  your DB dump"


but don't do make && make install on producton machines

sooner or later your build will get overwritten by a yum-update with 
the old version + patches because rpm don't know anything about your 
override


https://wiki.centos.org/HowTos/SetupRpmBuildEnvironment




I can put redis in thew yum exception list

--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: Trying Bayes / Redis

[Marc Perkel]

sa-learn --dump magic

0.000  0  3  0  non-token data: bayes db version
0.000  0  69569  0  non-token data: nspam
0.000  0  88747  0  non-token data: nham
0.000  0  0  0  non-token data: ntokens
0.000  0  0  0  non-token data: oldest atime
0.000  0  0  0  non-token data: newest atime
0.000  0  0  0  non-token data: last journal 
sync atime

0.000  0  0  0  non-token data: last expiry atime
0.000  0  0  0  non-token data: last expire 
atime delta
0.000  0  0  0  non-token data: last expire 
reduction count



# Server
redis_version:3.0.5
redis_git_sha1:
redis_git_dirty:0
redis_build_id:a0e516305b2572d8
redis_mode:standalone
os:Linux 2.6.32-042stab112.15 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:2085
run_id:f2436709b788f7a9b6a043f6ed90a49512c049d2
tcp_port:6379
uptime_in_seconds:685
uptime_in_days:0
hz:10
lru_clock:7100234
config_file:/etc/redis.conf

# Clients
connected_clients:550
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:905452784
used_memory_human:863.51M
used_memory_rss:924852224
used_memory_peak:964409608
used_memory_peak_human:919.73M
used_memory_lua:177152
mem_fragmentation_ratio:1.02
mem_allocator:jemalloc-3.6.0

# Persistence
loading:0
rdb_changes_since_last_save:98170
rdb_bgsave_in_progress:1
rdb_last_save_time:1449940748
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:7
rdb_current_bgsave_time_sec:1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

# Stats
total_connections_received:3806
total_commands_processed:1302390
instantaneous_ops_per_sec:2133
total_net_input_bytes:52637812
total_net_output_bytes:15806201
instantaneous_input_kbps:80.48
instantaneous_output_kbps:23.47
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:194094
keyspace_misses:65528
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:44672
migrate_cached_sockets:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:5.97
used_cpu_user:34.09
used_cpu_sys_children:4.85
used_cpu_user_children:57.29

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=3,expires=0,avg_ttl=0
db2:keys=9277218,expires=4,avg_ttl=2546306553

Re: Trying Bayes / Redis

[Marc Perkel]

On 12/12/15 15:28, Benny Pedersen wrote:
On December 12, 2015 5:49:27 PM Marc Perkel 
 wrote:



I can put redis in thew yum exception list


so much precompiled problems, why not upgrade to centos 7 ?




Because I'd have to upgrade 50 servers for consistency and if I do that 
I'll probably try something other than centos.


--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: Trying Bayes / Redis

[Benny Pedersen]

On December 13, 2015 1:16:17 AM Marc Perkel  
wrote:



Because I'd have to upgrade 50 servers for consistency and if I do that
I'll probably try something other than centos.


okay, it just not how i would solve +1 server farms in gentoo, here i would 
emerge --buildpkgonly on master, and then emerge --usepkgonly on all 50, no 
need to use distcc, its just for me another world


back to my problems with fuglu ebuild, its added to maintainer wanted in 
gentoo now if others like to help

Re: Trying Bayes / Redis

[Benny Pedersen]

On December 12, 2015 5:49:27 PM Marc Perkel  
wrote:



I can put redis in thew yum exception list


so much precompiled problems, why not upgrade to centos 7 ?

Re: Trying Bayes / Redis

[Reindl Harald]

Am 13.12.2015 um 00:28 schrieb Benny Pedersen:

On December 12, 2015 5:49:27 PM Marc Perkel
 wrote:


I can put redis in thew yum exception list


so much precompiled problems


blablub nonsense as usual


why not upgrade to centos 7 ?


because it's the purpose of a LTS not have major updates for a ton of 
packages all the time?




signature.asc
Description: OpenPGP digital signature

Re: Trying Bayes / Redis

[Marc Perkel]

So far so good. Thanks for your help. It looks like it's actually 
working. Have about 10k spam/ham tokens so far.

Re: Trying Bayes / Redis

[Marc Perkel]

On 12/11/15 14:59, Axb wrote:

On 12/11/2015 11:55 PM, Marc Perkel wrote:

So far so good. Thanks for your help. It looks like it's actually
working. Have about 10k spam/ham tokens so far.



sa-learn --dump magic should show if it's "alive"

For safety, don't take your eyes off memory usage and play with:

bayes_token_ttl
bayes_seen_ttl

I've set
bayes_seen_ttl 1


Curious what your "redis-cli info" looks like after a while of 
running... adn if you're happy with speed and results...


keep us posted...

Axb




0.000  0  3  0  non-token data: bayes db version
0.000  0  14814  0  non-token data: nspam
0.000  0  11933  0  non-token data: nham
0.000  0  0  0  non-token data: ntokens
0.000  0  0  0  non-token data: oldest atime
0.000  0  0  0  non-token data: newest atime
0.000  0  0  0  non-token data: last journal 
sync atime

0.000  0  0  0  non-token data: last expiry atime
0.000  0  0  0  non-token data: last expire 
atime delta
0.000  0  0  0  non-token data: last expire 
reduction count


redis_version:2.4.10
redis_git_sha1:
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:472
uptime_in_seconds:4846
uptime_in_days:0
lru_clock:284149
used_cpu_sys:29.75
used_cpu_user:91.39
used_cpu_sys_children:4.31
used_cpu_user_children:49.34
connected_clients:671
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:185037872
used_memory_human:176.47M
used_memory_rss:182960128
used_memory_peak:193339376
used_memory_peak_human:184.38M
mem_fragmentation_ratio:0.99
mem_allocator:jemalloc-2.2.5
loading:0
aof_enabled:0
changes_since_last_save:42332
bgsave_in_progress:0
last_save_time:1449876339
bgrewriteaof_in_progress:0
total_connections_received:110460
total_commands_processed:9275568
expired_keys:0
evicted_keys:0
keyspace_hits:1821764
keyspace_misses:793471
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:16204
vm_enabled:0
role:master
db2:keys=2012232,expires=4


--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400

Re: Trying Bayes / Redis

[Axb]

On 12/11/2015 11:55 PM, Marc Perkel wrote:

So far so good. Thanks for your help. It looks like it's actually
working. Have about 10k spam/ham tokens so far.



sa-learn --dump magic should show if it's "alive"

For safety, don't take your eyes off memory usage and play with:

bayes_token_ttl
bayes_seen_ttl

I've set
bayes_seen_ttl 1


Curious what your "redis-cli info" looks like after a while of 
running... adn if you're happy with speed and results...


keep us posted...

Axb

Re: Trying Bayes / Redis

[Ian Zimmerman]

On 2015-12-11 14:29 -0800, Marc Perkel wrote:

> Anyone using this rule timing plugin? Having trouble getting it to
> work. Just wondering if it's worth it?
> 
> Mail::SpamAssassin::Plugin::RuleTimingRedis

I use it and I have no trouble now.  But I remember I had to disable the
LUA scripting stuff when I set it up, it wouldn't work even though my
Redis version should be recent enough to support it.

-- 
Please *no* private copies of mailing list or newsgroup messages.
Rule 420: All persons more than eight miles high to leave the court.