Hi,
I'm trying to use lsyncd + csync2 to keep a bunch of web cache servers
(for testing 3 right now) running in AWS in sync. I'm following the
great tutorial on
https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/
(kudos to Floren, in case he reads). I got it basically running (and
almost understood what I did), but I encounter (at least) two problems.
The second one later about...
I'm dealing with quite a lot of files in /mnt/tst_vol_aws/sat_p_cache
distributed in various subdirectories 3 levels down from there. I can't
influence the layout easily, I would take it as given for now. The test
setup already has 0.8 Mio files (the final setup is supposed to hold
5Mio files and more) and files are added quite fast, especially in the
beginning when the cache is cold, give or take 120/s, distributed evenly
across all 3 machines. They are all rather small, below 50KB, so the
test volume as about 30GB, the full set would roughly have up to 1TB.
The web cache is adding constantly files to the folders, removal is not
yet planned.
Floren's setup invokes "/usr/sbin/csync2 -C $cfg -x" on a batch of
events within 1s delay. With all 3 servers completely in sync, no file
changes in progress, that command already runs for >1min. Running
"/usr/sbin/csync2 -c -r -v /mnt/tst_vol_aws/sat_p_cache" on the full
data set of 5Mio just took almost 8h (though that's on NFS which could
slow down things a bit). Needless to say that this does not keep things
in sync. The csync2's SQLite DB is already about 236MB for the 0.8Mio
files, for the full 5Mio set its 1.4GB, I'm not sure if that could
become a bottleneck. I tried putting it on tmpfs to no avail.
Is that performance in general expected on that data set? Was csync2
designed to work on that amount of files? I'm using m1.xlarge instances
with 4x instance store in a RAID0, with an XFS on top of it. I could
consider switch to larger instances (I need to anyway, for other
reasons), if that would bring improvements.
I tried to improve Floren's /etc/lsyncd.conf a bit by trying to give
csync2 a hint which files to focus on:
...
action = function(inlet)
local config = inlet.getConfig()
local elist = inlet.getEvents(function(event)
return event.etype ~= "Init"
end)
local directory = string.sub(config.source, 1, -2)
local paths = elist.getPaths(function(etype, path)
return directory .. path
end)
log("Normal", "Processing syncing list: ", table.getn(paths))
spawn(elist, "/usr/sbin/csync2", "-C", config.syncid, "-x",
paths)
end,
...
which would invoke "/usr/sbin/csync2 -C $cfg -x $long_list_of_files" on
every event (if I understood things correctly). I did not have the
impression that this speeds up things. I quickly see the number of
files to be synchronized beyond 1000 and encounter very soon:
Thu Aug 1 09:14:25 2013 Normal: --- OVERFLOW in event queue ---
Thu Aug 1 09:14:25 2013 Normal: --- HUP signal, resetting ---
Then lsyncd starts with init again which does a full csync2 -x run which
causes the machine to miss even more new files.
Any advice how to speed up things is much appreciated.
Kind regards,
Frank.
_______________________________________________
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2
