Hey Rob and Tom,
> I would definitely try using --inplace and/or --append and/or
> --append-verify. And please report back how it goes, if that resolves
> the issue then we can just add a note to the documentation so people
> will know about this gotcha.
I did some more tests now; but the results vary. First the good news:
Using "--inplace" or "--append" works nicely as long as the log file is
not rotated/truncated. Changes are picked up by heka, all good...
However, as soon as a log file is truncated/rotated, you cannot use
rsync --append or --append-verify anymore to sync it -- because the file
on the target is bigger than on the source, it will be skipped
altogether. So that leaves us with rsync without any options that way.
As fas as I understand, Heka does *only* detect logfile rotation when:
a) the file size *shrinks* and b) the file handle is different.
While I think that generally is sensible for Heka, this breaks the rsync
way of syncing log files; as one would essentially need to do the following:
IF target file size > source file size
# logrotate took place
do a normal rsync; adding a new file, and replacing the target file
completely - leading to a new file handle and Heka to pick up the file
correctly
ELSE
do rsync --inplace (or --append); as otherwise, Heka won't see the new
messages.
END
I wrote a small script (see below) which does exactly that; maybe it's
useful for others as well. This allows to effectively sync log files in
a way that Heka picks them up.
Are *both* conditions needed for Heka to work, or would just a) work
without problems? This would solve the rsync issue effectively in the
long run, and remove the need for any additional scripts.
Greets,
Sebastian
-- log file sync bash script --
#!/bin/bash
localSourceFile=$1
remoteHost=$2
remoteTargetFile=$3
ls -l testTarget/test.log | awk -F" " '{ print $5 }'
localFilesize=`ls -l $localSourceFile | awk -F' ' '{ print $5 }'`
echo "ssh $remoteHost ls -l $remoteTargetFile | | awk -F' ' '{ print $5 }'"
remoteFilesize=`ssh $remoteHost du $remoteTargetFile | cut -f1`
if [ "$localFilesize" -lt "$remoteFilesize" ]; then
rsync $localSourceFile $remoteHost:$remoteTargetFile
else
rsync --append $localSourceFile $remoteHost:$remoteTargetFile
fi
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
