On Wed, Aug 26, 2009 at 12:55 PM, Jamie Lokier wrote:
> Why don't they just use Unison?
Because it doesn't work well. It is seriously unreliable with large
files, on Linux or Windows. It also has a horrifying tendency to
corrupt its own state databases, lock up, or exit unexpectedly. It
also doesn
JW wrote:
> On Monday 13 July 2009 14:18:38 Ryan Malayter wrote:
> > It would be a big boost for large files if rsync "remembered" the
> > hashes on each end, so it didn't have to re-read the files on every
> > run if the files were unchanged. This is a feature that rsync's
> > developers have reje
On Monday 13 July 2009 14:18:38 Ryan Malayter wrote:
> It would be a big boost for large files if rsync "remembered" the
> hashes on each end, so it didn't have to re-read the files on every
> run if the files were unchanged. This is a feature that rsync's
> developers have rejected, since rsync is
On Wed, Jul 15, 2009 at 12:54 PM, Jamie Lokier wrote:
> It still has to send the hashes, which can be slow for a large file.
> So it would be even better to cache on the sending side hashes of
> files on the receiving side, perhaps indexed by the receiving side's
> MD5 of the whole file.
The hashe
Ryan Malayter wrote:
> So, when transferring a large file, it goes something like this from
> the sender's perspective:
> 1) sending file list
> 2) receving file list
> 3) file is different! Recevier, please give me some hashes
> 4) got hashes
> 5) begin transfer, calculating my hashes and co
On Tue, Jul 14, 2009 at 5:17 PM, Carlos Carvalho wrote:
> Hash calculation is very fast; rsync has a negligible cpu consumption.
Hash calculation for the receiver is usually disk-bound, But rsync has
massive CPU consumption in certain cases. When using -Z on a fast
network. I have seen rsync becom
Carlos Carvalho wrote:
>
> Ryan Malayter (malay...@gmail.com) wrote on 14 July 2009 17:00:
> >On Mon, Jul 13, 2009 at 4:54 PM, Jamie
> Lokier wrote:
> >>
> >> Remembering hashes doesn't make any difference to speed, if the
> >> bottleneck is the sending side.
> >
> >Except that in the rsync
Ryan Malayter (malay...@gmail.com) wrote on 14 July 2009 17:00:
>On Mon, Jul 13, 2009 at 4:54 PM, Jamie Lokier wrote:
>>
>> Remembering hashes doesn't make any difference to speed, if the
>> bottleneck is the sending side.
>
>Except that in the rsync pipeline, the reading the destination file
On Mon, Jul 13, 2009 at 4:54 PM, Jamie Lokier wrote:
>
> Remembering hashes doesn't make any difference to speed, if the
> bottleneck is the sending side.
Except that in the rsync pipeline, the reading the destination file to
get hashes happens BEFORE the sender reads its file. And the sender
calc
Ryan Malayter wrote:
> Your log file indicates that rsync is indeed working as designed
> finding lots of data matches:
>
>Literal data: 123736377 bytes
>Matched data: 17889663500 bytes
>
> This means that rsync only had to transfer 118 MB instead of 16+ GB.
> It does this by trading CPU
Your log file indicates that rsync is indeed working as designed
finding lots of data matches:
Literal data: 123736377 bytes
Matched data: 17889663500 bytes
This means that rsync only had to transfer 118 MB instead of 16+ GB.
It does this by trading CPU and disk operations for network bytes
-Original Message-
> From: rsync-bounces+tony=servacorp@lists.samba.org
> [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On
> Behalf Of Krzysztof Nosek
> Sent: Friday, July 10, 2009 7:48 AM
> To: malay...@gmail.com; rsync@lists.samba.org
> Subject: Re: How
Hello,
So I do not think the basic data structure is the problem,
unless mysql hotcopy does something really strange like inserting a
timestamp or other changing data info every few KB in the output stream.
No, really, mysqlhotcopy performs just a raw file system copy of
/var/lib/mysql ta
On Jul 8, 2009 5:12am, Krzysztof Nosek wrote:
Anyway, I'm slowly becoming convinced that one has to remove the file
blobs
from the database entirely, moving them to the flat structure in a
directory which
would be rsynced separately. There is such feature in Mantis, as far as I
know.
I thi
Hi,
thanks for your tips... even if I'm still stuck with my problem :)
* Does the output of mysqldump compress well with gzip?
No it doesn't. The biggest table we talk about here consists mainly of
large binary blobs which are media files in compressed formats such as
jpg, avi, png, etc.
Bu
> "Krzysztof" == Krzysztof Nosek writes:
Krzysztof> First, I was dumping the database with mysqldump to an
Krzysztof> uncompressed dump and rsyncing it. Such a dump would
Krzysztof> use some 34G of disk space. [...]
Some simple suggestions in addition to the one about experimenti
On Jul 6, 2009 4:41am, Krzysztof Nosek wrote:
I'm having a very hard time rsyncing efficiently a MySQL database which
contains very large binary blobs.
How to do it better? I thought that rsync would be magically able to
extract changes from my data. However, it looks like it diffs the 17G
17 matches
Mail list logo
