Re: svnsync crashes on a huge commit

2013-08-18 Thread Ben Reser
On 8/15/13 8:15 PM, Ben Reser wrote: > Not sure if this is really complete. I don't think the directory case > or the property case really should be using the dst_rev_pool. In fact > I really don't understand why the replay_ctx->file_pool exists as well > as the info->pool. I think we should pr

Re: svnsync crashes on a huge commit

2013-08-16 Thread Philip Martin
Ben Reser writes: > I'm inclined to take a more conservative approach. The following patch > should fix most of this problem on 1.8.x without pulling the new XML > parsing implementation onto 1.8.x. > > [[[ > Index: subversion/libsvn_ra_serf/replay.c > ===

Re: svnsync crashes on a huge commit

2013-08-15 Thread Ben Reser
On 7/28/13 1:52 AM, Ben Reser wrote: > Even if the change that is resolving the problem on trunk isn't viable > for backport, that still doesn't mean we won't fix the problem till > 1.9.x, it just means a fix that will work for 1.8.x needs to be > identified. I looked into merging the trunk change

Re: svnsync crashes on a huge commit

2013-07-28 Thread Ben Reser
On Sun, Jul 28, 2013 at 1:17 AM, Anatoly Zapadinsky wrote: > It is not a suggestion or a feature request. We are discussing a bug > which was introduced in 1.8 during the transition to serf. There is a > bugfix in trunk. How could it be disputed to merge it to 1.8 branch or > not? You have to post

Re: svnsync crashes on a huge commit

2013-07-28 Thread Anatoly Zapadinsky
On Sat, Jul 27, 2013 at 12:14 PM, Ben Reser wrote: >> I am sorry to open this thread again. But it seems that this revision >> was not included in 1.8.1. Will it be included in 1.8.2? > > It hasn't been nominated at this point. I haven't looked at the > change to know if I'd be willing to nomina

Re: svnsync crashes on a huge commit

2013-07-27 Thread Ben Reser
On Fri, Jul 26, 2013 at 2:56 PM, Anatoly Zapadinsky wrote: > I am sorry to open this thread again. But it seems that this revision > was not included in 1.8.1. Will it be included in 1.8.2? It hasn't been nominated at this point. I haven't looked at the change to know if I'd be willing to nomina

Re: svnsync crashes on a huge commit

2013-07-26 Thread Anatoly Zapadinsky
I am sorry to open this thread again. But it seems that this revision was not included in 1.8.1. Will it be included in 1.8.2? I want to use svnsync on windows and compiling trunk on windows looks like a nightmare after some superficial googling. On Sat, Jul 20, 2013 at 10:31 PM, Anatoly Zapadins

Re: svnsync crashes on a huge commit

2013-07-20 Thread Anatoly Zapadinsky
I confirm that the trunk do not consume more than 50Mb on this commit. And also it is subjectively faster. You should merge this bugfix to 1.8 branch. On Fri, Jul 19, 2013 at 2:32 PM, Philip Martin wrote: > Philip Martin writes: > >> I see those values with 1.8.x but the problem appears to be fi

Re: svnsync crashes on a huge commit

2013-07-19 Thread Philip Martin
Philip Martin writes: > I see those values with 1.8.x but the problem appears to be fixed on > trunk where the memory use is remains more or less constant at 100MB > virtual, 25MB resident. It's r1499863, the rewrite of ra_serf's replay code, that fixes the memory use. Reverse merging that revi

Re: svnsync crashes on a huge commit

2013-07-18 Thread Philip Martin
I see those values with 1.8.x but the problem appears to be fixed on trunk where the memory use is remains more or less constant at 100MB virtual, 25MB resident. Philip Martin writes: > I see smaller numbers on my 64-bit Linux box. r18 is the worst at about > 550MB. > > Anatoly Zapadinsky write

Re: svnsync crashes on a huge commit

2013-07-18 Thread Philip Martin
I see smaller numbers on my 64-bit Linux box. r18 is the worst at about 550MB. Anatoly Zapadinsky writes: > I've just tried this at home using console apps from tortoise > distributive (1.8.0 r1490375). Computer is running Windows2003Server > 32bit version. > > svncync already allocated 660 Mb t

Re: svnsync crashes on a huge commit

2013-07-18 Thread Stefan Fuhrmann
On Thu, Jul 18, 2013 at 4:08 PM, Anatoly Zapadinsky wrote: > >> > Does the memory usage already build up while transmitting the > >> > data (i.e. while the dots are being shown) or mainly during the > >> > commit stage, i.e. the during short delay after the last dot and > >> > the "Committed revis

Re: svnsync crashes on a huge commit

2013-07-18 Thread Anatoly Zapadinsky
>> > Does the memory usage already build up while transmitting the >> > data (i.e. while the dots are being shown) or mainly during the >> > commit stage, i.e. the during short delay after the last dot and >> > the "Committed revision xyz" message? >> >> On transmitting data stage. During the commi

Re: svnsync crashes on a huge commit

2013-07-18 Thread Stefan Fuhrmann
On Thu, Jul 18, 2013 at 3:10 PM, Anatoly Zapadinsky wrote: > On Thu, Jul 18, 2013 at 3:24 PM, Stefan Fuhrmann > wrote: > > > > > > Have you tried it with any newer Windows version (post-W2k3)? > > There might be an address space fragmentation issue between > > CRT and OS memory management. > > Ye

Re: svnsync crashes on a huge commit

2013-07-18 Thread Anatoly Zapadinsky
On Thu, Jul 18, 2013 at 3:24 PM, Stefan Fuhrmann wrote: > > > Have you tried it with any newer Windows version (post-W2k3)? > There might be an address space fragmentation issue between > CRT and OS memory management. > Yes. I've tried it on Ubuntu 12.08 (1.8 subversion was compiled from download

Re: svnsync crashes on a huge commit

2013-07-18 Thread Stefan Fuhrmann
On Thu, Jul 18, 2013 at 11:26 AM, Anatoly Zapadinsky wrote: > I've just tried this at home using console apps from tortoise > distributive (1.8.0 r1490375). Computer is running Windows2003Server > 32bit version. > I can't reproduce this on LINUX. I will run this later today on Win7 32 bits. Have

Re: svnsync crashes on a huge commit

2013-07-18 Thread Anatoly Zapadinsky
I've just tried this at home using console apps from tortoise distributive (1.8.0 r1490375). Computer is running Windows2003Server 32bit version. svncync already allocated 660 Mb transmitting file data for the 5th revision. Memory is deallocated after every new revision is transferred, so it is no

RE: svnsync crashes on a huge commit

2013-07-18 Thread Bert Huijben
From: Anatoly Zapadinsky [mailto:zapadin...@gmail.com] Sent: woensdag 17 juli 2013 14:04 To: Lieven Govaerts Cc: Subversion Development Subject: Re: svnsync crashes on a huge commit I've run 64 bit version on virtual machine and access fsfs repository on host windows computer through smb. It'

Re: svnsync crashes on a huge commit

2013-07-18 Thread Anatoly Zapadinsky
On Wed, Jul 17, 2013 at 6:01 PM, Stefan Fuhrmann wrote: > > > All that said, it may still be necessary or at least be useful > to retry opening any FSFS file (in case of network glitches > etc.). Because there opening files should never fail as part > of the normal operation, retrying when it does

RE: svnsync crashes on a huge commit

2013-07-18 Thread Bert Huijben
ag 17 juli 2013 14:04 To: Lieven Govaerts Cc: Subversion Development Subject: Re: svnsync crashes on a huge commit I've run 64 bit version on virtual machine and access fsfs repository on host windows computer through smb. It's because 32bit version of svnsync crashed on host windows

Re: svnsync crashes on a huge commit

2013-07-17 Thread Anatoly Zapadinsky
I've run 64 bit version on virtual machine and access fsfs repository on host windows computer through smb. It's because 32bit version of svnsync crashed on host windows machine on this huge commit with out of memory exception. I've found an easy way to reproduce this excessive memory use in svnsy

Re: svnsync crashes on a huge commit

2013-07-17 Thread Philip Martin
Stefan Fuhrmann writes: > read_next_ids accesses a txn-local file which should never > be accessed by multiple processes / threads at the same > time. It also gets closed explicitly such that there should be > no pool / file handle lifetime issue. I imagine an http client could do multiple PUTs

Re: svnsync crashes on a huge commit

2013-07-17 Thread Stefan Fuhrmann
On Wed, Jul 17, 2013 at 1:49 PM, Lieven Govaerts wrote: > (bringing this to dev) > > Devs, > > Anatoly sent me some more info in this issue and log files of svnsync > 1.8.0+serf 1.2.1 with logging enabled. > > He's running svnsync from a https repository (ra_serf) to a local > repository (ra_loca

Re: svnsync crashes on a huge commit

2013-07-17 Thread Lieven Govaerts
(bringing this to dev) Devs, Anatoly sent me some more info in this issue and log files of svnsync 1.8.0+serf 1.2.1 with logging enabled. He's running svnsync from a https repository (ra_serf) to a local repository (ra_local). This on an Ubuntu 12.08 VM with 64-bit binary. In the log files I rec