Re: git-p4 out of memory for very large repository

2013-09-07 Thread Pete Wyckoff
cmt...@gmail.com wrote on Fri, 06 Sep 2013 15:03 -0400: Finally, I claim success! Unfortunately I did not try either of the OOM score or strace suggestions - sorry! After spending so much time on this, I've gotten to the point that I'm more interested in getting it to work than in figuring

Re: git-p4 out of memory for very large repository

2013-09-06 Thread Corey Thompson
On Mon, Sep 02, 2013 at 08:42:36PM +0100, Luke Diamand wrote: I guess you could try changing the OOM score for git-fast-import. change /proc/pid/oomadj. I think a value of -31 would make it very unlikely to be killed. On 29/08/13 23:46, Pete Wyckoff wrote: I usually just do git p4 sync

Re: git-p4 out of memory for very large repository

2013-09-02 Thread Luke Diamand
I guess you could try changing the OOM score for git-fast-import. change /proc/pid/oomadj. I think a value of -31 would make it very unlikely to be killed. On 29/08/13 23:46, Pete Wyckoff wrote: cmt...@gmail.com wrote on Wed, 28 Aug 2013 11:41 -0400: On Mon, Aug 26, 2013 at 09:47:56AM -0400,

Re: git-p4 out of memory for very large repository

2013-08-29 Thread Pete Wyckoff
cmt...@gmail.com wrote on Wed, 28 Aug 2013 11:41 -0400: On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote: You are correct that git-fast-import is killed by the OOM killer, but I was unclear about which process was malloc()ing so much memory that the OOM killer got invoked (as

Re: git-p4 out of memory for very large repository

2013-08-28 Thread Corey Thompson
On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote: You are correct that git-fast-import is killed by the OOM killer, but I was unclear about which process was malloc()ing so much memory that the OOM killer got invoked (as other completely unrelated processes usually also get

Re: git-p4 out of memory for very large repository

2013-08-26 Thread Corey Thompson
On Sun, Aug 25, 2013 at 11:50:01AM -0400, Pete Wyckoff wrote: Modern git, including your version, do streaming reads from p4, so the git-p4 python process never even holds a whole file's worth of data. You're seeing git-fast-import die, it seems. It will hold onto the entire file contents.

Re: git-p4 out of memory for very large repository

2013-08-25 Thread Pete Wyckoff
cmt...@gmail.com wrote on Fri, 23 Aug 2013 07:48 -0400: On Fri, Aug 23, 2013 at 08:16:58AM +0100, Luke Diamand wrote: On 23/08/13 02:12, Corey Thompson wrote: Hello, Has anyone actually gotten git-p4 to clone a large Perforce repository? Yes. I've cloned repos with a couple of Gig

Re: git-p4 out of memory for very large repository

2013-08-23 Thread Luke Diamand
On 23/08/13 02:12, Corey Thompson wrote: Hello, Has anyone actually gotten git-p4 to clone a large Perforce repository? Yes. I've cloned repos with a couple of Gig of files. I have one codebase in particular that gets to about 67%, then consistently gets get-fast-import (and often times a

Re: git-p4 out of memory for very large repository

2013-08-23 Thread Corey Thompson
On Fri, Aug 23, 2013 at 08:16:58AM +0100, Luke Diamand wrote: On 23/08/13 02:12, Corey Thompson wrote: Hello, Has anyone actually gotten git-p4 to clone a large Perforce repository? Yes. I've cloned repos with a couple of Gig of files. I have one codebase in particular that gets to

Re: git-p4 out of memory for very large repository

2013-08-23 Thread Corey Thompson
On Fri, Aug 23, 2013 at 07:48:56AM -0400, Corey Thompson wrote: Sorry, I guess I could have included more details in my original post. Since then, I have also made an attempt to clone another (slightly more recent) branch, and at last had success. So I see this does indeed work, it just seems

Re: git-p4 out of memory for very large repository

2013-08-23 Thread Luke Diamand
I think I've cloned files as large as that or larger. If you just want to clone this and move on, perhaps you just need a bit more memory? What's the size of your physical memory and swap partition? Per process memory limit? On 23 Aug 2013 12:59, Corey Thompson cmt...@gmail.com wrote: On

Re: git-p4 out of memory for very large repository

2013-08-23 Thread Corey Thompson
On Fri, Aug 23, 2013 at 08:42:44PM +0100, Luke Diamand wrote: I think I've cloned files as large as that or larger. If you just want to clone this and move on, perhaps you just need a bit more memory? What's the size of your physical memory and swap partition? Per process memory limit? The

git-p4 out of memory for very large repository

2013-08-22 Thread Corey Thompson
Hello, Has anyone actually gotten git-p4 to clone a large Perforce repository? I have one codebase in particular that gets to about 67%, then consistently gets get-fast-import (and often times a few other processes) killed by the OOM killer. I've found some patches out there that claim to