Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-10 Thread Antoine Pitrou
Le Sat, 10 Sep 2011 05:02:09 +0200, Jesus Cea a écrit : > > I have committed a few patches in the last hours to get my buildbots > "green", back again. The memory used was <500MB, compared with >4GB > before the "-j". > > Could you reconfigure my buildbots to be able to run all the six (2.7, > 3

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-09 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/09/11 05:02, Jesus Cea wrote: > I have committed a few patches in the last hours to get my > buildbots "green", back again. The memory used was <500MB, compared > with >4GB before the "-j". One of my patches solves a "process leak" in multiproce

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-09 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/09/11 19:04, Antoine Pitrou wrote: >> On 08/09/11 09:18, Antoine Pitrou wrote: >>> Ok, I've added "-j4", let's how that works. >> >> It is not helping. it is taking tons of memory yet. > > That's rather strange. Is it for every test or a few se

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-09 Thread Antoine Pitrou
Le Fri, 09 Sep 2011 17:14:18 +0200, Jesus Cea a écrit : > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 08/09/11 09:18, Antoine Pitrou wrote: > > Ok, I've added "-j4", let's how that works. > > It is not helping. it is taking tons of memory yet. That's rather strange. Is it for every t

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-09 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/09/11 09:18, Antoine Pitrou wrote: > Ok, I've added "-j4", let's how that works. It is not helping. it is taking tons of memory yet. >> Another option would be to have a single Python process and >> "fork" for each test. That would launch each

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-09 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/09/11 09:18, Antoine Pitrou wrote: > Ok, I've added "-j4", let's how that works. It is not helping. it is taking tons of memory yet. >> Another option would be to have a single Python process and >> "fork" for each test. That would launch each

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-08 Thread Antoine Pitrou
Hello Jesus, > I sorry to bother you with these details > and waste of time, but could you possibly change my buildbot > configurarion to launch, let's say, 4 test processes in parallel, just > for testing? Ok, I've added "-j4", let's how that works. > Another option would be to have a single P

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-07 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/09/11 14:32, Antoine Pitrou wrote: > If "-j" is used, tests get run in a separate process each, so that > approach might be an answer. Antoine, I think this would be the answer. Each test would be a bit slower, because I would launch a new pyth

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-07 Thread Antoine Pitrou
On Wed, 07 Sep 2011 13:38:23 +0200 Jesus Cea wrote: > > So, the problem is that a) "make test" takes quite a bit of RAM and b) > the buildbot forks some "big" processes, so the virtual memory needed > is BIG. Note that buildbots run "make buildbottest", not "make test". > So I have two question

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-07 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/09/11 07:27, Nick Coghlan wrote: > It may be the case that with the reduced memory limit, your > machine may not be able to run concurrent slaves for 2.7, 3.2 and > 3.x as I believe it does now. Antoine has changed the buildmaster configuration

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-05 Thread Nick Coghlan
On Tue, Sep 6, 2011 at 3:19 PM, Jesus Cea wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 06/09/11 07:02, Jesus Cea wrote: >> On 06/09/11 06:59, Jesus Cea wrote: >>> Thanking for helping me with this. >> >> BTW, it is 7AM in Spain now. I am going bed. I will check this >> thread ag

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-05 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/09/11 07:02, Jesus Cea wrote: > On 06/09/11 06:59, Jesus Cea wrote: >> Thanking for helping me with this. > > BTW, it is 7AM in Spain now. I am going bed. I will check this > thread again tomorrow. Thanks for your time and effort. This is > very

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-05 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/09/11 06:59, Jesus Cea wrote: > Thanking for helping me with this. BTW, it is 7AM in Spain now. I am going bed. I will check this thread again tomorrow. Thanks for your time and effort. This is very frustrating, moreover because it was working v

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-05 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/09/11 06:46, Nick Coghlan wrote: > The server actually looks it may be in a generally unhappy state, > perhaps due to previous builds that failed without cleaning up > after themselves properly. How many python processes do you see > hanging aro

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-05 Thread Nick Coghlan
On Tue, Sep 6, 2011 at 2:27 PM, Jesus Cea wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 06/09/11 06:19, Nick Coghlan wrote: >> Is it possible your buildbot is set up to run the bigmem tests? >> IIRC, those would work correctly with 32 GB, but die a horrible >> death with only 4 G

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-05 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/09/11 06:19, Nick Coghlan wrote: > Is it possible your buildbot is set up to run the bigmem tests? > IIRC, those would work correctly with 32 GB, but die a horrible > death with only 4 GB available. How can I check that?. I am seen multiple pyt

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-05 Thread Nick Coghlan
On Tue, Sep 6, 2011 at 2:09 PM, Jesus Cea wrote: >> Can you tell if it's something recent or it has always been like >> that? > > I can't tell. My host has restricted me recently to 4GB RAM max (no > swap), and the buildbot is failing now, but I don't know if using so > much memory is something re

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-05 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/09/11 20:14, Antoine Pitrou wrote: > On Fri, 02 Sep 2011 19:53:37 +0200 Jesus Cea wrote: >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 >> >> On 02/09/11 17:57, Jesus Cea wrote: >>> The build hangs or die with a "out of memory" error, >>> eve

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-02 Thread Antoine Pitrou
On Fri, 02 Sep 2011 19:53:37 +0200 Jesus Cea wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 02/09/11 17:57, Jesus Cea wrote: > > The build hangs or die with a "out of memory" error, eventually. > > A simple "make test" with python not compiled with "pydebug" and > skipping all t

Re: [Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-02 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/09/11 17:57, Jesus Cea wrote: > The build hangs or die with a "out of memory" error, eventually. A simple "make test" with python not compiled with "pydebug" and skipping all the optional tests (like zip64) is taking up to 300MB of RAM. Python 2

[Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

2011-09-02 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 A single instance of buildbot in the OpenIndiana buildbot is eating 1.4GB of RAM and 3.8GB of SWAP and growing. The build hangs or die with a "out of memory" error, eventually. This is 100% reproducible. Everytime I force a build thru the buildbot co