[PyKDE] [PATCH] reduce PyQt concatenated build time on SMP
Hi Phil, in order to saturate multiple CPUs on SMP systems, I've created the attached patch. The problem is, that the qt module consists of 228 single modules, and build of its concatenated module outlasts all others by a few minutes here. With it applied, I was able to reduce the rpm build time from 16:15 to 13.30 on my Dual P3/1000. Since it is created on top of my former patch, it will apply with offsets, if applied first. Thanks, Pete--- build.py.new 2003-03-24 11:40:11.0 +0100 +++ build.py 2003-03-24 12:11:55.0 +0100 @@ -1282,23 +1282,42 @@ srclist = buf[match.start(0):match.end(0)] srclist = srclist[13:-1] srclist = string.replace(srclist,\\\n\t,) srclist = string.split(srclist, ) +if len(srclist) 100: +# special case for huge modules: create two modules + +h = len(srclist)/2 +# Concatenate the files. +d = open(mname + huge2.cpp,w) + +for cppfile in srclist[:h]: +f = open(cppfile,r) +d.write(f.read()) +f.close() + +d.close() + +srclist = srclist[h:] + +# Replace the C++ file names in the project file. +buf = re.sub(pat,SOURCES = + mname + huge2.cpp + mname + huge.cpp\n,buf) +else: +# Replace the C++ file names in the project file. +buf = re.sub(pat,SOURCES = + mname + huge.cpp\n,buf) + # Concatenate the files. d = open(mname + huge.cpp,w) for cppfile in srclist: f = open(cppfile,r) d.write(f.read()) f.close() d.close() -# Replace the C++ file names in the project file. -buf = re.sub(pat,SOURCES = + mname + huge.cpp\n,buf) - f = open(pro + .new,w) f.write(buf) f.close() os.remove(pro)
Re: [PyKDE] [PATCH] reduce PyQt concatenated build time on SMP
On Monday 24 March 2003 11:53 am, Hans-Peter Jansen wrote: Hi Phil, in order to saturate multiple CPUs on SMP systems, I've created the attached patch. The problem is, that the qt module consists of 228 single modules, and build of its concatenated module outlasts all others by a few minutes here. With it applied, I was able to reduce the rpm build time from 16:15 to 13.30 on my Dual P3/1000. Since it is created on top of my former patch, it will apply with offsets, if applied first. But doesn't this slow it down for everybody with a UP machine? Phil ___ PyKDE mailing list[EMAIL PROTECTED] http://mats.gmd.de/mailman/listinfo/pykde
Re: [PyKDE] [PATCH] reduce PyQt concatenated build time on SMP
On Monday 24 March 2003 13:05, Phil Thompson wrote: On Monday 24 March 2003 11:53 am, Hans-Peter Jansen wrote: Hi Phil, in order to saturate multiple CPUs on SMP systems, I've created the attached patch. The problem is, that the qt module consists of 228 single modules, and build of its concatenated module outlasts all others by a few minutes here. With it applied, I was able to reduce the rpm build time from 16:15 to 13.30 on my Dual P3/1000. Since it is created on top of my former patch, it will apply with offsets, if applied first. But doesn't this slow it down for everybody with a UP machine? Phil Well, yes, I have to admit. But the performance hit is less then the improvement, aka 19:53 - 21:29 UP, AMD 2000+ (~8%), 16:15 - 13:30 (~20%). Would you accept a combined cleaned up patch with different lower case options for this and the lib stuff? Pete ___ PyKDE mailing list[EMAIL PROTECTED] http://mats.gmd.de/mailman/listinfo/pykde
Re: [PyKDE] [PATCH] reduce PyQt concatenated build time on SMP
On Mon, Mar 24, 2003 at 03:09:17PM +0100, Hans-Peter Jansen wrote: On Monday 24 March 2003 13:05, Phil Thompson wrote: On Monday 24 March 2003 11:53 am, Hans-Peter Jansen wrote: Hi Phil, in order to saturate multiple CPUs on SMP systems, I've created the attached patch. The problem is, that the qt module consists of 228 single modules, and build of its concatenated module outlasts all others by a few minutes here. With it applied, I was able to reduce the rpm build time from 16:15 to 13.30 on my Dual P3/1000. Since it is created on top of my former patch, it will apply with offsets, if applied first. But doesn't this slow it down for everybody with a UP machine? Phil Well, yes, I have to admit. But the performance hit is less then the improvement, aka 19:53 - 21:29 UP, AMD 2000+ (~8%), 16:15 - 13:30 (~20%). Would you accept a combined cleaned up patch with different lower case options for this and the lib stuff? If you are proposing this, why don't you generalize and code something like the -j switch (parallel build switch) for make. In this case there is no penalty for UP machines (by default one single huge.cpp), and the guys with 4 processors will feel happy, too :-) Gerard PS: I don't have 4 processors. ___ PyKDE mailing list[EMAIL PROTECTED] http://mats.gmd.de/mailman/listinfo/pykde
Re: [PyKDE] [PATCH] reduce PyQt concatenated build time on SMP
On Monday 24 March 2003 2:50 pm, Gerard Vermeulen wrote: On Mon, Mar 24, 2003 at 03:09:17PM +0100, Hans-Peter Jansen wrote: On Monday 24 March 2003 13:05, Phil Thompson wrote: On Monday 24 March 2003 11:53 am, Hans-Peter Jansen wrote: Hi Phil, in order to saturate multiple CPUs on SMP systems, I've created the attached patch. The problem is, that the qt module consists of 228 single modules, and build of its concatenated module outlasts all others by a few minutes here. With it applied, I was able to reduce the rpm build time from 16:15 to 13.30 on my Dual P3/1000. Since it is created on top of my former patch, it will apply with offsets, if applied first. But doesn't this slow it down for everybody with a UP machine? Phil Well, yes, I have to admit. But the performance hit is less then the improvement, aka 19:53 - 21:29 UP, AMD 2000+ (~8%), 16:15 - 13:30 (~20%). Would you accept a combined cleaned up patch with different lower case options for this and the lib stuff? If you are proposing this, why don't you generalize and code something like the -j switch (parallel build switch) for make. In this case there is no penalty for UP machines (by default one single huge.cpp), and the guys with 4 processors will feel happy, too :-) Yes - I'd be happy with this. It also helps those compilers (aka MSVC) that choke on such a large C++ file but would probably handle... python build.py -c -j 10 Phil ___ PyKDE mailing list[EMAIL PROTECTED] http://mats.gmd.de/mailman/listinfo/pykde
Re: [PyKDE] [PATCH] reduce PyQt concatenated build time on SMP
On Mon, Mar 24, 2003 at 02:58:30PM +, Phil Thompson wrote: Yes - I'd be happy with this. It also helps those compilers (aka MSVC) that choke on such a large C++ file but would probably handle... python build.py -c -j 10 I thought that MS C++ compiler featured header precompilation. In such case (I'm praising for G++ to have this soon - they say not before 3.4), -c shouldn't make a big difference at all. ___ PyKDE mailing list[EMAIL PROTECTED] http://mats.gmd.de/mailman/listinfo/pykde
Re: [PyKDE] [PATCH] reduce PyQt concatenated build time on SMP
On Monday 24 March 2003 15:58, Phil Thompson wrote: On Monday 24 March 2003 2:50 pm, Gerard Vermeulen wrote: If you are proposing this, why don't you generalize and code something like the -j switch (parallel build switch) for make. In this case there is no penalty for UP machines (by default one single huge.cpp), and the guys with 4 processors will feel happy, too :-) Yes - I'd be happy with this. It also helps those compilers (aka MSVC) that choke on such a large C++ file but would probably handle... python build.py -c -j 10 One remark: the semantics will differ: while make -j 10 would run 10 build processes in parallel, this command should split large modules into 10 pieces. Is this fine with you? Phil Pete ___ PyKDE mailing list[EMAIL PROTECTED] http://mats.gmd.de/mailman/listinfo/pykde
Re: [PyKDE] [PATCH] reduce PyQt concatenated build time on SMP
On Monday 24 March 2003 4:00 pm, Ricardo Javier Cardenes Medina wrote: On Mon, Mar 24, 2003 at 02:58:30PM +, Phil Thompson wrote: Yes - I'd be happy with this. It also helps those compilers (aka MSVC) that choke on such a large C++ file but would probably handle... python build.py -c -j 10 I thought that MS C++ compiler featured header precompilation. In such case (I'm praising for G++ to have this soon - they say not before 3.4), -c shouldn't make a big difference at all. Actually the time consuming part on Windows is the link. If anybody knows of any tricks to speed this up I'd be most grateful. Phil ___ PyKDE mailing list[EMAIL PROTECTED] http://mats.gmd.de/mailman/listinfo/pykde
