Re: [PyKDE] How to contribute to sip ?
Here is what we're using as a non-Python alternative... sed "s/ \* Generated by SIP.*/ \*/" -i tmp/*.h tmp/*.cpp diff -ruN src tmp | patch -d src Sadly: sed, diff and patch on solaris are a little different, so this solution is not quite as portable as it needs to be... Nigel ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
Experimenting here with diff and patch... diff -ruN out tmp | patch -d out We don't have any Python dependency in our build toolchain. Maybe I'll just add another step to strip out the lines with timestamps... Here is what we're using as a non-Python alternative... (We use Cygwin for building on win32/64) sed "s/ \* Generated by SIP.*/ \*/" -i tmp/*.h tmp/*.cpp diff -ruN src tmp | patch -d src Nigel ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
Experimenting here with diff and patch... diff -ruN out tmp | patch -d out However, there is one issue, perhaps SIP could provide an option _not_ to put timestamps in the wrapper code comments... :-) Why not use Gerard's script which handles this? We don't have any Python dependency in our build toolchain. Maybe I'll just add another step to strip out the lines with timestamps... Nigel ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
Phil Thompson <[EMAIL PROTECTED]> wrote: >> My experience is that GCC cannot handle PyQt3 single-file version and takes >> like 20 minutes to compile it, while I can compile PyQt3 single-file with >> MSVC in roughly 100 seconds. Are you saying that this scenario is going to >> change with PyQt4? > > Well it obviously depends on compiler versions... > > MSVC v6 - can't handle large files. > > MSVC v7 (ie. 2003) - my current Windows compiler, but I've never tried large > files. > > GCC 3.4.4 - my current Linux compiler, handles large file. > > GCC 4 - never tried it. > > What Windows compiler are you using? I'm using MSVC 7.1 (ie 2003). It can handle large files pretty well. As I said, I can compile PyQt3 single-file in roughly 1:20 minutes (linking included) without using more than 70-80Mb of memory (with optimizations enabled). I'm on a Athlon XP 2200 (can't remember what's that in Mhz, sorry :). Also, I can tell you that I never managed to finish a PyQt3 single-file compilation on "gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)" (FC4). I know there used to be a bug which exponentially slowed down compilation of PyKDE (so I suppose it's SIP-related), but that was fixed in gcc 4.0.3. Even with that patch, I still hit 512Mb memory limit, the computer starts swapping and compilation times go too high. -- Giovanni Bajo ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
On Tuesday 28 February 2006 09:18, Giovanni Bajo wrote: > Phil Thompson <[EMAIL PROTECTED]> wrote: > > With one source file (plus lots of header files) the whole of PyQt4 takes >> 2 minutes 50 seconds on my 2GHz Opteron. Microsoft compilers struggle with > > such large source files. > My experience is that GCC cannot handle PyQt3 single-file version and takes > like 20 minutes to compile it, while I can compile PyQt3 single-file with > MSVC in roughly 100 seconds. Are you saying that this scenario is going to > change with PyQt4? gcc 4.0.1 appears to have problems with large PyQt and PyKDE "single" files. A couple of users have reported here that gcc 4.0.3 doesn't have that problem. gcc 3.3 seems to fail with really large cpp files - the kdeui module won't compile and PyKDE always splits that in 2. All of the other files (incl PyQt) work. The other limitation is the amount of free memory available, as the concatenated files use a lot of RAM. Jim ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
But, only sip knows what files will be generated because it depends of the input sip file(s) content. Moreover, only sip knows the exact structure of the comments it adds at the begining of each file. This comments must not be taken into account when comparing files (it contains date). If the comment structure is changed in sip, the comparison algorithm must change. This is why I think that this modification has something to do with what sip does.On 2/28/06, Phil Thompson <[EMAIL PROTECTED] > wrote:On Tuesday 28 February 2006 3:10 pm, Yann Cointepas wrote:> Hi, >> What is the procedure to propose modification to sip ?You've just followed it. :)> I need a feature> that I am ready to program. I would like sip to have an option to generate > a C++ file only if it does not exists or if it is different from the> existing one. This would greatly improve compilation time (using make> dependencies for instance) when modifying a small part of a sip project. I > looked at the code of sip 4.3.2 and it seem that there is not so much work> to do. I am ready to:>> - Describe the changes I would like to do> - Implement these changes in whatever sip version is appropriate > - Give (for free) these changes to sip developpers>> Tell me if it can be useful for other people, otherwise I will keep using a> Python script that generate the sip files in a temporary directory before > copying only modified ones.I wouldn't accept this change as it doesn't really have anything to do withwhat SIP does (although I readily admit that some of SIP's existingfunctionality also falls into that category). My recommended solution would be to use some sort of wrapper as you arecurrently doing. You could always put it on the Wiki.Phil-- Yann CointepasTel: +33 1 69 86 78 52 CEA - SHFJ Fax: +33 1 69 86 77 864, place du General Leclerc91401 Orsay Cedex France ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
On 28.02.06 18:18:42, Giovanni Bajo wrote: > Phil Thompson <[EMAIL PROTECTED]> wrote: > > > With one source file (plus lots of header files) the whole of PyQt4 takes > 2 > > minutes 50 seconds on my 2GHz Opteron. Microsoft compilers struggle with > > such large source files. > > My experience is that GCC cannot handle PyQt3 single-file version and takes > like 20 minutes to compile it, It takes <10 minutes here (Pentium-M 1.4GHz) with 4 files and debug mode. The latter is important because else gcc takes up more memory than is installed on my system, not sure why though. > MSVC in roughly 100 seconds. Are you saying that this scenario is going to > change with PyQt4? PyQt4 is a bit faster than PyQt3 here and also (IIRC) doesn't have a problem when building without debug symbols. Andreas -- It was all so different before everything changed. ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
On Tuesday 28 February 2006 5:18 pm, Giovanni Bajo wrote: > Phil Thompson <[EMAIL PROTECTED]> wrote: > > With one source file (plus lots of header files) the whole of PyQt4 takes > > 2 > > > minutes 50 seconds on my 2GHz Opteron. Microsoft compilers struggle with > > such large source files. > > My experience is that GCC cannot handle PyQt3 single-file version and takes > like 20 minutes to compile it, while I can compile PyQt3 single-file with > MSVC in roughly 100 seconds. Are you saying that this scenario is going to > change with PyQt4? Well it obviously depends on compiler versions... MSVC v6 - can't handle large files. MSVC v7 (ie. 2003) - my current Windows compiler, but I've never tried large files. GCC 3.4.4 - my current Linux compiler, handles large file. GCC 4 - never tried it. What Windows compiler are you using? PyQt3 takes 4 minutes 40 seconds on my system. Phil ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
On Tuesday 28 February 2006 5:25 pm, Nigel Stewart wrote: > > A wrapper that generated the .sip files into a temporary directory, > > compared them against previously generated files and replaced them if > > they were different isn't very difficult to write. > > Phil, > > Experimenting here with diff and patch... > diff -ruN out tmp | patch -d out > > However, there is one issue, perhaps SIP could > provide an option _not_ to put timestamps in > the wrapper code comments... :-) > > /* > * Interface wrapper code. > * > * Generated by SIP 4.3.2 (4.3.2-SIP-4_3-365) on Tue Feb 28 11:23:21 2006 > */ Why not use Gerard's script which handles this? Phil ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
A wrapper that generated the .sip files into a temporary directory, compared them against previously generated files and replaced them if they were different isn't very difficult to write. Phil, Experimenting here with diff and patch... diff -ruN out tmp | patch -d out However, there is one issue, perhaps SIP could provide an option _not_ to put timestamps in the wrapper code comments... :-) /* * Interface wrapper code. * * Generated by SIP 4.3.2 (4.3.2-SIP-4_3-365) on Tue Feb 28 11:23:21 2006 */ Nigel ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
Phil Thompson <[EMAIL PROTECTED]> wrote: > With one source file (plus lots of header files) the whole of PyQt4 takes 2 > minutes 50 seconds on my 2GHz Opteron. Microsoft compilers struggle with > such large source files. My experience is that GCC cannot handle PyQt3 single-file version and takes like 20 minutes to compile it, while I can compile PyQt3 single-file with MSVC in roughly 100 seconds. Are you saying that this scenario is going to change with PyQt4? -- Giovanni Bajo ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
On Tuesday 28 February 2006 4:30 pm, Giovanni Bajo wrote: > Nigel Stewart <[EMAIL PROTECTED]> wrote: > >> What does this have to do with SIP again? I don't remember GCC not > >> generating the object files if the source code is not changed. > > > > A compiler takes one input file and creates one output file. > > Actually not really, since the input can also contain an arbitrary large > number of header files. Not that this matters for your point, though. > > > The granularity of SIP is different, it's taking multiple > > inputs (via %Import) and creating multiple outputs. This > > design poses a "choke point". I think being time-stamp > > friendly is a way for SIP to compensate for this. > > I see. Can't you call SIP once for each .sip file, and make it generate one > class at a time? Isn't that what %Import is for? No. The output of a SIP "run" is the source of a single Python module, implemented as a number of C++ source files. You have control over the number (and therefore size) of source files. With one source file (plus lots of header files) the whole of PyQt4 takes 2 minutes 50 seconds on my 2GHz Opteron. Microsoft compilers struggle with such large source files. Phil ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
>> uic and moc avoid touching files unnecessarily, and so could SIP. I wasn't aware of uic and mock behaving like that. True, it's qmake that is uic and moc dependency aware... A wrapper that generated the .sip files into a temporary directory, compared them against previously generated files and replaced them if they were different isn't very difficult to write. I'll have a fiddle in our gmake Makefile to try this suggestion... I wonder if diff can handle it... Nigel ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
On Tue, 28 Feb 2006 10:18:31 -0600
Nigel Stewart <[EMAIL PROTECTED]> wrote:
>
> >> It could be argued that SIP is part of a broader toolchain
> >
> > What does this have to do with SIP again? I don't remember GCC not
> > generating the object files if the source code is not changed.
>
> A compiler takes one input file and creates one output file.
>
> The granularity of SIP is different, it's taking multiple
> inputs (via %Import) and creating multiple outputs. This
> design poses a "choke point". I think being time-stamp
> friendly is a way for SIP to compensate for this.
>
> > So, why can't you configure your build system so that it doesn't call SIP in
> > the first place if the header file has not changed?
>
> Here is the scenario - the header foo.h is changed, triggering
> SIP to regenerate the bindings. In most cases, the change to
> foo.h doesn't affect the code that SIP outputs. Of the 250
> .cpp files generated by SIP, 10 depend on foo.h. However,
> all 250 are rebuilt due to SIP changing the time stamp.
>
> That's a 25x slowdown that provides me more time to surf the
> PyKDE mailing list... :-)
>
> Perhaps our situation is not typical, it doesn't affect our
> developers that _depend_ on our SIP bindings. But it certainly
> affects development _upstream_ of the SIP bindings.
>
> Imagine someone hacking away at Qt and depending on PyQt
> based regression tests to know if they've broken anything...
> That's my day-to-day situation in a nutshell...
>
My configure.py scripts put the the output of sip in a temporary
directory and they use the following function to copy the new files
to the build directory:
def lazy_copy_file(source, target):
'''Lazy copy a file to another file:
- check for a SIP time stamp to skip,
- check if source and target do really differ,
- copy the source file to the target if they do,
- return True on copy and False on no copy.
'''
if not os.path.exists(target):
shutil.copy2(source, target)
return True
sourcelines = open(source).readlines()
targetlines = open(target).readlines()
# global length check
if len(sourcelines) != len(targetlines):
shutil.copy2(source, target)
return True
# skip a SIP time stamp
if (len(sourcelines) > 3
and sourcelines[3].startswith(' * Generated by SIP')
):
line = 4
else:
line = 0
# line by line check
while line < len(sourcelines):
if sourcelines[line] != targetlines[line]:
shutil.copy2(source, target)
return True
line = line + 1
return False
# lazy_copy_file()
The function is invoked by code like this:
lazy_copies = 0
for pattern in ('*.c', '*.cpp', '*.h', '*.sbf'):
for source in glob.glob(os.path.join(tmp_dir, pattern)):
target = os.path.join(build_dir, os.path.basename(source))
if lazy_copy_file(source, target):
print "Copy %s -> %s." % (source, target)
lazy_copies += 1
print "%s file(s) lazily copied." % lazy_copies
If you are using g++, you should look into ccache.samba.org.
This gives additional speed-ups of about a factor 10.
Gerard
___
PyKDE mailing listPyKDE@mats.imk.fraunhofer.de
http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
Nigel Stewart <[EMAIL PROTECTED]> wrote: >> What does this have to do with SIP again? I don't remember GCC not >> generating the object files if the source code is not changed. > > A compiler takes one input file and creates one output file. Actually not really, since the input can also contain an arbitrary large number of header files. Not that this matters for your point, though. > The granularity of SIP is different, it's taking multiple > inputs (via %Import) and creating multiple outputs. This > design poses a "choke point". I think being time-stamp > friendly is a way for SIP to compensate for this. I see. Can't you call SIP once for each .sip file, and make it generate one class at a time? Isn't that what %Import is for? -- Giovanni Bajo ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
On Tuesday 28 February 2006 3:50 pm, Nigel Stewart wrote: > > I wouldn't accept this change as it doesn't really have anything to do > > with what SIP does (although I readily admit that some of SIP's existing > > functionality also falls into that category). > > Phil, > > It could be argued that SIP is part of a broader toolchain (in our case, > gmake and qmake) that would benefit from SIP taking some care not to > touch unchanged files. uic and moc avoid touching files unnecessarily, > and so could SIP. I wasn't aware of uic and mock behaving like that. If so it's a dumb thing to do. > Currently, our sip-based python bindings are the bottleneck in our > incremental builds. We bypass this as much as possible, (cheating the > dependencies) but we shouldn't need to be doing that kind of workaround... Agreed - but there is a right place to fix things, and this isn't it. A wrapper that generated the .sip files into a temporary directory, compared them against previously generated files and replaced them if they were different isn't very difficult to write. Phil ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
It could be argued that SIP is part of a broader toolchain What does this have to do with SIP again? I don't remember GCC not generating the object files if the source code is not changed. A compiler takes one input file and creates one output file. The granularity of SIP is different, it's taking multiple inputs (via %Import) and creating multiple outputs. This design poses a "choke point". I think being time-stamp friendly is a way for SIP to compensate for this. So, why can't you configure your build system so that it doesn't call SIP in the first place if the header file has not changed? Here is the scenario - the header foo.h is changed, triggering SIP to regenerate the bindings. In most cases, the change to foo.h doesn't affect the code that SIP outputs. Of the 250 .cpp files generated by SIP, 10 depend on foo.h. However, all 250 are rebuilt due to SIP changing the time stamp. That's a 25x slowdown that provides me more time to surf the PyKDE mailing list... :-) Perhaps our situation is not typical, it doesn't affect our developers that _depend_ on our SIP bindings. But it certainly affects development _upstream_ of the SIP bindings. Imagine someone hacking away at Qt and depending on PyQt based regression tests to know if they've broken anything... That's my day-to-day situation in a nutshell... Nigel ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
Nigel Stewart <[EMAIL PROTECTED]> wrote: >> I wouldn't accept this change as it doesn't really have anything to do with >> what SIP does (although I readily admit that some of SIP's existing >> functionality also falls into that category). > > Phil, > > It could be argued that SIP is part of a broader toolchain (in our case, > gmake and qmake) that would benefit from SIP taking some care not to > touch unchanged files. uic and moc avoid touching files unnecessarily, > and so could SIP. > > Currently, our sip-based python bindings are the bottleneck in our > incremental builds. We bypass this as much as possible, (cheating the > dependencies) but we shouldn't need to be doing that kind of workaround... What does this have to do with SIP again? I don't remember GCC not generating the object files if they source code is not changed. It's up to the build process to detect if the source file is changed and call the compiler (SIP, uic, gcc, ecc.) if necessary. Usually, that is done by checking the date of the last modification. More complex build system like scons use intelligent MD5 signatures (which ignore comments in source files and whatnot). So, why can't you configure your build system so that it doesn't call SIP in the first place if the header file has not changed? -- Giovanni Bajo ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
I wouldn't accept this change as it doesn't really have anything to do with what SIP does (although I readily admit that some of SIP's existing functionality also falls into that category). Phil, It could be argued that SIP is part of a broader toolchain (in our case, gmake and qmake) that would benefit from SIP taking some care not to touch unchanged files. uic and moc avoid touching files unnecessarily, and so could SIP. Currently, our sip-based python bindings are the bottleneck in our incremental builds. We bypass this as much as possible, (cheating the dependencies) but we shouldn't need to be doing that kind of workaround... Nigel ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
On Tuesday 28 February 2006 3:10 pm, Yann Cointepas wrote: > Hi, > > What is the procedure to propose modification to sip ? You've just followed it. :) > I need a feature > that I am ready to program. I would like sip to have an option to generate > a C++ file only if it does not exists or if it is different from the > existing one. This would greatly improve compilation time (using make > dependencies for instance) when modifying a small part of a sip project. I > looked at the code of sip 4.3.2 and it seem that there is not so much work > to do. I am ready to: > > - Describe the changes I would like to do > - Implement these changes in whatever sip version is appropriate > - Give (for free) these changes to sip developpers > > Tell me if it can be useful for other people, otherwise I will keep using a > Python script that generate the sip files in a temporary directory before > copying only modified ones. I wouldn't accept this change as it doesn't really have anything to do with what SIP does (although I readily admit that some of SIP's existing functionality also falls into that category). My recommended solution would be to use some sort of wrapper as you are currently doing. You could always put it on the Wiki. Phil ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] How to contribute to sip ?
> I would like sip to have an option to generate a C++ file only if it does not exists or if it is different from the existing one. This would greatly improve compilation time ... That would be very good indeed. Does it need to be optional? We would see some immediate benefits to our build time... Nigel ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
