Re: [review] [PyKDE] PyQT module size
Phil Thompson [EMAIL PROTECTED] wrote: I think I'm just going to go with the other users' recommendation of nabbing the latest sip and compiling with some different flags with GCC4 and see if that gets me some quick wins. My mind is too small and fragile to attempt the changes you proposed above without screwing things up or causing myself more time than its worth (Thanks just the same for the options though) Which is kind of how I view pyqt in general - A time saver. As I understand it, GCC4 won't give any improvement if you are using the latest SIP (although it would be nice to have this confirmed). Thanks to Ulli, modules now only export a single symbol - just using the latest SIP by itself should give you the improvements. Yeah, I had missed that. I assume this is being done with a linker script. The new features of GCC4 are handy if you need to export multiple C++ classes, and you don't want to mention hundreds of mangled names in a linker script (including vtables, etc.), or manually mark each and every member function and member variable with a GCC-specific __attribute__. But this is not the case here. -- Giovanni Bajo ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [review] [PyKDE] PyQT module size
I think I'm just going to go with the other users' recommendation of nabbing the latest sip and compiling with some different flags with GCC4 and see if that gets me some quick wins. My mind is too small and fragile to attempt the changes you proposed above without screwing things up or causing myself more time than its worth (Thanks just the same for the options though) Which is kind of how I view pyqt in general - A time saver. As I understand it, GCC4 won't give any improvement if you are using the latest SIP (although it would be nice to have this confirmed). Thanks to Ulli, modules now only export a single symbol - just using the latest SIP by itself should give you the improvements. I do have one last question for you though. One user seemed to purport, and I haven't tested hard enough yet to confirm it, that when two python strings are concatenated: x = x + y That a new string is created, leaving the old reference to x in memory. With python's garbage collection as it is, this doesn't seem to be correct, but I wanted to ensure there wasn't some special way of handling cleanup that I need to take into account. Often you find that your are keeping an extra reference elsewhere. Use sys.getrefcount() to check. Phil ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
C++ data (Re: [PyKDE] PyQT module size)
Hello,
I wrote a version of mem.py in C++, and have put the data side by side in a
spreadsheet. It appears that a PyQt program more overhead has in terms of
libraries over C++, roughly 8Mb in this test for me. When running, the memory
usage pattern seems to mirror the C++ program with perhaps a little bit of
overhead per object, ~10% (ballpark figure).
What is needed now is some tests that create lots of Qt objects, and then see
how object creation affects memory usage.
Using Mandrake Linux 10.1, Qt-3.3.3, SIP 4.0.1 (I think, from kdebindings
3.3.1)
cheers,
--
Simon Edwards | Guarddog Firewall
[EMAIL PROTECTED] | http://www.simonzone.com/software/
Nijmegen, The Netherlands | ZooTV? You made the right choice.
PyQt_memory_usage.xls
Description: application/msexcel
/*
*/
#include qapplication.h
#include qstringlist.h
#include qpushbutton.h
#include qfile.h
void ProcMem() {
QStringList lines;
QFile file(/proc/self/status);
if(file.open( IO_ReadOnly ) ) {
QTextStream stream( file );
QString line;
while ( !stream.atEnd() ) {
line = stream.readLine(); // line of text excluding '\n
if(line.startsWith(V)) {
printf( %s\n, line.latin1() );
}
lines += line;
}
file.close();
}
}
int main( int argc, char ** argv ) {
printf(\nInitial values\n);
ProcMem();
printf(\nCreate QApplication\n);
QApplication *qapp = new QApplication( argc, argv );
ProcMem();
printf(\nCreate Pushbutton\n);
QPushButton *b = new QPushButton(Hello World, 0);
qapp-setMainWidget(b);
ProcMem();
printf(\nShow Pushbutton\n);
b-show();
ProcMem();
printf(\nConnect Pushbutton signal\n);
qapp-connect(b, SIGNAL(clicked()), qapp, SLOT(quit()));
ProcMem();
printf(Enter Mainloop\n);
qapp-exec();
ProcMem();
printf(Mainloop finished\n);
ProcMem();
printf(Delete Pushbutton\n);
delete b;
ProcMem();
printf(Delete Application\n);
delete qapp;
ProcMem();
return 0;
}
___
PyKDE mailing listPyKDE@mats.imk.fraunhofer.de
http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
[PyKDE] PyQT module size
We've been using PyQT/PyKDE here at Linspire for some time now to develop a handful of applications. One of these is Lsongs, and it has grown quite large and has a very large memory footprint. We're trying to reduce this a bit in various ways (switching to gstreamer from libxine for instance) and I was curious if there was a way to reduce the footprint size of pyqt/pykde. The shared object files themselves are quite large - Larger than their C++ counterparts. I'm totally ignorant on python bindings so this may just be the way it is and that's fine. Just curious if there's a way to shrink these down a bit. Thanks, -Brian -- This message contains information which may be confidential and privileged. Unless you are the addressee (or authorized to receive for the addressee), you may not use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender and delete the message. Thank you. ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] PyQT module size
On Thu, 2005-02-17 at 11:47 -0800, Brian Thomason wrote: We've been using PyQT/PyKDE here at Linspire for some time now to develop a handful of applications. One of these is Lsongs, and it has grown quite large and has a very large memory footprint. We're trying to reduce this a bit in various ways (switching to gstreamer from libxine for instance) and I was curious if there was a way to reduce the footprint size of pyqt/pykde. The shared object files themselves are quite large - Larger than their C++ counterparts. I fought this battle while at a previous position (admittedly on windows, not linux). The only solution I could find was to compile QT with the absolute minimum set of classes required for my application. This got QT down to about 2Megs and PyQT down to about 3.5 or so. I can't remember the exact figures, but it doesn't matter since you are using a different compiler. I do recall that it was more than half for me, but I wasn't using several large modules. The other horrible hack you can do to change this is to manually edit the main sip file and remove (or comment out) the includes for the classes you don't need. This gives you a fully functional QT, but a hobbled pyqt. (And then I would suggest changing the name of qt.pyd to linQt.pyd or some such so that users can install their own full version of py qt) We also hit memory foot print issues, and the first thing I will say is watch your python strings! string A + string B creates a new string, so concatenating using the + operator uses a lot more memory than you think. Once you've done that, look on the web for options on optimizing memory with python. There are plenty of resources. If you are still having memory problems, the only thing I can suggest (and this is what we did), is explictly delete as many variables as you can. This also can help. If you get _really_ desperate, you can force garbage collection in a seperate (python) thread, but that lead to some stability issues for me. Finally, if you really are at the end of the loop, if there is a call like win32's SetWorkingSetSize that writes unused objects to virtual memory, you can sprinkle that at strategic locations or even in a background loop. (Probably need to expose whatever call it would be.) Of course there is a pretty large performance hit here, but it can be very effective in freeing up physical ram for other apps. It also isn't cross-platform, but considering that you work for a linux company, I doubt that is the end of the world. :) In my situation (a video conferencing suite), this was very useful as my UI application's performance was virtually irrelevant as along as the video and networking apps were running well. (long story, everything was in a seperate app so if one part failed, the conference could continue). None of this is probably what you want to hear. AFAICT, there is no magic bullet that shrinks PyQTs memory down. What I can say if applying everything except the SetWorkingSetSize, we went from about 50 Megs while in conference to about 20 Megs while in conference. We could have saved more in a couple of areas specific to our application (the skinning engine, and our interactions with zip files), but money ran out before we could do that. I'm totally ignorant on python bindings so this may just be the way it is and that's fine. Just curious if there's a way to shrink these down a bit. Thanks, -Brian -- This message contains information which may be confidential and privileged. Unless you are the addressee (or authorized to receive for the addressee), you may not use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender and delete the message. Thank you. ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde -- --- Rob Myddrin Knapp [EMAIL PROTECTED] http://www.myddrin.com ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] PyQT module size
Brian Thomason [EMAIL PROTECTED] wrote: We've been using PyQT/PyKDE here at Linspire for some time now to develop a handful of applications. One of these is Lsongs, and it has grown quite large and has a very large memory footprint. We're trying to reduce this a bit in various ways (switching to gstreamer from libxine for instance) and I was curious if there was a way to reduce the footprint size of pyqt/pykde. The shared object files themselves are quite large - Larger than their C++ counterparts. I'm totally ignorant on python bindings so this may just be the way it is and that's fine. Just curious if there's a way to shrink these down a bit. One low-hanging fruit could be recompiling them through the upcoming GCC 4 using the new -fvisibility=hidden -fvisibility-inline-hidden options. By making only the needed symbols visibile (which AFAIK it is just one symbol for a Python extension module), the shared object size will reduce somehow, will be much much faster to load and will also be optimized better. See http://gcc.gnu.org/gcc-4.0/changes.html and http://www.nedprod.com/programs/gccvisibility.html for further reference. I'm sure there is already some infrastructure for this, as the windows version of qt.pyd exports just one visible symbol. -- Giovanni Bajo ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] PyQT module size
On Thursday 17 February 2005 20:47, Brian Thomason wrote: We've been using PyQT/PyKDE here at Linspire for some time now to develop a handful of applications. One of these is Lsongs, and it has grown quite large and has a very large memory footprint. We're trying to reduce this a bit in various ways (switching to gstreamer from libxine for instance) and I was curious if there was a way to reduce the footprint size of pyqt/pykde. The shared object files themselves are quite large - Larger than their C++ counterparts. Since this is an actual concern of mine, too, I ran some tests with the attached script, mainly to explore the impact of importing: from qt import Q... versus from qt import * To conclude, it is quite small, but see yourself: On SuSE 9.0, 2.4.21-273-athlon: sip version: 4.1.1 Qt version: 3.2.1 PyQt version: 3.13 the difference is about 80k :-( VmSize:63072 kB VmLck: 0 kB VmRSS: 16152 kB VmData: 4380 kB VmStk:44 kB VmExe: 4 kB VmLib: 21640 kB VmSize:63108 kB VmLck: 0 kB VmRSS: 16236 kB VmData: 4376 kB VmStk:44 kB VmExe: 4 kB VmLib: 21676 kB On SuSE 9.2, 2.6.8-24.11: sip version: 4.0.1 Qt version: 3.3.3 PyQt version: 3.12 (all from kdebindings3-python-3.3.0-13): VmSize:26216 kB VmLck: 0 kB VmRSS: 14148 kB VmData: 4336 kB VmStk:56 kB VmExe: 2 kB VmLib: 19606 kB VmSize:26352 kB VmLck: 0 kB VmRSS: 14156 kB VmData: 4472 kB VmStk:56 kB VmExe: 2 kB VmLib: 19606 kB If some kind soul could explain the huge difference in VmSize between 2.4 and 2.6 from /proc/pid/status, I would feel much better.. Pete import sys if len(sys.argv) 1 and sys.argv[1] == '-a': from qt import * else: from qt import QApplication, QPushButton, SIGNAL, qApp a = QApplication(sys.argv) b = QPushButton(Hello World, None) a.setMainWidget(b) b.show() a.connect(b, SIGNAL(clicked()), qApp.quit) a.exec_loop() ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
Re: [PyKDE] PyQT module size
On Friday 18 February 2005 18:00, Hans-Peter Jansen wrote:
and here's a more detailed comparison from the two:
9.0: 9.2:
sip version: 4.1.1 sip version: 4.0.1
Qt version: 3.2.1 Qt version: 3.3.3
PyQt version: 3.13 PyQt version: 3.12
Python: 2.3Python: 2.3.4
Initial Values Initial Values
VmSize: 4928 kBVmSize: 4276 kB
VmData: 1404 kBVmData: 852 kB
VmRSS: 2772 kBVmRSS: 2508 kB
VmLib: 2984 kBVmLib: 2886 kB
Import qt Import qt
VmSize:22464 kB17536 kBVmSize:21652 kB17376 kB
VmData: 1584 kB 180 kBVmData: 1200 kB 348 kB
VmRSS: 9336 kB 6564 kBVmRSS: 8252 kB 5744 kB
VmLib: 19552 kB16568 kBVmLib: 19190 kB16304 kB
Create Application Create Application
VmSize:60972 kB38508 kBVmSize:23912 kB 2260 kB
VmData: 2692 kB 1108 kBVmData: 2492 kB 1292 kB
VmRSS: 13656 kB 4320 kBVmRSS: 11020 kB 2768 kB
VmLib: 21252 kB 1700 kBVmLib: 19230 kB 40 kB
Create Pushbutton Create Pushbutton
VmSize:60972 kB0 kBVmSize:23904 kB -8 kB
VmData: 2692 kB0 kBVmData: 2484 kB -8 kB
VmRSS: 13824 kB 168 kBVmRSS: 11188 kB 168 kB
VmLib: 21252 kB0 kBVmLib: 19230 kB0 kB
Show PushbuttonShow Pushbutton
VmSize:62964 kB 1992 kBVmSize:26356 kB 2452 kB
VmData: 4272 kB 1580 kBVmData: 4476 kB 1992 kB
VmRSS: 15896 kB 2072 kBVmRSS: 13836 kB 2648 kB
VmLib: 21640 kB 388 kBVmLib: 19606 kB 376 kB
Connect Pushbutton signal Connect Pushbutton signal
VmSize:62964 kB0 kBVmSize:26356 kB0 kB
VmData: 4272 kB0 kBVmData: 4476 kB0 kB
VmRSS: 15924 kB 28 kBVmRSS: 13860 kB 24 kB
VmLib: 21640 kB0 kBVmLib: 19606 kB0 kB
Enter Mainloop Enter Mainloop
VmSize:63136 kB 172 kBVmSize:26356 kB0 kB
VmData: 4404 kB 132 kBVmData: 4476 kB0 kB
VmRSS: 16260 kB 336 kBVmRSS: 14196 kB 336 kB
VmLib: 21676 kB 36 kBVmLib: 19606 kB0 kB
Mainloop finished Mainloop finished
VmSize:63136 kB0 kBVmSize:26356 kB0 kB
VmData: 4404 kB0 kBVmData: 4476 kB0 kB
VmRSS: 16260 kB0 kBVmRSS: 14196 kB0 kB
VmLib: 21676 kB0 kBVmLib: 19606 kB0 kB
Delete Pushbutton Delete Pushbutton
VmSize:63128 kB -8 kBVmSize:26356 kB0 kB
VmData: 4396 kB -8 kBVmData: 4476 kB0 kB
VmRSS: 16260 kB0 kBVmRSS: 14200 kB4 kB
VmLib: 21676 kB0 kBVmLib: 19606 kB0 kB
Delete Application Delete Application
VmSize:38132 kB -24996 kBVmSize:25688 kB -668 kB
VmData: 4396 kB0 kBVmData: 4268 kB -208 kB
VmRSS: 16028 kB -232 kBVmRSS: 13848 kB -352 kB
VmLib: 21288 kB -388 kBVmLib: 19230 kB -376 kB
Funny are the tremendous movements in QApplication ctor/dtor on the
left.
Pete
#! /usr/bin/env python
import sys, os
class ProcMem:
def __init__(self):
self.pid = os.getpid()
self.initial = self.getValues()
self.showValues(self.initial)
def getValues(self):
val = {}
fp = open(/proc/%s/status % self.pid, r)
for l in fp:
if l.startswith(Vm):
k, v = l.split()[:2]
if k in (VmData:;, VmLib:, VmRSS:, VmSize:):
val[k] = int(v)
return val
def showValues(self, val, val2 = None):
for k in VmSize:, VmData:;, VmRSS:, VmLib::
if val2:
print %-7s %8s kB %8s kB % (k, val[k], val[k]-val2[k])
else:
print %-7s %8s kB % (k, val[k])
def diffValues(self):
val = self.getValues()
self.showValues(val, self.initial)
self.initial = val
print Initial Values
pm = ProcMem()
print Import qt
import qt
pm.diffValues()
print Create Application
a = qt.QApplication(sys.argv)
pm.diffValues()
print Create Pushbutton
b = qt.QPushButton(Hello World, None)
a.setMainWidget(b)
pm.diffValues()
print Show Pushbutton
b.show()
pm.diffValues()
print Connect Pushbutton signal
a.connect(b, qt.SIGNAL(clicked()), qt.qApp.quit)
pm.diffValues()
print Enter Mainloop
a.exec_loop()
pm.diffValues()
print Mainloop finished
pm.diffValues()
print Delete Pushbutton
del b
Re: [review] [PyKDE] PyQT module size
On Friday 18 February 2005 06:38, Brian Thomason wrote: We've been using PyQT/PyKDE here at Linspire for some time now to develop a handful of applications. One of these is Lsongs, and it has grown quite large and has a very large memory footprint. We're trying to reduce this a bit in various ways (switching to gstreamer from libxine for instance) and I was curious if there was a way to reduce the footprint size of pyqt/pykde. The shared object files themselves are quite large - Larger than their C++ counterparts. I'm totally ignorant on python bindings so this may just be the way it is and that's fine. Just curious if there's a way to shrink these down a bit. Short answer: no. Long answer: I agree that if you look at the module sizes as they sit on disk, it looks pretty horrendous. A simple KDE plugin program written using PyKDE looks like it will use something in the 15-20MB range if you add up the .so sizes. In reality though, the actual memory usage, especially since KDE is already running and its and Qt's libs are already loaded, isn't all that bad. I seem to recall the actual overhead for a fairly large program using several PyQt modules and 5 or 6 PyKDE modules is somewhere around 6-8MB according to 'free'. Another way to look at it is that the *first* app has a lot of overhead, but each successive PyQt/PyKDE app has very little - use them more :) Still another way to view it is that getting to QApplication or KApplication is very expensive, but everything after that is really cheap. Most of the code is pretty much a fixed amount of overhead per method and that's written very efficiently by sip. The tradeoff then becomes size of code vs. completeness of code - ie, the only way to make PyQt or PyKDE smaller is by throwing out methods and classes. Throwing out methods is possible; throwing out classes can eventually become a dependency nightmare. In either case, it's pretty hard to take a general purpose development system and predict what users will find useful or necessary down the road, so the philosophy (at least for PyKDE) has been to include everything that can possibly be included. Another solution would be to make more and smaller modules, but then it becomes a problem for users to figure out which module contains which classes and methods (and it's not an easy task for us either). I've discussed this in the past, and the conclusion is usually well, nobody ever complains about it - this is literally (to my knowledge) the first complaint we've ever had. It's a question of how much you're willing to pay for the convenience of writing Qt/KDE code in Python, and it seems like the market price in KB is pretty high (and memory is pretty cheap). There are ways that a custom set of bindings could be developed for your applications that only expose the methods/classes from PyQt/PyKDE that you really need. It would involve writing a custom set of C++ wrappers for the PyQt/PyKDE APIs and then binding those. Depending on what you need, that could be as small as a few hundred K for everything. For an application of any complexity (eg menus/toolbar/statusbar and a selection of various widgets, std file dialogs, etc) it might turn out to be quite a bit of work though, and the resulting names wouldn't look like Qt/KDE anymore. You could also try taking the PyQt/PyKDE and throwing stuff out - that might save 30% or so, depending on what you need to leave in. However, when you do a menu using actions (esp standard actions) in KDE, for example, there's a lot of underlying code that you never see that PyKDE needs to bind (that's what a wrapper can get rid of). Lastly, it seems to me there's probably a different approach that could access the Qt/KDE C++ libs without the per method overhead and be quite small, but it's likely to execute considerably slower and be pretty complex, esp when you get into things like virtual method overloading or mapping types between Python and C/C++. The latter might even destroy the initial size advantage. I'm also not sure if that would be possible if the symbols are stripped from the .so libs being accessed - don't know enough about .so structure to really say. Jim ___ PyKDE mailing listPyKDE@mats.imk.fraunhofer.de http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
