Re: [HACKERS] build environment: a different makefile
Peter, finally I had a chance to check it out. One word: perfect! Thanks Paul On 25-feb-2008, at 19:09, Peter Eisentraut wrote: Am Mittwoch, 6. Februar 2008 schrieb Paul van den Bogaard: I was hoping someone in the community already has a makefile that just creates object files from C-sources directly that I can use to try out the effect of in-lining to the performance of postgres. This is now the default in 8.4devel. Let us know what you find out. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings - Paul van den Bogaard [EMAIL PROTECTED] ISV-E -- ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone:+31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax:+31 334 515 001 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build environment: a different makefile
Am Mittwoch, 6. Februar 2008 schrieb Paul van den Bogaard: I was hoping someone in the community already has a makefile that just creates object files from C-sources directly that I can use to try out the effect of in-lining to the performance of postgres. This is now the default in 8.4devel. Let us know what you find out. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] build environment: a different makefile
On Feb 6, 2008 11:12 PM, Tom Lane [EMAIL PROTECTED] wrote: I've sometimes wondered whether the SUBSYS.o files really offer any advantage compared to just linking all the individual .o files. They certainly eat disk space, but perhaps they save some time ... or perhaps not, especially in a one-off build. Getting rid of them would certainly make building OSX universal binaries easier. I suppose that we might fall foul of command line length limits on some platforms :-(. The output of your find command amounts to nearly 11000 characters in HEAD. What do other large build systems do? /D ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] build environment: a different makefile
On Thu, Feb 07, 2008 at 08:09:24AM +, Dave Page wrote: On Feb 6, 2008 11:12 PM, Tom Lane [EMAIL PROTECTED] wrote: I've sometimes wondered whether the SUBSYS.o files really offer any advantage compared to just linking all the individual .o files. They certainly eat disk space, but perhaps they save some time ... or perhaps not, especially in a one-off build. Getting rid of them would certainly make building OSX universal binaries easier. I suppose that we might fall foul of command line length limits on some platforms :-(. The output of your find command amounts to nearly 11000 characters in HEAD. What do other large build systems do? FWIW, the MSVC build ends up writing the list of object files to a temp file and then having the linker read that list. (This is all done behind the scenes though, it's not something we made up) IIRC the gcc linker can also take the commandline from a file instead of the actual commandline, which should be workable I think. //Magnus ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] build environment: a different makefile
Peter Eisentraut napsal(a): Paul van den Bogaard wrote: The SunStudio compiler we are using fortunately has an option for this. Unfortunately there are restrictions. One restriction I face is its inability to deal with ld -rs. These are used in the build environment to create all the SUBSYS.o object files. I was hoping someone in the community already has a makefile that just creates object files from C-sources directly that I can use to try out the effect of in-lining to the performance of postgres. I don't know if anyone has a makefile for it, but the following seems to work for me: pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/pg83/pg-install/lib' -Wl,-E $(find -name *.o | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres If you find that the optimizations you are hoping for are useful, I'm sure we could put an option of that sort somewhere in the makefiles. Peter, Suns studio performs inline optimization on -xO3 level. Optimization levels are different from GCC. Maximal level is -xO5. I think Paul plays with xipo flag which requires at least -xO4. See http://docs.sun.com/app/docs/doc/819-5265/bjapp?a=view Zdenek ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] build environment: a different makefile
Am Donnerstag, 7. Februar 2008 schrieb Dave Page: What do other large build systems do? I strongly suspect that they just fail unless run with GNU tools or some other well-defined tool set. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] build environment: a different makefile
Zdenek is right. Normal inlining using -O3 or higher means inlining within the source file. I currently try to see the effect of inlining over all the sources. The -xipo flag is specific to the Sun Studio (version 12) suite. It creates large objects that now include meta data for the final linking stage. During this linking stage all this stuff is read, analyzed an the object files are changed. Later this can be expanded by compiler profiling and/or other trickery. BTW the actual binary does *not* include all that metadata and is just bigger due to all that code being inlined in those objects. This trick has been done for other applications and we find in general significant approvement. Not a guarantee though ... However my suite has a dependecy on dtrace so the initial idea I saw from Peter is not complete. Currently creating a quick and dirty build script. Just to get that ipo-ing done and tested. I'll keep you all updated on the results. If we decide it is too good to be excluded can we start thinking about an adaption of the build environement. Hope this sound fair and acceptable to all. Thanks so far for all that feedback Cheers Paul BTW I build and test on a 16 SPARC @1350MHz V890. 64 bit mode, currently 16GB of shared memory, and 7 disk arrays (RAID0). This to exclude the IO part of the equation as much as possible, so we can focus on CPU related matter: the need for faster (smarter) code pathes and/or scalability issues like ProcArrayLock contention. On 7-feb-2008, at 11:08, Zdenek Kotala wrote: Peter Eisentraut napsal(a): Paul van den Bogaard wrote: The SunStudio compiler we are using fortunately has an option for this. Unfortunately there are restrictions. One restriction I face is its inability to deal with ld -rs. These are used in the build environment to create all the SUBSYS.o object files. I was hoping someone in the community already has a makefile that just creates object files from C-sources directly that I can use to try out the effect of in-lining to the performance of postgres. I don't know if anyone has a makefile for it, but the following seems to work for me: pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer- arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno- strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/ pg83/pg-install/lib' -Wl,-E $(find -name *.o | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/ port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto - lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres If you find that the optimizations you are hoping for are useful, I'm sure we could put an option of that sort somewhere in the makefiles. Peter, Suns studio performs inline optimization on -xO3 level. Optimization levels are different from GCC. Maximal level is -xO5. I think Paul plays with xipo flag which requires at least -xO4. See http://docs.sun.com/app/docs/doc/819-5265/bjapp?a=view Zdenek ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster - Paul van den Bogaard [EMAIL PROTECTED] ISV-E -- ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone:+31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax:+31 334 515 001 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] build environment: a different makefile
Magnus Hagander [EMAIL PROTECTED] writes: On Thu, Feb 07, 2008 at 08:09:24AM +, Dave Page wrote: What do other large build systems do? FWIW, the MSVC build ends up writing the list of object files to a temp file and then having the linker read that list. (This is all done behind the scenes though, it's not something we made up) IIRC the gcc linker can also take the commandline from a file instead of the actual commandline, which should be workable I think. Hmm. I'm not sure that's universal, but if it is then a simple incremental improvement on what we're doing now would be to replace the SUBSYS.o files with subsys include files that just list all the .o files to be included. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] build environment: a different makefile
Currently trying to enhance the way we can make binaries that run on Solaris. One thing I found was a scalability bottleneck in the use of the ProcArrayLock. (this one has also been reported by a couple of my colleagues). One big user of this lock is GetSnapshotData. After it has taken this lock it does its work and releases it again. While it is holding the lock it is not doing any system calls and the lock holding process is barely preempted. The only way to make this code faster is making the code use less CPU cycles to achieve its goal. One way is having the compiler do some strong code in-lining. The SunStudio compiler we are using fortunately has an option for this. Unfortunately there are restrictions. One restriction I face is its inability to deal with ld -rs. These are used in the build environment to create all the SUBSYS.o object files. I was hoping someone in the community already has a makefile that just creates object files from C-sources directly that I can use to try out the effect of in-lining to the performance of postgres. Any other hints to achieve my goal are welcome too, of-course. Please note that in-lining is done in both the compiler and the linker. Thanks, Paul - Paul van den Bogaard [EMAIL PROTECTED] ISV-E -- ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone:+31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax:+31 334 515 001 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] build environment: a different makefile
Paul van den Bogaard wrote: The SunStudio compiler we are using fortunately has an option for this. Unfortunately there are restrictions. One restriction I face is its inability to deal with ld -rs. These are used in the build environment to create all the SUBSYS.o object files. I was hoping someone in the community already has a makefile that just creates object files from C-sources directly that I can use to try out the effect of in-lining to the performance of postgres. I don't know if anyone has a makefile for it, but the following seems to work for me: pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/pg83/pg-install/lib' -Wl,-E $(find -name *.o | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres If you find that the optimizations you are hoping for are useful, I'm sure we could put an option of that sort somewhere in the makefiles. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] build environment: a different makefile
Peter Eisentraut [EMAIL PROTECTED] writes: I don't know if anyone has a makefile for it, but the following seems to work for me: pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/pg83/pg-install/lib' -Wl,-E $(find -name *.o | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres If you find that the optimizations you are hoping for are useful, I'm sure we could put an option of that sort somewhere in the makefiles. I've sometimes wondered whether the SUBSYS.o files really offer any advantage compared to just linking all the individual .o files. They certainly eat disk space, but perhaps they save some time ... or perhaps not, especially in a one-off build. I suppose that we might fall foul of command line length limits on some platforms :-(. The output of your find command amounts to nearly 11000 characters in HEAD. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
