Changing optimization level to -O1 with Clang/LLVM, was: Fun with LevelDB, LLVM and OpenCL
Hi, I've downgraded the optimization level from -O2 to -O1 with Clang/LLVM as -O2 seems to generate non-working code, in particular in kernel/gc.hpp:gc_current_pool(). For some reason the memory pool is not set correctly in that function with -O2. I suspect some pointer aliasing issue confusing the optimizer (as if I split that 10-line function into several smaller functions using additional variables for intermediate results then the generated code works) but for now feel safer to go with -O1. -- Jean-Sebastien On Tue, Sep 6, 2011 at 9:40 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Mon, Sep 5, 2011 at 10:24 PM, dsh daniel.hais...@googlemail.com wrote: Have to look at it during the weekend again or so. Didn't have much time recently cause I had to focus on other stuff. I think I will disable each broken dependency in the FreeBSD port and thus build Tuscany only with the proposed set of minimal dependency for now. Cheers Daniel Sounds good! -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
On Sun, Aug 14, 2011 at 2:28 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Thu, Aug 11, 2011 at 8:14 PM, dsh daniel.hais...@googlemail.com wrote: ... My attempt to provide an overview what works on FreeBSD (8.2) and what not: * autoconf, automake, libtool, doxygen, gcc can be installed from the FreeBSD ports collection * at the time there's only Apache HTTPD 2.2 available in the FreeBSD ports collection. I started creating an additional Apache HTTPD 2.3 port [0] but did not finish it yet * APR 2 - I sent an problem report [1] including a new port for the most recent APR 2 version * OpenSSL can be used from the FreeBSD base system * pcre-8.12 can be used from the FreeBSD ports collection * expat-2.0.1_1 can be used from the FreeBSD ports collection * memcached-1.4.5 can be installed from the FreeBSD ports collection * libevent-1.4.14b_2 can be installed from the FreeBSD ports collection * libxml2-2.7.8_1 can be installed from the FreeBSD ports collection * curl-7.21.3_2 can be installed from the FreeBSD ports collection (including SSL support) * I am currently in the process of upgrading the existing FreeBSD SpiderMonkey 1.7.x port [2], [3] * tinycdb-0.77 is available from the FreeBSD ports collection * ap20-mod_security-2.5.13 is available from the FreeBSD ports collection (not sure whether this one is too old) * I started to create an Apache Axis2/C port for FreeBSD but hit an issue which may be a bug [4] * I started to create an Apache Qpid port for FreeBSD put hit an issue which may be a bug [5], [6] * ap20-mod_auth_openid-0.5 is available from the FreeBSD ports collection including support for libopkele-2.0.2_1 * liboauth-0.9.4 is available from the FreeBSD ports collection * I think Apache Vysper can be installed relatively easy from the binary distribution and thus I decided to not create a separate FreeBSD port * Didn't create a FreeBSD port for Libstrophe yet cause I don't know how to pull sources directly from Github while building a FreeBSD port * postgresql-client-8.4.8 is available from the FreeBSD ports collection * scribe-2.2_3 is available from the FreeBSD ports collection * thrift-0.4.0,1 is available from the FreeBSD ports collection * I created and submitted a new FreeBSD port for Apache Libcloud using a PR [7] [0] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=284897+0+current/freebsd-ports [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=159658 [2] http://people.apache.org/~dsh/projects/fbsd-ports/spidermonkey185/ [3] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=343488+0+current/freebsd-ports [4] http://mail-archives.apache.org/mod_mbox/axis-c-user/201108.mbox/%3ccac0wh9akzrvc3-qzhg3thmfhnvi20kzerb0zp94hq_x+d_f...@mail.gmail.com%3E [5] https://issues.apache.org/jira/browse/QPID-2549 [6] http://mail-archives.apache.org/mod_mbox/qpid-dev/201108.mbox/%3CCAC0wh9ZjuK0Ns_2LnAbM=oufhcg2gavsb99+pkpuybrtr6k...@mail.gmail.com%3E [7] http://www.freebsd.org/cgi/query-pr.cgi?pr=159703 Wow, that's great progress! ... with a pretty long list of dependencies :) If there's too many issues with the dependencies it should be easy to scale down to the short list I mentioned earlier [9]. ... So I guess it's going to take a while to figure how to build the complete list of dependencies with Clang/LLVM. :) In the meantime the few changes in SVN revision r1164964 should allow a subset of Tuscany to be built with Clang/LLVM. The macos/macos-install script can be used to build the Tuscany runtime, a subset of its components and their dependencies on a clean Mac OS X 10.6.7 without requiring anything else than Xcode 4.1. The dependencies are built with the Xcode 4.1 GCC compiler. Tuscany is built with the Xcode 4.2 Clang/LLVM compiler. Hope this helps. -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Have to look at it during the weekend again or so. Didn't have much time recently cause I had to focus on other stuff. I think I will disable each broken dependency in the FreeBSD port and thus build Tuscany only with the proposed set of minimal dependency for now. Cheers Daniel On Tue, Sep 6, 2011 at 12:16 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Sun, Aug 14, 2011 at 2:28 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Thu, Aug 11, 2011 at 8:14 PM, dsh daniel.hais...@googlemail.com wrote: ... My attempt to provide an overview what works on FreeBSD (8.2) and what not: * autoconf, automake, libtool, doxygen, gcc can be installed from the FreeBSD ports collection * at the time there's only Apache HTTPD 2.2 available in the FreeBSD ports collection. I started creating an additional Apache HTTPD 2.3 port [0] but did not finish it yet * APR 2 - I sent an problem report [1] including a new port for the most recent APR 2 version * OpenSSL can be used from the FreeBSD base system * pcre-8.12 can be used from the FreeBSD ports collection * expat-2.0.1_1 can be used from the FreeBSD ports collection * memcached-1.4.5 can be installed from the FreeBSD ports collection * libevent-1.4.14b_2 can be installed from the FreeBSD ports collection * libxml2-2.7.8_1 can be installed from the FreeBSD ports collection * curl-7.21.3_2 can be installed from the FreeBSD ports collection (including SSL support) * I am currently in the process of upgrading the existing FreeBSD SpiderMonkey 1.7.x port [2], [3] * tinycdb-0.77 is available from the FreeBSD ports collection * ap20-mod_security-2.5.13 is available from the FreeBSD ports collection (not sure whether this one is too old) * I started to create an Apache Axis2/C port for FreeBSD but hit an issue which may be a bug [4] * I started to create an Apache Qpid port for FreeBSD put hit an issue which may be a bug [5], [6] * ap20-mod_auth_openid-0.5 is available from the FreeBSD ports collection including support for libopkele-2.0.2_1 * liboauth-0.9.4 is available from the FreeBSD ports collection * I think Apache Vysper can be installed relatively easy from the binary distribution and thus I decided to not create a separate FreeBSD port * Didn't create a FreeBSD port for Libstrophe yet cause I don't know how to pull sources directly from Github while building a FreeBSD port * postgresql-client-8.4.8 is available from the FreeBSD ports collection * scribe-2.2_3 is available from the FreeBSD ports collection * thrift-0.4.0,1 is available from the FreeBSD ports collection * I created and submitted a new FreeBSD port for Apache Libcloud using a PR [7] [0] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=284897+0+current/freebsd-ports [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=159658 [2] http://people.apache.org/~dsh/projects/fbsd-ports/spidermonkey185/ [3] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=343488+0+current/freebsd-ports [4] http://mail-archives.apache.org/mod_mbox/axis-c-user/201108.mbox/%3ccac0wh9akzrvc3-qzhg3thmfhnvi20kzerb0zp94hq_x+d_f...@mail.gmail.com%3E [5] https://issues.apache.org/jira/browse/QPID-2549 [6] http://mail-archives.apache.org/mod_mbox/qpid-dev/201108.mbox/%3CCAC0wh9ZjuK0Ns_2LnAbM=oufhcg2gavsb99+pkpuybrtr6k...@mail.gmail.com%3E [7] http://www.freebsd.org/cgi/query-pr.cgi?pr=159703 Wow, that's great progress! ... with a pretty long list of dependencies :) If there's too many issues with the dependencies it should be easy to scale down to the short list I mentioned earlier [9]. ... So I guess it's going to take a while to figure how to build the complete list of dependencies with Clang/LLVM. :) In the meantime the few changes in SVN revision r1164964 should allow a subset of Tuscany to be built with Clang/LLVM. The macos/macos-install script can be used to build the Tuscany runtime, a subset of its components and their dependencies on a clean Mac OS X 10.6.7 without requiring anything else than Xcode 4.1. The dependencies are built with the Xcode 4.1 GCC compiler. Tuscany is built with the Xcode 4.2 Clang/LLVM compiler. Hope this helps. -- Jean-Sebastien
Re: Thread local storage with CLang/LLVM, was: Fun with LevelDB, LLVM and OpenCL
On Wed, Aug 10, 2011 at 8:11 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Wed, Aug 10, 2011 at 6:45 AM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien, On Wed, Aug 10, 2011 at 6:45 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: That's what I meant when I mentioned 'convert usages of __thread to Posix thread TLS calls' in my previous post [1]. __thread works for me with GCC on Mac OS X (using the GCC build from the HPC project [2]), but causes a compile error with Clang/LLVM. It's listed as a limitation of Clang on Mac OS X in the Clang user manual (look for __thread on this page [3]). Does Works for me mean it compiles for you just fine AND during runtime you are getting the expected behavior or does it just mean the code compiles just fine if using GCC? I am asking cause I googled a bit last night and from what I found I had the feeling thread local isn't really supported on OS X during runtime. Hi Daniel! __thread compiles and behaves as expected for me on OS X with GCC 4.6 (the version referenced in the macos/macos-install script). I've added some test logic to kernel/parallel-test.cpp to help verify that (see SVN r1156460). Parallel-test built with GCC works, showing that __thread is correctly implemented by GCC on OS X. After commenting __thread on parallel-test.cpp:line 134, parallel-test breaks with an assertion error, showing that the test is actually effective at testing thread local storage. I'm hoping that this additional test code will help you when you implement thread local storage over the pthread API. In the meantime you don't have to worry about it if you --disable-threads :) Hi Daniel, In SVN revision 1162472 I've replaced the references to __thread by a new portable implementation of thread local storage using the POSIX pthread key_create, key_delete, setspecific and getspecific functions. See the perthread_ptr class in kernel/parallel.hpp, the tests in kernel/parallel-test.cpp and other usages throughout the code base. Compared to __thread, it also works on class and struct fields and can take an initialization function responsible for creating the thread specific value. Hope that helps... We shouldn't need to use --disable-threads anymore with CLang/LLVM. -- Jean-Sebastien
Re: Running on parallel hardware, was: Fun with LevelDB, LLVM and OpenCL
On Sun, Aug 14, 2011 at 9:30 PM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien, the old SpiderMonkey port [10] already contains support for nspr - so that end should be covered cause I am using the old port descriptor files as a basis for the new 1.8.5 port. That's the type of embedded board [11] I am considering. It comes with an FPGA processor that accompanies the main CPU that I want to leverage to optimize certain operations (maybe that scenario is similar to what you are doing with offloading CPU cycles to the GPU). [11] http://soekris.com/net6501.htm [10] http://www.freebsd.org/cgi/ports.cgi?query=^spidermonkeystype=allsektion=all Cheers Daniel Yes it's similar. I'm targeting two different platforms: - a cluster of energy efficient Atom + nvidia Ion fanless mini-itx boards + flash memory (pretty cheap computing power) - EC2 spot GPU instances loaded with high end nvidia Tesla Fermi gpus (not cheap but great for spot stress testing) -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Hi, looks like Qpid experiences a similar thread-local issue on OS X: https://issues.apache.org/jira/browse/QPID-2206 On Mon, Aug 15, 2011 at 6:30 AM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien, the old SpiderMonkey port [10] already contains support for nspr - so that end should be covered cause I am using the old port descriptor files as a basis for the new 1.8.5 port. That's the type of embedded board [11] I am considering. It comes with an FPGA processor that accompanies the main CPU that I want to leverage to optimize certain operations (maybe that scenario is similar to what you are doing with offloading CPU cycles to the GPU). [11] http://soekris.com/net6501.htm [10] http://www.freebsd.org/cgi/ports.cgi?query=^spidermonkeystype=allsektion=all Cheers Daniel On Sun, Aug 14, 2011 at 11:28 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Thu, Aug 11, 2011 at 8:14 PM, dsh daniel.hais...@googlemail.com wrote: ... My attempt to provide an overview what works on FreeBSD (8.2) and what not: * autoconf, automake, libtool, doxygen, gcc can be installed from the FreeBSD ports collection * at the time there's only Apache HTTPD 2.2 available in the FreeBSD ports collection. I started creating an additional Apache HTTPD 2.3 port [0] but did not finish it yet * APR 2 - I sent an problem report [1] including a new port for the most recent APR 2 version * OpenSSL can be used from the FreeBSD base system * pcre-8.12 can be used from the FreeBSD ports collection * expat-2.0.1_1 can be used from the FreeBSD ports collection * memcached-1.4.5 can be installed from the FreeBSD ports collection * libevent-1.4.14b_2 can be installed from the FreeBSD ports collection * libxml2-2.7.8_1 can be installed from the FreeBSD ports collection * curl-7.21.3_2 can be installed from the FreeBSD ports collection (including SSL support) * I am currently in the process of upgrading the existing FreeBSD SpiderMonkey 1.7.x port [2], [3] * tinycdb-0.77 is available from the FreeBSD ports collection * ap20-mod_security-2.5.13 is available from the FreeBSD ports collection (not sure whether this one is too old) * I started to create an Apache Axis2/C port for FreeBSD but hit an issue which may be a bug [4] * I started to create an Apache Qpid port for FreeBSD put hit an issue which may be a bug [5], [6] * ap20-mod_auth_openid-0.5 is available from the FreeBSD ports collection including support for libopkele-2.0.2_1 * liboauth-0.9.4 is available from the FreeBSD ports collection * I think Apache Vysper can be installed relatively easy from the binary distribution and thus I decided to not create a separate FreeBSD port * Didn't create a FreeBSD port for Libstrophe yet cause I don't know how to pull sources directly from Github while building a FreeBSD port * postgresql-client-8.4.8 is available from the FreeBSD ports collection * scribe-2.2_3 is available from the FreeBSD ports collection * thrift-0.4.0,1 is available from the FreeBSD ports collection * I created and submitted a new FreeBSD port for Apache Libcloud using a PR [7] [0] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=284897+0+current/freebsd-ports [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=159658 [2] http://people.apache.org/~dsh/projects/fbsd-ports/spidermonkey185/ [3] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=343488+0+current/freebsd-ports [4] http://mail-archives.apache.org/mod_mbox/axis-c-user/201108.mbox/%3ccac0wh9akzrvc3-qzhg3thmfhnvi20kzerb0zp94hq_x+d_f...@mail.gmail.com%3E [5] https://issues.apache.org/jira/browse/QPID-2549 [6] http://mail-archives.apache.org/mod_mbox/qpid-dev/201108.mbox/%3CCAC0wh9ZjuK0Ns_2LnAbM=oufhcg2gavsb99+pkpuybrtr6k...@mail.gmail.com%3E [7] http://www.freebsd.org/cgi/query-pr.cgi?pr=159703 Wow, that's great progress! ... with a pretty long list of dependencies :) If there's too many issues with the dependencies it should be easy to scale down to the short list I mentioned earlier [9]. BTW js-1.8.5 also needs nspr-4.8.x to correctly work in multiple threads. I just ran into that today while playing with massive multi-threading in the context of the OpenCL integration. [9] http://marc.info/?l=tuscany-devm=131308575106061w=2 ... I am interested in Apache Tuscany native cause I want to figure out, whether it would be possible to use OSGi like modules (i.e. provided by Apache Celix) for SCA components similar to what I did with Java in a Redbook I wrote last year [8]. Besides that I want to elaborate whether Apache Celix And Apache Tuscany (native) can be used on embedded devices such as router/firewall devices and what the use cases may be and thus my interest in creating all those various FreeBSD ports. Sounds interesting! [8] http://www.redbooks.ibm.com/abstracts/sg247911.html?Open -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
On Thu, Aug 11, 2011 at 8:14 PM, dsh daniel.hais...@googlemail.com wrote: ... My attempt to provide an overview what works on FreeBSD (8.2) and what not: * autoconf, automake, libtool, doxygen, gcc can be installed from the FreeBSD ports collection * at the time there's only Apache HTTPD 2.2 available in the FreeBSD ports collection. I started creating an additional Apache HTTPD 2.3 port [0] but did not finish it yet * APR 2 - I sent an problem report [1] including a new port for the most recent APR 2 version * OpenSSL can be used from the FreeBSD base system * pcre-8.12 can be used from the FreeBSD ports collection * expat-2.0.1_1 can be used from the FreeBSD ports collection * memcached-1.4.5 can be installed from the FreeBSD ports collection * libevent-1.4.14b_2 can be installed from the FreeBSD ports collection * libxml2-2.7.8_1 can be installed from the FreeBSD ports collection * curl-7.21.3_2 can be installed from the FreeBSD ports collection (including SSL support) * I am currently in the process of upgrading the existing FreeBSD SpiderMonkey 1.7.x port [2], [3] * tinycdb-0.77 is available from the FreeBSD ports collection * ap20-mod_security-2.5.13 is available from the FreeBSD ports collection (not sure whether this one is too old) * I started to create an Apache Axis2/C port for FreeBSD but hit an issue which may be a bug [4] * I started to create an Apache Qpid port for FreeBSD put hit an issue which may be a bug [5], [6] * ap20-mod_auth_openid-0.5 is available from the FreeBSD ports collection including support for libopkele-2.0.2_1 * liboauth-0.9.4 is available from the FreeBSD ports collection * I think Apache Vysper can be installed relatively easy from the binary distribution and thus I decided to not create a separate FreeBSD port * Didn't create a FreeBSD port for Libstrophe yet cause I don't know how to pull sources directly from Github while building a FreeBSD port * postgresql-client-8.4.8 is available from the FreeBSD ports collection * scribe-2.2_3 is available from the FreeBSD ports collection * thrift-0.4.0,1 is available from the FreeBSD ports collection * I created and submitted a new FreeBSD port for Apache Libcloud using a PR [7] [0] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=284897+0+current/freebsd-ports [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=159658 [2] http://people.apache.org/~dsh/projects/fbsd-ports/spidermonkey185/ [3] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=343488+0+current/freebsd-ports [4] http://mail-archives.apache.org/mod_mbox/axis-c-user/201108.mbox/%3ccac0wh9akzrvc3-qzhg3thmfhnvi20kzerb0zp94hq_x+d_f...@mail.gmail.com%3E [5] https://issues.apache.org/jira/browse/QPID-2549 [6] http://mail-archives.apache.org/mod_mbox/qpid-dev/201108.mbox/%3CCAC0wh9ZjuK0Ns_2LnAbM=oufhcg2gavsb99+pkpuybrtr6k...@mail.gmail.com%3E [7] http://www.freebsd.org/cgi/query-pr.cgi?pr=159703 Wow, that's great progress! ... with a pretty long list of dependencies :) If there's too many issues with the dependencies it should be easy to scale down to the short list I mentioned earlier [9]. BTW js-1.8.5 also needs nspr-4.8.x to correctly work in multiple threads. I just ran into that today while playing with massive multi-threading in the context of the OpenCL integration. [9] http://marc.info/?l=tuscany-devm=131308575106061w=2 ... I am interested in Apache Tuscany native cause I want to figure out, whether it would be possible to use OSGi like modules (i.e. provided by Apache Celix) for SCA components similar to what I did with Java in a Redbook I wrote last year [8]. Besides that I want to elaborate whether Apache Celix And Apache Tuscany (native) can be used on embedded devices such as router/firewall devices and what the use cases may be and thus my interest in creating all those various FreeBSD ports. Sounds interesting! [8] http://www.redbooks.ibm.com/abstracts/sg247911.html?Open -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Hi Jean-Sebastien, the old SpiderMonkey port [10] already contains support for nspr - so that end should be covered cause I am using the old port descriptor files as a basis for the new 1.8.5 port. That's the type of embedded board [11] I am considering. It comes with an FPGA processor that accompanies the main CPU that I want to leverage to optimize certain operations (maybe that scenario is similar to what you are doing with offloading CPU cycles to the GPU). [11] http://soekris.com/net6501.htm [10] http://www.freebsd.org/cgi/ports.cgi?query=^spidermonkeystype=allsektion=all Cheers Daniel On Sun, Aug 14, 2011 at 11:28 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Thu, Aug 11, 2011 at 8:14 PM, dsh daniel.hais...@googlemail.com wrote: ... My attempt to provide an overview what works on FreeBSD (8.2) and what not: * autoconf, automake, libtool, doxygen, gcc can be installed from the FreeBSD ports collection * at the time there's only Apache HTTPD 2.2 available in the FreeBSD ports collection. I started creating an additional Apache HTTPD 2.3 port [0] but did not finish it yet * APR 2 - I sent an problem report [1] including a new port for the most recent APR 2 version * OpenSSL can be used from the FreeBSD base system * pcre-8.12 can be used from the FreeBSD ports collection * expat-2.0.1_1 can be used from the FreeBSD ports collection * memcached-1.4.5 can be installed from the FreeBSD ports collection * libevent-1.4.14b_2 can be installed from the FreeBSD ports collection * libxml2-2.7.8_1 can be installed from the FreeBSD ports collection * curl-7.21.3_2 can be installed from the FreeBSD ports collection (including SSL support) * I am currently in the process of upgrading the existing FreeBSD SpiderMonkey 1.7.x port [2], [3] * tinycdb-0.77 is available from the FreeBSD ports collection * ap20-mod_security-2.5.13 is available from the FreeBSD ports collection (not sure whether this one is too old) * I started to create an Apache Axis2/C port for FreeBSD but hit an issue which may be a bug [4] * I started to create an Apache Qpid port for FreeBSD put hit an issue which may be a bug [5], [6] * ap20-mod_auth_openid-0.5 is available from the FreeBSD ports collection including support for libopkele-2.0.2_1 * liboauth-0.9.4 is available from the FreeBSD ports collection * I think Apache Vysper can be installed relatively easy from the binary distribution and thus I decided to not create a separate FreeBSD port * Didn't create a FreeBSD port for Libstrophe yet cause I don't know how to pull sources directly from Github while building a FreeBSD port * postgresql-client-8.4.8 is available from the FreeBSD ports collection * scribe-2.2_3 is available from the FreeBSD ports collection * thrift-0.4.0,1 is available from the FreeBSD ports collection * I created and submitted a new FreeBSD port for Apache Libcloud using a PR [7] [0] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=284897+0+current/freebsd-ports [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=159658 [2] http://people.apache.org/~dsh/projects/fbsd-ports/spidermonkey185/ [3] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=343488+0+current/freebsd-ports [4] http://mail-archives.apache.org/mod_mbox/axis-c-user/201108.mbox/%3ccac0wh9akzrvc3-qzhg3thmfhnvi20kzerb0zp94hq_x+d_f...@mail.gmail.com%3E [5] https://issues.apache.org/jira/browse/QPID-2549 [6] http://mail-archives.apache.org/mod_mbox/qpid-dev/201108.mbox/%3CCAC0wh9ZjuK0Ns_2LnAbM=oufhcg2gavsb99+pkpuybrtr6k...@mail.gmail.com%3E [7] http://www.freebsd.org/cgi/query-pr.cgi?pr=159703 Wow, that's great progress! ... with a pretty long list of dependencies :) If there's too many issues with the dependencies it should be easy to scale down to the short list I mentioned earlier [9]. BTW js-1.8.5 also needs nspr-4.8.x to correctly work in multiple threads. I just ran into that today while playing with massive multi-threading in the context of the OpenCL integration. [9] http://marc.info/?l=tuscany-devm=131308575106061w=2 ... I am interested in Apache Tuscany native cause I want to figure out, whether it would be possible to use OSGi like modules (i.e. provided by Apache Celix) for SCA components similar to what I did with Java in a Redbook I wrote last year [8]. Besides that I want to elaborate whether Apache Celix And Apache Tuscany (native) can be used on embedded devices such as router/firewall devices and what the use cases may be and thus my interest in creating all those various FreeBSD ports. Sounds interesting! [8] http://www.redbooks.ibm.com/abstracts/sg247911.html?Open -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
On Wed, Aug 10, 2011 at 9:17 AM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien, On Wed, Aug 10, 2011 at 5:26 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: Thanks for the link to js 185, I'll try it on Ubuntu, Redhat and Mac OS X too, happy to switch to it if it works on these systems, but I'd like to avoid installing it in the default system directories (hoping there's a way to specify the install dir with --prefix). The below link as well contains information on how to build mozjs: http://www.mongodb.org/display/DOCS/Building+Spider+Monkey Cheers Daniel Hi Daniel, Thanks for the link to js-1.8.5 and the build info. SVN revision r1156555 now uses SpiderMonkey 1.8.5. The INSTALL doc, macos-install and ubuntu-install* scripts have been updated accordingly (the build instructions from the MongoDB doc caused compile errors for me, but configure --prefix=... in the js/src directory worked fine). The js and json modules have been ported to the SpiderMonkey 1.8.5 API. A few JSON test files have been adapted to minor JSON formatting changes in 1.8.5, and all the test cases run successfully for me with make check, on Mac OS X, Ubuntu and Redhat. That's with GCC... hoping they'll work as well with Clang... -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Hi Jean-Sebastien, On Thu, Aug 11, 2011 at 7:53 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: thanks for your feedback. You are right as long as Tuscany native hasn't been released as a pre-build package that can be installed on an operating system it doesn't make sense to use libraries as dependencies coming pre-packaged with a particular OS or package management system. Instead I should setup my own staging area containing all the various dependencies. Tho, in the long run I am interested in pushing those dependencies as FreeBSD ports into the FreeBSD ports tree cause there you could have development snapshots of a certain library as a FreeBSD port which would allow to use the dependencies coming with the FreeBSD ports tree instead of manually compiled ones (we did that with DSPAM development snapshots in the past). That would be really cool! My attempt to provide an overview what works on FreeBSD (8.2) and what not: * autoconf, automake, libtool, doxygen, gcc can be installed from the FreeBSD ports collection * at the time there's only Apache HTTPD 2.2 available in the FreeBSD ports collection. I started creating an additional Apache HTTPD 2.3 port [0] but did not finish it yet * APR 2 - I sent an problem report [1] including a new port for the most recent APR 2 version * OpenSSL can be used from the FreeBSD base system * pcre-8.12 can be used from the FreeBSD ports collection * expat-2.0.1_1 can be used from the FreeBSD ports collection * memcached-1.4.5 can be installed from the FreeBSD ports collection * libevent-1.4.14b_2 can be installed from the FreeBSD ports collection * libxml2-2.7.8_1 can be installed from the FreeBSD ports collection * curl-7.21.3_2 can be installed from the FreeBSD ports collection (including SSL support) * I am currently in the process of upgrading the existing FreeBSD SpiderMonkey 1.7.x port [2], [3] * tinycdb-0.77 is available from the FreeBSD ports collection * ap20-mod_security-2.5.13 is available from the FreeBSD ports collection (not sure whether this one is too old) * I started to create an Apache Axis2/C port for FreeBSD but hit an issue which may be a bug [4] * I started to create an Apache Qpid port for FreeBSD put hit an issue which may be a bug [5], [6] * ap20-mod_auth_openid-0.5 is available from the FreeBSD ports collection including support for libopkele-2.0.2_1 * liboauth-0.9.4 is available from the FreeBSD ports collection * I think Apache Vysper can be installed relatively easy from the binary distribution and thus I decided to not create a separate FreeBSD port * Didn't create a FreeBSD port for Libstrophe yet cause I don't know how to pull sources directly from Github while building a FreeBSD port * postgresql-client-8.4.8 is available from the FreeBSD ports collection * scribe-2.2_3 is available from the FreeBSD ports collection * thrift-0.4.0,1 is available from the FreeBSD ports collection * I created and submitted a new FreeBSD port for Apache Libcloud using a PR [7] [0] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=284897+0+current/freebsd-ports [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=159658 [2] http://people.apache.org/~dsh/projects/fbsd-ports/spidermonkey185/ [3] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=343488+0+current/freebsd-ports [4] http://mail-archives.apache.org/mod_mbox/axis-c-user/201108.mbox/%3ccac0wh9akzrvc3-qzhg3thmfhnvi20kzerb0zp94hq_x+d_f...@mail.gmail.com%3E [5] https://issues.apache.org/jira/browse/QPID-2549 [6] http://mail-archives.apache.org/mod_mbox/qpid-dev/201108.mbox/%3CCAC0wh9ZjuK0Ns_2LnAbM=oufhcg2gavsb99+pkpuybrtr6k...@mail.gmail.com%3E [7] http://www.freebsd.org/cgi/query-pr.cgi?pr=159703 We may be able to do that if people like you start to help out... as I only have little free evening / weekend / vacation time for this. I am interested in Apache Tuscany native cause I want to figure out, whether it would be possible to use OSGi like modules (i.e. provided by Apache Celix) for SCA components similar to what I did with Java in a Redbook I wrote last year [8]. Besides that I want to elaborate whether Apache Celix And Apache Tuscany (native) can be used on embedded devices such as router/firewall devices and what the use cases may be and thus my interest in creating all those various FreeBSD ports. [8] http://www.redbooks.ibm.com/abstracts/sg247911.html?Open Cheers Daniel
Re: Fun with LevelDB, LLVM and OpenCL
Hi Jean-Sebastien, On Wed, Aug 10, 2011 at 6:45 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: That's what I meant when I mentioned 'convert usages of __thread to Posix thread TLS calls' in my previous post [1]. __thread works for me with GCC on Mac OS X (using the GCC build from the HPC project [2]), but causes a compile error with Clang/LLVM. It's listed as a limitation of Clang on Mac OS X in the Clang user manual (look for __thread on this page [3]). Does Works for me mean it compiles for you just fine AND during runtime you are getting the expected behavior or does it just mean the code compiles just fine if using GCC? I am asking cause I googled a bit last night and from what I found I had the feeling thread local isn't really supported on OS X during runtime. Are you getting a compile error with GCC too? or just with Clang? I forced myself to not install GCC on OS X if possible and thus the compile errors I am getting are those reported by Clang. For now you could just run configure without --enable-threads, then later try to convert the few references to __thread to Posix thread TLS calls (i.e. pthread_key_create, pthread_getspecific, pthread_setspecific, pthread_key_delete). Yep I am not using --enable-threads or I set it explicitly to --disable-threads to avoid the compile error for the time being. I will try to look into your Posix thread TLS calls suggestion today (I as well have to get APR V2 and back out the symlinks I set so far). Many thanks and best wishes! Cheers Daniel
Re: Fun with LevelDB, LLVM and OpenCL
Hi Jean-Sebastien, thanks for your feedback. You are right as long as Tuscany native hasn't been released as a pre-build package that can be installed on an operating system it doesn't make sense to use libraries as dependencies coming pre-packaged with a particular OS or package management system. Instead I should setup my own staging area containing all the various dependencies. Tho, in the long run I am interested in pushing those dependencies as FreeBSD ports into the FreeBSD ports tree cause there you could have development snapshots of a certain library as a FreeBSD port which would allow to use the dependencies coming with the FreeBSD ports tree instead of manually compiled ones (we did that with DSPAM development snapshots in the past). on another subject - Is there a new Tuscany native release planned anytime soon? M3 seems to date back to 2007. I am asking cause at least at the time you would plan to ship a new release -- and maybe even pre-build packages for certain Linux/Unix distributions such as DEBs or RPMs -- you would have to use the packages available on a certain OS. Cheers Daniel On Wed, Aug 10, 2011 at 5:26 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Tue, Aug 9, 2011 at 3:53 PM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien Delfino, some things I noticed during the configure process (I am using MacPorts): * I think it would help to point out in the INSTALL file that mozjs can be downloaded at - ftp://ftp.mozilla.org/pub/mozilla.org/js/ * On OSX Lion xulrunner won't install from MacPorts and thus one would end up without a proper mozjs installation * If compiling mozjs from the prio mentioned URL, it ends up to be installed as /opt/local/lib/libmozjs185.1.0.0.dylib which causes the configure process to fail * I thus symlinked /opt/local/lib/libmozjs185.1.0.0.dylib to /opt/local/lib/libmozjs.dylib I usually run configure --with-js-include=$build/tracemonkey-bin/include/js --with-js-lib=$build/tracemonkey-bin/lib with $build pointing to a non-system directory; ... after having built trace monkey like this: curl -OL http://hg.mozilla.org/tracemonkey/archive/e4364736e170.tar.gz mv e4364736e170.tar.gz tracemonkey-e4364736e170.tar.gz tar xzf tracemonkey-e4364736e170.tar.gz cd tracemonkey-e4364736e170/js/src $build/autoconf-2.13-bin/bin/autoconf ./configure --prefix=$build/tracemonkey-bin make make install ... and on Mac OS X this additional step: install_name_tool -id $build/tracemonkey-bin/lib/libmozjs.dylib $build/tracemonkey-bin/lib/libmozjs.dylib I've never had much luck installing libmozjs in the system directories for all kinds of reasons. On Ubuntu and Redhat I had conflicts with already installed incompatible versions, dependencies dragged by xulrunner, and IIRC similar issues on Mac OS X with macports too. Thanks for the link to js 185, I'll try it on Ubuntu, Redhat and Mac OS X too, happy to switch to it if it works on these systems, but I'd like to avoid installing it in the default system directories (hoping there's a way to specify the install dir with --prefix). More generally I try to stay away from using pre-built packaged system dependencies for the following reasons: - most of the times they system dependencies conflict with the versions of the dependencies I want, for example Ubuntu 10.10 ships with Apache httpd 2.2.16 which is really old compared to the 2.3.10 version used by Tuscany; - using the system dependencies shipped with the operating system means versions on different systems, different behavior and significant effort to port and test with all these different versions; - if I need to debug or patch them I'll have to recompile them anyway, and if I do that then I'll have to install them in a non-system directory to not mess up my system; After all, that's the beauty of open-source, you can get the source, compile it yourself and install it wherever is most convenient for you... - if I need to install Tuscany and its dependencies on a new machine (an EC2 VM for example), it's much easier to build and install all the dependencies in the same directory as Tuscany, zip that directory up, and transfer that zip to the target VMs without altering its system directories. * The configure process is looking for -lapr-2 instead of -lapr-1 (I have APR 1.4.5_1 installed) * I thus had to symlink /opt/local/lib/libapr-1.0.dylib to /opt/local/lib/libapr-2.dylib Yes I've been using apr-2 as it's required by httpd's mod-session for crypto. See the related email discussion on the httpd dev list there: http://markmail.org/thread/dbavlf55ywf2a33h I don't think that APR 1.4.5 includes the crypto API required by httpd 2.3.x mod_session_crypto yet, so we're stuck with APR trunk for now, until APR releases that API and httpd ports to that release. If you like I could try to update the INSTALL document accordingly and attach the patch to a JIRA. That would help, but what
Re: Fun with LevelDB, LLVM and OpenCL
IMHO what you need is a python script who is independent from the OS which it will cope your dependencies as maven does for building all the stuff. 2011/8/10 dsh daniel.hais...@googlemail.com: Hi Jean-Sebastien, thanks for your feedback. You are right as long as Tuscany native hasn't been released as a pre-build package that can be installed on an operating system it doesn't make sense to use libraries as dependencies coming pre-packaged with a particular OS or package management system. Instead I should setup my own staging area containing all the various dependencies. Tho, in the long run I am interested in pushing those dependencies as FreeBSD ports into the FreeBSD ports tree cause there you could have development snapshots of a certain library as a FreeBSD port which would allow to use the dependencies coming with the FreeBSD ports tree instead of manually compiled ones (we did that with DSPAM development snapshots in the past). on another subject - Is there a new Tuscany native release planned anytime soon? M3 seems to date back to 2007. I am asking cause at least at the time you would plan to ship a new release -- and maybe even pre-build packages for certain Linux/Unix distributions such as DEBs or RPMs -- you would have to use the packages available on a certain OS. Cheers Daniel On Wed, Aug 10, 2011 at 5:26 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Tue, Aug 9, 2011 at 3:53 PM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien Delfino, some things I noticed during the configure process (I am using MacPorts): * I think it would help to point out in the INSTALL file that mozjs can be downloaded at - ftp://ftp.mozilla.org/pub/mozilla.org/js/ * On OSX Lion xulrunner won't install from MacPorts and thus one would end up without a proper mozjs installation * If compiling mozjs from the prio mentioned URL, it ends up to be installed as /opt/local/lib/libmozjs185.1.0.0.dylib which causes the configure process to fail * I thus symlinked /opt/local/lib/libmozjs185.1.0.0.dylib to /opt/local/lib/libmozjs.dylib I usually run configure --with-js-include=$build/tracemonkey-bin/include/js --with-js-lib=$build/tracemonkey-bin/lib with $build pointing to a non-system directory; ... after having built trace monkey like this: curl -OL http://hg.mozilla.org/tracemonkey/archive/e4364736e170.tar.gz mv e4364736e170.tar.gz tracemonkey-e4364736e170.tar.gz tar xzf tracemonkey-e4364736e170.tar.gz cd tracemonkey-e4364736e170/js/src $build/autoconf-2.13-bin/bin/autoconf ./configure --prefix=$build/tracemonkey-bin make make install ... and on Mac OS X this additional step: install_name_tool -id $build/tracemonkey-bin/lib/libmozjs.dylib $build/tracemonkey-bin/lib/libmozjs.dylib I've never had much luck installing libmozjs in the system directories for all kinds of reasons. On Ubuntu and Redhat I had conflicts with already installed incompatible versions, dependencies dragged by xulrunner, and IIRC similar issues on Mac OS X with macports too. Thanks for the link to js 185, I'll try it on Ubuntu, Redhat and Mac OS X too, happy to switch to it if it works on these systems, but I'd like to avoid installing it in the default system directories (hoping there's a way to specify the install dir with --prefix). More generally I try to stay away from using pre-built packaged system dependencies for the following reasons: - most of the times they system dependencies conflict with the versions of the dependencies I want, for example Ubuntu 10.10 ships with Apache httpd 2.2.16 which is really old compared to the 2.3.10 version used by Tuscany; - using the system dependencies shipped with the operating system means versions on different systems, different behavior and significant effort to port and test with all these different versions; - if I need to debug or patch them I'll have to recompile them anyway, and if I do that then I'll have to install them in a non-system directory to not mess up my system; After all, that's the beauty of open-source, you can get the source, compile it yourself and install it wherever is most convenient for you... - if I need to install Tuscany and its dependencies on a new machine (an EC2 VM for example), it's much easier to build and install all the dependencies in the same directory as Tuscany, zip that directory up, and transfer that zip to the target VMs without altering its system directories. * The configure process is looking for -lapr-2 instead of -lapr-1 (I have APR 1.4.5_1 installed) * I thus had to symlink /opt/local/lib/libapr-1.0.dylib to /opt/local/lib/libapr-2.dylib Yes I've been using apr-2 as it's required by httpd's mod-session for crypto. See the related email discussion on the httpd dev list there: http://markmail.org/thread/dbavlf55ywf2a33h I don't think that APR 1.4.5 includes the crypto API required by httpd 2.3.x mod_session_crypto yet, so we're stuck
Re: Fun with LevelDB, LLVM and OpenCL
Hi Jean-Sebastien, On Wed, Aug 10, 2011 at 5:26 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: Thanks for the link to js 185, I'll try it on Ubuntu, Redhat and Mac OS X too, happy to switch to it if it works on these systems, but I'd like to avoid installing it in the default system directories (hoping there's a way to specify the install dir with --prefix). The below link as well contains information on how to build mozjs: http://www.mongodb.org/display/DOCS/Building+Spider+Monkey Cheers Daniel
Re: Fun with LevelDB, LLVM and OpenCL
On Wed, Aug 10, 2011 at 6:45 AM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien, On Wed, Aug 10, 2011 at 6:45 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: That's what I meant when I mentioned 'convert usages of __thread to Posix thread TLS calls' in my previous post [1]. __thread works for me with GCC on Mac OS X (using the GCC build from the HPC project [2]), but causes a compile error with Clang/LLVM. It's listed as a limitation of Clang on Mac OS X in the Clang user manual (look for __thread on this page [3]). Does Works for me mean it compiles for you just fine AND during runtime you are getting the expected behavior or does it just mean the code compiles just fine if using GCC? I am asking cause I googled a bit last night and from what I found I had the feeling thread local isn't really supported on OS X during runtime. Hi Daniel! __thread compiles and behaves as expected for me on OS X with GCC 4.6 (the version referenced in the macos/macos-install script). I've added some test logic to kernel/parallel-test.cpp to help verify that (see SVN r1156460). Parallel-test built with GCC works, showing that __thread is correctly implemented by GCC on OS X. After commenting __thread on parallel-test.cpp:line 134, parallel-test breaks with an assertion error, showing that the test is actually effective at testing thread local storage. I'm hoping that this additional test code will help you when you implement thread local storage over the pthread API. In the meantime you don't have to worry about it if you --disable-threads :) -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Hi Jean-Sebastien Delfino, some things I noticed during the configure process (I am using MacPorts): * I think it would help to point out in the INSTALL file that mozjs can be downloaded at - ftp://ftp.mozilla.org/pub/mozilla.org/js/ * On OSX Lion xulrunner won't install from MacPorts and thus one would end up without a proper mozjs installation * If compiling mozjs from the prio mentioned URL, it ends up to be installed as /opt/local/lib/libmozjs185.1.0.0.dylib which causes the configure process to fail * I thus symlinked /opt/local/lib/libmozjs185.1.0.0.dylib to /opt/local/lib/libmozjs.dylib * The configure process is looking for -lapr-2 instead of -lapr-1 (I have APR 1.4.5_1 installed) * I thus had to symlink /opt/local/lib/libapr-1.0.dylib to /opt/local/lib/libapr-2.dylib If you like I could try to update the INSTALL document accordingly and attach the patch to a JIRA. I am doing the dependency setup in parallel on FreeBSD 8.2 to see whether I would run into similar issues (for instance the one of having a -lapr-1 installed instead of the expected -lapr-2). We could then decide whether we want to improve the auto* tools infrastructure or not (or even migrate to CMake). PS: I am planning to create a FreeBSD port for Apache Tuscany (native) cause I feel having to setup all of these dependencies is kind of daunting. Cheers Daniel On Mon, Aug 8, 2011 at 7:05 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Sun, Aug 7, 2011 at 6:35 PM, dsh daniel.hais...@googlemail.com wrote: Just a little update: OSX Lion with Xcode 4.1: * i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) * Apple clang version 2.1 OSX Lion with Xcode 4.2: * i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 * Apple clang version 3.0 OSX Snow Leopard with Xcode 4.0.2: * i686-apple-darwin10-llvm-gcc-4.2 (GCC) 4.2.1 * Apple clang version 2.0 based on LLVM 2.9svn PS: There exists an Apple Xcode version 4.2 for OSX Snow Leopard which we could use as a fallback if 4.0.2 turns out to be unusable PPS: I really love these virtualized OSX envs I setup recently because they allow to have all those different setups and configurations :) Cheers Daniel OK, here's what I'm using on OS X Lion: Xcode Version 4.1 Build 4B110 macbook-air:~$ g++ --version i686-apple-darwin11-llvm-g++-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) macbook-air:~$ clang++ --version Apple clang version 2.1 (tags/Apple/clang-163.7.1) (based on LLVM 3.0svn) Target: x86_64-apple-darwin11.0.0 Thread model: posix -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Looks like OS X does not support thread local storage. I.e. __thread causes an compile error and thus one would not need to use --enable-threads or would need to disable them explicitely using --disable-threads. Is this a known issue on OS X? Cheers Daniel On Wed, Aug 10, 2011 at 12:53 AM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien Delfino, some things I noticed during the configure process (I am using MacPorts): * I think it would help to point out in the INSTALL file that mozjs can be downloaded at - ftp://ftp.mozilla.org/pub/mozilla.org/js/ * On OSX Lion xulrunner won't install from MacPorts and thus one would end up without a proper mozjs installation * If compiling mozjs from the prio mentioned URL, it ends up to be installed as /opt/local/lib/libmozjs185.1.0.0.dylib which causes the configure process to fail * I thus symlinked /opt/local/lib/libmozjs185.1.0.0.dylib to /opt/local/lib/libmozjs.dylib * The configure process is looking for -lapr-2 instead of -lapr-1 (I have APR 1.4.5_1 installed) * I thus had to symlink /opt/local/lib/libapr-1.0.dylib to /opt/local/lib/libapr-2.dylib If you like I could try to update the INSTALL document accordingly and attach the patch to a JIRA. I am doing the dependency setup in parallel on FreeBSD 8.2 to see whether I would run into similar issues (for instance the one of having a -lapr-1 installed instead of the expected -lapr-2). We could then decide whether we want to improve the auto* tools infrastructure or not (or even migrate to CMake). PS: I am planning to create a FreeBSD port for Apache Tuscany (native) cause I feel having to setup all of these dependencies is kind of daunting. Cheers Daniel On Mon, Aug 8, 2011 at 7:05 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Sun, Aug 7, 2011 at 6:35 PM, dsh daniel.hais...@googlemail.com wrote: Just a little update: OSX Lion with Xcode 4.1: * i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) * Apple clang version 2.1 OSX Lion with Xcode 4.2: * i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 * Apple clang version 3.0 OSX Snow Leopard with Xcode 4.0.2: * i686-apple-darwin10-llvm-gcc-4.2 (GCC) 4.2.1 * Apple clang version 2.0 based on LLVM 2.9svn PS: There exists an Apple Xcode version 4.2 for OSX Snow Leopard which we could use as a fallback if 4.0.2 turns out to be unusable PPS: I really love these virtualized OSX envs I setup recently because they allow to have all those different setups and configurations :) Cheers Daniel OK, here's what I'm using on OS X Lion: Xcode Version 4.1 Build 4B110 macbook-air:~$ g++ --version i686-apple-darwin11-llvm-g++-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) macbook-air:~$ clang++ --version Apple clang version 2.1 (tags/Apple/clang-163.7.1) (based on LLVM 3.0svn) Target: x86_64-apple-darwin11.0.0 Thread model: posix -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
On Tue, Aug 9, 2011 at 3:53 PM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien Delfino, some things I noticed during the configure process (I am using MacPorts): * I think it would help to point out in the INSTALL file that mozjs can be downloaded at - ftp://ftp.mozilla.org/pub/mozilla.org/js/ * On OSX Lion xulrunner won't install from MacPorts and thus one would end up without a proper mozjs installation * If compiling mozjs from the prio mentioned URL, it ends up to be installed as /opt/local/lib/libmozjs185.1.0.0.dylib which causes the configure process to fail * I thus symlinked /opt/local/lib/libmozjs185.1.0.0.dylib to /opt/local/lib/libmozjs.dylib I usually run configure --with-js-include=$build/tracemonkey-bin/include/js --with-js-lib=$build/tracemonkey-bin/lib with $build pointing to a non-system directory; ... after having built trace monkey like this: curl -OL http://hg.mozilla.org/tracemonkey/archive/e4364736e170.tar.gz mv e4364736e170.tar.gz tracemonkey-e4364736e170.tar.gz tar xzf tracemonkey-e4364736e170.tar.gz cd tracemonkey-e4364736e170/js/src $build/autoconf-2.13-bin/bin/autoconf ./configure --prefix=$build/tracemonkey-bin make make install ... and on Mac OS X this additional step: install_name_tool -id $build/tracemonkey-bin/lib/libmozjs.dylib $build/tracemonkey-bin/lib/libmozjs.dylib I've never had much luck installing libmozjs in the system directories for all kinds of reasons. On Ubuntu and Redhat I had conflicts with already installed incompatible versions, dependencies dragged by xulrunner, and IIRC similar issues on Mac OS X with macports too. Thanks for the link to js 185, I'll try it on Ubuntu, Redhat and Mac OS X too, happy to switch to it if it works on these systems, but I'd like to avoid installing it in the default system directories (hoping there's a way to specify the install dir with --prefix). More generally I try to stay away from using pre-built packaged system dependencies for the following reasons: - most of the times they system dependencies conflict with the versions of the dependencies I want, for example Ubuntu 10.10 ships with Apache httpd 2.2.16 which is really old compared to the 2.3.10 version used by Tuscany; - using the system dependencies shipped with the operating system means versions on different systems, different behavior and significant effort to port and test with all these different versions; - if I need to debug or patch them I'll have to recompile them anyway, and if I do that then I'll have to install them in a non-system directory to not mess up my system; After all, that's the beauty of open-source, you can get the source, compile it yourself and install it wherever is most convenient for you... - if I need to install Tuscany and its dependencies on a new machine (an EC2 VM for example), it's much easier to build and install all the dependencies in the same directory as Tuscany, zip that directory up, and transfer that zip to the target VMs without altering its system directories. * The configure process is looking for -lapr-2 instead of -lapr-1 (I have APR 1.4.5_1 installed) * I thus had to symlink /opt/local/lib/libapr-1.0.dylib to /opt/local/lib/libapr-2.dylib Yes I've been using apr-2 as it's required by httpd's mod-session for crypto. See the related email discussion on the httpd dev list there: http://markmail.org/thread/dbavlf55ywf2a33h I don't think that APR 1.4.5 includes the crypto API required by httpd 2.3.x mod_session_crypto yet, so we're stuck with APR trunk for now, until APR releases that API and httpd ports to that release. If you like I could try to update the INSTALL document accordingly and attach the patch to a JIRA. That would help, but what would help even more , I think, would be to have a script that downloads / checks out the dependencies and builds them end to end on a particular system. The single INSTALL doc may get really big if we start to add to it all the different instructions to build off system dependencies on Ubuntu, Redhat, Mac OS X, FreeBSD etc... There's already a script to build everything on Ubuntu 10.x there: https://svn.apache.org/repos/asf/tuscany/sca-cpp/trunk/ubuntu/ubuntu-install There's another one for Mac OS X Leopard and Lion there: https://svn.apache.org/repos/asf/tuscany/sca-cpp/trunk/macos/macos-install ... which by the way should help you build on MacOS without fighting with any system dependencies. With that script, you should be able to build on a pristine Mac OS system + GCC 4.4+ from Macport or other, without polluting your system directories with any other dependencies at all... as all required dependencies are built from source by the script. I am doing the dependency setup in parallel on FreeBSD 8.2 to see whether I would run into similar issues (for instance the one of having a -lapr-1 installed instead of the expected -lapr-2). We could then decide whether we want to improve the auto* tools infrastructure or not (or even
Re: Fun with LevelDB, LLVM and OpenCL
On Sun, Aug 7, 2011 at 6:35 PM, dsh daniel.hais...@googlemail.com wrote: Just a little update: OSX Lion with Xcode 4.1: * i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) * Apple clang version 2.1 OSX Lion with Xcode 4.2: * i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 * Apple clang version 3.0 OSX Snow Leopard with Xcode 4.0.2: * i686-apple-darwin10-llvm-gcc-4.2 (GCC) 4.2.1 * Apple clang version 2.0 based on LLVM 2.9svn PS: There exists an Apple Xcode version 4.2 for OSX Snow Leopard which we could use as a fallback if 4.0.2 turns out to be unusable PPS: I really love these virtualized OSX envs I setup recently because they allow to have all those different setups and configurations :) Cheers Daniel OK, here's what I'm using on OS X Lion: Xcode Version 4.1 Build 4B110 macbook-air:~$ g++ --version i686-apple-darwin11-llvm-g++-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) macbook-air:~$ clang++ --version Apple clang version 2.1 (tags/Apple/clang-163.7.1) (based on LLVM 3.0svn) Target: x86_64-apple-darwin11.0.0 Thread model: posix -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Just a little update: OSX Lion with Xcode 4.1: * i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) * Apple clang version 2.1 OSX Lion with Xcode 4.2: * i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 * Apple clang version 3.0 OSX Snow Leopard with Xcode 4.0.2: * i686-apple-darwin10-llvm-gcc-4.2 (GCC) 4.2.1 * Apple clang version 2.0 based on LLVM 2.9svn PS: There exists an Apple Xcode version 4.2 for OSX Snow Leopard which we could use as a fallback if 4.0.2 turns out to be unusable PPS: I really love these virtualized OSX envs I setup recently because they allow to have all those different setups and configurations :) Cheers Daniel On Sun, Aug 7, 2011 at 1:52 AM, dsh daniel.hais...@googlemail.com wrote: On Sun, Aug 7, 2011 at 12:34 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: The configure.ac language is a little cryptic so let me know if you need help with that detection + option setup, as I've already done similar things several times. Ok I'll let you know if I struggle with any of the GNU Build Tools files such as the autoconf files. I worked on Gnome, Avahi, wzdftpd, PostgreSQL, OpenLDAP and some other autoconf based projects back in 2005/2006 or so. Thus I hope it doesn't feel too alienating to me :) Thanks and best wishes! Cheers Daniel
Re: Fun with LevelDB, LLVM and OpenCL
Hi Jean-Sebastien, if you like I could take a look at LLVM/clang. From my point of view it not only integrates more nicely with Xcode but you would as well get rid of dynamically linking against things like libgcc* which sometimes triggers licensing discussions if talking to attorneys. If you think it would make sense I could also have a look at replacing the autotools stuff (automake) by CMake. The latter would generate IDE project setups such as that for Xcode on the fly and Xcode provides the same level of abstraction like you would get it from GNU autotools et al. PS: I am on vacation till Aug. 15th and wanted to do some C/C++ coding anyway so maybe that's a nice match and additional excercise I could have. Cheers Daniel On Sat, Aug 6, 2011 at 7:55 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: Hi all, I'm on vacation for a few days and I'm going to have a little bit of free time to hack around Tuscany C++. I've been thinking about the following: a) Integrate LevelDB [1] as a key-value store component. I've been experimenting with LevelDB and I think it'll be a useful complement to the TinyCDB based store (best for constant data) and the PostgreSQL based store (best for flexible query). b) Use the LLVM based Clang compiler [2] as an alternative to GCC, useful on MacOS as it'll enable building without anything extra on top of Xcode 4.1. c) More challenging and interesting too, analyze a composite and transform it into a set of OpenCL [3][4][5][6] kernel functions running on a mix of CPUs and GPUs (giving you a lot of cheap computing power to run your SCA compositions). In particular convert a graph of synchronous service invocations into chained asynchronous tasks and determine the optimum partition of the composite into OpenCL kernels, work groups and work items while minimizing the data transfers between OpenCL kernels. I only have a few days to bootstrap these projects and will probably spend most of my energy on the OpenCL one as I want to try a few innovative ideas (including a functional programming monad based approach to model the interactions around composites and help the sync to async transformation, and another idea to have the component implementations do meta-programming and return their computations as kernel programs instead of the actual computation results) so if anybody is interested in working with me on this stuff ping me on gtalk jsdelfino, the #tuscany IRC channel or even better post a message here... [1] http://code.google.com/p/leveldb/ [2] http://clang.llvm.org/ [3] http://www.khronos.org/opencl/ [4] http://developer.apple.com/library/mac/#documentation/Performance/Conceptual/OpenCL_MacProgGuide/Introduction/Introduction.html [5] http://developer.nvidia.com/opencl [6] http://developer.amd.com/zones/OpenCLZone/Pages/default.aspx -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Btw, concerning OpenCL I recently found this IBM research project [1]. It not only abstracts OpenCL but OpenMP, CUDA and FPGA programming too. So you don't have to deal with low level library programming. Just thought I share that link with you (I am not recommending using that project as part of Apache Tuscany).. [1] https://researcher.ibm.com/researcher/view_project.php?id=122 Cheers Daniel On Sat, Aug 6, 2011 at 7:55 AM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: Hi all, I'm on vacation for a few days and I'm going to have a little bit of free time to hack around Tuscany C++. I've been thinking about the following: a) Integrate LevelDB [1] as a key-value store component. I've been experimenting with LevelDB and I think it'll be a useful complement to the TinyCDB based store (best for constant data) and the PostgreSQL based store (best for flexible query). b) Use the LLVM based Clang compiler [2] as an alternative to GCC, useful on MacOS as it'll enable building without anything extra on top of Xcode 4.1. c) More challenging and interesting too, analyze a composite and transform it into a set of OpenCL [3][4][5][6] kernel functions running on a mix of CPUs and GPUs (giving you a lot of cheap computing power to run your SCA compositions). In particular convert a graph of synchronous service invocations into chained asynchronous tasks and determine the optimum partition of the composite into OpenCL kernels, work groups and work items while minimizing the data transfers between OpenCL kernels. I only have a few days to bootstrap these projects and will probably spend most of my energy on the OpenCL one as I want to try a few innovative ideas (including a functional programming monad based approach to model the interactions around composites and help the sync to async transformation, and another idea to have the component implementations do meta-programming and return their computations as kernel programs instead of the actual computation results) so if anybody is interested in working with me on this stuff ping me on gtalk jsdelfino, the #tuscany IRC channel or even better post a message here... [1] http://code.google.com/p/leveldb/ [2] http://clang.llvm.org/ [3] http://www.khronos.org/opencl/ [4] http://developer.apple.com/library/mac/#documentation/Performance/Conceptual/OpenCL_MacProgGuide/Introduction/Introduction.html [5] http://developer.nvidia.com/opencl [6] http://developer.amd.com/zones/OpenCLZone/Pages/default.aspx -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
On Sat, Aug 6, 2011 at 2:14 AM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien, if you like I could take a look at LLVM/clang. From my point of view it not only integrates more nicely with Xcode but you would as well get rid of dynamically linking against things like libgcc* which sometimes triggers licensing discussions if talking to attorneys. If you think it would make sense I could also have a look at replacing the autotools stuff (automake) by CMake. The latter would generate IDE project setups such as that for Xcode on the fly and Xcode provides the same level of abstraction like you would get it from GNU autotools et al. PS: I am on vacation till Aug. 15th and wanted to do some C/C++ coding anyway so maybe that's a nice match and additional excercise I could have. Cheers Daniel Hi Daniel, Cool! I'll be happy to work with you on this! Porting Tuscany to Clang shouldn't be too hard. To build on Mac OS X Snow Leopard or Lion, take a look at the INSTALL doc and the macos/macos-install script, which builds Tuscany and all its dependencies from source. The dependencies can be built with the GCC 4.2 shipped with Xcode. Tuscany needs GCC 4.4+ as it requires some C++0x features. To build with CLang/LLVM you'll probably just have to set CC=/usr/bin/clang, CXX=/usr/bin/clang++, then add -std=libc++ to the Tuscany configure.ac script. Then you'll have to adjust bits of the code, in particular to convert usages of __thread to Posix thread TLS calls, and perhaps fix up some of the C++0x variadic templates which showed compile errors last time I tried. I had tried an earlier version of Clang but Xcode 4.1 now ships Clang/LLVM 3.0, which should support everything that's needed. To remove all the dependencies on the GCC libs, the next step would be to build all the Tuscany dependencies with Clang as well. Most of their builds are Autoconf/Automake based so you could try to just run them with the CC and CXX environment variables set to point to clang and clang++. That should work for most... and then go from there... About CMake, I don't know it well, so I'm not sure but my guess is that'll be more work as there's 44 Makefile.am configs in the source tree, unless you have a way to convert them quasi-automatically to CMake equivalent configs... Have a nice vacation :) For any questions, issues etc look for me on gtalk, irc #tuscany or post here on the list. -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
Hi Jean-Sebastien, for which Xcode versions shall we be heading for? 4.1 or 4.2 on Lion (I have 4.2 dev preview 5 on one of my Lion systems to develop iOS 5 apps) and 4.0.2 on snow leopard? I would then check which would be the min LLVM/clang version that needs to be supported. Shall we try to keep GCC compatibility if possible (e.g. should it be possible to build Tuscany on platforms where there's no LLVM distribution available)? On FreeBSD for instance it's your own decision whether to compile the system's user-land utilities etc. either using GCC or LLVM/clang. FreeBSD makes this possible by either setting CC to CC=gcc or CC=clang and so on in /etc/make.conf. Maybe we could use a similar mechanism... Concerning CMake - That has for now low priority to me I would focus on LLVM and if that is done on CMake. Cheers Daniel On Sat, Aug 6, 2011 at 8:14 PM, Jean-Sebastien Delfino jsdelf...@apache.org wrote: On Sat, Aug 6, 2011 at 2:14 AM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien, if you like I could take a look at LLVM/clang. From my point of view it not only integrates more nicely with Xcode but you would as well get rid of dynamically linking against things like libgcc* which sometimes triggers licensing discussions if talking to attorneys. If you think it would make sense I could also have a look at replacing the autotools stuff (automake) by CMake. The latter would generate IDE project setups such as that for Xcode on the fly and Xcode provides the same level of abstraction like you would get it from GNU autotools et al. PS: I am on vacation till Aug. 15th and wanted to do some C/C++ coding anyway so maybe that's a nice match and additional excercise I could have. Cheers Daniel Hi Daniel, Cool! I'll be happy to work with you on this! Porting Tuscany to Clang shouldn't be too hard. To build on Mac OS X Snow Leopard or Lion, take a look at the INSTALL doc and the macos/macos-install script, which builds Tuscany and all its dependencies from source. The dependencies can be built with the GCC 4.2 shipped with Xcode. Tuscany needs GCC 4.4+ as it requires some C++0x features. To build with CLang/LLVM you'll probably just have to set CC=/usr/bin/clang, CXX=/usr/bin/clang++, then add -std=libc++ to the Tuscany configure.ac script. Then you'll have to adjust bits of the code, in particular to convert usages of __thread to Posix thread TLS calls, and perhaps fix up some of the C++0x variadic templates which showed compile errors last time I tried. I had tried an earlier version of Clang but Xcode 4.1 now ships Clang/LLVM 3.0, which should support everything that's needed. To remove all the dependencies on the GCC libs, the next step would be to build all the Tuscany dependencies with Clang as well. Most of their builds are Autoconf/Automake based so you could try to just run them with the CC and CXX environment variables set to point to clang and clang++. That should work for most... and then go from there... About CMake, I don't know it well, so I'm not sure but my guess is that'll be more work as there's 44 Makefile.am configs in the source tree, unless you have a way to convert them quasi-automatically to CMake equivalent configs... Have a nice vacation :) For any questions, issues etc look for me on gtalk, irc #tuscany or post here on the list. -- Jean-Sebastien
Re: Fun with LevelDB, LLVM and OpenCL
On Sat, Aug 6, 2011 at 2:02 PM, dsh daniel.hais...@googlemail.com wrote: Hi Jean-Sebastien, for which Xcode versions shall we be heading for? 4.1 or 4.2 on Lion (I have 4.2 dev preview 5 on one of my Lion systems to develop iOS 5 apps) and 4.0.2 on snow leopard? I would then check which would be the min LLVM/clang version that needs to be supported. If I remember correctly 4.0.2 didn't have all the required C++0x support (I didn't research it extensively though), but I'm pretty sure that 4.1 and 4.2 are based on LLVM 3.0+ and should work. I have Xcode 4.1 on Lion right now, but if you only manage to make it work with Xcode 4.2 that's OK with me too, I'll just upgrade to 4.2 as well :). Shall we try to keep GCC compatibility if possible (e.g. should it be possible to build Tuscany on platforms where there's no LLVM distribution available)? On FreeBSD for instance it's your own decision whether to compile the system's user-land utilities etc. either using GCC or LLVM/clang. FreeBSD makes this possible by either setting CC to CC=gcc or CC=clang and so on in /etc/make.conf. Maybe we could use a similar mechanism... Yes, I'm interested in keeping compatibility with GCC for some time to run it on my EC2 and home servers, which don't have LLVM yet... Best would be to detect availability of LLVM in configure.ac and automatically point CC and CXX to it when available, and have a --with-llvm=yes/no configure option to override that if needed (like with the other configure options listed by configure --help). The configure.ac language is a little cryptic so let me know if you need help with that detection + option setup, as I've already done similar things several times. Concerning CMake - That has for now low priority to me I would focus on LLVM and if that is done on CMake. Sounds good! -- Jean-Sebastien
Fun with LevelDB, LLVM and OpenCL
Hi all, I'm on vacation for a few days and I'm going to have a little bit of free time to hack around Tuscany C++. I've been thinking about the following: a) Integrate LevelDB [1] as a key-value store component. I've been experimenting with LevelDB and I think it'll be a useful complement to the TinyCDB based store (best for constant data) and the PostgreSQL based store (best for flexible query). b) Use the LLVM based Clang compiler [2] as an alternative to GCC, useful on MacOS as it'll enable building without anything extra on top of Xcode 4.1. c) More challenging and interesting too, analyze a composite and transform it into a set of OpenCL [3][4][5][6] kernel functions running on a mix of CPUs and GPUs (giving you a lot of cheap computing power to run your SCA compositions). In particular convert a graph of synchronous service invocations into chained asynchronous tasks and determine the optimum partition of the composite into OpenCL kernels, work groups and work items while minimizing the data transfers between OpenCL kernels. I only have a few days to bootstrap these projects and will probably spend most of my energy on the OpenCL one as I want to try a few innovative ideas (including a functional programming monad based approach to model the interactions around composites and help the sync to async transformation, and another idea to have the component implementations do meta-programming and return their computations as kernel programs instead of the actual computation results) so if anybody is interested in working with me on this stuff ping me on gtalk jsdelfino, the #tuscany IRC channel or even better post a message here... [1] http://code.google.com/p/leveldb/ [2] http://clang.llvm.org/ [3] http://www.khronos.org/opencl/ [4] http://developer.apple.com/library/mac/#documentation/Performance/Conceptual/OpenCL_MacProgGuide/Introduction/Introduction.html [5] http://developer.nvidia.com/opencl [6] http://developer.amd.com/zones/OpenCLZone/Pages/default.aspx -- Jean-Sebastien
