Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On 10/14/06, Bill Spotz <[EMAIL PROTECTED]> wrote: > I would like to second the notion of converging on a single MPI > interface. My parallel project encapsulates most of the inter- > processor communication within higher-level objects because the lower- > level communication patterns can usually be determined from higher- > level data structures. But still, there are times when a user would > like access to the lower-level MPI interface. Using mpi4py, you have access to almost all MPI internals directly from the Python side, with an API really similar to MPI-2 C++ bindigs. This is a feature I've not seen in other Python bindings for MPI. I think this is really important for developers and people learning MPI; you do not need to learn a new API. -- Lisandro Dalcín --- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On 10/13/06, eric <[EMAIL PROTECTED]> wrote: > Brian Granger wrote: > > keeping mpi4py as a separate project. > > Is there any chance it could be > > hosted at mip4py.scipy.org? > > > Fine from our side... > > eric > Can anybody help setting up mip4py.scipy.org? I really do not have experience with SVN. What should I do? -- Lisandro Dalcín --- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On 10/15/06, A. M. Archibald <[EMAIL PROTECTED]> wrote: > So, well, any suggestion for a name? pylab is already in use by > matplotlib (for some reason), as is Scientific Python (and numpy, > Numeric and numarray are obviously already confusing). I always thought 'pylab' was an odd name, so I always do from matplotlib import pylab as plot I think the matplotlib folks would be willing to change the name of pylab if it were seen as something that would be good for the community. I kind of wonder why they didn't call it 'matplot', personally. --bb - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On Sat, Oct 14, 2006 at 10:50:36AM -0600, Charles R Harris wrote: > So, well, any suggestion for a name? pylab is already in use by > matplotlib (for some reason), as is Scientific Python (and numpy, > Numeric and numarray are obviously already confusing). >supernumpy? I think we should rather build upon scipy, rather than numpy, if we want to build upon an existing name. Numpy is low-level array manipulation, scipy is all the fancy math that goes above, so it make more sens to derive a name on scipy. Besides scipy.org is already a good rallying point for scientific computing with python How about scipylab ! :-> -- Gaël - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On 10/14/06, A. M. Archibald <[EMAIL PROTECTED]> wrote: On 14/10/06, Gael Varoquaux <[EMAIL PROTECTED]> wrote:> On Sat, Oct 14, 2006 at 06:58:45AM -0600, Bill Spotz wrote: I agree. Moreover, being picked for such integration work would helpencourage people to converge on one MPI interface (for example). There's some discussion of such a package at NumpyProConPage on thewiki (and in particular at http://www.scipy.org/PyLabAwaits ). The"scipy superpack" (which is only for Windows, as I understand it) is a sort of beginning.So, well, any suggestion for a name? pylab is already in use bymatplotlib (for some reason), as is Scientific Python (and numpy,Numeric and numarray are obviously already confusing). supernumpy?Chuck - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On 14/10/06, Gael Varoquaux <[EMAIL PROTECTED]> wrote: > On Sat, Oct 14, 2006 at 06:58:45AM -0600, Bill Spotz wrote: > > I would like to second the notion of converging on a single MPI > > interface. My parallel project encapsulates most of the inter- > > processor communication within higher-level objects because the lower- > > level communication patterns can usually be determined from higher- > > level data structures. But still, there are times when a user would > > like access to the lower-level MPI interface. > > I think we are running in the same old problem of scipy/a super-package > of scientific tools. It is important to keep numpy/scipy modular. People > may want to install either one without an MPI interface. However it is > nice that if somebody just wants "everything" without having to choose > he can download that "everything" from scpy.org and that it comes well > bundled together. So all we need to do is find a name for that > super-package, put it together with good integration and add it to > scipy.org. > > My 2 cents, > > Gaël I agree. Moreover, being picked for such integration work would help encourage people to converge on one MPI interface (for example). There's some discussion of such a package at NumpyProConPage on the wiki (and in particular at http://www.scipy.org/PyLabAwaits ). The "scipy superpack" (which is only for Windows, as I understand it) is a sort of beginning. So, well, any suggestion for a name? pylab is already in use by matplotlib (for some reason), as is Scientific Python (and numpy, Numeric and numarray are obviously already confusing). A. M. Archibald - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On Sat, Oct 14, 2006 at 06:58:45AM -0600, Bill Spotz wrote: > I would like to second the notion of converging on a single MPI > interface. My parallel project encapsulates most of the inter- > processor communication within higher-level objects because the lower- > level communication patterns can usually be determined from higher- > level data structures. But still, there are times when a user would > like access to the lower-level MPI interface. I think we are running in the same old problem of scipy/a super-package of scientific tools. It is important to keep numpy/scipy modular. People may want to install either one without an MPI interface. However it is nice that if somebody just wants "everything" without having to choose he can download that "everything" from scpy.org and that it comes well bundled together. So all we need to do is find a name for that super-package, put it together with good integration and add it to scipy.org. My 2 cents, Gaël - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On Oct 14, 2006, at 2:58 AM, [EMAIL PROTECTED] wrote: > On 10/13/06, Lisandro Dalcin <[EMAIL PROTECTED]> wrote: >> This post is surely OT, but I cannot imagine a better place to >> contact >> people about this subject. Please, don't blame me. >> >> Any people here interested in NumPy/SciPy + MPI? > > I've been working on a Dynamic Bayesian Network (DBN) toolkit for > some time > (called Mocapy, freely available from sourceforge https:// > sourceforge.net/projects/mocapy/). The thing is almost entirely > implemented in Python, and currently uses Numeric and pyMPI. I > routinely train DBNs from 100.000s of observations on our 240 CPU > cluster. > > I'm in the proces of porting Mocapy to numpy. I assume pyMPI will > also work with numpy, but I haven't tried it out yet. Would be > great if scipy came with default MPI support, especially since > pyMPI does not seem to be actively developed anymore. I would like to second the notion of converging on a single MPI interface. My parallel project encapsulates most of the inter- processor communication within higher-level objects because the lower- level communication patterns can usually be determined from higher- level data structures. But still, there are times when a user would like access to the lower-level MPI interface. I haven't tried to make my project compatible with any of the several MPI python interfaces out there, because it isn't clear to me which one is most widely used. If one were to emerge -- and even better, if the various independent projects were to then combine their efforts in an open source environment (the way Numeric and numarray are converging to NumPy) -- then this choice would be easy. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370Email: [EMAIL PROTECTED] ** - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
On 10/13/06, Lisandro Dalcin <[EMAIL PROTECTED]> wrote: > This post is surely OT, but I cannot imagine a better place to contact > people about this subject. Please, don't blame me. > > Any people here interested in NumPy/SciPy + MPI? I've been working on a Dynamic Bayesian Network (DBN) toolkit for some time (called Mocapy, freely available from sourceforge https://sourceforge.net/projects/mocapy/). The thing is almost entirely implemented in Python, and currently uses Numeric and pyMPI. I routinely train DBNs from 100.000s of observations on our 240 CPU cluster. I'm in the proces of porting Mocapy to numpy. I assume pyMPI will also work with numpy, but I haven't tried it out yet. Would be great if scipy came with default MPI support, especially since pyMPI does not seem to be actively developed anymore. Cheers, -Thomas Thomas Hamelryck, Post-doctoral researcher Bioinformatics center Institute of Molecular Biology and Physiology University of Copenhagen Universitetsparken 15 - Bygning 10 DK-2100 Copenhagen Ø Denmark Homepage: http://www.binf.ku.dk/Protein_structure - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
Brian Granger wrote: > Just as a data point. > > I have used mpi4py before and have built it on many systems ranging > from my macbook to NERSC supercomputers. In my opinion it is > currently the best python mpi bindings available. Lisandro has done a > fantastic job with this. Also Fernando and I have worked hard to make > sure that mpi4py works with the new parallel capabilities of IPython. > > I would love to see mpi4py hosted in a public repository for others to > contribute. I think this would really solidify mpi4py as a top notch > mpi interface. But, my only concern is that there might be many folks > who want to use mpi4py who don't need scipy. I am one of those folks > - I don't necessarily need scipy on the NERSC supercomputers, but I do > need mpi4py. Because of this, I would probably still recommend > keeping mpi4py as a separate project. Is there any chance it could be > hosted at mip4py.scipy.org? > Fine from our side... eric > I strongly encourage others to try it out. Installation is is easy. > > Brian Granger > > On 10/13/06, Lisandro Dalcin <[EMAIL PROTECTED]> wrote: > >> This post is surely OT, but I cannot imagine a better place to contact >> people about this subject. Please, don't blame me. >> >> Any people here interested in NumPy/SciPy + MPI? From some time ago, >> I've been developing mpi4py (first release at SF) and I am really near >> to release a new version. >> >> This package exposes an API almost identical to MPI-2 C++ bindings. >> Almost all MPI-1 and MPI-2 features (even one-sided communications and >> parallel I/O) are fully supported for any object exposing >> single-segment buffer interface, an only some of them for >> communication of general Python objects (with the help of >> pickle/marshal). >> >> The posibility of constructing any user-defined MPI datatypes, as well >> as virtual topologies (specially cartesian), can be really nice for >> anyone interested in parallel multidimensional array procesing. >> >> Before the next release, I would like to wait for any comment, You can >> contact me via private mail to get a tarbal with latest developments, >> or we can have some discussion here, if many of you consider this a >> good idea. In the long term, I would like to see mpi4py integrated as >> a subpackage of SciPy. >> >> Regards, >> >> -- >> Lisandro Dalcín >> --- >> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) >> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) >> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) >> PTLC - Güemes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 >> ___ >> Scipy-dev mailing list >> [EMAIL PROTECTED] >> http://projects.scipy.org/mailman/listinfo/scipy-dev >> >> > > - > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ___ > Numpy-discussion mailing list > Numpy-discussion@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-dev] NumPy/SciPy + MPI for Python
Just as a data point. I have used mpi4py before and have built it on many systems ranging from my macbook to NERSC supercomputers. In my opinion it is currently the best python mpi bindings available. Lisandro has done a fantastic job with this. Also Fernando and I have worked hard to make sure that mpi4py works with the new parallel capabilities of IPython. I would love to see mpi4py hosted in a public repository for others to contribute. I think this would really solidify mpi4py as a top notch mpi interface. But, my only concern is that there might be many folks who want to use mpi4py who don't need scipy. I am one of those folks - I don't necessarily need scipy on the NERSC supercomputers, but I do need mpi4py. Because of this, I would probably still recommend keeping mpi4py as a separate project. Is there any chance it could be hosted at mip4py.scipy.org? I strongly encourage others to try it out. Installation is is easy. Brian Granger On 10/13/06, Lisandro Dalcin <[EMAIL PROTECTED]> wrote: > This post is surely OT, but I cannot imagine a better place to contact > people about this subject. Please, don't blame me. > > Any people here interested in NumPy/SciPy + MPI? From some time ago, > I've been developing mpi4py (first release at SF) and I am really near > to release a new version. > > This package exposes an API almost identical to MPI-2 C++ bindings. > Almost all MPI-1 and MPI-2 features (even one-sided communications and > parallel I/O) are fully supported for any object exposing > single-segment buffer interface, an only some of them for > communication of general Python objects (with the help of > pickle/marshal). > > The posibility of constructing any user-defined MPI datatypes, as well > as virtual topologies (specially cartesian), can be really nice for > anyone interested in parallel multidimensional array procesing. > > Before the next release, I would like to wait for any comment, You can > contact me via private mail to get a tarbal with latest developments, > or we can have some discussion here, if many of you consider this a > good idea. In the long term, I would like to see mpi4py integrated as > a subpackage of SciPy. > > Regards, > > -- > Lisandro Dalcín > --- > Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) > Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) > Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) > PTLC - Güemes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > ___ > Scipy-dev mailing list > [EMAIL PROTECTED] > http://projects.scipy.org/mailman/listinfo/scipy-dev > - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
