This is always the problem with big packages and can't really suggest much.
If a package has a lot of Python code, ensuring that it has .pyc files to facilitate quicker loading is one thing you can do, but I am guess here that those packages are primarily C extensions and so that shouldn't be an issue. Anyway, this is a question better off asked on the scipy/numpy forms. Graham On 30 November 2011 06:58, Hang-A <[email protected]> wrote: > Hi, Grapham! > > Thanks for your reference. > I read it multiple times to understand what is going on. > > Here is a new problem. > After setting up the WSGIApplicationGroup directive, > at least my application runs. > The new problem is that the speed of importing scipy or numpy is still > very slow. > Do you think this problem is still related to the use of the > simplified GIL api? > Even in the terminal, it takes a longer time to import scipy or numpy > on our cluster server running on Cent OS 6. > > -- Cluster server running on Cent OS 6 > time python -c 'import scipy' > real 0m9.734s > user 0m0.268s > sys 0m1.188s > time python -c 'import numpy' > real 0m3.192s > user 0m0.169s > sys 0m0.387s > > -- my desktop running Mac OS X Leopard > time python -c 'import scipy' > real 0m2.385s > user 0m0.178s > sys 0m0.260s > time python -c 'import numpy' > real 0m0.149s > user 0m0.081s > sys 0m0.066s > > Do you have any ideas about how to increase the speed of scipy/numpy > import? > > Thanks! > > > On Nov 7, 1:00 pm, Graham Dumpleton <[email protected]> > wrote: >> For some context as to why it helps see: >> >> http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simpli... >> >> Graham >> >> On 8 November 2011 03:39, Hang-A <[email protected]> wrote: >> >> >> >> >> >> >> >> > That was it. >> > Thanks, Graham! >> >> > Myung-Hwa >> >> > On Nov 5, 5:42 pm, Graham Dumpleton <[email protected]> >> > wrote: >> >> Try setting: >> >> >> WSGIApplicationGroup %{GLOBAL} >> >> >> for that WSGI application in Apache configuration. >> >> >> Likely you are using a third party C extension module in there >> >> somewhere which isn't safe to use in sub interpreter. >> >> >> That directive will force your application to run in main Python >> >> interpreter within the process. >> >> >> Graham >> >> >> On 6 November 2011 06:56, Hang-A <[email protected]> wrote: >> >> >> > Hello, list! >> >> >> > First of all, this will be a long message, due to the complexity of >> >> > our environment. >> >> > So, please be patient with my question. >> >> >> > I am trying to run a simple django application in a cluster >> >> > environment. >> >> > And, my application hangs while it imports scipy.linalg, and both >> >> > scipy and apache do not write out error messages. >> >> > When I run my application in my local python shell, it imports >> >> > scipy.linalg. But, somehow it does not when it is run by apache. >> >> > So, after reading this message, please share any ideas about how to >> >> > debug this problem or new solutions to address this issue or deploy my >> >> > application. >> >> >> > Now, let me explain our current setup. >> >> > 1. OS >> >> > -- The server is a compute cluster where each node runs centos 6 that >> >> > was installed from a clean version of centos6 minimal. >> >> > 2. Apache >> >> > -- Apache 2.2 was also manually installed from one of default linux >> >> > repository. >> >> > To be specific, it was installed from its source code together with >> >> > httpd-dev. >> >> > 3. Python >> >> > -- Python 2.7.2 was also installed from its source code across all >> >> > nodes in the cluster. Its source code was downloaded from python.org's >> >> > ftp. >> >> > 4. Python packages: nose, numpy, scipy >> >> > -- Nose 1.1.2 was downloaded from pypi.python.org and installed from >> >> > its source code. >> >> > -- numpy 1.6.1 was downloaded and installed from a linux repository. >> >> > When building numpy, gnu95 fortran complier was used. >> >> > -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas >> >> > from their source code. >> >> > ----- atlas was from sourceforge's 3.8.4 stable version. To compile >> >> > altas, gcc was used. >> >> > ----- lapack and blas was obtained from netlib.org's repository. To >> >> > compile the package of lapack and blas, gforan was used. >> >> > ----- Finally, after exporting paths to blas, lapack, and atlas, >> >> > scipy-0.9.0 was installed from its source code. >> >> > scipy was obtained from sourceforge.net's repository. >> >> >> > All of the above were installed in the same way across all nodes in >> >> > our cluster. >> >> > Since I am the only user of the cluster who needs to run python web >> >> > applications, >> >> > I installed python virtualenv package in my local directory. >> >> > Within my virtual environment, django-1.3 and pysal-1.2 (our own >> >> > package) were installed. >> >> > To deploy my web applications, we used mod_wsgi. >> >> > mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. >> >> >> > My application is as follows: >> >> > import os, sys, site, time >> >> > print >> sys.stderr, str(time.time()) + ':' + str(time.ctime()) >> >> >> > root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), >> >> > '..')) >> >> > root_dir += '/myunghwa_python2' >> >> > print >> sys.stderr, root_dir >> >> > print >> sys.stderr, os.listdir(root_dir) >> >> > site.addsitedir(os.path.join(root_dir, 'lib/python2.7/site-packages')) >> >> > sys.path.append(root_dir) >> >> > print >> sys.stderr, sys.path >> >> > print >> sys.stderr, os.listdir(sys.path[-2]) >> >> >> > os.environ['PYTHON_EGG_CACHE'] = '/home/mhwang3/repos/django/python- >> >> > eggs' >> >> > os.environ['DJANGO_SETTINGS_MODULE'] = 'wsgi.settings' >> >> >> > def application(environ, start_response): >> >> > status = '200 OK' >> >> > output = 'Hello!' >> >> > try: >> >> > t1 = time.time() >> >> > print >> sys.stderr, "Going to import numpy: " + >> >> > str(time.time()) + ":" + str(time.time() - t1) >> >> > import numpy >> >> > print >> sys.stderr, "imported numpy" >> >> > print >> sys.stderr, "Going to import scipy" >> >> > import scipy >> >> > print >> sys.stderr, "imported scipy >> >> > from:"+str(scipy.__path__) >> >> > print >> sys.stderr, "Going to import scipy.linalg" + >> >> > str(time.time()) + ':' + str(time.time() - t1) >> >> > import scipy.stats >> >> > print >> sys.stderr, "imported scipy.linalg: " + >> >> > str(time.time()) + ':' + str(time.time() - t1) >> >> > print >> sys.stderr, "Going to import pysal" >> >> > import pysal >> >> > print >> sys.stderr, "imported pysal" >> >> > except: >> >> > print >> sys.stderr, "exception" >> >> > raise >> >> >> > response_headers = [('Content-type', 'text/plain'), >> >> > ('Content-Length', str(len(output)))] >> >> > start_response(status, response_headers) >> >> >> > return [output] >> >> >> > Basically, it is a 'hello world' application that tests if numpy, >> >> > scipy, and pysal can be imported. >> >> > In the attached file, lines 4-9 are just adding paths to django and >> >> > pysal so that apache knows where to find these packages. >> >> > Also, to let apache know where to find atlas-related packages, the >> >> > path to those packages was added to the LD_LIBRARY_PATH environment >> >> > variable in the /etc/sysconfig/httpd file. >> >> >> > When I first ran my application, it just hung and wrote no message. >> >> > So, across scipy.linalg modules, I added print out statements to >> >> > figure out at which point the import was broken. >> >> > Here is the messages I got when I imported scipy.linalg in my local >> >> > python shell. >> >> > ######################## >> >> > starting linalg.__init__ >> >> > pre __init__.__doc__ >> >> > pre __init__.__version__ >> >> > pre __init__.misc >> >> > pre __init__.basic >> >> > ####################### >> >> > Starting basic >> >> > pre basic.flinalg >> >> > pre basic.lapack >> >> > pre basic.misc >> >> > pre basic.scipy.linalg >> >> > pre basic.decomp_svd >> >> > pre __init__.decomp >> >> > ################ >> >> > starting decomp >> >> > pre decomp.array et al. >> >> > pre decomp.calc_lwork >> >> > pre decomp.LinAlgError >> >> > pre decomp.get_lapack_funcs >> >> > pre decomp.get_blas_funcs >> >> > #################### >> >> > Starting blas >> >> > pre blas.scipy.linalg.fblas >> >> > pre blas.scipy.linalg.cblas >> >> > pre __init__.decomp_lu >> >> > pre __init__.decomp_cholesky >> >> > pre __init__.decomp_qr >> >> > ################# >> >> > Starting special_matrices >> >> > pre special_matrices.math >> >> > pre special_matrices.np >> >> > pre __init__.decomp_svd >> >> > pre __init__.decomp_schur >> >> > ################## >> >> > starting schur... >> >> > pre decomp_schur.misc >> >> > pre decomp_schur.LinAlgError >> >> > pre decomp_schur.get_lapack_funcs >> >> > pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 >> >> > schur testing >> >> > pre __init__.matfuncs >> >> > ##################### >> >> > Starting matfuncs >> >> > pre matfuncs. asarray et al >> >> > pre matfuncs.matrix >> >> > pre matfuncs.np >> >> > pre matfuncs.misc >> >> > pre matfuncs.basic >> >> > pre matfuncs.special_matrices >> >> > pre matfuncs.decomp >> >> > pre matfuncs.decomp_svd >> >> > pre matfuncs.decomp_schur >> >> > pre __init__.blas >> >> > pre __init__.special_matrices >> >> > When scipy.linalg is successfully imported, I should get these >> >> > messages. >> >> > But, when my web application tried to import scipy.linalg, the output >> >> > messages stop at line 41. >> >> > At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py >> >> > was already imported at line 16, scipy ignores it and continues to >> >> > import other modules in my local shell. >> >> > But, somehow, in apache-mod_wsgi environment, scipy failed to ignore >> >> > or reload decomp.py and seems to kill my web application. >> >> > This is really odd, because python does not give any message about >> >> > this error and neither does apache. apache just hangs without sending >> >> > out any response. >> >> > Since lapack and blas functions were imported successfully, the >> >> > problem seems not related to path setup. >> >> >> > If anyone in the list has any insights into or experience into this >> >> > kind of symptom, >> >> > please share your insights and experience. In particular, debugging >> >> > techniques or less-known installation/compilation problems would be >> >> > helpful. >> >> > I feel like I am at a dead end. So, please help me. >> >> >> > Thanks for reading this post. >> >> > I will look forward to your responses. >> >> >> > -- Myunghwa-Hwang Hwang >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "modwsgi" group. >> >> > To post to this group, send email to [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]. >> >> > For more options, visit this group >> >> > athttp://groups.google.com/group/modwsgi?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "modwsgi" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/modwsgi?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
