[sage-devel] Re: mpmath and plot incompatibility
On 3/1/11 1:37 PM, Fredrik Johansson wrote: On Tue, Mar 1, 2011 at 3:43 PM, Jason Grout wrote: On 3/1/11 3:58 AM, Fredrik Johansson wrote: On Tue, Mar 1, 2011 at 4:30 AM, Francois Bissey wrote: On Mon, 28 Feb 2011 at 01:26PM -0800, jtyard wrote: I'm running sage 4.6.1 and cannot use plot after loading mpmath. Namely, running from mpmath import * plot(lambda t: sin(2*pi*t), [1, 4]) produces no output in the notebook. I'm loading mpmath because I need to make plots of theta functions, and the page http://mpmath.googlecode.com/svn/trunk/doc/build/functions/elliptic.html# jacobi-theta-functions tells me to first run "from mpmath import *" to gain access to jtheta. I'm not 100% sure what's going on, but I do see that mpmath has its own plot function, and your "from mpmath import *" line overwrites Sage's usual plot function with mpmath's. This works: sage: import mpmath sage: plot(lambda t: mpmath.sin(2*pi*t), (1,4)) So I guess the question is, why isn't mpmath's plot function working? Interestingly mpmath-0.16 and 0.17 have support for matplotlib so presumably those can use matplotlib for plot. mpmath basically justs calls pylab.show(), which doesn't do anything in the notebook (nor in the Sage interpreter). We could hack mpmath in Sage to be more clever. Alternatively, perhaps a better solution would to make pylab.show() work in Sage. Is this feasible? If you compile with SAGE_MATPLOTLIB_GUI (and you have the prerequisites so that one of the GUI backends is built), then I believe pylab.show() works just fine. Ah, nice. Still, it would be nice if matplotlib could be used to generate plots directly in the notebook, by default. All that needs to happen is pylab.show() generate an image file. The notebook will then pick up that image file and display it automatically. Thanks, Jason -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: mpmath and plot incompatibility
On Tue, Mar 1, 2011 at 3:43 PM, Jason Grout wrote: > On 3/1/11 3:58 AM, Fredrik Johansson wrote: >> >> On Tue, Mar 1, 2011 at 4:30 AM, Francois Bissey >> wrote: On Mon, 28 Feb 2011 at 01:26PM -0800, jtyard wrote: > > I'm running sage 4.6.1 and cannot use plot after loading mpmath. > Namely, running > >> from mpmath import * >> plot(lambda t: sin(2*pi*t), [1, 4]) > > produces no output in the notebook. I'm loading mpmath because I need > to make plots of theta functions, and the page > > > > http://mpmath.googlecode.com/svn/trunk/doc/build/functions/elliptic.html# > jacobi-theta-functions > > tells me to first run "from mpmath import *" to gain access to jtheta. I'm not 100% sure what's going on, but I do see that mpmath has its own plot function, and your "from mpmath import *" line overwrites Sage's usual plot function with mpmath's. This works: sage: import mpmath sage: plot(lambda t: mpmath.sin(2*pi*t), (1,4)) So I guess the question is, why isn't mpmath's plot function working? >>> Interestingly mpmath-0.16 and 0.17 have support for matplotlib so >>> presumably >>> those can use matplotlib for plot. >> >> mpmath basically justs calls pylab.show(), which doesn't do anything >> in the notebook (nor in the Sage interpreter). We could hack mpmath in >> Sage to be more clever. Alternatively, perhaps a better solution would >> to make pylab.show() work in Sage. Is this feasible? > > If you compile with SAGE_MATPLOTLIB_GUI (and you have the prerequisites so > that one of the GUI backends is built), then I believe pylab.show() works > just fine. Ah, nice. Still, it would be nice if matplotlib could be used to generate plots directly in the notebook, by default. Fredrik -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: profiling Sage startup
On Tue, Mar 1, 2011 at 10:07 AM, David Kirkby wrote: > On 1 March 2011 15:38, daly wrote: >> Sage could follow the same technique used by Gimp. >> Gimp loads a huge number of files at startup and can >> take over a minute to start on small systems. They >> have a startup splash screen that shows the files >> which are being loaded. That way the user sees progress >> during startup. This wouldn't make sense for the command line version, though one could have a ... "progress bar." I still think we should shoot for < 1 second, which is totally obtainable. > I personally find Sage starts a lot quicker than Gimp, but perhaps > this depends on ones system. Sage is taking 21 s to start on my laptop > (dual core 2.0 GHz), compared to Mathematica's 7 seconds. I don't have > Gimp on here. > > On my workstation at home (3.33 GHz quad core Xeon) Sage is taking > only a few seconds. > > Is it possible to open these files read-only? I don't know if that > could save time on any file systems? I could image on something like > NFS, it could be potentially quicker if the system does not have to > consider the possibility Sage might modify the files. I bet Python already does this for loading modules. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: profiling Sage startup
On Mar 1, 10:13 am, Robert Bradshaw
wrote:
> On Tue, Mar 1, 2011 at 12:48 AM, Johan S. R. Nielsen
> See lazy-import. Doing this for everything may incur significant
> delays the first time a function is called (rather than before the
> prompt) and there are issues with Sage being fragile about the order
> in which some modules are implemented, but yes, it's possible and
> largely implemented.
>
> - Robert
i have played a bit more with lazy_import and looks promising.
On my computer, a cold start of sage is around 10 seconds.
I start a fresh session and look in the global namespace names
belonging to a module containing sage and save this information in a
pickle file
{{{
import pickle
L = copy(globals())
pepinillo = {}
for name in L:
if hasattr(L[name], '__module__') and L[name].__module__:
ambient_module = L[name].__module__
if 'sage' in ambient_module:
if pepinillo.has_key(ambient_module):
pepinillo[ambient_module].append(name)
else:
pepinillo[ambient_module] = [name]
f = open('pepinillo.sobj','w')
pickle.dump(pepinillo, f)
f.close()
}}}
pepinillo.sobj is only 70K
Now, I start an ipython session and load the information of pepinillo
as lazy_objects
{{{
import pickle
L = pickle.load(open('pepinillo.sobj'))
from sage.misc.lazy_import import lazy_import
for ambient_module in L:
for name in L[ambient_module]:
lazy_import(ambient_module, name)
}}}
This is instantaneous less than 0.5 seconds
Now I have more or less a sage session. It is not usable, there is no
preparser. There is no working ZZ, RR or CC so most commands complain.
ZZ = Integer() produces a segfault. var('x') also segfaults.
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Re: [sage-devel] Re: profiling Sage startup
On Tue, Mar 1, 2011 at 4:56 AM, luisfe wrote:
> On Mar 1, 1:32 pm, "Johan S. R. Nielsen"
> wrote:
>> On Mar 1, 10:13 am, Robert Bradshaw
>> wrote:
>>
>> Nice! I weren't aware of this module. When you get a good idea,
>> there's a good chance that someone else thought of it before ;-) I
>> like the fact that one can dynamically hack into an object's
>> namespace :-D However, lazy_import seems not to be used much (only 2-3
>> places) currently in the Sage startup (or did I grep wrongly?). Was it
>> never the intention or is it due to the overhead?
>
> Yes, it is not used much. Ideally, the way of improving sage
> startuptime would be:
>
> 1.- On first execution by a user, create a pickle of lazy_import of
> the global names and save it in the .sage directory. This should not
> be slower than current startup.
>
> 2.- Then, following executions of sage would only load that pickle
> instead of reading hundreds of different files. This will translates
> the delay from the start to the first call of the function. But note
> that you will never call each global name so globally should be
> faster...
>
> 3.- That pickle should be updated under certain conditions. By a
> command, each time sage is updated etc.
That essentially what it does, though not of the global module because
that would just be shifting the wait time from startup to first
invocation.
> Note that currently there are lazy_import(...) in the startup of sage,
> but this command does not avoid disk access of the file, only
> execution/compilation of the module to import. That is why it has
> currently low impact.
Disk access of a, or a couple, files is not the problem. Even on a
slow filesystem, that's below human perception. It's disk access of
1000s of files that's the problem.
>> Also, I didn't proofread the entire lazy_import code, but the
>> implementation seems to differ from my idea in a significant way: the
>> LazyImport object keeps wrapping the imported objects. My thought was
>> that the first time the imported object was accessed, it would replace
>> itself with the original in the global namespace, and then forward the
>> call. This way, there will be zero overhead in all later calls.
>
> No, the lazy_import object keeps wrapping the original object, but
> when accessing the lazy_import object it imports the real object in
> the namespace. So, if the lazy_import has not been referenced by
> another object then it will be collected as garbage. I do not have a
> running sage right now, but if you experiment you will see that after
> calling the object the lazy_import object disappears.
>
> It should be something like:
>
> {{{
> sage: from sage.misc.lazy_import import lazy_import
> sage: lazy_import('sage.rings.all', 'ZZ')
> sage: a = ZZ
> sage: type(ZZ)
>
> sage: ZZ(4.0)
> 4
> sage: type(ZZ)
>
> sage: type(a)
>
> }}}
Should be? It is (in the latest version of Sage :-).
- Robert
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Re: [sage-devel] Re: profiling Sage startup
On Tue, Mar 1, 2011 at 4:32 AM, Johan S. R. Nielsen
wrote:
> On Mar 1, 10:13 am, Robert Bradshaw
> wrote:
>> On Tue, Mar 1, 2011 at 12:48 AM, Johan S. R. Nielsen
>>
>>
>>
>> wrote:
>> > On Feb 23, 11:03 pm, Jason Grout wrote:
>> >> On 2/23/11 3:56 PM, Robert Bradshaw wrote:
>>
>> >> > On Wed, Feb 23, 2011 at 1:47 PM, Jason Grout
>> >> > wrote:
>> >> >> On 2/23/11 3:06 PM, Robert Bradshaw wrote:
>>
>> >> >>> On Wed, Feb 23, 2011 at 11:34 AM, William Stein
>> >> >>> wrote:
>>
>> >> On Wed, Feb 23, 2011 at 10:57 AM, Jason Grout
>> >> wrote:
>>
>> >> > On 2/23/11 12:28 PM, William Stein wrote:
>>
>> >> >> At lunch yesterday Robert Bradshaw made the interesting suggestion
>> >> >> to
>> >> >> read the docs for importlib
>> >> >> (http://docs.python.org/dev/library/importlib.html) and write a
>> >> >> customized import hook, so that every time during Sage startup
>> >> >> that a
>> >> >> module is imported, the import is done from a single big in-memory
>> >> >> zip
>> >> >> file instead of done using the filesystem. If this can be made
>> >> >> to
>> >> >> work, it would be a huge win for slow filesystems. The basic
>> >> >> problem
>> >> >> is that some filesystems are fast but have huge*latency*.
>>
>> >> > Is it a big win primarily because the zip file contents can be read
>> >> > in
>> >> > and
>> >> > cached by us? I'm just trying to understand it better.
>>
>> >> Which would you rather do on a high latency filesystem:
>>
>> >> (1) Read/stat 20,000 little files, or
>> >> (2) Read exactly one 40MB file.
>>
>> >> > Is this the same idea as Jar files in java?
>>
>> >> I don't know.
>>
>> >> >>> Yep. In that case the "high latency file system" was a webserver.
>>
>> >> > You mean likehttp://docs.python.org/library/zipimport.html?
>>
>> >> Cool.
>>
>> >> >>> Note that this should just involve putting the zip file first in the
>> >> >>> python path.
>>
>> >> I don't know for a fact that Robert Bradshaw's suggestion will be a
>> >> big win, since nobody has tried this yet. But I'm optimistic. The
>> >> idea would be to make a zip archive of
>> >> $SAGE_ROOT/local/lib/python/site-packages (say), and do *all* imports
>> >> using that massive zip archive.
>>
>> >> >>> I'm optimistic too. This would, of course, make more sense for
>> >> >>> system-wide installs than development versions, but the former are
>> >> >>> more likely to be on a non-local filesystem anyways.
>>
>> >> >> Sounds like it is time for a trial!
>>
>> >> >> I created a directory of 2000 .py files and an __init__.py file to
>> >> >> make it a
>> >> >> module
>>
>> >> >> for i in range(2000):
>> >> >> with open('importtest/test_%s.py'%i,'w') as f:
>> >> >> f.write("VALUE=%s\n"%i)
>> >> >> with open('importtest/__init__.py','w') as f:
>> >> >> f.write(' ')
>>
>> >> >> Then I imported each of these so that .pyc files were created.
>>
>> >> >> for i in range(2000):
>> >> >> exec 'import importtest.test_%s'%i
>>
>> >> >> Okay, then I copied the directory and zipped it up (in the shell now):
>>
>> >> >> $ cp -r importtest zipimporttest
>> >> >> $ zip -r tmp.zip zipimporttest
>> >> >> $ rm -rf zipimporttest
>>
>> >> >> One nice side effect is that the zip file is less than one MB, while
>> >> >> the
>> >> >> directory of python files is around 16M.
>>
>> >> >> Now for the test. Here are my two scripts. One imports each module
>> >> >> in the
>> >> >> directory and adds up the VALUE in each module:
>>
>> >> >> % cat mytest.py
>> >> >> s=0
>> >> >> for i in range(2000):
>> >> >> exec 'import importtest.test_%s as tt'%i
>> >> >> s+=tt.VALUE
>> >> >> print s
>>
>> >> >> The other first adds the zip to the front of sys.path and then does
>> >> >> the same
>> >> >> imports and summing, but using the zipped module:
>>
>> >> >> % cat mytestzip.py
>> >> >> import sys
>> >> >> sys.path.insert(0,'./tmp.zip')
>> >> >> s=0
>> >> >> for i in range(2000):
>> >> >> exec 'import zipimporttest.test_%s as tt'%i
>> >> >> s+=tt.VALUE
>> >> >> print s
>>
>> >> >> And now for the timings:
>>
>> >> >> % time sage -python mytest.py
>> >> >> Detected SAGE64 flag
>> >> >> Building Sage on OS X in 64-bit mode
>> >> >> 1999000
>> >> >> sage -python mytest.py 0.26s user 1.47s system 75% cpu 2.282 total
>>
>> >> >> % time sage -python mytestzip.py
>> >> >> Detected SAGE64 flag
>> >> >> Building Sage on OS X in 64-bit mode
>> >> >> 1999000
>> >> >> sage -python mytestzip.py 0.21s user 0.11s system 99% cpu 0.327 total
>>
>> >> >> It looks like the zip is a clear winner in this case. And this is
>> >> >> with the
>> >> >> directory presumably in the FS cache.
>>
>> >> > Cool. Given the CPU was pegged at 99%, have you tried using an
>> >> > uncompressed zip file? It'd have more data to read, but less to do
>> >> > with it once it's read.
>>
>> >> In my case, using zip -0 (n
Re: [sage-devel] Re: profiling Sage startup
On 1 March 2011 15:38, daly wrote: > Sage could follow the same technique used by Gimp. > Gimp loads a huge number of files at startup and can > take over a minute to start on small systems. They > have a startup splash screen that shows the files > which are being loaded. That way the user sees progress > during startup. I personally find Sage starts a lot quicker than Gimp, but perhaps this depends on ones system. Sage is taking 21 s to start on my laptop (dual core 2.0 GHz), compared to Mathematica's 7 seconds. I don't have Gimp on here. On my workstation at home (3.33 GHz quad core Xeon) Sage is taking only a few seconds. Is it possible to open these files read-only? I don't know if that could save time on any file systems? I could image on something like NFS, it could be potentially quicker if the system does not have to consider the possibility Sage might modify the files. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: profiling Sage startup
Sage could follow the same technique used by Gimp. Gimp loads a huge number of files at startup and can take over a minute to start on small systems. They have a startup splash screen that shows the files which are being loaded. That way the user sees progress during startup. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Error installing mpir-1.2.2.p2 (building sage-4.6.1/sage-4.6.2) on (32bit) Core Duo MacBook running OS X 10.6
Hi there, so I'm running Mac OS X 10.6 (Snow Leopard) on a first generation MacBook (Yeah, the one with the old 32bit Intel Core Duo processor). Now obviously I cannot run the precompiled sage-4.6.1-OSX-64bit-10.6-i386-Darwin binary, which is why I chose to build sage from source. During the build process (of sage-4.6.1) I ran into the following error: Making all in mpn /bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo fib_table | sed 's/_$//'`-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c -o fib_table.lo fib_table.c mkdir .libs gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_fib_table -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c fib_table.c -fno-common -DPIC -o .libs/fib_table.o /bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo mp_bases | sed 's/_$//'`-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c -o mp_bases.lo mp_bases.c gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_mp_bases -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c mp_bases.c -fno-common -DPIC -o .libs/mp_bases.o /bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo add | sed 's/_$//'`-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c -o add.lo add.c gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_add -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c add.c -fno-common -DPIC -o .libs/add.o /bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo add_1 | sed 's/_$//'`-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c -o add_1.lo add_1.c gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_add_1 -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c add_1.c -fno-common -DPIC -o .libs/add_1.o /bin/sh ../libtool --mode=compile --tag=CC ../mpn/m4-ccas --m4="m4" gcc -std=gnu99 -c -DHAVE_CONFIG_H-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo add_n | sed 's/_$//'` -I. -I. -I.. `test -f 'add_n.asm' || echo './'`add_n.asm ../mpn/m4-ccas --m4=m4 gcc -std=gnu99 -c -DHAVE_CONFIG_H -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -D__GMP_WITHIN_GMP -I.. -DOPERATION_add_n -I. -I. -I.. add_n.asm -fno-common -DPIC -o .libs/add_n.o m4 -DHAVE_CONFIG_H -D__GMP_WITHIN_GMP -DOPERATION_add_n -DPIC add_n.asm >tmp-add_n.s gcc -std=gnu99 -c -DHAVE_CONFIG_H -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -D__GMP_WITHIN_GMP -I.. -DOPERATION_add_n -I. -I. -I.. tmp-add_n.s -fno-common -DPIC -o .libs/add_n.o /bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo sub | sed 's/_$//'`-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c -o sub.lo sub.c gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_sub -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c sub.c -fno-common -DPIC -o .libs/sub.o /bin/sh ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo sub_1 | sed 's/_$//'`-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c -o sub_1.lo sub_1.c gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_sub_1 -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -c sub_1.c -fno-common -DPIC -o .libs/sub_1.o /bin/sh ../libtool --mode=compile --tag=CC ../mpn/m4-ccas --m4="m4" gcc -std=gnu99 -c -DHAVE_CONFIG_H-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo sub_n | sed 's/_$//'` -I. -I. -I.. `test -f 'sub_n.asm' || echo './'`sub_n.asm ../mpn/m4-ccas --m4=m4 gcc -std=gnu99 -c -DHAVE_CONFIG_H -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -D__GMP_WITHIN_GMP -I.. -DOPERATION_sub_n -I. -I. -I.. sub_n.asm -fno-common -DPIC -o .libs/sub_n.o m4 -DHAVE_CONFIG_H -D__GMP_WITHIN_GMP -DOPERATION_sub_n -DPIC sub_n.asm >tmp-sub_n.s gcc -std=gnu99 -c -DHAVE_CONFIG_H -m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -D__GMP_WITHIN_GMP -I.. -DOPERATION_sub_n -I. -I. -I.. tmp-sub_n.s -fno-common -DPIC -o .libs/sub_n.o /bin/sh ../libtool --mode=compile --tag=CC ../mpn/m4-ccas --m4="m4" gcc -std=gnu99 -c -DHAVE_CONFIG_H-m32 -O2 -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo mul_1 | sed 's/_$//'` -I. -I. -I.. `test -f 'mul_1.asm' || echo './'`mul_1.asm ../mpn/m4-ccas --m4=m4 gcc -st
Re: [sage-devel] Re: Making Google Analytics Public
I didn't mean to suggest changing systems. Both Analytics and Piwik are installed on my sites. Posting summaries would be nice. Would you be willing to give individual viewing access? http://www.google.com/support/analytics/bin/answer.py?hl=en&answer=55500 On Tue, Mar 1, 2011 at 4:34 AM, Harald Schilly wrote: > I'm pretty sure there are some legal reasons why analytics is not public and > changing our analytics system after we have collected years of data is not > wise (apart from the fact that we would need to maintain yet another > system). There are weekly summary reports that we could publish online, how > about that? > H > > -- > To post to this group, send an email to sage-devel@googlegroups.com > To unsubscribe from this group, send an email to > sage-devel+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/sage-devel > URL: http://www.sagemath.org > -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: profiling Sage startup
On Mar 1, 3:43 pm, "Johan S. R. Nielsen"
wrote:
> On Mar 1, 1:56 pm, luisfe wrote:
>
>
>
>
>
> > No, the lazy_import object keeps wrapping the original object, but
> > when accessing the lazy_import object it imports the real object in
> > the namespace. So, if the lazy_import has not been referenced by
> > another object then it will be collected as garbage. I do not have a
> > running sage right now, but if you experiment you will see that after
> > calling the object the lazy_import object disappears.
>
> > It should be something like:
>
> > {{{
> > sage: from sage.misc.lazy_import import lazy_import
> > sage: lazy_import('sage.rings.all', 'ZZ')
> > sage: a = ZZ
> > sage: type(ZZ)
> >
> > sage: ZZ(4.0)
> > 4
> > sage: type(ZZ)
> >
> > sage: type(a)
> >
>
> > }}}
>
> I just did the above in 4.6.1 and got:
> {{{
> sage: lazy_import('sage.rings.all','ZZ')
> sage: a = ZZ
> sage: type(ZZ)
>
> sage: ZZ(4.0)
> 4
> sage: type(ZZ)
> }}}
>
> And naturally, the same type for a. It seems nothing I do to the ZZ
> object will unwrap it from the LazyImporter; otherwise, what you say
> would naturally be true, and that was exactly what I thought would be
> neat to do. Even though the wrapping is very thin, if each and every
> callable was wrapped as such, it would probably be quite a large
> performance-penalty.
>
> Johan
Oops, my mistake, I am using a developper version. Try with sage-4.6.2
(to be released this week) the improvement I am talking about was
merged in 4.6.2.alpha3
Also lazy_import('sage.rings.all','ZZ') does not need disk access, is
lazy_import('sage.rings'all','*') what needs to read the file.
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
[sage-devel] Re: profiling Sage startup
On Mar 1, 1:56 pm, luisfe wrote: > > Yes, it is not used much. Ideally, the way of improving sage > startuptime would be: > > 1.- On first execution by a user, create a pickle of lazy_import of > the global names and save it in the .sage directory. This should not > be slower than current startup. > > 2.- Then, following executions of sage would only load that pickle > instead of reading hundreds of different files. This will translates > the delay from the start to the first call of the function. But note > that you will never call each global name so globally should be > faster... > > 3.- That pickle should be updated under certain conditions. By a > command, each time sage is updated etc. This idea seems to be related to the zip-file and is orthogonal to which callables should be imported lazily. Is this faster than the zip- file? Perhaps one could zip all the pickles, unzip the file in memory and unpickle from this stream ;-) > > Note that currently there are lazy_import(...) in the startup of sage, > but this command does not avoid disk access of the file, only > execution/compilation of the module to import. That is why it has > currently low impact. I might have overlooked something, but as far as I can see, lazy_import does no disk access until the lazily imported callable is accessed. The initialiser of the object does nothing but set a few fields, and only in the private _get_object is the actual import done; this function is called whenever some access is needed. Anyway, a lazy importer should and could surely have this feature. Also, where is the lazy_import used, apart from in sage.sets.family and sage.categories.sets_cat? Johan -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: mpmath and plot incompatibility
On 3/1/11 3:58 AM, Fredrik Johansson wrote: On Tue, Mar 1, 2011 at 4:30 AM, Francois Bissey wrote: On Mon, 28 Feb 2011 at 01:26PM -0800, jtyard wrote: I'm running sage 4.6.1 and cannot use plot after loading mpmath. Namely, running from mpmath import * plot(lambda t: sin(2*pi*t), [1, 4]) produces no output in the notebook. I'm loading mpmath because I need to make plots of theta functions, and the page http://mpmath.googlecode.com/svn/trunk/doc/build/functions/elliptic.html# jacobi-theta-functions tells me to first run "from mpmath import *" to gain access to jtheta. I'm not 100% sure what's going on, but I do see that mpmath has its own plot function, and your "from mpmath import *" line overwrites Sage's usual plot function with mpmath's. This works: sage: import mpmath sage: plot(lambda t: mpmath.sin(2*pi*t), (1,4)) So I guess the question is, why isn't mpmath's plot function working? Interestingly mpmath-0.16 and 0.17 have support for matplotlib so presumably those can use matplotlib for plot. mpmath basically justs calls pylab.show(), which doesn't do anything in the notebook (nor in the Sage interpreter). We could hack mpmath in Sage to be more clever. Alternatively, perhaps a better solution would to make pylab.show() work in Sage. Is this feasible? If you compile with SAGE_MATPLOTLIB_GUI (and you have the prerequisites so that one of the GUI backends is built), then I believe pylab.show() works just fine. Jason -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: profiling Sage startup
On Mar 1, 1:56 pm, luisfe wrote:
>
> No, the lazy_import object keeps wrapping the original object, but
> when accessing the lazy_import object it imports the real object in
> the namespace. So, if the lazy_import has not been referenced by
> another object then it will be collected as garbage. I do not have a
> running sage right now, but if you experiment you will see that after
> calling the object the lazy_import object disappears.
>
> It should be something like:
>
> {{{
> sage: from sage.misc.lazy_import import lazy_import
> sage: lazy_import('sage.rings.all', 'ZZ')
> sage: a = ZZ
> sage: type(ZZ)
>
> sage: ZZ(4.0)
> 4
> sage: type(ZZ)
>
> sage: type(a)
>
>
> }}}
I just did the above in 4.6.1 and got:
{{{
sage: lazy_import('sage.rings.all','ZZ')
sage: a = ZZ
sage: type(ZZ)
sage: ZZ(4.0)
4
sage: type(ZZ)
}}}
And naturally, the same type for a. It seems nothing I do to the ZZ
object will unwrap it from the LazyImporter; otherwise, what you say
would naturally be true, and that was exactly what I thought would be
neat to do. Even though the wrapping is very thin, if each and every
callable was wrapped as such, it would probably be quite a large
performance-penalty.
Johan
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
[sage-devel] Re: profiling Sage startup
On Mar 1, 1:32 pm, "Johan S. R. Nielsen"
wrote:
> On Mar 1, 10:13 am, Robert Bradshaw
> wrote:
>
> Nice! I weren't aware of this module. When you get a good idea,
> there's a good chance that someone else thought of it before ;-) I
> like the fact that one can dynamically hack into an object's
> namespace :-D However, lazy_import seems not to be used much (only 2-3
> places) currently in the Sage startup (or did I grep wrongly?). Was it
> never the intention or is it due to the overhead?
Yes, it is not used much. Ideally, the way of improving sage
startuptime would be:
1.- On first execution by a user, create a pickle of lazy_import of
the global names and save it in the .sage directory. This should not
be slower than current startup.
2.- Then, following executions of sage would only load that pickle
instead of reading hundreds of different files. This will translates
the delay from the start to the first call of the function. But note
that you will never call each global name so globally should be
faster...
3.- That pickle should be updated under certain conditions. By a
command, each time sage is updated etc.
Note that currently there are lazy_import(...) in the startup of sage,
but this command does not avoid disk access of the file, only
execution/compilation of the module to import. That is why it has
currently low impact.
> Also, I didn't proofread the entire lazy_import code, but the
> implementation seems to differ from my idea in a significant way: the
> LazyImport object keeps wrapping the imported objects. My thought was
> that the first time the imported object was accessed, it would replace
> itself with the original in the global namespace, and then forward the
> call. This way, there will be zero overhead in all later calls.
No, the lazy_import object keeps wrapping the original object, but
when accessing the lazy_import object it imports the real object in
the namespace. So, if the lazy_import has not been referenced by
another object then it will be collected as garbage. I do not have a
running sage right now, but if you experiment you will see that after
calling the object the lazy_import object disappears.
It should be something like:
{{{
sage: from sage.misc.lazy_import import lazy_import
sage: lazy_import('sage.rings.all', 'ZZ')
sage: a = ZZ
sage: type(ZZ)
sage: ZZ(4.0)
4
sage: type(ZZ)
sage: type(a)
}}}
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
[sage-devel] Re: Making Google Analytics Public
I'm pretty sure there are some legal reasons why analytics is not public and changing our analytics system after we have collected years of data is not wise (apart from the fact that we would need to maintain yet another system). There are weekly summary reports that we could publish online, how about that? H -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: profiling Sage startup
On Mar 1, 10:13 am, Robert Bradshaw
wrote:
> On Tue, Mar 1, 2011 at 12:48 AM, Johan S. R. Nielsen
>
>
>
> wrote:
> > On Feb 23, 11:03 pm, Jason Grout wrote:
> >> On 2/23/11 3:56 PM, Robert Bradshaw wrote:
>
> >> > On Wed, Feb 23, 2011 at 1:47 PM, Jason Grout
> >> > wrote:
> >> >> On 2/23/11 3:06 PM, Robert Bradshaw wrote:
>
> >> >>> On Wed, Feb 23, 2011 at 11:34 AM, William Stein
> >> >>> wrote:
>
> >> On Wed, Feb 23, 2011 at 10:57 AM, Jason Grout
> >> wrote:
>
> >> > On 2/23/11 12:28 PM, William Stein wrote:
>
> >> >> At lunch yesterday Robert Bradshaw made the interesting suggestion
> >> >> to
> >> >> read the docs for importlib
> >> >> (http://docs.python.org/dev/library/importlib.html) and write a
> >> >> customized import hook, so that every time during Sage startup that
> >> >> a
> >> >> module is imported, the import is done from a single big in-memory
> >> >> zip
> >> >> file instead of done using the filesystem. If this can be made to
> >> >> work, it would be a huge win for slow filesystems. The basic
> >> >> problem
> >> >> is that some filesystems are fast but have huge*latency*.
>
> >> > Is it a big win primarily because the zip file contents can be read
> >> > in
> >> > and
> >> > cached by us? I'm just trying to understand it better.
>
> >> Which would you rather do on a high latency filesystem:
>
> >> (1) Read/stat 20,000 little files, or
> >> (2) Read exactly one 40MB file.
>
> >> > Is this the same idea as Jar files in java?
>
> >> I don't know.
>
> >> >>> Yep. In that case the "high latency file system" was a webserver.
>
> >> > You mean likehttp://docs.python.org/library/zipimport.html?
>
> >> Cool.
>
> >> >>> Note that this should just involve putting the zip file first in the
> >> >>> python path.
>
> >> I don't know for a fact that Robert Bradshaw's suggestion will be a
> >> big win, since nobody has tried this yet. But I'm optimistic. The
> >> idea would be to make a zip archive of
> >> $SAGE_ROOT/local/lib/python/site-packages (say), and do *all* imports
> >> using that massive zip archive.
>
> >> >>> I'm optimistic too. This would, of course, make more sense for
> >> >>> system-wide installs than development versions, but the former are
> >> >>> more likely to be on a non-local filesystem anyways.
>
> >> >> Sounds like it is time for a trial!
>
> >> >> I created a directory of 2000 .py files and an __init__.py file to make
> >> >> it a
> >> >> module
>
> >> >> for i in range(2000):
> >> >> with open('importtest/test_%s.py'%i,'w') as f:
> >> >> f.write("VALUE=%s\n"%i)
> >> >> with open('importtest/__init__.py','w') as f:
> >> >> f.write(' ')
>
> >> >> Then I imported each of these so that .pyc files were created.
>
> >> >> for i in range(2000):
> >> >> exec 'import importtest.test_%s'%i
>
> >> >> Okay, then I copied the directory and zipped it up (in the shell now):
>
> >> >> $ cp -r importtest zipimporttest
> >> >> $ zip -r tmp.zip zipimporttest
> >> >> $ rm -rf zipimporttest
>
> >> >> One nice side effect is that the zip file is less than one MB, while the
> >> >> directory of python files is around 16M.
>
> >> >> Now for the test. Here are my two scripts. One imports each module in
> >> >> the
> >> >> directory and adds up the VALUE in each module:
>
> >> >> % cat mytest.py
> >> >> s=0
> >> >> for i in range(2000):
> >> >> exec 'import importtest.test_%s as tt'%i
> >> >> s+=tt.VALUE
> >> >> print s
>
> >> >> The other first adds the zip to the front of sys.path and then does the
> >> >> same
> >> >> imports and summing, but using the zipped module:
>
> >> >> % cat mytestzip.py
> >> >> import sys
> >> >> sys.path.insert(0,'./tmp.zip')
> >> >> s=0
> >> >> for i in range(2000):
> >> >> exec 'import zipimporttest.test_%s as tt'%i
> >> >> s+=tt.VALUE
> >> >> print s
>
> >> >> And now for the timings:
>
> >> >> % time sage -python mytest.py
> >> >> Detected SAGE64 flag
> >> >> Building Sage on OS X in 64-bit mode
> >> >> 1999000
> >> >> sage -python mytest.py 0.26s user 1.47s system 75% cpu 2.282 total
>
> >> >> % time sage -python mytestzip.py
> >> >> Detected SAGE64 flag
> >> >> Building Sage on OS X in 64-bit mode
> >> >> 1999000
> >> >> sage -python mytestzip.py 0.21s user 0.11s system 99% cpu 0.327 total
>
> >> >> It looks like the zip is a clear winner in this case. And this is with
> >> >> the
> >> >> directory presumably in the FS cache.
>
> >> > Cool. Given the CPU was pegged at 99%, have you tried using an
> >> > uncompressed zip file? It'd have more data to read, but less to do
> >> > with it once it's read.
>
> >> In my case, using zip -0 (no compression) gives:
>
> >> % time sage -python mytestzip.py
> >> Detected SAGE64 flag
> >> Building Sage on OS X in 64-bit mode
> >> 1999000
> >> sage -python mytestzip.py 0.20s user 0.10s system 99% cpu 0.309 total
>
> >> S
Re: [sage-devel] mpmath and plot incompatibility
On Tue, Mar 1, 2011 at 4:30 AM, Francois Bissey wrote: >> On Mon, 28 Feb 2011 at 01:26PM -0800, jtyard wrote: >> > I'm running sage 4.6.1 and cannot use plot after loading mpmath. >> > Namely, running >> > >> > > from mpmath import * >> > > plot(lambda t: sin(2*pi*t), [1, 4]) >> > >> > produces no output in the notebook. I'm loading mpmath because I need >> > to make plots of theta functions, and the page >> > >> > >> > http://mpmath.googlecode.com/svn/trunk/doc/build/functions/elliptic.html# >> > jacobi-theta-functions >> > >> > tells me to first run "from mpmath import *" to gain access to jtheta. >> >> I'm not 100% sure what's going on, but I do see that mpmath has its own >> plot function, and your "from mpmath import *" line overwrites Sage's >> usual plot function with mpmath's. >> >> This works: >> >> sage: import mpmath >> sage: plot(lambda t: mpmath.sin(2*pi*t), (1,4)) >> >> So I guess the question is, why isn't mpmath's plot function working? >> > Interestingly mpmath-0.16 and 0.17 have support for matplotlib so presumably > those can use matplotlib for plot. mpmath basically justs calls pylab.show(), which doesn't do anything in the notebook (nor in the Sage interpreter). We could hack mpmath in Sage to be more clever. Alternatively, perhaps a better solution would to make pylab.show() work in Sage. Is this feasible? Fredrik -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: profiling Sage startup
On Tue, Mar 1, 2011 at 12:48 AM, Johan S. R. Nielsen
wrote:
> On Feb 23, 11:03 pm, Jason Grout wrote:
>> On 2/23/11 3:56 PM, Robert Bradshaw wrote:
>>
>>
>>
>> > On Wed, Feb 23, 2011 at 1:47 PM, Jason Grout
>> > wrote:
>> >> On 2/23/11 3:06 PM, Robert Bradshaw wrote:
>>
>> >>> On Wed, Feb 23, 2011 at 11:34 AM, William Stein
>> >>> wrote:
>>
>> On Wed, Feb 23, 2011 at 10:57 AM, Jason Grout
>> wrote:
>>
>> > On 2/23/11 12:28 PM, William Stein wrote:
>>
>> >> At lunch yesterday Robert Bradshaw made the interesting suggestion to
>> >> read the docs for importlib
>> >> (http://docs.python.org/dev/library/importlib.html) and write a
>> >> customized import hook, so that every time during Sage startup that a
>> >> module is imported, the import is done from a single big in-memory zip
>> >> file instead of done using the filesystem. If this can be made to
>> >> work, it would be a huge win for slow filesystems. The basic problem
>> >> is that some filesystems are fast but have huge*latency*.
>>
>> > Is it a big win primarily because the zip file contents can be read in
>> > and
>> > cached by us? I'm just trying to understand it better.
>>
>> Which would you rather do on a high latency filesystem:
>>
>> (1) Read/stat 20,000 little files, or
>> (2) Read exactly one 40MB file.
>>
>> > Is this the same idea as Jar files in java?
>>
>> I don't know.
>>
>> >>> Yep. In that case the "high latency file system" was a webserver.
>>
>> > You mean likehttp://docs.python.org/library/zipimport.html?
>>
>> Cool.
>>
>> >>> Note that this should just involve putting the zip file first in the
>> >>> python path.
>>
>> I don't know for a fact that Robert Bradshaw's suggestion will be a
>> big win, since nobody has tried this yet. But I'm optimistic. The
>> idea would be to make a zip archive of
>> $SAGE_ROOT/local/lib/python/site-packages (say), and do *all* imports
>> using that massive zip archive.
>>
>> >>> I'm optimistic too. This would, of course, make more sense for
>> >>> system-wide installs than development versions, but the former are
>> >>> more likely to be on a non-local filesystem anyways.
>>
>> >> Sounds like it is time for a trial!
>>
>> >> I created a directory of 2000 .py files and an __init__.py file to make
>> >> it a
>> >> module
>>
>> >> for i in range(2000):
>> >> with open('importtest/test_%s.py'%i,'w') as f:
>> >> f.write("VALUE=%s\n"%i)
>> >> with open('importtest/__init__.py','w') as f:
>> >> f.write(' ')
>>
>> >> Then I imported each of these so that .pyc files were created.
>>
>> >> for i in range(2000):
>> >> exec 'import importtest.test_%s'%i
>>
>> >> Okay, then I copied the directory and zipped it up (in the shell now):
>>
>> >> $ cp -r importtest zipimporttest
>> >> $ zip -r tmp.zip zipimporttest
>> >> $ rm -rf zipimporttest
>>
>> >> One nice side effect is that the zip file is less than one MB, while the
>> >> directory of python files is around 16M.
>>
>> >> Now for the test. Here are my two scripts. One imports each module in
>> >> the
>> >> directory and adds up the VALUE in each module:
>>
>> >> % cat mytest.py
>> >> s=0
>> >> for i in range(2000):
>> >> exec 'import importtest.test_%s as tt'%i
>> >> s+=tt.VALUE
>> >> print s
>>
>> >> The other first adds the zip to the front of sys.path and then does the
>> >> same
>> >> imports and summing, but using the zipped module:
>>
>> >> % cat mytestzip.py
>> >> import sys
>> >> sys.path.insert(0,'./tmp.zip')
>> >> s=0
>> >> for i in range(2000):
>> >> exec 'import zipimporttest.test_%s as tt'%i
>> >> s+=tt.VALUE
>> >> print s
>>
>> >> And now for the timings:
>>
>> >> % time sage -python mytest.py
>> >> Detected SAGE64 flag
>> >> Building Sage on OS X in 64-bit mode
>> >> 1999000
>> >> sage -python mytest.py 0.26s user 1.47s system 75% cpu 2.282 total
>>
>> >> % time sage -python mytestzip.py
>> >> Detected SAGE64 flag
>> >> Building Sage on OS X in 64-bit mode
>> >> 1999000
>> >> sage -python mytestzip.py 0.21s user 0.11s system 99% cpu 0.327 total
>>
>> >> It looks like the zip is a clear winner in this case. And this is with
>> >> the
>> >> directory presumably in the FS cache.
>>
>> > Cool. Given the CPU was pegged at 99%, have you tried using an
>> > uncompressed zip file? It'd have more data to read, but less to do
>> > with it once it's read.
>>
>> In my case, using zip -0 (no compression) gives:
>>
>> % time sage -python mytestzip.py
>> Detected SAGE64 flag
>> Building Sage on OS X in 64-bit mode
>> 1999000
>> sage -python mytestzip.py 0.20s user 0.10s system 99% cpu 0.309 total
>>
>> So just a slight savings.
>>
>> Jason
>
>
> I had an orthorgonal thought, though I'm not sure it's completely
> possible. Insted of actually loading the real functions/classes etc.,
> couldn't we fast-load (or generate) stub-versions of all these, which
> when called would load
[sage-devel] Re: profiling Sage startup
On Feb 23, 11:03 pm, Jason Grout wrote:
> On 2/23/11 3:56 PM, Robert Bradshaw wrote:
>
>
>
> > On Wed, Feb 23, 2011 at 1:47 PM, Jason Grout
> > wrote:
> >> On 2/23/11 3:06 PM, Robert Bradshaw wrote:
>
> >>> On Wed, Feb 23, 2011 at 11:34 AM, William Stein
> >>> wrote:
>
> On Wed, Feb 23, 2011 at 10:57 AM, Jason Grout
> wrote:
>
> > On 2/23/11 12:28 PM, William Stein wrote:
>
> >> At lunch yesterday Robert Bradshaw made the interesting suggestion to
> >> read the docs for importlib
> >> (http://docs.python.org/dev/library/importlib.html) and write a
> >> customized import hook, so that every time during Sage startup that a
> >> module is imported, the import is done from a single big in-memory zip
> >> file instead of done using the filesystem. If this can be made to
> >> work, it would be a huge win for slow filesystems. The basic problem
> >> is that some filesystems are fast but have huge*latency*.
>
> > Is it a big win primarily because the zip file contents can be read in
> > and
> > cached by us? I'm just trying to understand it better.
>
> Which would you rather do on a high latency filesystem:
>
> (1) Read/stat 20,000 little files, or
> (2) Read exactly one 40MB file.
>
> > Is this the same idea as Jar files in java?
>
> I don't know.
>
> >>> Yep. In that case the "high latency file system" was a webserver.
>
> > You mean likehttp://docs.python.org/library/zipimport.html?
>
> Cool.
>
> >>> Note that this should just involve putting the zip file first in the
> >>> python path.
>
> I don't know for a fact that Robert Bradshaw's suggestion will be a
> big win, since nobody has tried this yet. But I'm optimistic. The
> idea would be to make a zip archive of
> $SAGE_ROOT/local/lib/python/site-packages (say), and do *all* imports
> using that massive zip archive.
>
> >>> I'm optimistic too. This would, of course, make more sense for
> >>> system-wide installs than development versions, but the former are
> >>> more likely to be on a non-local filesystem anyways.
>
> >> Sounds like it is time for a trial!
>
> >> I created a directory of 2000 .py files and an __init__.py file to make it
> >> a
> >> module
>
> >> for i in range(2000):
> >> with open('importtest/test_%s.py'%i,'w') as f:
> >> f.write("VALUE=%s\n"%i)
> >> with open('importtest/__init__.py','w') as f:
> >> f.write(' ')
>
> >> Then I imported each of these so that .pyc files were created.
>
> >> for i in range(2000):
> >> exec 'import importtest.test_%s'%i
>
> >> Okay, then I copied the directory and zipped it up (in the shell now):
>
> >> $ cp -r importtest zipimporttest
> >> $ zip -r tmp.zip zipimporttest
> >> $ rm -rf zipimporttest
>
> >> One nice side effect is that the zip file is less than one MB, while the
> >> directory of python files is around 16M.
>
> >> Now for the test. Here are my two scripts. One imports each module in the
> >> directory and adds up the VALUE in each module:
>
> >> % cat mytest.py
> >> s=0
> >> for i in range(2000):
> >> exec 'import importtest.test_%s as tt'%i
> >> s+=tt.VALUE
> >> print s
>
> >> The other first adds the zip to the front of sys.path and then does the
> >> same
> >> imports and summing, but using the zipped module:
>
> >> % cat mytestzip.py
> >> import sys
> >> sys.path.insert(0,'./tmp.zip')
> >> s=0
> >> for i in range(2000):
> >> exec 'import zipimporttest.test_%s as tt'%i
> >> s+=tt.VALUE
> >> print s
>
> >> And now for the timings:
>
> >> % time sage -python mytest.py
> >> Detected SAGE64 flag
> >> Building Sage on OS X in 64-bit mode
> >> 1999000
> >> sage -python mytest.py 0.26s user 1.47s system 75% cpu 2.282 total
>
> >> % time sage -python mytestzip.py
> >> Detected SAGE64 flag
> >> Building Sage on OS X in 64-bit mode
> >> 1999000
> >> sage -python mytestzip.py 0.21s user 0.11s system 99% cpu 0.327 total
>
> >> It looks like the zip is a clear winner in this case. And this is with the
> >> directory presumably in the FS cache.
>
> > Cool. Given the CPU was pegged at 99%, have you tried using an
> > uncompressed zip file? It'd have more data to read, but less to do
> > with it once it's read.
>
> In my case, using zip -0 (no compression) gives:
>
> % time sage -python mytestzip.py
> Detected SAGE64 flag
> Building Sage on OS X in 64-bit mode
> 1999000
> sage -python mytestzip.py 0.20s user 0.10s system 99% cpu 0.309 total
>
> So just a slight savings.
>
> Jason
I had an orthorgonal thought, though I'm not sure it's completely
possible. Insted of actually loading the real functions/classes etc.,
couldn't we fast-load (or generate) stub-versions of all these, which
when called would load and replace themselves with the real version
and then run it. I'm not completely sure it's possible with Python,
but Python is pretty flexible so perhaps there is a way; in
particular, I don't know how Python supports reflect
[sage-devel] Re: Documentation of prcedures with decorator
I don't know what influence there is from Cython, but for general python-functions, decorators need to decorate their return functions with @sage_wraps (see e.g. sage.plots.contour_plot.contour_plot) to get the correct doc-strings. Setting these could be done manually in the decorator, but it is the intention that sage_wraps should coherently take care of all this and related stuff, so this should always be used instead of manually setting these. If decorated Cython functions are special, perhaps we should make a Cython-version of sage_wraps to fix this in the same, coherent manner. Even when using sage_wraps, Sphinx still uses the argument strings of the decorator when compiling the documentation. I posted a patch (Trac #9976) for this several months ago, which patches Sphinx and enhances the capabilities of sage_wraps to fix this, but there hasn't been a reviewer. Regards, Johan On Mar 1, 8:48 am, Robert Bradshaw wrote: > On Mon, Feb 28, 2011 at 11:39 PM, Simon King wrote: > > Hi Robert, > > > On 1 Mrz., 00:15, Robert Bradshaw > > wrote: > >> ... > >> > Namely, cached_method tries to obtain certain attributes common to > >> > python functions from its argument. func_defaults is just one among > >> > others. > > >> We don't have as much freedom here, as C-defined and python-defined > >> functions are quite different (standard library) objects. In this > >> case, cached_method should recognize > >> method_descriptors/builtin_function_or_method types and handle them > >> differently (if possible). > > > Where is the source code for method_descriptor or > > builtin_function_or_method? What is needed to be cimported from where, > > in order to access the attributes corresponding to func_defaults and > > friends? > > The source code is in the Python library itself, > e.g.http://svn.python.org/projects/python/trunk/Objects/methodobject.c > There's no direct analogue to many of these fields... > > - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
