On Saturday, September 16, 2017 at 6:18:17 AM UTC-7, Simon King wrote:
>
> Hi Thierry,
>
> On 2017-09-16, Thierry <sage-goo...@lma.metelu.net <javascript:>> wrote:
> > How does raw "exec" behaves with your large file ?
> >
> > sage: with open('your_file.txt') as f:
> > ....: exec(preparse(f.read()))
>
> Time for my preparser that translates the gap readable into
> Python readable data is about 3 minutes. The result is a string
> s that I am using in the following.
>
> Time for Sage's preparser:
> sage: %time sp = preparse("D = "+s)
> CPU times: user 4min 33s, sys: 1.08 s, total: 4min 34s
> Wall time: 4min 34s
>
> The attempt to exec(sp) soon exhausted 15 GB, Ctrl-C didn't work,
> and my laptop swapped so badly that I couldn't even open a new
> terminal to do "pkill python".
>
> (if you're in a terminal you have a chance that "ctrl-Z" does work, which
stops the process. You can then "kill %1" or something like that)
I've tried some benchmarks along the lines of
sage: %time s=preparse("s="+(dict((i,i) for i in range(100000))).__str__())
CPU times: user 1.13 s, sys: 6.23 ms, total: 1.14 s
Wall time: 1.13 s
sage: %time exec(s)
CPU times: user 773 ms, sys: 98.1 ms, total: 871 ms
Wall time: 864 ms
The main thing that I noticed was that execution time (in both steps)
scales about linearly in the length, so the python parser seems to be using
an algorithm of the right order.
Note that "exec" will *compile* the expression (i.e., write a straight-line
program for the cpython virtual machine to assemble the data structure) and
then execute it, so it needs at least twice the memory.
Running the code doesn't seem to be the bottleneck, though:
sage: %time s=preparse("def L(): return "+(dict((i,i) for i in
range(100000))).__str__())
CPU times: user 1.15 s, sys: 20.3 ms, total: 1.17 s
Wall time: 1.16 s
sage: %time exec(s)
CPU times: user 772 ms, sys: 80.7 ms, total: 853 ms
Wall time: 857 ms
sage: %time v=L()
CPU times: user 47.4 ms, sys: 5.4 ms, total: 52.8 ms
Wall time: 45.9 ms
I don't know if the performance above indicates acceptable behaviour for
your use case. If it doesn't then it would seem the python parsing strategy
is just not suitable for your application. If it does, then there is
something else that's slow. Perhaps running the code exposes some
inefficient constructors in sage?
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
