On Sun, Aug 23, 2009 at 10:37 PM, Brian Granger <ellisonbg....@gmail.com>wrote:
> William,
>
> Thanks for clarifying some of the details of pexpect. I do really want to
> understand this because I am starting to use the notebook more and currently
> IPython's parallel stuff works fine (there are a few things that need to be
> fixed on our side to make it easier though).
>
> Moreover, as long as the worksheet and the notebook server are
>> distinct processes (as they should be, IMHO), the difference between
>> using pexpect, or xmlrpc, or anything else, for them to communicate is
>> completely and totally irrelevant, since it is a black box to the
>> entire rest of the program.
>>
>
> I agree with you that for the rest of the program (the notebook) this
> detail is completely hidden. But I guess I don't quite follow your
> statement that the differences of using pexpect/twisted to manage this are
> irrelevant. In my mind there is a big different between pexpect and
> twisted:
>
Sorry. I really only meant to convey precisely the part of the statement
that you agree with. I didn't express myself clearly enough. I should
have added that this is only irrelevant for the rest of the notebook code.
That's all I meant.
>
>
> * pexpect simply controls and observes the worksheet process (I now
> understand that this can be asynchronous). The worksheet process doesn't
> have *any* custom code to enable this to work and probably doesn't even
> import pexpect (unless it does so for a completely different reason - like
> talking to Mathematica, etc.).
>
That is correct. The worksheet processes are pure Python processes that do
not need to use pexpect at all themselves, except to talk to other systems
of they so choose. Also, the communication is precisely as powerful as
communication between *you* and a program you are using interactively at the
command line. For example, if you are sitting there using the program,
then you go make a cup of tea, the program can't do anything at all to get
your attention. In the same way, a worksheet process under pexpect will be
completely ignored by the notebook server process until the notebook server
process chooses to look at the worksheet process.
>
>
> * To get Twisted to make two processes talk over TCP/IP (I am ignoring
> Twisted's ability to talk to a process in the same manner as pexpect, which
> I think it might be able to do - are you thinking of this?) *both* processes
> must start the Twisted reactor. Thus, if you wanted the notebook server and
> a worksheet process to talk over xmlrpc or pb, the worksheet process must be
> re-designed to run the Twisted reactor. To use IPython or dsage in a
> context like that, the Twisted reactor and user code must be run in
> different threads. This is a super subtle aspect of using Twisted in a
> blocking manner like the dsage and IPython clients do. I can give more
> details about this aspect if needed.
>
That sounds right to me. I have never thought of using twisted in the
same way as pexpect. Yes, using xmlrpc would require putting the worksheet
process in a special listening mode and would either require two threads or
having the worksheet's side of xmlrpc simply go dead whenever the worksheet
is working, which would be completely unacceptable given that users need to
see the output as it is produced. So yes, two threads are needed.
Also, to correct another possible misconception, communication between
>> a processes and a subprocess using pexpect is not blocking. The
>> master processes can listen for however long it wants to the
>> subprocess, then stop listening. That's why when you do
>>
>> for i in range(10):
>> sleep(1)
>> print(i)
>>
>> in the Sage notebook, you see the output as it is computed. The
>> notebook server just uses pexpect to "peak" at the output of the
>> subprocess doing the actual work and look to see what has been output
>> so far.
>>
>
> Not that you mention this I vaguely remember this about pexpect.
>
>
>> Another misconception is that pexpect is restricted to local
>> processes. It's easy to control a process via pexpect over the
>> network via ssh. This has been in Sage since 2005, and can already
>> be used for worksheet subprocesses *now* as long as you have a shared
>> filesystem (just use the server_pool option). Here is an example on
>> the command line. I have ssh keys setup so I can do "ssh
>> sage.math.washington.edu" and login without typing a password. I
>> start Sage on my laptop in a coffee shop, and make a connection to a
>> remote Sage that gets started running on sage.math, and I run a
>> calculation.
>>
>
> Wow, I had no idea that this was possible! I will definitely keep this in
> mind as it is very nice.
>
>
>> flat:sageuse wstein$ sage
>> ----------------------------------------------------------------------
>> | Sage Version 4.1.1, Release Date: 2009-08-14 |
>> | Type notebook() for the GUI, and license() for information. |
>> ----------------------------------------------------------------------
>> sage: s = Sage(server="sage.math.washington.edu")
>> No remote temporary directory (option server_tmpdir) specified, using
>> /tmp/ on sage.math.washington.edu
>> sage: s.eval("2+2")
>> '4'
>> sage: s.eval("os.system('uname -a')")
>> 'Linux sage.math.washington.edu 2.6.24-23-server #1 SMP Wed Apr 1
>> 22:14:30 UTC 2009 x86_64 GNU/Linux\n0'
>> sage:
>>
>>
>> The above used pexpect. You can even interact with remote objects:
>>
>> sage: e = s("EllipticCurve([1..5])")
>> sage: e.rank()
>> 1
>>
>> You can do the same with Mathematica, etc. by the way:
>>
>> sage: s = Mathematica(server="sage.math.washington.edu")
>> sage: s("Factorial[50]")
>> 30414093201713378043612608166064768844377641568960512000000000000
>>
>>
>> Compare my laptop to sage.math's mathematica:
>>
>> sage: s("Timing[Factorial[10^6]][[1]]") # sage.math
>> 1.1099999999999999
>> sage: mathematica("Timing[Factorial[10^6]][[1]]") # laptop
>> 0.8902620000000001
>>
>> (I guess Mathematica 7.0 is faster at factorials than Mathematica 6.0.)
>>
>> This tests latency:
>>
>> sage: timeit('s.eval("2+2")') # over web via ssh
>> 5 loops, best of 3: 56.3 ms per loop
>> sage: timeit('mathematica.eval("2+2")') # local
>> 625 loops, best of 3: 209 µs per loop
>>
>> Of course latency is long over the net, since I'm in a random coffee shop.
>>
>> This remote server stuff has been in sage since 2005, and hasn't been
>> changed in the slightest bit since then. That's why I'm advertising
>> it now, since it would be cool to see some people work on it and
>> improve it. For example, for people without ssh keys, one could
>> *easily* make it so the following works:
>>
>
> This is truly great stuff and I am glad you are advertising it. I need to
> look at pexpect more because it might come in useful for some things we do
> with IPython. We really like Twisted, but for somethings Twisted doesn't
> make sense.
>
I should expand this out into a nice ReST document and put it in the Sage
reference manual.
> So there is still some potential to the pseudotty approach to
>> controlling processes. The main drawback in my mind is that it
>> works differently (and maybe not so well) on Windows (though it does
>> actually work, but via the "Console API").
>>
>
> Question then: are you planning on continuing to use pexpect to
> communicate between the notebook server and worksheet or are you planning on
> moving to Twisted for that?
>
By far the fastest path to Sage-on-windows right now is via Cygwin, and for
that pexpect works perfectly well. For native MSVC, we did port pexpect,
and it might work. Frankly, I don't really have a plan for the MSVC
notebook communicating with worksheets. Hopefully whatever happens, if it
is a good solution it'll just be a black box somewhere, and it won't impact
anything else in the system. So one could do everything else first.
William
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel-unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://www.sagemath.org
-~----------~----~----~----~------~----~------~--~---
