Re: [sugar] python activities startup

[Tomeu Vizoso]

On Fri, 2008-02-08 at 14:44 +0100, Tomeu Vizoso wrote:
> On Fri, 2008-02-08 at 13:41 +0100, Tomeu Vizoso wrote:
> > On Thu, 2008-02-07 at 16:46 +, Simon McVittie wrote:
> > > A better solution would be for Rainbow to avoid SessionBus()
> entirely,
> > > and instead use an instance of the superclass,
> dbus.bus.BusConnection. This
> > > does not have the weird caching behaviour at all (one call to the
> > > constructor = one instance = one D-Bus connection). The code that
> gets run
> > > after forking is still free to use the shared SessionBus.
> > 
> > Rainbow is not using the session bus at all, just the system one.
> > 
> > But when the child tries to instantiate a SessionBus object, I get
> this stack trace:
> > 
...
> > 
> > : 
> > org.freedesktop.DBus.Error.NoServer: Failed to connect to socket 
> > /tmp/dbus-7dk02BRXOZ: Connection refused
> > 
> > 
> > This is the value of the DBUS_SESSION_BUS_ADDRESS env var:
> > unix:path=/tmp/olpc-session-bus,guid=fdf37ebd35a585d12382a50047ac3034
> > 
> > unix:path=/tmp/olpc-session-bus is specified
> > in /etc/dbus-1/session-olpc.conf:
> > 
> > 
> >   
> >   session
> > 
> >   unix:path=/tmp/olpc-session-bus
> > 
> >   
> > ...
> 
> When running inside rainbow, strace shows this:
> 
> [pid  2370] connect(0, {sa_family=AF_FILE, path=@/tmp/dbus-XQcfxlvcoY}, 23) = 
> -1 ECONNREFUSED (Connection refused)
> 
> But on the console (works fine):
> 
> connect(3, {sa_family=AF_FILE, path="/tmp/olpc-session-bus"}, 23) = 0

I'm sorry about the noise.

When Rainbow was started, the env var DBUS_SESSION_BUS_ADDRESS wasn't
set until the clone() happened. By looking at dbus code, the bus
addresses are initialized at startup, so it kept using the bad address.

Setting DBUS_SESSION_BUS_ADDRESS before running Rainbow fixed this
issue.

So, no known D-Bus problems right now.

Tomeu

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Tomeu Vizoso]

On Fri, 2008-02-08 at 13:41 +0100, Tomeu Vizoso wrote:
> On Thu, 2008-02-07 at 16:46 +, Simon McVittie wrote:
> > A better solution would be for Rainbow to avoid SessionBus() entirely,
> > and instead use an instance of the superclass, dbus.bus.BusConnection. This
> > does not have the weird caching behaviour at all (one call to the
> > constructor = one instance = one D-Bus connection). The code that gets run
> > after forking is still free to use the shared SessionBus.
> 
> Rainbow is not using the session bus at all, just the system one.
> 
> But when the child tries to instantiate a SessionBus object, I get this stack 
> trace:
> 
> /usr/lib/python2.5/site-packages/sugar/presence/presenceservice.py in 
> __init__(self= (sugar+presence+presenceservice+PresenceService at 0x86f1370)>, 
> allow_offline_iface=False)
>  73 
>  74 # Get a connection to the session bus
> ---> 75 self._bus = dbus.SessionBus()
> self._bus = undefined
> global dbus.SessionBus = 
>  76 self._bus.add_signal_receiver(self._name_owner_changed_cb,
>  77 signal_name="NameOwnerChanged",
> 
> /usr/lib/python2.5/site-packages/dbus/_dbus.py in __new__(cls= 'dbus._dbus.SessionBus'>, private=False, mainloop=None)
> 211 """
> 212 return Bus.__new__(cls, Bus.TYPE_SESSION, private=private,
> --> 213mainloop=mainloop)
> mainloop = None
> 214 
> 215 class StarterBus(Bus):
> 
> /usr/lib/python2.5/site-packages/dbus/_dbus.py in __new__(cls= 'dbus._dbus.SessionBus'>, bus_type=0, private=False, mainloop=None)
> 100 raise ValueError('invalid bus_type %s' % bus_type)
> 101 
> --> 102 bus = BusConnection.__new__(subclass, bus_type, 
> mainloop=mainloop)
> bus = undefined
> global BusConnection.__new__ = 
> subclass = 
> bus_type = 0
> mainloop = None
> 103 
> 104 bus._bus_type = bus_type
> 
> /usr/lib/python2.5/site-packages/dbus/bus.py in __new__(cls= 'dbus._dbus.SessionBus'>, address_or_type=0, mainloop=None)
> 123 
> 124 def __new__(cls, address_or_type=TYPE_SESSION, mainloop=None):
> --> 125 bus = cls._new_for_bus(address_or_type, mainloop=mainloop)
> bus = undefined
> cls._new_for_bus =  0x8256f44>
> address_or_type = 0
> mainloop = None
> 126 
> 127 # _bus_names is used by dbus.service.BusName!
> 
> : org.freedesktop.DBus.Error.NoServer: 
> Failed to connect to socket /tmp/dbus-7dk02BRXOZ: Connection refused
> 
> 
> This is the value of the DBUS_SESSION_BUS_ADDRESS env var:
> unix:path=/tmp/olpc-session-bus,guid=fdf37ebd35a585d12382a50047ac3034
> 
> unix:path=/tmp/olpc-session-bus is specified
> in /etc/dbus-1/session-olpc.conf:
> 
> 
>   
>   session
> 
>   unix:path=/tmp/olpc-session-bus
> 
>   
> ...

When running inside rainbow, strace shows this:

[pid  2370] connect(0, {sa_family=AF_FILE, path=@/tmp/dbus-XQcfxlvcoY}, 23) = 
-1 ECONNREFUSED (Connection refused)

But on the console (works fine):

connect(3, {sa_family=AF_FILE, path="/tmp/olpc-session-bus"}, 23) = 0

Tomeu

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Tomeu Vizoso]

On Thu, 2008-02-07 at 16:46 +, Simon McVittie wrote:
> A better solution would be for Rainbow to avoid SessionBus() entirely,
> and instead use an instance of the superclass, dbus.bus.BusConnection. This
> does not have the weird caching behaviour at all (one call to the
> constructor = one instance = one D-Bus connection). The code that gets run
> after forking is still free to use the shared SessionBus.

Rainbow is not using the session bus at all, just the system one.

But when the child tries to instantiate a SessionBus object, I get this stack 
trace:

/usr/lib/python2.5/site-packages/sugar/presence/presenceservice.py in 
__init__(self=, 
allow_offline_iface=False)
 73 
 74 # Get a connection to the session bus
---> 75 self._bus = dbus.SessionBus()
self._bus = undefined
global dbus.SessionBus = 
 76 self._bus.add_signal_receiver(self._name_owner_changed_cb,
 77 signal_name="NameOwnerChanged",

/usr/lib/python2.5/site-packages/dbus/_dbus.py in __new__(cls=, private=False, mainloop=None)
211 """
212 return Bus.__new__(cls, Bus.TYPE_SESSION, private=private,
--> 213mainloop=mainloop)
mainloop = None
214 
215 class StarterBus(Bus):

/usr/lib/python2.5/site-packages/dbus/_dbus.py in __new__(cls=, bus_type=0, private=False, mainloop=None)
100 raise ValueError('invalid bus_type %s' % bus_type)
101 
--> 102 bus = BusConnection.__new__(subclass, bus_type, 
mainloop=mainloop)
bus = undefined
global BusConnection.__new__ = 
subclass = 
bus_type = 0
mainloop = None
103 
104 bus._bus_type = bus_type

/usr/lib/python2.5/site-packages/dbus/bus.py in __new__(cls=, address_or_type=0, mainloop=None)
123 
124 def __new__(cls, address_or_type=TYPE_SESSION, mainloop=None):
--> 125 bus = cls._new_for_bus(address_or_type, mainloop=mainloop)
bus = undefined
cls._new_for_bus = 
address_or_type = 0
mainloop = None
126 
127 # _bus_names is used by dbus.service.BusName!

: org.freedesktop.DBus.Error.NoServer: 
Failed to connect to socket /tmp/dbus-7dk02BRXOZ: Connection refused


This is the value of the DBUS_SESSION_BUS_ADDRESS env var:
unix:path=/tmp/olpc-session-bus,guid=fdf37ebd35a585d12382a50047ac3034

unix:path=/tmp/olpc-session-bus is specified
in /etc/dbus-1/session-olpc.conf:


  
  session

  unix:path=/tmp/olpc-session-bus

  
...

Any ideas?

Thanks,

Tomeu


___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Ivan Krstić]

Hi Jukka,

On Feb 6, 2008, at 11:13 AM, Klaus Weidner wrote:
> Would it be feasible/helpful to use the "uncore" approach used by  
> Emacs?
> - launch Python interpreter, load and initialize modules
> - force the interpreter to dump core
> - convert the core file to an executable file that has all the modules
>  loaded and initialized already
> - activities use this preloaded executable instead of the plain python
>  interpreter


per this thread:


we're looking into startup-time optimizations for Python on OLPC XO  
laptops. We had discussed this in the past at some length, and I  
recall you mentioning you had folks working on something similar. Did  
your effort get anywhere? Any notes to share?

Thanks!

Cheers,

--
Ivan Krstić <[EMAIL PROTECTED]> | http://radian.org

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Ivan Krstić]

On Feb 7, 2008, at 3:06 PM, Frank Ch. Eigler wrote:
> To what extent could prelinking, which should improve this aspect,
> be used on OLPC?

It was tested and found to have no measurable performance impact.

--
Ivan Krstić <[EMAIL PROTECTED]> | http://radian.org

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Frank Ch. Eigler]

[EMAIL PROTECTED] writes:

> [...]  I can't comment on python modules, but I've seen similar
> issue on shared libraries with firewall proxies that did a fork for
> each connection. [...]  if you can get the linking and
> initialization step down to once per boot it will bee a huge win.

To what extent could prelinking, which should improve this aspect,
be used on OLPC?

- FChE
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Simon McVittie]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Wed, 06 Feb 2008 at 23:06:19 -0500, Michael Stone wrote:
> Next, according to dbus-python's dbus/_dbus.py, the SessionBus returned
> when you call SessionBus() is cached in 
> 
>   Bus._shared_instances[BUS_SESSION]

This is not an API guarantee; neither is it API at all. I accept no
responsibility if the entire platform mysteriously crashes after a
dbus-python upgrade...

It ought to be sufficient for Rainbow to call close() on the SessionBus
after forking, but before executing the "target" code. This should remove it
from _shared_instances automatically, causing subsequent calls to SessionBus()
to create a new instance, although that code-path is not well tested and you
might need some patches or a newer version of dbus-python.

A better solution would be for Rainbow to avoid SessionBus() entirely,
and instead use an instance of the superclass, dbus.bus.BusConnection. This
does not have the weird caching behaviour at all (one call to the
constructor = one instance = one D-Bus connection). The code that gets run
after forking is still free to use the shared SessionBus.

This may require some care in the libraries that Rainbow uses - they
should all be able to accept a BusConnection argument rather than
implicitly using the shared SessionBus, but they don't necessarily get
this right. Libraries that can only use the shared SessionBus should
probably be considered buggy anyway, though.

Simon
-BEGIN PGP SIGNATURE-

iD8DBQFHqzX8WSc8zVUw7HYRAg6uAJ0XPz2C0az/DQ1XCAqNXZokzgFpeACdEfT3
MCpT4HahnUjTgG/rN1UFELw=
=zVsd
-END PGP SIGNATURE-
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[david]

On Wed, 6 Feb 2008, Benjamin M. Schwartz wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Tomeu Vizoso wrote:
>> By the data in #5228, looks like more than 50% of time is spent
>> importing modules. dbus, telepathy and pygtk make for more than 30% of
>> _total_ startup time.
>
> There has been much talk about fork()ing the python interpreter for each
> activity instead of starting a new interpreter.  AFAIK, the python interpreter
> only does nontrivial work the first time a module is imported in each
> interpreter.  By this logic, if we use a forked interpreter and import these
> modules at the root, it seems that the import will only have to happen once at
> Sugar startup, rather than once for each Activity.
>
> I would appreciate a comment on this from someone who knows more about the
> relevant issues, like the interaction of CoW memory and python modules.

I can't comment on python modules, but I've seen similar issue on shared 
libraries with firewall proxies that did a fork for each connection. when 
they used a bunch of shared libraries, the cost of library setup at each 
fork was pretty significant (enough to cut the throughput by a factor of 
10 compared to a staticly linked version)

if you can get the linking and initialization step down to once per boot 
it will bee a huge win.

David Lang
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Michael Stone]

Tomeu,

> I have hacked the rainbow service in the following way:

Please publish your code so that we can have a look at it.

> - rainbow preimports pygtk, telepathy, dbus and some slow sugar modules.
> - after cloning, reconnect to X.
> - instead of execvpe sugar-activity, directly execute the code.
> 
> Python activities have the expected speedup, from 7 seconds to 3.

What is costing us the remaining 3 seconds? 

> Haven't measured the memory usage improvement yet.

If python dirties every page when it updates refcounts, as is asserted
in the 'lessons' section of Andy Wingo's memory-profiling post [1], then
it seems that our strategies for improving memory usage in
Python-based activities are to change this behavior in our python
interpreter or to reduce the number of pages which are mapped by our
python-based activities.

However, please let us know if your measurements support or refute
"Wingo's hypothesis" that we will see essentially zero net memory
savings.

> The visible problems right now are that the D-Bus connection to the
> session bus is stale (how can we reconnect?) and that the activity uses
> the default gtk theme and not the Sugar one. These two should be quite
> easily solvable.

The gtk theme is controlled by the setting of the 'GTK2_RC_FILES'
environment variable. Setting

  GTK2_RC_FILES=/usr/share/sugar/data/sugar-xo.gtkrc

before importing gtk should do the trick.

Next, according to dbus-python's dbus/_dbus.py, the SessionBus returned
when you call SessionBus() is cached in 

  Bus._shared_instances[BUS_SESSION]

(where BUS_SESSION is a constant from _dbus_bindings). 

Consequently, if you make a new BusConnection and save it as

  Bus._shared_instances[BUS_SESSION]

post-fork(), you should be good to go.  

> I think Michael, Scott and Chris played in the past with this idea. What
> was your conclusion? Anybody saw any flaw with this approach? Security
> implications?

The conclusion was that it was unlikely to result in memory savings.
Since we really wanted memory savings, nobody tried it out. Thanks for
stepping up to the plate - now let's see if we can make it shippable.

Best,

Michael

[1]: 
http://wingolog.org/archives/2007/11/27/reducing-the-footprint-of-python-applications
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Tomeu Vizoso]

On Wed, 2008-02-06 at 09:59 -0500, Benjamin M. Schwartz wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Tomeu Vizoso wrote:
> > If I understand correctly how that fork() would work, what we would be
> > saving in that case is the reading of modules, but the python code at
> > the module scope would need to be executed anyway.
> 
> I am no expert, but, without evidence, my current suspicion is the other way.
> Consider:
> 
> import dbus
> import os
> 
> def launch_activity(some_activity)
> if os.fork()  == 0:
> import some_activity
> x = some_activity.SomeActivity()
> else:
> return
> 
> - From the perspective of the child process, the modules have already been
> imported, and are even still in scope.  It would be strange for the Python
> semantics if the imports were somehow "forgotten" in the child.

You are right.

I have hacked the rainbow service in the following way:

- rainbow preimports pygtk, telepathy, dbus and some slow sugar modules.
- after cloning, reconnect to X.
- instead of execvpe sugar-activity, directly execute the code.

Python activities have the expected speedup, from 7 seconds to 3.

Haven't measured the memory usage improvement yet.

The visible problems right now are that the D-Bus connection to the
session bus is stale (how can we reconnect?) and that the activity uses
the default gtk theme and not the Sugar one. These two should be quite
easily solvable.

I think Michael, Scott and Chris played in the past with this idea. What
was your conclusion? Anybody saw any flaw with this approach? Security
implications?

Thanks,

Tomeu

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Simon McVittie]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Wed, 06 Feb 2008 at 10:13:23 -0600, Klaus Weidner wrote:
> Would it be feasible/helpful to use the "uncore" approach used by Emacs?
[...]
> Of course, any initialization done initially must not depend on external
> state, so for example any connections to the X server or other network
> connections would need to be delayed

D-Bus connections, too. If a process forks or does a core/uncore cycle after
opening a D-Bus connection (including, but not limited to, any call to
dbus-python's dbus.Bus(), dbus.SessionBus() etc.), chaos is likely to ensue.

Regards,
Simon
-BEGIN PGP SIGNATURE-

iD8DBQFHqe2ZWSc8zVUw7HYRAhKIAKDFXV5n2QKZeBeTW1rxL1G3bHo9SwCghprv
ChdeTcKOMpYXdzdxM1fFXHc=
=Bb4Z
-END PGP SIGNATURE-
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Klaus Weidner]

On Wed, Feb 06, 2008 at 09:31:02AM -0500, Benjamin M. Schwartz wrote:
> Tomeu Vizoso wrote:
> > By the data in #5228, looks like more than 50% of time is spent
> > importing modules. dbus, telepathy and pygtk make for more than 30% of
> > _total_ startup time.
> 
> There has been much talk about fork()ing the python interpreter for each
> activity instead of starting a new interpreter.  AFAIK, the python interpreter
> only does nontrivial work the first time a module is imported in each
> interpreter.  By this logic, if we use a forked interpreter and import these
> modules at the root, it seems that the import will only have to happen once at
> Sugar startup, rather than once for each Activity.
> 
> I would appreciate a comment on this from someone who knows more about the
> relevant issues, like the interaction of CoW memory and python modules.

(Disclaimer: I know nothing at all about Python internals)

Would it be feasible/helpful to use the "uncore" approach used by Emacs?

- launch Python interpreter, load and initialize modules

- force the interpreter to dump core

- convert the core file to an executable file that has all the modules
  loaded and initialized already

- activities use this preloaded executable instead of the plain python
  interpreter

Of course, any initialization done initially must not depend on external
state, so for example any connections to the X server or other network
connections would need to be delayed or reopened when running the final
application. If that's where most of the time or effort is spent, this
approach would not be helpful.

-Klaus
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Benjamin M. Schwartz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Tomeu Vizoso wrote:
> If I understand correctly how that fork() would work, what we would be
> saving in that case is the reading of modules, but the python code at
> the module scope would need to be executed anyway.

I am no expert, but, without evidence, my current suspicion is the other way.
Consider:

import dbus
import os

def launch_activity(some_activity)
if os.fork()  == 0:
import some_activity
x = some_activity.SomeActivity()
else:
return

- From the perspective of the child process, the modules have already been
imported, and are even still in scope.  It would be strange for the Python
semantics if the imports were somehow "forgotten" in the child.
- --Ben
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHqctEUJT6e6HFtqQRAnuJAKCERl12jDpsYovWMEoCTKmea1NvLgCfdXOg
wBt9No0/TmALQ/k3AHxdgyg=
=13qT
-END PGP SIGNATURE-
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Tomeu Vizoso]

On Wed, 2008-02-06 at 09:31 -0500, Benjamin M. Schwartz wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Tomeu Vizoso wrote:
> > By the data in #5228, looks like more than 50% of time is spent
> > importing modules. dbus, telepathy and pygtk make for more than 30% of
> > _total_ startup time.
> 
> There has been much talk about fork()ing the python interpreter for each
> activity instead of starting a new interpreter.  AFAIK, the python interpreter
> only does nontrivial work the first time a module is imported in each
> interpreter.  By this logic, if we use a forked interpreter and import these
> modules at the root, it seems that the import will only have to happen once at
> Sugar startup, rather than once for each Activity.
> 
> I would appreciate a comment on this from someone who knows more about the
> relevant issues, like the interaction of CoW memory and python modules.

AFAIK, what is taking most of the time here is the code at the module
level that is executed the first time that module is imported, and not
the reading and loading of modules. I think Michael Stone can confirm
this.

If I understand correctly how that fork() would work, what we would be
saving in that case is the reading of modules, but the python code at
the module scope would need to be executed anyway.

Tomeu

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: [sugar] python activities startup

[Tomeu Vizoso]

On Wed, 2008-02-06 at 15:20 +0100, Ryan Pavlik wrote:
> Tomeu Vizoso wrote:
> > Hi,
> >
> > as we all know, activities that use the python API (most of them) start
> > up very slow in the XO, a trivial one launching in 7 seconds.
> >
> > http://dev.laptop.org/ticket/5228
> >
> > I don't know yet if performance work will land in update.2 or in
> > update.3, but now may be a good moment to summarize what we know and see
> > which are our options.
> >
> > By the data in #5228, looks like more than 50% of time is spent
> > importing modules. dbus, telepathy and pygtk make for more than 30% of
> > _total_ startup time.
> >
> > In that ticket is attached a file that lists the 161 modules imported at
> > startup. I doubt most of those modules are actually used during startup,
> > and many won't be used neither during the activity lifetime, they are
> > just imported because one module we actually use *may* need it at some
> > point.
> >
> > It's clear that we would prefer to not pay this price upfront at
> > startup, but rather that the needed initializations happened during
> > runtime as (if) needed.
> >
> > I would like to take the chance to ask to Guido what's his opinion on
> > this, as I'm sure this has been discussed in the python community at
> > some point. How could we move all these initializations from import time
> > to use time?
> >
> > I see the following options:
> >
> > - Reduce the number of things that are done at startup. We probably
> > could write to the datastore, initialize sharing and creating the D-Bus
> > service at a later point after startup. These changes look to me as
> > quite invasive and somewhat risky. The benefit from the user point of
> > view is not clear, as we would like those things to happen as soon as
> > possible after the activity window has been brought up.
> >
> > - Modify the modules that do significant initializations at import time
> > so that code is executed lazily as needed. Those modules will be inside
> > python itself, dbus, pygtk, telepathy, sugar, etc. The drawback of this
> > approach is that we'll need to maintain forks for some time.
> >
> > - Hack the import logic in python so that the module-level code is
> > executed only when some function or method from that module is called or
> > a variable is set. I'm not sure if this is possible by just using the
> > existing hooks or if we would need to patch python. In case this brings
> > some incompatibilities, we could activate this "fast mode" through some
> > flag.
> >
> > I wonder too if python 2.6 and 3.0 will bring some improvement in these
> > areas?
> >
> > Thanks,
> >
> > Tomeu
> >
> >
> > ___
> > Sugar mailing list
> > [EMAIL PROTECTED]
> > http://lists.laptop.org/listinfo/sugar
> >
> >   
> Though it's not very elegant, you can import anywhere and I believe the 
> import is then available everywhere (within that module).  You might be 
> able to just move the import statements to right before they are needed 
> - I believe multiple imports are ignored, so if imported functions are 
> needed in multiple places, import in each of those (or the first to 
> execute if it is deterministic).  Not really pretty, but it might do the 
> trick I think...

This would help only for the imports in sugar modules that are not
actually used during startup. Sadly, none of the modules that appear in
the profiles as significant fall into this category.

Tomeu

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: python activities startup

[Benjamin M. Schwartz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Tomeu Vizoso wrote:
> By the data in #5228, looks like more than 50% of time is spent
> importing modules. dbus, telepathy and pygtk make for more than 30% of
> _total_ startup time.

There has been much talk about fork()ing the python interpreter for each
activity instead of starting a new interpreter.  AFAIK, the python interpreter
only does nontrivial work the first time a module is imported in each
interpreter.  By this logic, if we use a forked interpreter and import these
modules at the root, it seems that the import will only have to happen once at
Sugar startup, rather than once for each Activity.

I would appreciate a comment on this from someone who knows more about the
relevant issues, like the interaction of CoW memory and python modules.

- --Ben
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHqcSmUJT6e6HFtqQRAsahAJsG4pa1WMFbBX+C/+z+KuCXxGuB8ACfXC05
PKkSUnLaK3iHeWgI4l5ZPks=
=Lx/w
-END PGP SIGNATURE-
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

python activities startup

[Tomeu Vizoso]

Hi,

as we all know, activities that use the python API (most of them) start
up very slow in the XO, a trivial one launching in 7 seconds.

http://dev.laptop.org/ticket/5228

I don't know yet if performance work will land in update.2 or in
update.3, but now may be a good moment to summarize what we know and see
which are our options.

By the data in #5228, looks like more than 50% of time is spent
importing modules. dbus, telepathy and pygtk make for more than 30% of
_total_ startup time.

In that ticket is attached a file that lists the 161 modules imported at
startup. I doubt most of those modules are actually used during startup,
and many won't be used neither during the activity lifetime, they are
just imported because one module we actually use *may* need it at some
point.

It's clear that we would prefer to not pay this price upfront at
startup, but rather that the needed initializations happened during
runtime as (if) needed.

I would like to take the chance to ask to Guido what's his opinion on
this, as I'm sure this has been discussed in the python community at
some point. How could we move all these initializations from import time
to use time?

I see the following options:

- Reduce the number of things that are done at startup. We probably
could write to the datastore, initialize sharing and creating the D-Bus
service at a later point after startup. These changes look to me as
quite invasive and somewhat risky. The benefit from the user point of
view is not clear, as we would like those things to happen as soon as
possible after the activity window has been brought up.

- Modify the modules that do significant initializations at import time
so that code is executed lazily as needed. Those modules will be inside
python itself, dbus, pygtk, telepathy, sugar, etc. The drawback of this
approach is that we'll need to maintain forks for some time.

- Hack the import logic in python so that the module-level code is
executed only when some function or method from that module is called or
a variable is set. I'm not sure if this is possible by just using the
existing hooks or if we would need to patch python. In case this brings
some incompatibilities, we could activate this "fast mode" through some
flag.

I wonder too if python 2.6 and 3.0 will bring some improvement in these
areas?

Thanks,

Tomeu


___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel