[kamaelia-list] Re: Autoloading Components based on Imports
On Tuesday 30 June 2009 22:16:41 John Eikenberry wrote:
> What is wrong with simply importing the most common things in
> Kamaelia/__init__.py so they'd be available to import off the top level
> package? Why all the autoload() magic in the original? It was the latter
> that I was mainly disagreeing with, not the simplified importing.
I realised that the issue was with the magic, rather than the overall effect.
With regard to putting the most common things in Kamaelia/__init__.py, you
*could* do that, but I think you can only do that properly in certain
restricted usecases.
Then you end up with a static list, and actually risk making things more
confusing because you end up wondering why some things are there, but not
this other thing.
For example, if I give you a choice of 10 appetisers, 10 main courses, and 10
desserts, I can guarantee that some things will be more popular than most,
but I can also guarantee that many people will choose outliers from the norm
for at least one of those courses. A pipeline with 3 components is the same.
So how do you choose what's most popular? You can't realistically.
However, if the alternative is considered just too whacky to even think of
accepting, then this maxim seems appropriate: "In the face of ambiguity,
refuse the temptation to guess."
On Tuesday 30 June 2009 21:38:30 Gloria W wrote:
> > Essentially I'm wondering how to best implement the equivalent of
> > $PATH for KamaeliaComponents.
> >
>
> I think this is perfectly satisfied by virtualenv, which generates a
> setup.pth file.
> This path file can be checked in and used as part of a project.
>
> easy_install modifies the setup.pth file in /your/python/site-packages,
> while virtualenv creates local setup.pth files which can extend the
> global file.
>
> A "make install" command could roll it out into a "work" directory where
> Kamaelia tests are placed.
>
> This isn't considered magic, because it is a well-known and used
> convention for isolated test environments.
I _think_ you're mis-interpretting what I mean by $PATH - I could be wrong.
Whilst Kamaelia isn't the shell, there are lots of similarities, and for some
of them it appears that implementing some more may be useful. The unix path
is a good example.
Taking a silly shell example. Suppose I want to add all the sizes of video up
using the shell I can do this:
/media/disk/Europython09/20090701> echo `ls -l |grep mp4|awk '{print $5}'`|
sed -e "s/ /+/g"|bc
19267846144
I don't have to do this:
ECHO=echo #builtin
LS=/bin/ls
GREP=/usr/bin/grep
AWK=/usr/bin/awk
SED=/usr/bin/sed
BC=/usr/bin/bc
$ECHO `$LS -l |$GREP mp4|$AWK '{print $5}'`|$SED -e "s/ /+/g"|$BC
Now of course, I *can* do this, and indeed, many sysadmins will agree that
doing this can be a good idea - especially if the definitions are in a
separate file allowing profiles.
This latter version is however directly akin to this:
from Kamaelia.Util.Backplane import Backplane, PublishTo, SubscribeTo
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Chassis.ServerCore import ServerCore
from Kamaelia.File.Append import Append
from Kamaelia.Util.PureTransformer import PureTransformer
And is inherently a useful thing for all the reasons stated. But suppose we
wanted it to run just like the unix command line. We'd need to provide the
equivalent of this:
~> echo $PATH
/opt/kde3/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/jvm/jre/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/gnome/bin:/usr/lib/qt3/bin
Where rather than finding programs, or finding python modules (.pth files) I'd
prefer something that can look for individual components.
Now that could look like this:
from Kamaelia import Backplane, PublishTo, SubscribeTo, \
Pipeline, ServerCore, Append, PureTransformer
Or it could look like this:
Backplane, PublishTo, SubscribeTo, Pipeline, ServerCore, Append,
PureTransformer = Kamaelia.Find("Backplane", "PublishTo", "SubscribeTo",
"Pipeline", "ServerCore", "Append", "PureTransformer")
or this:
Backplane, PublishTo, SubscribeTo, Pipeline, ServerCore, Append,
PureTransformer = Kamaelia.Find("Backplane", "PublishTo", "SubscribeTo",
"Pipeline", "ServerCore", "Append", "PureTransformer")
But my preference is something like:
from < thing > import Backplane, PublishTo, SubscribeTo, \
Pipeline, ServerCore, Append, PureTransformer
NB, this kind of thinking implies that these should be possible too:
>>> Kamaelia.Which("UnixProcess")
Kamaelia.File.UnixProcess.UnixProcess
>>> Kamaelia.Man("UnixProcess")
===
UnixProcess
===
Launch another unix process and communicate with it via its standard input
and output, by using the "inbox" and "outbox" of this component.
Example Usage
-
Which you can nearly do:
>>> import Kamaelia.File.UnixProcess
>>> Kamaelia.File.UnixProcess
>>>
[kamaelia-list] Re: Autoloading Components based on Imports
Michael Sparks wrote: > I must admit that I'm more than a little surprised at the overall > consensus here, but it's a very clear one. > > As you can imagine I'm in favour of the idea for obvious reasons... > but I'd be a fool if I asked for advice on what people thought and > then ignored it. > > For now, I'll say that this idea... > > from Kamaelia import RandomStuffYouHaveNoIdeaWhereFrom > > ... is dead in the water for now. What is wrong with simply importing the most common things in Kamaelia/__init__.py so they'd be available to import off the top level package? Why all the autoload() magic in the original? It was the latter that I was mainly disagreeing with, not the simplified importing. -- John Eikenberry [...@zhar.net - http://zhar.net] [PGP public key @ http://zhar.net/jae_at_zhar_net.gpg] __ "Perfection is attained, not when no more can be added, but when no more can be removed." -- Antoine de Saint-Exupery signature.asc Description: Digital signature
[kamaelia-list] Re: Autoloading Components based on Imports
> Essentially I'm wondering how to best implement the equivalent of > $PATH for KamaeliaComponents. > I think this is perfectly satisfied by virtualenv, which generates a setup.pth file. This path file can be checked in and used as part of a project. easy_install modifies the setup.pth file in /your/python/site-packages, while virtualenv creates local setup.pth files which can extend the global file. A "make install" command could roll it out into a "work" directory where Kamaelia tests are placed. This isn't considered magic, because it is a well-known and used convention for isolated test environments. Hope this helps, Gloria --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Autoloading Components based on Imports
I must admit that I'm more than a little surprised at the overall consensus here, but it's a very clear one. As you can imagine I'm in favour of the idea for obvious reasons... but I'd be a fool if I asked for advice on what people thought and then ignored it. For now, I'll say that this idea... from Kamaelia import RandomStuffYouHaveNoIdeaWhereFrom ... is dead in the water for now. But I think the idea can be reworked in a way that works. FWIW, where I'm coming from is here: ~> echo $PATH /opt/kde3/bin:/home/zathras/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/jvm/jre/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/gnome/bin:/usr/lib/qt3/bin ~> which ls /bin/ls Or an example of something that's a pain: $ echo $PATH /bin:/sbin:/usr/bin:/usr/sbin $ which enable /usr/bin/enable $ enable enable . enable : enable [ enable alias enable bg enable bind enable break enable builtin enable cd enable command enable compgen ... (this output of enable is not the declared one by which, but the shell builtin "enable") Essentially I'm wondering how to best implement the equivalent of $PATH for KamaeliaComponents. However, FWIW, it won't be thrown *in* without more discussion, and not in the form I suggested, given various reactions :-) Cheers :-) Michael. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Autoloading Components based on Imports
Gloria W wrote: > On 06/23/2009 06:27 AM, Sylvain Hellegouarch wrote: > > I appreciate the fact that the current verbosity means I know where > > things are coming from and never worry about name clashing. > > > > - Sylvain > > > I have to admit, and I agree. I did something similar in PHP5 with > autoloading (we had no choice, really, because of the limitations of PHP > and it's internal global scoping). much magic was done, and it was > convenient but confusing. > > I think Python developers, whether they realize it or not, come to this > language with the expectation of being given a superb tools to do their > own magic. Dynamic module loading may cause great confusion. But a > static solution is actually excellent, IMHO. It saves one the effort of > discovering the import chain, but at the same time gives us the ability > to change it as needed. +1 on avoiding magic. I like my libraries simple and explicit. Leave the magic for me to do locally if I want it. -- John Eikenberry [...@zhar.net - http://zhar.net] [PGP public key @ http://zhar.net/jae_at_zhar_net.gpg] __ "Perfection is attained, not when no more can be added, but when no more can be removed." -- Antoine de Saint-Exupery signature.asc Description: Digital signature
[kamaelia-list] Re: Autoloading Components based on Imports
On 06/23/2009 06:27 AM, Sylvain Hellegouarch wrote: > Matt Hammond a écrit : > >>> So some questions arise: >>> 1. Good idea or not? >>> 2. Really? Could be viewed as bad mojo& messing about one step too >>> far. >>> 3. If OK, should it be static? ie the list of what gets imported and >>> handles what dealt with by a table lookup in Kamaelia/__init__.py ? >>> 4. Or should it go "OK, I was imported here, I'll rummage around in all >>> my >>> subdirectories, in this overall order" >>> 5. Do 3, then 4, if the name wasn't found in 3. >>> 6. The other way round ? >>> 7. Do we allow extra search paths for the case of 4 ? (think sys.path >>> for >>> modules) >>> 8. How about allowing extra lookup tables to be added in the case of 3? >>> 9. lots more possibilities. >>> >>> >> I can see it would be much less verbose and that is a *good* thing! If >> nothing else, from writing examples in documentation, where brevity is >> highly desireable, adding all those import statements can be tedious and >> ugly. >> >> > True and yet I welcome that verbosity. I did a bit of Ruby and the first > thing that annoyed me was that there was lots of magic going on in > module finding and since the language allows you to change things rather > dramatically at so many levels I ended up being lost and frustrated quickly. > > I appreciate the fact that the current verbosity means I know where > things are coming from and never worry about name clashing. > > - Sylvain > I have to admit, and I agree. I did something similar in PHP5 with autoloading (we had no choice, really, because of the limitations of PHP and it's internal global scoping). much magic was done, and it was convenient but confusing. I think Python developers, whether they realize it or not, come to this language with the expectation of being given a superb tools to do their own magic. Dynamic module loading may cause great confusion. But a static solution is actually excellent, IMHO. It saves one the effort of discovering the import chain, but at the same time gives us the ability to change it as needed. Another possibility could be to provide virtualenv configs for people who choose to use them, where the setup.pth file contains the proper sys path, and all modules are pulled in, etc. This can be a beautiful solution, since I can set up different dev environments that won't conflict or get too heavy when compared to my others. People who want to purely DIY can avoid the virtualenv config altogether. What do you think? Gloria --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Autoloading Components based on Imports
Matt Hammond a écrit : >> So some questions arise: >>1. Good idea or not? >>2. Really? Could be viewed as bad mojo & messing about one step too >> far. >>3. If OK, should it be static? ie the list of what gets imported and >>handles what dealt with by a table lookup in Kamaelia/__init__.py ? >>4. Or should it go "OK, I was imported here, I'll rummage around in all >> my >>subdirectories, in this overall order" >>5. Do 3, then 4, if the name wasn't found in 3. >>6. The other way round ? >>7. Do we allow extra search paths for the case of 4 ? (think sys.path >> for >>modules) >>8. How about allowing extra lookup tables to be added in the case of 3? >>9. lots more possibilities. >> > > I can see it would be much less verbose and that is a *good* thing! If > nothing else, from writing examples in documentation, where brevity is > highly desireable, adding all those import statements can be tedious and > ugly. > True and yet I welcome that verbosity. I did a bit of Ruby and the first thing that annoyed me was that there was lots of magic going on in module finding and since the language allows you to change things rather dramatically at so many levels I ended up being lost and frustrated quickly. I appreciate the fact that the current verbosity means I know where things are coming from and never worry about name clashing. - Sylvain --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Autoloading Components based on Imports
> So some questions arise: >1. Good idea or not? >2. Really? Could be viewed as bad mojo & messing about one step too > far. >3. If OK, should it be static? ie the list of what gets imported and >handles what dealt with by a table lookup in Kamaelia/__init__.py ? >4. Or should it go "OK, I was imported here, I'll rummage around in all > my >subdirectories, in this overall order" >5. Do 3, then 4, if the name wasn't found in 3. >6. The other way round ? >7. Do we allow extra search paths for the case of 4 ? (think sys.path > for >modules) >8. How about allowing extra lookup tables to be added in the case of 3? >9. lots more possibilities. I can see it would be much less verbose and that is a *good* thing! If nothing else, from writing examples in documentation, where brevity is highly desireable, adding all those import statements can be tedious and ugly. I would also worry about the situation where two or more components share the same (leaf) name. Eg: Kamaelia.Chassis.Pipeline Kamaelia.Experimental.Chassis.Pipeline There are also other components which, although they do not clash now, look likely to later, eg: Kamaelia.Audio.Codec.PyMedia.Decoder.Decoder I suppose I'm asking, will someone's code break X months/years later when more components are added to teh repository with the same name? This concern therefore makes me worry this could be a short term gain which causes future difficulties. But maybe this is still a good idea and the solution is to develop a stricter component naming policy? (and retrospectively rename existing components to fit it) Another perspective: If one of the problems this is trying to solve is disagreement over where a component should be located in the hierarchy; this could be solved by symlinking - ie. let the component be in both places in the hierarchy where it is believed to make sense. However, I guess this could end up being highly confusing! I can see it would make sense if you consider the hierarchy as a hierarchical classification scheme, rather than a packaging/encapulsation thing - eg. more like categories on Ebay than packages in java. -- | Matt Hammond | | [anything you like unless it bounces] 'at' matthammond 'dot' org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
