Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Phillip J. Eby wrote:
>> It goes in the .egg-info directory. This way elsewhere you can say:
>>
>>application = SomeApplication[feature1]
>
>
> I like this a lot, although for a different purpose than the format
> Chris and I were talking about.
Yes, this proposal really just simplifies a part of that application
deployment configuration, it doesn't replace it. Though it might make
other standardization less important.
> I see this fitting into that format as
> maybe:
>
>[feature1 from SomeApplication]
># configuration here
>
>
>> And it's quite unambiguous. Note that there is *no* "configuration" in
>> the egg-info file, because you can't put any configuration related to a
>> deployment in an .egg-info directory, because it's not specific to any
>> deployment. Obviously we still need a way to get configuration in
>> there, but lets say that's a different matter.
>
>
> Easily fixed via what I've been thinking of as the "deployment
> descriptor"; I would call your proposal here the "import map".
> Basically, an import map describes a mapping from some sort of feature
> name to qualified names in the code.
Yes, it really just gives you a shorthand for the factory configuration
variable.
> I have an extension that I would make, though. Instead of using
> sections for features, I would use name/value pairs inside of sections
> named for the kind of import map. E.g.:
>
> [wsgi.app_factories]
> feature1 = somemodule:somefunction
> feature2 = another.module:SomeClass
> ...
>
> [mime.parsers]
> application/atom+xml = something:atom_parser
> ...
I assume mime.parsers is just a theoretical example of another kind of
service a package can provide? But yes, this seems very reasonable, and
even allows for loosely versioned specs (e.g., wsgi.app_factories02,
which returns factories with a different interface; or maybe something
like foo.configuration_schema, an optional entry point that returns the
configuration schema for an application described elsewhere).
This kind of addresses the issue where the module structure of a package
becomes an often unintentional part of its external interface. It feels
a little crude in that respect... but maybe not. Is it worse to do:
from package.module import name
or:
name = require('Package').load_entry_point('service_type', 'name')
OK, well clearly the second is worse ;) But if that turned into a
single function call:
name = load_service('Package', 'service_type', 'name')
It's not that bad. Maybe even:
name = services['Package:service_type:name']
Though service_type feels extraneous to me. I see the benefit of being
explicit about what the factory provides, but I don't see the benefit of
separating namespaces; the name should be unambiguous. Well... unless
you used the same name to group related services, like the configuration
schema and the application factory itself. So maybe I retract that
criticism.
> In addition to specifying the entry point, each entry in the import map
> could optionally list the "extras" that are required if that entry point
> is used.
> It could also issue a 'require()' for the corresponding feature if it
> has any additional requirements listed in the extras_require dictionary.
I figured each entry point would just map to a feature, so the
extra_require dictionary would already have entries.
> So, I'm thinking that this would be implemented with an entry_points.txt
> file in .egg-info, but supplied in setup.py like this:
>
> setup(
> ...
> entry_points = {
> "wsgi.app_factories": dict(
> feature1 = "somemodule:somefunction",
> feature2 = "another.module:SomeClass [extra1,extra2]",
> ),
> "mime.parsers": {
> "application/atom+xml": "something:atom_parser
> [feedparser]"
> }
> },
> extras_require = dict(
> feedparser = [...],
> extra1 = [...],
> extra2 = [...],
> )
> )
I think I'd rather just put the canonical version in .egg-info instead
of as an argument to setup(); this is one place where using Python
expressions isn't a shining example of clarity. But I guess this is
fine too; for clarity I'll probably start writing my setup.py files with
variable assignments, then a setup() call that just refers to those
variables.
>> Open issues? Yep, there's a bunch. This requires the rest of the
>> configuration to be done quite lazily.
>
>
> Not sure I follow you; the deployment descriptor could contain all the
> configuration; see the Web-SIG post I made just previous to this one.
Well, when I proposed that the factory be called with zero arguments,
that wouldn't allow any configuration to be passed in.
>> I don't think
>> this is useful without the other pieces (both in front of this
>> configuration file and behind it) but maybe we can think abo
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 02:12 PM 7/24/2005 -0500, Ian Bicking wrote:
>This kind of addresses the issue where the module structure of a package
>becomes an often unintentional part of its external interface. It feels a
>little crude in that respect... but maybe not. Is it worse to do:
>
> from package.module import name
>
>or:
>
> name = require('Package').load_entry_point('service_type', 'name')
>
>OK, well clearly the second is worse ;) But if that turned into a single
>function call:
>
> name = load_service('Package', 'service_type', 'name')
>
>It's not that bad. Maybe even:
>
> name = services['Package:service_type:name']
The actual API I have implemented in my CVS working copy is:
the_object = load_entry_point('Project', 'group', 'name')
which seems pretty clean to me. You can also use
dist.load_entry_point('group','name') if you already have a distribution
object for some reason. (For example, if you use an activation listener to
get callbacks when distributions are activated on sys.path.)
To introspect an entry point or check for its existence, you can use:
entry_point = get_entry_info('Project', 'group', 'name')
which returns either None or an EntryPoint object with various
attributes. To list the entry points of a group, or to list the groups,
you can use:
# dictionary of group names to entry map for each kind
group_names = get_entry_map('Project')
# dictionary of entry names to corresponding EntryPoint object
entry_names = get_entry_map('Project', 'group')
These are useful for dynamic entry points.
>Though service_type feels extraneous to me. I see the benefit of being
>explicit about what the factory provides, but I don't see the benefit of
>separating namespaces; the name should be unambiguous.
You're making the assumption that the package author defines the entry
point names, but that's not the case for application plugins; the
application will define entry point names and group names for the
application's use, and some applications will need multiple groups. Groups
might be keyed statically (i.e. a known set of entry point names) or
dynamically (the keys are used to put things in a table, e.g. a file
extension handler table).
>>In addition to specifying the entry point, each entry in the import map
>>could optionally list the "extras" that are required if that entry point
>>is used.
>>It could also issue a 'require()' for the corresponding feature if it has
>>any additional requirements listed in the extras_require dictionary.
>
>I figured each entry point would just map to a feature, so the
>extra_require dictionary would already have entries.
The problem with that is that asking for a feature that's not in
extras_require is an InvalidOption error, so this would force you to define
entries in extras_require even if you have no extras involved. It would
also make for redundancies when entry points share an extra. I also don't
expect extras to be used as frequently as entry points.
>>So, I'm thinking that this would be implemented with an entry_points.txt
>>file in .egg-info, but supplied in setup.py like this:
>> setup(
>> ...
>> entry_points = {
>> "wsgi.app_factories": dict(
>> feature1 = "somemodule:somefunction",
>> feature2 = "another.module:SomeClass [extra1,extra2]",
>> ),
>> "mime.parsers": {
>> "application/atom+xml": "something:atom_parser [feedparser]"
>> }
>> },
>> extras_require = dict(
>> feedparser = [...],
>> extra1 = [...],
>> extra2 = [...],
>> )
>> )
>
>I think I'd rather just put the canonical version in .egg-info instead of
>as an argument to setup(); this is one place where using Python
>expressions isn't a shining example of clarity. But I guess this is fine
>too; for clarity I'll probably start writing my setup.py files with
>variable assignments, then a setup() call that just refers to those variables.
The actual syntax I'm going to end up with is:
entry_points = {
"wsgi.app_factories": [
"feature1 = somemodule:somefunction",
"feature2 = another.module:SomeClass [extra1,extra2]",
]
}
Which is still not great, but it's a bit simpler. If you only have one
entry point, you can use:
entry_points = {
"wsgi.app_factories": "feature = somemodule:somefunction",
}
Or you can use a long string for each group:
entry_points = {
"wsgi.app_factories": """
# define features for blah blah
feature1 = somemodule:somefunction
feature2 = another.module:SomeClass [extra1,extra2]
"""
}
Or even list everything in one giant string:
entry_points = """
[wsgi.app_factories]
# define features for blah blah
feature1 = somemodule:somefu
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 02:12 PM 7/24/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>However, after more thought, I think that the "next application" argument >>should be a keyword argument too, like 'wsgi_next' or some such. This >>would allow a factory to have required arguments in its signature, e.g.: >> def some_factory(required_arg_x, required_arg_y, optional_arg="foo", >> ): >> ... >>The problem with my original idea to have the "next app" be a positional >>argument is that it would prevent non-middleware applications from having >>any required arguments. > >I think it's fine to declare the next_app keyword argument as special, and >promise (by convention) to always pass it in with that name. Actually, now that we have the "entry points" capability in pkg_resources (I just checked it in), we could simply have middleware components looked up in 'wsgi.middleware_factories' and applications looked up in 'wsgi.application_factories'. If a factory can be used for both, you can always list it in both places. Entry points have 1001 uses... I can imagine applications defining entry point groups for URL namespaces. For example, Trac has URLs like /changesets and /roadmap, and these could be defined via a trac.navigation entry point group, e.g.: [trac.navigation] changesets = some.module:foo roadmap = other.module:bar And then people could easily create plugin projects that add additional navigation components. (Trac already has an internal extension point system to do things rather like this, but entry points are automatically discoverable without any prior knowledge of what modules to import.) There are other frameworks out there (e.g. PyBlosxom), both web and non-web, that could really do nicely with having a standard way to do this kind of thing, rather than having to roll their own. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Phillip J. Eby wrote:
> The actual syntax I'm going to end up with is:
>
> entry_points = {
> "wsgi.app_factories": [
> "feature1 = somemodule:somefunction",
> "feature2 = another.module:SomeClass [extra1,extra2]",
> ]
> }
That seems weird to put the assignment inside a string, instead of:
entry_points = {
'wsgi.app_factories': {
'app': 'somemodule:somefunction',
},
}
Also, is there any default name? Like for a package that distributes
only one application. Or these just different spellings for the same thing?
--
Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Sorry, I think I may have lost track of where we were going wrt the
deployment spec. Specifically, I don't know how we got to using eggs
(which I'd really like to, BTW, they're awesome conceptually!) from
where we were in the discussion about configuring a WSGI pipeline. What
is a "feature"? What is an "import map"? "Entry point"? Should I just
get more familiar with eggs to understand what's being discussed here or
did I miss a few posts?
On Sun, 2005-07-24 at 12:49 -0400, Phillip J. Eby wrote:
> [cc:ed to distutils-sig because much of the below is about a new egg
> feature; follow-ups about the web stuff should stay on web-sig]
>
> At 04:04 AM 7/24/2005 -0500, Ian Bicking wrote:
> >So maybe here's a deployment spec we can start with. It looks like:
> >
> >[feature1]
> >someapplication.somemodule.some_function
> >
> >[feature2]
> >someapplication.somemodule.some_function2
> >
> >You can't get dumber than that! There should also be a "no-feature"
> >section; maybe one without a section identifier, or some special section
> >identifier.
> >
> >It goes in the .egg-info directory. This way elsewhere you can say:
> >
> >application = SomeApplication[feature1]
>
> I like this a lot, although for a different purpose than the format Chris
> and I were talking about. I see this fitting into that format as maybe:
>
> [feature1 from SomeApplication]
> # configuration here
>
>
> >And it's quite unambiguous. Note that there is *no* "configuration" in
> >the egg-info file, because you can't put any configuration related to a
> >deployment in an .egg-info directory, because it's not specific to any
> >deployment. Obviously we still need a way to get configuration in
> >there, but lets say that's a different matter.
>
> Easily fixed via what I've been thinking of as the "deployment descriptor";
> I would call your proposal here the "import map". Basically, an import map
> describes a mapping from some sort of feature name to qualified names in
> the code.
>
> I have an extension that I would make, though. Instead of using sections
> for features, I would use name/value pairs inside of sections named for the
> kind of import map. E.g.:
>
> [wsgi.app_factories]
> feature1 = somemodule:somefunction
> feature2 = another.module:SomeClass
> ...
>
> [mime.parsers]
> application/atom+xml = something:atom_parser
> ...
>
> In other words, feature maps could be a generic mechanism offered by
> setuptools, with a 'Distribution.load_entry_point(kind,name)' API to
> retrieve the desired object. That way, we don't end up reinventing this
> idea for dozens of frameworks or pluggable applications that just need a
> way to find a few simple entry points into the code.
>
> In addition to specifying the entry point, each entry in the import map
> could optionally list the "extras" that are required if that entry point is
> used.
> It could also issue a 'require()' for the corresponding feature if it has
> any additional requirements listed in the extras_require dictionary.
>
> So, I'm thinking that this would be implemented with an entry_points.txt
> file in .egg-info, but supplied in setup.py like this:
>
> setup(
> ...
> entry_points = {
> "wsgi.app_factories": dict(
> feature1 = "somemodule:somefunction",
> feature2 = "another.module:SomeClass [extra1,extra2]",
> ),
> "mime.parsers": {
> "application/atom+xml": "something:atom_parser [feedparser]"
> }
> },
> extras_require = dict(
> feedparser = [...],
> extra1 = [...],
> extra2 = [...],
> )
> )
>
> Anyway, this would make the most common use case for eggs-as-plugins very
> easy: an application or framework would simply define entry points, and
> plugin projects would declare the ones they offer in their setup script.
>
> I think this is a fantastic idea and I'm about to leap into implementing
> it. :)
>
>
> >This puts complex middleware construction into the function that is
> >referenced. This function might be, in turn, an import from a
> >framework. Or it might be some complex setup specific to the
> >application. Whatever.
> >
> >The API would look like:
> >
> >wsgiapp = wsgiref.get_egg_application('SomeApplication[feature1]')
> >
> >Which ultimately resolves to:
> >
> >wsgiapp = some_function()
> >
> >get_egg_application could also take a pkg_resources.Distribution object.
>
> Yeah, I'm thinking that this could be implemented as something like:
>
> import pkg_resources
>
> def get_wsgi_app(project_name, app_name, *args, **kw):
> dist = pkg_resources.require(project_name)[0]
> return dist.load_entry_point('wsgi.app_factories',
> app_name)(*args,**kw)
>
> with all the heavy lifting happening in the pkg_resources.Distribution
> class,
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Chris McDonough wrote: > Sorry, I think I may have lost track of where we were going wrt the > deployment spec. Specifically, I don't know how we got to using eggs > (which I'd really like to, BTW, they're awesome conceptually!) from > where we were in the discussion about configuring a WSGI pipeline. What > is a "feature"? What is an "import map"? "Entry point"? Should I just > get more familiar with eggs to understand what's being discussed here or > did I miss a few posts? It wouldn't hurt to read up on eggs. It's not obvious how they fit here, and it's taken me a while to figure it out. But specifically: * Eggs are packages. Packages can have optional features. Those features can have additional requirements (external packages) that the base package does not have. Package specifications are spelled like "PackageName>=VERSION_NUMBER[FeatureName]" * Import maps and entry points are new things we're discussing now. They are kind of the same thing; basically an entry point maps a logical specification (like a 'wsgi.app_factory' named 'foo') to a actual import statement. That's the configuration file: [wsgi.app_factory] app = mymodule.wsgi:make_app Which means to get an object "app" which fulfills the spec "wsgi.app_factory" you would do "from mymodule.wsgi import make_app" Eggs have an PackageName.egg-info directory, where configuration files can go, and pkg_resources (which is part of setuptools, and associated with easy_install, and defines the require() function) can find and parse them. -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 07:26 PM 7/24/2005 -0500, Ian Bicking wrote:
>Phillip J. Eby wrote:
>>The actual syntax I'm going to end up with is:
>> entry_points = {
>> "wsgi.app_factories": [
>> "feature1 = somemodule:somefunction",
>> "feature2 = another.module:SomeClass [extra1,extra2]",
>> ]
>> }
>
>That seems weird to put the assignment inside a string, instead of:
>
>entry_points = {
> 'wsgi.app_factories': {
> 'app': 'somemodule:somefunction',
> },
>}
It turned out that EntryPoint objects really want to know their 'name' for
ease of use in various APIs, and it also made it really easy to do stuff
like "map(EntryPoint.parse, lines)" to get a list of entry points from a
list of lines.
>Also, is there any default name?
Huh?
> Like for a package that distributes only one application. Or these
> just different spellings for the same thing?
I don't understand you. The most minimal way to specify a single entry
point in setup() is with:
entry_points = """
[groupname.here]
entryname = some.thing:here
"""
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Phillip J. Eby wrote: >> Like for a package that distributes only one application. Or these >> just different spellings for the same thing? > > > I don't understand you. The most minimal way to specify a single entry > point in setup() is with: > > entry_points = """ > [groupname.here] > entryname = some.thing:here > """ Basically, in the (I think common) case where a package only provides one entry point, do we have to choose an arbitrary entry name. Like, a package that implements one web application; it seems like that application would have to be named. Maybe that name could match the package name, or a fixed name we agree upon, but otherwise it adds another name to the mix. -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 08:35 PM 7/24/2005 -0400, Chris McDonough wrote: >Sorry, I think I may have lost track of where we were going wrt the >deployment spec. Specifically, I don't know how we got to using eggs >(which I'd really like to, BTW, they're awesome conceptually!) from >where we were in the discussion about configuring a WSGI pipeline. What >is a "feature"? What is an "import map"? "Entry point"? Should I just >get more familiar with eggs to understand what's being discussed here or >did I miss a few posts? I suggest this post as the shortest architectural introduction to the whole egg thang: http://mail.python.org/pipermail/distutils-sig/2005-June/004652.html It explains pretty much all of the terminology I'm currently using, except for the new terms invented today... Entry points are a new concept, invented today by Ian and myself. Ian proposed having a mapping file (which I dubbed an "import map") included in an egg's metadata, and then referring to named entries from a pipeline descriptor, so that you don't have to know or care about the exact name to import. The application or middleware factory name would be looked up in the egg's import map in order to find the actual factory object. I took Ian's proposal and did two things: 1) Generalized the idea to a concept of "entry points". An entry point is a name that corresponds to an import specification, and an optional list of "extras" (see terminology link above) that the entry point may require. Entry point names exist in a namespace called an "entry point group", and I implied that the WSGI deployment spec would define two such groups: wsgi.applications and wsgi.middleware, but a vast number of other possibilities for entry points and groups exist. In fact, I went ahead and implemented them in setuptools today, and realized I could use them to register setup commands with setuptools, making it extensible by any project that registers entry points in a 'distutils.commands' group. 2) I then proposed that we extend our deployment descriptor (.wsgi file) syntax so that you can do things like: [foo from SomeProject] # configuration here What this does is tell the WSGI deployment API to look up the "foo" entry point in either the wsgi.middleware or wsgi.applications entry point group for the named project, according to whether it's the last item in the .wsgi file. It then invokes the factory as before, with the configuration values as keyword arguments. This proposal is of course an *extension*; it should still be possible to use regular dotted names as section headings, if you haven't yet drunk the setuptools kool-aid. But, it makes for interesting possibilities because we could now have a tool that reads a WSGI deployment descriptor and runs easy_install to find and download the right projects. So, you could potentially just write up a descriptor that lists what you want and the server could install it, although I think I personally would want to run a tool explicitly; maybe I'll eventually add a --wsgi=FILENAME option to EasyInstall that would tell it to find out what to install from a WSGI deployment descriptor. That would actually be pretty cool, when you realize it means that all you have to do to get an app deployed across a bunch of web servers is to copy the deployment descriptor and tell 'em to install stuff. You can always create an NFS-mounted cache directory where you put pre-built eggs, and EasyInstall would just fetch and extract them in that case. Whew. Almost makes me wish I was back in my web apps shop, where this kind of thing would've been *really* useful to have. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 08:49 PM 7/24/2005 -0500, Ian Bicking wrote:
>Chris McDonough wrote:
>>Sorry, I think I may have lost track of where we were going wrt the
>>deployment spec. Specifically, I don't know how we got to using eggs
>>(which I'd really like to, BTW, they're awesome conceptually!) from
>>where we were in the discussion about configuring a WSGI pipeline. What
>>is a "feature"? What is an "import map"? "Entry point"? Should I just
>>get more familiar with eggs to understand what's being discussed here or
>>did I miss a few posts?
>
>It wouldn't hurt to read up on eggs. It's not obvious how they fit here,
>and it's taken me a while to figure it out. But specifically:
>
>* Eggs are packages. Packages can have optional features.
I've taken to using the term "project" to mean a collection of packages,
scripts, data files, etc., wrapped with a setup script. In order to avoid
confusion with other kinds of "features" and "options", the official term
for those things is now "extras". An "extra" is some optional capability
of a project that may incur additional requirements.
> Those features can have additional requirements (external packages)
> that the base package does not have. Package specifications are spelled
> like "PackageName>=VERSION_NUMBER[FeatureName]"
Actually, it's "ProjectName[extra,...]>=version", and you can list multiple
version operators, like "FooBar>1.2,<2.1,==2.6,>3.0" to mean versions
between 1.2 and 2.1 exclusive, and anything *after* 3.0, but 2.6 was okay
too. :)
I'm proposing that for WSGI entry points, we allow everything but the
[extras_list] in a section heading, e.g.:
[wiki from FooBarWiki>=2.0]
would mean what it looks like it does. By the way, all this version
parsing, dependency checking, PyPI-finding, auto-download and build from
source or binary stuff already exists; it's not a hypothetical
pie-in-the-sky proposal.
>* Import maps and entry points are new things we're discussing now. They
>are kind of the same thing; basically an entry point maps a logical
>specification (like a 'wsgi.app_factory' named 'foo') to a actual import
>statement. That's the configuration file:
>
> [wsgi.app_factory]
> app = mymodule.wsgi:make_app
>
>Which means to get an object "app" which fulfills the spec
>"wsgi.app_factory" you would do "from mymodule.wsgi import make_app"
>
>Eggs have an PackageName.egg-info directory, where configuration files can
>go, and pkg_resources (which is part of setuptools, and associated with
>easy_install, and defines the require() function) can find and parse them.
Yes, and with the CVS HEAD version of setuptools you can now specify a
project's entry point map in it setup script, and it will generate the
entry point file in the project's .egg-info directory, and parse it at
runtime when you request lookup of an entry point. There's an API in
pkg_resources that lets you do:
factory = load_entry_point("ProjectName", "wsgi.app_factory", "app")
which will do the same as if you had said "from mymodule.wsgi import
make_app as factory".
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Thanks... I'm still confused about high level requirements so please try to be patient with me as I try get back on track. These are the requirements as I understand them: 1. We want to be able to distribute WSGI applications and middleware (presumably in a format supported by setuptools). 3. We want to be able to configure a WSGI application in order to create an application instance. 2. We want a way to combine configured instances of those applications into pipelines and start an "instance" of a pipeline. Are these requirements the ones being discussed? If so, which of the config file formats we've been discussing matches which requirement? Thanks, - C On Sun, 2005-07-24 at 22:24 -0400, Phillip J. Eby wrote: > At 08:35 PM 7/24/2005 -0400, Chris McDonough wrote: > >Sorry, I think I may have lost track of where we were going wrt the > >deployment spec. Specifically, I don't know how we got to using eggs > >(which I'd really like to, BTW, they're awesome conceptually!) from > >where we were in the discussion about configuring a WSGI pipeline. What > >is a "feature"? What is an "import map"? "Entry point"? Should I just > >get more familiar with eggs to understand what's being discussed here or > >did I miss a few posts? > > I suggest this post as the shortest architectural introduction to the whole > egg thang: > > http://mail.python.org/pipermail/distutils-sig/2005-June/004652.html > > It explains pretty much all of the terminology I'm currently using, except > for the new terms invented today... > > Entry points are a new concept, invented today by Ian and myself. Ian > proposed having a mapping file (which I dubbed an "import map") included in > an egg's metadata, and then referring to named entries from a pipeline > descriptor, so that you don't have to know or care about the exact name to > import. The application or middleware factory name would be looked up in > the egg's import map in order to find the actual factory object. > > I took Ian's proposal and did two things: > > 1) Generalized the idea to a concept of "entry points". An entry point is > a name that corresponds to an import specification, and an optional list of > "extras" (see terminology link above) that the entry point may > require. Entry point names exist in a namespace called an "entry point > group", and I implied that the WSGI deployment spec would define two such > groups: wsgi.applications and wsgi.middleware, but a vast number of other > possibilities for entry points and groups exist. In fact, I went ahead and > implemented them in setuptools today, and realized I could use them to > register setup commands with setuptools, making it extensible by any > project that registers entry points in a 'distutils.commands' group. > > 2) I then proposed that we extend our deployment descriptor (.wsgi file) > syntax so that you can do things like: > > [foo from SomeProject] > # configuration here > > What this does is tell the WSGI deployment API to look up the "foo" entry > point in either the wsgi.middleware or wsgi.applications entry point group > for the named project, according to whether it's the last item in the .wsgi > file. It then invokes the factory as before, with the configuration values > as keyword arguments. > > This proposal is of course an *extension*; it should still be possible to > use regular dotted names as section headings, if you haven't yet drunk the > setuptools kool-aid. But, it makes for interesting possibilities because > we could now have a tool that reads a WSGI deployment descriptor and runs > easy_install to find and download the right projects. So, you could > potentially just write up a descriptor that lists what you want and the > server could install it, although I think I personally would want to run a > tool explicitly; maybe I'll eventually add a --wsgi=FILENAME option to > EasyInstall that would tell it to find out what to install from a WSGI > deployment descriptor. > > That would actually be pretty cool, when you realize it means that all you > have to do to get an app deployed across a bunch of web servers is to copy > the deployment descriptor and tell 'em to install stuff. You can always > create an NFS-mounted cache directory where you put pre-built eggs, and > EasyInstall would just fetch and extract them in that case. > > Whew. Almost makes me wish I was back in my web apps shop, where this kind > of thing would've been *really* useful to have. > ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
BTW, a simple example that includes proposed solutions for all of these requirements would go a long way towards helping me (and maybe others) understand how all the pieces fit together. Maybe something like: - Define two simple WSGI components: a WSGI middleware and a WSGI application. - Describe how to package each as an indpendent egg. - Describe how to configure an instance of the application. - Describe how to configure an instance of the middleware - Describe how to string them together into a pipeline. - C On Mon, 2005-07-25 at 02:33 -0400, Chris McDonough wrote: > Thanks... > > I'm still confused about high level requirements so please try to be > patient with me as I try get back on track. > > These are the requirements as I understand them: > > 1. We want to be able to distribute WSGI applications and middleware > (presumably in a format supported by setuptools). > > 3. We want to be able to configure a WSGI application in order > to create an application instance. > > 2. We want a way to combine configured instances of those > applications into pipelines and start an "instance" of a pipeline. > > Are these requirements the ones being discussed? If so, which of the > config file formats we've been discussing matches which requirement? > > Thanks, > > - C > > On Sun, 2005-07-24 at 22:24 -0400, Phillip J. Eby wrote: > > At 08:35 PM 7/24/2005 -0400, Chris McDonough wrote: > > >Sorry, I think I may have lost track of where we were going wrt the > > >deployment spec. Specifically, I don't know how we got to using eggs > > >(which I'd really like to, BTW, they're awesome conceptually!) from > > >where we were in the discussion about configuring a WSGI pipeline. What > > >is a "feature"? What is an "import map"? "Entry point"? Should I just > > >get more familiar with eggs to understand what's being discussed here or > > >did I miss a few posts? > > > > I suggest this post as the shortest architectural introduction to the whole > > egg thang: > > > > http://mail.python.org/pipermail/distutils-sig/2005-June/004652.html > > > > It explains pretty much all of the terminology I'm currently using, except > > for the new terms invented today... > > > > Entry points are a new concept, invented today by Ian and myself. Ian > > proposed having a mapping file (which I dubbed an "import map") included in > > an egg's metadata, and then referring to named entries from a pipeline > > descriptor, so that you don't have to know or care about the exact name to > > import. The application or middleware factory name would be looked up in > > the egg's import map in order to find the actual factory object. > > > > I took Ian's proposal and did two things: > > > > 1) Generalized the idea to a concept of "entry points". An entry point is > > a name that corresponds to an import specification, and an optional list of > > "extras" (see terminology link above) that the entry point may > > require. Entry point names exist in a namespace called an "entry point > > group", and I implied that the WSGI deployment spec would define two such > > groups: wsgi.applications and wsgi.middleware, but a vast number of other > > possibilities for entry points and groups exist. In fact, I went ahead and > > implemented them in setuptools today, and realized I could use them to > > register setup commands with setuptools, making it extensible by any > > project that registers entry points in a 'distutils.commands' group. > > > > 2) I then proposed that we extend our deployment descriptor (.wsgi file) > > syntax so that you can do things like: > > > > [foo from SomeProject] > > # configuration here > > > > What this does is tell the WSGI deployment API to look up the "foo" entry > > point in either the wsgi.middleware or wsgi.applications entry point group > > for the named project, according to whether it's the last item in the .wsgi > > file. It then invokes the factory as before, with the configuration values > > as keyword arguments. > > > > This proposal is of course an *extension*; it should still be possible to > > use regular dotted names as section headings, if you haven't yet drunk the > > setuptools kool-aid. But, it makes for interesting possibilities because > > we could now have a tool that reads a WSGI deployment descriptor and runs > > easy_install to find and download the right projects. So, you could > > potentially just write up a descriptor that lists what you want and the > > server could install it, although I think I personally would want to run a > > tool explicitly; maybe I'll eventually add a --wsgi=FILENAME option to > > EasyInstall that would tell it to find out what to install from a WSGI > > deployment descriptor. > > > > That would actually be pretty cool, when you realize it means that all you > > have to do to get an app deployed across a bunch of web servers is to copy > > the deployment descriptor and te
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Actually, let me give this a shot.
We package up an egg called helloworld.egg. It happens to contain
something that can be used as a WSGI component. Let's say it's a WSGI
application that always returns 'Hello World'. And let's say it also
contains middleware that lowercases anything that passes through before
it's returned.
The implementations of these components could be as follows:
class HelloWorld:
def __init__(self, app, **kw):
pass # nothing to configure
def __call__(self, environ, start_response):
start_response('200 OK', [])
return ['Hello World']
class Lowercaser:
def __init__(self, app, **kw):
self.app = app
# nothing else to configure
def __call__(self, environ, start_response):
for chunk in self.app(environ, start_response):
yield chunk.lower()
An import map would ship inside of the egg-info dir:
[wsgi.app_factories]
helloworld = helloworld:HelloWorld
lowercaser = helloworld:Lowercaser
So we install the egg and this does nothing except allow it to be used
from within Python.
But when we create a "deployment descriptor" like so in a text editor:
[helloworld from helloworld]
[lowercaser from helloworld]
... and run some "starter" script that parses that as a pipeline,
creates the two instances, wires them together, and we get a running
pipeline?
Am I on track?
OK, back to Battlestar Galactica ;-)
On Mon, 2005-07-25 at 02:40 -0400, Chris McDonough wrote:
> BTW, a simple example that includes proposed solutions for all of these
> requirements would go a long way towards helping me (and maybe others)
> understand how all the pieces fit together. Maybe something like:
>
> - Define two simple WSGI components: a WSGI middleware and a WSGI
> application.
>
> - Describe how to package each as an indpendent egg.
>
> - Describe how to configure an instance of the application.
>
> - Describe how to configure an instance of the middleware
>
> - Describe how to string them together into a pipeline.
>
> - C
>
>
> On Mon, 2005-07-25 at 02:33 -0400, Chris McDonough wrote:
> > Thanks...
> >
> > I'm still confused about high level requirements so please try to be
> > patient with me as I try get back on track.
> >
> > These are the requirements as I understand them:
> >
> > 1. We want to be able to distribute WSGI applications and middleware
> > (presumably in a format supported by setuptools).
> >
> > 3. We want to be able to configure a WSGI application in order
> > to create an application instance.
> >
> > 2. We want a way to combine configured instances of those
> > applications into pipelines and start an "instance" of a pipeline.
> >
> > Are these requirements the ones being discussed? If so, which of the
> > config file formats we've been discussing matches which requirement?
> >
> > Thanks,
> >
> > - C
> >
> > On Sun, 2005-07-24 at 22:24 -0400, Phillip J. Eby wrote:
> > > At 08:35 PM 7/24/2005 -0400, Chris McDonough wrote:
> > > >Sorry, I think I may have lost track of where we were going wrt the
> > > >deployment spec. Specifically, I don't know how we got to using eggs
> > > >(which I'd really like to, BTW, they're awesome conceptually!) from
> > > >where we were in the discussion about configuring a WSGI pipeline. What
> > > >is a "feature"? What is an "import map"? "Entry point"? Should I just
> > > >get more familiar with eggs to understand what's being discussed here or
> > > >did I miss a few posts?
> > >
> > > I suggest this post as the shortest architectural introduction to the
> > > whole
> > > egg thang:
> > >
> > > http://mail.python.org/pipermail/distutils-sig/2005-June/004652.html
> > >
> > > It explains pretty much all of the terminology I'm currently using,
> > > except
> > > for the new terms invented today...
> > >
> > > Entry points are a new concept, invented today by Ian and myself. Ian
> > > proposed having a mapping file (which I dubbed an "import map") included
> > > in
> > > an egg's metadata, and then referring to named entries from a pipeline
> > > descriptor, so that you don't have to know or care about the exact name
> > > to
> > > import. The application or middleware factory name would be looked up in
> > > the egg's import map in order to find the actual factory object.
> > >
> > > I took Ian's proposal and did two things:
> > >
> > > 1) Generalized the idea to a concept of "entry points". An entry point
> > > is
> > > a name that corresponds to an import specification, and an optional list
> > > of
> > > "extras" (see terminology link above) that the entry point may
> > > require. Entry point names exist in a namespace called an "entry point
> > > group", and I implied that the WSGI deployment spec would define two such
> > > groups: wsgi.applications and wsgi.middleware, but a vast number of other
> > > possibilities for entry points and groups exist. In fact, I went ahead
> > > and
> > > implement
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 03:02 AM 7/25/2005 -0400, Chris McDonough wrote:
>Actually, let me give this a shot.
>
>We package up an egg called helloworld.egg. It happens to contain
>something that can be used as a WSGI component. Let's say it's a WSGI
>application that always returns 'Hello World'. And let's say it also
>contains middleware that lowercases anything that passes through before
>it's returned.
>
>The implementations of these components could be as follows:
>
>class HelloWorld:
> def __init__(self, app, **kw):
> pass # nothing to configure
>
> def __call__(self, environ, start_response):
> start_response('200 OK', [])
> return ['Hello World']
I'm thinking that an application like this wouldn't take an 'app'
constuctor parameter, and if it takes no configuration parameters it
doesn't need **kw, but good so far.
>class Lowercaser:
> def __init__(self, app, **kw):
> self.app = app
> # nothing else to configure
>
> def __call__(self, environ, start_response):
> for chunk in self.app(environ, start_response):
> yield chunk.lower()
Again, no need for **kw if it doesn't take any configuration, but okay.
>An import map would ship inside of the egg-info dir:
>
>[wsgi.app_factories]
>helloworld = helloworld:HelloWorld
>lowercaser = helloworld:Lowercaser
I'm thinking it would be more like:
[wsgi.middleware]
lowercaser = helloworld:Lowercaser
[wsgi.apps]
helloworld = helloworld:HelloWorld
and you'd specify it in the setup script as something like this:
setup(
#...
entry_points = {
'wsgi.apps': ['helloworld = helloworld:HelloWorld']
'wsgi.middleware': ['lowercaser = helloworld:Lowercaser']
}
)
(And the CVS version of setuptools already supports this.)
>So we install the egg and this does nothing except allow it to be used
>from within Python.
>
>But when we create a "deployment descriptor" like so in a text editor:
>
>[helloworld from helloworld]
>
>[lowercaser from helloworld]
Opposite order, though; the lowercaser comes first because it's the
middleware; the application would always come last, because they're listed
in the order in which they receive data, just like a pipes-and-filters
command line.
>... and run some "starter" script that parses that as a pipeline,
... possibly using a #! line if you're using CGI or FastCGI with Apache or
some other non-Python webserver.
>creates the two instances, wires them together, and we get a running
>pipeline?
>
>Am I on track?
Definitely.
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Great. Given that, I've created the beginnings of a more formal
specification:
WSGI Deployment Specification
-
I use the term "WSGI component" in here as shorthand to indicate all
types of WSGI implementations (application, middleware).
The primary deployment concern is to create a way to specify the
configuration of an instance of a WSGI component within a
declarative configuration file. A secondary deployment concern is
to create a way to "wire up" components together into a specific
deployable "pipeline".
Pipeline Descriptors
Pipeline descriptors are file representations of a particular WSGI
"pipeline". They include enough information to configure,
instantiate, and wire together WSGI apps and middleware components
into one pipeline for use by a WSGI server. Installation of the
software which composes those components is handled separately.
In order to define a pipeline, we use a ".ini"-format configuration
file conventionally named '.wsgi'. This file may
optionally be marked as executable and associated with a simple UNIX
interpreter via a leading hash-bang line to allow servers which
employ stdin and stdout streams (ala CGI) to run the pipeline
directly without any intermediation. For example, a deployment
descriptor named 'myapplication.wsgi' might be composed of the
following text::
#!/usr/bin/runwsgi
[mypackage.mymodule.factory1]
quux = arbitraryvalue
eekx = arbitraryvalue
[mypackage.mymodule.factory2]
foo = arbitraryvalue
bar = arbitraryvalue
Section names are Python-dotted-path names (or setuptools "entry
point names" described in a later section) which represent
factories. Key-value pairs within a given section are used as
keyword arguments to the factory that can be used as configuration
for the component being instantiated.
All sections in the deployment descriptor describe 'middleware'
except for the last section, which must describe an application.
Factories which construct middleware must return something which is
a WSGI "callable" by implementing the following API::
def factory(next_app, [**kw]):
""" next_app is the next application in the WSGI pipeline,
**kw is optional, and accepts the key-value pairs
that are used in the section as a dictionary, used
for configuration """
Factories which construct middleware must return something which is
a WSGI "callable" by implementing the following API::
def factory([**kw]):
**kw is optional, and accepts the key-value pairs
that are used in the section as a dictionary, used
for configuration """
A deployment descriptor can also be parsed from within Python. An
importable configurator which resides in 'wsgiref' exposes a
function that accepts a single argument, "configure"::
>>> from wsgiref.runwsgi import parse_deployment
>>> appchain = parse_deployment('myapplication.wsgi')
'appchain' will be an object representing the fully configured
"pipeline". 'parse_deployment' is guaranteed to return something
that implements the WSGI "callable" API described in PEP 333.
Entry Points
On Mon, 2005-07-25 at 10:39 -0400, Phillip J. Eby wrote:
> At 03:02 AM 7/25/2005 -0400, Chris McDonough wrote:
> >Actually, let me give this a shot.
> >
> >We package up an egg called helloworld.egg. It happens to contain
> >something that can be used as a WSGI component. Let's say it's a WSGI
> >application that always returns 'Hello World'. And let's say it also
> >contains middleware that lowercases anything that passes through before
> >it's returned.
> >
> >The implementations of these components could be as follows:
> >
> >class HelloWorld:
> > def __init__(self, app, **kw):
> > pass # nothing to configure
> >
> > def __call__(self, environ, start_response):
> > start_response('200 OK', [])
> > return ['Hello World']
>
> I'm thinking that an application like this wouldn't take an 'app'
> constuctor parameter, and if it takes no configuration parameters it
> doesn't need **kw, but good so far.
>
>
> >class Lowercaser:
> > def __init__(self, app, **kw):
> > self.app = app
> > # nothing else to configure
> >
> > def __call__(self, environ, start_response):
> > for chunk in self.app(environ, start_response):
> > yield chunk.lower()
>
> Again, no need for **kw if it doesn't take any configuration, but okay.
>
>
> >An import map would ship inside of the egg-info dir:
> >
> >[wsgi.app_factories]
> >helloworld = helloworld:HelloWorld
> >lowercaser = helloworld:Lowercaser
>
> I'm thinking it would be more like:
>
> [wsgi.middleware]
> lowercaser = helloworld:Lowercaser
>
> [wsgi.apps]
> helloworld = helloworld:HelloWorld
>
> and you'd specify it in the setup script as something like this:
>
> setup(
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 12:59 PM 7/25/2005 -0400, Chris McDonough wrote:
> In order to define a pipeline, we use a ".ini"-format configuration
Ultimately I think the spec will need a formal description of what that
means exactly, including such issues as a PEP 263-style "encoding"
specifier, and the precise format of values. But I'm fine with adding all
that myself, since I'm going to have to specify it well enough to create a
parser anyway.
With respect to the format, I'm actually leaning towards either treating
the settings as Python assignment statements (syntactically speaking) or
restricting the values to being single-token Python literals (i.e.,
numbers, strings, or True/False/None, but not tuples, lists, or other
expressions).
Interestingly enough, I think you could actually define the entire format
in terms of standard Python-language tokens, it's just the higher-level
syntax that differs from Python's. Although actually using the "tokenize"
module to scan it would mean that all lines' content would need to start
exactly at the left margin, with no indentation. Probably not a big deal,
though.
The syntax would probably be something like:
pipeline::= section*
section ::= heading assignment*
heading ::= '[' qname trailer ']' NEWLINE
assignment ::= NAME '=' value NEWLINE
qname ::= NAME ('.' NAME) *
trailer ::= "from" requirement | "options"
value ::= NUMBER | STRING | "True" | "False" | "None"
requirement ::= NAME versionlist?
versionlist ::= versionspec (',' versionspec)*
versionspec ::= relop STRING
relop ::= "<" | "<=" | "==" | "!=" | ">=" | ">"
The versions would have to be strings in order to avoid problems parsing
e.g '2.1a4' as a number. And if we were going to allow structures like
tuples or lists or dictionaries, then we'd need to expand on 'value' a
little bit, but not as much as if we allowed arbitrary expressions.
> file conventionally named '.wsgi'. This file may
> optionally be marked as executable and associated with a simple UNIX
> interpreter via a leading hash-bang line to allow servers which
> employ stdin and stdout streams (ala CGI) to run the pipeline
> directly without any intermediation.
For that matter, while doing development and testing, the interpreter could
be something like "#!invoke peak launch wsgifile", to launch the app in a
web browser from a localhost http server. (Assuming I added a "wsgifile"
command to PEAK, of course.)
> Factories which construct middleware must return something which is
> a WSGI "callable" by implementing the following API::
>
> def factory(next_app, [**kw]):
> """ next_app is the next application in the WSGI pipeline,
> **kw is optional, and accepts the key-value pairs
> that are used in the section as a dictionary, used
> for configuration """
Note that you can also just list the parameter names you take, or no
parameter names at all. I don't want to imply that you *have* to use kw,
because it's fairly easy to envision simple middleware components that only
take two or three parameters, or maybe even just one (e.g., their config
file name).
> Factories which construct middleware must return something which is
> a WSGI "callable" by implementing the following API::
You probably meant "application" or "terminal application" here. (Or
whatever term we end up with for an application that isn't middleware.
> A deployment descriptor can also be parsed from within Python. An
> importable configurator which resides in 'wsgiref' exposes a
> function that accepts a single argument, "configure"::
>
> >>> from wsgiref.runwsgi import parse_deployment
> >>> appchain = parse_deployment('myapplication.wsgi')
>
> 'appchain' will be an object representing the fully configured
> "pipeline". 'parse_deployment' is guaranteed to return something
> that implements the WSGI "callable" API described in PEP 333.
Or raise SyntaxError for a malformed descriptor file, or ImportError if an
application import failed or an entry point couldn't be found, or
DistributionNotFound if a needed egg couldn't be found, or VersionConflict
if it needs a conflicting version.
Or really it could raise anything if one of the factories failed, come to
think of it.
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 01:35 PM 7/25/2005 -0400, Phillip J. Eby wrote: >heading ::= '[' qname trailer ']' NEWLINE Oops. That should've been "trailer?", since the trailer is optional. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Phillip J. Eby wrote: > At 12:59 PM 7/25/2005 -0400, Chris McDonough wrote: > >> In order to define a pipeline, we use a ".ini"-format configuration > > > Ultimately I think the spec will need a formal description of what that > means exactly, including such issues as a PEP 263-style "encoding" > specifier, and the precise format of values. But I'm fine with adding all > that myself, since I'm going to have to specify it well enough to create a > parser anyway. Incidentally I have a generic ini parser here: http://svn.w4py.org/home/ianb/wsgikit_old_config/iniparser.py I suspect I'm doing the character decoding improperly (line-by-line instead of opening the file with the given character encoding), but otherwise it's been sufficiently generic and workable, and should allow for doing more extensive parsing of things like section headers. -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
Hi All,
I'm a bit late coming to all this and didn't really see the benefits of
the new format over what we already do so I set out to contrast new and
old to demonstrate why it wasn't *that* useful. I've since changed my
mind and think it is great but here is the contrasting I did anyway. I'd
be pleased to hear all the glaring errors :-)
Here is a new example: we want to have an application that returns a
GZip encoded "hello world" string after it has been made lowercase by
case changer middleware taking a parameter newCase. The GZip middleware
is an optional feature of the modules in wsgiFilters.egg and the
CaseChanger middleware and HelloWorld application are in the helloworld.egg.
The classes look like this:
class HelloWorld:
def __call__(self, environ, start_response):
start_response('200 OK', [('Content-type','text/plain')])
return ['Hello World']
class CaseChanger:
def __init__(self, app, newCase):
self.app = app
self.newCase = newCase
def __call__(self, environ, start_response):
for chunk in self.app(environ, start_response):
if self.newCase == 'lower':
yield chunk.lower()
else:
yield chunk
Class GZip:
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
# Do clever things with headers here (omitted)
for chunk in self.app(environ, start_response):
yeild gzip(chunk)
The way we would write our application at the moment is as follows:
from pkg_resources import require
require('helloworld >= 0.2')
from helloworld import Helloworld
require('wsgiFilters[GZip] == 1.4.3')
from wsgiFilters import GZip
pipeline = GZip(
app = CaseChanger(
app = HelloWorld(),
newCase = 'lowercase',
)
)
With pipeline itself somehow being executed as a WSGI application.
The new way is like this (correct me if I'm wrong)
The modules have egg_info files like this respectively defining the
"entry points":
wsgiFilters.egg:
[wsgi.middleware]
gzipper = GZip:GZip
helloworld.egg:
[wsgi.middleware]
cs = helloworld:CaseChanger
[wsgi.app]
myApp = helloworld:HelloWorld
We would then write an "import map" (below) based on the "deployment
descriptors" in the .eggs used to describe the "entry points" into the
eggs. The order the "pipeline" would be built is the same as in the
Python example eg middleware first then application.
[gzipper from wsgiFilters[GZip] == 1.4.3]
[cs from helloworld >= 0.2 ]
newCase = 'lower'
[myApp from helloworld >= 0.2]
It is loaded using an as yet unwritten modules which uses a factory
returning a middleware pipeline equivalent to what would be produced in
the Python example (is this very last bit correct?)
Doing things this new way has the following advantages:
* We have specified explicitly in the setup.py of the eggs that the
middleware and applications we are importing are actually middleware and
an application
* It is simpler for a non-technical user.
* There are lots of other applications for the ideas being discussed
It has the following disadvantages:
* We are limited as to what we can use as variable names. Existing
middleware would need customising to only accept basic parameters.
* We require all WSGI coders to use the egg format.
* Users can't customise the middleware in the configuration file (eg by
creating a derived class etc and you lose flexibility).
* If we use a Python file we can directly import and manipulate the
pipeline (I guess you can do this anyway once your factory has returned
the pipeline)
Both methods are the same in that
* We have specified the order of the pipeline and the middleware and
applications involved
* Auto-downloading and installation of middleware and applications based
on version requirements is possible (thanks to PJE's eggs)
* We have specified which versions of modules we require.
* Both could call a script such as wsgi_CGI.py wsgi_mod_python.py etc to
execute the WSGI pipeline so both method's files could be distributed as
a single file and would auto download their own dependencies.
Other ideas:
Is it really necessary to be able to give an entry point a name? If not
because we know what we want to import anyway, we can combine the
deployment descriptor into the import map:
[GZip:GZip from wsgiFilters[GZip] == 1.4.3]
We can then simplify the deployment descriptor like this:
[wsgi.middleware]
GZip:GZip
And then remove the colons and give a fully qualified Python-style path:
[GZip.GZip from wsgiFilters[GZip] == 1.4.3]
and
[wsgi.middleware]
GZip.GZip
Is this not better? Why do you need to assign names to entry points?
Although writing a middleware chain is dead easy for a Python
programmer, it isn't for the end user and if you compare the end user
files from this example I know which one I'd rather explain to someone.
So although this dep
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
At 10:54 PM 7/25/2005 +0100, James Gardner wrote: >The new way is like this (correct me if I'm wrong) > >The modules have egg_info files like this respectively defining the >"entry points": > >wsgiFilters.egg: > >[wsgi.middleware] >gzipper = GZip:GZip Almost; this one should be: [wsgi.middleware] gzipper = GZip:GZip [GZip] So that using gzipper doesn't require specifying "extras" in the pipeline descriptor. See below. >helloworld.egg: > >[wsgi.middleware] >cs = helloworld:CaseChanger > >[wsgi.app] >myApp = helloworld:HelloWorld > > >We would then write an "import map" (below) based on the "deployment >descriptors" in the .eggs used to describe the "entry points" into the >eggs. Actually, the new thing you write is the deployment descriptor or pipeline descriptor. The "import map" is the thing you put in the eggs' setup.py to list the entry points offered by the eggs. >The order the "pipeline" would be built is the same as in the >Python example eg middleware first then application. > >[gzipper from wsgiFilters[GZip] == 1.4.3] >[cs from helloworld >= 0.2 ] >newCase = 'lower' >[myApp from helloworld >= 0.2] You wouldn't need the [GZip] part if it were declared with the entry point, as I showed above. >It is loaded using an as yet unwritten modules which uses a factory >returning a middleware pipeline equivalent to what would be produced in >the Python example (is this very last bit correct?) Yes. The order in the file is the order in which the items are invoked by the controlling server. >Doing things this new way has the following advantages: >* We have specified explicitly in the setup.py of the eggs that the >middleware and applications we are importing are actually middleware and >an application >* It is simpler for a non-technical user. >* There are lots of other applications for the ideas being discussed > >It has the following disadvantages: >* We are limited as to what we can use as variable names. Existing >middleware would need customising to only accept basic parameters. This depends a lot on the details of the .ini-like format, which are still up in the air. >* We require all WSGI coders to use the egg format. Not so; you can use [GZip.GZip] as a section header in order to do just a plain ol' import. >* Users can't customise the middleware in the configuration file (eg by >creating a derived class etc and you lose flexibility). No, but all they have to do is create a Python file and refer to it, and they are thereby encouraged to separate code from configuration. :) >* If we use a Python file we can directly import and manipulate the >pipeline (I guess you can do this anyway once your factory has returned >the pipeline) Yep. >Both methods are the same in that >* We have specified the order of the pipeline and the middleware and >applications involved >* Auto-downloading and installation of middleware and applications based >on version requirements is possible (thanks to PJE's eggs) One difference here: the .ini format is parseable to determine what eggs are needed without executing arbitrary code. >Other ideas: > >Is it really necessary to be able to give an entry point a name? Yes. Entry points are a generic setuptools mechanism now, and they have names. However, this doesn't mean they all have to be exported by an egg's import map. >If not >because we know what we want to import anyway, we can combine the >deployment descriptor into the import map: > >[GZip:GZip from wsgiFilters[GZip] == 1.4.3] We could perhaps still allow that format; the format is still being discussed. However, this would just be a function of the .wsgi file, and doesn't affect the concept of "entry points". It's just naming a factory directly instead of accessing it via an entry point. >We can then simplify the deployment descriptor like this: > >[wsgi.middleware] >GZip:GZip If you don't care about the entry point, you can just not declare one. But you can't opt out of naming them. >Why do you need to assign names to entry points? Because other things that use entry points need names. For example, setuptools searches a "distutils.commands" entry point group to find commands that extend the normal setup commands. It certainly doesn't know in advance what commands the eggs are going to provide. The question you should be asking is, "Why do we have to use entry points to specify factories?", and the answer is, "we don't". :) >Although writing a middleware chain is dead easy for a Python >programmer, it isn't for the end user and if you compare the end user >files from this example I know which one I'd rather explain to someone. Yep. Ian gets the credit for further simplifying my "sequence of [factory.name] sections" proposal by coming up with the idea of having named entry points declared in an egg. I then took the entry points idea to its logical conclusion, even refactoring setuptools to use them for its own extensibility. >So althoug
Re: [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config
[cc:'d to distutils-sig because this is mostly about cool uses for the new EntryPoint facility of setuptools/pkg_resources] At 12:01 AM 7/26/2005 +0100, James Gardner wrote: >Hi Phillip, > >>There's one other advantage: this format will hopefully become as >>successful as WSGI itself in adoption by servers and applications. >>Hopefully within a year or so, *the* normal way to deploy a Python web >>app will be using a .wsgi file. >> >>Beyond that, we can hopefully begin to see "Python" rather than >>"framework X" as being what people write their web apps with. > >Well that would be absolutely wonderful but also looking fairly likely >which is great news. I've got to say a massive thank you for the eggs >format and easy install as well.. Python was really crying out for it and >it will be phenomenally useful. I've split all my code up as a result >because there is no need to worry about people having to install lots of >packages if it is all done automatically. > >One thought: I'd ideally like to be able to backup a WSGI deployment to >allow it to be easily redeployed on another server with a different >configuration or just restored in the event of data loss. This would >probably just involve making a zip file of all data files (including an >SQL dump) and then redistributing it with the .wsgi file. Have you had any >thoughts on how that could be achieved or is that something you wouldn't >want the .wsgi file to be used for? Whatever software installed the >dependencies of the .wsgi file would need to be aware of the data file and >what to do with it, perhaps simply by calling an install handler >somewhere? Umm, all getting a bit complicated but I just wondered if you >had had any thoughts of that nature? Well, you could define another set of entry point groups, like "wsgi.middleware.backup_handlers", which would contain entry points with the same names as in middleware, that would get called with the same configuration arguments as the application factories, but would then do some kind of backing up. Similarly, you could have an entry point group for restoration functions. These would have to defined by the same egg as the one with the factory, of course, although perhaps we could make the entry point names be the entry point targets instead of using the original entry point names. That additional level of indirection would let one egg define backup and restore services for another's factories. Perhaps the backup functions would return the name of a directory tree to back up, and the restore functions would receive some kind of zipfile or archive. Obviously that's a totally unbaked idea that would need a fair amount of thought, but there's nothing stopping anybody from fleshing it out as a tool and publishing a spec for the entry points it uses. >Oh sorry, another quick question: Is there any work underway auto-document >eggs using some of the docutils code if an appropriate specifier is made >in the egg_info file saying the egg is in restructured text or similar? >Would that be something you would be willing to include as a feature of >easy_install or is it a bit too messy? I'd love to be able to distribute a >.wsgi file and have all the documentation for the downloaded modules auto >created. If only some of the modules supported it it would still be quite >handy. I'm having a little trouble envisioning what you mean exactly. All that's really coming across is the idea that "there's some way to generate documentation from eggs". I'd certainly like to be able to see tools like epydoc or pydoc support generating documentation for an egg. However, there's a fair amount of balkanization in how you specify inputs for Python documentation tools, not unlike the previous balkanization of web servers and web apps/frameworks. Maybe somebody will come up with a similar lingua franca for documentation tools. With respect to adding more cool features to setup(), I plan to add a couple of entry point groups to setuptools that would support what you have in mind, though. There's already a distutils.commands group that allows you to register setup commands, but I also plan to add egg_info.writers and distutils.setup_args. The setup_args entry points would have the name of a parameter you'd like setup() to have, and would be a function that would get called on the Distribution object during its finalize_options() (so you can validate the argument). The egg_info.writers group will define entry points for writing metadata files as part of the egg_info command. Last, but not least, I'll add a 'setup_requires' argument to setup() that will specify eggs that need to be present for the setup script to run. With these three things in place, tools like the build_utils or py2exe and py2app won't have to monkeypatch the distutils in order to install themselves; they can instead just define entry points for setup() args and the new commands they add. And
