Re: os.path.normpath

[Graham Dumpleton]

Hari Sekhon wrote ..
> [EMAIL PROTECTED] wrote:
> > [EMAIL PROTECTED] wrote:
> >   
> >> I am using a windows box and passing a string like "../foo/../foo2"
> to
> >> normpath which then returns "..\\foo2". But if this string is going
> >> into a webpage link it should really be "../foo".
> >>
> >> Is there any way to tell os.path.normpath to act like we are an a unix
> >> style box?
> >> 
> >
> > Use posixpath.normpath() instead.
> >
> >   
> I can't seem to find posixpath in the docs, but I can import posixpath
> and a dir shows it does indeed have a normpath.

Quoting:

  http://www.python.org/doc/2.3.5/lib/node750.html

it says:

  posixpath
  -- Implementation of os.path on POSIX.

What it provides is therefore documented by os.path module. The posixpath
module is still accessible on non POSIX platforms though.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mod_python installation problem ..severity High

[Graham Dumpleton]

boney wrote:
> On doing this and duly following the installation and testing
> instructions at www.modpython.org and then pointing the url as
> http://localhost/test/mptest.py, i get the following error on the
> server log file:
>
> [Tue Nov 14 15:17:47 2006] [error] make_obcallback: could not import
> mod_python.apache.\n
> [Tue Nov 14 15:17:47 2006] [error] make_obcallback: Python path being
> used "['C:Program FilesApache
> GroupApache2binpython24.zip', '.DLLs', '.lib',
> '.libplat-win', '.liblib-tk', 'C:Program
> FilesApache GroupApache2bin']".
> [Tue Nov 14 15:17:47 2006] [error] python_handler: no interpreter
> callback found.
> [Tue Nov 14 15:17:47 2006] [error] [client 127.0.0.1] python_handler:
> Can't get/create interpreter.

Your registry settings for Python do not include the site-packages
directory where mod_python typically gets installed. Thus, trying
updating the registry to add the site-packages directory. As a guide,
see the following post from the mod_python mailing list archive as a
guide.


http://www.modpython.org/pipermail/mod_python/2006-September/021979.html

This is for Python 2.5, but should give an indicator of what to look
for. Update the PythonPath entry.

For any mod_python help in the future, suggest you subscribe to the
mod_python mailing list and ask your question there as there is a
higher concentration of people there who are familiar with the
software. Details of the mod_python mailing list are on the mod_python
web site.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python, WSGI, legacy web application

[Graham Dumpleton]

Ben Finney wrote:
> Howdy all,
>
> I'm working on a web application that is starting to gain a lot of
> back-end code written in Python. However, all the current interface
> code is written in legacy PHP. I'd like to slowly introduce new
> features as Python WSGI programs.
>
> Is it possible to write a Python WSGI program that talks to a PHP
> program as its "back end"? Where can I find out how to do this,
> preferably with examples?
>
> The ideal here is to keep all the existing code as is, but write
> little or no new PHP code. Instead, iteratively change the interface,
> replacing pieces of the monolithic legacy PHP interface with modular
> WSGI programs over time.

Look at mod_python for Apache. If you use it correctly you can on a
page by page basis as need be, replace the existing PHP pages with
equivalents written using Python. You could do this by programming
right at the level of mod_python, or again, if done right by using WSGI
on top of mod_python. If you need to maintain compatibility of URLs,
you could even do things so that even though URLs use .php, that it
maps to Python code underneath, thus easing any transition.

If you are interested in this path, the mod_python mailing list may be
a good place to go to discuss the mod_python aspects of this. The
mod_python mailing list details are on the mod_python web site at
www.modpython.org.


Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: WSGI with mod_python (was: Python, WSGI, legacy web application)

[Graham Dumpleton]

Paul Boddie wrote:
> Rob De Almeida wrote:
> > Ben Finney wrote:
> > > I was under the impression that WSGI in mod_python was a rather kludgy
> > > way to do WSGI, but I don't know what the alternatives are. CGI?
> > > Python http server (e.g. CherryPy)? Something else?
> >
> > You can use FastCGI or SCGI too, with Apache, lighttpd or Cherokee.
>
> I think the motivation behind suggesting an Apache solution was that
> you'd be able to migrate the PHP resources you already have running in
> Apache (I assume, since PHP can run in other places these days) to
> mod_python whilst staying within the Apache environment, rather than
> having to maintain a number of different environments at the same time.
> In other words, you'd write your replacement resources using WSGI (or
> whatever) on mod_python (for performance), CGI (for relative
> simplicity), or some other connection technology, and then it'd just be
> a matter of changing the various directives and having Apache figure it
> out.

Correct.

As example, imagine you have written a mod_python handler which itself
interprets how to map a URL to something to implement the URL. This
might map to a WSGI application or to some system of basic mod_python
handlers.

Within the .htaccess file of the directory where all your PHP files
live you could then write:

  PythonHandler myphppagereplacementhandler | .php

At this point nothing will happen, but then one could do the following:

  
  SetHandler mod_python
  

For the one page called 'index.php' the mod_python handler would be
called instead of PHP. As a Python equivalent for each PHP page is
written, just need to trigger the mod_python handler to be used by
using the Files directive.

One could also have different handlers for each page and use Apache to
make the selection if wanted to:

  
  SetHandler mod_python
  PythonHandler myphppagereplacementshandler::index
  

Now I am sure that some will say it looks messy, but as far as trying
to do a progressive replacement of pages and maintain URLs, it is
probably the quickest way. It should be said that any progressive
migration like this is likely to be a bit messy.

Don't like this, then another way using Apache might be to use
mod_rewrite to remap URLs to new URLs which use Python code. Using
mod_rewrite can be a pain though. Yet another way may be to use
mod_proxy to selectively forward URLs through to a separate back end
web server if you are Apache phobic and want to use a web server
written in pure Python.

Overall, Apache/mod_python has a lot to offer, but from what I have
seen most Python web frameworks simply uses it as a jumping off point
and not much else.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Mod_python vs. application server like CherryPy?

[Graham Dumpleton]

Vincent Delporte wrote:
> On 5 Dec 2006 17:05:06 -0800, "fumanchu" <[EMAIL PROTECTED]> wrote:
> >In a nutshell, mod_python gives you
> >access from Python to the Apache API, whereas CherryPy and friends give
> >you their own API.
>
> I didn't know Apache had an API of its own, or that it was even needed
> when writing a web application in Python. What does it provide in
> addition to Python/mod_python?

Although Apache does have a quite considerable underlying API of its
own, mod_python doesn't actually provide direct access to much of it
from Python code. What mod_python does provide is still adequate for
getting basic stuff done, but Apache could perhaps be better harnessed
if all the Apache API were exposed and available.

Where the power of Apache comes into play is the ability to compose
together the functionality of different Apache modules to build up an
application. That is, you aren't just doing everything in Python code.
That said though, this doesn't mean you have to go off and write code
in another language such as C. This is because the Apache modules are
glued together through the Apache configuration files with some
features also being able to be enabled from Python code running under
mod_python.

In some respects you need to see the whole of Apache as being a
platform for building a web application. Unfortunately, most Python web
application developers don't see that and simply use Apache as a
convenient hopping off point for the main content handling phase of a
request. Even where people do write stuff which makes use of mod_python
as more than just a hopping off point, more often than not they still
work within just mod_python and don't bring in other parts of Apache to
any degree.

For example, consider an extreme case such as WSGI. Through a goal of
WSGI being portability it effectively ignores practically everything
that Apache has to offer. Thus although Apache offers support for
authentication and authorisation, a WSGI user would have to implement
this functionality themselves or use a third party WSGI component that
does it for them. Another example is Apache's support for enabling
compression of content returned to a client. The WSGI approach is again
to duplicate that functionality. Similarly with other Apache features
such as URL rewriting, proxying, caching etc etc.

Although WSGI is an extreme case because of the level it pitches at,
other systems such as CherryPy and Django aren't much different as they
effectively duplicate a lot of stuff that could be achieved using more
basic functionality of Apache as well. Once one starts to make use of
the underlying Apache functionality, you are binding yourself to Apache
though and your stuff isn't portable to other web servers. Also, your
job in part becomes more about integrating stuff to come up with a
solution, rather than just writing pure Python code, something that
many Python coders possibly wouldn't find appealing. :-)

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Mod_python vs. application server like CherryPy?

[Graham Dumpleton]

Vincent Delporte wrote:
> On 6 Dec 2006 14:55:58 -0800, "Graham Dumpleton"
> <[EMAIL PROTECTED]> wrote:
> >Although WSGI is an extreme case because of the level it pitches at,
> >other systems such as CherryPy and Django aren't much different as they
> >effectively duplicate a lot of stuff that could be achieved using more
> >basic functionality of Apache as well.
>
> Mmm... So how can I use those goodies from Apache? Just through their
> configuration files, or do I have to somehow call them from Python?
>
> Is the fact that Python developers tend to ignore resources in Apach
> due to difficulties in making calls from Python, making the scripts
> unpythonic?

It depends on what you are doing. For example, if you need to do URL
rewriting, you use the mod_rewrite module for Apache, not that it is
the most pleasant thing to use. If you need to proxy through some
subset of requests to a downstream web server, use mod_proxy. If you
need to compress content going back to clients, use mod_deflate. If you
need to do caching you use mod_cache.

How to configure each of these from the Apache configuration files, you
need to look at the Apache httpd documentation at httpd.apache.org.

Some, like mod_proxy and mod_deflate can be triggered from within
mod_python although finding the magic required isn't always straight
forward. How you setup responses can also control mod_cache.

If anything, the reason that Python developers tend to ignore a lot of
what Apache has to offer is that it means understanding Apache. The
Apache documentation isn't always the easiest thing to understand and
for some things it even requires looking at the Apache source code to
work out how to do things. The mod_python documentation at the moment
doesn't help either, as it doesn't provide much in the way of recipes
for doing things. The new mod_python wiki will hopefully address that
over time, but right now the mod_python mailing lists are the best it
gets.

In terms of whether it is 'unpythonic', what should be more important
is whether it gets the job done in a way that makes best use of what is
available. If you want something to be 'pythonic', then using Apache as
a framework probably isn't what you want, as as I said previously it
becomes more about integrating things rather than writing pure Python
code.

Getting perhaps back to the answer you were seeking right back at the
start, that is if you are new to web application and development and
Python, then you may well be better of just using a higher level
framework as they will make it easier and isolate you from any pains in
understanding Apache and how to use it properly.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: apache & mod_python

[Graham Dumpleton]

Maxim Sloyko wrote:
> m.banaouas wrote:
>
> > Can i install and use  "Apache 2.2.3" & "mod_python 3.2.10" (most recent
> > versions) without facing any known major issue ?

Only that to use Apache 2.2 you must have mod_python 3.2.10 or later,
older versions of mod_python do not work with the more recent version
of Apache.

> Works fine for me.
> The only "known major issue" you can face is general non-threadsafety
> of Python interpreter. So, if you are using Apache MPM, you have to
> allow for it, or use framework that does it for you.

You answer here is a bit misleading. There are no issues with the
thread safety of the Python interpreter per-se. However, as with any
Python application, whether mod_python is used or not, if multiple
threads are being used your application code may have to be written so
as to cope with access to code/data from multiple threads at the same
time. This is not anything unique to mod_python.

Whether this will be an issue or not is dependent on which Apache MPM
is used, not that the Apache MPM is used as one will always be used.
The specific Apache MPMs where multithreading has to be taken into
consideration are "winnt" on Windows platforms and "worker" on UNIX
platforms. What this all means is that when these MPMs are used there
can be concurrent request handlers executing in distinct threads within
the same Apache processes. Thus, where common data is accessed, it has
to be adequately protected.

For further details on the Apache/mod_python process/interpreter/thread
model, see:


http://www.dscpl.com.au/wiki/ModPython/Articles/TheProcessInterpreterModel

That said, there are sometimes threading issues with mod_python but it
is more to do with the fact that mod_python can use multiple
interpreter instances within a single process. Also, the problems are
not a problem in mod_python, but result from third party C modules for
Python being used which take the simple path of using the simplified
thread API for working with the GIL. Such modules often will not work
properly when used from secondary Python interpreter instances. In
these cases it is a simple matter of telling mod_python to handle
requests which are dependent on those modules within the main Python
interpreter, ie., the very first one which gets created. Where third
party C modules use the full thread API for Python properly, it doesn't
matter which interpreter instance they are used from.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Mod_python vs. application server like CherryPy?

[Graham Dumpleton]

Damjan wrote:
> > For example, consider an extreme case such as WSGI. Through a goal of
> > WSGI being portability it effectively ignores practically everything
> > that Apache has to offer. Thus although Apache offers support for
> > authentication and authorisation, a WSGI user would have to implement
> > this functionality themselves or use a third party WSGI component that
> > does it for them.
>
> OTOH
> WSGI auth middleware already supports more auth methods than apache2 itself.

A distinction needs to be made here. HTTP supports Basic and Digest
authentication mechanisms, both of which are in Apache by default. The
authentication mechanism though needs to be seen as distinct from the
auth provider, that is, who actually validates that a user is valid.
Apache 2.2 separates these two things with there being a pluggable auth
provider facility with backend implementations for such things as
passwd like files, dbm database, ldap and using mod_dbd various SQL
databases. Because it is pluggable it can be extended to support any
sort of auth provider implementation you want. It is even technically
possibly to write an auth provider which makes used of Python to
perform the check using some custom system, although mod_python itself
doesn't provide this yet, so you need to roll your own auth module to
do it.

Even as far as authentication mechanisms go, you aren't limited to just
those as the fact that you can provide a complete authentication and/or
authorisation handler means you can implement other as well. You might
for example build on top of SSL and use client certificates to control
access, or you could use HTTP forms based login and sessions. These
custom mechanisms could also use the auth provider plugin system so
that they aren't dependent on one particular user validation mechanism.

Now, when you say that WSGI already supports more auth methods than
Apache 2 itself, are you referring to how the login/authentication is
handled over HTTP, or how client validation is handled, ie., auth
providers.

I am not being critical here, asking more to build my knowledge of WSGI
and what capabilities it does have.

> > Similarly with other Apache features
> > such as URL rewriting, proxying, caching etc etc.
>
> Well, not everybody can use Apache ... and again there's already WSGI
> middleware that's more flexible than the Apache modules for most of the
> features you mention.
>
> It's not that I think mod_python doesn't have uses.. I just think it's not
> practical to make python web applications targeted solely to mod_python.

For what you are doing that may not be practical, but in a lot of other
places it would be a more than reasonable way of going about it. To
clarify though, I am not talking about building something which is
completed implemented within the context of mod_python alone and not
even something that is written in Python exclusively. What I have been
talking about is using Apache as a whole as the platform.

Thus, taking authentication as an example, for basic forms of
authentication it is best to use the standard mod_auth and auth
provider facilities of Apache to do it. For more complicated mechanisms
such as using HTTP form, more customisation is usually required and
this might be implemented using Python under mod_python but could just
as well be implemented in mod_perl. A main thing to note though is that
if Apache authentication handlers are written properly, it can be used
to span not just mod_python but apply to static files served by Apache,
as well as handlers which serve up content using other languages
modules such as PHP, mod_perl.

This is where I said it can be more about integration rather than
writing pure Python code, because if you use Apache and mod_python
properly, you don't end having to write code within a monoculture
consisting of one implementation language as you are if you use WSGI.
Instead, you can pick and choose the modules and languages which make
the most sense as far as allowing you to implement something in the
easiest way possible given what you have and the environment you are
operating in

All I can say is that since this is a Python language list, that many
here will be here because they want to program specifically in Python
and nothing else. Others though may well see the bigger picture and the
realities of working with big systems, especially in a corporate
environment, where one doesn't have a choice but to deal with code
developed over time and in different languages with a result that most
of the time you are writing more glue and than actual application code.
Thus, in an ideal world you might be able to write in Python and
nothing else, but it isn't always an ideal world.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mod_python.so is garbled mod_python.so is garbled

[Graham Dumpleton]

blbmdsmith wrote:
> Has anyone seen the following error while starting httpd:
>
> Starting httpd: httpd: Syntax error on line 54 of
> /usr/local/apache2/conf/httpd.conf: API module structure
> `python_module' in file /usr/local/apache/modules/mod_python.so is
> garbled - perhaps this is not an Apache module DSO
>
> I am running python2.5 with apache server 2.2.3, using
> mod_python-3.2.10
> I ran mod_python configure with-apxs= /usr/local/apache/bin/apxs
> --with-python=/usr/bin/python
>
> I have configured httpd.conf with the follwing lines:
>
> LoadModule python_module /usr/local/apache/modules/mod_python.so
>
> 
> AllowOverride FileInfo
> AddHandler mod_python .py
> PythonHandler mptest
> PythonDebug On
> 
>
> Thanks,
> Bill
>
> I have posted the same message on the mod_python group.  Is this a
> better group to post his message?

The "mod_python" Google group doesn't get used. You should be
subscribing to and posting on the mod_python mailing list for best
chances of a response. Mailing list details are on the mod_python web
site.

Should you have perhaps used:

  --with-apxs=/usr/local/apache2/bin/apxs

Ie., is the version of Apache you compiled for actually the version you
are running it with?

Do you have multiple versions of Apache installed on your system?

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Apache 2.2.3 and mod_python 3.2.10

[Graham Dumpleton]

m.banaouas wrote:
> sorry, I give here the right paths:
>
> I installed Apache 2.2.3 and mod_python 3.2.10 on WinXP plateform
> I configured mod_python via httpd.conf:
> LoadModule python_module modules/mod_python.so
>
> but my script folder configuration doesn't work correctely:
>
> Alias /myfolder D:/myfolder
>
> 
>  Order allow,deny
>  Allow from all
>  AddHandler mod_python .py
>  PythonHandler mod_python.publisher
>  PythonDebug On
> 
>
> for test, this is a sample script d:\myfolder\test.py
> # test.py
> #
> from mod_python import apache
> #
> def hello(name=None):
>  if name:
>  return 'Hello, %s!' % name.capitalize()
>  else:
>  return 'Hello there!'
> #
> def handler(req):
>req.content_type = 'text/plain'
>req.write("from handler test, Hello World!")
>return apache.OK
>
> when I access to the url  http://localhost/myfolder/test.py,
> I obtain source test.py listing but not the rendering of handler or
> hello method.
>
> with url http://localhost/myfolder/test.py/hello, I obtain :
> =>The requested URL /myfolder/test.py/hello was not found on this server.
>
> It seems like something is missing ... but what ?
>
> thanks for any help

Get it working for a normal handler first and don't use
mod_python.publisher.

For a set of instructions on how to get a simple handler working,
including descriptions of common problems and how to debug it, see:


http://www.dscpl.com.au/wiki/ModPython/Articles/GettingModPythonWorking

Once you have confirmed that your installation works, then consider
looking at mod_python.publisher.

I'd also suggest that you get yourself on the mod_python user mailing
list and seek help there as the mod_python community there is much
larger. Mailing list details are on the mod_python web site.

BTW, don't you mean:

  

if that is what Alias uses?

Also, don't mix normal handlers and mod_python.publisher functions in
the same module, and don't use files called 'test.py' as there is a
standard Python module by the same name and if things aren't set up
right you can be picking up the wrong module.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Page layouts in mod_python?

[Graham Dumpleton]

Bruno Desthuilliers wrote:
> Michael a écrit :
> > Hey everyone,
> >
> > Is it possible to automatically insert headers/footers using
> > mod_python?
> > I will be not be using PSP's, so I cannot use the PSP/include solution.
> > Furthermore, the header will be dynamic; it won't be a static HTML
> > page.
> >
> > In short, I've been looking for a page layout facility using
> > mod_python.
> > (Similar to the layout capability in Ruby on Rails.)
>
> mod_python is mainly a librairy exposing the apache API to Python - not
> a full-stack db-to-web framework. What you want here is a HTML
> templating system. There are quite a lot available in Python - look for
> SimpleTAL, Genshi, Myghty, Jinja, Cheetah, etc...

Contrary to the above advise, you could actually use mod_python to do
what you want and you do not need a full stack to do it. Whether you
would want to use just mod_python is another question and the answer
would depend on a bit on what form the HTML files you want to modify
are in or what is generating them.

The first option available is to write an Apache output filter using
mod_python which takes the content of the page and searches for the
 and  tags and inserts at those points the appropriate
header/footer. Using this scheme, because you are looking for tags
already within the HTML page, you do not need to modify the original
HTML source files. Because it is an output filter, the technique will
work on both static files and on files which have been generated
dynamically by PHP, mod_perl, mod_python or any other Apache response
handler capable of generating dynamic pages.

For this first technique, you do not even have to use mod_python if you
didn't want to, as there are other Apache modules which have been
designed to specifically use this technique to insert headers and
footers in pages. These other modules are written in C and so are
probably going to be quicker as well as more flexible without you
having to do too much hard work.

The second option, although not strictly mod_python specific, is to
enable server side include processing for HTML files. For this to work
you would need to modify the original HTML source files or what is
generating them to add in appropriate SSI directives to include the
header/footer. Again, because SSI is performed as an output filter,
source files can be static or dynamically generated.

If using basic SSI, the include of the header/footer to get dynamic
information would be done as a virtual include. That is, it generates
an Apache subrequest to a handler which generates the header/footer.
This handler could be implemented as a CGI script, mod_python handler,
mod_perl handler or even PHP.

Alternatively, in mod_python 3.3 (soon to be released), one can use the
new ability to use Python code in conjunction with SSI. For dynamic
header/footer snippets, this would involve adding callouts to pages to
Python code setup using mod_python.

For an example of header/footer generation using Python code, see:


http://www.dscpl.com.au/wiki/ModPython/Articles/BasicsOfServerSideIncludes

If you particularly want to use mod_python, for further discussion on
this topic and what may be best for what you want to do, I would
suggest you might take the conversation over to the mod_python user
mailing list. Details for the list are on the mod_python web site.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ANNOUNCE: Mod_python 3.3.0b (Beta)

[Graham Dumpleton]

derekl00 wrote:
> Gregory (Grisha) Trubetskoy wrote:
> > The Apache Software Foundation and The Apache HTTP Server Project are
> > pleased to announce the 3.3.0b (Beta) release of mod_python.
>
> How long does it usually take for these things to make there way into
> the Fedora (or other distro) repositories?

Given that this is a beta, I would hope that the distro people don't
get on to it quickly or at all. It was a right pain the last time a
beta of mod_python was released and RPMs got distributed, as it took us
a lot longer to get rid of the beta version out of the pipeline when
the non beta was actually released.

Thus, if you really want to try out this beta, compile it from source
code, otherwise wait for the official non beta release. If you don't
like compiling from source, then you possibly aren't the sort of person
who should be using this beta version in the first place, especially
when compilation on different platforms is always a part of what we
want tested in releasing a beta.
 

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Mod_python

[Graham Dumpleton]

Maxim Sloyko wrote:
> Lad wrote:
> > In my web application I use Apache and mod_python.
> > I allow users to upload huge files( via HTTP FORM , using POST method)
> > I would like to store the file directly on a hard disk and not to
> > upload the WHOLE huge file into  server's memory first.
> > Can anyone suggest a solution?
>
> The only solution you need is Apache and mod_python :)
> I mean,  Apache won't load a huge POST request into its memory no
> matter what. All file uploads will be stored in temporary files. Under
> mod_python (provided you use FieldStorage) you'll need to deal only
> with 'file' objects.

Note though that the default behaviour of FieldStorage is to store
files in whatever directory the 'tempfile' module uses. This may not
always be acceptable, especially with very large files, as the user may
want to have the files actually reside elsewhere in some special
directory and having to make a copy of the file, if a different file
system, may in itself cause issues.

Thus, provided you are using a recent version of mod_python, you still
may have to customise how files are handled using the FieldStorage file
callbacks to have the file stored in the desired location from the
outside.

The other alternative to FieldStorage is to write a custom
Apache/mod_python input filter to process uploads. An example of this
is Tramline (http://www.infrae.com/products/tramline).

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: per interpreter storage for C extensions

[Graham Dumpleton]

Chris Mellon wrote:
> I'm not that familiar with
> mod_python but I'm surely each python interpreter is in a different
> thread (if not process) than the others.

No.

In mod_python there can be multiple distinct interpreter instances in
each Apache child process. If using a multithreaded Apache MPM, there
can be multiple active threads in each Apache child process and more
than one thread can be executing within any one particular interpreter
instance within a process at the same time.

For more details about the Apache/mod_python process/interpreter models
read:


http://www.dscpl.com.au/wiki/ModPython/Articles/TheProcessInterpreterModel

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about the "new" module

[Graham Dumpleton]

[EMAIL PROTECTED] wrote:
> Gabriele> I'm using Python 2.5 to develop a simple MVC framework based
> Gabriele> on mod_python. To load my controllers, I create new modules
> Gabriele> using the "new" module like this:
>
> Gabriele> # 
> Gabriele> my_module = new.module("random_name")
> Gabriele> my_module.__file__ = module_path
> Gabriele> exec open(module_path, "r") in my_module.__dict__
>
> Gabriele> then I initialize the class defined inside the module and call
> Gabriele> a method of this class based on the HTTP request.
>
> Why use the new module?  Why not call __import__() or execfile()?  Details
> on their use are here:
>
> http://docs.python.org/dev/lib/built-in-funcs.html

Or why not use mod_python.apache.import_module() from mod_python
itself. It is designed as a way of users importing specific modules,
including the capability for modules to be reloaded if they have been
changed. See:


http://www.dscpl.com.au/wiki/ModPython/Articles/BasicsOfModuleImporting

as well as the mod_python documentation.

Is recommended though to use mod_python 3.3 as soon as you can though
if you want the module reloading to always work. You can find
descriptions of problems with the module reloading in older versions of
mod_python at:


http://www.dscpl.com.au/wiki/ModPython/Articles/ModuleImportingIsBroken

Note that module importer in mod_python 3.3 has been enhanced some what
over prior versions in addition to fixing problems. An important new
feature is being able to load modules based on their path with modules
with same names in different directories being properly distinguished,
something which is quite useful where modules are used in web
applications to represent pages. You may want to look at documentation
for it at:


http://www.modpython.org/live/mod_python-3.3.0b/doc-html/pyapi-apmeth.html

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiple python interpreters within the same process

[Graham Dumpleton]

On Jun 10, 9:07 am, Josiah Carlson <[EMAIL PROTECTED]>
wrote:
> André wrote:
> > On Jun 9, 5:00 pm, "Marcin Kalicinski" <[EMAIL PROTECTED]> wrote:
> >> How do I use multiple Python interpreters within the same process?
>
> >> I know there's a function Py_NewInterpreter. However, how do I use 
> >> functions
> >> like Py_RunString etc. with it? They don't take any arguments that would
> >> tell on which interpreter to run the string...?
>
> >> Marcin
>
> > You may want to look at the code 
> > modulehttp://docs.python.org/lib/module-code.html
>
> That's completely unrelated.
>
> To answer Marcin's question, from what I understand, running multiple
> Python interpreters is not supported.  There are various issues with
> object sharing and refcounts, etc., and are unlikely to be fixed soon if
> ever.

I really don't know why people keep propagating this myth that you
can't run multiple Python interpreters. :-(

The Python C API has supported multiple Python interpreter instances
within a single process for a long time. If you write your C program
correctly there isn't a problem. The only real limitation is that
different Python interpreter instances can't use different versions of
a C extension module as they when loaded are shared across all Python
interpreter instances.

C extension modules can also cause other problems as well, but this
isn't the fault of Python but of the module writers. Specifically, if
a C extension module is written to only use the simplified API for GIL
locking it can only be used with the first Python interpreter instance
created. If they use the wider GIL locking API properly then there is
no problem with using it in other Python interpreter instances.
Another issue although one which you aren't likely to come across
unless you are doing really tricky stuff, is where a C extension
module was written so as to assume that once it is loaded into a
Python interpreter instance, that that interpreter will not be
destroyed. From what I have seen Pyrex generated C extension modules
possibly have this latter problem and will cause a process to crash if
a Python interpreter instance is destroyed and then a new one created
in its place.

As proof that all this stuff can work, have a look at mod_python and
mod_wsgi. Both make use of multiple Python interpreter instances in a
single process. The mod_wsgi documentation even points out the above
problems with C extension modules in latter sections of:

  http://code.google.com/p/modwsgi/wiki/ApplicationIssues

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How does mod_python know where's Python installed ?

[Graham Dumpleton]

On Jun 13, 7:03 am, arorap <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I recently setupmod_pythonsuccessfully and things work smooth.
> However, I do not remember tellingmod_pythonwhere to find Python
> installation. There's not environment variable which gives that
> information. As such how doesmod_pythonknow where to find Python ?
> When I print sys.path in my scripts it contains my python installation
> folder. I'm wondering how does Apache/mod_pythonget this
> information ?
>
> I'm running Apache 2.2 on windows.

Rather than duplicate here, for a long explanation see the comments in
the file:

  Modules/getpath.c

in the Python source code.

In short though, it searches for 'python' executable on PATH and from
that tries to determine where the corresponding 'lib' directory is for
the version of Python that mod_python is linked against.

Because it looks in PATH, this will not always work when people have
multiple versions of Python installed on their system with different
base directories. See:

  https://issues.apache.org/jira/browse/MODPYTHON-225

for further details in respect of that problem.

Graham


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: cgi.FieldStorage() not working on Windows

[Graham Dumpleton]

On Jun 13, 1:17 am, arorap <[EMAIL PROTECTED]> wrote:
> I've mod_php installed with Apache 2.2. In one of my folders, I'm
> using the cgihandler as the PythonHandler as my target host runs
> python only as CGI. Here cgi.FieldStorage() doesn't seem to work. I
> can see the form data in sys.stdin but cgi.FieldStorage() returns an
> empty dictionary. Here's the code for the test script I am posting to
> -
>
> --
> #!/usr/bin/python
>
> import os
> import cgi
> import sys
>
> print "Content Type: text/plain\n\n"
> print "Hello CGI World !\n"
>
> for key in os.environ:
>   print key + "= " + os.environ[key]
>
> print cgi.FieldStorage()
>
> print sys.stdin.read()
> --
>
> And here's the output I see ..
>
> --
> Hello CGI World !
>
> HTTP_REFERER=http://learnpython/form.htm
> SERVER_SOFTWARE= Apache/2.2.4 (Win32)mod_python/3.3.1 Python/2.5.1
> SCRIPT_NAME= /mptest.py
> SERVER_SIGNATURE=
> REQUEST_METHOD= POST
> SERVER_PROTOCOL= HTTP/1.1
> QUERY_STRING= abc=ayz
> PATH= C:\Program Files\Internet Explorer;;C:\WINDOWS\system32;C:
> \WINDOWS;C:\WINDOWS\System32\Wbem;q:\bin;m:\cm\clearcase\bin;M:\PERL\NT
> \EXEC\BIN;m:\cm\clearcase\bin\nt;M:\Perl\NT\EXEC\BIN;m:\perl\nt\exec
> \bin;m:\cm\clearcase\utils;q:\bin;m:\opus;m:\tvcs;C:\highc331\bin;C:
> \Program Files\Rational\ClearCase\bin;C:\Program Files\Rational\common
> CONTENT_LENGTH= 86
> HTTP_USER_AGENT= Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
> SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
> HTTP_CONNECTION= Keep-Alive
> SERVER_NAME= learnpython
> REMOTE_ADDR= 127.0.0.1
> PATHEXT= .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
> SERVER_PORT= 80
> SERVER_ADDR= 127.0.0.1
> DOCUMENT_ROOT= D:/Projects/LearnPython/www
> COMSPEC= C:\WINDOWS\system32\cmd.exe
> SCRIPT_FILENAME= D:/Projects/LearnPython/www/mptest.py
> SERVER_ADMIN= [EMAIL PROTECTED]
> HTTP_HOST= learnpython
> SystemRoot= C:\WINDOWS
> HTTP_CACHE_CONTROL= no-cache
> REQUEST_URI= /mptest.py?abc=ayz
> HTTP_ACCEPT= */*
> WINDIR= C:\WINDOWS
> GATEWAY_INTERFACE= Python-CGI/1.1
> REMOTE_PORT= 1081
> HTTP_ACCEPT_LANGUAGE= en-us
> CONTENT_TYPE= application/x-www-form-urlencoded
> HTTP_ACCEPT_ENCODING= gzip, deflate
>
> FieldStorage(None, None, [])
>
> firstName=puneet&address=hawaii
> --
>
> I am posting to this script using a form with two text fields named
> firstName and address.
>
> any clue where am I going wrong ?

You don't need mod_python/cgihandler to run CGI scripts. Rather than
bring mod_python into the picture and confuse things, set up Apache to
run your script as a traditional CGI script instead.

BTW, the fact that mod_python is loaded means that CGI scripts aren't
the only way of using Python available to you as you seem to think.
So, suggest you do some research as to what the differences are
between CGI and mod_python.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: cgi.FieldStorage() not working on Windows

[Graham Dumpleton]

On Jun 13, 12:58 pm, arorap <[EMAIL PROTECTED]> wrote:
> Thanks for your reply.
>
> The reason I want to run it as CGI (even though mod_php is available
> on my local computer

Why do you keep mentioning mod_php, surely you mean mod_python.

> is that the target machine to which I will
> finally be uploading my scripts runs CGI.
>
> cgihandler should work just like CGI.

I wouldn't rely on it being exactly the same. The way it works uses a
number of kludges. Also, the mod_python.cgihandler code in mod_python
doesn't really get much attention from mod_python developers anymore
and not sure if it was even specifically retested when mod_python 3.3
was released.

> Any clue why the
> cgi.FieldStorage()might not be working ?

Have no idea why it doesn't work as works as written on MacOS X even
when mod_python.cgihandler is used.

You'll have to get someone else who has Windows to try it. You might
be better off going to the mod_python mailing list to get help, or
just use plain old CGI instead since using mod_python isn't really
going to gain you much anyway.

Graham

> On Jun 12, 7:59 pm, Graham Dumpleton <[EMAIL PROTECTED]>
> wrote:
>
> > On Jun 13, 1:17 am,arorap<[EMAIL PROTECTED]> wrote:
>
> > > I've mod_php installed with Apache 2.2. In one of my folders, I'm
> > > using the cgihandler as the PythonHandler as my target host runs
> > > python only as CGI. Here cgi.FieldStorage() doesn't seem to work. I
> > > can see the form data in sys.stdin but cgi.FieldStorage() returns an
> > > empty dictionary. Here's the code for the test script I am posting to
> > > -
>
> > > --
> > > #!/usr/bin/python
>
> > > import os
> > > import cgi
> > > import sys
>
> > > print "Content Type: text/plain\n\n"
> > > print "Hello CGI World !\n"
>
> > > for key in os.environ:
> > >   print key + "= " + os.environ[key]
>
> > > print cgi.FieldStorage()
>
> > > print sys.stdin.read()
> > > --
>
> > > And here's the output I see ..
>
> > > --
> > > Hello CGI World !
>
> > > HTTP_REFERER=http://learnpython/form.htm
> > > SERVER_SOFTWARE= Apache/2.2.4 (Win32)mod_python/3.3.1 Python/2.5.1
> > > SCRIPT_NAME= /mptest.py
> > > SERVER_SIGNATURE=
> > > REQUEST_METHOD= POST
> > > SERVER_PROTOCOL= HTTP/1.1
> > > QUERY_STRING= abc=ayz
> > > PATH= C:\Program Files\Internet Explorer;;C:\WINDOWS\system32;C:
> > > \WINDOWS;C:\WINDOWS\System32\Wbem;q:\bin;m:\cm\clearcase\bin;M:\PERL\NT
> > > \EXEC\BIN;m:\cm\clearcase\bin\nt;M:\Perl\NT\EXEC\BIN;m:\perl\nt\exec
> > > \bin;m:\cm\clearcase\utils;q:\bin;m:\opus;m:\tvcs;C:\highc331\bin;C:
> > > \Program Files\Rational\ClearCase\bin;C:\Program Files\Rational\common
> > > CONTENT_LENGTH= 86
> > > HTTP_USER_AGENT= Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
> > > SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
> > > HTTP_CONNECTION= Keep-Alive
> > > SERVER_NAME= learnpython
> > > REMOTE_ADDR= 127.0.0.1
> > > PATHEXT= .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
> > > SERVER_PORT= 80
> > > SERVER_ADDR= 127.0.0.1
> > > DOCUMENT_ROOT= D:/Projects/LearnPython/www
> > > COMSPEC= C:\WINDOWS\system32\cmd.exe
> > > SCRIPT_FILENAME= D:/Projects/LearnPython/www/mptest.py
> > > SERVER_ADMIN= [EMAIL PROTECTED]
> > > HTTP_HOST= learnpython
> > > SystemRoot= C:\WINDOWS
> > > HTTP_CACHE_CONTROL= no-cache
> > > REQUEST_URI= /mptest.py?abc=ayz
> > > HTTP_ACCEPT= */*
> > > WINDIR= C:\WINDOWS
> > > GATEWAY_INTERFACE= Python-CGI/1.1
> > > REMOTE_PORT= 1081
> > > HTTP_ACCEPT_LANGUAGE= en-us
> > > CONTENT_TYPE= application/x-www-form-urlencoded
> > > HTTP_ACCEPT_ENCODING= gzip, deflate
>
> > > FieldStorage(None, None, [])
>
> > > firstName=puneet&address=hawaii
> > > --
>
> > > I am posting to this script using a form with two text fields named
> > > firstName and address.
>
> > > any clue where am I going wrong ?
>
> > You don't need mod_python/cgihandler to run CGI scripts. Rather than
> > bring mod_python into the picture and confuse things, set up Apache to
> > run your script as a traditional CGI script instead.
>
> > BTW, the fact that mod_python is loaded means that CGI scripts aren't
> > the only way of using Python available to you as you seem to think.
> > So, suggest you do some research as to what the differences are
> > between CGI and mod_python.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Text-To-Speech for the Mac (OS X)

[Graham Dumpleton]

On Jun 21, 9:41 am, Brian <[EMAIL PROTECTED]> wrote:
> Hello,
>
> Does anyone know how to get Python to be able to perform text-to-speech
> abilities for the Mac (OS X)?  I have been searching Google, but have
> not found any helpful solutions or resources yet.

os.system('say read the man page for "say"')

:-)

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Using PSE under Win32

[Graham Dumpleton]

On Jun 24, 1:13 am, Eduardo Dobay <[EMAIL PROTECTED]> wrote:
> Hello, I've been playing around withmod_pythonthese days (using
> Publisher and PSP), and it has been working smoothly under Windows XP
> (using Apache 2.2). But when I installed PSE and went to use it 
> withmod_python, it didn't work. The error I get whenever I try to load a
> PSE page is:
>
> Traceback (most recent call last):
>
>   File "C:\Python25\lib\site-packages\mod_python\importer.py", line
> 1537, in HandlerDispatch
> default=default_handler, arg=req, silent=hlist.silent)
>
>   File "C:\Python25\lib\site-packages\mod_python\importer.py", line
> 1229, in _process_target
> result = _execute_target(config, req, object, arg)
>
>   File "C:\Python25\lib\site-packages\mod_python\importer.py", line
> 1128, in _execute_target
> result = object(arg)
>
> TypeError: 'module' object is not callable
>
> I thought it could be some incompatibility issue between PSE andmod_python, 
> but I tried both installing the PSE binary and building
> the sources, and it didn't make a difference. Has anyone out there had
> success using PSE under Windows?
>
> (Just for the record, I did install matching versions, at least for
> Apache (2.2.3), Python (2.5) andmod_python(3.3.1). PSE doesn't seem
> to have a strict version requirement.)

What do you have PythonHandler set to and what does it identify? The
error above suggests that whatever you identify as the handler is not
a callable object like a function. Are you perhaps identifying some
sort of PSE template object as target when you shouldn't be? Post your
handler code so we can see it.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: server wide variables

[Graham Dumpleton]

On Jun 26, 1:29 am, Jay Sonmez <[EMAIL PROTECTED]> wrote:
> I want to be able to save some server variables as long as Apache runs
> on the server (mod_python).
>
> How is that possible in Python?

It depends on whether you expect all Apache child processes for that
server to be able to access the data and for the data to be able to
survive the killing off and start up of individual Apache child
processes.

For background on some of the issues read:

  http://www.dscpl.com.au/wiki/ModPython/Articles/TheProcessInterpreterModel

Saving data in os.environ is not recommended even if they have to
survive just within the context of that specific interpreter within
that process, as os.environ access and update is not thread protected
and Apache child processes may be multithreaded.

So, read that referenced document and then perhaps explain you
expectations a bit better.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: server wide variables

[Graham Dumpleton]

On Jun 26, 10:29 am, Graham Dumpleton <[EMAIL PROTECTED]>
wrote:
> On Jun 26, 1:29 am, Jay Sonmez <[EMAIL PROTECTED]> wrote:
>
> > I want to be able to save some server variables as long as Apache runs
> > on the server (mod_python).
>
> > How is that possible in Python?
>
> It depends on whether you expect all Apache child processes for that
> server to be able to access the data and for the data to be able to
> survive the killing off and start up of individual Apache child
> processes.
>
> For background on some of the issues read:
>
>  http://www.dscpl.com.au/wiki/ModPython/Articles/TheProcessInterpreter...
>
> Saving data in os.environ is not recommended even if they have to
> survive just within the context of that specific interpreter within
> that process, as os.environ access and update is not thread protected
> and Apache child processes may be multithreaded.

Before someone decides to shoot me down, yes I know that dictionaries
themselves are thread safe and thus os.environ itself is thread safe.
In making that comment I am looking more at the higher level locking
aspects of a combined set of data, ie., the same as you might have for
global data in a module and what is required to protect that in the
presence of multiple threads. The os.environ dictionary being what it
is, people may not appreciate the need that locking may still be
required if one is going to use it as a means of holding data that can
change over time. In short, os.environ simply shouldn't be used for
that anyway.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Changing default output object for

[Graham Dumpleton]

I am sure someone has probably thought of this and there are good
reasons why it is a bad idea or not possible, but, is there any
technical reason why one couldn't somehow replace PRINT_ITEM byte
code, with combination of byte codes which inserted in a reference to
a file like object and then invoked PRINT_ITEM_TO byte code instead.

In other words, the aim is allow users code to use 'print' but change
where the output goes when no target file is provided.

I do know that one can replace sys.stdout with another file object in
the simple case, or in the more complicated case with a file like
object that determines where to redirect output based on the calling
thread identity or some over criteria, but wanted to see if some one
could do something without modifying sys.stdout.

One particular example where I thought this might be interesting is to
allow in a web templating system where Python code blocks can be
embedded in markup, the use of the 'print' statement without a user
having to know that they really need to direct output to a special
file object, eg. 'print >> response'. Procedure might thus be to load
Python code snippets and compile to byte code. Fixup byte code to
change PRINT_ITEM to PRINT_ITEM_TO with other byte codes added to
insert appropriate reference to file like object being supplied
through data only passed to code object when executed. Cache code for
ongoing use. Invoke code object as required, supplying reference to
file object for output.

In some template systems such as in web.py they replace sys.stdout
with a magic object which based on the identity of the thread making
calls to the methods of sys.stdout, will redirect the output back to a
registered response object, but instead of doing it this way, could
byte code manipulation provide another way.

Anyway, just curious as to whether it would be possible or not and if
someone has an example of doing it using byteplay or similar would
love to see what is required.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mod_python & doc file system layout

[Graham Dumpleton]

On Jul 5, 12:22 am, Jan Danielsson <[EMAIL PROTECTED]> wrote:
> Hello all,
>
>This is probably more of an apache question, but I'm guessing there
> will be other mod_python-beginners who are wondering the same thing.
>
>Let's say I have a web app called MyApp. It uses the usual images and
> style sheets. I keep it all in
> ~/projects/MyApp/{styles,images,index.py,.htaccess}. I configure up
> apache to use virtual hosts, and point it to MyApp.
>
>Now when I access myapp.local (which points to 127.0.0.1, obviously),
> my index.py is parsed properly. However, any access made to /images or
> /styles end up being handled by mod_python, which isn't good, because it
> won't load images or style sheets.
>
>I have solved this by moving index.py (and the associated .htaccess)
> to a subdirectory "main".
>
>I guess one solution would be to make an access to / on the virtual
> server redirect to main.
>
>Another solution, I guess, would be to make the python script load
> the images and style sheets.. A solution which I find to be particularly
> ugly. :-)
>
>Any mod_python users out there with any other solutions?

In the subdirectories containing your static files add a .htaccess
file containing:

  SetHandler None

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Client-side cookies on Python in Mac OSX

[Graham Dumpleton]

On Jul 13, 12:14 pm, Adrian Petrescu <[EMAIL PROTECTED]> wrote:
> Hi, all. I'm writing an app for OS X; therefore I'd prefer to use only
> the default python install that comes with Tiger. For the moment,
> however, this means:
>
> NaviOSX:~ adrianpetrescu$ python -V
> Python 2.3.5
>
> Therefore, I was not surprised to find out that cookielib did not
> exist here, since I knew that it was a 2.4+ feature.
>
> However, I *was* shocked to find out that ClientCookie, which I'd
> thought was a Python 2.0+ feature, also cannot be found:
>
> >>> import ClientCookie
>
> Traceback (most recent call last):
>   File "", line 1, in ?
> ImportError: No module named ClientCookie
>
> Why would Apple go out of their way to remove this functionality from
> their shipped version of Python? More importantly, considering I need
> to programatically access a website that uses cookies for
> authentication, how can I do this in OSX's Python install? Do they
> provide some other library they'd prefer you to use?
>
> I'm sure SOMEONE in the world has used cookies on Macs so I'm hoping
> there is a solution for this...

They didn't remove it, it was never there in the first place.

The ClientCookie site says 'Python 2.0 or above is required'. It
doesn't say it is included with Python distributions. The package
still has to be installed separately. See:

  http://wwwsearch.sourceforge.net/ClientCookie/

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Installing mod_python on mac os 10.4.7

[Graham Dumpleton]

On Jul 15, 2:47 am, 7stud <[EMAIL PROTECTED]> wrote:
> Themod_pythonmanual says this under section 2.1 Prerequisites:
>
> --
> In order to compilemod_pythonyou will need to have the include files
> for both Apache and Python, as well as the Python library installed on
> your system. If you installed Python and Apache from source, then you
> already have everything needed. However, if you are using prepackaged
> software (e.g. Red Hat Linux RPM, Debian, or Solaris packages from
> sunsite, etc) then chances are, you have just the binaries and not the
> sources on your system. Often, the Apache and Python include files and
> libraries necessary to compilemod_pythonare part of separate
> ``development'' package. If you are not sure whether you have all the
> necessary files, either compile and install Python and Apache from
> source, or refer to the documentation for your system on how to get
> the development packages.
> -
>
> I installed Apache from source using these instructions:
>
> http://switch.richard5.net/isp-in-a-box-v2/installing-apache-on-mac-o...
>
> but I used a package to install python 2.4.  The package was from
> here:
>
> http://www.pythonmac.org/packages/
>
> and it's a "Universal binary version of Python that runs natively on
> PPC and Intel systems."
>
> But my mac came with Python 2.3.5 pre-installed, so I wonder if I
> already have the necessary "include files for both Apache and Python,
> as well as the Python library" already installed.

Have you actually tried to install mod_python? That would be the
quickest way of finding out.

Because you are using an alternate Apache than the OS supplied one,
you will need to use the --with-apxs option to configure when building
Python. Unless you really need Python 2.4, it is easier to use the OS
supplied version of Python. If you must use an alternate version, use
the --with-python option to configure for mod_python to tell it which
version. Depending on where that Python version is installed, you may
also have to fiddle the Apache 'envvars' file as well to get it to
work.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Installing mod_python on mac os 10.4.7

[Graham Dumpleton]

On Jul 15, 10:06 am, Graham Dumpleton <[EMAIL PROTECTED]>
wrote:
> On Jul 15, 2:47 am, 7stud <[EMAIL PROTECTED]> wrote:
>
>
>
> > Themod_pythonmanual says this under section 2.1 Prerequisites:
>
> > --
> > In order to compilemod_pythonyou will need to have the include files
> > for both Apache and Python, as well as the Python library installed on
> > your system. If you installed Python and Apache from source, then you
> > already have everything needed. However, if you are using prepackaged
> > software (e.g. Red Hat Linux RPM, Debian, or Solaris packages from
> > sunsite, etc) then chances are, you have just the binaries and not the
> > sources on your system. Often, the Apache and Python include files and
> > libraries necessary to compilemod_pythonare part of separate
> > ``development'' package. If you are not sure whether you have all the
> > necessary files, either compile and install Python and Apache from
> > source, or refer to the documentation for your system on how to get
> > the development packages.
> > -
>
> > I installed Apache from source using these instructions:
>
> >http://switch.richard5.net/isp-in-a-box-v2/installing-apache-on-mac-o...
>
> > but I used a package to install python 2.4.  The package was from
> > here:
>
> >http://www.pythonmac.org/packages/
>
> > and it's a "Universal binary version of Python that runs natively on
> > PPC and Intel systems."
>
> > But my mac came with Python 2.3.5 pre-installed, so I wonder if I
> > already have the necessary "include files for both Apache and Python,
> > as well as the Python library" already installed.
>
> Have you actually tried to install mod_python? That would be the
> quickest way of finding out.
>
> Because you are using an alternate Apache than the OS supplied one,
> you will need to use the --with-apxs option to configure when building
> Python.

Whoops, --with-apxs option is to configure for mod_python, not Python.

> Unless you really need Python 2.4, it is easier to use the OS
> supplied version of Python. If you must use an alternate version, use
> the --with-python option to configure for mod_python to tell it which
> version. Depending on where that Python version is installed, you may
> also have to fiddle the Apache 'envvars' file as well to get it to
> work.
>
> Graham


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Third party script debugging on remote server ...

[Graham Dumpleton]

After calling whatever it is that is writing to standard output/error,
do:

  import sys
  sys.stdout.flush()
  sys.stderr.flush()

This should cause any buffered data to be immediately flushed to the
main Apache error log, ie., don't look in any virtual host logs, check
the main one.

Graham

On Apr 19, 9:15 am, dbee <[EMAIL PROTECTED]> wrote:
> Right. I've got a really, really annoying/difficult/time consuming
> problem with my development environment. I'm using django to build a
> web app with paypal integration. My server is hosted remotely, and it
> is receiving IPN (payment notifications) POST requests from Paypal.
> I've checked on google and irc and this is my last shot at solving
> this before I go mad ... :-(
>
> The problem is that I can't debug those POST requests. Browser
> debugging is out of the question obviously cause I'm not at the
> computer, ( and it doesn't have X ).
>
> I've tried cgitb but that doesn't seem to work at all ...
>
> import cgitb; cgitb.enable(display=0,logdir='/tmp/',format='plain')
> import cgitb; cgitb.enable()
>
> ... neither of those commands have any effect. Although I do log other
> parts of the script to /tmp, so I know that it's reachable...
>
> mod_pythonabsolutely refuses to error_log to the apache error_log. I
> have restarted it and it still won't flush whatever error buffer it
> may ( or may not ) have stored.
>
> I can re-constitute the requests in my browser using a GET request.
> But frankly, that's kinda messy - there are lots of paypal IPN
> combinations and I may have to integrate other applications with
> paypal. So ideally speaking I'm looking for a proper debugging
> environment for this kind of thing ...
>
> Basically, I either wantmod_pythonto start error_logging properly,
> or I want some type of working traceback environment to be available.
> Help !
>
> Server version: Apache/2.0.52
> Server built:   Aug 13 2006 03:29:43
> CentOS4.x: (RedHat Clone)mod_python.i386  3.1.3-5.1
> installed
>
> # httpd.conf
>
> 
>
>  ServerName mydomain.biz
>  ServerAliaswww.mydomain.biz
>  SetHandlermod_python
>  PythonPath "['/home/babo/django'] + sys.path"
>  PythonHandler django.core.handlers.modpython
>  SetEnv DJANGO_SETTINGS_MODULE mydomain.settings
>
>   
>SetHandler None
>Options None
>   
>
> 
>
> My python.conf: ( seems pretty normal )
>
> #
> #Mod_pythonis a module that embeds the Python language interpreter
> # within the server, allowing Apache handlers to be written in Python.
> #
>
> LoadModule python_module modules/mod_python.so
>
> # Override type-map handler for /var/www/manual
> 
> 
> SetHandler default-handler
> 
> 
>
> # This will cause files beneath /var/www/html with the extension .spam
> # to be handled by the Python script /var/www/html/eggs.py
> #
> #
> #AddHandler python-program .spam
> #PythonHandler eggs
> #
>
> # This will cause all requests to the /python heirachy of your
> # webserver to be handled by the python script /path/to/myhandler.py
> #
> #
> #SetHandler python-program
> #PythonPath "sys.path + ['/path/to']"
> #PythonHandler myhandler
> #
>
> # This will cause all requests to the /python heirachy of your
> # webserver to be handled bymod_python'sPublisher handler
> # (seehttp://localhost/manual/mod/mod_python/hand-pub.html)
> #
> # This will cause the output of all requests to files beneath
> # /var/www/html with the extension .flt to be filtered through
> # the Python script /var/www/html/filter.py
> #
> #
> #PythonOutputFilter filter MYFILTER
> #AddOutputFilter MYFILTER .flt
> #


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Third party script debugging on remote server ...

[Graham Dumpleton]

> Hi Graeme,
>
> Thanks for the suggestion. Unluckily for me, it doesn't seem to be
> working though ...
>
> I've tried it a number of different ways. I guess if I put the code
> after an exception, then the code won't be called.
> So I ran an error in the python script and then I called the
> sys.stderr.flush() from the python shell. No luck though.
>
> I called on it's own, still no luck,
>
> Then I tried ...
>
> try:
>
> raise Exception('Please log this error')
>
> except:
>
> import sys
>
> sys.stderr.flush()
> sys.stdout.flush()
>
> The error log still has no error info in it :-(  
>
> Is this problem unique to me ?

Those calls don't generate output themselves and exception details
don't get output at the point of a try/except block. If you want to
generate to Apache error log details of exception from try/except,
use:

  import traceback
  import sys

  try:
 xxx
  except:
 traceback.print_exc()
 sys.stderr.flush()

Using these flush functions from interactive debugger isn't going to
show you anything as this trick is only needed with mod_python. All
the functions do is force the output of any buffered data that was
written out previously.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python cgi problem with textarea

[Graham Dumpleton]

On Apr 22, 11:09 pm, placid <[EMAIL PROTECTED]> wrote:
> On Apr 22, 4:08 pm, Adrian Smith <[EMAIL PROTECTED]> wrote:
>
>
>
> > This may be more a cgi thing than a Python one, but I'm trying to get
> > this page:
>
> >http://adrian10.phpwebhosting.com/trial.html
>
> > consisting basically of this:
>
> > 
> > 
> > 
> > 
>
> > ...to print out the contents of the textarea with this cgi script:
>
> > #!/usr/bin/python
> > import cgi
> > print "Content-type: text/html\n"
> > form = cgi.FieldStorage()
> > print form["essay"].value
>
> > ...and I get an internal server error if I have any spaces in the
> > textarea, which is really going to limit its usefulness to me. Oddly,
> > it seems to work for a friend in the UK who's looked at it, but it
> > doesn't work for me here in Japan.
>
> i just tried it and its working. here it is
>
> http://yallara.cs.rmit.edu.au/~bevcimen/form.html
>
> maybe the internal server error is because mod_python isn't installed
> assuming your using Apache as your web server

You do not need mod_python installed to be able to run CGI scripts,
thus has nothing to do with mod_python.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: re-importing modules

[Graham Dumpleton]

On May 1, 2:17 pm, Steven D'Aprano <[EMAIL PROTECTED]>
wrote:
> On Tue, 01 May 2007 00:32:20 +, John Nagle wrote:
> > [EMAIL PROTECTED] wrote:
>
> >>>In addition to the warning that reload() does not recursively reload
> >>>modules that the reloaded module depends on, be warned that reloading a
> >>>module does not magically affect any functions or objects from the old
> >>>version that you may be holding on to.
>
> > Maybe reloading modules should be deprecated.  The semantics
> > are awful, and it interferes with higher-performance implementations.
>
> I'd hate for reload to disappear, it is great for interactive
> development/debugging, at least under some circumstances. (If you have
> complex and tangled class hierarchies, it might not be powerful enough.)
>
> As for the semantics being awful, I disagree. reload() does exactly
> what it claims to do, no more, no less. I wouldn't expect reload(module1)
> to reload modules 2 through 5 just because they were imported by module1.
> If I wanted them reloaded, I'd say so.

That it doesn't reload a parent when a child changes may be fine in an
interactive debugger, but can cause problems if not done where
automatic reloading is being done in a long running web application
where you want to avoid server restarts. For that very reason, the new
importer in mod_python 3.3 tracks parent/child relationships between
modules and is able to import a parent module when a child changes.
The new importer does a lot more besides that as well, including
distinguishing modules by full path rather than purely by name. For
some details on the mod_python importer see documentation for
import_module() in:

  http://www.modpython.org/live/current/doc-html/pyapi-apmeth.html

To understand why the new importer does some of the things it does, it
may be helpful to read all the problems the previous importer was
causing:

  http://www.dscpl.com.au/wiki/ModPython/Articles/ModuleImportingIsBroken

Getting module importing with reloading working is frightfully tricky
in an application where multiple threads corresponding to concurrent
web requests are executing as one can't be reloading on top of
modules where code may be executing, one must also avoid triggering a
reload of a specific module half way through a request where the same
module is encountered twice and lots, plus lots of other gotchas.

Graham





-- 
http://mail.python.org/mailman/listinfo/python-list

Re: re-importing modules

[Graham Dumpleton]

On May 1, 3:51 pm, Paul Rubin <http://[EMAIL PROTECTED]> wrote:
> Graham Dumpleton <[EMAIL PROTECTED]> writes:
> > That it doesn't reload a parent when a child changes may be fine in an
> > interactive debugger, but can cause problems if not done where
> > automatic reloading is being done in a long running web application
> > where you want to avoid server restarts.
>
> Oh that sounds horrid.  If you really have to do something like that,
> it's time to consider adding serious hot-patching capability as I
> believe Erlang has done.  Better is to start a new server process and
> transfer the live session data and active connections to it through
> IPC mechanisms.  I've had an open RFE for many years (#814689) about
> ancillary messages on AF_UNIX sockets, which let you pass file
> descriptors (such as those attached to open TCP connections) between
> processes.  It actually came up in conversation with someone about
> something I'm currently working on, so maybe I'll have reason to try
> coding it sometime soon (that's not at all definite though).

It may sound horrible but mod_python's original module importer had
lots of problems because of things not being reloaded when they should
have in a consistent manner. People weren't going to want to give up
the reload feature altogether as restarting Apache all the time is
viewed as a worse solution, so the only practical solution was at
least make it reliable and this is one of the things that was required
to do it. The number of complaints against mod_python module importing
and reloading problems has basically vanished now with mod_python 3.3
and the whole thing is so much more stable now.

Do note that this reloading mechanism isn't applied across all Python
modules, only mod_python handler and associated modules which are used
within the context of URL/document space of your web server. Thus it
is quite specialised and has quite specific use case scenarios which
are well understood. In that context the whole mechanism works fine.
Thus, please try not to pass judgment on it without full understanding
the problem space it operates in and how it internally works. :-)

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: WSGI spec clarification regarding exceptions

[Graham Dumpleton]

On May 10, 8:26 am, Adam Atlas <[EMAIL PROTECTED]> wrote:
> I'm trying to figure out if there's any defined behaviour in PEP 333
> for instances where an application returns an iterable as usual
> without error, but that iterable's next() method eventually raises an
> exception. Since any data theretofore returned by the iterable must be
> assumed to have already been written to the client, thus making it
> impossible to replace the response with a 500 error or somesuch, does
> the gateway just absorb the exception and cut off the response there?
> It seems like that's pretty much all it could do, but I'm wondering if
> that's explicitly specified anywhere (I couldn't find anything about
> that in the PEP).

Because the WSGI specification requires that a WSGI adapter for a web
server always explicitly perform a flush after each string yielded
from the iterator then simply cutting off the response at that point
is all one can do as the first time a flush is done any response
status and headers will also have to be written out.

Now depending on the web server being used, all the client may see is
the truncated page, or it might also see some form of error page
appended to it by the underlying web server. For example, in Apache,
if the WSGI adapter returns HTTP_INTERNAL_SERVER_ERROR back to the
server, the server disregards whether anything has already been sent
and tries to generate a 500 error page. Since the response status and
headers have already been flushed though, the original status is
received by the client, but the Apache error page content is still
sent. Thus you might get output looking like:

Hello World!


200 OK

OK
The server encountered an internal error or
misconfiguration and was unable to complete
your request.
Please contact the server administrator,
 [EMAIL PROTECTED] and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.
More information about this error may be available
in the server error log.

Apache/2.2.2 (Unix) mod_wsgi/1.0-TRUNK Python/2.3.5 Server at
localhost Port 8002


It is actually hard to know what to do here. There are ways one could
stop the appending of the error page content, but is returning just
the truncated page any better. At least the error page content in
there highlights an issue even if status wasn't 500, but then not
being 500, the page content could get cached.

This is where one wanders whether the WSGI way of always flushing is a
good idea. It may be better to always use buffering unless one
specifically knows that one wants to do streaming of data where size
could be large or take some time to produce.

Anyway, for WSGI matters, you are probably better off bringing them up
on the Python WEB-SIG list.

  http://www.python.org/community/sigs/current/web-sig/

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: WSGI spec clarification regarding exceptions

[Graham Dumpleton]

On May 10, 12:07 pm, Graham Dumpleton <[EMAIL PROTECTED]>
wrote:
> On May 10, 8:26 am, Adam Atlas <[EMAIL PROTECTED]> wrote:
>
> > I'm trying to figure out if there's any defined behaviour in PEP 333
> > for instances where an application returns an iterable as usual
> > without error, but that iterable's next() method eventually raises an
> > exception. Since any data theretofore returned by the iterable must be
> > assumed to have already been written to the client, thus making it
> > impossible to replace the response with a 500 error or somesuch, does
> > the gateway just absorb the exception and cut off the response there?
> > It seems like that's pretty much all it could do, but I'm wondering if
> > that's explicitly specified anywhere (I couldn't find anything about
> > that in the PEP).
>
> Because the WSGI specification requires that a WSGI adapter for a web
> server always explicitly perform a flush after each string yielded
> from the iterator then simply cutting off the response at that point
> is all one can do as the first time a flush is done any response
> status and headers will also have to be written out.
>
> Now depending on the web server being used, all the client may see is
> the truncated page, or it might also see some form of error page
> appended to it by the underlying web server. For example, in Apache,
> if the WSGI adapter returns HTTP_INTERNAL_SERVER_ERROR back to the
> server, the server disregards whether anything has already been sent
> and tries to generate a 500 error page. Since the response status and
> headers have already been flushed though, the original status is
> received by the client, but the Apache error page content is still
> sent. Thus you might get output looking like:
>
> Hello World!
> 
> 
> 200 OK
> 
> OK
> The server encountered an internal error or
> misconfiguration and was unable to complete
> your request.
> Please contact the server administrator,
>  [EMAIL PROTECTED] and inform them of the time the error occurred,
> and anything you might have done that may have
> caused the error.
> More information about this error may be available
> in the server error log.
> 
> Apache/2.2.2 (Unix) mod_wsgi/1.0-TRUNK Python/2.3.5 Server at
> localhost Port 8002
> 
>
> It is actually hard to know what to do here. There are ways one could
> stop the appending of the error page content, but is returning just
> the truncated page any better. At least the error page content in
> there highlights an issue even if status wasn't 500, but then not
> being 500, the page content could get cached.
>
> This is where one wanders whether the WSGI way of always flushing is a
> good idea. It may be better to always use buffering unless one
> specifically knows that one wants to do streaming of data where size
> could be large or take some time to produce.
>
> Anyway, for WSGI matters, you are probably better off bringing them up
> on the Python WEB-SIG list.
>
>  http://www.python.org/community/sigs/current/web-sig/

BTW, forgot to mention that one can always create a WSGI middleware
component that does buffering and which only sends the complete
response. If an error occurs while accumulating the response, then you
can return a 500 status and error page of your own making.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is wsgi ready for prime time?

[Graham Dumpleton]

On May 18, 5:31 am, Stefan Sonnenberg-Carstens
<[EMAIL PROTECTED]> wrote:
> IMHO WSGI is _only_ a new way of talking to webservers, like apache.
> It is as low-level as (f)cgi, so don't expect too much support at this
> stage -
> indeed a module like the cgi one in the std lib would be nice.
> As google uses it (mod_wsgi), I would suspect you can use it.

So people don't get the wrong impression, mod_wsgi is merely hosted on
the Google code site. This does not mean that Google uses it, nor does
Google have anything to do with its development.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is wsgi ready for prime time?

[Graham Dumpleton]

On May 18, 5:31 am, Stefan Sonnenberg-Carstens
<[EMAIL PROTECTED]> wrote:
> IMHO WSGI is _only_ a new way of talking to webservers, like apache.
> It is as low-level as (f)cgi, so don't expect too much support at this
> stage -
> indeed a module like the cgi one in the std lib would be nice.
> As google uses it (mod_wsgi), I would suspect you can use it.

So people don't get the wrong impression, mod_wsgi is merely hosted on
the Google code site. This does not mean that Google uses it, nor does
Google have anything to do with its development.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mod_python performs several magnitudes slower than PHP?

[Graham Dumpleton]

On May 20, 10:01 am, John Nagle <[EMAIL PROTECTED]> wrote:
> That's puzzling, because withmod_python, you're only invoking
> the compiler once per Apache restart.  With CGI programs, you pay
> the loading penalty on every request.
>
> John Nagle
>
> [EMAIL PROTECTED] wrote:
> > Recently I've had to move my site to a new dedicated server running
> > FreeBSD 6.1. After installing apache 2.0.59, python 2.4.4 and
> >mod_python3.3.1, I decided to bench a script in PHP vs one in Python.
> > I found out that for some reason, mymod_pythonwas performing
> > extremely slow - magnitudes slower than it should. I scowered the
> > internet for hours and asked a few friends and still haven't been able
> > to find a solution to the problem.
>
> > frommod_pythonimport apache
>
> > def handler(req):
> >   for i in xrange(1000):
> > print >> req, "Yeah"
> >   return apache.OK
>
> > and...
>
> >  >   for ($i = 0; $i < 1000; $i++)
> > echo "Yeah\n" ;
> > ?>
>
> > when I ran ab on both using 1000 requests and a concurrency of 10, i
> > got these results:
>
> > python- Requests per second:21.37 [#/sec] (mean)
> > php- Requests per second:1008.37 [#/sec] (mean)
>
> > Any ideas would really be appreciated... I'm on my last leg.

The original poster also asked the same question on the mod_python
mailing list. As pointed out there, the examples aren't equivalent as
the mod_python example would have resulted in data being flushed to
the client after every call to 'print'.

A more suitable example for comparison would have been:

  def handler(req):
   for i in xrange(1000):
 req.write('Yeah\n, 0)
   return apache.OK

The second argument of 0 to req.write() will allow Apache to buffer
data in an appropriate way and avoid lots of socket writes with small
amounts of data, plus avoid triggering of Apache's filter mechanism
every time.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mod_python performs several magnitudes slower than PHP?

[Graham Dumpleton]

On May 21, 5:51 pm, Winfried Tilanus <[EMAIL PROTECTED]> wrote:
> On 05/20/2007 Graham Dumpleton wrote:
>
> Hi,
>
> > A more suitable example for comparison would have been:
>
> And are there any benchmarks with this new version available? Just
> curious...

Unless someone else posts that specific example comparing it to PHP on
the same system, then you might instead look at:

  http://www.modpython.org/pipermail/mod_python/2007-May/023654.html

This shows the original posters Python example compared to another way
of doing it, not what I posted, which would be even quicker, ie.,
using Python itself to do the buffering.

Do note that such benchmarks are pretty meaningless as you will never
achieve such throughputs once you actually load on top your Django,
TurboGears, Pylons or other mega web framework application. Such
applications are huge and carry a lot of overhead which dwarfs any
overhead of mod_python itself.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: need advice on building core code for python and PHP

[Graham Dumpleton]

On May 25, 5:24 am, aspineux <[EMAIL PROTECTED]> wrote:
> On 24 mai, 19:33, Szabolcs Nagy <[EMAIL PROTECTED]> wrote:
>
> > > Is there a way I could code the base (core) code in Python and have
> > > PHP call it?  I've really liked using SQLAlchemy and there are other
>
> > * quick and dirty solution:
> > in a shell:
> >   $ python yourscript.py pipe_out
> > in the php script:
> >   fwrite(pipe_in, input_data);
> >   results = fread(pipe_out, sizeof_results);
>
> > * simple and nice solution:
> >   do not ever use php
>
> Write a CGI wrapper around your python script, and publish it using 
> mod_python.
> And make the appropriate http requests from PHP.

You do not need mod_python to host CGI scripts written in Python, they
are two separate things.

Depending on the complexity of what you are doing, you might be better
off writing a backend server in Python that incorporates an XML-RPC
server. Your PHP script can then use XML-RPC client to communicate to
the backend Python server to do the real work. Over time you could
even transition your web pages to being done in Python instead. In
doing this your back end Python server doesn't have to change, you
just make XML-RPC calls from the Python code for the web pages in
place of where you would be doing it with PHP initially. You also
wouldn't be restricted to web based front ends, you could also use GUI
based front end as well.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: need advice on building core code for python and PHP

[Graham Dumpleton]

On May 30, 11:24 am, digimotif <[EMAIL PROTECTED]> wrote:
> On May 24, 5:01 pm, Graham Dumpleton <[EMAIL PROTECTED]>
> wrote:
>
>
>
> > On May 25, 5:24 am, aspineux <[EMAIL PROTECTED]> wrote:
>
> > > On 24 mai, 19:33, Szabolcs Nagy <[EMAIL PROTECTED]> wrote:
>
> > > > > Is there a way I could code the base (core) code in Python and have
> > > > > PHP call it?  I've really liked using SQLAlchemy and there are other
>
> > > > * quick and dirty solution:
> > > > in a shell:
> > > >   $ python yourscript.py pipe_out
> > > > in the php script:
> > > >   fwrite(pipe_in, input_data);
> > > >   results = fread(pipe_out, sizeof_results);
>
> > > > * simple and nice solution:
> > > >   do not ever use php
>
> > > Write a CGI wrapper around your python script, and publish it 
> > > usingmod_python.
> > > And make the appropriate http requests from PHP.
>
> > You do not needmod_pythonto host CGI scripts written in Python, they
> > are two separate things.
>
> > Depending on the complexity of what you are doing, you might be better
> > off writing a backend server in Python that incorporates an XML-RPC
> > server. Your PHP script can then use XML-RPC client to communicate to
> > the backend Python server to do the real work. Over time you could
> > even transition your web pages to being done in Python instead. In
> > doing this your back end Python server doesn't have to change, you
> > just make XML-RPC calls from the Python code for the web pages in
> > place of where you would be doing it with PHP initially. You also
> > wouldn't be restricted to web based front ends, you could also use GUI
> > based front end as well.
>
> > Graham
>
> This sounds more like the direction I should go.  Is XML-RPC the only
> technology allowing this sort of setup?  If I understand correctly, it
> would basically mean going to a three tiered application approach.
> I'd have the database, the python xml-rpc server, and the gui/web
> interfaces.  I'd also want to make sure I'm implementing technology
> that will scale well.

There is also Pyro, but by using that you limit yourself to Python
clients as far as I know whereas XML-RPC has clients for lots of
different languages. Other more complicated options are Corba and SOAP
frameworks.

Graham


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Installing mod_python on mac os 10.4.7

[Graham Dumpleton]

On Jul 29, 7:26 am, 7stud <[EMAIL PROTECTED]> wrote:
> On Jul 14, 8:34 pm, Graham Dumpleton <[EMAIL PROTECTED]>
> wrote:
>
>
>
> > On Jul 15, 10:06 am, Graham Dumpleton <[EMAIL PROTECTED]>
> > wrote:
>
> > > On Jul 15, 2:47 am, 7stud <[EMAIL PROTECTED]> wrote:
>
> > > > Themod_pythonmanual says this under section 2.1 Prerequisites:
>
> > > > --
> > > > In order to compilemod_pythonyou will need to have the include files
> > > > for both Apache and Python, as well as the Python library installed on
> > > > your system. If you installed Python and Apache from source, then you
> > > > already have everything needed. However, if you are using prepackaged
> > > > software (e.g. Red Hat Linux RPM, Debian, or Solaris packages from
> > > > sunsite, etc) then chances are, you have just the binaries and not the
> > > > sources on your system. Often, the Apache and Python include files and
> > > > libraries necessary to compilemod_pythonare part of separate
> > > > ``development'' package. If you are not sure whether you have all the
> > > > necessary files, either compile and install Python and Apache from
> > > > source, or refer to the documentation for your system on how to get
> > > > the development packages.
> > > > -
>
> > > > I installed Apache from source using these instructions:
>
> > > >http://switch.richard5.net/isp-in-a-box-v2/installing-apache-on-mac-o...
>
> > > > but I used a package to install python 2.4.  The package was from
> > > > here:
>
> > > >http://www.pythonmac.org/packages/
>
> > > > and it's a "Universal binary version of Python that runs natively on
> > > > PPC and Intel systems."
>
> > > > But my mac came with Python 2.3.5 pre-installed, so I wonder if I
> > > > already have the necessary "include files for both Apache and Python,
> > > > as well as the Python library" already installed.
>
> > > Have you actually tried to installmod_python? That would be the
> > > quickest way of finding out.
>
> > > Because you are using an alternate Apache than the OS supplied one,
> > > you will need to use the --with-apxs option to configure when building
> > > Python.
>
> > Whoops, --with-apxs option is to configure formod_python, not Python.
>
> > > Unless you really need Python 2.4, it is easier to use the OS
> > > supplied version of Python. If you must use an alternate version, use
> > > the --with-python option to configure formod_pythonto tell it which
> > > version. Depending on where that Python version is installed, you may
> > > also have to fiddle the Apache 'envvars' file as well to get it to
> > > work.
>
> > > Graham
>
> Hi,
>
> Thanks for the response.  Sorry, I didn't respond in a timely manner.
>
> I thought I would give your suggestion a shot and just try installing
> mod_python and see what happens, so I installed mod_python 3.3.1 with
> this command:
>
> $ ./configure --with-apxs=/Library/Apache2/bin/apxs
>
> Here is the output:
>
> 
> checking for gcc... gcc
> checking for C compiler default output file name... a.out
> checking whether the C compiler works... yes
> checking whether we are cross compiling... no
> checking for suffix of executables...
> checking for suffix of object files... o
> checking whether we are using the GNU C compiler... yes
> checking whether gcc accepts -g... yes
> checking for gcc option to accept ANSI C... none needed
> checking for ar... ar
> checking for a BSD-compatible install... /usr/bin/install -c
> checking whether make sets $(MAKE)... yes
> checking for main in -lm... yes
> checking for an ANSI C-conforming const... yes
> checking your blood pressure... a bit high, but we can proceed
> configure: checking whether apxs is available...
> checking for --with-apxs... /Library/Apache2/bin/apxs executable, good
> checking Apache version... 2.2.4
> checking for Apache libexec directory... /Library/Apache2/modules
> checking for Apache include directory... -I/Library/Apache2/include
> checking for --with-python... no
> checking for python... /Library/Frameworks/Python.framework/Versions/
> Current/bin/python
> checking Python version... 2.4
> checking Python install prefix... /Library/Frameworks/Python.framework/
> Versions/2.4
> checking checking where python libraries are installed... /Library/
> Frameworks/Python.framework/Versions/2.4/lib/pytho

Re: Installing mod_python on mac os 10.4.7

[Graham Dumpleton]

On Jul 30, 2:37 am, 7stud <[EMAIL PROTECTED]> wrote:
> On Jul 29, 4:07 am, Graham Dumpleton <[EMAIL PROTECTED]>
> wrote:
>
> > Have you got an appropriate LoadModule directive in Apache
> > configuration to load the mod_python module? That you get this error
> > is indicative of mod_python module not being loaded. You can check the
> > Apache error logs to see if mod_python got loaded.
>
> > See:
>
> >  http://www.dscpl.com.au/wiki/ModPython/Articles/GettingModPythonWorking
>
> > for more help on debugging.
>
> That article says the LoadModule directive should be:
>
> >LoadModule python_module libexec/mod_python.so
>
> I saw this note in /Library/Apache2/conf/http.conf
>
> # Configuration and logfile names: If the filenames you specify for
> many
> # of the server's control files begin with "/" (or "drive:/" for
> Win32), the
> # server will use that explicit path.  If the filenames do *not* begin
> # with "/", the value of ServerRoot is prepended -- so "logs/foo.log"
> # with ServerRoot set to "/Library/Apache2" will be interpreted by the
> # server as "/Library/Apache2/logs/foo.log".
>
> I have this server root in /Library/Apache2/conf/httpd.conf:
>
> ServerRoot "/Library/Apache2"
>
> So in the Load Module directive, the actual path to the mod_python.so
> file is:
>
> /Library/Apache2/libexec/mod_python.so
>
> But I checked and I don't have a /Library/Apache2/libexec directory.

Run 'apxs' for the version of Apache you are using to work out what
the actual modules directory should be:

$ /usr/local/apache-2.2.4/bin/apxs -q LIBEXECDIR
/usr/local/apache-2.2.4/modules

If there is no mod_python.so in there then you can't have done the
mod_python install properly. Go back to the mod_python source code and
rerun:

  sudo make install_dso

When you do that it will even tell you exactly what the LoadModule
line you should use is.

Also ensure you actually installed the Python module components of
mod_python. You can redo that by running:

  sudo make install_py_lib

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Destruction of generator objects

[Graham Dumpleton]

On Aug 8, 8:28 am, Stefan Bellon <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I'm generating a binding from Python to C using SWIG. On the C side I
> have iterators over some data structures. On the Python side I
> currently use code like the following:
>
> def get_data(obj):
> result = []
> iter = make_iter(obj)
> while more(iter):
> item = next(iter)
> result.append(item)
> destroy(iter)
> return result
>
> Now I'd like to transform it to a generator function like the following
> in order to make it more memory and time efficient:
>
> def get_data(obj):
> iter = make_iter(obj)
> while more(iter):
> yield next(iter)
> destroy(iter)
>
> But in the generator case, I have a problem if the generator object is
> not iterated till the StopIteration occurs, but if iteration is stopped
> earlier. In that case, the C iterator's destroy is not called, thus the
> resource is not freed.
>
> Is there a way around this? Can I add some sort of __del__() to the
> generator object so that in case of an early destruction of the
> generator object, the external resource is freed as well?
>
> I'm looking forward to hearing your hints!
>
> --
> Stefan Bellon

Perhaps read through:

  http://www.python.org/dev/peps/pep-0325/
  http://www.python.org/dev/peps/pep-0342/

The latter superseding the first.

Based on these what the WSGI specification:

  http://www.python.org/dev/peps/pep-0333/

did was mandate that the consumer of data from the generator
explicitly call the close() method on the generator no matter whether
all data was consumed or not, or whether an exception occurred.

result = application(environ, start_response)
try:
for data in result:
if data:# don't send headers until body appears
write(data)
if not headers_sent:
write('')   # send headers now if body was empty
finally:
if hasattr(result,'close'):
result.close()

That the consumer called close() allowed generators to provide a
close() method to cleanup resources even though older version of
Python was being used which didn't support automatic means of close()
being called. In other words, it allowed the required cleanup and was
forward compatible with newer versions of Python.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: some import / namespace questions

[Graham Dumpleton]

On Aug 9, 1:11 am, stef mientki <[EMAIL PROTECTED]> wrote:
> hello,
>
> I'm working on a rather strange program,
> that is partially build dynamically,
> and where users can create plugins,
> without knowing any of the details of the core program.
>
> Of course this leads to some weird bugs,
> so one of the things I want to view/log is the order (and even better
> the way) modules are imported.

Set and export the environment variable:

  PYTHONVERBOSE=1

in your environment before you run Python. This will generate lots of
internal information about what Python is up to, including details
about module importing, where it is searching for modules etc etc. For
example:

>>> import socket
# /System/Library/Frameworks/Python.framework/Versions/2.3/lib/
python2.3/socket.pyc matches /System/Library/Frameworks/
Python.framework/Versions/2.3/lib/python2.3/socket.py
import socket # precompiled from /System/Library/Frameworks/
Python.framework/Versions/2.3/lib/python2.3/socket.pyc
dlopen("/System/Library/Frameworks/Python.framework/Versions/2.3/lib/
python2.3/lib-dynload/_socket.so", 2);
import _socket # dynamically loaded from /System/Library/Frameworks/
Python.framework/Versions/2.3/lib/python2.3/lib-dynload/_socket.so
dlopen("/System/Library/Frameworks/Python.framework/Versions/2.3/lib/
python2.3/lib-dynload/_ssl.so", 2);
import _ssl # dynamically loaded from /System/Library/Frameworks/
Python.framework/Versions/2.3/lib/python2.3/lib-dynload/_ssl.so
import errno # builtin

Output goes to standard error. There can be quite a lot, so you may
want to redirect standard error to a file.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Overriding Thread.join in derived class.

[Graham Dumpleton]

If one creates a thread using threading.Thread and makes it a daemon
by calling setDaemon(), then when Python is exiting it will not
attempt to call join() on that thread and wait for it to complete
first.

The problem with this is that the daemonised thread will continue to
run while atexit register callbacks are being called and while Python
is being subsequently destroyed

The result is that the daemonised thread may access application
components that have been cleaned up and in the worst case as Python
progresses with destroying interpreters it could try accessing things
like sys.modules which may no longer exist.

End result is that it could result in a Python exception at some
point, evidenced by messages starting with:

Exception in thread Thread-1 (most likely raised during interpreter
shutdown)

or:

Unhandled exception in thread started by >

In order to avoid this, do people think that as an alternative to
creating daemonised thread that it would be reasonable to create a
derived Thread class for the particular task which overrides
Thread.join() method to set some flag or otherwise take some action
that the thread would notice to tell it to stop, and then call base
class Thread.join().

This way you can flag the thread to shutdown automatically when Python
is going around calling join() on non daemonised threads.

Or am I missing the obvious as far as ways of flagging threads to tell
them to stop?

Note that this is where Python is being used embedded in a C program.
There is no Python main function where code can be put to signal
threads to otherwise stop.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Overriding Thread.join in derived class.

[Graham Dumpleton]

On Aug 20, 5:40 am, Gabriel Genellina <[EMAIL PROTECTED]> wrote:
> On 18 ago, 04:31, Graham Dumpleton <[EMAIL PROTECTED]> wrote:
>
>
>
> > If one creates a thread using threading.Thread and makes it a daemon
> > by calling setDaemon(), then when Python is exiting it will not
> > attempt to call join() on that thread and wait for it to complete
> > first. [...]
> > End result is that it could result in a Python exception at some
> > point, evidenced by messages starting with:
> > Exception in thread Thread-1 (most likely raised during interpreter
> > shutdown) [...]
>
> > In order to avoid this, do people think that as an alternative to
> > creating daemonised thread that it would be reasonable to create a
> > derived Thread class for the particular task which overrides
> > Thread.join() method to set some flag or otherwise take some action
> > that the thread would notice to tell it to stop, and then call base
> > class Thread.join().
>
> > This way you can flag the thread to shutdown automatically when Python
> > is going around calling join() on non daemonised threads.
>
> > Or am I missing the obvious as far as ways of flagging threads to tell
> > them to stop?
>
> > Note that this is where Python is being used embedded in a C program.
> > There is no Python main function where code can be put to signal
> > threads to otherwise stop.
>
> This last statement is important. You need "some" way to tell the
> threads to stop working. Usually the thread's run() method is a big
> loop; you need to tell it somehow to exit the loop. By example, if you
> are using a Queue, put a sentinel object in it; or use an Event
> object; or at least a global variable.
> Overriding thread.join() as you suggest above may be a good place to
> do that, if you don't have another chance.

I already use the Queue method to flag the thread to stop. In practice
the question is more theoretical at the moment as I am able to stop
the thread by registering an atexit callback which stuffs something on
the queue and everything works okay.

The question was me more thinking out loud and wandering what other
peoples opinions were in it.

What makes this stuff even more tricky is that prior to Python 2.5,
the joining with non daemonised threads was done in a callback
registered with the atexit module. This meant it could occur in the
middle of other atexit callbacks being called and thus no absolute
certainty as to ordering.

In Python 2.5, the joining with non daemonised threads was moved to a
distinct phase before atexit callbacks were called. This was good for
non daemonised threads as it gave you more certainty. Still less for
daemonised thread though as although can use atexit callbacks to
trigger a thread to shutdown and then wait on it if necessary, still
doing stuff when some atexit callbacks may have already been called.

Thus why I was hunting for a slightly more predictable method for
daemonised threads, although would be dependent on Python 2.5 or later
being used if you absolutely need it to occur before atexit callbacks
called.

Also, just to make it even more complicated, when using Python in an
embedded manner and using secondary sub interpreters, destroying a
secondary sub interpreter does not result in the atexit callbacks
being called, nor joining with non daemonised threads (at least for <
2.5, can't remember for certain about 2.5 but don't think it does it
either). Thus, if you need these things to occur for secondary sub
interpreters, you must write code yourself to trigger both the thread
joining and atexit callbacks. :-(

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python error on Mac

[Graham Dumpleton]

On Aug 26, 12:58 pm, Clover <[EMAIL PROTECTED]> wrote:
> When trying to do some things on my Mac (starting Lyx, compiling Latex
> via TextMate) I get this error:
>
> python: execv:
> /Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python:
> No such file or directory
>
> I (and people on Lyx and TextMate lists) are at a complete loss as to
> why this is happening. I didn't (at least intentionally) fiddle with
> Python setup.

The default operating system supplied version of Python on recent
versions of MacOS X is 2.3.5. Unless you have specifically installed
2.5, it will not exist. So, if those applications have it hardwired to
use Python 2.5 they will not work.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: some problems with mod_python

[Graham Dumpleton]

On Aug 28, 6:13 am, Johan <[EMAIL PROTECTED]> wrote:
> Hi
>
> I have installed and tested this on centos, fedora and freebsd all
> give the same problem so I guess I missed some steps.
>
> I have compiled bot apache (2.2.4) and mod_python (3.3.1) according to
> the docs and no problem with this.
> But when I have made everything about testing mod_python an browse 
> tohttp://server/testand there expecting to see "Hello world" I instead
> get an index of contents in this directory. If I go to  
> http://server/test/mptest.py
> it works. No errors in any log either.
>
> What Have I missed?
>
> This I added to httpd.conf
> 
> AllowOverride All
> AddHandler mod_python .py
> PythonHandler mptest
> PythonDebug On
> 
>
> this is my mptest.py:
> from mod_python import apache
>
> def handler(req):
> req.content_type = 'text/plain'
> req.write("Hello World!")
> return apache.OK
>
> /johan

You need to look at the Apache documentation for the difference
between AddHandler and SetHandler directives. If you don't want to
read the Apache documentation and learn more about it, at least read:

http://www.dscpl.com.au/wiki/ModPython/Articles/SetHandlerVersusAddHandler

Graham


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to use os.putenv() ?

[Graham Dumpleton]

On Aug 30, 11:21 am, [EMAIL PROTECTED] wrote:
> >>> import os
>
> >>> os.environ['PATH']
>
> 'C:\\WINNT\\system32;C:\\WINNT;C:\\WINNT\\System32\\Wbem;%C:\\WINNT%\
> \system32;%C:\\WINNT%;%C:\\WINNT%\\System32\\Wbem'
>
> >>> os.putenv('PATH', 'C:\\WINNT\\system32')
>
> >>> os.environ['PATH']
>
> 'C:\\WINNT\\system32;C:\\WINNT;C:\\WINNT\\System32\\Wbem;%C:\\WINNT%\
> \system32;%C:\\WINNT%;%C:\\WINNT%\\System32\\Wbem'
>
>
>
> What am I doing wrong?  How do I change the value of an environment
> variable?

What you are missing is that os.environ is only populated from the
global process environment at process startup.

If you update os.environ the changes will be pushed into the global
process environment as well. But if you use os.putenv() instead,
bypassing os.environ, the changes will not show in os.environ.

To confirm that the global process environment is being updated, use
os.getenv().

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Module for mod_python

[Graham Dumpleton]

On Sep 6, 9:32 am, rieh25 <[EMAIL PROTECTED]> wrote:
> Is it such a bad idea that it doesn't deserve a reply?

You only posted the question six hours ago. Maybe the people who might
want to comment are asleep.

BTW, have you done an analysis of the various existing database object
relational mapper systems for Python that already exist? There are
also some systems for automatic web application forms generation based
on database data as well.

How is what you have in mind different to or not covered by these
existing packages?

If you describe how what you want to do isn't satisfied by the
existing systems then people may be more interested in commenting. If
you aren't aware of these existing systems then perhaps use Google to
do some research on them first.

Graham

> rieh25 wrote:
>
> > I've been thinking about a module (actually I have it partially
> > implemented in Zope), that would do the following things:
>
> > - Read the structure of a MySql database (fill a dictionary with it)
>
> > In order to:
>
> > - Quickly create detail/filter/update forms given a table name (without
> > specifying the fields because they have already been detected)
> > - Keep state of record selection. In order to implement Master-Detail form
> > processing, with unlimited levels of parent-child table relations, which
> > have also been detected already (foreign keys)
> > - Management of interrelation between security and menus (a user can only
> > access certain parts of the application)
>
> > I think that Ruby on Rails does something similar, but still I thing it
> > would be interesting to implement it in Python. What do you think?
>
> --
> View this message in 
> context:http://www.nabble.com/Module-for-mod_python-tf4386823.html#a12513012
> Sent from the Python - python-list mailing list archive at Nabble.com.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 3K or Python 2.9?

[Graham Dumpleton]

On Sep 12, 2:14 pm, Ben Finney <[EMAIL PROTECTED]>
wrote:
> Paul Rubin  writes:
> > TheFlyingDutchman <[EMAIL PROTECTED]> writes:
> > > Python user and advocate Bruce Eckel is disappointed with the
> > > additions (or lack of additions) in Python 3:
>
> > >http://www.artima.com/weblogs/viewpost.jsp?thread=214112
>
> > That article is pretty weak.
>
> It is responded to by Guido here:
>
>"A Response to Bruce Eckel"
> http://www.artima.com/weblogs/viewpost.jsp?thread=214325>

In that blog, Guido says:

"""Concurrency: It seems we're now happily out exploring here. I'm
looking forward to benchmarks showing that PP or similar (or
dissimilar!) solutions actually provide a performance gain. Another
route I'd like to see explored is integrating one such solution into
an existing web framework (or perhaps as WSGI middleware) so that web
applications have an easy way out without redesigning their
architecture."""

Maybe I don't fully understand where Guido is coming from, but
solutions for spreading web applications across multiple processes
have been available for a long time in solutions such as mod_python
and mod_fastcgi. With a view to improving further on these solutions
mod_wsgi has also been created.

All these solutions either use the multi process nature of the web
server, or themselves use multiple daemon processes to which requests
are distributed by Apache. End result is that one can make better use
of multi processor or multi core machines. Also, when using multi
threaded Apache worker MPM, because a lot of stuff is not even done in
Python code, such as static file serving, multiple cores can even be
used within the same process. Thus, in the larger context of how
Apache is implemented and what web applications provide, the GIL isn't
as big a problem as some like to believe it is as far as preventing
proper utilisation of the machines resources.

FWIW, I have blogged my own response to Guido's comment above at:

http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html

Now over the years I have seen a lot of Python developers showing
quite a dislike for using Python integrated with Apache. As a result
the last thing people seem to want to do is fix such solutions up and
make them work better. Reality is though that unless a very good
solution for hosting Python with Apache comes up, you will probably
never see good cheap commodity web hosting for Python. Older solutions
simply aren't safe to use or are hard to set up and manage.

Creating lots of distinct Python processes and proxying to them, like
the purists would like to see, simply isn't going to happen as such
setups are too hard to manage and use up too much resources on a large
scale. Web hosting companies want something simple which they can
integrate into their existing PHP focused Apache installations and
which don't chew up huge amounts of additional resources, thereby
forcing a reduction in their site densities. To that end, we still
have a way to go.

An older blog entry of mine where I have covered these problems is:

http://blog.dscpl.com.au/2007/07/commodity-shared-hosting-and-modwsgi.html

Graham


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: re-point mod_python - is it possible?

[Graham . Dumpleton]

On Mar 7, 3:08 pm, "leland" <[EMAIL PROTECTED]> wrote:
> i've upgraded my RHEL3 box to run python 2.3. problem is, mod_python
> still points to the old python 2.2. is there any way to tell
> mod_python that i've got a new version of python installed, and
> to use that version now?
>
> any help would be great - thanks!

First off, your existing mod_python will only work against Python 2.2.
You cannot simply coax it into trying to use Python 2.3. If you have
completely replaced Python 2.2 with 2.3 such that 2.3 is now the
default Python version, then all you need to do is recompile
mod_python from source code, or install prebuilt mod_python package
compiled against 2.3.

If however you now have both Python 2.2 and 2.3 installed at the same
time in different locations, it can get a bit more tricky. If this is
the case and even if you reinstall mod_python which has been compiled
against 2.3, you can have problems by fact that the way Python
initialises itself involves searching for 'python' executable in PATH.
If the version it finds is actually that for Python 2.2, even though
mod_python is built against 2.3, it may attempt to try and use Python
library from 2.2. If this is the case, you need to ensure that the
PATH environment used by Apache when started finds 'python' for 2.3
first.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Configuration: Apache + mod_python

[Graham . Dumpleton]

On Mar 8, 9:50 pm, "Danilo" <[EMAIL PROTECTED]> wrote:
> Hi there,
>
> is it possible to create a rewrite rule to send every server-request
> to the directory /py? But only if the file does not exists on the
> server.
>
> This is my mod_python section of the apache config-file.
>
> 
> SetHandler python-program
> PythonHandler django.core.handlers.modpython
> PythonPath "['/var/www/mydomain.com/htdocs/py'] + sys.path"
> SetEnv DJANGO_SETTINGS_MODULE myapp.settings
> PythonDebug Off
> 

For the more general case of where a HTTP 404 error would otherwise be
returned, indicating that a resource could not be found, as opposed to
an actual physical file, you can just use:

  ErrorDocument 404 /py

This would be simpler than using mod_rewrite. I can't remember though
whether the handler when triggered in this case can change the
response status to something other than 404.

You could use mod_rewrite if you really must, but not sure how it
would interact with virtual resources managed by some handler where no
actual file exists. To be practical you would probably want to
restrict the scope of mod_rewrite to specific contexts.

Quoting an example from very good book "The Definitive Guide to Apache
mod_rewrite", you can do something similar to:

  RewriteEngine On
  # If its not here ...
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  # Look here instead ...
  RewriteRule ^/images/(.*) /pics/$1 [PT]

In this case it is causing lookups for images to be made in two
places, but your case wouldn't be much different.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Configuration: Apache + mod_python

[Graham . Dumpleton]

On Mar 9, 12:02 am, "Danilo" <[EMAIL PROTECTED]> wrote:
> On 8 Mrz., 12:18, [EMAIL PROTECTED] wrote:
>
>
>
> > On Mar 8, 9:50 pm, "Danilo" <[EMAIL PROTECTED]> wrote:
>
> > > Hi there,
>
> > > is it possible to create a rewrite rule to send every server-request
> > > to the directory /py? But only if the file does not exists on the
> > > server.
>
> > > This is my mod_python section of the apache config-file.
>
> > > 
> > > SetHandler python-program
> > > PythonHandler django.core.handlers.modpython
> > > PythonPath "['/var/www/mydomain.com/htdocs/py'] + sys.path"
> > > SetEnv DJANGO_SETTINGS_MODULE myapp.settings
> > > PythonDebug Off
> > > 
>
> > For the more general case of where a HTTP 404 error would otherwise be
> > returned, indicating that a resource could not be found, as opposed to
> > an actual physical file, you can just use:
>
> >   ErrorDocument 404 /py
>
> > This would be simpler than using mod_rewrite. I can't remember though
> > whether the handler when triggered in this case can change the
> > response status to something other than 404.
>
> > You could use mod_rewrite if you really must, but not sure how it
> > would interact with virtual resources managed by some handler where no
> > actual file exists. To be practical you would probably want to
> > restrict the scope of mod_rewrite to specific contexts.
>
> > Quoting an example from very good book "The Definitive Guide to Apache
> > mod_rewrite", you can do something similar to:
>
> >   RewriteEngine On
> >   # If its not here ...
> >   RewriteCond %{REQUEST_FILENAME} !-f
> >   RewriteCond %{REQUEST_FILENAME} !-d
> >   # Look here instead ...
> >   RewriteRule ^/images/(.*) /pics/$1 [PT]
>
> > In this case it is causing lookups for images to be made in two
> > places, but your case wouldn't be much different.
>
> > Graham
>
> The rewrite rule works, but now every request ist send to /py.
> This is my .conf:
>
> 
> DocumentRoot /var/www/mydomain.com/htdocs
> ServerName mydomain.com
> ServerAliaswww.mydomain.com
>
> 
> SetHandler python-program
> PythonHandler django.core.handlers.modpython
> PythonPath "['/var/www/mydomain.com/htdocs/py'] + sys.path"
> SetEnv DJANGO_SETTINGS_MODULE myapp.settings
> PythonDebug Off
> 
>
> RewriteEngine On
> # If its not here...
> RewriteCond %{REQUEST_FILENAME} !-f
> RewriteCond %{REQUEST_FILENAME} !-d
> # Look here instead...
> RewriteRule (.*) /py$1 [PT]
>
> ErrorLog /var/www/mydomain.com/logs/error.log
> CustomLog /var/www/mydomain.com/logs/access.log common
> 
>
> Any ideas what is wrong?

I did say you would probably need to restrict the scope of the
mod_rewrite rule to a specific context. In particular, put it inside
of a Directory directive corresponding to the file system directory
where your files live. Where you have it as the moment,
REQUEST_FILENAME probably will not resolve to anything as Apache
hasn't yet matched it to the filesystem. Thus:

  

RewriteEngine On
# If its not here...
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# Look here instead...
RewriteRule (.*) /py$1 [PT]


  

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Configuration: Apache + mod_python

[Graham Dumpleton]

On Mar 9, 7:09 pm, "Danilo" <[EMAIL PROTECTED]> wrote:
> On 8 Mrz., 22:23, [EMAIL PROTECTED] wrote:
>
>
>
> > On Mar 9, 12:02 am, "Danilo" <[EMAIL PROTECTED]> wrote:
>
> > > On 8 Mrz., 12:18, [EMAIL PROTECTED] wrote:
>
> > > > On Mar 8, 9:50 pm, "Danilo" <[EMAIL PROTECTED]> wrote:
>
> > > > > Hi there,
>
> > > > > is it possible to create a rewrite rule to send every server-request
> > > > > to the directory /py? But only if the file does not exists on the
> > > > > server.
>
> > > > > This is mymod_pythonsection of the apache config-file.
>
> > > > > 
> > > > > SetHandler python-program
> > > > > PythonHandler django.core.handlers.modpython
> > > > > PythonPath "['/var/www/mydomain.com/htdocs/py'] + sys.path"
> > > > > SetEnv DJANGO_SETTINGS_MODULE myapp.settings
> > > > > PythonDebug Off
> > > > > 
>
> > > > For the more general case of where a HTTP 404 error would otherwise be
> > > > returned, indicating that a resource could not be found, as opposed to
> > > > an actual physical file, you can just use:
>
> > > >   ErrorDocument 404 /py
>
> > > > This would be simpler than using mod_rewrite. I can't remember though
> > > > whether the handler when triggered in this case can change the
> > > > response status to something other than 404.
>
> > > > You could use mod_rewrite if you really must, but not sure how it
> > > > would interact with virtual resources managed by some handler where no
> > > > actual file exists. To be practical you would probably want to
> > > > restrict the scope of mod_rewrite to specific contexts.
>
> > > > Quoting an example from very good book "The Definitive Guide to Apache
> > > > mod_rewrite", you can do something similar to:
>
> > > >   RewriteEngine On
> > > >   # If its not here ...
> > > >   RewriteCond %{REQUEST_FILENAME} !-f
> > > >   RewriteCond %{REQUEST_FILENAME} !-d
> > > >   # Look here instead ...
> > > >   RewriteRule ^/images/(.*) /pics/$1 [PT]
>
> > > > In this case it is causing lookups for images to be made in two
> > > > places, but your case wouldn't be much different.
>
> > > > Graham
>
> > > The rewrite rule works, but now every request ist send to /py.
> > > This is my .conf:
>
> > > 
> > > DocumentRoot /var/www/mydomain.com/htdocs
> > > ServerName mydomain.com
> > > ServerAliaswww.mydomain.com
>
> > > 
> > > SetHandler python-program
> > > PythonHandler django.core.handlers.modpython
> > > PythonPath "['/var/www/mydomain.com/htdocs/py'] + 
> > > sys.path"
> > > SetEnv DJANGO_SETTINGS_MODULE myapp.settings
> > > PythonDebug Off
> > > 
>
> > > RewriteEngine On
> > > # If its not here...
> > > RewriteCond %{REQUEST_FILENAME} !-f
> > > RewriteCond %{REQUEST_FILENAME} !-d
> > > # Look here instead...
> > > RewriteRule (.*) /py$1 [PT]
>
> > > ErrorLog /var/www/mydomain.com/logs/error.log
> > > CustomLog /var/www/mydomain.com/logs/access.log common
> > > 
>
> > > Any ideas what is wrong?
>
> > I did say you would probably need to restrict the scope of the
> > mod_rewrite rule to a specific context. In particular, put it inside
> > of a Directory directive corresponding to the file system directory
> > where your files live. Where you have it as the moment,
> > REQUEST_FILENAME probably will not resolve to anything as Apache
> > hasn't yet matched it to the filesystem. Thus:
>
> >   
>
> > RewriteEngine On
> > # If its not here...
> > RewriteCond %{REQUEST_FILENAME} !-f
> > RewriteCond %{REQUEST_FILENAME} !-d
> > # Look here instead...
> > RewriteRule (.*) /py$1 [PT]
>
> >   
>
> > Graham
>
> Thank you.
>
> the RewriteCond just needs the absolute path:
>
> RewriteEngine On
> # If its not here...
> RewriteCond /var/www/btsgroup.de/htdocs/%{REQUEST_FILENAME} !-f
> RewriteCond /var/www/btsgroup.de/htdocs/%{REQUEST_FILENAME} !-d
> # Look here instead...
> RewriteRule (.*) /py$1 [PT]

Doing that would probably be considered bad practice. I think the
problem was I neglected to mention you would have to change your
RewriteRule to add a slash when used in Directory directive. Ie., use:



RewriteEngine On
# If its not here...
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# Look here instead...
RewriteRule (.*) /py/$1 [PT]



Note the slash after /py.

This works for me when I test it.


Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: merits of Lisp vs Python

[Graham Dumpleton]

On Mar 11, 12:31 pm, [EMAIL PROTECTED] (John J. Lee) wrote:
> John Nagle <[EMAIL PROTECTED]> writes:
> > John J. Lee wrote:
> > > John Nagle <[EMAIL PROTECTED]> writes:
> > > [...]
>
> > >>Python, on the other hand, is uphill all the way.  Constant trouble
> > >>with version issues, especially with C components called from Python.
> > >>MySQLdb, M2Crypto, SSL - they all have platform/version
> > >>incompatibility problems.  I just spent three days making M2Crypto
> > >>work on a new Linux server with a different Red Hat version.
> > >>Neither Python's packaging tools nor the platform's packaging
> > >>tools deal adequately with these issues.
> > > [...]
> > > You haven't been usingmod_python, by any chance?
> > > John
>
> > No, haven't started to deal with that yet.  Still using
> > CGI.  Not sure whether to usemod_pythonor fastcgi for the
> > small requests where the Python load time swamps the time
> > to do one SQL select and reply.  Comments?
>
> mod_pythonrelies on an unsupported feature of Python, namedly
> multiple interpreters --> risk of more pain with C extensions.

As usual, those bashing up on mod_python tend not to really know what
they are talking about. :-(

Although multiple Python interpreters cannot be manipulated directly
from within a Python script, they can from the Python C API and it is
very much a feature of Python and has been for a long time.

The only issue with multiple sub interpreters in respect of C
extension modules is that implementers of those C extension modules
take the easy path and use the simplified thread API for GIL locking.
The consequence of them doing that is that their C extension module
may not work when used in anything but the first interpreter created
by Python. If instead of using the simplified thread API for GIL
locking they used other parts of the Python threading API as
appropriate and did thread lock handling properly for multiple
interpreters, there would not be an issue.

Even so, to get such a C extension module working in the context of
mod_python simply means telling mod_python to run that particular
application in the first interpreter instance by specifying the
mod_python directive:

  PythonInterpreter main_interpreter

Thus, the problem is not mod_python at all, but that the C extension
modules implementer didn't bother to implement their module so as to
be usable within a system such as mod_python where multiple
interpreters are used. Further, mod_python even provides a way to work
around the problems with such third party C extension modules.

If you feel that this is not the case and mod_python is still broken
in some way, please properly explain what the problem is.

PS. Yes I do know that forcing using of main interpreter in mod_python
only helps solve this problem in mod_python 3.2 and later and did not
help with older versions.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: merits of Lisp vs Python

[Graham Dumpleton]

On Mar 15, 7:22 am, [EMAIL PROTECTED] (John J. Lee) wrote:
> "Graham Dumpleton" <[EMAIL PROTECTED]> writes:
> > On Mar 11, 12:31 pm, [EMAIL PROTECTED] (John J. Lee) wrote:
> [...]
> > >mod_pythonrelies on an unsupported feature of Python, namely
> > > multiple interpreters --> risk of more pain with C extensions.
>
> > As usual, those bashing up onmod_pythontend not to really know what
> > they are talking about. :-(
>
> > Although multiple Python interpreters cannot be manipulated directly
> > from within a Python script, they can from the Python C API and it is
> > very much a feature of Python and has been for a long time.
>
> I didn't dispute that multiple interpreters are a feature of Python.
> I said that they are an unsupported feature -- at least, Martin
> v. Loewis says they're "just broken", which is close enough for me:
>
> http://groups.google.co.uk/group/comp.lang.python/browse_thread/threa...
>
> """
> In any case, it looks like that the "multiple interpreters" feature of
> Python is just broken.
> """

For mod_python at least, the issues described there do not present as
a problem because in mod_python sub interpreters are never destroyed
while the process is running. Thus as far as the implementation of
mod_python goes it is fine to key off the pointer to the interpreter
as one knows interpreters will never go away.

In the more general case I can see that it may be an issue for some
small percentage of C extension modules which may want to cache
information per interpreter. This though would only be the case if
they wanted to exclusively hold everything directly within the C code
space. Although it would perceivably slow access down, there is
nothing to stop such a module instantiating its own pseudo module
within the context of a sub interpreter using PyImport_AddModule().
This Python module could then be used to hold objects created by
PyCObject_FromVoidPtr() with access later being had by using
PyCObject_AsVoidPtr() after having looked up the object in the module.
As necessary, when creating these objects, the C extension module can
associate a cleanup function to be called when the object is destroyed
by virtue of the sub interpreter being destroyed. Thus the C extension
module would be able to cope with sub interpreters coming an going.

So, more work is required and some may be concerned about efficiency
and that a user may screw with the cached data from Python code, but
it is possible to have per interpreter information with a C extension
module.

> [...]> Even so, to get such a C extension module working in the context of
> >mod_pythonsimply means tellingmod_pythonto run that particular
> > application in the first interpreter instance by specifying the
> >mod_pythondirective:
>
> >   PythonInterpreter main_interpreter
>
> [...]
>
> Is it possible to askmod_pythonto start separate processes to serve
> requests, rather than "separate" interpreters?  We couldn't see a way.

Within the context of Apache it is possible to write modules which
could spawn off a separate daemon process to which requests could
later be passed for processing. An example of this included with
Apache is something like mod_cgid. Other examples of modules which
allow requests to be farmed off to separate processes, although they
work a bit differently, are mod_fastcgi, mod_scgi and mod_proxy_ajp.

To do something similar with mod_python isn't really practical because
of the way it hooks into more than just the response handling phase of
Apache. This concept though is being investigated for some version of
mod_wsgi after the initial version has been released. When
implemented, it would allow mod_wsgi to either handle WSGI
applications in the same style as mod_python within the Apache child
processes, or optionally farm the request off to a separate process
running either as the same user as Apache or a different user, in the
same style as mod_fastcgi and mod_scgi. The difference with mod_wsgi
would be that it would be one self contained module all managed from
Apache and would not require a separate executable or daemon process
to Apache to be run to which it would communicate with as the separate
daemon processes would ultimately just be a fork of the Apache parent
process and be running code from the Apache module itself. Also the
application scripts would be exactly the same no matter which mode it
was running in so no need to modify them in any special way to run
under the separate process.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C extension providing... Python's own API?

[Graham Dumpleton]

On Mar 27, 11:02 am, "Adam Atlas" <[EMAIL PROTECTED]> wrote:
> On Mar 26, 4:55 pm, "Matimus" <[EMAIL PROTECTED]> wrote:
>
> > I think that is what the "code" module is for. Maybe not exactly what
> > you were expecting, but the capability you describe is already there.
> > Being able to access its own interpreter is one of the things that
> > makes Python a  dynamic language.
>
> That's not really what I'm after. Believe me, I've searched the
> standard library in length and breadth. :) The `code` module runs in
> the same interpreter, basically executing as though it were just a
> separate module (except interactively). What I'm interested in is
> accessing Python's C API for CREATING interpreters -- i.e. whatmod_pythonor 
> anything else embedding the Python interpreter would do.
> Creating a whole environment, not just a module; it has its own `sys`
> parameters, its own `__builtin__`, and so on, so you can safely mess
> with those without changing them in the parent interpreter.
>
> So far, I've tried ctypes, and it doesn't work; then I noticed the
> docs said it wouldn't anyway. But I think I'll try writing a C
> extension module. This could be interesting for sandboxing &c.

Having played around in that area more than most I would say it would
be entirely possible to create a C extension module for Python that
would allow one to create additional sub interpreters. The question
only becomes how does one want to make use of it and make the
interface look like.

A simple interface may be to direct that in named sub interpreter
import this module. If named sub interpreter didn't exist then it
would be created. To have code run in the separate interpreter on an
ongoing basis, the imported module could create a thread. Personally
though I don't like triggering side affects from module import, so
might be better to say import this module and if that is successful
then call named object in that module with specified arguments. The
latter is better in that it allows data to be passed across as well,
although you would want to limit to basic data types for various
reasons.

The only tricky issues in this are ensuring that thread state is given
up when calling out of the first interpreter and ensuring that new
thread state is created against the new interpreter before calling
into it. You can't use the same thread state as then you get into
issues with exceptions about running in restricted mode.

Another issue to contend with is whether you allow sub interpreters to
be deleted. This can be tricky for various reasons. First is that when
deleting sub interpreters anything registered with atexit.register()
isn't run as that only happens for main Python interpreter, but one
can ensure they are by calling sys.exitfunc() within that sub
interpreter. Even then, if you have separate threads running in that
sub interpreter and they don't register an atexit function to allow
themselves to be killed, it could hang the attempt to delete the sub
interpreter. Even if that works, you may find that some third party
extension modules can't cope with sub interpreters being killed
through caching of the interpreter reference, something that might end
up point elsewhere after the sub interpreter is killed.

There are also other little minor details that one has to worry about
like providing a sys.argv as sub interpreters don't have that but some
Python modules blindly expect to be able to access it even though they
aren't the actual main function of a program. Also possibly need to
restrict access to sys.stdin from sub interpreters and only allow main
interpreter to use it etc etc.

Anyway, what you want to do is something I have thought about before
in relation to  mod_wsgi and whether there is any use for the ability
to call between sub interpreters. In the context of mod_wsgi though
such a thing is probably just an avenue for creating more mischief,
especially within a hosting environment where different peoples
applications may be running in different sub interpreters.

On the other hand, it might be useful in a standalone Python based
WSGI web server which you have more direct control over.  It might
take a bit of design work as to how to do it in practice, but you
could create different sub interpreters through the module for
distinct WSGI applications. The main interpreter could be running the
web server and somehow then hand off a WSGI environ etc off to a
manager module in the other sub interpreter that then deals with
communicating with the WSGI application in that sub interpreter for
that specific request. You could conceivably have a hand off
arrangement whereby when an application code base changes that you
start routing requests for URL subset to new instance of application
in new sub interpreter and kill off old sub interpreter when able to.
Would certainly be an interesting area to look at. For mod_wsgi at
least though I ruled out allowing sub interpreters to be killed off
and allowing reloading of a complete appl

Re: Python C extension providing... Python's own API?

[Graham Dumpleton]

On Mar 27, 12:41 pm, "Graham Dumpleton" <[EMAIL PROTECTED]>
wrote:
> On the other hand, it might be useful in a standalone Python based
> WSGI web server which you have more direct control over.  It might
> take a bit of design work as to how to do it in practice, but you
> could create different sub interpreters through the module for
> distinct WSGI applications. The main interpreter could be running the
> web server and somehow then hand off a WSGI environ etc off to a
> manager module in the other sub interpreter that then deals with
> communicating with the WSGI application in that sub interpreter for
> that specific request. You could conceivably have a hand off
> arrangement whereby when an application code base changes that you
> start routing requests for URL subset to new instance of application
> in new sub interpreter and kill off old sub interpreter when able to.
> Would certainly be an interesting area to look at. For mod_wsgi at
> least though I ruled out allowing sub interpreters to be killed off
> and allowing reloading of a complete application by such a handoff
> mechanism as too dangerous a feature in an ISP based web hosting
> environment as user code could too easily hang the Apache process when
> trying to kill off the sub interpreter.

One more thought. It would actually be quite cute if you could make
this whole encapsulation of pushing a WSGI request into a distinct sub
interpreter a WSGI middleware component. That way you could just drop
it into any existing web server infrastructure that supports WSGI. Now
I'm getting really interested to have a play. :-)

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to get IP address of client from CGI module?

[Graham Dumpleton]

On Apr 11, 12:22 pm, John Nagle <[EMAIL PROTECTED]> wrote:
>The documentation for Python's CGI module doesn't seem to say how to get
> the IP address of the client.  Don't see an obvious way to get that info
> from reading the source, either. Ideas?
>
> John Nagle

Read the CGI RFC.

  http://www.ietf.org/rfc/rfc3875

Specifically section 4.1.8.

This value should be supplied by way of variable in os.environ as are
all the CGI variables.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: lxml + mod_python: cannot unmarshal code objects in restricted execution mode

[Graham Dumpleton]

On Sep 13, 11:05 pm, Dmitri Fedoruk <[EMAIL PROTECTED]> wrote:
> Hello everyone,
>
> I'm developing a mod_python application that is based on XML\XSLT
> transforming.
>
> I used 4Suite libraries for that, but as the speed was unacceptable
> for me, I switched to lxml. Everything became much easier and 10 times
> faster, but I've encountered the subject problem.
>
> In brief - all my data and xslt are stored and transferred in UTF-8.
> With 4Suite everything was fine all the time. With lxml it works fine
> from the console, but inside mod_python it occasionaly dies, ~ one
> time out of three. Strange - the same code with the same data works or
> dies by its own means.
>
> As far as I have found, there was a similar problem with PyXML and
> encodings module, but there was no clear solution.
>
> So, my configuration is the following:
> Python 2.5.1
> Server version: Apache/2.2.4 (FreeBSD)
> mod_python-3.3.1
>
> And the relevant parts of my code are these:
>
> def extApplyXslt(xslt, data, logger ):
> try:
> strXslt = urllib2.urlopen(xslt).read()
> # i have to read the xslt url to the python string
> except urllib2.HTTPError, e:
>...
> except urllib2.URLError, e:
>.
> try:
> xslt_parser = etree.XMLParser()
> xslt_parser.resolvers.add( PrefixResolver("XSLT") )
>
> # and now I have to use the string; a more elegant solution,
> anyone?
> f = StringIO(strXslt)
> xslt_doc = etree.parse(f, xslt_parser)
>
> # and here where the problem comes
> transform = etree.XSLT(xslt_doc)
>
> except Exception, exc:
> logger.log(logging.CRITICAL, exc.__str__() )
>
> try:
>   result_tree = transform(data)
>   return etree.tostring(result_tree, 'utf-8')
> except Exception, exc:
>   print "xslt processing error!", exc.__str__()
>   return ""
>
> It dies with the message 'cannot unmarshal code objects in restricted
> execution mode'. By profiling I detected the point where problem
> occurs:
>  transform = etree.XSLT(xslt_doc)
>
> So, I would be grateful for any suggestions how to get rid of this.
> I'd really like to use lxml. Maybe I should initialize the xslt
> processor in somehow other way?
>
> Thanks in advance,
> Dmitri

Try forcing mod_python to run your code in the first interpreter
instance created by Python.

  PythonInterpreter main_interpreter

If that still doesn't work then you are hitting problems in mod_python
related to how it does GIL management or how the third party package
is implemented and whether it is compatible with use in secondary
Python interpreters. For details see:

If it still doesn't work and your application is a WSGI application
and you aren't using mod_python specific features, only choice at this
point would be to use mod_fastcgi or mod_wsgi instead. If using
mod_wsgi, you would still need to tell it to run your code in first
interpreter instance created by Python.

  WSGIApplicationGroup %{GLOBAL}

For some details on issues related to this problem, see sections
'Python Simplified GIL State API' and 'Multiple Python Sub
Interpreters' in mod_wsgi document:

  http://code.google.com/p/modwsgi/wiki/ApplicationIssues

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python,SWIG and libjvm

[Graham Dumpleton]

On Sep 21, 9:00 am, sapsi <[EMAIL PROTECTED]> wrote:
> Hello,
> I'm not sure if this the correct list but here goes (and sorry for the
> noise). I've been attempting to wrap python around libhdfs.
> So far so good (i've attached the SWIG template at the end). The
> compilation works without errors and the shared objects do have
> references to all the functions.
>
> However, when importing into python
>
> import pyhdfs
>
> i get the following error:
>
> Traceback (most recent call last):
>   File "", line 1, in ?
>   File "/home/sguha/tmp/b/hdfs.py", line 7, in ?
> import _hdfs
> ImportError: libjvm.so: cannot open shared object file: No such file
> or directory
>
> However, libjvm.so is located in /home/sguha/mine/jdk1.6.0_02/jre/lib/
> amd64/server which is present in the PYTHONPATH and sys.path.
> I can understand it complaining while loading the libjvm.so
> but python doesn't locate it even when in the path.
>
> As an aside, the build command for hdfs.c has "   -ljvm  " - this
> should affect SWIG, right? Meaning if my program (e.g) links to a
> library and i want a wrapper around my program (to python) i need SWIG
> templates for my headers only and not for the linked library...right?
> ( I guess the answer is i dont otherwise that would be quite a
> nightmare of work)
>
> Thank you for your time
> Saptarshi
>
> Attachments:
> SWIG template
> %module pyhdfs
>
> %{
> #include "hdfs.h"
> %}
>
> %include "hdfs.h"

Your libjvm.so isn't being found because it isn't in a standard system
library directory. You have a few choices to solve this:

1. Install libjvm.so into /lib or /usr/lib. You may be able to get
away with putting it in /usr/local/lib if your OS is setup to look
there.

2. If a Linux/Solaris box, do:

  LD_LIBRARY_PATH=/home/sguha/mine/jdk1.6.0_02/jre/lib/amd64/server
  export LD_LIBRARY_PATH

then run your application. This will tell you application to look in
that directory for the shared library at run time.

3. If on Linux/Solaris, compile that library directory into the Python
module .so by setting before you compile your Python module:

  LD_RUN_PATH=/home/sguha/mine/jdk1.6.0_02/jre/lib/amd64/server
  export LD_RUN_PATH

This only needs to be done at compile time, but means your module will
always expect it to be in that location.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mod_python preprocess/filter before proxy

[Graham Dumpleton]

Yes, use PythonInputFilter directive to specify an input filter.

  http://www.modpython.org/live/current/doc-html/dir-filter-if.html

Input filters which modify the length of the data may be an issue
though when doing proxying however.

If you cant work it out, possibly more appropriate to take your
questions to the official mod_python mailing list referenced on the
mod_python site.

Graham

On Sep 27, 9:26 pm, David Sánchez Martín <[EMAIL PROTECTED]> wrote:
> Hi!
>
>   I've seen the message below in this python list, that seems to be
> unanswered.
>
>   I'm trying to do the pretty same thing.
>
>   There's a way to preprocess the request with a mod_python handler and then
> proxying it with mod_proxy?
>
>   Thank you very much in advance, and sorry for trying to revive such an old
> message.
>
> Cheers.
>
>
>
>
>
> >whale whale at mycameo.com
> >Tue Mar 25 05:49:42 CET 2003
>
> >I use Apache2.0.44 and mod_python3.0.1.
> >I add ProxyPass and PythonHandler directives in httpd.conf
> >## httpd.conf
> >ProxyPass  /test/  http://www.test.com
> >ProxyPassReverse  /test/  http://www.test.com
> >
> >AddHandler python-program .htm .html
> >PythonHandler mptest
> >
>
> >And mptest.py:
> >from mod_python import apache
> >def handler(req):
> >req.write('Hello!!')
> >return apache.OK
>
> >ex. My apache server iswww.server.com.
> >When I browsed the local server request(ex.
> >http://www.server.com/1.html),
> >I received 'Hello!!' response, the python handler worked well.
> >But when I browse the proxy request(ex.
> >http://www.server.com/test/1.html),
> >the response I received is the original content of
> >http://www.test.com/1.html, not 'Hello!!'.
> >The proxy requests didn't seem to be processed by mod_python handler.
> >How could I pre-process the proxy request before mod_proxy module?
> >Thanks a lot!!
>
> ---
> David Sanchez Martin
> Administrador de Sistemas
> [EMAIL PROTECTED]
> GPG Key ID: 0x37E7AC1F
>
> E2000 Nuevas Tecnologías
> Tel : +34 902 19 61 77
>
>  David Sánchez Martín.vcf
> 1KDownload
>
>  smime.p7s
> 1KDownload


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PYTHONPATH on OS X

[Graham Dumpleton]

On Oct 11, 8:00 am, "mhearne808[insert-at-sign-here]gmail[insert-dot-
here]com" <[EMAIL PROTECTED]> wrote:
> I'm missing something major here.  I'm trying to add a directory to my
> python path using the PYTHONPATH environment variable, and it's being
> ignored by the Python interactive shell.
>
> Below is a capture of what I did.  Note that my newfolder appears
> nowhere on the list of directories in sys.path.  How do I get Python
> to pay attention to my shell variables?
>
> Using bash on OS X 10.4.10.
>
> %:~ user$ echo $PYTHONPATH
>
> %:~ user$ PYTHONPATH=/Users/user/newfolder
> %:~ user$ echo $PYTHONPATH
> /Users/user/newfolder
> %:~ user$ python
> Python 2.5.1 (r251:54863, Aug 10 2007, 10:46:58)
> [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.>>> 
> import sys
> >>> sys.path
>
> ['', '/usr/local/lib/python2.5/site-packages/
> setuptools-0.7a1dev_r56320-py2.5.egg', '/usr/local/lib/python2.5/site-
> packages/ipython1-0.9alpha2-py2.5.egg', '/usr/local/lib/python2.5/site-
> packages/SQLAlchemy-0.4.0beta5-py2.5.egg', '/usr/local/lib/python2.5/
> site-packages/MySQL_python-1.2.2-py2.5-macosx-10.3-i386.egg', '/usr/
> local/lib/python25.zip', '/usr/local/lib/python2.5', '/usr/local/lib/
> python2.5/plat-darwin', '/usr/local/lib/python2.5/plat-mac', '/usr/
> local/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/lib/
> python2.5/lib-tk', '/usr/local/lib/python2.5/lib-dynload', '/usr/local/
> lib/python2.5/site-packages', '/usr/local/lib/python2.5/site-packages/
> PIL']

You need to export the environment variable.

  export PYTHONPATH

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Paste and WSGI 2.0 [WAS: Yet another comparison of Python Web Frameworks]

[Graham Dumpleton]

On Oct 14, 6:46 pm, Michele Simionato <[EMAIL PROTECTED]>
wrote:
> Now, since you are here, there is an unrelated question that I want to
> ask you, concerning the future of Paste with respect to WSGI 2.0.
> I do realize that at this stage WSGI 2.0, is only a draft

Hmmm, not sure where people keep getting this idea that there exists a
WSGI 2.0 draft.

The only thing that exists that I know of is:

  http://www.wsgi.org/wsgi/WSGI_2.0

This is at this point merely a collection of ideas for change, or
areas where the existing WSGI 1.0 is deficient. There are a various
things around this where decisions would need to be made or solutions
developed before one could even consider calling it a draft. I
certainly wouldn't necessarily go off making decisions based on what
may or may not be in WSGI 2.0. :-)

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem calling python modules from python apache handler

[Graham Dumpleton]

On Oct 17, 8:00 am, mannewalis <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have the following setup...
>
> Apache 2.2.3, Python 2.5 and mod_python 3.3.1
>
> I have configured apache to call a python script when fetching .py
> resources...so far so good... I did the hello world script in the
> mod_python docs and it works great.
>
> Now when I want to add some meat and potatoes and call some other
> python code from the script things get pear shaped.. I try to import
> the module I want, but it never can find it... I always get the
> error...
>
> ImportError: No module named ???
>
> no matter what I do. I have moved my python modules around, to no
> avail. I've even put them in the same directory as the handler script
> and even that doesn't work.
>
> I am fairly new to python, so I expect there is some search-path-
> fu(tm) that I am presently unaware of. Can anybody explain how to get
> this working?
>
> Thanks
> Justin
>
> For example...
>
> I have the following folders...
>
> \python25\lib\test\
>
> with the script mytest.py
>
> In my apache hander I try to import this script
>
> from test import mytest

First, done call your package directory 'test'. There is a module in
Python called that and if it finds the Python module first it will not
find your package.

Second, you can't dump Python packages in the same directory as your
mod_python handler code file, they will not be found. Place them
outside of the Apache document tree and use PythonPath directive as
described in documentation to indicate where they are located.

Also ensure you read documentation for import_module() in:

  http://www.modpython.org/live/current/doc-html/pyapi-apmeth.html

This covers issues around module loading in mod_python.

Graham


> and get the error..
>
> MOD_PYTHON ERROR
>
> ProcessId:  4988
> Interpreter:'localhost'
>
> ServerName: 'localhost'
> DocumentRoot:   'C:/Program Files/EasyPHP 2.0b1/www'
>
> URI:'/foo.py'
> Location:   None
> Directory:  'C:/Program Files/EasyPHP 2.0b1/www/'
> Filename:   'C:/Program Files/EasyPHP 2.0b1/www/foo.py'
> PathInfo:   ''
>
> Phase:  'PythonHandler'
> Handler:'query'
>
> Traceback (most recent call last):
>
>   File "C:\Python25\lib\site-packages\mod_python\importer.py", line
> 1537, in HandlerDispatch
> default=default_handler, arg=req, silent=hlist.silent)
>
>   File "C:\Python25\lib\site-packages\mod_python\importer.py", line
> 1202, in _process_target
> module = import_module(module_name, path=path)
>
>   File "C:\Python25\lib\site-packages\mod_python\importer.py", line
> 296, in import_module
> log, import_path)
>
>   File "C:\Python25\lib\site-packages\mod_python\importer.py", line
> 680, in import_module
> execfile(file, module.__dict__)
>
>   File "C:\Program Files\EasyPHP 2.0b1\www\query.py", line 2, in
> 
> from test import mytest
>
> ImportError: No module named test
>
> MODULE CACHE DETAILS
>
> Accessed:   Tue Oct 16 14:50:22 2007
> Generation: 4
>
> _mp_9bae44ea3d2a76746732f5d982e3e12d {
>   FileName: 'C:\\Program Files\\EasyPHP 2.0b1\\www\\query.py'
>   Instance: 13 [RELOAD]
>   Generation:   4 [ERROR]
>   Modified: Mon Oct 15 19:54:19 2007
>   Imported: Mon Oct 15 19:50:44 2007
>
> }


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MOD_PYTHON + packages reloading

[Graham Dumpleton]

On Oct 18, 6:55 pm, Duncan Booth <[EMAIL PROTECTED]> wrote:
> [EMAIL PROTECTED] wrote:
> > I came across annoying problem during my fun with mod_python. I turned
> > out that mod_python load package only onca and don't care about any
> > changes to it. Obviously it makes sense on production server but
> > during development is more then annoying.
>
> Have you read the mod_python manual?
>
> http://www.modpython.org/live/current/doc-html/dir-other-par.html
>
> 5.4.8 PythonAutoReload
>
> Syntax: PythonAutoReload {On, Off}
> Default: PythonAutoReload On
> Context: server config, virtual host, directory, htaccess
> Override: not None
> Module: mod_python.c
>
> If set to Off, instructs mod_python not to check the modification date
> of the module file.
>
> By default, mod_python checks the time-stamp of the file and reloads the
> module if the module's file modification date is later than the last
> import or reload. This way changed modules get automatically reimported,
> eliminating the need to restart the server for every change.
>
> Disabling autoreload is useful in production environment where the
> modules do not change; it will save some processing time and give a
> small performance gain.

Have you read the other document:

  http://www.modpython.org/live/current/doc-html/pyapi-apmeth.html

In particular read the documentation for the import_module() function.
It clearly describes in what limited cases module reloading is
performed. It does not do it for all modules as many mistakenly
believe. It does not do any form of module reloading on Python
packages because it is impossible to do properly due to the fact that
cyclic dependencies can occur with how packages are generally
structured.

For a description of some of the issues and why mod_python 3.3 works
like it does, see the big list of problems related to module reloading
in older versions of mod_python:

  http://www.dscpl.com.au/wiki/ModPython/Articles/ModuleImportingIsBroken

So, what is in 3.3 as described in documentation for import_module()
is as good as it gets. As long as you work within the constraints it
imposes you can go a long way, but don't expect miracles.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to tell if a script has been run by Pythonw.exe instead of Python.exe?

[Graham Dumpleton]

On Oct 19, 7:56 am, Metalone <[EMAIL PROTECTED]> wrote:
> In particular I want to know how to tell if reading and writing to the
> console can occur.
> Something like
> sys.isConsolePresent()

Have you tried:

  sys.stdin.isatty()
  sys.stdout.isatty()

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Monitoring external processes

[Graham Dumpleton]

On Oct 23, 3:09 pm, [EMAIL PROTECTED] wrote:
> Hi,
>
> Is there a way to track external processes launched by python on the
> Mac? I am using subprocess module to launch the process.
>
> Thanks
> Sunil

If using Python 2.3/2.4, you can use os.wait().

If using Python 2.5, there is also have os.wait3() and os.wait4().

See the operating system manual pages for the difference. Ie.,

  man wait4

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: sys.path not properly initialized (was: PyImport_ImportModule/embedding: surprising behaviors)

[Graham Dumpleton]

On Oct 26, 6:53 am, David Abrahams <[EMAIL PROTECTED]> wrote:
> > David Abrahams wrote:
> >> I'm seeing highly surprising (and different!) behaviors of
> >> PyImport_ImportModule on Linux and Windows when used in a program with
> >> python embedding.
>
> >> On Linux, ...
>
> 
>
> Unfortunately, nothing you have written below or on the pages you
> reference seems to help in Windows.  Here's a simple program that
> demonstrates:
>
>  import.c
> 1KDownload
>
>   'import site' failed; use -v for traceback
>   Traceback (most recent call last):
> File "", line 1, in 
>   ImportError: __import__ not found
>
> >> I can work around the problem by setting PYTHONPATH to point to the
> >> python library directory:
>
> >>   set PYTHONPATH=c:\Python25\Lib
>
> > This happens because Python calculates the initial import path by
> > looking for an executable file named "python" along PATH.
>
> C:\Python25, which contains python.exe, is in the PATH.  If it's
> really looking for an executable named "python" along PATH, that can
> never succeed on Windows, since (I think) only files ending in .exe,
> .bat, and .cmd are executable there.
>
> > You can
> > change this by calling Py_SetProgramName(filename) before calling
> > Py_Initialize(). This is documented in API reference manual:
>
> Uncomment that line in my program above and you'll see it makes
> absolutely no difference.
>
> >http://docs.python.org/api/embedding.html
>
> > That page also describes a few hooks that you can overwrite to
> > modify the initial search path. They are described in more detail
> > on this page:
>
> >http://docs.python.org/api/initialization.html
>
> The only thing mentioned there that seems to have any effect at all is
>
>   set PYTHONHOME=C:\Python25
>
> and even then, it only eliminates the first line of the error, which
> then reads:
>
>   Traceback (most recent call last):
> File "", line 1, in 
>   ImportError: __import__ not found
>
> My only known workaround is to set PYTHONPATH.  This just doesn't seem
> right; isn't anyone doing embedding under Windows?

The problem with not finding site.py on Windows will often come up
when you have gone through a cycle of installing/removing different
versions of Python and the registry entries get mucked. Also can be a
problem where Python was not installed as administrator originally and
now trying to run stuff as user different to what Python was installed
as. A lot of the time problems go away by deinstalling Python and
reinstalling as administrator. Also watch out for where you have some
third party application which provides its own version of Python and
that version or its DLL is being found first in your PATH.

Graham


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mod_python, ElementTree and Aapche 2.0

[Graham Dumpleton]

On Oct 28, 3:02 am, Rajarshi <[EMAIL PROTECTED]> wrote:
> Hi, this is a slightly vague question but I'm really puzzled as to
> when I write a mod_python (3.1.3) program that makes use of
> ElementTree and call it via a URL, the program simply stops when I do
> something like
>
> s = # some XML document in a string
> root = XML(s)
>
> There is no exception at all - the browser just shows a blank page
>
> But running the code outside of the web server makes it run fine.
>
> This is on a RHEL 4 machine, with Apache 2.0.52 and Python 2.3.4
>
> Has anybody ever seen this type of behavior?

Try reading:

  http://www.dscpl.com.au/wiki/ModPython/Articles/ExpatCausingApacheCrash

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mod_python and Content-Type

[Graham Dumpleton]

Paul Rudin wrote:
> I'm have a little experiment with mod_python. I'm trying to figure out
> how to get hold of the original Content-Type header.
>
> In my config file I have:
>
> 
>   AddHandler mod_python .py
>   PythonHandler atomserv
>   PythonDebug On
>   PythonAutoReload On
> 
>
> The file atomserv.py in that directory has a function called handler
> that gets called as I expect, but on entry to that function
> req.content_type is "text/x-python" whereas my test client sent a
> different Content-Type header.
>
> So, how can I either persuade the apache/mod_python machinery to
> preserve the original content_type, or else how can I get hold of it?

All headers which come from the client are available through the
'headers_in'
attribute of the request object. Eg.

  def handler(req):
ct = req.headers_in.get('Content-Type')
...

The 'content_type' attribute is set by Apache based on its
determination of
what the content type of the response will be based on looking at the
extension type used on the matched resource specified by the URL.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: xml.dom.minidom.parseString segmentation fault on mod_python

[Graham Dumpleton]

On Jan 26, 10:00 pm, "Ziga Seilnacht" <[EMAIL PROTECTED]>
wrote:
> On Jan 26, 10:41 am, [EMAIL PROTECTED] wrote:
>
> > Python 2.4.4
> >mod_python3.2.10  +  Apache 2.0
>
> > def index( req, **params ):
> > from xml.dom.minidom import parseString
> > doc = parseString( "whatever" )
>
> > => blank screen, _no_any_exception_; Apache error_log:
> > [Fri Jan 26 10:18:48 2007] [notice] child pid 17596 exit signal
> > Segmentation fault (11)
>
> > Outsidemod_pythoncode works well. Any ideas? I would be 
> > grateful.http://www.python.org/sf/1558223http://www.python.org/sf/1295808http://www.python.org/sf/1075984
>
> Try to compile all your dependencies against the same version of
> Expat or upgrade to python 2.5.

For further information read:


http://www.dscpl.com.au/wiki/ModPython/Articles/ExpatCausingApacheCrash

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python does not play well with others

[Graham Dumpleton]

On Feb 4, 1:05 pm, Paul Rubin  wrote:
> "Paul Boddie" <[EMAIL PROTECTED]> writes:
>> Probably the biggest inhibitor, as far as I can see, has been the
>> server technology chosen. Many hosting providers have historically
>> offered no better than CGI for Python, whilst PHP runs within Apache
>> itself, and it has previously been stated that mod_python has been
>> undesirable with regard to isolating processes from each other.
>> Consequently, a number of Python people seem to have held out for
>> other "high performance" solutions, which various companies now offer.
>
> Your point that shared hosting with Python isn't so easy because of
> insufficient isolation between apps is valid.  Maybe Python 3.0 can do
> something about that and it seems like a valid thing to consider while
> fleshing out the 3.0 design.

To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.

First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process. These
distinct sub interpreters can be linked to different parts of the URL
namespace. This means that it is possible to host more than one
mod_python application where each executes within in their own
distinct sub interpreter. The outcome of this is that each application
can have their own sys.path, own os.environ, own sets of modules and
potentially with different versions of some module.

Maintaining separation using sub interpreters eliminates the bulk of
problems with applications interfering which each other at least
within a process. Some problems can still arise though where third
party extension modules for Python aren't written so as to be usable
from multiple sub interpreters at the same time however. This is not a
failing of mod_python though, but a failing of the module writers.

The main area where interference can occur is where applications needs
to write to the file system. This is because all code will be
executing as the user that Apache runs as. Thus, distinct applications
could overwrite each others data within the file system. On one level
this just means that applications need to be configured to always use
their own part of the file system. For example, if using the support
in mod_python for sessions, the distinct applications should perhaps
use separate session databases rather than use the same common
database. Ultimately though, a rogue application could write where
ever it wants to, but from what I know (and could be wrong), this
isn't different to other languages systems within Apache such as PHP
and mod_perl.

Another possibility for interference is where an application simply
does something bad like get stuck in a tight loop or consume lots of
memory. Such an event can possibly interfere with other applications,
even across language boundaries, however, how bad the impact will be
depend on what MPM is used by Apache.

If "prefork" MPM is used, then the request being handled by that
application is the only thing which would be running within that child
process at that particular time. Thus, if it stops the functioning of
just that one process it doesn't matter as Apache will just farm
requests off to other child processes. If that initial child process
crashes because of the problem, then again it doesn't matter as Apache
will just create another child process to replace it and will
otherwise keep running.

If the "worker" MPM is used the impact can be greater as there could
be other requests being handled concurrently within the same child
process and the performance of those requests may be hindered. If the
worst happens and the child process crashes, only other requests being
handled within that specific child process would be affected, those in
other child processes would again continue unaffected as would Apache
as a whole.

The worst case is the "winnt" MPM (ie., Windows boxes). This is
because there is only one Apache process and thus a rogue application
can affect the whole Apache web server.

It should be highlighted though that this class of problem is not
unique to Python or mod_python as you could get rogue code in a PHP
page or mod_perl application just as easily.

What it all really comes down to is that the only features that are
really missing are the ability for distinct applications to run as
distinct users, or for applications to run inside of some sort of
chroot environment. Some aspects of this are addressed by FCGI and
SCGI, but again lack of this feature within mod_python itself is not
unique to it and that as far as I know is also going to be an issue
for other language systems for Apache such as PHP or mod_perl.

Having said all that, perhaps those who are complaining about lack of
support for specific features in mod_python can now clarify what
actually you are talking about. At the moment the brief comments being
made seem to possi

Re: Python does not play well with others

[Graham Dumpleton]

On Feb 5, 9:45 am, John Nagle <[EMAIL PROTECTED]> wrote:
> Graham Dumpleton wrote:
> > On Feb 4, 1:05 pm, Paul Rubin <http://[EMAIL PROTECTED]> wrote:
>
> >>"Paul Boddie" <[EMAIL PROTECTED]> writes:
>
> >>>Probably the biggest inhibitor, as far as I can see, has been the
> >>>server technology chosen. Many hosting providers have historically
> >>>offered no better than CGI for Python, whilst PHP runs within Apache
> >>>itself, and it has previously been stated that mod_python has been
> >>>undesirable with regard to isolating processes from each other.
> >>>Consequently, a number of Python people seem to have held out for
> >>>other "high performance" solutions, which various companies now offer.
>
> >>Your point that shared hosting with Python isn't so easy because of
> >>insufficient isolation between apps is valid.  Maybe Python 3.0 can do
> >>something about that and it seems like a valid thing to consider while
> >>fleshing out the 3.0 design.
>
> > To clarify some points about mod_python, since these posts do not
> > properly explain the reality of the situation and I feel people are
> > getting the wrong impression.
>
> > First off, when using mod_python it is possible to have it create
> > multiple sub interpreters within each Apache child process.
>
>  Realistically, mod_python is a dead end for large servers,
> because Python isn't really multi-threaded.  The Global Python
> Lock means that a multi-core CPU won't help performance.

That is not true if 'prefork' MPM is used for Apache which is how most
people seem to run it. This is because each Apache child process only
run one request at a time and so there isn't normally going to be any
contention on the GIL at all. The only case where there would be
contention in this arrangement is if the request handlers within
Apache had spawned off distinct threads themselves to do stuff. Even
then, in this arrangement the main request handler is usually not
doing anything and is just waiting for the created thread to finish
what it was doing. Thus if only one thread was spawned to do some work
or a blocking operation to allow the main thread to timeout, then
again there isn't really any contention as only one thread is actually
doing anything. If you are trying to embed very intensive operations
with threads within Apache then I would suggest it is not really the
best design you could use anyway as such things would be much better
farmed off to a long running backend process using XML-RPC or some
other interprocess communication mechanism.

If one is using the "worker" MPM then yes there will be some
contention if multiple requests are being handled by mod_python at the
same time within the same Apache child process. The downside of this
is lessened however by the fact that there are still multiple Apache
child processes and Apache will spread requests across all the Apache
child processes, thus the amount that may be running concurrently
within any one process is less.

The basic problem of GIL contention here is no different to a single
Python backend process which is handling everything behind Apache. In
some respects the Apache approach actually works better as there are
multiple processes spreading the load. Your comment on the GIL is
therefore partly unjustified in that respect for Apache and
mod_python. Your statement in some respect still stands for Python
itself when run as a single process, but you linked it to mod_python
and Apache which lessens the impact through its architecture of using
multiple child processes.

Finally we have 'winnt' MPM, again, because this is all in the one
process you will have GIL contention in a much more substantial
manner. However, I'd suggest that most wouldn't choose Apache on
Windows as a major deployment platform.

>  FastCGI, though, can get all the CPUs going.  It takes more
> memory, though, since each instance has a full copy of Python
> and all the libraries in use.

How is that any different to Apache child processes. Each Apache child
process has a full copy of Python and the libraries in use. Each
Apache child process can be making use of different CPUs. Further,
static file requests, plus other requests against PHP, mod_perl etc
can when mod_python is also running be on separate CPUs within the
same child process when 'worker' MPM is being used. Thus you haven't
lost all forms or parallelism that may be possible, it is only within
the mod_python world that there will be some GIL contention and only
with 'worker' and 'winnt' MPMs, not 'prefork'. It isn't going to lock
out non Python stuff from making use of additional CPUs.

>  (Fas

Re: Python does not play well with others

[Graham Dumpleton]

On Feb 6, 4:52 am, John Nagle <[EMAIL PROTECTED]> wrote:
> [EMAIL PROTECTED] wrote:
> > John Nagle wrote:
>
> >>Graham Dumpleton wrote:
>
> >>>On Feb 4, 1:05 pm, Paul Rubin <http://[EMAIL PROTECTED]> wrote:
>
> >>>>"Paul Boddie" <[EMAIL PROTECTED]> writes:
> >> Realistically, mod_python is a dead end for large servers,
> >>because Python isn't really multi-threaded.  The Global Python
> >>Lock means that a multi-core CPU won't help performance.
>
> > The GIL doesn't affect seperate processes, and any large server that
> > cares about stability is going to be running a pre-forking MPM no
> > matter what language they're supporting.
>
>Pre-forking doesn't reduce load; it just improves responsiveness.
> You still pay for loading all the modules on every request.  For
> many AJAX apps, the loading cost tends to dominate the transaction.
>
>FastCGI, though, reuses the loaded Python (or whatever) image.
> The code has to be written to be able to process transactions
> in sequence (i.e. don't rely on variables intitialized at load),
> but each process lives for more than one transaction cycle.
> However, each process has a copy of the whole Python system,
> although maybe some code gets shared.

As someone else pointed out, your understanding of how mod_python
works within Apache is somewhat wrong. I'll explain some things a bit
further to make it clearer for you.

When the main Apache process (parent) is started it will load all the
various Apache modules including that for mod_python. Each of these
modules has the opportunity to hook into various configuration phases
to perform actions. In the case of mod_python it will hook into the
post config phase and initialise Python which will in turn setup all
the builtin Python modules.

When Apache forks off child processes each of those child processes
will inherit Python already in an initialised state and also the
initial Python interpreter instance which was created, This therefore
avoids the need to perform initialisation of Python every time that a
child process is created.

In general this initial Python interpreter instance isn't actually
used though, as the default strategy of mod_python is to allocate
distinct Python interpreter instances for each VirtualHost, thereby at
least keeping applications running in distinct VirtualHost containers
to be separate so they don't interfere with each other.

Yes, these per VirtualHost interpreter instances will only be created
on demand in the child process when a request arrives which
necessitates it be created and so there is some first time setup for
that specific interpreter instance at that point, but the main Python
initialisation has already occurred so this is minor. Most
importantly, once that interpreter instance is created for the
specific VirtualHost in the child process it is retained in memory and
used from one request to the next. If the handler for a request loads
in Python modules, those Python modules are retained in memory and do
not have to be reloaded on each request as you believe.

If you are concerned about the fact that you don't specifically know
when an interpreter instance will be first created in the child
process, ie., because it would only be created upon the first request
arriving that actually required it, you can force interpreter
instances to be created as soon as the child process has been created
by using the PythonImport directive. What this directive allows you to
do is specify a Python module that should be preloaded into a specific
interpreter instance as soon as the child process is created. Because
the interpreter will need to exist, it will first be created before
the module is loaded thereby achieving the effect of precreating the
specific named interpreter instance.

So as to make sure you don't think that that first interpreter
instance created in the parent and inherited by the child processes is
completely wasted, it should be pointed out that the first interpreter
instance created by Python is sort of special. In general it shouldn't
matter, but there is one case where it does. This is where a third
party extension module for Python has not been written so as to work
properly in a context where there are multiple sub interpreters.
Specifically, if a third party extension module used the simplified
API for GIL locking one can have problems using that module in
anything but the first interpreter instance created by Python. Thus,
the first instance is retained and in some cases it may be necessary
to force your application to run within the context of that
interpreter instance to get it to work where using such a module. If
you have to do this for multiple applications running under different
VirtualHost containers you loose your separation 

Re: Python does not play well with others

[Graham Dumpleton]

On Feb 6, 5:39 am, Paul Rubin  wrote:
> John Nagle <[EMAIL PROTECTED]> writes:
> > > The GIL doesn't affect seperate processes, and any large server that
> > > cares about stability is going to be running a pre-forking MPM no
> > > matter what language they're supporting.
>
> >Pre-forking doesn't reduce load; it just improves responsiveness.
> > You still pay for loading all the modules on every request.  For
> > many AJAX apps, the loading cost tends to dominate the transaction.
>
> I think the idea is that each pre-forked subprocess has its own
> mod_python that services multiple requests serially.  

And where 'worker' MPM is used, each child process can be handling
multiple concurrent requests at the same time. Similarly on Windows
although there is only one process.

> New to me is the idea that you can have multiple separate Python
> interpreters in a SINGLE process (mentioned in another post).  I'd
> thought that being limited to one interpreter per process was a
> significant and hard-to-fix limitation of the current CPython
> implementation that's unlikely to be fixed earlier than 3.0.

No such limitation exists with mod_python as it does all the
interpreter creation and management at the Python C API level. The one
interpreter per process limitation is only when using the standard
'python' runtime executable and you are doing everything in Python
code.

Graham


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python does not play well with others

[Graham Dumpleton]

On Feb 6, 8:57 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
> On Feb 5, 12:52 pm, John Nagle <[EMAIL PROTECTED]> wrote:
>
>
>
> > [EMAIL PROTECTED] wrote:
> > > John Nagle wrote:
>
> > >>Graham Dumpleton wrote:
>
> > >>>On Feb 4, 1:05 pm, Paul Rubin <http://[EMAIL PROTECTED]> wrote:
>
> > >>>>"Paul Boddie" <[EMAIL PROTECTED]> writes:
> > >> Realistically,mod_pythonis a dead end for large servers,
> > >>because Python isn't really multi-threaded.  The Global Python
> > >>Lock means that a multi-core CPU won't help performance.
>
> > > The GIL doesn't affect seperate processes, and any large server that
> > > cares about stability is going to be running a pre-forking MPM no
> > > matter what language they're supporting.
>
> >Pre-forking doesn't reduce load; it just improves responsiveness.
> > You still pay for loading all the modules on every request.
>
> No, you don't.  Each server is persistent and serves many requests--
> it's not at all like CGI, and it reuses the loaded Python image.
>
> So if you have, say, an expensive to load Python module, that will
> only be executed once for each server you start...e.g. if you have
> Apache configured to accept up to 50 connections, the module will be
> run at most 50 times; once each of the 50 processes has started up,
> they stick around until you restart Apache, unless you've configured
> apache to only serve X requests in one process before restarting it.
> (The one major feature thatmod_python_is_ missing is the ability to
> do some setup in the Python module prior to forking.  That would make
> restarting Apache somewhat nicer).

There would be a few issues with preloading modules before the main
Apache child process performed the fork.

The first is whether it would be possible for code to be run with
elevated privileges given that the main Apache process usually is
started as root. I'm not sure at what point it switches to the special
user Apache generally runs as and whether in the main process the way
this switch is done is enough to prevent code getting back root
privileges in some way, so would need to be looked into.

The second issue is that there can be multiple Python interpreters
ultimately created depending on how URLs are mapped, thus it isn't
just an issue with loading a module once, you would need to create all
the interpreters you think might need it and preload it into each. All
this will blow out the memory size of the main Apache process.

There is also much more possibility for code, if it runs up extra
threads, to interfere with the operation of the Apache parent process.
One particular area which could be a problem is where Apache wants to
do a restart, as it will attempt to unload the mod_python module and
reload it. Right now this may not be an issue as mod_python does the
wrong thing and doesn't shutdown Python allowing it to be
reinitialised when mod_python is reloaded, but in mod_wsgi (when
mod_python isn't also being loaded), it will shutdown Python. If there
is user code executing in a thread within the parent process this may
actually stop mod_wsgi from cleanly shutting down Python thus causing
Apache to hang.

All up, the risks of loading extra modules in the parent process
aren't worth it and could just result in things being less stable.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python does not play well with others

[Graham Dumpleton]

On Feb 6, 9:15 am, Paul Rubin <http://[EMAIL PROTECTED]> wrote:
> "Graham Dumpleton" <[EMAIL PROTECTED]> writes:
> > Yes, these per VirtualHost interpreter instances will only be created
> > on demand in the child process when a request arrives which
> > necessitates it be created and so there is some first time setup for
> > that specific interpreter instance at that point, but the main Python
> > initialisation has already occurred so this is minor.
>
> Well ok, but what if each of those interpreters wants to load, say,
> the cookie module?  Do you have separate copies of the cookie module
> in each interpreter?  Does each one take the overhead of loading the
> cookie module?

Each interpreter instance will have its own copy of any Python based
code modules. You can't avoid this as Python code is so modifiable
that they have to be separate else you would be modifying the same
instance as used by a different interpreter which could screw up the
other applications view of the world. The whole point of having
separate interpreters is to avoid applications trampling on each
other. If you really are concerned about multiple loading, use the
PythonInterpreter directive to specifically say that applications
running under different VirtualHost containers should use the same
interpreter.

Note though, that although you can run multiple applications in one
interpreter in many cases, it may not be able to be done in others.
For example, it is not possible to run two instances of Django within
the one interpreter instance. The first reason as to why this can't be
done is that Django expects certain information about its
configuration to come from os.environ. Since there is only one
os.environ it can't have two different values for each application at
the same time. Some may argue that in 'prefork' you could just change
os.environ to be correct for the application for the current request
and this effectively is what the mod_python adapter for Django does,
but this will fail when 'worker' MPM or Windows is used. I suspect
this is the where the idea that Django can't be run on 'worker' MPM
came from. Although the documentation for Django suggests it is a
mod_python problem, it is actually a Django problem. This use of
os.environ by Django also means that Django isn't a well behaved WSGI
application component. :-(

> It would be neat if there was a way of including
> frequently used modules in the shared text segment of the
> interpreters, as created during the initial build process.  GNU Emacs
> used to do something like that with a contraption called "unexec" (it
> could dump out parts of its data segment into a pure (shared)
> executable that you could then run without the overhead of loading all
> those modules) but the capability went away as computers got faster
> and it became less common to have a lot of Emacs instances weighing
> down timesharing systems.  Maybe it's time for a revival of those
> techniques.

I don't see it as being applicable. Do note that provided there are
precompiled byte code files for .py files then load time is at least
reduced because Python doesn't have to recompile the code. This
actually can be quite significant.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python does not play well with others

[Graham Dumpleton]

On Feb 6, 10:15 am, Paul Rubin <http://[EMAIL PROTECTED]> wrote:
> "Graham Dumpleton" <[EMAIL PROTECTED]> writes:
> > There is also much more possibility for code, if it runs up extra
> > threads, to interfere with the operation of the Apache parent process.
>
> Certainly launching any new threads should be postponed til after the
> fork.  

Except that you can't outright prevent it from being done as a Python
module could create the threads as a side effect of the module import
itself. I guess though if you load a module which does that and it
screws things up, then you have brought it on yourself as it would
have been your choice to make mod_python load it in the first place if
the feature was there. :-)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python web frameworks

[Graham Dumpleton]

On Nov 21, 2:33 am, Istvan Albert <[EMAIL PROTECTED]> wrote:
> On Nov 20, 9:42 am, "Diez B. Roggisch" <[EMAIL PROTECTED]> wrote:
>
> > > 12/7. Django comes with its own little server so that you don't have
> > > to set up Apache on your desktop to play with it.
>
> > I was rather shocked to learn that django only has this tiny server and does
> > not come with a stand-alone server
>
> Alas it is even worse than that, the development server is single
> threaded and that can be a big problem when developing sites that make
> multiple simultaneous requests at the same time (say if one of those
> requests stalls for some reason). It is trivial to add multi threading
> via a mixin (takes about two lines of code) but it does not seem to be
> a priority to do so.
>
> For large traffic though, Django is better than just about anything
> other framework because it is built as multiprocess framework through
> mod_python (rather than threaded).

This only holds if actually hosted on Apache. As Django these days
supports WSGI interface there is nothing to stop it being run with
other hosting solutions that support WSGI. So, you could host it under
paster or CherryPy WSGI servers. You could even run it under CGI if
you were really desperate using a CGI-WSGI adapter. So, it isn't
strictly correct to say it is as a multiprocess framework specifically
for mod_python, although the developers will admit in the first
instance that they didn't design the internals with multithreading in
mind. That said, there aren't believed to be any multithreading issues
in Django itself at this time.

> So there is no global interpreter
> lock, thread switching etc.

Use of worker MPM (multiprocess+multithreaded) in Apache in place of
prefork MPM (multiprocess+single threaded) is also still more than
acceptable a solution to hosting Python web applications using either
mod_python or mod_wsgi.

People keep pushing this barrow about the GIL and multithreading being
a huge problem, when in the context of Apache it is isn't, at least
not to the degree people make out. The reason for this is that when
using worker MPM it sill acts as a multi process web server even
though each process is also multithreaded. Within those worker MPM
child processes there is also a lot going on that doesn't involve
Python code nor the GIL, for example initial request process and
serving up of static files etc.

Result is that the Python GIL is no impediment when using Apache on
UNIX to making good use of multiple processors or cores, even when
Apache worker MPM is used.

For where I have talked about this before see:
 http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python web frameworks

[Graham Dumpleton]

On Nov 21, 1:37 pm, BartlebyScrivener <[EMAIL PROTECTED]> wrote:
> On Nov 20, 3:39 pm, Graham Dumpleton <[EMAIL PROTECTED]>
> wrote:
>
> > This only holds if actually hosted on Apache. As Django these days
> > supports WSGI interface there is nothing to stop it being run with
> > other hosting solutions that support WSGI. So, you could host it under
> > paster or CherryPy WSGI servers. You could even run it under CGI if
> > you were really desperate using a CGI-WSGI adapter. So, it isn't
> > strictly correct to say it is as a multiprocess framework specifically
> > for mod_python, although the developers will admit in the first
> > instance that they didn't design the internals with multithreading in
> > mind. That said, there aren't believed to be any multithreading issues
> > in Django itself at this time.
>
> > People keep pushing this barrow about the GIL and multithreading being
> > a huge problem, when in the context of Apache it is isn't, at least
> > not to the degree people make out. The reason for this is that when
> > using worker MPM it sill acts as a multi process web server even
> > though each process is also multithreaded. Within those worker MPM
> > child processes there is also a lot going on that doesn't involve
> > Python code nor the GIL, for example initial request process and
> > serving up of static files etc.
>
> > Result is that the Python GIL is no impediment when using Apache on
> > UNIX to making good use of multiple processors or cores, even when
> > Apache worker MPM is used.
>
> I understand about a fifth of this exchange but I'm glad it's here so
> I can follow links and search on the terminology. I couldn't tell from
> earlier posts if mod_python was good or bad.

Version 3.3 of mod_python fixed up a lot of issues that existed with
older versions of mod_python. There are still a lot of issues in
mod_python unfixed.

  https://issues.apache.org/jira/browse/MODPYTHON

In the main people will not run into these issues, of if they do, the
incidence of them causing a direct or significant impact is low, or
with people just tolerating the problems.

If you want to be where hosting with Apache is heading, then look at
mod_wsgi (http://www.modwsgi.org) instead. People will say I am biased
because I wrote it, but I was also the main person who did the more
recent work on fixing up mod_python and am more aware than others of
what problems still exist in mod_python.

To be frank, unless some white knight comes along and dives into
mod_python and fixes up the remaining issues, then you probably will
not see any significant future updates to mod_python and it will just
stagnate. I certainly will not be devoting much time to mod_python any
more.

Part of the problem with mod_python is that the code base has grown
over time and is long overdue for a complete rethink, which is in part
what mod_wsgi was about, ie., making the code and configuration a lot
simpler and safer for use in web hosting environments.

Thus mod_wsgi takes aspects of what mod_python does, combining it with
aspects of how FASTCGI solutions work. This gives the option of
embedding a Python application in Apache for maximum speed, or using
daemon processes as means of being able to better separate multiple
applications.

Most importantly, mod_wsgi supports WSGI directly, making it
reasonably trivial to run any Python web framework or application
which supports the WSGI standard.

> The Django book says: "Apache with mod_python currently is the most
> robust setup for using Django on a production server."
>
> Is that true?

I would say that that is now debatable. Overall mod_wsgi is probably a
better package in terms of what it has to offer. Only thing against
mod_wsgi at this point is peoples willingness to accept something that
is new in conjunction with Linux distributions and web hosting
companies being slow to adopt new packages.

Various people are quite happily using mod_wsgi. Users of mod_wsgi
range from people trying to run it in memory constrained VPS systems,
right up to major sites serving up to between 3-4 million hits a day.

There have been a few odd things come up since the initial release
which have since been fixed, but the core is showing itself to be very
solid.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python web frameworks

[Graham Dumpleton]

On Nov 23, 4:00 am, Istvan Albert <[EMAIL PROTECTED]> wrote:
> On Nov 21, 12:15 am, Graham Dumpleton <[EMAIL PROTECTED]>
> wrote:
>
> > I would say that that is now debatable. Overall mod_wsgi is probably a
> > better package in terms of what it has to offer. Only thing against
> > mod_wsgi at this point is peoples willingness to accept something that
> > is new in conjunction with Linux distributions and web hosting
> > companies being slow to adopt new packages.
>
> Yes that is to be expected, many people want someone else to pay the
> early adopter's costs. Nonetheless mod_wsgi seems like the right
> direction to move the python world.
>
> One confounding factor that may slow its adoption could be the need of
> running plain old CGI in an efficient way. I'm not sure how that fits
> into the WSGI picture.

Do note that using mod_wsgi doesn't preclude you still running plain
old CGI scripts using mod_cgi or mod_cgid. As to making it more
efficient, one can go two paths on this.

The preferred path would be to put in the effort to convert the Python
CGI application to WSGI. If one still needs to run it as CGI with
other hosting solutions, then use a CGI-WSGI adapter.

Second, albeit a bit of a kludge, just like mod_python.cgihandler is a
kludge, is to emulate the CGI environment on top of WSGI. This would
work if using single threaded Apache prefork MPM, or using mod_wsgi
daemon mode with multiple processes but where each is single threaded.
It wouldn't be practical though to do it for multithread Apache worker
MPM, or multithreaded daemon processes with mod_wsgi daemon mode.
Because of how Python leaks environment variables between sub
interpreters, you would also only want to be running one sub
interpreter within a process. This would be fine if using mod_wsgi
daemon mode as different CGI scripts could be delegated to run in
different daemon processes as necessary to keep them separate, but may
not be practical if using embedded mode if hosting a range of other
WSGI applications at the same time in embedded mode.

So, it is doable, but effort would be better off expended at least
converting the Python CGI script to WSGI instead. It would save a lot
of trouble in the long run, especially with CGI scripts which weren't
designed to be run multiple times in same memory context, ie., where
they don't clean up after themselves.

If someone really wanted to host an existing CGI script under WSGI
where the same process was used over and over, and had good reasons
for doing so, it wouldn't be hard for me to come up with a workable
adapter that allowed it.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Running unmodified CGI scripts persistently under mod_wsgi.

[Graham Dumpleton]

On Nov 23, 8:49 am, Graham Dumpleton <[EMAIL PROTECTED]>
wrote:
> On Nov 23, 4:00 am, Istvan Albert <[EMAIL PROTECTED]> wrote:
>
>
>
> > On Nov 21, 12:15 am, Graham Dumpleton <[EMAIL PROTECTED]>
> > wrote:
>
> > > I would say that that is now debatable. Overall mod_wsgi is probably a
> > > better package in terms of what it has to offer. Only thing against
> > > mod_wsgi at this point is peoples willingness to accept something that
> > > is new in conjunction with Linux distributions and web hosting
> > > companies being slow to adopt new packages.
>
> > Yes that is to be expected, many people want someone else to pay the
> > early adopter's costs. Nonetheless mod_wsgi seems like the right
> > direction to move the python world.
>
> > One confounding factor that may slow its adoption could be the need of
> > running plain old CGI in an efficient way. I'm not sure how that fits
> > into the WSGI picture.
>
> Do note that using mod_wsgi doesn't preclude you still running plain
> old CGI scripts using mod_cgi or mod_cgid. As to making it more
> efficient, one can go two paths on this.
>
> The preferred path would be to put in the effort to convert the Python
> CGI application to WSGI. If one still needs to run it as CGI with
> other hosting solutions, then use a CGI-WSGI adapter.
>
> Second, albeit a bit of a kludge, just like mod_python.cgihandler is a
> kludge, is to emulate the CGI environment on top of WSGI. This would
> work if using single threaded Apache prefork MPM, or using mod_wsgi
> daemon mode with multiple processes but where each is single threaded.
> It wouldn't be practical though to do it for multithread Apache worker
> MPM, or multithreaded daemon processes with mod_wsgi daemon mode.
> Because of how Python leaks environment variables between sub
> interpreters, you would also only want to be running one sub
> interpreter within a process. This would be fine if using mod_wsgi
> daemon mode as different CGI scripts could be delegated to run in
> different daemon processes as necessary to keep them separate, but may
> not be practical if using embedded mode if hosting a range of other
> WSGI applications at the same time in embedded mode.
>
> So, it is doable, but effort would be better off expended at least
> converting the Python CGI script to WSGI instead. It would save a lot
> of trouble in the long run, especially with CGI scripts which weren't
> designed to be run multiple times in same memory context, ie., where
> they don't clean up after themselves.
>
> If someone really wanted to host an existing CGI script under WSGI
> where the same process was used over and over, and had good reasons
> for doing so, it wouldn't be hard for me to come up with a workable
> adapter that allowed it.

I had a play with this and got an initial solution working. It allows
mod_wsgi to be pointed at a directory of existing unmodified Python
CGI scripts and it will run them, maintaining the code for the CGI
script in memory between requests. One doesn't even have to rename
scripts to have a .py extension or follow Python module naming
conventions like mod_python required with its CGI handler.

Now, where as CGI script would run say at 10 requests per second for a
simple hello world program, using mod_wsgi to manage the CGI scripts
one could achieve 450+ requests per second with embedded mode of
mod_wsgi, and 250+ requests per second with daemon mode. Not as quick
as what can be achieved for an equivalent WSGI specific version of the
script, which runs at 900+ requests per second for embedded mode and
500+ requests for daemon mode, but still a lot better than a plain old
CGI script which requires a new process for each request.

One would still need to see how it works for more complicated Python
CGI scripts. Also may need to be tweaked to allow one to configure its
behaviour to suit the CGI script. For example, should additional
modules imported by the script be deleted out of sys.modules when done
or preserved for next request. Similarly, should the globals and/or
locals of the script be preserved between requests. Whether one would
need to adjust such behaviour would depend on the nature of the
specific CGI script.

The other question is whether there is even a demand for this. Do
people want to be able to take unmodified Python CGI scripts and try
to run them persistently in this way, or would they be better off
converting them to proper WSGI applications.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Different kinds of Import Errors

[Graham Dumpleton]

On Nov 28, 12:35 am, Thomas Guettler <[EMAIL PROTECTED]> wrote:
> If you look at this code, you see there are two kind of ImportErrors:
>
> 1. app_name has no attribute or file managment.py: That's OK.
> 2. managment.py exists, but raises an ImportError: That's not OK: reraise
>
> # Import the 'management' module within each installed app, to 
> register
> # dispatcher events.
> for app_name in settings.INSTALLED_APPS:
> try:
> __import__(app_name + '.management', {}, {}, [''])
> except ImportError, exc:
> if exc.args[0]!='No module named management':
> raise
>
> I am searching a better solution, since in a future version of python
> the string 'No module namend management' might be changed.
>
> Any better solution?

Perhaps check for the presence of the module in sys.modules.

  if (app_name + '.management') in sys.modules:
raise

If not there, module couldn't be found. If module there but exception
occurred then some other import related error occurred.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Running unmodified CGI scripts persistently under mod_wsgi.

[Graham Dumpleton]

On Nov 29, 2:36 am, Thomas Guettler <[EMAIL PROTECTED]> wrote:
> Istvan Albert schrieb:
>
> > It will be awesome ifmod_wsgican run CGI without invoking python on
> > each access.
>
> For SCGI there is something like this: cgi2scgi: it is small executable 
> written in C,
> which connects to a running SCGI server.
>
> Executing this small binary on every request is no big overhead.
>
> Nevertheless I never used it.

That isn't the same as what is being talked about here with unmodified
CGI scripts running under mod_wsgi. That program you are talking about
is more in line with what Ian was talking about, but would still
require the CGI script to be converted to work with SCGI directly or
WSGI and then use a SCGI/WSGI adapter.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Different kinds of Import Errors

[Graham Dumpleton]

On Dec 1, 12:24 am, Thomas Guettler <[EMAIL PROTECTED]> wrote:
> Sorry, but this does not work. If there is an ImportError
> during importing the existing module, it won't get inserted
> into sys.modules. I just tried it with a small example.
>
> An other solution would be to inspect the traceback. If the
> app_name+'.management' is in it, it exists.
>
> Graham Dumpleton schrieb:
>
> > On Nov 28, 12:35 am, Thomas Guettler <[EMAIL PROTECTED]> wrote:
> >> If you look at this code, you see there are two kind of ImportErrors:
>
> >> 1. app_name has no attribute or file managment.py: That's OK.
> >> 2. managment.py exists, but raises an ImportError: That's not OK: reraise
>
> >> # Import the 'management' module within each installed app, to 
> >> register
> >> # dispatcher events.
> >> for app_name in settings.INSTALLED_APPS:
> >> try:
> >> __import__(app_name + '.management', {}, {}, [''])
> >> except ImportError, exc:
> >> if exc.args[0]!='No module named management':
> >> raise
>
> >> I am searching a better solution, since in a future version of python
> >> the string 'No module namend management' might be changed.
>
> >> Any better solution?
>
> > Perhaps check for the presence of the module in sys.modules.
>
> >   if (app_name + '.management') in sys.modules:
> > raise
>
> > If not there, module couldn't be found. If module there but exception
> > occurred then some other import related error occurred.

What example did you use to test it? What version of Python are you
using?

Curious, as at least for Python 2.3 it behaves as I described with the
simple examples I tested. I do vaguely remember behaviour related to
this changing though, so possible that newer version works
differently.

First try case of importing module that doesn't exist:

>>> import sys
>>> import z
Traceback (most recent call last):
  File "", line 1, in ?
ImportError: No module named z
>>> sys.modules['z']
Traceback (most recent call last):
  File "", line 1, in ?
KeyError: 'z'

Thus ImportError occurred, and not in sys.modules. Therefore module
must not have been able to be found.

Now lets try existing module, but non ImportError case first:

$ cat a.py
print 'a1'
xxx
print 'a2'

>>> import sys
>>> sys.modules['a']
Traceback (most recent call last):
  File "", line 1, in ?
KeyError: 'a'
>>> import a
a1
Traceback (most recent call last):
  File "", line 1, in ?
  File "a.py", line 2, in ?
xxx
NameError: name 'xxx' is not defined
>>> sys.modules['a']


Thus, although an exception occurred in importing file, the module is
still in sys.modules.

Now lets try for an ImportError for where target module exists:

$ cat a.py
print 'a1'
import z
print 'a2'

>>> import sys
>>> import a
a1
Traceback (most recent call last):
  File "", line 1, in ?
  File "a.py", line 2, in ?
import z
ImportError: No module named z
>>> sys.modules['a']


In this case ImportError occurred but is in sys.modules, thus
ImportError can only have been related to a sub module import as top
level module did actually exist.

I'll have to try this at work next week on newer version of Python
than my Mac has.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Mod python set cookie for all subdomains

[Graham Dumpleton]

On Dec 3, 11:44 am, Abandoned <[EMAIL PROTECTED]> wrote:
> Hi..
> I set cookie fromwww.domain.combut i'cant read this cookie from
> subdomain.domain.com
> How can i set cookie for all subdomains ?
>
> My set cookie code is:
> cookie = Cookie.Cookie('Login-Data', data)
> cookie.expires = time.time() + 992
> Cookie.add_cookie(req, cookie)

Having created the cookie do:

  cookie.domain = 'domain.com'

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weird embedding problem

[Graham Dumpleton]

On Dec 7, 5:01 pm, DavidM <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I'm embedding python in a C prog which is built as a linux shared lib.
>
> The prog is linked against libpython, and on startup, it calls
> Py_Initialize().
>
> The prog imports a pure-python script. The script starts up ok, but when
> it imports the 'math' module, the import fails with:
>
> Traceback (most recent call last):
>   File "/home/david/work/video/myprogs/dvedit/test/frei0rfx1.py", line 10, in 
> 
> import math
> ImportError: /usr/lib/python2.5/lib-dynload/math.so: undefined symbol: 
> PyExc_ValueError
> Failed to import math
>
> Any ideas of how to work around this?
>
> Please note - the program I'm writing *must* be built as a shared lib, so
> the usual practice of 'extend, don't embed' is just not an option here.
>
> Thoughts?

Are you actually linking your C program against the Python library?

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weird embedding problem

[Graham Dumpleton]

On Dec 7, 11:44 pm, DavidM <[EMAIL PROTECTED]> wrote:
> On Fri, 07 Dec 2007 00:53:15 -0800, Graham Dumpleton wrote:
> > Are you actually linking your C program against the Python library?
>
> Yes. Refer OP:
>
> >> I'm embedding python in a C prog which is built as a linux shared lib.
> >> The prog is linked against libpython, and on startup, it calls
> >> Py_Initialize().

What was the compiler link line you used for your program/shared
library then?

Have you confirmed using 'ldd' command that your program/shared
library do actually link against libpython?

Your original description is confusing. You say your program is built
as a shared library. Can you properly explain the full process you are
using to build your code and if you are producing a shared library for
your program how then is that shared library being used with an actual
executable? Be specific about where the program main() is located?

Some platforms don't necessarily find symbols from shared library 'A'
linked against a program executable if the main() is actually obtained
from a shared library 'B" linked to the same program and where the
symbols required from 'A' are only used in 'B'. What is required is to
link library 'A' against library 'B'.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Running unmodified CGI scripts persistently under mod_wsgi.

[Graham Dumpleton]

On Dec 9, 12:26 am, Michael Ströder <[EMAIL PROTECTED]> wrote:
> Jeffrey Froman wrote:
>
> > I'd still be interested in a mod_wsgi wrapper for 3rd-party CGI scripts.
>
> I doubt that this is possible, not because of the interface. But
> conventional CGI scripts are implemented with the assumption of being
> stateless. You would have to completely reinitialize them for each hit.
> Without knowledge about the internal CGI script processing this would
> mean reinitializing the whole Python run-time environment. So IMHO
> there's no real gain. Just my 2 c.

One doesn't necessarily need to reinitialise the whole Python run time
environment. The hack that mod_python uses is to remember what Python
modules had been loaded before first request. At the end of the
request it will delete from sys.modules anything that was added since
the beginning of the request. It also replaces os.environ in its
entirety at the start of each request as well.

Yes, it may not still work for all CGI scripts, eg., C extension
modules may be a problem, but may be adequate for many. In fact, for
some class of CGI scripts, deleting all those modules from sys.modules
may not be necessary, provided you at least cause the main CGI script
file to at least be reinterpreted/reimported. What degree of cleaning
out the environment could be a configurable parameter of the WSGI/CGI
bridge.

So, although it can be done, it is the need to use strange hacks like
this that means it may just be better to convert CGI script to work as
WSGI, as more guarantee of success.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Deploying embedded Python

[Graham Dumpleton]

On Dec 18, 11:07 am, Andreas Raab <[EMAIL PROTECTED]> wrote:
> Hi -
>
> I'm currently looking into a few deployment issues with our embedded
> Python interpreter and I'm looking for any information about deploying
> embedded Python that people may have. Specifically, I'm looking for the
> following information:
>
> 1) How to define a useful subset of the stdlib that can serve as an
> initial basis for the installation but later allows upgrade to the
> "full" library if desirable. In other words, I'd like to deploy a small
> subset of the stdlib to begin with (simply because of size constraints)
> which may later be extended to a full stdlib if this is desirable. Has
> someone done this before? I'd love to have a small "Python.zip"
> cross-platform stdlib surrogate that just gets shipped with the product.
> If not, what is the right starting point for analyzing the dependencies
> inside the stdlib?
>
> 2) How to isolate the embedded interpreter from environmental effects. I
> have found that on occasion, the interpreter would pick up "stray"
> installations which can cause weird problems. Which environmental
> settings affect the startup of an embedded Python interpreter?

PYTHONHOME environment variable, or if embedded in C application use
Py_SetPythonHome() before calling Py_Intialize(). This can be used to
ensure that specific Python installation is used as source of
configuration and modules.

> How does
> one work around/remove those dependencies? Is there any information
> available about how exactly the startup works?

Yes, the source code. :-)

> What is being read/loaded
> in which order etc?

Set PYTHONVERBOSE environment variable to have Python output a lot of
information about what it is doing at startup.

Graham

> 3) General advice about deploying embedded Python. Pointers to web
> sites, general experience (good or bad) etc. are all very welcome.
>
> Thanks,
>- Andreas

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: More than one interpreter per process?

[Graham Dumpleton]

On Dec 19, 2:37 am, sturlamolden <[EMAIL PROTECTED]> wrote:
> On 18 Des, 10:24, Roger Binns <[EMAIL PROTECTED]> wrote:
>
> > You can.  Have a look at mod_python andmod_wsgiwhich does exactly
> > this.  But extension modules that use the simplified GIL api don't work
> > with them (well, if at all).
>
> mod_python implements use Py_NewInterpreter() to create sub-
> interpreters. They all share the same GIL. The GIL is declared static
> in ceval.c, and shared for the whole process. But ok, if
> PyEval_AquireLock() would take a pointer to a 'sub-GIL', sub-
> interpreters could run concurrent on SMPs. But it would require a
> separate thread scheduler in each subprocess.

In current versions of Python it is possible for multiple sub
interpreters to access the same instance of a Python object which is
notionally independent of any particular interpreter. In other words,
sharing of objects exists between sub interpreters. If you remove the
global GIL and make it per sub interpreter then you would loose this
ability. This may have an impact of some third party C extension
modules, or in embedded systems, which are able to cache simple Python
data objects for use in multiple sub interpreters so that memory usage
is reduced.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: More than one interpreter per process?

[Graham Dumpleton]

On Dec 18, 8:24 pm, Roger Binns <[EMAIL PROTECTED]> wrote:
> sturlamolden wrote:
> > If one can have more than one interpreter in a single process,
>
> You can.  Have a look at mod_python andmod_wsgiwhich does exactly
> this.  But extension modules that use the simplified GIL api don't work
> with them (well, if at all).

When using mod_wsgi there is no problem with C extension modules which
use simplified GIL API provided that one configures mod_wsgi to
delegate that specific application to run in the context of the first
interpreter instance created by Python.

In theory the same should be the case for mod_python but there is
currently a bug in the way that mod_python works such that some C
extension modules using simplified API for the GIL still don't work
even when made to run in first interpreter.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: More than one interpreter per process?

[Graham Dumpleton]

On Dec 19, 3:07 pm, Roger Binns <[EMAIL PROTECTED]> wrote:
> Graham Dumpleton wrote:
> > When using mod_wsgi there is no problem with C extension modules which
> > use simplified GIL API provided that one configures mod_wsgi to
> > delegate that specific application to run in the context of the first
> > interpreter instance created by Python.
>
> Graham, I've asked you before but never quite got a straight answer.

Maybe because it isn't that simple. :-)

> What *exactly* should extension authors change their code to in order to
> be fully compatible?  For example how should the C function below be
> changed:
>
> void somefunc(void)
> {
>   PyGILState_STATE gilstate=PyGILState_Ensure();
>
>   abc();
>
>   Py_BEGIN_ALLOW_THREADS
>  def();
>   Py_END_ALLOW_THREADS
>
>   ghi();
>
>   PyGILState_Release(gilstate);
>
> }

What you do depends on what the overall C extension module does. It
isn't really possible to say how the above may need to be changed as
there is a great deal of context which is missing as far as knowing
how that function comes to be called. Presented with that function in
isolation I can only say that using simplified GIL API is probably the
only way of doing it and therefore can only be used safely against the
first interpreter created by Python.

If the direction of calling for a C extension module is always Python
code into C code and that is far as it goes, then none of this is an
issue as you only need to use Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS.

The problem case is where C code needs to callback into Python code
and you are not using simplified GIL API in order to be able to
support multiple sub interpreters. The easiest thing to do here is to
cache a thread state object for the interpreter instance when you
first obtained the handle for the object which allows you to interact
with C extension module internals. Later when a callback from C to
Python code needs to occur then you lookup the cached thread state
object and use that as the argument to PyEval_AcquireThread().

As example see:

  http://svn.dscpl.com.au/ose/trunk/software/python/opydispatch.cc

The thread state object is cached when handle to an instance is first
created. Any callbacks which are registered remember the interpreter
pointer and then that is used as key to lookup up the cached thread
state.

This code was done a long time ago. It is possible that it needs to be
revised based on what has been learnt about simplified GIL API.

This way of doing things will also not work where it is possible that
sub interpreters are created and then later destroyed prior to process
exit because of the fact that the interpreter pointer is cached. But
then, enough C extension modules have this problem that recycling sub
interpreters in a process isn't practical anyway.

Note that the indicated file uses a global cache. The agent.hh/
opyagent.cc files at that same location implement a more complicated
caching system based on individual objects.

The software which this is from is probably a more extreme example of
what is required, but your example was too simple to draw any
conclusions from.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Detecting memory leaks on apache, mod_python

[Graham Dumpleton]

On Dec 21, 7:42 pm, Ilias Lazaridis <[EMAIL PROTECTED]> wrote:
> On Dec 19, 5:40 am, Ilias Lazaridis <[EMAIL PROTECTED]> wrote:
>
>
>
> > On Dec 17, 8:41 am, Ilias Lazaridis <[EMAIL PROTECTED]> wrote:
>
> > > How to detect memory leaks of python programms, which run in an
> > > environment like this:
>
> > >  * Suse Linux 9.3
> > >  * Apache
> > >  *mod_python
>
> > > The problem occoured after some updates on the infrastructure. It's
> > > most possibly caused by trac and it's dependencies, but several
> > > components of the OS where updated, too.
>
> > > Any nice tools which play with the above constellation?
>
> > > Thank's for any hints!
>
> > No tool available to detect memory leaks for python applications?
>
> So, anyone who hit's on this thread via a search will think
>
> a) that there's really no memory leak detection for python
> b) that this community is not very helpful

Comments like (b) will not help your chances one bit in respect of
getting an answer from anyone.

Maybe you should read:

  http://www.catb.org/~esr/faqs/smart-questions.html

Two things to note in here. First, choose your forum appropriately and
secondly show some courtesy rather than making accusations against the
community if no one answers.

If you want to see perhaps how you might be viewed by the open source
community when you make such a comment, perhaps also watch:

  http://video.google.com/videoplay?docid=-4216011961522818645

Now, since you think this is a Trac problem, why don't you go ask on
the Trac mailing list.

  http://groups.google.com/group/trac-users?lnk=li

Even a search of that forum will most likely yield previous
discussions about growing memory use of Trac as a result of things
like Python wrappers for Subversion or certain database adapters. It
may be a case of using a different version, or in some cases
configuration of your hosting environment, if using Apache, to recycle
Apache child processes after a set number of requests so as to restore
process sizes back to a low level.

So, do some research first in the correct places and then perhaps ask
any additional questions in the correct place also.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to protect directory traversal in mod_python based custom apps

[Graham Dumpleton]

On Dec 24, 10:34 pm, "Ravi Kumar" <[EMAIL PROTECTED]> wrote:
> hi :)
> I was trying to develop a custommod_pythonbased web-site, just
> today. the problem I got
> though i liked themod_python'sfeature of mapping and calling
> functions in python script by parsing the url.
> I mean,http://localhost/site/member/list?no=100
>
> would call site/member.py page's function list with arguments no=100.
> Thats a feature i liked.
> But PROBLEM 01:
> i have included in index.py a css link to say something media/base.css
> now when same page comes with URL index.py/index the URL becomes
> false. I am finding some better way to overcome this.
> Placing all CSS as static served is not a good idea,(like if CSS is
> dynamically generated).
> So according to you, what should be a better approach to this problem.

The mod_python.publisher code is arguably broken in the way it handles
the trailing slash problem.

For some discussion on the issue see:

  http://www.modpython.org/pipermail/mod_python/2006-March/020501.html

This includes some code which might be modified and used in a stack
handler arrangement to give you a relative anchor point to use on
URLs.

> PROBLEM 02:
> How can I prevent directory traversal.
> Take the case, i have five subdirs in dir 'site' named :
> components
> modules
> config
> templates
>
> and a file loader.py
>
> when a request comes as loader.py/pagename?renderType=xhtml
> it would call the function pagename which loads the pages from subdir
> 'templates' resolves the added components in pages from subdir
> 'components' where components uses custom modules from 'modules' and
> so on. Configuration subdir contains various configuration files in
> .py and .xml
>
> I don't want visitors to traverse and get list of all those subdirs.
> Those sub-dirs actually should no way be traversable online.
> Though I can prevent it using apache .htaccess and access directives
> in apache config.
>
> But many hosting server, apache config can't be edited (or maybe some
> situation). Then how can i block traversing the directory (what sort
> of implementation)
> Referring to CodeIgnitor PHP Framework, they places index.php in every
> dir. thats doesn't seem a good idea, and if a person calls the pages
> providing the right path, they are able to execute files in the
> framework,  though since those configs and other files doesn't return
> anything, tere is no result.

If the ISP gives you some directory space which isn't part of the
exposed document tree, then simply move those subdirectories from the
document tree outside to the additional space you have. Then refer to
the files from there.

If you can't do that because the document tree is all you have, then
one remaining hack is to rename all the files in the subdirectories to
begin with '.ht' prefix. This would generally work as default Apache
configuration is to forbid access to any files starting with '.ht'
prefix.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What is the best way to do dynamic imports ?

[Graham Dumpleton]

On Dec 31, 1:24 am, [EMAIL PROTECTED] wrote:
> Hi list and python gurus :-)
>
> I'm playing with somemod_pythonand web development. And in me code I
> need to do som dynamic imports.
> Right now I just do a:
>
> exec 'import '+some_modulename
>
> But it seems to easy, is there a "dark side" to doing it this way?
> (memory use,processing ,etc)
> And have to I check if the modul is already loaded?
>
> Another thing is how to call my dynamic imported moduls.
> Now I use exec (as with my modules), like this:
>
> exec 'newclass = '+classname+'()'
> newclass.somefunction()
>
> Again it seems to easy. Is there a better/proper way to do it?
>
> Do anybody now a good howto or tutorial to this?
>
> Many thanks and hope you all have a happy new year :-)
>
> /marc

If using mod_python, you should use mod_python's own dynamic module
importer. See the documentation for mod_python.apache.import_module()
in:

  http://www.modpython.org/live/current/doc-html/pyapi-apmeth.html

There is no need to use __import__ directly. By using mod_python's way
of doing things you can benefit from automatic module reloading
provided certain constraints met.

Graham
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python setup not working on Windows XP

[Graham Dumpleton]

On Jan 8, 5:31 pm, Tim Roberts <[EMAIL PROTECTED]> wrote:
> Gowri <[EMAIL PROTECTED]> wrote:
>
> >I am new to Python and am trying to setup Apache to serve Python using
> >mod_python. I'm using a Windows XP box. here is a list of steps i
> >followed for the installation:
>
> >1. Installed Apache 2.2.6
> >2. Installed Python 2.5.1
> >3. Installedmod_python3.3.1
>
> >I then included the line
> >LoadModule python_module modules/mod_python.so in httpd.conf
>
> >I had this one line python file (print "Hello") in htdocs of Apache.
>
> Did you put it in a file called "hello.py"?  Did you create an AddHandler
> for .py files?  Did you create a PythonHandler referring to hello.py?

And did you (OP) read the mod_python documentation enough to know that
'print "Hello" is in no way going to work with mod_python. You cannot
just throw an arbitrary bit of Python code in a file using 'print'
statements and it will somehow magically work. You need to write your
code to the mod_python APIs.

Graham

-- 
http://mail.python.org/mailman/listinfo/python-list