Re: Alternative to python -u for binary upload to cgi on windows?

[cameron . walsh]

On Dec 14, 6:58 pm, Cameron Walsh <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> Using a pythoncgiscript such as the one below to handle uploaded
> binary files will end up with atruncatedfile (truncates when it hits
> ^Z) on Windows systems.  On linux systems the code works and the file is
> nottruncated.
>
> One solution for Windows is to use the -u flag, i.e.
> #!C:\Python\python.exe -u
>
> Is there another fix that doesn't require python to be run in unbuffered
> mode?  What performance hits would I expect for running in unbuffered mode?

Answer to the first part:

import msvcrt, sys, os
msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)

# other problems may also be fixed with:
# msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Alternative to python -u for binary upload to cgi on windows?

[Cameron Walsh]

Hi all,

Using a python cgi script such as the one below to handle uploaded 
binary files will end up with a truncated file (truncates when it hits 
^Z) on Windows systems.  On linux systems the code works and the file is 
not truncated.

One solution for Windows is to use the -u flag, i.e.
#!C:\Python\python.exe -u

Is there another fix that doesn't require python to be run in unbuffered 
mode?  What performance hits would I expect for running in unbuffered mode?

Best regards,

Cameron.


Example code:

import cgi, os.path as path
form = cgi.FieldStorage()
new_file = form.getvalue("new_file",None)
if new_file is not null:
   fname = path.split(form["new_file"].filename)[-1]
   fle = open(fname,"wb")
   fle.write(new_file)
   fle.close()
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: webbrowser

[Cameron Walsh]

Antonio Ceballos wrote:
> Hello,
> 
> I am trying to open a URL on a new browser or new tab from an HTML page 
> created from a python cgi script. 

import cgitb; cgitb.enable()
# Will save you hours of debugging.  Prints error messages and 
exceptions to the client, wrapped in pretty html.

On Apache in my localhost, it works, both
> with Internet Explorer and Firefox. However, when I upload the script to a 
> remote server, it does not. A 500 Internal Server Error is displayed on the 
> browser.

Check your server logs - they give more information on the cause of the 
error.

> 
> The (simplified) piece of code is as follows:
> 
> import webbrowser
> webbrowser.open_new(http://www.google.com)

try: webbrowser.open_new("http://www.google.com";) instead.

But the main problem is that you cannot make a web client open their 
browser.  What your code does is make whichever computer is running the 
cgi code (i.e. the server) open a browser pointing to http://www.google.com.

It works on your home machine, because you have a graphical interface 
running and python can open whatever browser it chooses.  The code would 
work if it wasn't a cgi file (or requested by a web browser and executed 
by a python handler on the server in any way).


Why not make the cgi file print the javascript that would make the 
client open another window?


> 
> The server runs python 2.3. I am using python 2.5 in my localhost.

The webbrowser module is present in python 2.3 but it is not what you 
want.  You might also have other unrelated problems in the code.  Use 
import cgitb; cgitb.enable() and check your server logs.


Hope that helps,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: manually cutting a picture

[Cameron Walsh]

Amit Khemka wrote:

> Cut image by "m X m" grid (bigger the m, the more varied shapes you
> would be able to generate), this will give you m*m square pieces. With
> each piece store a vector which represents the polygon (say by storing
> co-ordinates of the corners).
> Now visualize this set of pieces as a graph with an edge between
> adjacent pieces. Now depending on the final number of pieces that you
> want to generate (N), you traverse the graph and for each node:
> 
> 1. "Merge" a node with a randomly selected neighbor
> 
> Repeat step 1 until you have N pieces left
> 
Doesn't that have the fairly likely possibility of ending up with 1 very 
large piece and n-1 very small pieces?  If you iterate over the graph 
merging with a neighbour at random, then each node has a 50% chance of 
being merged with a node that has already been merged.

*-*-* *-*
|   | |
* *-*-* *

* * * * * etc. could be the result of the first 8 steps.  Already every 
node touched is part of the same group.  Or have I misunderstood your 
algorithm?

A random region growing approach might work, with seeds scattered evenly 
throughout the image.  That would be prone to orphaning squares inside 
larger objects, but that may be what you want.  It would not be easy to 
program.

What I would do is break the graph in to m*m squares, then for each 
edge, randomly select a means of joining them - blob sticking out of 
piece A, or hole for blob in piece A; move hole/blob along the edge by a 
random amount, move edge in or out to get different sized pieces.  For 
each square this was applied to, it would apply the reverse sizing to 
the square touching on that edge.

I'd restrict in/out movement of edges and movement of blob/hole along 
the edge to a specific range, to avoid problems where the blob of one 
square is stuck on the furthest corner, but the piece touching that 
corner has had its edge moved in too far so the blob would overlap two 
squares.

Let me know how it goes, it sounds like a fun problem.

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python good for data mining?

[Cameron Walsh]

Jens wrote:
> 
> Thanks a lot! I'm not sure I completely understand your description of
> how to integrate Python with, say PHP. Could you please give a small
> example? I have no experience with Python web development using CGI.
> How easy is it compared to web development in PHP?
> 
> I still havent't made my mind up about the choice of programming
> language for my data mining project. I think it's a difficult
> decision. My heart tells me "Python" and my head tells me "Java" :-)
> 

My C++ lecturer used to tell us "'C++ or Java?' is never the question. 
For that matter, Java is never the answer."

As for python and cgi, it's pretty simple.  Instead of a .php file to be 
handled by the php-handler, you have a .cgi file which is handled by the 
cgi-handler.  Set the action of your html form to the .cgi file.  At the 
top of the .cgi file, you'll need a line like:

#!/usr/bin/env python

Which tells it to use python as the interpreter.  You'll need a few imports:

import cgi
import cgitb; cgitb.enable() # for debugging - it htmlises
 # your exceptions and error messages.
print """Content-type: text/html; charset="iso-8859-1";\n"""
# You need that line or something similar so the browser knows what to 
do with the output of the script.

Everything that's printed by the python script goes straight to the 
client's browser, so the script will have to print html.  The cgi module 
handles form data, typically formdata = cgi.FieldStorage() will be 
filled when a form is sent to the script.  print it and see what's in it.

 From here, there's a huge number of tutorials on python and cgi on the 
web and I'm tired.

Best of luck,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to iterate through each set

[Cameron Walsh]

Tom_chicollegeboy wrote:
> I figured out problem. here is my code. now it works as it should!
> Thank you everyone!
>
I decided my 4th clue earlier was too much, so I removed it before 
posting.  It looks like you got it anyway =)

You've now solved it the way the course instructor intended you to solve 
it.  I would have done it this way:

def speed():
 journeys = []
 for line in open('speed.in', 'r').readlines():
 line = list(int(i) for i in line.split())
 if len(line) == 1:
 if line[0] == -1:
 break
 time = 0
 journeys.append(0)
 else:
 journeys[-1] += (line[1]-time) * line[0]
 time = line[1]
 for journey in journeys:
 print journey

speed()
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to iterate through each set

[Cameron Walsh]

Tom_chicollegeboy wrote:
> 
> 
> Here is my code:
> 
> def speed():
> infile = open('speed.in', 'r')
> line = infile.readline()
> read = int(line)
> print line
> i = 0
> t0 = 0
> v = 0
> 
> while i line = infile.readline()
> list = line.split()
> s = int(list[0])
> t1 = int(list[1])
> t = t1- t0
> v += s*t
> i+=1
> t0 = t1
> print v
> 
> It works only for first set. How should I implement another solution
> that would move program to read next set? I think I should implement
> nested while look, with first that reads until -1 and under it another
> while loop that reads n times sets. But I have no idea how. Can you
> help me?
> 

Clues:

1.)  Your program works for one block.  As you suggested you now need to 
loop over some of your code until you reach an end condition.
2.)  The end condition has been given to you already in the text and the 
file.
3.)  This can be done with two more lines of code.  One line to add a 
loop of some sort, checking for the end condition; and one line which 
brings you closer to the end condition after each block, and prepares 
you for the next block.

I hope this has given you enough to go on without confusing you or 
giving you the answer outright.

Now a few general pointers unrelated to your assignment:

1.)  list is a keyword in python.  Don't call your variables list.
2.)  You've left the file open at the end of your program.  Yes, Python 
will close it once the program has finished, but it's good practice to 
close it yourself as soon as you've finished with it.  This won't be a 
problem in this assignment, but it may be in later assignments.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python good for data mining?

[Cameron Walsh]

Jens wrote:
> I'm starting a project in data mining, and I'm considering Python and
> Java as possible platforms.
> 
> I'm concerned by performance. Most benchmarks report that Java is
> about 10-15 times faster than Python, and my own experiments confirms
> this. I could imagine this to become a problem for very large
> datasets.

If most of the processing is done with SQL calls, this shouldn't be an 
issue.  I've known a couple of people at Sydney University who were 
using Python for data mining.  I think they were using sqlite3 and MySQL.

> 
> How good is the integration with MySQL in Python?

Never tried it, but a quick google reveals a number of approaches you 
could try - the MySQLdb module, MySQL for Python, etc.

> 
> What about user interfaces? How easy is it to use Tkinter for
> developing a user interface without an IDE? And with an IDE? (which
> IDE?)

WxPython was recommended to me when I was learning how to create a GUI. 
It has more features than Tkinter and a more native look and feel across 
platforms.  With WxPython it was fairly easy to create a multi-pane, 
tabbed interface for a couple of programs, without using an IDE.  The 
demos/tutorials were fantastic.

> 
> What if I were to use my Python libraries with a web site written in
> PHP, Perl or Java - how do I integrate with Python?

Possibly the simplest way would be python .cgi files.  The cgi and cgitb 
modules allow form data to be read fairly easily.  Cookies are also 
fairly simple.  For a more complicated but more customisable approach, 
you could look in to the BaseHTTPServer module or a socket listener of 
some sort, running that alongside the webserver publicly or privately. 
Publicly you'd have links from the rest of your php/whatever pages to 
the python server.  Privately the php/perl/java backend would request 
data from the local python server before feeding the results back 
through the main server (apache?) to the client.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to do the mapping btw numpy arrayvalues and matrix columns

[Cameron Walsh]

[EMAIL PROTECTED] wrote:
> hi
> i am looking for some info about mapping  btw values in an array and
> corresponding columns of a matrix
> 
> i have an numpy array=[11.0,33.0,22.0,55.0,44.0]
> and a numpy matrix object=
>matrix(([1.3,2.5,3.2,6.7,3.1],
>   [9.7,5.6,4.8,2.5,2.2],
>   [5.1,3.7,9.6,3.1,6.7],
>   [5.6,3.3,1.5,2.4,8.5]))
> the first value of array(ie 11.0) is related to the first column of
> matrix  and so on..
> i wish to create a mapping btw each val of array and corresponding col
> of matrix..and then i want to sort the array and retrieve the matrix
> columns for some values of sorted array..can anyone advise how to go
> about it..
> 
> dn
> 

If this were a less helpful mailing list I'd say something like "there 
is also a numpy mailing list and you've overwritten the object class."

Anyway, the following would work, but it's not going to be fast for 
millions of elements:

from numpy import matrix, asarray
obj = matrix(([1.3,2.5,3.2,6.7,3.1],
   [9.7,5.6,4.8,2.5,2.2],
   [5.1,3.7,9.6,3.1,6.7],
   [5.6,3.3,1.5,2.4,8.5]))
ar = asarray(obj)
val_to_col_list = []
for row in ar:
 for ind,val in enumerate(row):
 val_to_col_list.append((val,ind))
val_to_col_list.sort()

If instead you require a map, such that each value maps to a list of the 
columns it appears in, you could try the following:

val_col_map = {}
for row in ar:
 for col,val in enumerate(row):
 tmplist=val_col_map.get(val,[])
 tmplist.append(col)
 val_col_map[val]=tmplist
val_keys = val_col_map.keys()
val_keys.sort()

val_keys is now a sorted list of unique values from your original 
matrix.  Use these values as keys for val_col_map.

Eww... but it works.  You'll want to be really careful with floating 
point numbers as keys, see http://docs.python.org/tut/node16.html for 
more details.

Best of luck,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python in academics?

[Cameron Walsh]

sandipm wrote:
> seeing posts from students on group. I am curious to know, Do they
> teach python in academic courses in universities?
> 
Sydney University teaches user interface design, some data mining and 
some natural language processing in Python.  Software development is 
still largely a Java or C affair.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Logging output from python

[Cameron Walsh]

Barry wrote:
> Hi, guys
> 
> Basiclly, it is automated testing system. There is a main python script
> that handles the testing campagin. This main script will call another
> script that will in turn runs a few hundered individual python scripts.
> 
> 
> Here is my problem. I want to log everything displayed in the screen
> after I start the main python script. Things include unhandled
> exceptions , message from print statement and other sources.
> Basically, if it is displayed on the screen, I want to log it..
> 
> 
> It might not be a pythons specific problem. Does anyone know a small
> tool does that job?
> 
> 
> Thanks.
> 

If it's on linux you can just redirect the screen output to a file:

python initialfile.py 1>stdout.txt 2>stderr.txt

or if you want standard out and standard error to go to the same file:

python initialfile.py 1>output.txt 2>output.txt

or if you don't want to see anything on standard error:

python initialfile.py 1>output.txt 2>/dev/null


As for windows, I'll test it now...

It turns out you can at least redirect the output to a file, I'm not
sure what it does with standard error or even if it exists or not.

python initialfile.py > output.txt

should work.


Hope it helps,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: newb: Join two string variables

[Cameron Walsh]

johnny wrote:
> How do I join two string variables?
> I  want to do:  download_dir + filename.
> download_dir=r'c:/download/'
> filename =r'log.txt'
> 
> I want to get something like this:
> c:/download/log.txt
> 

Hi Johnny,

This is actually two questions:

1.)  How do I concatenate strings
2.)  How do I concatenate pathnames?

Answers:

1.)

>>> download_dir="C:/download/"
>>> filename="log.txt"
>>> path_to_file=download_dir+filename
>>> path_to_file
'C:/download/log.txt'

2.)

>>> import os
>>> filename='log.txt'
>>> path_to_file = os.path.join("C:/download",filename)
>>> path_to_file
'C:/download/log.txt'


Help on function join in module posixpath:

join(a, *p)
Join two or more pathname components, inserting '/' as needed



Hope it helps,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PIL throws exception when reading bitmap/pnm data

[Cameron Walsh]

Cameron Walsh wrote:
> Hi all,
> 
> I'm trying to extract the data from a bitmap or .pnm file using the
> following code:
> 
> import Image
> img = Image.open("test.bmp","r")
> data=img.getdata()
> 
> Unfortunately I get the following exception on Linux, but not on Windows:
> 
>>>> data=img.getdata()
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/usr/local/lib/python2.5/site-packages/PIL/Image.py", line 796,
> in getdata
> self.load()
>   File "/usr/local/lib/python2.5/site-packages/PIL/ImageFile.py", line
> 147, in load
> self.map = mmap.mmap(file.fileno(), size)
> EnvironmentError: [Errno 19] No such device
> 

> 
> 
> Thanks and regards,
> 
> Cameron.


Hi all,

It turns out the problem only arose when the file was on an NTFS drive
mounted as NTFS-3g.  Copying it to my home directory fixed the problem.

Who do I send the bug report to, the ntfs-3g or the PIL guys?

Regards,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

PIL throws exception when reading bitmap/pnm data

[Cameron Walsh]

Hi all,

I'm trying to extract the data from a bitmap or .pnm file using the
following code:

import Image
img = Image.open("test.bmp","r")
data=img.getdata()

Unfortunately I get the following exception on Linux, but not on Windows:

>>> data=img.getdata()
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.5/site-packages/PIL/Image.py", line 796,
in getdata
self.load()
  File "/usr/local/lib/python2.5/site-packages/PIL/ImageFile.py", line
147, in load
self.map = mmap.mmap(file.fileno(), size)
EnvironmentError: [Errno 19] No such device


At this time, I cannot provide the full bitmap for copyright reasons,
but I can provide some information about the bitmap:

[EMAIL PROTECTED]:~$ file test.bmp
test.bmp: PC bitmap data, Windows 3.x format, 1000 x 1000 x 8

The same code works for .ppm images in the same folder:

[EMAIL PROTECTED]:~$ convert test.bmp test.ppm
[EMAIL PROTECTED]:~$ python
import Image
img=Image.open("test.ppm")
data=img.getdata()

But does not work for .pnm images in the same folder:

[EMAIL PROTECTED]:~$ convert test.bmp test.pnm
[EMAIL PROTECTED]:~$ python
import Image
img=Image.open("test.pnm")
data=img.getdata()
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.5/site-packages/PIL/Image.py", line 796,
in getdata
self.load()
  File "/usr/local/lib/python2.5/site-packages/PIL/ImageFile.py", line
147, in load
self.map = mmap.mmap(file.fileno(), size)
EnvironmentError: [Errno 19] No such device




PIL Version on Linux and Windows:

$Id: Image.py 2337 2005-03-25 07:50:30Z fredrik $


How can I avoid this error without converting each image into a
different format?


Thanks and regards,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: splitting a long string into a list

[Cameron Walsh]

ronrsr wrote:
> still having a heckuva time with this.
> 
> here's where it stand - the split function doesn't seem to work the way
> i expect it to.
> 
> 
> longkw1,type(longkw):   Agricultural subsidies; Foreign
> aid;Agriculture; Sustainable Agriculture - Support; Organic
> Agriculture; Pesticides, US, Childhood Development, Birth Defects;
>  1
> 
> longkw.replace(',',';')
> 
> Agricultural subsidies; Foreign aid;Agriculture; Sustainable
> Agriculture - Support; Organic Agriculture; Pesticides, US, Childhood
> Development

Here you have discovered that string.replace() returns a string and does
NOT modify the original string.  Try this for clarification:

>>> a="DAWWIJFWA,dwadw;djwkajdw"
>>> a
'DAWWIJFWA,,dwadw;djwkajdw'
>>> a.replace(",",";")
'DAWWIJFWA;;dwadw;djwkajdw'
>>> a
'DAWWIJFWA,,dwadw;djwkajdw'
>>> b = a.replace(',',';')
>>> b
'DAWWIJFWA;;dwadw;djwkajdw'



> 
> 
>  kw = longkw.split("; ,")#kw is now a list of len 1

Yes, because it is trying to split longkw wherever it finds the whole
string "; '" and NOT wherever it finds ";" or " " or ",".  This has been
stated before by NickV, Duncan Booth, Fredrik Lundh and Paul McGuire
amongst others. You will need to do either:

a.)

# First split on every semicolon
a = longkw.split(";")
b = []
# Then split those results on whitespace
#(the default action for string.split())
for item in a:
  b.append(item.split())
# Then split on commas
kw = []
for item in b:
kw.append(item.split(","))

or b.)

# First replace commas with spaces
longkw = longkw.replace(",", " ")
# Then replace semicolons with spaces
longkw = longkw.replace(";", " ")
# Then split on white space, (default args)
kw = longkw.split()


Note that we did:
longkw = longkw.replace(",", " ")
and not just:
longkw.replace(",", " ")


You will find that method A may give empty strings as some elements of
kw.  If so, use method b.


Finally, if you have further problems, please please do the following:

1.)  Provide your input data clearly, exactly as you have it.
2.)  Show exactly what you want the output to be, including any special
cases.
3.)  If something doesn't work the way you expect it to, tell us how you
expect it to work so we know what you mean by "doesn't work how I expect
it to"
4.)  Read all the replies carefully and if you don't understand the
reply, ask for clarification.
5.)  Read the help functions carefully - what the input parameters have
to be and what the return value will be, and whether or not it changes
the parameters or original object.  Strings are usually NOT mutable so
any functions that operate on strings tend to return the result as a new
string and leave the original string intact.

I really hope this helps,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: splitting a long string into a list

[Cameron Walsh]

ronrsr wrote:
> I have a single long string - I'd like to split it into a list of
> unique keywords. Sadly, the database wasn't designed to do this, so I
> must do this in Python - I'm having some trouble using the .split()
> function, it doesn't seem to do what I want it to - any ideas?
> 
> thanks very much for your help.
> 
> r-sr-
> 
> 
> longstring = ''

What do you want it to do?  Split on each semicolon?

a = longstring.split(";")
for element in a:
  print element

Agricultural subsidies
 Foreign aidAgriculture
Sustainable Agriculture - Support
 Organic Agriculture
 Pesticides, US,Childhood Development, Birth Defects
 Toxic ChemicalsAntibiotics,AnimalsAgricultural Subsidies, Global
TradeAgriculturalSubsidiesBiodiversityCitizen
ActivismCommunityGardensCooperativesDietingAgriculture,
CottonAgriculture, GlobalTradePesticides, MonsantoAgriculture,
SeedCoffee, HungerPollution,Water, FeedlotsFood PricesAgriculture,
WorkersAnimal Feed,
Corn,PesticidesAquacultureChemicalWarfareCompostDebtConsumerismFearPesticides,
US, Childhood Development,Birth DefectsCorporate Reform,  Personhood
(Dem. Book)Corporate Reform, Personhood, Farming (Dem. Book)Crime Rates,
Legislation,EducationDebt, Credit CardsDemocracyPopulation,
WorldIncomeDemocracy,Corporate Personhood, Porter Township (Dem.
Book)DisasterReliefDwellings, SlumsEconomics, MexicoEconomy,
LocalEducation,ProtestsEndangered Habitat, RainforestEndangered
SpeciesEndangeredSpecies, Extinctionantibiotics, livestockAgricultural
subsidies
Foreign aid
Agriculture
 Sustainable Agriculture - Support
 OrganicAgriculture
 Pesticides, US, Childhood Development, Birth Defects
Toxic Chemicals


I think the problem arises because your string has the following problems:

1.)  Inconsistent spaces between words (some are non-existent)
2.)  Inconsistent separators between elements (sometimes semi-colons,
sometimes commas, but commas appear to belong to elements, sometimes no
clear separator at all)

Basically, this problem is not solvable by computer with currently
available resources.  There is no way Python or anything else can know
which words are meant to be together and which are not, when there are
no separators between elements and no separators between words within
those elements.

You need to find a new way of generating the string, or do it by hand.

How did you get the string?

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: reading csv problem

[Cameron Walsh]

[EMAIL PROTECTED] wrote:
> Hello,
> 
> I use csv to take information from file.
> import csv
> reader = csv.reader(open('t.csv'))
> 
> for row in reader:
> print row # it is perfectly OK
> 
> -
> But If I use this code I have problem
> import csv
> reader = csv.reader(open('t.csv'))
> 
> for row in reader:
> print row # it is perfectly OK
> for row in reader:
> print row # it is not printed on the monitor???
> 
> Why does only the first print row work here?
> 
> TIA,
> ajikoe
> 

Because reader is an iterator and has no means of going back to its
beginning.

You could make a list out of it:
lines = list(csv.reader(open('t.csv')))
for row in lines:
  print row
for row in lines:
  print row

Hope it helps,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Printing to file, how do I do it efficiently?

[Cameron Walsh]

Robert Kern wrote:
> Cameron Walsh wrote:
>> Hi all,
>>
>> I have a numpy.array of 89x512x512 uint8's, set up with code like this:
> 
> numpy questions are best asked on the numpy list, not here.

At first I thought it was a generic python question, since it had more
to do with writing array data to file rather than the specific format of
the array data.

> 
>> data=numpy.array([],dtype="uint8")
>> data.resize((89,512,512))
> 
> You might want to look at using numpy.empty() here, instead.
> 

Thanks!

[...]
>> I'm guessing that the slow part is the fact that I am converting the
>> data to character format and writing it one character at a time.  What
>> is a better way of doing this, or where should I look to find a better way?
> 
> data.tostring()
> 

And here I see I was wrong, it was a numpy question.  I assumed the
tostring() method would produce the same output as printing the array to
the screen by just calling "data".  But of course, that would be the job
of the __repr__() method.

It is now ridiculously fast (<1second).  Thank you for your help.

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Printing to file, how do I do it efficiently?

[Cameron Walsh]

Hi all,

I have a numpy.array of 89x512x512 uint8's, set up with code like this:


data=numpy.array([],dtype="uint8")
data.resize((89,512,512))
# Data filled in about 4 seconds from 89 image slices



I first tried writing this data to a binary raw format (for use in a
program called Drishti) as follows:

# The slow bit:
volumeFile=file("/tmp/test.raw","wb")
for z in xrange(Z):
for y in xrange(Y):
for x in xrange(X):
volumeFile.write("%c" %(data[z,y,x]))
volumeFile.close()

That took about 39 seconds.

My second attempt was as follows:
volumeFile = open("/tmp/test2.raw","wb")
data.resize((X*Y*Z)) # Flatten the array
for i in data:
volumeFile.write("%c" %i)
data.resize((Z,Y,X))
volumeFile.close()

This took 32 seconds.  (For those of you unfamiliar with numpy, the
data.resize() operations take negligible amounts of time, all it does is
allow the data to be accessed differently.)

I'm guessing that the slow part is the fact that I am converting the
data to character format and writing it one character at a time.  What
is a better way of doing this, or where should I look to find a better way?


Thanks,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python to tell what is the IP of my PC .

[Cameron Walsh]

Gabriel Genellina wrote:
> At Wednesday 8/11/2006 21:18, Nicolas G wrote:
> 
>> > How can I use python to get the real IP address of my DSL router (when
>> > my PC is part of the local home LAN) ?
>>
>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/162994
>>
>>
>> import
>>  win32api raise the error "module not exsit"

http://sourceforge.net/projects/pywin32/

>> and
>> socket.gethostbyname(name)the router doesn't have a hostname.

use the hostname you create on noip.com or dyndns.com etc.


Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python cgi Apache os.system()

[Cameron Walsh]

[EMAIL PROTECTED] wrote:
> Hello :)
> 
> I have installed Apache on windows...
> The server work well, and my python script also
> 
> but when I want in my python script to run a System command like
> os.system(my_command) the script doesn't do anything!
> 
> here the error.log
> 
> [Wed Nov 08 14:19:30 2006] [error] [client 127.0.0.1] The system cannot
> find the drive specified.\r, referer:
> http://127.0.0.1/cgi-bin/extract_source.py
> 
> When i run the python script manually it works!
> 
> i think it's a user access probleme but i don't know how to resolve it
> !
> 
> please help  
> thanks
> 

Without knowing what "my_command" is, all I can suggest is that
"my_command" might be referring to a file that is in your user path, but
not in the path of the user used to run apache or python (which could be
"nobody").

Depending on how securely your server is configured, you may be able to
access whichever file it is looking for if you specify the full path to
the file, or the relative path from the server root directory (watch out
for hardlinks/softlinks.)

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Cards deck problem

[Cameron Walsh]

[EMAIL PROTECTED] wrote:
> Dennis Lee Bieber wrote:
>> On Fri, 27 Oct 2006 03:48:25 GMT, Michael Naunton
>> <[EMAIL PROTECTED]> declaimed the following in comp.lang.python:
>>
>>> This may seem pendantic, but CS is mostly about thinking about (and thus
>>   Someday I should arrange to do a lost-wax casting of a
>> /pendant/ with the definition of /pedant/ on it 
> 
> Why not a /pedant/ with a description of /pendant/ on it?
> 
>> --
>>  WulfraedDennis Lee Bieber   KD6MOG
>>  [EMAIL PROTECTED]   [EMAIL PROTECTED]
>>  HTTP://wlfraed.home.netcom.com/
>>  (Bestiaria Support Staff:   [EMAIL PROTECTED])
>>  HTTP://www.bestiaria.com/
> 
As fun as it would be, I haven't found too many /pedants/ who want to 
have the definition of /pendant/ written on them.

You've just given me an idea for an online T-Shirt printing business. 
Thank you, and your royalties will be in the post soon, just provide me 
with your credit card details.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to Split Chinese Character with backslash representation?

[Cameron Walsh]

limodou wrote:
> On 10/27/06, Wijaya Edward <[EMAIL PROTECTED]> wrote:
>>
>> Thanks but my intention is to strictly use regex.
>> Since there are separator I need to include as delimiter
>> Especially for the case like this:
>>
>> >>> str = '\xc5\xeb\xc7\xd5\xbc--FOO--BAR'
>> >>> field = list(str)
>> >>> print field
>> ['\xc5', '\xeb', '\xc7', '\xd5', '\xbc', '-', '-', 'F', 'O', 'O', '-', 
>> '-', 'B', 'A', 'R']
>>
>> What we want as the output is this instead:
>> ['\xc5', '\xeb', '\xc7', '\xd5', '\xbc','FOO','BAR]
>>
>> What's the best way to do it?
>>
> If the case is very simple, why not just replace '_' with '', for example:
> 
> str.replace('-', '')
> 
Except he appears to want the Chinese characters as elements of the 
list, and English words as elements of the list.  Note carefully the 
last two elements in his desired list.  I'm still puzzling this one...
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to Split Chinese Character with backslash representation?

[Cameron Walsh]

Wijaya Edward wrote:
> Hi all,
> 
> I was trying to split a string that 
> represent chinese characters below:
>  
>  
 str = '\xc5\xeb\xc7\xd5\xbc'
 print str2,
> ???
 fields2 = split(r'\\',str)
 print fields2,
> ['\xc5\xeb\xc7\xd5\xbc']
> 
> But why the split function here doesn't seem
> to do the job for obtaining the desired result:
>  
> ['\xc5','\xeb','\xc7','\xd5','\xbc']
>  

Depends on what you want to do with them:

 >>> string = '\xc5\xeb\xc7\xd5\xbc'
 >>> for char in string:
 print char


Å
ë
Ç
Õ
¼
 >>> list_of_characters = list(string)
 >>> list_of_characters
['\xc5', '\xeb', '\xc7', '\xd5', '\xbc']
 >>> for char in string:
 char


'\xc5'
'\xeb'
'\xc7'
'\xd5'
'\xbc'
 >>> for char in list_of_characters:
 print char


Å
ë
Ç
Õ
¼
 >>> string[3]
'\xd5'
 >>> string[1:3]
'\xeb\xc7'

Basically, you characters are already separated into a list of 
characters, that's effectively what a string is (but with a few more 
methods applicable only to lists of characters, not to other lists).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Insert Content of a File into a Variable

[Cameron Walsh]

Wijaya Edward wrote:
> Hi,
>  
> How can we slurp all content of a single file 
> into one variable?
>  
> I tried this:
>  
 myfile_content = open('somefile.txt')
 print myfile_content,
> 
> 
>  
> But it doesn't print the content of the file.

 >>> help(open)
Help on built-in function open in module __builtin__:

open(...)
 open(name[, mode[, buffering]]) -> file object

 Open a file using the file() type, returns a file object.

 >>> help(file)
Help on class file in module __builtin__:

class file(object)
  |  file(name[, mode[, buffering]]) -> file object



  |read(...)
  |  read([size]) -> read at most size bytes, returned as a string.
  |
  |  If the size argument is negative or omitted, read until EOF is
  |  reached.
  |  Notice that when in non-blocking mode, less data than what was
  |  requested
  |  may be returned, even if no size parameter was given.



  |readlines(...)
  |  readlines([size]) -> list of strings, each a line from the file.
  |
  |  Call readline() repeatedly and return a list of the lines so read.
  |  The optional size argument, if given, is an approximate bound on the
  |  total number of bytes in the lines returned.



Those sound useful...

Or alternatively:

 >>> my_file = open('somefile.txt')
 >>> print myfile_content,

 >>> #Hmmm, that's not what I wanted, what can I do with this?
 >>> dir(my_file)
['__class__', '__delattr__', '__doc__', '__enter__', '__exit__', 
'__getattribute__', '__hash__', '__init__', '__iter__', '__new__', 
'__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 
'close', 'closed', 'encoding', 'fileno', 'flush', 'isatty', 'mode', 
'name', 'newlines', 'next', 'read', 'readinto', 'readline', 'readlines', 
'seek', 'softspace', 'tell', 'truncate', 'write', 'writelines', 
'xreadlines']
 >>> #That read attribute looks interesting...
 >>> my_file.read

 >>> my_file.read()

Ye gads!  I wish I'd chosen a shorter file!  Or used a variable to put 
it in! 

 >>> text = my_file.read()
 >>> print text

Ye gads!  I wish I'd chosen a shorter file!  


You might also try:
 >>> lines = my_file.readlines()
 >>> for line in lines:
 >>> print line



Hope it helps,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting by item_in_another_list

[Cameron Walsh]

Steven Bethard wrote:
> 
> As you noted, you'll get an error if you try to concatenate B as a set 
> to the list.
> 
> Steve

Whoops, remind me to check things work before I type them.  In the mean 
time, here are some more interesting timing results:

With a larger data set, 500 elements instead of 100, the times were 
almost the same:

python -m timeit -s "A=range(500); B=range(40,60,2); B_set=set(B)" 
"A=B+list(x for x in A if x not in B_set)"
1 loops, best of 3: 103 usec per loop

python -m timeit -s "A=range(500); B=range(40,60,2); B_set = set(B)" 
"A.sort(key=B_set.__contains__, reverse=True)"
1 loops, best of 3: 99.2 usec per loop

But for the last recommended one:

python -m timeit -s "A=range(500); B=range(40,60,2); C = set(A) -set(B)" 
"A=B+filter(C.__contains__, A)"
1 loops, best of 3: 38.3 usec per loop

Much faster!  But that does not take into account the set creation and 
set difference calculations.  Let's try including that:

python -m timeit -s "A=range(500); B=range(40,60,2)" "C=set(A)-set(B)" 
"a=B+filter(C.__contains__, A)"
1 loops, best of 3: 105 usec per loop

and compare to our other two versions including set creation:

python -m timeit -s  "A=range(500); B=range(40,60,2)" "B_set = set(B); 
A=B+list(x for x in A if X not in B_set"
1 loops, best of 3: 105 usec per loop

python -m timeit -s  "A=range(500); B=range(40,60,2)" "B_set = set(B); 
A.sort(key=B_set.__contains__, reverse=True)"
1 loops, best of 3: 101 usec per loop

Pretty similar again.

However, converting each element to a string first ("A=list(str(x) for x 
in range(500)" etc.), making it more similar to my data, gave:

102usec per loop for the A.sort() method
132usec per loop for the A=B+filter() method
104usec per loop for the A=B+list() method

It is interesting to note the increased time taken by the set 
difference/filter method.  It appears not to like strings as much as 
ints.  We can also shave another 7 usec off the filter time:

python -m timeit -s "A=list(str)x) for x in range(500)); B=list(str(x) 
for x in range(40,60,2)); not_in_B_set=lambda x: x not in B_set" 
"B_set=set(B); A=B+filter(not_in_B_set, A)"
1 loops, best of 3: 125 usec per loop

In conclusion, it appears I should use the clearest of all the methods, 
since for the data sizes I am dealing with, they are of approximately 
the same speed, and the time taken is negligible anyway.

Thanks for all your help,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting by item_in_another_list

[Cameron Walsh]

Paul McGuire wrote:
> "J. Clifford Dyer" <[EMAIL PROTECTED]> wrote in message 
> news:[EMAIL PROTECTED]
>> ZeD wrote:
>>> Paul Rubin wrote:
>>>
> A = [0,1,2,3,4,5,6,7,8,9,10]
> B = [2,3,7,8]
>
> desired_result = [2,3,7,8,0,1,4,5,6,9,10]
 How about:

   desired_result = B + sorted(x for x in A if x not in B)
>>> this. is. cool.
>>>
>> Cool, yes, but I'm not entirely sure it does what the OP wanted.  Partly 
>> because I'm not entirely sure what the OP wanted.  Counter example:
>>
>> Given these variables:
>>
>> A = [0,1,2,3,4,5,6,8,9,10]  # Note 7 is missing
>> B = [2,3,7,8]
>>
>> which of the following should the function yield?
>>
> 
> From the original post:
> 
> "I have two lists, A and B, such that B is a subset of A."
> 
> So this is not a case that needs to be supported.
> 
> I envisioned something like the OP had a sequence of items to start with, 
> did a random sampling from the list, and wanted to move the sampled items to 
> the front of the list.
> 
> -- Paul
> 
> 
> 

The sequence of items I had was a list of files to be processed in a 
particular order.  I noted that the order in which the files were placed 
into the list might not match the desired order of processing, and that 
the filenames might not be sortable by filename into this desired order 
(such are the vagaries of human file naming.)

So yes, Paul is correct.

However, Cliff is also correct in that the solution provided is not what 
I (the OP) wanted.  The solution provided is half right, in that it 
places the elements also in B before the elements only in A, but it 
sorts the remaining elements only in A instead of keeping the original 
order.

A more correct solution (at least in terms of answering this particular 
problem) would be:

desired_result = B + list(x for x in A if x not in B)
rather than
desired_result = B + sorted(x for x in A if x not in B)

This solution is also guaranteed to work if for some reason the sort 
algorithm cannot be guaranteed to be stable in the future or on some 
other implementation of Python.


Which brings me to the question, would this solution:

B = set(B)
A = B + list(x for x in A if x not in B)

be faster than this solution:

B = set(B)
A.sort(key=B.__contains__, reverse=True)


My guess is yes, since while the __contains__ method is only run once 
for each object, the list still does not require sorting.

Thanks everyone for all your helpful replies,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting by item_in_another_list

[Cameron Walsh]

Delaney, Timothy (Tim) wrote:
> Cameron Walsh wrote:
> 
>> Hi,
>>
>> I have two lists, A and B, such that B is a subset of A.
>>
>> I wish to sort A such that the elements in B are at the beginning of
>> A, and keep the existing order otherwise, i.e. stable sort.  The
>> order of elements in B will always be correct.
>>
>> for example:
>>
>> A = [0,1,2,3,4,5,6,7,8,9,10]
>> B = [2,3,7,8]
>>
>> desired_result = [2,3,7,8,0,1,4,5,6,9,10]
> 
> c = set(B)
> a.sort(key=c.__contains__, reverse=True)
> 
> Tim Delaney

Depressingly simple.  I like it.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python GUIs comparison (want)

[Cameron Walsh]

[EMAIL PROTECTED] wrote:
> Now i began to learn GUI programming. There are so many
> choices of GUI in the python world, wxPython, pyGTK, PyQT,
> Tkinter, .etc, it's difficult for a novice to decide, however.
> Can you draw a comparison among them on easy coding, pythonish design,
> beautiful and generous looking, powerful development toolkit, and
> sufficient documentation, .etc.
> It's helpful for a GUI beginner.
> Thank you.
> 
> 
> :)Sorry for my poor english.
> 

I googled "python gui compare" a while back and got 
www.awaretek.com/toolkits.html as the first result.

Every variation on the values I entered seemed to point me to wxPython, 
which I'm still using now.  However, they seem to think that EasyGUI is 
the easiest to learn, but that it suffers on "Maturity, documentation, 
breadth of widget selection".

All the best,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting by item_in_another_list

[Cameron Walsh]

Paul McGuire wrote:
> "Cameron Walsh" <[EMAIL PROTECTED]> wrote in message 
> news:[EMAIL PROTECTED]
>> Hi,
>>
>> I have two lists, A and B, such that B is a subset of A.
>>
>> I wish to sort A such that the elements in B are at the beginning of A, 
>> and keep the existing order otherwise, i.e. stable sort.  The order of 
>> elements in B will always be correct.
>>
>> for example:
>>
>> A = [0,1,2,3,4,5,6,7,8,9,10]
>> B = [2,3,7,8]
>>
>> desired_result = [2,3,7,8,0,1,4,5,6,9,10]
>>
>>
>> At the moment I have defined a comparator function:
>>
>> def sort_by_in_list(x,y):
>> ret = 0
>> if x in B:
>> ret -= 1
>> if y in B:
>> ret += 1
>> return ret
>>
>> and am using:
>>
>> A.sort(sort_by_in_list)
>>
>> which does produce the desired results.
>>
>> I do now have a few questions:
>>
>> 1.)  Is this the most efficient method for up to around 500 elements? If 
>> not, what would be better?
>> 2.)  This current version does not allow me to choose a different list for 
>> B.  Is there a bind_third function of some description that I could use to 
>> define a new function with 3 parameters, feed it the third (the list to 
>> sort by), and have the A.sort(sort_by_in_list) provide the other 2 
>> variables?
>>
> 
> Think in Python.  Define a function to take the list, and have that function 
> return the proper comparison function.  This gives me the chance to also 
> convert the input list to a set, which will help in scaling up my list to 
> hundreds of elements.  See below.
> 
> -- Paul
> 
> 
> def sort_by_in_list(reflist):
> reflist = set(reflist)
> def sort_by_in_list_(x,y):
> ret = 0
> if x in reflist: ret -= 1
> if y in reflist: ret += 1
> return ret
> return sort_by_in_list_
> 
> A = [0,1,2,3,4,5,6,7,8,9,10]
> B = [2,3,7,8]
> A.sort( sort_by_in_list(B) )
> print A
> 
> Gives:
> [2, 3, 7, 8, 0, 1, 4, 5, 6, 9, 10]
> 
> 

Looks like our answers crossed-over.  Must learn about sets in Python...

Thanks very much,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting by item_in_another_list

[Cameron Walsh]

Cameron Walsh wrote:
> Hi,
> 
> I have two lists, A and B, such that B is a subset of A.
> 
> I wish to sort A such that the elements in B are at the beginning of A, 
> and keep the existing order otherwise, i.e. stable sort.  The order of 
> elements in B will always be correct.
> 
> for example:
> 
> A = [0,1,2,3,4,5,6,7,8,9,10]
> B = [2,3,7,8]
> 
> desired_result = [2,3,7,8,0,1,4,5,6,9,10]
> 
> 
> At the moment I have defined a comparator function:
> 
> def sort_by_in_list(x,y):
> ret = 0
> if x in B:
> ret -= 1
> if y in B:
> ret += 1
> return ret
> 
> and am using:
> 
> A.sort(sort_by_in_list)
> 
> which does produce the desired results.
> 
> I do now have a few questions:
> 
> 1.)  Is this the most efficient method for up to around 500 elements? If 
> not, what would be better?
> 2.)  This current version does not allow me to choose a different list 
> for B.  Is there a bind_third function of some description that I could 
> use to define a new function with 3 parameters, feed it the third (the 
> list to sort by), and have the A.sort(sort_by_in_list) provide the other 
> 2 variables?
> 
> 
> Regards to all,
> 
> Cameron.


Well I found an answer to the second question with the following:

 >>> A=[0,1,2,3,4,5,6,7,8,9,10]
 >>> B=[2,3,7,8]
 >>> def sort_by_in_list(in_list):
def ret_function(x,y):
ret = 0
if x in in_list:
ret -= 1
if y in in_list:
ret += 1
return ret
return ret_function

 >>> A.sort(sort_by_in_list(B))
 >>> A
[2, 3, 7, 8, 0, 1, 4, 5, 6, 9, 10]


Hope it helps someone,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Sorting by item_in_another_list

[Cameron Walsh]

Hi,

I have two lists, A and B, such that B is a subset of A.

I wish to sort A such that the elements in B are at the beginning of A, 
and keep the existing order otherwise, i.e. stable sort.  The order of 
elements in B will always be correct.

for example:

A = [0,1,2,3,4,5,6,7,8,9,10]
B = [2,3,7,8]

desired_result = [2,3,7,8,0,1,4,5,6,9,10]


At the moment I have defined a comparator function:

def sort_by_in_list(x,y):
ret = 0
if x in B:
ret -= 1
if y in B:
ret += 1
return ret

and am using:

A.sort(sort_by_in_list)

which does produce the desired results.

I do now have a few questions:

1.)  Is this the most efficient method for up to around 500 elements? 
If not, what would be better?
2.)  This current version does not allow me to choose a different list 
for B.  Is there a bind_third function of some description that I could 
use to define a new function with 3 parameters, feed it the third (the 
list to sort by), and have the A.sort(sort_by_in_list) provide the other 
2 variables?


Regards to all,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Cannot import a module from a variable

[Cameron Walsh]

Gabriel Genellina wrote:
> At Wednesday 18/10/2006 22:51, Cameron Walsh wrote:
> 
>> previous_directory = os.getcwd()
>> try:
>> os.chdir(directory)
>> [ ... ]
>> return modules
>> finally:
>> os.chdir(previous_directory)
>>
>> Woah, that actually works?  Having the "finally" after the "return"?
>> That could make some things easier, and some things harder...
> 
> Note that moving the return statement after the finally does *exactly* 
> the same thing, generates shorter code, and is a lot more readable (IMHO).
> 
> 

I wholeheartedly agree, the above version is hideous.  It was a 
copy-paste error that got it after the return statement and I was 
surprised to see it actually worked.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Cannot import a module from a variable

[Cameron Walsh]

Hi,

This has actually been answered in a previous post ("user modules"
started by myself), for which I was very grateful.  I have since
expanded on their solutions to create the following code, of which parts
or all may be useful.  You'll probably be most interested in the last
part of the code, from "# Now we actually import the modules" onwards.

>>> import os
>>> import glob
>>> # import_extension_modules(directory)
>>> # Imports all the modules in the sub-directory "directory"
>>> # "directory" MUST be a single word and a sub-directory of that
>>> # name MUST exist.
>>> # Returns then as a dictionary of {"module_name":module}
>>> # This works for both .py (uncompiled modules)
>>> # and .pyc (compiled modules where the source code is not given)
>>> # TODO: Fix limitations above, clean up code.
>>> #
>>> def import_extension_modules(directory):
previous_directory = os.getcwd()
try:
os.chdir(directory)
uncompiled_files = glob.glob("*.py")
compiled_files = glob.glob("*.pyc")
all_files = []
modules = {}
for filename in uncompiled_files:
all_files.append(filename[0:-3]) # Strip off the ".py"
for filename in compiled_files:
# Add any pre-compiled modules without source code.
filename = filename[0:-4] # Strip off the ".pyc"
if filename not in all_files:
all_files.append(filename)
if "__init__" in all_files:
# Remove any occurrences of __init__ since it does
# not make sense to import it.
all_files.remove("__init__")
# Now we actually import the modules
for module_name in all_files:
# The last parameter must not be empty since we are using
# import blah.blah
# format because we want modules not packages.
# see 'help(__import__)' for details.
module = __import__("%s.%s" %(directory,module_name), None,
None,["some string to make this a non-empty list"])
modules[module.__name__] = module
return modules
finally:
os.chdir(previous_directory)

>>> user_modules = import_extension_modules("user_modules")
>>> user_modules
{'user_modules.funky_module': }

Woah, that actually works?  Having the "finally" after the "return"?
That could make some things easier, and some things harder...

Hope that helps,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to share session with IE

[Cameron Walsh]

I just thought, your original question was whether or not it was 
possible to share your browser session with IE.  Unless you do this 
explicitly, you may require a different login for your Python program 
and for your IE user.  If the Python program does not get the same 
cookie as used by IE, or vice-versa, and tries to login, you may find 
this resets the login in the other browser.

Oh and at risk of starting a flame war get a real browser.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to share session with IE

[Cameron Walsh]

John J. Lee wrote:
> "Bernard" <[EMAIL PROTECTED]> writes:
>> zdp wrote:
> [...]
>>> However, now I need to process some pages by a python program. When I
>>> use urllib.urlopen(theurl), I can only get a page which told me I need
>>> login. I think It's reasonable, becuase I wasn't in a loggined session
>>> which as IE did.
>>>
>>> So how can I do my job? I want to get the right webpage by the url. I
>>> have search answers from the groups but didn't get clear answer. Should
>>> I use win32com or urllib? Any reply or information is appreciate. Hope
>>> I put it clear.
> 
>> You can do the same thing as IE on your forum using urllib2 and
>> cookielib. In short you need to code a small webcrawler. I can give you
>> my browser module if necessary.
>> You might not have the time to fiddle with the coding part or my
>> browser module so you can also use this particularly useful module :
>> http://wwwsearch.sourceforge.net/mechanize/
>> The documentation is pretty clear for an initiated python programmer.
>> If it's not your case, I'd recommend to read some ebooks on the python
>> language first to get use to it.
> 
> In particular, if you're following the approach Bernard suggests, you
> can either:
> 
> 1. Log in every time your program runs, by going through the sequence
>of clicks, pages, etc. that you would use in a browser to log in.
> 
> 2. Once only (or once a month, or whatever), log in by hand using IE
>with a "Remember me"-style feature (if the website offers that) --
>where the webapp asks the browser to save the cookie rather than
>just keeping it in memory until you close your browser.  Then your
>program can load the cookies from your real browser's cookie store
>using this:
> 
> http://wwwsearch.sourceforge.net/mechanize/doc.html#browsers
> 
> 
> There are other alternatives too, but they depend on knowing a little
> bit more about how cookies and web apps work, and may or may not work
> depending on what exactly the server does.  I'm thinking specifically
> here of saving *session* cookies (the kind that usually go away when
> you close your browser) in a file -- but the server may not like them
> when you send them back the next time, depending how much time has
> elapsed since the last run.  Of course, you can always detect the
> "need to login" condition, and react accordingly.
> 
> 
> John
> 


Another option instead of making your program run through a series of 
clicks and text inputs, which is difficult to program, is to browse the 
html source until you find the name of the script that processes the 
login, and use python to request the page with the necessary form fields 
encoded in the request.  Request something like
http://www.targetsite.com/login.cgi?username=pyuser&password="fhqwhgads";
This format is not guaranteed to work, since the login script or server 
might only support one of GET and POST.  If this is the case, creating 
the request is slightly more involved and to be honest I haven't looked 
into how to do it.

Thereafter, you will have to pass the environment to every page request 
so the server can read the cookie.  Which brings me to question whether 
or not it is possible to do this manually once, export the environment 
variable to a file, and reload this file each time the program is run. 
Or to generate the cookie in the environment yourself.  Quite frankly 
any server application that allows the client to control whether or not 
they have logged in sucks, but I've seen a fair few that do.[citation 
required]

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: user modules

[Cameron Walsh]

Juho Schultz wrote:
> Juho Schultz wrote:
>> Cameron Walsh wrote:
>>> Hi,
>>>
>>> I'm writing a python program to analyse and export volumetric data.  To
>>> make development and extension easier, and to make it more useful to the
>>> public when it is released (LGPL), I would like to enable users to place
>>> their own python files in a "user_extensions" directory.  These files
>>> would implement a common interface in order for the main program to be
>>> able to read them and execute the necessary code.
>>>
>>> My question is what is the best way of implementing this?
>>>
>>> I have investigated importing them as modules, but unless the user
>>> modifies the main program I cannot see how the main program can learn of
>>> the existence of specific modules.
>>>
>> One simple solution would be a shell script that adds user_extensions
>> (or whatever) to $PYTHONPATH and then starts your main program.
> 
> Sorry... I was typing faster than reading or thinking.
> 
> You could have a __init__.py file within user_extensions with
> __all__ = ["package1", "package2"]
> 
> If you want every python file within some directory in here, you can
> auto-generate the __init__.py file in user_extension before importing.
> (Or you could have some sort of tester for new .py files detected and
> only after you are sure it works, add it.)
> 
> from user_extensions import *
> 
> would import everything mentioned in __all__. You also have access to
> their names through
> user_extensions.__all__
> 
> The last step would be to put the modules into a list. After the
> import,
> 
> user_ext_list = [eval(elem) for elem in user_extensions.__all__ ]
> for ext in user_ext_list:
> ext.initialize()
> ext.get_tab_window()
> 

Thanks Juho,

Your solution also worked a charm.  I like the suggestion of a tester 
before adding a module to the __all__ list.

Thanks for teaching me about the eval() function, I've been wondering 
how to turn a string into a python command.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: user modules

[Cameron Walsh]

Tuomas wrote:
> Cameron Walsh wrote:
>> Hi,
>>
>> I'm writing a python program to analyse and export volumetric data.  
>> To make development and extension easier, and to make it more useful 
>> to the public when it is released (LGPL), I would like to enable users 
>> to place their own python files in a "user_extensions" directory.  
>> These files would implement a common interface in order for the main 
>> program to be able to read them and execute the necessary code.
>>
>> My question is what is the best way of implementing this?
>>
>> I have investigated importing them as modules, but unless the user 
>> modifies the main program I cannot see how the main program can learn 
>> of the existence of specific modules.
>>
>> For example:
>>
>> from user_modules import *
>> # directory 'user_modules' contains __init__.py allowing this
>> # From here I would need a list of the imported modules, in order to
>> # execute a common command on each of them, such as
>>
>> for module in imported_modules:
>> module.initialise()
>> module.get_tab_window()
>>
>>
>> How do I get from the first bit to the second bit, or is there a 
>> better way of obtaining the functionality I need?
>>
>>
>> --Cameron.
> 
> import os
> 
> files=os.listdir('user_modules')
> tabs=[]
> for fle in files:
> if fle.endswith('.py'):
> module=__import__(fle[0:-3], 'user_modules', None,
>['initialise', 'get_tab_window'])
> module.initialise()
> tabs.append(module.get_tab_window())
> 
> *not tested*
> 
> print __import__.__doc__
> __import__(name, globals, locals, fromlist) -> module
> 
> Import a module.  The globals are only used to determine the context;
> they are not modified.  The locals are currently unused.  The fromlist
> should be a list of names to emulate ``from name import ...'', or an
> empty list to emulate ``import name''.
> When importing a module from a package, note that __import__('A.B', ...)
> returns package A when fromlist is empty, but its submodule B when
> fromlist is not empty.
> 
> Tuomas
> 

Thanks Tuomas, your solution worked a charm, with two changes:
1.)  Check I'm not trying to import __init__.py
2.)  module=__import__(fle[0:-3], 'user_modules', None,
['initialise', 'get_tab_window'])
  had to be changed to:
  module=__import__("user_modules.%s" %fle[0:-3], None, None,
['initialise', 'get_tab_window'])

For some reason it wasn't setting the context properly, I'll have to 
learn more about the function.

Thanks for teaching me how to read the documentation properly, about how 
private keywords like "import" are defined, and for providing a solution 
to a problem I'm likely to face many more times.

Best regards,

Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list

user modules

[Cameron Walsh]

Hi,

I'm writing a python program to analyse and export volumetric data.  To 
make development and extension easier, and to make it more useful to the 
public when it is released (LGPL), I would like to enable users to place 
their own python files in a "user_extensions" directory.  These files 
would implement a common interface in order for the main program to be 
able to read them and execute the necessary code.

My question is what is the best way of implementing this?

I have investigated importing them as modules, but unless the user 
modifies the main program I cannot see how the main program can learn of 
the existence of specific modules.

For example:

from user_modules import *
# directory 'user_modules' contains __init__.py allowing this
# From here I would need a list of the imported modules, in order to
# execute a common command on each of them, such as

for module in imported_modules:
module.initialise()
module.get_tab_window()


How do I get from the first bit to the second bit, or is there a better 
way of obtaining the functionality I need?


--Cameron.
-- 
http://mail.python.org/mailman/listinfo/python-list