Re: bug with os.rename in 2.4.1?

2009-04-28 Thread Nick Craig-Wood
t123  wrote:
>  It's running on solaris 9.  Here is some of the code.  It's actually
>  at the beginning of the job.  The files are ftp'd over.  The first
>  thing that happens is that the files get renamed before any processing
>  of the file.  And when it fails, it always fails at the first file,
>  comm.dat.  What I can't understand is why the inconsistent behavior.
> 
>  try:
>  if os.path.exists(paths.xferin_dir+'/COMM.DAT'):
>os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/

This code is inherently racy... What if two copies of your code
started simultaneously?  They might both run the os.path.exists but
only one will succeed in the os.rename.

You could write instead

try:

os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/COMM.DAT'+'.0')
except OSError:
pass

Which isn't racy.  Or if you wanted to be more thorough

import errno
try:

os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/COMM.DAT'+'.0')
    except OSError, e:
if e.errno != errno.ENOENT:
raise

The traceback should show the exact problem though.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Large data arrays?

2009-04-23 Thread Nick Craig-Wood
Ole Streicher  wrote:
>  Hi Nick,
> 
>  Nick Craig-Wood  writes:
> > mmaps come out of your applications memory space, so out of that 3 GB
> > limit.  You don't need that much RAM of course but it does use up
> > address space.
> 
>  Hmm. So I have no chance to use >= 2 of these arrays simultaniously?
> 
> > Sorry don't know very much about numpy, but it occurs to me that you
> > could have two copies of your mmapped array, one the transpose of the
> > other which would then speed up the two access patterns enormously.
> 
>  That would be a solution, but it takes twice the amount of address
>  space (which seems already to be the limiting factor). In my case (1.6
>  GB per array), I could even not use one array. 

You don't need them mapped at the same time so you could get away with
just one copy mapped.

Also you can map the array in parts and use dramatically less address
space.

>  Also, I would need to fill two large files at program start: one for
>  each orientation (row-wise or column-wise). Depending on the input
>  data (which are also either row-wise or column-wise), the filling of
>  the array with opposite direction would take a lot of time because of
>  the inefficiencies.
> 
>  For that, using both directions probably would be not a good
>  solution. What I found is the "Morton layout" which uses a kind of
>  fractal interleaving and sound not that complicated.

It sounds cool!

>  But I have no idea on how to turn it into a "numpy" style: can I
>  just extend from numpy.ndarray (or numpy.memmap), and which
>  functions/methods then need to be overwritten? The best would be
>  ofcourse that someone already did this before that I could use
>  without trapping in all these pitfalls which occur when one
>  implements a very generic algorithm.

I'd start by writing a function which took (x, y) in array
co-ordinates and transformed that into (z) remapped in the Morton
layout.

Then instead of accessing array[x][y] you access
morton_array[f(x,y)].  That doesn't require any subclassing and is
relatively easy to implement. I'd try that and see if it works first!

Alternatively you could install a 64bit OS on your machine and use my
scheme!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Memory footpring of python objects

2009-04-23 Thread Nick Craig-Wood
BlueBird  wrote:
>  I have a program that manages several thousands instances of one
>  object. To reduce memory
>  consumption, I want of course that specific object to have the
>  smallest memory footpring possible.
> 
>  I have a few ideas that I want to experiment with, like using
>  __slots__, using a tuple or using a dict. My
>  question is: how do I know the memory footprint of a given python
>  object ? I could not find any
>  builtin functions for this in the documentation.

I managed to 1/3 the memory requirement of our program by using
__slots__ on the most common class (several hundred thousand
instances!).

When doing these optimisations I ran a repeatable script and measured
the total memory usage using the OS tools (top in my case).

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Large data arrays?

2009-04-23 Thread Nick Craig-Wood
Ole Streicher  wrote:
>  for my application, I need to use quite large data arrays 
>  (100.000 x 4000 values) with floating point numbers where I need a fast
>  row-wise and column-wise access (main case: return a column with the sum 
>  over a number of selected rows, and vice versa).
> 
>  I would use the numpy array for that, but they seem to be
>  memory-resistent. So, one of these arrays would use about 1.6 GB
>  memory which far too much. So I was thinking about a memory mapped
>  file for that. As far as I understand, there is one in numpy.
> 
>  For this, I have two questions: 
> 
>  1. Are the "numpy.memmap" array unlimited in size (resp. only limited
>  by the maximal file size)? And are they part of the system's memory
>  limit (~3GB for 32bit systems)?

mmaps come out of your applications memory space, so out of that 3 GB
limit.  You don't need that much RAM of course but it does use up
address space.

>  2. Since I need row-wise as well as column-wise access, a simple usage
>  of a big array as memory mapped file will probably lead to a very poor
>  performance, since one of them would need to read values splattered
>  around the whole file. Are there any "plug and play" solutions for
>  that? If not: what would be the best way to solve this problem? 
>  Probably, one needs to use someting like the "Morton layout" for the
>  data. Would one then build a subclass of memmap (or ndarray?) that
>  implements this specific layout? How would one do that? (Sorry, I am
>  still a beginner with respect to python).

Sorry don't know very much about numpy, but it occurs to me that you
could have two copies of your mmapped array, one the transpose of the
other which would then speed up the two access patterns enormously.

You needn't mmap the two arrays (files) at the same time either.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: pyflakes, pylint, pychecker - and other tools

2009-04-23 Thread Nick Craig-Wood
Colin J. Williams  wrote:
>  Esmail wrote:
> > What is the consensus of the Python community regarding these
> > code checkers?
> > 
> > In particular, are the stylistic recommendations that
> > pylint makes considered sensible/valid?
> 
>  pylint seems a bit heavy handled, a bit 
>  too much PEP 8, which was intended as a 
>  guide, rather than a prescription.

I've been very happy with pychecker.  I found pylint a bit too fussy
(rather like the original C lint!)

Note that if you run pychecker from emacs (M-x compile, then
"pychecker module_name") you can then click in its output window to go
to the correct line of code.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: A Special Thanks

2009-04-21 Thread Nick Craig-Wood
norseman  wrote:
>  I'm one of those that tries to get an outline of the project and then 
>  puts in code as things become clear. Once the basics are working 
>  reasonably I go back and organize the thing for maintainability. Then 
>  finish flushing it out. It is the one stage I dread the most.
> 
>  Why not organize it up front?  Because I don't always have the whole pie 
>  at the outset.
> 
>  In changing to Python I had a bigger learning curve than I realized at 
>  the start.  When I finally got my "pieces" accomplishing what I wanted 
>  it became time to start looking at its structure.
> 
>  I did the cut and paste into a followable form and ran the basic to 
>  check for the usual errors, omissions and outright flaws. This was a 
>  major reorganization.
> 
>  IT RAN FLAWLESSLY THE FIRST TRY!  UNBELIEVABLE! (at least for me)

Python has a well deserved reputation for being executable
pseudo-code.  It is the only language I've ever used where when you
write the program it is quite likely to work the first time.

Python also converted me to using unit tests.  If you add unit tests
into your methodology above then when you re-organize (or refactor to
use the modern jargon) the code you can be 100% sure that you didn't
break anything which is a wonderful feeling.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: How to create a virtual serial port?

2009-04-19 Thread Nick Craig-Wood
Grant Edwards  wrote:
>  On 2009-04-12, JanC  wrote:
> > Grant Edwards wrote:
> >
> >> On 2009-04-10, Stuart Davenport  wrote:
> >>
> >>> I am trying to work out if its possible, to create a virtual serial
> >>> port with Python?
> >>
> >> On Linux: no.
> >
> > I wonder if there is no way to emulate ptys from userspace?
> 
>  Didn't I just answer that question?
> 
>  On Linux: no.

Actually you could do it with an LD_PRELOAD library

Intercept open("/dev/ttyS0",...).  You'd need to intercept ioctl(),
read(), write(), close() etc too.

Messy but possible.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: What IDE support python 3.0.1 ?

2009-04-16 Thread Craig
Well i use netbean is alot better i think and it work with 2.6 and 3.0

--- On Thu, 4/16/09, mousemeat  wrote:

From: mousemeat 
Subject: Re: What IDE support python 3.0.1 ?
To: python-list@python.org
Date: Thursday, April 16, 2009, 4:41 AM

Use eclipse with the pydev module.  I use python(x,y) which is a big
bundle of most of the python stuff you could possibly want (including
scientific stuff, but its not mandatory to use it) configured to work
together.  It uses python 2.5.

You can have the best of both worlds.  Search for 'from __future__
import'  to see how to get 3.0 features into 2.x, minimizing the
eventual upgrade hassles.
--
http://mail.python.org/mailman/listinfo/python-list



  --
http://mail.python.org/mailman/listinfo/python-list


Re: interacting with an updatedb generated data file within python

2009-04-16 Thread Nick Craig-Wood
birdsong  wrote:
>  Does anybody have any recommendations on how to interact with the data
>  file that updatedb generates?  I'm running through a file list in
>  sqlite that I want to check against the file system. updatedb is
>  pretty optimized for building an index and storing it, but I see no
>  way to query the db file other than calling locate itself.  This would
>  require me to fork and exec for every single file I want to verify -
>  I'd be better off doing the stat myself in that case, but I'd really
>  rather let updatedb build the index for me.

Hmm..

There are several different implementations of locate and I'm not sure
they all use the same database.  On this ubuntu machine the database
is only root readable also.

$ ls -l /var/lib/mlocate/mlocate.db
-rw-r- 1 root mlocate 7013090 2009-04-14 07:54 /var/lib/mlocate/mlocate.db

>  I searched high and low for any sort of library that is well suited
>  for reading these data files, but I've found nothing for any language
>  other than the source for locate and updatedb itself.

You can use this to extract the database from the locate database

from subprocess import Popen, PIPE
from time import time

start = time()

all_files = set()
p = Popen(["locate", "*"], stdout=PIPE)
for line in p.stdout:
path = line[:-1]
all_files.add(path)

print "Found", len(all_files), "files in", time()-start, "seconds"

This builds a set of all the files on the filesystem and prints

Found 314492 files in 1.152987957 seconds

on my laptop, using about 19 MB total memory

You could easily enough put that into an sqlite table instead of a set().

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Best way to extract from regex in if statement

2009-04-16 Thread Nick Craig-Wood
Paul McGuire  wrote:
>  On Apr 3, 9:26 pm, Paul Rubin <http://phr...@nospam.invalid> wrote:
> > bwgoudey  writes:
> > > elif re.match("^DATASET:\s*(.+) ", line):
> > >         m=re.match("^DATASET:\s*(.+) ", line)
> > >         print m.group(1))
> >
> > Sometimes I like to make a special class that saves the result:
> >
> >   class Reg(object):   # illustrative code, not tested
> >      def match(self, pattern, line):
> >         self.result = re.match(pattern, line)
> >         return self.result
> >
>  I took this a little further, *and* lightly tested it too.
> 
>  Since this idiom makes repeated references to the input line, I added
>  that to the constructor of the matching class.
> 
>  By using __call__, I made the created object callable, taking the RE
>  expression as its lone argument and returning a boolean indicating
>  match success or failure.  The result of the re.match call is saved in
>  self.matchresult.
> 
>  By using __getattr__, the created object proxies for the results of
>  the re.match call.
> 
>  I think the resulting code looks pretty close to the original C or
>  Perl idiom of cascading "elif (c=re_expr_match("..."))" blocks.
> 
>  (I thought about cacheing previously seen REs, or adding support for
>  compiled REs instead of just strings - after all, this idiom usually
>  occurs in a loop while iterating of some large body of text.  It turns
>  out that the re module already caches previously compiled REs, so I
>  left my cacheing out in favor of that already being done in the std
>  lib.)
> 
> 
>  import re
> 
>  class REmatcher(object):
>  def __init__(self,sourceline):
>  self.line = sourceline
>  def __call__(self, regexp):
>  self.matchresult = re.match(regexp, self.line)
>  self.success = self.matchresult is not None
>  return self.success
>  def __getattr__(self, attr):
>  return getattr(self.matchresult, attr)

That is quite similar to the one I use...

"""
Matcher class encapsulating a call to re.search for ease of use in conditionals.
"""

import re

class Matcher(object):
"""
Matcher class

m = Matcher()

if m.search(r'add (\d+) (\d+)', line):
do_add(m[0], m[1])
elif m.search(r'mult (\d+) (\d+)', line):
do_mult(m[0], m[1])
elif m.search(r'help (\w+)', line):
show_help(m[0])

"""
def search(self, r, s):
"""
Do a regular expression search and return if it matched.
"""
self.value = re.search(r, s)
return self.value
def __getitem__(self, n):
"""
Return n'th matched () item.

Note so the first matched item will be matcher[0]
"""
return self.value.group(n+1)
def groups(self):
"""
Return all the matched () items.
"""
return self.value.groups()

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: HTML Generation

2009-04-16 Thread Nick Craig-Wood
J Kenneth King  wrote:
>  Stefan Behnel  writes:
> > See here for another example that uses lxml.html:
> >
> > http://codespeak.net/lxml/lxmlhtml.html#creating-html-with-the-e-factory
> >
> > Stefan
> 
>  Ah, looks good. Have never used nor finished the example I had given --
>  only meant as inspiration. I'm not surprised it has been done by someone
>  else.

Something more like what the OP wanted is HTMLgen.

I'm not sure it is maintained any more, but I used it quite a lot a
few years ago.

Debian still have the source and a package here

  http://packages.debian.org/sid/python-htmlgen

But I think its original website is gone.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


python docs redirect on python.org is old

2009-04-04 Thread Brandon Craig Rhodes
When I visit ...

   http://www.python.org/doc/lib/lib.html

... I get redirected to ...

   http://www.python.org/doc/2.5.2/lib/lib.html

... which seems a bit old.

-- 
Brandon Craig Rhodes   bran...@rhodesmill.org   http://rhodesmill.org/brandon

--
http://mail.python.org/mailman/listinfo/python-list


Re: Sending SMS using python script

2009-04-03 Thread Craig
There's a Python wrapper to the Skype API here:
http://sourceforge.net/projects/skype4py/
On Linux I've used the PyGTK GUI that uses this. It's called
SkySentials here:
http://www.kolmann.at/philipp/linux/skysentials/

Craig

On Apr 3, 6:50 am, "ISF (Computer Scientists without Frontiers,
Italy)"  wrote:
> On 2 Apr, 06:41, guptha  wrote:
>
>
>
> > hi group,
> > my application needs to sendSMSoccasionally to all the clients  .Is
> > there any library in python that supports in sendingSMS.
> > I like to conform few information i gathered in this regard.
>
> > I can sendSMSby two ways
>
> > 1. SendingSMSusing Email clients
> > 2. Usingsmsgateway to send message(we can implementSMSGateway API
> > 's ,provided by vendor and ,sendSMS-- we will be charged
> > accordingly )
>
> > In case of First approach
> > 1. We can make use of libgamil library to sendSMSusing gmail ( I
> > ref :http://blog.datasingularity.com/?p=63)
> > i suppose sendingsmsthrough gmail is not supported in India
> >  2. Can we use Skype4py library,
>
> > In case of second approach
> > 1. Is there any way to sendSMSfor free inside India ,or ,Any freeSMSgateway 
> > providers in India
> > Any information regarding this is appreciable
> >   Thanks
>
> A friend from India sent me this hint:
>
> the following link can be explored further for sending SMS on pyS60
> (python for symbian OS)http://mobilenin.com/pys60/menu.htm
>
> wkr,
> Aldo

--
http://mail.python.org/mailman/listinfo/python-list


Re: Beazley on Generators

2009-04-01 Thread Craig Allen
this is great, thanks... we have used generators to create something
akin to a cooperative tasking environment... not to implement
multitasking, but to be able to control low level data processing
scripts.  These scripts, written as generators, yield control to a
control loop which then can pause, resume, abort, or change the state
of shared context objects which the script uses as it's input and
output space.  E.g. the control loop can see there is intermediate
output which an operator (managing a data reduction pipeline) might
want to see.

I can see from the first few slide I need to understand this. It
already seems clear that there are ways to improve our approach to
what we have done, though the overall approach is solid and works
well.

anyway thanks.

--
http://mail.python.org/mailman/listinfo/python-list


Re: double/float precision question

2009-04-01 Thread Nick Craig-Wood
TP  wrote:
>  Hi everybody,
> 
>  Try the following python statements:
> 
> >>> "%.40f" % 0.222
>  '0.098864108374982606619596'
> >>> float( 0.222)
>  0.1
> 
>  It seems the first result is the same than the following C program:
>  
>  #include 
> 
>  int main(void)
>  {
>  double a = 0.222;
> 
>  printf( "%.40f\n", a );
>  return 0;
>  }
>  #
> 
>  My problem is the following:
>  * the precision "40" (for example) is given by the user, not by the
>  programmer.
>  * I want to use the string conversion facility with specifier "e", that
>  yields number is scientific format; so I cannot apply float() on the result
>  of "%.40e" % 0.222, I would lost the scientific
>  format.
> 
>  Is there any means to obtain the full C double in Python, or should I limit
>  the precision given by the user (and used in "%.*e") to the one of a Python
>  float?

Python floats are actually C doubles (as you proved yourself with your
little test program).

Eg

>>> 1.+2.**-52
1.0002

>>> 1.+2.**-53
1.0

Indicating that python floats have about 52 bits of precision, so are
definitely what C calls doubles.

When you do

>>> float( 0.222)
0.1

Python prints as many decimal places as are significant in the answer.

This is covered in the FAQ

http://www.python.org/doc/faq/general/#why-are-floating-point-calculations-so-inaccurate

If you want more precision use the built in decimal module or the
third party gmpy module.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: A design problem I met again and again.

2009-04-01 Thread Nick Craig-Wood
一首诗  wrote:
>  But I think the first step to resolve a problem is to describe it.  In
>  that way, I might find the answer myself

:-) That is a great saying!

To answer your original question, split your code up into sections
that can be tested independently.  If you can test code in a isolated
way then it belongs in a class / module of its own.

If you have a class that is too big, then factor independent classes
out of it until it is the right size.  That is easier said than done
and may require some creativity on your part.  It will pay dividends
though as the level of abstraction in your program will rise.

I've noticed some programmers think in big classes and some think in
small classes.  Train yourself to do the other thing and your
programming will improve greatly!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Ordered Sets

2009-03-30 Thread Nick Craig-Wood
Aahz  wrote:
> I find the trick of using a Python list to store the doubly-linked
> list difficult to understand (as opposed to the usual mechanism of a
> node class).  I understand why it was done (time and space
> efficiency), but I also still feel emotionally that it's somewhat
> sick and perverted.  I probably would feel more comfortable if the
> doubly-linked list were abstracted out and commented.  Heck, the
> whole thing needs unit tests.

Hmmm... Lets compare the two.

Here is the length three list

$ python
Python 2.5.2 (r252:60911, Jan  4 2009, 17:40:26)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> L = []
>>> for i in xrange(100):
... L.append([1,2,3])
...
>>> import os
>>> os.getpid()
28134
>>>

(from top)
28134 ncw   20   0 58860  53m 1900 S0  2.6   0:02.62 python

vs a Node class with __slots__ (for efficiency)

$ python
Python 2.5.2 (r252:60911, Jan  4 2009, 17:40:26)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class Node(object):
... __slots__ = ["prev", "next", "this"]
... def __init__(self, prev, next, this):
... self.prev = prev
... self.next = next
... self.this = this
...
>>> Node(1,2,3)
<__main__.Node object at 0xb7e897cc>
>>> Node(1,2,3).prev
1
>>> L = []
>>> for i in xrange(100):
... L.append(Node(1,2,3))
...
>>> import os
>>> os.getpid()
28203
>>>

(from top)
28203 ncw   20   0 43364  38m 1900 S0  1.9   0:04.41 python

So the Node class actually takes less memory 38 Mbytes vs 53 Mbytes for
the list.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Programming Python 4th Edition?

2009-03-29 Thread Nick Craig-Wood
Esmail  wrote:
>  prueba...@latinmail.com wrote:
> > It isn't a introduction to the Python language like "Learning Python",
> > it doesn't work as reference like "Python in a Nutshell", it doesn't
> > contain short idiomatic code like "Python Cookbook". What you are left
> > with is different application domains and how to apply Python to them.
> > The book is excellent if you want to do Network, GUI, Databases, etc.
> > but poor if you want to learn about Python the core language. The
> > title of the book should be changed from "Programming Python" to
> > "Applied Python" 
> 
>  I appreciate you taking the time to post. I agree with what you say.
>  I guess what appeals to me is the nearly encyclopedic nature of the
>  book .. and I am curious about scripting with python, so it seems to
>  have some good material on it (though I think there are newer modules
>  now available for this).
> 
>  It's good to hear what others think about this book, and others
>  too.

I read Programming Python as an experienced programmer and like you I
enjoyed the encyclopedic nature of it.  So if it appeals to you I'd
say go for it!

The fact that it doesn't use the latest version of python isn't a
problem - python doesn't change very quickly and emphasises backwards
compatibility, even for the jump to 3.x.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: New Python Runtime / Unladen Swallow

2009-03-28 Thread Nick Craig-Wood
Adonis  wrote:
>  Came across this article on Ars on a new LLVM (Low Level Virtual 
>  Machine) JIT compiler for Python being built by Google:
> 
>  
> http://arstechnica.com/open-source/news/2009/03/google-launches-project-to-boost-python-performance-by-5x.ars

Interesting article and site. The projct is called Unladen Swallow.
(I'd write a link but I can't copy and paste them on my phone!)

The idea seems to be to gradually transform python (2.6) into using
LLVM as its VM instead of the python VM it uses at the moment.

They've already managed to compile python into LLVM (there is a LLVM
version of gcc) and speed it up a bit.  The next step is to retarget the
python byte code generator into producing LLVM bytecode instead.

The advantage of targetting LLVM is that it is cross platform and
already has JIT and full compilers for many platforms.

They plan to fold their work back into CPython when done too.

Sounds like a project to keep an eye on!

>  Now the question is will this make Vista run faster?

Nothing could do that ;-)

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Interfacing python and C

2009-03-28 Thread Nick Craig-Wood
MRAB  wrote:
>  steve William wrote:
> > Hi All,
> > 
> > I'm using SWIG for the first time and I am facing some problems with 
> > user defined header files. I'm trying to use my own header file in a C 
> > program which would be interfaced with python.
> > 
> > The header file is test.h:
> > /#include 
> > 
> > int fact(int n) {
> >  if (n <= 1) return 1;
> >  else return n*fact(n-1);
> > }/
> >
> > The C program is test1.c:
> > /#include 
> > #include "test.h"
> > 
> > int calc_fact(int a)
> > {
> > return (fact(a));
> > }/
> > 
> > The interface file is test1.i:
> > /%module test1
> > 
> > %{
> > #include "stdio.h"
> > #include "test.h"
> >  %}
> > 
> > int calc_fact(int a);/
> > 
> > The commands that I used to generate the wrappers are:
> > /swig -python test1.i
> > gcc -c test1.c test1_wrap.c -I/usr/include/python2.5 
> > -I/usr/lib/python2.5/config
> > g++ -shared test1_wrap.o -o _test1.so/

gcc not g++ here

link test1.o also

gcc -shared test1.o test1_wrap.o -o test1.so

> > When I try to import test1, I get an error saying:
> > />>> import test1
> > Traceback (most recent call last):
> >   File "", line 1, in 
> >   File "test1.py", line 22, in 
> > import _test1
> > ImportError: ./_test1.so: undefined symbol: calc_fact/

Try nm or objdump on the .so to see what symbols are in it.

> > I'm not sure why this is happening because when I try without user 
> > defined header file, it works. Also the error says that it does not 
> > recognize /calc_fact/ which is a function that I want to access from 
> > python and is declared in the interface file.
> > 
> > Is there any specific way in which user defined headers need to be 
> > declared in the interface file? Should the user defined header be placed 
> > in the /usr/include  directory?
> > 
> > Any help on this is highly appreciated.

My advice to you is to compile the C stuff into a .so and use ctypes
instead of swig.  You then write the interface code in python not C
and you'll have a lot more fun!

cython is very useful in this area too provided you don't mind an
extra dependency.  If you are developing C code from scratch to
use with python, then write it in cython instead!

>  Should you be putting a function body in a header file?

No

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: How to port Python to an ARM running NucleusPlus OS

2009-03-28 Thread Nick Craig-Wood
Thomas Ng  wrote:
>  I'm planning to port the Python to run on an embedded device running
>  Nucleus+ OS on ARM7 processor.
> 
>  Can anyone help to advise how to get started?
> 
>  Where I am now: I've downloaded the Python 2.6 from SVN and able to
>  compile it using VC++. But no idea how to proceed from here. Since I
>  am not familiar with Linux, I would prefer to do the porting work on
>  Windows.

Well... Nucleus isn't Linux, it is a RTOS.  If it has enough POSIX
interfaces you probably can port python but I wouldn't have thought it
would be easy.  Try to compile python in the cross compiling
environment and see what happens!

However if you are running Nucleus with Linux and want to run python
in the Linux bit of it then I'd suggest to use the packages available
for the Linux side of it.  (Eg if it is running debian then apt-get
install python).

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: C extension using GSL

2009-03-28 Thread Nick Craig-Wood
Gabriel Genellina  wrote:
>  En Fri, 27 Mar 2009 03:10:06 -0300, jesse  escribió:
> 
> > I give up. I cannot find my memory leak! I'm hoping that someone out
> > there has come across something similar. Let me lay out the basic
> > setup:
> > [...]
> > 4) C: A PyList object, L,  is created (new reference!). This will hold
> > the solution vector for the ODE
> > [...]
> > 7) C: Return L to Python with return Py_BuildValue("N", L).
> 
>  I don't know the "N" format, but if Py_BuildValue returns a new reference,  
>  you're leaking a reference to the L list.
>  Why don't you return L directly?
> 
>  You can use sys.getrefcount in Python to see the reference count for an  
>  object:
> 
>  py> from sys import getrefcount as rc
>  py> x = object()
>  py> rc(x)
>  2   # the name x, and a temporary reference as parameter
>  py> rc([])
>  1   # only the temporary reference
>  py> x = y = []
>  py> rc(x)
>  3
>  py> x = ()
>  py> rc(x)
>  954   # the empty tuple is shared

That reminds me, you can use the gc module to show all your objects
that are in use, which can help with memory leaks.

eg something like

http://code.activestate.com/recipes/457665/

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Code anntotations (copyright, autor, etc) in your code

2009-03-28 Thread Nick Craig-Wood
mattia  wrote:
>  Hi all, which are the usual comments that you put at the beginning of 
>  your code to explain e.g. the author, the usage, the license etc?
> 
>  I've found useful someting like:
>  
> #-
>  # Name:About.py
>  # Purpose:
>  #
>  # Author:  
>  #
>  # Created: 2009
>  # Copyright:   (c) 2009
>  # Licence: GPL
>  
> #-
> 
>  others put something like
> 
>  __author__ = "Name Surname"
>  __year__   = 2009
> 
>  What do you use?

__version__ is suggested in PEP 8

http://www.python.org/dev/peps/pep-0008/

__author__,  __date__ and __credits__ are both understood by pydoc but
I haven't actually seen that in a document only in the source code!

I think the others are just conventions and are not actually used by
anything, but I'd be interested to be proved wrong!

I tend to use

__author__ = "Nick Craig-Wood "
__version__ = "$Revision: 5034 $"
__date__ = "$Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $"
__copyright__ = "Copyright (c) 2008 Nick Craig-Wood"

With __version__ and __date__ being version control tags (in this case
svn)

Pydoc produces this from the above

----
[snip]

DATA
__author__ = 'Nick Craig-Wood '
__copyright__ = 'Copyright (c) 2008 Nick Craig-Wood'
__date__ = '$Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $'
__version__ = '$Revision: 5034 $'

VERSION
5034

DATE
$Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $

AUTHOR
Nick Craig-Wood 


pydoc pydoc shows an example of what __credits__ looks like

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: c.l.py dead, news at 11 (was Re: Mangle function name with decorator?)

2009-03-27 Thread Nick Craig-Wood
Aahz  wrote:
>  Well, yes, but that's simply the nature of online fora (I originally
>  wrote "nature of Usenet", but I think it's more general than that).  From
>  my POV, if you're going to call it a "decline", you need to provide more
>  evidence than some people leaving and others arriving.  I think that the
>  sheer volume and quality of posts to c.l.py is evidence that c.l.py is
>  not declining.

c.l.py is my favourite usenet group and has been for some time.  I've
been doing usenet for 16 years now!

I enjoy reading about problems and problems solved.  I enjoy helping
people where I can and especially learning new things when helping
people - I think it has contributed enormously to my development as a
python programmer.

When I ask a question of c.l.py I find the answers to be informative
and enlightening.  Even after quite a few years of python programing
I'm still learning new things from c.l.py

As a long time usenet user I find it easy to ignore the occasional
flame wars.  Posters with the wrong sort of attitude are brought
gently into line by the majority.

If usenet groups had ratings I'd give c.l.py 5 stars.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: C extension using GSL

2009-03-27 Thread Nick Craig-Wood
jesse  wrote:
>  I give up. I cannot find my memory leak! I'm hoping that someone out
>  there has come across something similar. Let me lay out the basic
>  setup:
> 
>  I'm performing multiple simulations on a model. Each iteration
>  involves solving a system of differential equations. For this I use
>  the GNU Scientific Library (GSL) -- specifically the rk4imp ODE
>  solver. After the ODE is solved the array is returned to python and is
>  analyzed. As you may have guessed, my problem is that over the course
>  of the iterations the memory keeps climbing until python crashes.
> 
>  Note: *The extension does not keep running.* It returns object (a
>  list) and is done until the next iteration. AFAIK, any memory
>  allocated during execution of the extension should be released.
>  Question: Since the extension was run from within python is memory
>  allocated within an extension part of python's heap?

No, on the normal C heap

>  Would this have
>  an adverse or unpredictable affect on any memory allocated during the
>  running of the extension?

If the library passes you data and ownership of a heap block, you'll
need to free it.


> One hypothesis I have, since most other
>  possibilities have been eliminated, is that GSL's creation of it's own
>  data structures (eg., gsl_vector) is messing up Python's control of
>  the heap. Is this possible?

Only if GSL is buggy

> If so, how would one be able to fix this
>  issue?

Valgrind
 
>  It may help some nice folks out there who are good enough to look at
>  this post if I layout the flow, broken up by where stuff happens
>  (i.e., in the Python code or C code):
> 
>  1) Python: Set up the simulation and basic data structures (numpy
>  arrays with initial conditions, coupling matrices for the ODE's, dicts
>  with parameters, etc).
>  2) Python: Pass these to the C extension
>  3) C: Python objects passed in are converted to C arrays, floats,
>  etc.
>  4) C: A PyList object, L,  is created (new reference!). This will hold
>  the solution vector for the ODE
>  5) C: Initialize GSL ODE solver and stepper functions. Solve the ODE,
>  at each step use PyList_Append(L, current_state) to append the current
>  state to L.
>  6) C: After the ODE solver finishes, free GSL objects, free coupling
>  matrices, free last state vector.
>  7) C: Return L to Python with return Py_BuildValue("N", L).
>  8) Python: Convert returned list L to array A, delete L, work with A.
>  8.1) Python: Step 8) includes plotting. (I have a small suspicion that
>  matplotlib holds onto lots of data, but I've used clf() and close(fig)
>  on all plots, so I think I'm safe here. )
>  8.2) Python: save analysis results from A, save A. (At this point
>  there should be no more use of A. In fact, at point 8) in the next
>  iteration A is replaced by a new array.)
>  9) Python: Change any parameters or initial conditions and goto 1).

At every point a memory allocation is made check who owns the memory
and that it is freed through all possible error paths.

Valgrind will help you find the memory leaks.  It works well once
you've jumped the flaming hoops of fire that setting it up is!

Another thing you can try is run your process untill it leaks loads,
then make it dump core.  Examine the core dump with a hex editor and
see what it is full of!  This technique works suprisingly often.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Bash-like brace expansion

2009-03-24 Thread Nick Craig-Wood
Peter Waller  wrote:
>  Okay, I got fed up with there not being any (obvious) good examples of
>  how to do bash-like brace expansion in Python, so I wrote it myself.
>  Here it is for all to enjoy!

Interesting!

Might I suggest some unit tests?  You can then test all the corner
cases (unmatched brackets, empty brackets, etc) and be sure it works
exactly as specified.  doctest is cool for this kind of stuff.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: udp package header

2009-03-24 Thread Nick Craig-Wood
mete  wrote:
>  I got a problem. İ want to send udp package and get this package (server and 
>  clinet ). it's easy to python but i want to look the udp header how can i 
>  do ?

There is pretty much nothing in a UDP packet header except
from_address, to_address, from_port and to_port.  You should already
know to_address and to_port and socket.recv will give you the
from_address and from_port

socket.recvfrom(bufsize[, flags])

Receive data from the socket. The return value is a pair (string,
address) where string is a string representing the data received
and address is the address of the socket sending the data.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Using python 3 for scripting?

2009-03-23 Thread Nick Craig-Wood
Alan G Isaac  wrote:
>  On 3/22/2009 12:41 PM Chris Rebert apparently wrote:
> > 2.6.1, the latest non-3.x release is probably best. Most libraries
> > haven't been ported to 3.x yet, so Python 3 has yet to become
> > widespread.
> 
>  This seems slightly optimistic to me.  Until a week ago, there was
>  not a NumPy release for 2.6.  There is still not a SciPy release
>  for 2.6.  Most dismaying, the SimpleParse guys need a little help
>  compiling SimpleParse for 2.6 and have not yet gotten that help.
>  (Is anyone listening?)
> 
>  So 2.5.4 is still perhaps the safest bet, even though it is more
>  awkward for writing code close to Python 3 syntax.

I tend to target whatever is in Debian stable, which starting from
this month is 2.5 (recently upgraded from 2.4).

2.6 or 3.x is nowhere to be seen in Debian stable, testing or unstable
:-(

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Async serial communication/threads sharing data

2009-03-23 Thread Nick Craig-Wood
Jean-Paul Calderone  wrote:
>  On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood  
> wrote:
> >Jean-Paul Calderone  wrote:
> > [snip]
> >>
> >>  In the case of a TCP to serial forwarder, you don't actually have to
> >>  implement either a producer or a consumer, since both the TCP connection
> >>  and the serial connection are already both producers and consumers.  All
> >>  you need to do is hook them up to each other so that when the send buffer
> >>  of one fills up, the other one gets paused, and when the buffer is empty
> >>  again, it gets resumed.
> >
> >I eventually came up with this which seems to work, but I'm not sure
> >it is the best way of doing it as it had to mess about with the
> >twisted internals to get the number of bytes in the serial port
> >output buffer.
> 
>  This is sort of on the right track.  Here's how to do it without
>  touching implementation details:

Thank you for that!

See below for the complete prog with your suggested modifications.

That seems to solve the problem - I can see the writer pausing and
unpausing at the serial port rate.

  write 16446
  write 6430
  pause producing
  resume producing
  write 65536
  write 56724
  pause producing
  resume producing
  write 65536
  write 65536
  pause producing

It has exposed a problem with the sender not throttling properly now,
but I guess that is progress!

Thanks for your help

Here is the complete program FYI with your suggested mods.

#!/usr/bin/python
"""Transfer data between a serial port and one (or more) TCP
connections.
options:
-h, --help:this help
-p, --port=PORT: port, a number, default = 0 or a device name
-b, --baud=BAUD: baudrate, default 115200
-t, --tcp=PORT: TCP port number, default 1234
-l, --log: log data streams to 'snifter-0', 'snifter-1'
-L, --log_name=NAME: log data streams to '-0', '-1'
"""

import sys
import getopt
from twisted.internet import reactor, protocol, serialport
from zope.interface import implements
from twisted.internet import protocol, interfaces

# FIXME set serial buffer size? SEND_LIMIT

class SerialPort(protocol.Protocol):
"""Create a serial port connection and pass data from it to a
known list of
TCP ports."""

def __init__(self, port, reactor, baudrate, log_name=None):
self.tcp_ports = []
self.serial = serialport.SerialPort(self, reactor, port,
baudrate, rtscts=0)
self.serial.registerProducer(self, True)
self.paused = False
self.log = None
if log_name is not None:
self.log = file('%s-0' % log_name, 'w')

def add_tcp(self, tcp_port):
"""Add a TCPPort to those receiving serial data."""
if self.paused:
tcp_port.transport.pauseProducing()
self.tcp_ports.append(tcp_port)

def del_tcp(self, tcp_port):
"""Remove a TCPPort from the those receiving serial data."""
self.tcp_ports.remove(tcp_port)

def write(self, data):
"""Write data to the serial port."""
self.serial.write(data)
if self.log:
self.log.write(data)

def pauseProducing(self):
"""Pause producing event"""
print "pause producing"
self.paused = True
for port in self.tcp_ports:
port.transport.pauseProducing()

def resumeProducing(self):
"""Resume producing event"""
print "resume producing"
self.paused = False
for port in self.tcp_ports:
port.transport.resumeProducing()

def stopProducing(self):
"""Stop producing event"""
print "stop producing"

def dataReceived(self, data):
"""Pass any received data to the list of TCPPorts."""
for tcp_port in self.tcp_ports:
tcp_port.write(data)

class TCPPort(protocol.Protocol):
"""Create a TCP server connection and pass data from it to the
serial port."""

def __init__(self, serial, log_name, index):
"""Add this TCPPort to the SerialPort."""
self.serial = serial
self.serial.add_tcp(self)
self.log = None
if log_name is not None:
self.log = file('%s-%d' % (log_name, index+1), 'w')

def __del__(self):
"""Remove this TCPPort from the SerialPort."""
self.serial.del_tcp(self)

def dataReceived(self, data):
"""Pass received data

Re: Async serial communication/threads sharing data

2009-03-23 Thread Nick Craig-Wood
Jean-Paul Calderone  wrote:
>  On Sun, 22 Mar 2009 12:30:04 -0500, Nick Craig-Wood  
> wrote:
> >I wrote a serial port to TCP proxy (with logging) with twisted.  The
> >problem I had was that twisted serial ports didn't seem to have any
> >back pressure.  By that I mean I could pump data into a 9600 baud
> >serial port at 10 Mbit/s.  Twisted would then buffer the data for me
> >using 10s or 100s or Megabytes of RAM.  No data would be lost, but
> >there would be hours of latency and my program would use up all my RAM
> >and explode.
> >
> >What I wanted to happen was for twisted to stop taking the data when
> >the serial port buffer was full and to only take the data at 9600
> >baud.
> 
>  This is what Twisted's producers and consumers let you do.  There's a
>  document covering these features:
> 
>  http://twistedmatrix.com/projects/core/documentation/howto/producers.html
> 
>  In the case of a TCP to serial forwarder, you don't actually have to
>  implement either a producer or a consumer, since both the TCP connection
>  and the serial connection are already both producers and consumers.  All
>  you need to do is hook them up to each other so that when the send buffer
>  of one fills up, the other one gets paused, and when the buffer is empty
>  again, it gets resumed.

I eventually came up with this which seems to work, but I'm not sure
it is the best way of doing it as it had to mess about with the
twisted internals to get the number of bytes in the serial port
output buffer.

class SerialPort(protocol.Protocol):
"""Create a serial port connection and pass data from it to a known list of 
TCP ports."""

def __init__(self, port, reactor, baudrate, log_name=None):
self.tcp_ports = []
self.serial = serialport.SerialPort(self, reactor, port, baudrate, 
rtscts=0)
self.log = None
if log_name is not None:
self.log = file('%s-0' % log_name, 'w')

def write(self, data):
"""Write data to the serial port."""
self.serial.write(data)
if self.log:
self.log.write(data)
self.throttle()

def throttle(self):
"""
Pause the inputs if there is too much data in the output buffer
"""
bytes_in_buffer = len(self.serial.dataBuffer) + self.serial._tempDataLen
too_full = bytes_in_buffer > 1024
for tcp_port in self.tcp_ports:
if too_full:
tcp_port.transport.pauseProducing()
else:
tcp_port.transport.resumeProducing()
if too_full:
reactor.callLater(0.1, self.throttle)

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Async serial communication/threads sharing data

2009-03-22 Thread Nick Craig-Wood
Jean-Paul Calderone  wrote:
>  It's true that the serial port support in Twisted isn't the most used
>  feature. :)  These days, serial ports are on the way out, I think.  That
>  said, much of the way a serial port is used in Twisted is the same as the
>  way a TCP connection is used.  This means that the Twisted documentation
>  for TCP connections is largely applicable to using serial ports.  The only
>  major difference is how you set up the "connection".  You can see examples
>  of using the serial port support here (one of them seems to suggest that
>  it won't work on Windows, but I think this is a mistake):
> 
>http://twistedmatrix.com/projects/core/documentation/examples/gpsfix.py
>http://twistedmatrix.com/projects/core/documentation/examples/mouse.py
> 
>  The 2nd to last line of each example shows how to "connect" to the serial
>  port.  These basic documents describe how to implement a protocol.  Though
>  in the context of TCP, the protocol implementation ideas apply to working
>  with serial ports as well:
> 
>http://twistedmatrix.com/projects/core/documentation/howto/servers.html
>http://twistedmatrix.com/projects/core/documentation/howto/clients.html
> 
>  You can ignore the parts about factories, since they're not used with serial
>  ports.

I wrote a serial port to TCP proxy (with logging) with twisted.  The
problem I had was that twisted serial ports didn't seem to have any
back pressure.  By that I mean I could pump data into a 9600 baud
serial port at 10 Mbit/s.  Twisted would then buffer the data for me
using 10s or 100s or Megabytes of RAM.  No data would be lost, but
there would be hours of latency and my program would use up all my RAM
and explode.

What I wanted to happen was for twisted to stop taking the data when
the serial port buffer was full and to only take the data at 9600
baud.

I never did solve that problem :-(

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: script files with python (instead of tcsh/bash)?

2009-03-22 Thread Nick Craig-Wood
Esmail  wrote:
>  thanks for including the script, that really helps. Nice way of
>  finding files.

Python has lots of useful stuff like that!

>  Two quick questions:
> 
>  As a replacement for grep I would use the re module and its
>  methods?

The re module works on strings not files, but basically yes.

Note that the re module uses a superset of the "grep -E" regexps, and
they are almost identical to those used in perl and php.

Here is a simple grep in python

#!/usr/bin/python
import re
import sys
import fileinput
pattern = sys.argv.pop(1)
for line in fileinput.input():
if re.search(pattern, line):
print line.rstrip()

Save in a file called grep.py then you can do

$ ./grep.py dizzy *.py
self.dizzy = 0
if self.dizzy:

$ ls | ./grep.py '\d{2}'
linux-image-2.6.24-21-eeepc_2.6.24-21.39eeepc1_i386.deb
linux-ubuntu-modules-2.6.24-21-eeepc_2.6.24-21.30eeepc6_i386.deb

>  What about awk which I regularly use to extract fields based on position
>  but not column number, what should I be using in Python to do the
>  same?

I presume you mean something like this

   ... | awk '{print $2}'

In python, assuming you've got the line in the "line" variable, then

In python an equivalent of the above would be

import fileinput
for line in fileinput.input():
print line.split()[1]

Note that the fileinput module is really useful for making shell
command replacements!

>  The other things I need to do consist of moving files, manipulating file
>  names and piping outputs of one command to the next, so I'm digging into
>  the documentation as much as I can.

Read up on the os module and the subprocess module.  You'll find you
need to do much less piping with python as with shell because it has
almost everything you'll need built in.

Using built in functions is much quicker than fork()-ing an external
command too.

>  So much to learn, so little time (but so much fun!)

;-)

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: 3.0 - bsddb removed

2009-03-22 Thread Nick Craig-Wood
Sean  wrote:
>  Anyone got any thoughts about what to use as a replacement.  I need 
>  something (like bsddb) which uses dictionary syntax to read and write an 
>  underlying (fast!) btree or similar.

sqlite.

bsddb gave me no end of trouble with threads, but sqlite worked
brilliantly.

You would need to make a dictionary interface to sqlite, eg

  http://code.activestate.com/recipes/576638/

Or do something a bit simpler yourself.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: script files with python (instead of tcsh/bash)?

2009-03-22 Thread Nick Craig-Wood
Esmail  wrote:
>  I am wondering if anyone is using python to write script files?

Yes!

>  Right now I have a bigg'ish bash/tcsh script that contain some grep/awk
>  command plus various files are processed and created, renamed and
>  moved to specific directories. I also write out some gnuplot scripts
>  that later get executed to generate .jpg images.

Almost any script that contains a loop I convert into python.

>  In any case, the scripts are starting to look pretty hairy and I was
>  wondering if it would make sense to re-write them in Python. I am not
>  sure how suitable it would be for this.

With python you get the advantages of a language with a very clear
philosophy which is exceptionally easy to read and write.

Add a few classes and unit tests to your scripts and they will be
better than you can ever achieve with bash (IMHO).

>  I've looked around the web w/o much luck for some examples but come
>  short. Any comments/suggestions?

Here is a short script I wrote to recover my slrn spool when my disk
fills up.  I could have written it in bash but I think it is much
clearer as python, and doesn't shell out to any external commands.

It searches through the spool directory looking for empty .minmax
files.  It then finds the lowest and highest numeric file name (all
the files are numeric) and writes the minmax file with that.

#!/usr/bin/python
"""
rebuild all the .minmax files for slrnpull after the disk gets full
"""

import os

spool = "/var/spool/slrnpull"

def main():
for dirpath, dirnames, filenames in os.walk(spool):
if ".minmax" not in filenames:
continue
minmax_path = os.path.join(dirpath, ".minmax")
if os.path.getsize(minmax_path) != 0:
print "Skipping non empty %r" % minmax_path
continue
print dirpath
digits = [ int(f) for f in filenames if f.isdigit() ]
if not digits:
digits = [0]
digits.sort()
start = digits[0]
end = digits[-1]
f = open(minmax_path, "w")
    f.write("%s %s" % (start, end))
f.close()
print "done"

if __name__ == "__main__": main()


-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: cross compile Python to Linux-ARM

2009-03-19 Thread Nick Craig-Wood
jefm  wrote:
>  We are looking to use Python on an embedded Linux ARM system.
>  What I gather from googling the subject is that it is not that
>  straight forward (a fair amount of patching & hacking).
>  Nobody out there that has done it claims it is easy, which makes me
>  worried.
> 
>  I haven't seen a description on porting Python 2.6 or 3.0 yet. Is it
>  much different than for the earlier versions (the latest I have seem
>  is Python 2.5).
> 
>  Does it matter whether Python is cross compiled to Linux 2.4 or Linux
>  2.6 ?
> 
>  Can anyone point to a howto they know works well ?

I gave up trying to cross compile and just used the python from debian
ARM which works very well.

>  What are the chances of an 'officially' supported ARM-Linux Python
>  distribution ?

Depending on your definition of "officially" and "supported" :-

  http://packages.debian.org/lenny/arm/python2.5/download

>  (or is it safer to wait for industrial spec Intel Atom boards to avoid
>  the cross compilation altogether ?

We often compile stuff on our 200 MHz TS-7200 boards.  We NFS mount
the build environment, type make then go out for lunch ;-)

For builds of our application (which embeds python) we normally use
the headers and libraries from the debian ARM packages and cross
compile on something a big beefier.

>  How is the performance and stability of a working Python on an
>  embedded ARM-Linux system ?

Works very well.

>  Does cross compiling Python automatically include the standard
>  Python library, or is that yet another adventure ?

If you use the debian compiled version then you get the lot.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Python to Perl transalators

2009-03-18 Thread Nick Craig-Wood
Armin  wrote:
>  On Wednesday 18 March 2009 11:01:00 Boris Borcic wrote:
> > Armin wrote:
> > >> 
> > >> Why on earth would you want to? That'd be like translating Shakespeare
> > >> into a bad rap song!
> > >> 
> > >
> > > lol, actually I would prefer a rap song over Shakespeare, so your analogy
> > > doesn't work there ;)
> >
> > Why, some people do prefer Perl over Python, so what's wrong with the
> > analogy ?
> >
> 
>  That would mostly apply to people who haven't coded in Python for one reason 
>  or another.

Perhaps the OP is looking for something like this

  http://pleac.sourceforge.net/pleac_python/index.html

Which is a sort of Rosetta stone for perl and python ;-)

(The perl cookbook translated into python.)

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: packaging

2009-03-18 Thread Craig Allen

> andrew

thanks andrew, good advice, I should probably use that throughout our
code.

btw, hope the world is treating you well, long time no see...

-craig
--
http://mail.python.org/mailman/listinfo/python-list


packaging

2009-03-17 Thread Craig Allen
we have software we are putting into package form. So far, all the
code was in local py files and we imported between the modules as
you'd think.  Now with the package ("ourpackage") we are addressing
how import affects the importing module.

if "ourpackage" __init__.py itself does regular imports of the key
modules, like "ourmodule" (containing class OurClass)... it's a bit of
a pain for the user.

one imports ourpackage, and then need to get OurClass from ourmodule
still, i.e.:

  import ourpackage
  inst = ourpackage.ourmodule.OurClass()

Instead, I think we want "import package" to preserve the sort of
namespace our loose python files provided, so:

  import ourpackage
  inst = ourpackage.OurClass()

I think the way to do this, and it seems a legit use of a somewhat
dangerous form of import, to in ourpackage's __init__.py do this:

  from ourmodule import *

So that the second form works.  If anyone has a comment on that I'm
interested, either that it won't work, or to contradict my idea that a
wildcarded import is appropriate in this place as we are trying to
fill a flattened namespace.

But the reason I'm asking is that it's also been suggested that we
should include everything in a single module, so, ourpython1.py and
ourpython2.py, etc, all get moved to ourpython.py.  I very much
dislike that idea for various (probably obvious) reasons.

On the other hand I do want to adopt whatever are de facto standards
for this sort of thing (especially from the user pov but also from the
developer's)... so any comment are appreciated.  I've been using
python for a few years now but this is the first time we are forming
it in the shape of a proper package.

cheers and thanks.
-craig
--
http://mail.python.org/mailman/listinfo/python-list


Re: Special keyword argument lambda syntax

2009-03-14 Thread Nick Craig-Wood
Beni Cherniavsky  wrote:
>  This proposal outrageously suggests a special syntax for in-line
>  functions passed as keyword arguments::
> 
>  >>> sorted(range(9), key(n)=n%3)
>  [0, 3, 6, 1, 4, 7, 2, 5, 8]
> 
>  The claim is that such specialization hits a syntax sweet spot, and
>  that this use case is frequent enough to contemplate eventually making
>  it the only in-line function syntax.

-1 from me.

I think that lambda / inline functions should be discouraged as it
moves python away from, "there should be one-- and preferably only one
--obvious way to do it."  IMHO Guido was right in his original impulse
to kill this second class way of making functions...

I would write the above as

def compute_key(n):
"Compute the sort key so that x, y and z are true"
return n % 3
sorted(range(9), key=compute_key)

Which I think is clearer and more obvious.  It gives you the
opportunity for a docstring also.

Yes it is a bit more typing, but who wants to play "code golf" all
day?

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Raw String Question

2009-03-12 Thread Nick Craig-Wood
Jim Garrison  wrote:
>   >>> r"a\b"
>'a\\b'
>   >>> r"a\"
>SyntaxError: EOL while scanning string literal (, line 1)
>   >>> r"a\ "
>'a\\ '
>   >>> r"a\""
>'a\\"'
> 
>  It seems the parser is interpreting the backslash as an escape
>  character in a raw string if the backslash is the last character.
>  Is this expected?

Yes

http://docs.python.org/reference/lexical_analysis.html#string-literals

  Specifically, a raw string cannot end in a single backslash (since
  the backslash would escape the following quote character).

The usual way round this is like this

>>> r"a" "\\"
'a\\'
>>>

Which isn't terribly elegant, but it doesn't happen very often.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Why is lambda allowed as a key in a dict?

2009-03-11 Thread Craig Allen
On Mar 10, 1:39 pm, Paul Rubin <http://phr...@nospam.invalid> wrote:
> Craig Allen  writes:
> > it raises an interesting question about why doesn't it.  I can think
> > of practical answers to that, obviously, but in principle, if a
> > function compiles to exactly the same byte code, you obviously do not
> > need two copies of it, and like strings shouldn't an identical
> > function have the same id?
>
> Identical strings don't necessarily have the same id:
>
>     >>> a = "a"*1000
>     >>> b = "a"*1000
>     >>> id(a),id(b)
>     (137143648, 137144680)
>     >>> a==b
>     True

interesting, I thought they were supposed to.
--
http://mail.python.org/mailman/listinfo/python-list


Re: Ban Xah Lee

2009-03-11 Thread Craig Allen
> There you go: a 30-second psychological diagnosis by an
> electrical engineer based entirely on Usenet postings.  It
> doesn't get much more worthless than that...
>
> --
> Grant

rolf but interesting post nonetheless.  I have been really somewhat
fascinated by AS since I heard of it about a decade ago.  There are
many among us, with interesting ideas, occasionally savant level
insight into certain abstractions, which often they can not
communicate but which lie there for those that can communicate or come
to understand nonetheless.

having said that, none of this forgives rudeness or implies people
have to tolarate it due to a person's condition, or even due to trying
to help them achieve their potential (and thus get something
productive out of it ourselves as well)...  that is, if you have these
communications problems you have to realize it, thank god you are
functional, and just that alone will help you communicate.

me, also IANAP, also working from usenet and an asperger's book I read
(and google)...
--
http://mail.python.org/mailman/listinfo/python-list


Re: Is python worth learning as a second language?

2009-03-10 Thread Craig Allen
On Mar 9, 12:43 am, ZikO  wrote:
> Hi
>
> I hope I won't sound trivial with asking my question.
>
> I am a C++ programmer and I am thinking of learning something else
> because I know second language might be very helpful somehow. I have
> heard a few positive things about Python but I have never writen any
> single line in python so I do not know this language at all.
>
> Do you think python would be good complementary language for C++? Do you
> think it's worth learning it or let's say try Java? and how difficult it
> would be for me if I know C++ pretty well I would say?
>
> Thanks

I'm not even going to read the replies first because I have my own. I
was a C and C++ programmer exclusively for over a decade. During that
time I had a "whatever tool for the problem" approach to language
selection but C++ continued to be the best for the sort of thing I was
doing. During that time I continued to learn other languages at least
enough to consider them.  I appreciated the role of interpreted
languages were filling, but also never felt comfortable in them.

Python, imo, is an excellent language to learn as a C++ programmer.
It is relatively easy to extend with C/C++ and so works together well.
When you find yourself writing some configuration language, you'll be
able to use python instead.  Also, I have found that Python helped
open my mind a bit about object orientation and to realize that while
the compile-time decisions in C++ are great for the C/C++ linking
model, and provides a certain sort of power and control, that it also
really does (as many had complained to me) take a few blows at how you
really want OO to work.

So I love python's OO and things which can be done structurally in C++
(like metaclass programming) but with which the C++ syntax is not
cooperative, and which is very much harder to safely extend modularity
too (I could go into detail but why bother here, it's either already
clear what I mean or isn't that important)...

Finally, Python is a multiparadigmed language, like C++.  It very much
seems to believe in Bjarne's original trust the programmer outlook on
languages.   As a C++ programmer I enjoyed the responsibility and
power of choosing one's paradigms at the outset of a project. Such
choices are best made consciously, one learns a lot about their
project and develops a lot of philosophical standards for the design
and implementation by having to think first "what models and
programming paradigms will we adopt". It makes you think what sort of
code you will be writing often, and which idioms will be efficient and
maintainable.

Honestly, I've become more of a Python fan than I am really
comfortable with... it can't be as good as I think.

-craig
--
http://mail.python.org/mailman/listinfo/python-list


Re: Why is lambda allowed as a key in a dict?

2009-03-10 Thread Craig Allen

> I think the point is that function objects compare by object identity,
> so the two lambdas you use above are not equal even though they have the
> same code.

it raises an interesting question about why doesn't it.  I can think
of practical answers to that, obviously, but in principle, if a
function compiles to exactly the same byte code, you obviously do not
need two copies of it, and like strings shouldn't an identical
function have the same id?

This is merely philosophical, I don't see the value in making this so
(a slight optimizaton and perhaps better conformity to other python
features), but I do think it's an interesting question.

cheers.
--
http://mail.python.org/mailman/listinfo/python-list


Re: A better way to timeout a class method?

2009-03-09 Thread Nick Craig-Wood
John O'Hagan  wrote:
>  Is there a concise Pythonic way to write a method with a timeout?
> 
>  I did this:
> 
>  class Eg(object):
> 
>  def get_value(self, timeout):
> 
>  from threading  import Timer
>  self.flag = True
> 
>  def flag_off():
>  self.flag = False
>  timer = Timer(timeout, flag_off)
>  timer.start()
> 
>  while self.flag:
>  #do stuff for a long time to get some value
>  if condition:  #if we ever get the value
>  self.value = value
>  break
> 
>  but it seems hackish to have a special "flag" attribute just to manage the 
>  timeout within the the method.

How about something like this

from threading import Timer

class Duration(object):
def __init__(self, duration):
self.running = True
self.timer = Timer(duration, self.set_finished)
self.timer.setDaemon(True)
self.timer.start()
def set_finished(self):
self.running = False
def __nonzero__(self):
return self.running

from time import sleep

def long_function():
duration = Duration(5)
i = 0
print "starting"
while duration:
print i
i += 1
sleep(1)
print "finished"

long_function()

Which prints

starting
0
1
2
3
4
finished

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: A better way to timeout a class method?

2009-03-09 Thread Nick Craig-Wood
Marco Mariani  wrote:
>  John O'Hagan wrote:
> 
> > Is there a concise Pythonic way to write a method with a timeout?
> 
>  No need for threading. Just define a signal handler and call signal.alarm().
> 
>  See the example at the end of the page:
> 
>  http://docs.python.org/library/signal.html

Won't work on windows and there is only one sigalarm timer, so you
can't nest them :-(

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Any c header parser for generate ctypes module?

2009-03-09 Thread Nick Craig-Wood
Diez B. Roggisch  wrote:
>  Victor Lin wrote:
> > I am writing python binding for some c library. It is really a super
> > boring job. Copy... paste... modify.. copy paste...modify I am
> > wondering, I am a programmer, why I can't do this job like a
> > programmer? So I think the best way to write binding for those c
> > libraries, is to write a parser that parse header of c library and
> > generate ctypes python module automatically. My question is, is there
> > any available tools can achieve this? If not, what tool can I use to
> > such a job easily. I need a c parser, is there any C parser written in
> > python?
> 
>  GCCXML is usually used to create ctypes-structures from headers.

Look at

  http://pypi.python.org/pypi/ctypeslib/

And the h2xml and xml2py scripts that are part of it.

You'll need gccxml too as Diez pointed out.

A definite time saver!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Is python worth learning as a second language?

2009-03-09 Thread Nick Craig-Wood
ZikO  wrote:
>  I am a C++ programmer and I am thinking of learning something else 
>  because I know second language might be very helpful somehow. I have 
>  heard a few positive things about Python but I have never writen any 
>  single line in python so I do not know this language at all.
> 
>  Do you think python would be good complementary language for C++?

Yes

> Do you think it's worth learning it or let's say try Java?

Jave is basically C++ with the hard bits taken out.  I wouldn't
bother.

> and how difficult it would be for me if I know C++ pretty well I
> would say?

Go here

  http://www.diveintopython.org/

Download the PDF or buy the book.

You'll be writing your first python program in about 5 minutes!

We use lots of python embedded into our humungous C++ app.  Gradually
the dev team has moved from writing C++ to writing python as much of
the time as possible and C++ only when necessary.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Opening for Python Programmer at Newport Beach

2009-03-03 Thread Craig Allen
On Mar 3, 10:17 am, Cool Dude  wrote:
> Hello ,
> This is Aniket from Techclique, a New Jersey based software
> development and IT consulting firm providing top quality technical and
> software professionals on a permanent and contractual basis to
> Government and commercial customer including fortune 500 companies and
> most of states and federal governments. I am contacting you to see if
> you are comfortable to the skills mentioned below, please forward
> resume to this e-mail address ani...@techclique.com
>
> Python Programmer (someone with industry knowledge/asset mgt)
> Location: Newport Beach, CA (Need Local)
> Duration: 6+ months
> Note: Communication MUST be perfect!, Plus submit with 3 professional
> references
>
> Python programming skills (with web development and dynamically
> generated charts/plots in particular) and working knowledge of Linux/
> UNIX Shell Scripts and SQL.
> Knowledge of Python integration with C/C++ - a definite plus.
>
> Qualifications/Requirements
> * 3+ years of Python programming on Linux/Unix platform; recent (2007)
> required
need programmer to write bot that doesn't multipost... please submit
three references so I can check to see if you've ever multiposted on
usenet.

> * Programming skills building forms, lay-outs, charts, and graphing
> required
> * Designing, coding, and testing web based applications preferred.
> * Strong organizational, oral and written communications skills
> * High energy/self starter with the ability to work independently
> within the firm*s demanding and highly focused environment
>
> Thanks & Regards,
> Aniket
> Techclique Inc.
> Jersey City, NJ
> Email: ani...@techclique.com
> Yahoo IM : aniket_mitta...@yahoo.com
> Contact No : 732-357-3844

--
http://mail.python.org/mailman/listinfo/python-list


Re: Using xreadlines

2009-02-28 Thread Nick Craig-Wood
bearophileh...@lycos.com  wrote:
>  Brett Hedges:
> > My question is how do I go to a previous line in the file? xreadlines has a 
> > file.next() statement that gives the next line, and I need a statement that 
> > gives me the previous line.<
> 
>  In modern versions of Python you usually don't need xreadlines,
>  because files are iterable.
> 
>  If your files are small, you can just read all the lines in a list
>  with open(...).readlines(), and then just use the item of the list
>  with the n-1 index.

There is also the linecache module

  http://docs.python.org/library/linecache.html

Which may be useful...

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Extending Python Questions .....

2009-02-26 Thread Nick Craig-Wood
Ben  wrote:
>  On Feb 24, 11:31?am, Nick Craig-Wood  wrote:
> > So do you want to embed python into your code?
> >
> > I'm still not clear what you are trying to achieve with python, though
> > I have a better idea what SLAG is now!
> 
>  Actually no, I want to EXTEND python using the lower levels of S-lang
>  screen modules.

Ah!

>  My Modules are written in C and are a frame work for building pull-
>  down menus and data entry screens.
> 
>  Very nice for writing business applications. Think along the lines
>  of FoxPro and/or the "Screen" section in Cobol and you have a
>  pretty good idea of what i have done.

You've got several choices.

1) Write a python module in C which interfaces with your C code.

This is the traditional way, but it is quite time consuming and easy
to slip up with reference counting.  You need to learn the python API
to use this.

Will require compiling by your users (not a big hurdle if you make a
setup.py).

2) Write an interface with ctypes

If your C code presents itself as a shared object (dll/so) then this
is a really nice way of using it.  You don't have to write any C code
or worry about any reference counting - you do it all in python.

You'll need to learn how ctypes maps python onto C, but as you know C
well you won't find this difficult.

ctypes is included with python now.

3) Use Cython

Cython is a sort of hybrid C and python which can be used for
interfacing with C code.

This requires learning exactly what you can and can't do in Cython (it
is almost python but not quite).

This will require your users to install Cython and compile stuff.
Again easy if you make a setup.py but more stuff to install.

4) Use Swig

A solid approach, quicker than 1) but not as quick as 2) or 3)

This will require your users to install swig and a compiler.

...

My prefered approach is 2) ctypes at the moment.  I have used all 4 of
the above approaches in the past though!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Extending Python Questions .....

2009-02-24 Thread Nick Craig-Wood
Ben  wrote:
>  No, It uses the the S-lang for video, and input control. However, SLAG
>  is more of an abstract layer on top of that.
> 
>  It has a Structures that contains menus and screens (menumodule /
>  screenmodule). One LOADS them up with parameters.  such as creating
>  a new menu is like:
> 
>  OpenMenu( Company name, SubSystem, this program name, mode, bottom
>  status display) - Create initial menu structure Addtomenu(Menu
>  Block Set name, DISPLAY line, ID, type of program, password ID ) -
>  add to / update MENU blocks.  runMenu() - Displays the whole create
>  menu structure.
> 
>  The Menu structure is done in pull downs and scrollable blocks in a
>  TUI (text User Interface) and using the S-lang screen library is
>  fully mouseable.
> 
>  The screen module works mych the same way, but with the abiltity to
>  open and close and work within "Sub Screens".
> 
>  For those who do not know, S-lang is a interpreted language much
>  like Python. However, there is s direth of library modules. The
>  original S-lang started out as library of screen of keyboard
>  modules, but has been expanded
> 
>  My SLAG project does not care in reality WHICH or what language, it
>  is simply handling menu and screen control.

So do you want to embed python into your code?

I'm still not clear what you are trying to achieve with python, though
I have a better idea what SLAG is now!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: intermediate python csv reader/writer question from a beginner

2009-02-24 Thread Nick Craig-Wood
Learning Python  wrote:
>  anything related to csv, I usually use VB within excel to manipulate
>  the data, nonetheless, i finally got the courage to take a dive into
>  python.  i have viewed a lot of googled csv tutorials, but none of
>  them address everything i need.  Nonetheless, I was wondering if
>  someone can help me manipulate the sample csv (sample.csv) I have
>  generated:
> 
>  ,,
>  someinfo,,,
>  somotherinfo,,,
>  SEQ,Names,Test1,Test2,Date,Time,,
>  1,Adam,1,2,Monday,1:00 PM,,
>  2,Bob,3,4,Monday,1:00 PM,,
>  3,Charlie,5,6,Monday,1:00 PM,,
>  4,Adam,7,8,Monday,2:00 PM,,
>  5,Bob,9,10,Monday,2:00 PM,,
>  6,Charlie,11,12,Monday,2:00 PM,,
>  7,Adam,13,14,Tuesday,1:00 PM,,
>  8,Bob,15,16,Tuesday,1:00 PM,,
>  9,Charlie,17,18,Tuesday,1:00 PM,,
> 
>  into (newfile.csv):
> 
>  Adam-Test1,Adam-Test2,Bob-Test1,Bob-Test2,Charlie-Test1,Charlie-
>  Test2,Date,Time
>  1,2,3,4,5,6,Monday,1:00 PM
>  7,8,9,10,11,12,Monday,2:00 PM
>  13,14,15,16,17,18,Tuesday,1:00 PM
> 
>  note:
>  1. the true header doesn't start line 4 (if this is the case would i
>  have to use "split"?)
>  2. if there were SEQ#10-12, or 13-15, it would still be Adam, Bob,
>  Charlie, but with different Test1/Test2/Date/Time

I'm not really sure what you are trying to calculate, but this should
give you some ideas...

import csv
from collections import defaultdict

reader = csv.reader(open("sample.csv"))
result = defaultdict(list)
for row in reader:
# ignore unless first row is numeric
if not row or not row[0].isdigit():
continue
n, name, a, b, day, time = row[:6]
print "n=%r, name=%r, a=%r, b=%r, day=%r, time=%r" % (n, name, a,
b, day, time)
result[(day, time)].append(n)

writer = csv.writer(open("newfile.csv", "w"))
for key, values in result.iteritems():
day, time = key
values = values + [day, time]
writer.writerow(values)

This prints

n='1', name='Adam', a='1', b='2', day='Monday', time='1:00 PM'
n='2', name='Bob', a='3', b='4', day='Monday', time='1:00 PM'
n='3', name='Charlie', a='5', b='6', day='Monday', time='1:00 PM'
n='4', name='Adam', a='7', b='8', day='Monday', time='2:00 PM'
n='5', name='Bob', a='9', b='10', day='Monday', time='2:00 PM'
n='6', name='Charlie', a='11', b='12', day='Monday', time='2:00 PM'
n='7', name='Adam', a='13', b='14', day='Tuesday', time='1:00 PM'
n='8', name='Bob', a='15', b='16', day='Tuesday', time='1:00 PM'
n='9', name='Charlie', a='17', b='18', day='Tuesday', time='1:00 PM'

And leaves newfile.csv with the contents

1,2,3,Monday,1:00 PM
7,8,9,Tuesday,1:00 PM
4,5,6,Monday,2:00 PM

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Extending Python Questions .....

2009-02-23 Thread Nick Craig-Wood
Ben  wrote:
>  In My S-Lag Project called, SLAG, I have some function keys that get
>  mapped back to S-lang internal functions.
> 
>  My SLAG project works pretty much like Python (as does the S-Lang).
>  You write a S-lang script
>  that "imports" your extension. module - and all this gets run by the
>  shell/interpreter.
> 
>  I allow function keys to be mapped back internal function(s) inside of
>  the controlling program.
> 
>  My question is which Python C api Do I need to this with ? 

Do you mean like this?

  http://code.google.com/p/python-slang 

Not sure how well maintained it is though.

> Do I need to worry about my reference counting since the Python
> Program is in essence calling a function in itself?

I'm not sure I understand you here, but in general you don't need to
worry about reference counting in Python - it all happens behind the
scenes.  If you are writing a python extension in C then you do need
to worry about reference counting - a lot!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: can multi-core improve single funciton?

2009-02-23 Thread Nick Craig-Wood
sturlamolden  wrote:
>  On Feb 10, 8:38?am, Paul McGuire  wrote:
> 
> > Even worse than linear, the function is recursive, which as I
> > understand it, is inherently a no-no when looking for code that is
> > parallel-friendly.
> 
>  There is no way to parallelize Fibonacci numbers computed with linear
>  time complexity, as we must know fib(n-1) to compute fib(n). But if we
>  were to use Oyster's recursive version, which has geometric
>  complexity, one could parallelize with a 'forkjoin' scheme.
> 
>  Anyhow, this runs in amortized linear time and would be a lot faster:
> 
>  def fib(n):
> while True:
>try:
>   return fib.seq[n]
>except AttributeError:
>   fib.seq = [0, 1, 1]
>except IndexError:
>   fib.seq.append( fib.seq[-2] +
>fib.seq[-1] )

Or something which runs in O(1)...

from gmpy import mpf, mpz, digits
from math import log, sqrt

root5_float = sqrt(5)
phi_float = (1 + root5_float) / 2
log_phi_float = log(phi_float)
log_root5_float = log(root5_float)
log2 = log(2)

def fibonacci_noniterative(n):
"""
Work out n'th Fibonacci number in an noniterative fashion using Binet's 
formula
"""
# Work out magnitude of answer
bits = int(((n * log_phi_float - log_root5_float) / log2)*1.1)
if bits < 0:
bits = 0
root5 = mpf(5, bits).sqrt()
phi = (mpf(1, bits) + root5) / mpf(2, bits)
f_n = phi**n / root5
f_n = mpz(f_n + mpf(1, bits) / mpf(2, bits))
return long(f_n)

It runs in O(1) if the O we are talking about is large integer
arithmetic.  It actually runs in more like O(log(n)**1.5) time.  It is
a lot faster than the iterative approach too which runs in
O(log(n)**2) for any n > ~40.

Times to calculate F(n)

 n,  iterative, noniterative
 1,   0.03,   0.16
 2,   0.03,   0.14
 4,   0.03,   0.12
 8,   0.03,   0.16
16,   0.04,   0.09
32,   0.07,   0.10
64,   0.14,   0.10
   128,   0.32,   0.11
   256,   0.72,   0.14
   512,   0.000157,   0.19
  1024,   0.000361,   0.31
  2048,   0.000881,   0.66
  4096,   0.002504,   0.000178
  8192,   0.007640,   0.000521
 16384,   0.025362,   0.001572
 32768,   0.090633,   0.004701
 65536,   0.342724,   0.014132
131072,   1.335723,   0.045194
262144,   5.273360,   0.111201
524288,  20.791205,   0.301488
   1048576,  82.617938,   0.833644

Here is the rest of the program just in case anyone wants to play

def fibonacci_iterative(n):
"""
Work out n'th Fibonacci number in an iterative fashion
"""
a, b = 0, 1
while n > 0:
a, b = a + b, a
n -= 1
return a

if __name__ == "__main__":
# warmup
fibonacci_noniterative(100)
print " n,  iterative, noniterative"
for e in range(30):
i = 2 ** e
t0 = time()
f_iterative = fibonacci_iterative(i)
t_iterative = time() - t0
t0 = time()
f_noniterative = fibonacci_noniterative(i)
t_noniterative = time() - t0
print "%10d, %10.6f, %10.6f" % (i, t_iterative, t_noniterative)
if f_iterative != f_noniterative:
    print "Calculation error"
print "iterative", f_iterative
print "non iterative", f_noniterative
print "difference", f_iterative-f_noniterative

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Threading and tkinter

2009-02-20 Thread Craig Allen

> The statement
>
>     x=x+1
>
> (which, by the way, should stylistically be written
>
>     x = x + 1
>

yes I was wondering what "x=x+1" meant until you translated it... oh,
"x = x + 1" of course! I thought to myself.

Oh wait no I'm sarcastic.
--
http://mail.python.org/mailman/listinfo/python-list


Re: How do I declare global vars or class vars in Python ?

2009-02-20 Thread Nick Craig-Wood
Linuxguy123  wrote:
>  How do I do this in Python ?
> 
>  #
>  declare A,B
> 
>  function getA
>  return A
> 
>  function getB
>  return B
> 
>  function setA(value)
>   A = value
> 
>  function setB(value)
>   B = value
> 
>  main()
>  getA
>  getB
>  dosomething
>  setA(aValue)
>  setB(aValue)
>  
> 
>  The part I don't know to do is declare the variables, either as globals
>  or as vars in a class.  How is this done in Python without setting them
>  to a value ?

Variables can have any value in python so if you want to pre-declare
then you set them to None normally.

As a class :-

class Stuff(object):
def __init__(self):
self.A = None
self.B = None
def getA(self):
return self.A
def getB(self):
return self.B
def setA(self, value):
self.A = value
def setB(self, value):
self.B = value

>>> a = Stuff()
>>> print a.getA()
None
>>> print a.getB()
None
>>> # dosomething
... a.setA("aValue")
>>> a.setB("aValue")
>>> print a.getA()
aValue
>>> print a.getB()
aValue
>>> 

Note that in python we don't normally bother with getA/setA normally,
just use self.A, eg

class Stuff(object):
def __init__(self):
self.A = None
self.B = None
def main(self):
print self.A
print self.B
# dosomething
self.A = "aValue"
self.B = "aValue"
print self.A
print self.B

>>> a = Stuff()
>>> a.main()
None
None
aValue
aValue
>>>  

If you need (later) A to be a computed value then you turn it into a
property, which would look like this.  (Note the main method is
identical to that above).

class Stuff(object):
def __init__(self):
self._A = None
self.B = None
def _getA(self):
print "Getting A"
return self._A
def _setA(self, value):
print "Setting A"
self._A = value
A = property(_getA, _setA)
def main(self):
    print self.A
print self.B
# dosomething
self.A = "aValue"
self.B = "aValue"
print self.A
print self.B

>>> a = Stuff()
>>> a.main()
Getting A
None
None
Setting A
Getting A
aValue
aValue
>>>  

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Pythonic way to determine if a string is a number

2009-02-16 Thread Nick Craig-Wood
pyt...@bdurham.com  wrote:
>  Thanks for everyone's feedback. I believe my original post's code
>  (updated following my signature) was in line with this list's feedback.
> 
>  Christian: Thanks for reminding me about exponential formats. My updated
>  code accounts for these type of numbers. I don't need to handle inf or
>  nan values. My original code's except catches an explicit ValueError
>  exception per your concern about bare excepts.
> 
>  Malcolm
> 
> 
>  # str_to_num.py
> 
>  def isnumber( input ):
>  try:
>  num = float( input )
>  return True
> 
>  except ValueError:
>  return False

That is a fine solution.

Last time I had to solve this I had a specific format of number to
parse - I didn't want to include all the python formats.  This is what
I came up with...  This particular code returns the converted number
or 0.0 - adjust as you see fit!

import re

_float_pattern   = re.compile(r"^\s*([-+]?(\d*\.)?\d+([eE][-+]?\d+)?)")

def atof(value):
"""
Convert a string to an float in the same way the c-library atof()
does.  Ie only converting as much as it can and returning 0.0 for
an error.
"""
match = _float_pattern.search(value)
if match:
return float(match.group(1))
return 0.0

>>> atof("15.5 Sausages")
15.5
>>> atof("   17.2")
17.199
>>> atof("0x12")
0.0
>>> atof("8.3.2")
8.3007
>>> atof("potato")
0.0
>>>

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Found a very nice, small, cross-platform GUI toolkit for Python.

2009-02-16 Thread Nick Craig-Wood
laplacia...@gmail.com  wrote:
>  I think I just found the GUI toolkit for Python I've been searching
>  for. It seems to meet all of the following requirements:
> 
>* free software
>* small (I don't need batteries -- Python already comes with those.)
>* easy to use
>* actively maintained
>* cross-platform
>* easy to install
>* based on a stable and actively-maintained C library
>* does not depend on an external scripting language (for example,
>  Tcl)
>* well-documented
>* not too many dependencies
>* can easily integrate with PyOpenGL
>* support for accessibility
> 
>  and it's also written in Python.
> 
>  I have no idea how it's stayed under the radar in the Python community
>  for this long, yet here it is: [OcempGUI](http://ocemp.sourceforge.net/
>  gui.html). The C library it depends upon? [SDL](http://
>  www.libsdl.org/) (via [PyGame](http://www.pygame.org/news.html)).

Interesting!  One of the commercial apps I'm involved (C++ not python)
in uses SDL as its GUI with windows etc built on top of it.  It means
that it looks exactly the same on all supported platforms and since it
usually runs full screen that is fine.  I imagine this GUI toolkit
fits the same niche.

Presumably since it uses SDL then all the GUI will appear in one
window?  So windows within windows in the MDI style?

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: python in emacs

2009-02-15 Thread Craig
I would go to ubuntu linux if you can.

--- On Sun, 2/15/09, Diez B. Roggisch  wrote:

From: Diez B. Roggisch 
Subject: Re: python in emacs
To: python-list@python.org
Date: Sunday, February 15, 2009, 9:23 AM

kentand...@sbcglobal.net schrieb:
> When I visit a file with extension .py, emacs says "loading
> python...done", and gives me a "python" menu with options like "start
> interpreter" and "eval buffer". When I try to use one of these options
> emacs says "loading compile...done", then hangs and has to be shut down
> from the task manager. The Python folder is in my PATH variable and
> works fine from the command line (or with IDLE). I'm using emacs-22.3 on
> windows xp, and have the same problem with both python-2.2 and
> python-3.0. Anybody got any tips, or should I just give up and use some
> other editor?

Did you try setting the python interpreter with the full path?

Diez
--
http://mail.python.org/mailman/listinfo/python-list



  --
http://mail.python.org/mailman/listinfo/python-list


Re: Easier to wrap C or C++ libraries?

2009-02-15 Thread Nick Craig-Wood
Christian Heimes  wrote:
>  Hrvoje Niksic schrieb:
> > "Diez B. Roggisch"  writes:
> > 
> >> The answer is easy: if you use C, you can use ctypes to create a
> >> wrapper - with pure python, no compilation, no platform issues.
> > 
> > The last part is not true.  ctypes doesn't work on 64-bit
> > architectures, nor does it work when Python is built with non-gcc Unix
> > compilers
> 
>  ctypes works on 64bit systems, too. However it's a pain in the ... back
>  to write code with ctypes that works across all platforms. I used to use
>  ctypes for wrapper but eventually I switched to Cython.

What sort of problems have you had?

I find as long as I use the same types as the C code actually uses it
all works fine.  If on a 64 bit platform long is 64 bits then it will
be under ctypes too.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: encrypting lines from file with md5 module doesn't work?

2009-02-14 Thread Nick Craig-Wood
Canned  wrote:
>  I write a small script that read lines from plain text file and encrypt
>  each lines using md5 module. I have a small word list that contain 2000+
>  words, 1 word/line. Using the code below, I can save the output to
>  another file to use it with john the ripper (http://www.openwall.com).
> 
>  Here is the part that annoys me, john recognize the output as des and
>  not as md5. Using the original wordlist, I tried to crack the list but
>  nothing has come up after waiting for almost 9 hours. Can someone please
>  tell me what did I wrong? Why john don't recognize the output as md5?
>  I'm using python 2.5.1 and I'm python noob and also don't have any
>  knowledge about encryption.
> 
>  import sys, md5
> 
>  f = open(sys.argv[1])
>  obj = md5.new()
> 
>  for line in f:
>  if line[-1:] == '\n':
>  text = line[:-1]
>  obj.update(text),
>  print text + ':' + obj.hexdigest()
> 
>  f.close()
> 
> 
>  00:670b14728ad9902aecba32e22fa4f6bd
>  :c47532bbb1e2883c902071591ae1ec9b
>  11:bf874003f752e86e6d6ba6d6df1f24a2
>  :65a89de3000110bf37bcafdbd33df55a
>  121212:38a8eeb4dfb0f86aefea908365817c15
>  123123:f226a65a908909b83aed92661897d0c9

john cracks password files if I remember rightly.

md5 encoded password files don't look like that, they look like this

guest:$1$3nvOlOaw$vRWaitT8Ne4sMjf9NOrVZ.:13071:0:9:7:::

(not a real password line!)

You need to work out how to write that format.

>From memory: the "$1" bit means it is an md5 hash, the next
"$3nvOlOaw$" is the salt and the final "$vRWaitT8Ne4sMjf9NOrVZ." is
the md5 hash in some encoded format or other!  Some googling should
reveal the correct algorithm!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: String concatenation performance with +=

2009-02-14 Thread Nick Craig-Wood
Sammo  wrote:
>  String concatenation has been optimized since 2.3, so using += should
>  be fairly fast.
> 
>  In my first test, I tried concatentating a 4096 byte string 1000 times
>  in the following code, and the result was indeed very fast (12.352 ms
>  on my machine).
> 
>  import time
>  t = time.time()
>  mydata = ""
>  moredata = "A"*4096
>  for i in range(1000):
>  mydata += moredata # 12.352 ms
>  print "%0.3f ms"%(1000*(time.time() - t))
> 
>  However, I got a different result in my second test, which is
>  implemented in a class with a feed() method. This test took 4653.522
>  ms on my machine, which is 350x slower than the previous test!
> 
>  class StringConcatTest:
>  def __init__(self):
>  self.mydata = ""
> 
>  def feed(self, moredata):
>  self.mydata += moredata # 4653.522 ms
> 
>  test = StringConcatTest()
>  t = time.time()
>  for i in range(1000):
>  test.feed(moredata)
>  print "%0.3f ms"%(1000*(time.time() - t))
> 
>  Note that I need to do something to mydata INSIDE the loop, so please
>  don't tell me to append moredata to a list and then use "".join after
>  the loop.
> 
>  Why is the second test so much slower?

The optimized += depends on their being no other references to the
string.  Strings are immutable in python.  So append must return a new
string.  However the += operation was optimised to do an in-place
append if and only if there are no other references to the string.

You can see this demonstrated here

$ python -m timeit -s 'a="x"' 'a+="x"'
100 loops, best of 3: 0.231 usec per loop

$ python -m timeit -s 'a="x"; b=a' 's = a; a+="x"'
10 loops, best of 3: 30.1 usec per loop

You are keeping the extra reference in a class instance like this

$ python -m timeit -s 'class A(object): pass' -s 'a=A(); a.a="x"' 'a.a+="x"'
10 loops, best of 3: 30.7 usec per loop

Knowing that, this optimization suggests itself

$ python -m timeit -s 'class A(object): pass' -s 'a=A(); a.a="x"' 's = a.a; a.a 
= None; s += "x"; a.a = s'
100 loops, best of 3: 0.954 usec per loop

Or in your example

class StringConcatTest:
def __init__(self):
self.mydata = ""
def feed(self, moredata):
#self.mydata += moredata
s = self.mydata
del self.mydata
s += moredata
self.mydata = s

moredata = "A"*4096
test = StringConcatTest()
t = time.time()
for i in range(1000):
test.feed(moredata)

print "%0.3f ms"%(1000*(time.time() - t))

Before it was 3748.012 ms on my PC, afterwards it was 52.737 ms

However that isn't a perfect solution - what if something had another
reference on self.mydata?

You really want a non-immutable string for this use.  array.array
is a possibility

$ python -m timeit -s 'import array' -s 'a = array.array("c")' 'a.extend("x")'
10 loops, best of 3: 2.01 usec per loop

There are many other possibilities though like the mmap module.
-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Escaping my own chroot...

2009-02-13 Thread Nick Craig-Wood
Jean-Paul Calderone  wrote:
>  On Wed, 11 Feb 2009 09:31:56 -0600, Nick Craig-Wood  
> wrote:
> >r0g  wrote:
> >>  I'm writing a linux remastering script in python where I need to chroot
> >>  into a folder, run some system commands and then come out and do some
> >>  tidying up, un-mounting proc & sys etc.
> >>
> >>  I got in there with os.chroot() and I tried using that to get back out
> >>  but that didn't work so... is my script trapped in there forever now or
> >>  is there an un-hacky way to escape?
> >
> >No!
> 
>  If you still have root in the chroot (and you need root to get in there, so
>  it's not implausible that you will), then you can get out.  Googling for
>  "escape chroot" turns up lots of hits.  This page contains a fairly simple,
>  explicit description of how to get out of a chroot:
> 
> http://www.bpfh.net/simes/computing/chroot-break.html
> 
>  See the bulleted list in the "Breaking chroot()" section.  Since you also
>  control the process before the chroot happens, breaking out is even simpler
>  in your case (just open / before you chroot in the first place).  forking
>  before doing the chroot may still be a good idea, but it's not the only
>  solution.

I admit it can be done, but I'm not sure it isn't hacky!

#!/usr/bin/python

"""
Enter a chroot and escape again

Run as root
"""

import os
import sys

def ls(path):
"""List the path"""
print "Directory listing of %r" % path
for f in os.listdir(path):
print ">>", f

def main():
if len(sys.argv) < 2:
print >>sys.stderr, "Need directory to chroot to as an argument"
raise SystemExit(1)
chroot_dir = sys.argv[1]
print "Opening root"
root = os.open("/", os.O_RDONLY)
print "Before chroot"
ls("/")
print "Chrooting to %r" % chroot_dir
os.chroot(chroot_dir)
ls("/")
print "Breaking the chroot"
os.fchdir(root)
for i in range(100):
os.chdir("..")
os.chroot(".")
ls("/")
os.close(root)

if __name__ == "__main__":
main()


I ran this

$ mkdir chroot_test
$ touch chroot_test/in_the_chroot
$ sudo ./chroot_test.py chroot_test

And it produced this

Opening root
Before chroot
Directory listing of '/'
>> lost+found
>> home
>> bin
>> boot
>> proc
>> dev
>> etc
[snip]
Chrooting to 'chroot_test'
Directory listing of '/'
>> in_the_chroot
Breaking the chroot
Directory listing of '/'
>> lost+found
>> home
>> bin
>> boot
>> proc
>> dev
>> etc
[snip]


-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Escaping my own chroot...

2009-02-11 Thread Nick Craig-Wood
r0g  wrote:
>  I'm writing a linux remastering script in python where I need to chroot
>  into a folder, run some system commands and then come out and do some
>  tidying up, un-mounting proc & sys etc.
> 
>  I got in there with os.chroot() and I tried using that to get back out
>  but that didn't work so... is my script trapped in there forever now or
>  is there an un-hacky way to escape?

No!

>  If it is an OS restriction rather than my ignorance of python is there a
>  canonical way of dealing with it? Should I maybe fork a process, have it
>  do the chroot and wait for it to terminate before continuing?

That is the only way...

However this is a solved problem if you use schroot (which works very
well in my experience)

  http://packages.debian.org/sid/schroot
  http://www.debian-administration.org/articles/566

There are definitely debian, ubuntu and centos packages for it if you
look!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: MySQLdb and MySQL stored functions

2009-02-04 Thread Nick Craig-Wood
kurt.forrester@googlemail.com  wrote:
>  Any ideas on how to suppress the warning output:
>  __main__:1: Warning: No data - zero rows fetched, selected, or
>  processed

You can use the warnings module to suppress these I would have
thought.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Cross platform compilation?

2009-02-04 Thread Nick Craig-Wood
Christian Heimes  wrote:
>  John Harper schrieb:
> > I am trying to build Python to use in an embedded system which uses a 
> > ppc_440 CPU. The only information I've found is something written by 
> > Klaus Reimer a few years ago, which was based on Python 2.2. So far I 
> > seem to have successfully built Python itself, but building the 
> > extensions fails miserably with complaints about library format. It 
> > insists on referring to "i686" which is indeed the platform it is 
> > actually running on.
> > 
> > Before I try to reverse engineer completely setup.py, is there 
> > something obvious that needs to be done to get it to use the right tool 
> > chain?

I spent some time mucking about trying to do this for ARM.  I almost
got it to work using qemu but not quite!  What I did in the end was to
use a pre-compiled python (from debian) then copy the headers and
library files into the cross compiling environment so we could embed
python into our code and build our extensions.

> > More generally, it seems like this would be something that lots of 
> > people would want to do. I'm surprised there isn't support for it built 
> > into the distributed version of Python, without having to hack all the 
> > setup files...?
> 
>  I'm sorry to inform you that Python doesn't support cross platform
>  compilation. Python eats its own dog food and uses Python to compile
>  optional extension modules.

It isn't an impossible problem...

If you've ever cross compiled gcc you'll find that it first uses the
host compiler (possibly not gcc) to compile itself to make a gcc which
runs on the host but makes code for the target.  It then uses that new
compiler to compile a gcc for the target architecture.

I could imagine a similar scheme for python, but it would involve lots
of fiddling about and some commitment from the python maintainers.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Code critique xmlrpclib

2009-02-04 Thread Nick Craig-Wood
flagg  wrote:
>  On Feb 3, 7:32?am, Nick Craig-Wood  wrote:
> > flagg  wrote:
> > > ?This xmlrpc server is designed to parse dns zone files and then
> > > ?perform various actions on said files. \
> > > ?It uses dnspython, and xmlrpclib
> > > ? I'd like to know what some of the more experienced python users
> > > ?think. Where I could improve code, make it more efficient, whatever.
> > > ?All suggestions are welcome. ?This is my first functional python
> > > ?program, ?and I have tons of questions to go along with whatever
> > > ?suggestions. ? How could I do sanity checking; right now nothing
> > > ?checks the input to these functions, error-handling in this program is
> > > ?almost non-existant how do you integrate decent error handling into a
> > > ?piece of software..ill shut up now.
> >
> > I made a few comments, some about error handling! ?See below
> >
> >
> >
> > > ?import dns.zone
> > > ?from time import localtime, strftime, time
> > > ?import os, sys
> > > ?from dns.rdataclass import *
> > > ?from dns.rdatatype import *
> > > ?from string import Template
> > > ?import xmlrpclib
> > > ?from SimpleXMLRPCServer import SimpleXMLRPCServer
> > > ?from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
> >
> > > ?zoneDir = "/dns"
> >
> > > ?def openZoneFile(zoneFile):
> > > ? ? ?"""
> > > ? ? ?Opens a zone file. ?Then reads in zone file to dnspython zone
> > > ?object.
> > > ? ? ?"""
> > > ? ? ?global zone
> > > ? ? ?global host
> > > ? ? ?global domain
> > > ? ? ?domain = ".".join(zoneFile.split('.')[1:])
> > > ? ? ?host = ".".join(zoneFile.split('.')[:1])
> >
> > > ? ? ?try:
> > > ? ? ? ? ? ? ?zone = dns.zone.from_file(zoneDir + domain + '.zone',
> > > ?domain)
> > > ? ? ?except dns.exception, e:
> > > ? ? ? ? ? ? ?print e.__class__, e
> >
> > Don't use global variables!
> >
> > This function should probably return 3 things
> >
> > ? ? return zone, host, domain
> >
> > And whenever you use it, say
> >
> > ? ? zone, host, domain = openZoneFile(xxx)
> >
> > It should probably be a method of DNSFunctions. ?I'd avoid setting
> > self.zone etc as that is making a different sort of global data which
> > I'd avoid too.
> >
> > Also if the dns.zone.from_file didn't work (raises dns.exception) you
> > set the host and domain but the zone is left at its previous value
> > which probably isn't what you wanted. ?I'd let the error propagate
> > which I think will do something sensible for the xmlrpc.
> >
> >
> >
> > > ?class DNSFunctions:
> >
> > > ? ? ? # Not exposed to xml-rpc calls, used internally only.
> > > ? ? ?def _writeToZoneFile(self, record):
> > > ? ? ? ? ?"""
> > > ? ? ? ? ?Checks if zone file exists, if it does it write values to zone
> > > ?file
> > > ? ? ? ? ?"""
> >
> > > ? ? ? ? ?for (name, ttl, rdata) in zone.iterate_rdatas(SOA):
> > > ? ? ? ? ? ? ?new_serial = int(strftime('%Y%m%d00', localtime(time(
> > > ? ? ? ? ? ? ?if new_serial <= rdata.serial:
> > > ? ? ? ? ? ? ? ? ?new_serial = rdata.serial + 1
> > > ? ? ? ? ? ? ?rdata.serial = new_serial
> >
> > > ? ? ? ? ?if os.path.exists(zoneDir + str(zone.origin) + "zone"):
> > > ? ? ? ? ? ? ?f = open(zoneDir + str(zone.origin) + "zone", "w")
> > > ? ? ? ? ? ? ?zone.to_file(f)
> > > ? ? ? ? ? ? ?f.close()
> > > ? ? ? ? ?else:
> > > ? ? ? ? ? ? ?print "Zone: " + zone.origin + " does not exist, please
> > > ?add first"
> >
> > This should probably be raising an exception to be caught or
> > propagated back to the client via xmlrpc.
> >
> >
> >
> >
> >
> > > ? ? ?def showRecord(self, record):
> > > ? ? ? ? ?"""
> > > ? ? ? ? ?Shows a record for a given zone. Prints out
> > > ? ? ? ? ?TTL, IN, Record Type, IP
> > > ? ? ? ? ?"""
> >
> > > ? ? ? ? ?openZoneFile(record)
> >
> > > ? ? ? ? ?try:
> > > ? ? ? ? ? ? ?rdataset = zone.get_node(host)
> > > ? ? ? ? ? ? ?for rdata in rdataset:
> > > ? ? ? ? ? ? ? ? ?return str(rdata)
> > > ? ? ? ? ?except:
> > 

Re: Code critique xmlrpclib

2009-02-03 Thread Nick Craig-Wood
flagg  wrote:
>  This xmlrpc server is designed to parse dns zone files and then
>  perform various actions on said files. \
>  It uses dnspython, and xmlrpclib
>   I'd like to know what some of the more experienced python users
>  think. Where I could improve code, make it more efficient, whatever.
>  All suggestions are welcome.  This is my first functional python
>  program,  and I have tons of questions to go along with whatever
>  suggestions.   How could I do sanity checking; right now nothing
>  checks the input to these functions, error-handling in this program is
>  almost non-existant how do you integrate decent error handling into a
>  piece of software..ill shut up now.

I made a few comments, some about error handling!  See below

>  import dns.zone
>  from time import localtime, strftime, time
>  import os, sys
>  from dns.rdataclass import *
>  from dns.rdatatype import *
>  from string import Template
>  import xmlrpclib
>  from SimpleXMLRPCServer import SimpleXMLRPCServer
>  from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
> 
> 
>  zoneDir = "/dns"
> 
>  def openZoneFile(zoneFile):
>  """
>  Opens a zone file.  Then reads in zone file to dnspython zone
>  object.
>  """
>  global zone
>  global host
>  global domain
>  domain = ".".join(zoneFile.split('.')[1:])
>  host = ".".join(zoneFile.split('.')[:1])
> 
>  try:
>  zone = dns.zone.from_file(zoneDir + domain + '.zone',
>  domain)
>  except dns.exception, e:
>  print e.__class__, e

Don't use global variables!

This function should probably return 3 things

return zone, host, domain

And whenever you use it, say

zone, host, domain = openZoneFile(xxx)

It should probably be a method of DNSFunctions.  I'd avoid setting
self.zone etc as that is making a different sort of global data which
I'd avoid too.

Also if the dns.zone.from_file didn't work (raises dns.exception) you
set the host and domain but the zone is left at its previous value
which probably isn't what you wanted.  I'd let the error propagate
which I think will do something sensible for the xmlrpc.

>  class DNSFunctions:
> 
>   # Not exposed to xml-rpc calls, used internally only.
>  def _writeToZoneFile(self, record):
>  """
>  Checks if zone file exists, if it does it write values to zone
>  file
>  """
> 
>  for (name, ttl, rdata) in zone.iterate_rdatas(SOA):
>  new_serial = int(strftime('%Y%m%d00', localtime(time(
>  if new_serial <= rdata.serial:
>  new_serial = rdata.serial + 1
>  rdata.serial = new_serial
> 
>  if os.path.exists(zoneDir + str(zone.origin) + "zone"):
>  f = open(zoneDir + str(zone.origin) + "zone", "w")
>  zone.to_file(f)
>  f.close()
>  else:
>  print "Zone: " + zone.origin + " does not exist, please
>  add first"

This should probably be raising an exception to be caught or
propagated back to the client via xmlrpc.

> 
>  def showRecord(self, record):
>  """
>  Shows a record for a given zone. Prints out
>  TTL, IN, Record Type, IP
>  """
> 
>  openZoneFile(record)
> 
>  try:
>  rdataset = zone.get_node(host)
>  for rdata in rdataset:
>  return str(rdata)
>  except:
>  raise Exception("Record not found")
> 
> 
>  def changeRecord(self, record, type, target):
>  """
>  Changes a dns entry.
>  @param record: which record to chance
>  @param type:  what type of record, A, MX, NS, CNAME
>  @target: if CNAME target points to HOSTNAME, if A record
>  points to IP
>  """
>  openZoneFile(record)
>  try:
>  rdataset = zone.find_rdataset(host, rdtype=type)
>  except:
>  raise Exception("You must enter a valid record and type.
>  See --usage for examples")

Don't catch all exceptions - find out which exceptions are thrown.
Consider just letting it propagate - hopefully the find_rdataset error
is descriptive enough.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Does the Python community really follow the philospy of "Community Matters?"

2009-02-02 Thread Craig Allen
I think what you have found is a remarkable characteristic of this
language.  Somehow, perhaps something to do with guido or python
itself, python has a very strong non-dogmatic streak.  It's a relief
really.  If I were to pose a "is python or its community really xyz?"
I would wonder about the "one best way to do something".  One of my
first posts here was about three different ways I had built
singletons. As a result of that I was given a better way, but no one
said any of the other ways were "non-pythonic", it was more, "if you
use option one, make sure of this, if you use option two, don't forget
that"...

As such, python programmers are not there for you to feel superior.
If you want a language for a feeling of community, then it hardly
matters what language you choose, and if you want a good language,
it's secondary if the community gives you that feeling (you should get
it from the language itself).  And since there IS a python community
to go to for help, python has that covered without worrying about it
or making it a language feature.  It may seems cold, instead of giving
warm fuzzies, people will help you with tangible problems, but we are
not starting a commune.  "Uses a langague with sense of community that
advocates  for their language over others" is never in a spec.


-craig
--
http://mail.python.org/mailman/listinfo/python-list


Re: what IDE is the best to write python?

2009-02-01 Thread Craig
eclipse

--- On Sun, 2/1/09, Dennis Lee Bieber  wrote:

From: Dennis Lee Bieber 
Subject: Re: what IDE is the best to write python?
To: python-list@python.org
Date: Sunday, February 1, 2009, 3:31 AM

On Sat, 31 Jan 2009 23:42:42 -0800 (PST), "mcheun...@hotmail.com"
 declaimed the following in comp.lang.python:

> Hi all
>    what IDE is the best to write python?

    The one in which YOU are most comfortable.
-- 
    Wulfraed    Dennis Lee Bieber        KD6MOG
    wlfr...@ix.netcom.com        wulfr...@bestiaria.com
        HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff:        web-a...@bestiaria.com)
        HTTP://www.bestiaria.com/
--
http://mail.python.org/mailman/listinfo/python-list



  --
http://mail.python.org/mailman/listinfo/python-list


Re: Python 2.6's multiprocessing lock not working on second use?

2009-01-19 Thread Nick Craig-Wood
Jesse Noller  wrote:
> > Opened issue #4999 [http://bugs.python.org/issue4999] on the matter,
> > referencing this thread.
> 
>  Thanks, I've assigned it to myself. Hopefully I can get a fix put
>  together soonish, time permitting.

Sounds like it might be hard or impossible to fix to me.  I'd love to
be proved wrong though!

If you were thinking of passing time.time() /
clock_gettime(CLOCK_MONOTONIC) along in the Queue too, then you'll
want to know that it can differ by significant amounts on different
processors :-(

Good luck!

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Python 2.6's multiprocessing lock not working on second use?

2009-01-19 Thread Nick Craig-Wood
Gabriel Genellina  wrote:
>  En Fri, 16 Jan 2009 14:41:21 -0200, escribiste en el grupo
>  gmane.comp.python.general
> 
> > I ran a few tests on the new Python 2.6 multiprocessing module before
> > migrating a threading code, and found out the locking code is not
> > working well. In this case, a pool of 5 processes is running, each
> > trying to get the lock and releasing it after waiting 0.2 seconds
> > (action is repeated twice). It looks like the multiprocessing lock
> > allows multiple locking after the second pass. Running the exact same
> > code with threads works correctly.
> 
>  I've tested your code on Windows and I think the problem is on the Queue
>  class. If you replace the Queue with some print statements or write to a
>  log file, the sequence lock/release is OK.
>  You should file a bug report on http://bugs.python.org/

A little bit of instrumentation in the code reveals the problem.  The
Queue class doesn't always return the items in the order that they
were put in.  This should probably be either documented or fixed!  I
suspect it is impossible to fix for a multi-producer Queue though.

The first number is time since the program started.  Tested under linux.

 0.048810 [proc0] Got lock
 0.248679 [proc0] Released lock
 0.248858 [proc0] Got lock
 0.448666 [proc0] Released lock
 0.448859 [proc2] Got lock
 0.648639 [proc2] Released lock
 0.648893 [proc3] Got lock
 0.848633 [proc3] Released lock
 0.848767 [proc3] Got lock
 1.048635 [proc3] Released lock
 1.049090 [proc1] Got lock
 1.248617 [proc1] Released lock
 1.248743 [proc1] Got lock
 1.448634 [proc1] Released lock
 1.448810 [proc4] Got lock
 1.648674 [proc4] Released lock
 1.648831 [proc4] Got lock
 1.849867 [proc2] Got lock<--- out of order
 1.849679 [proc4] Released lock   <--- out of order
 2.048683 [proc2] Released lock

#!/usr/bin/python -*- coding: utf-8 -*-

from multiprocessing import Process, Queue, Lock
from Queue import Empty
from threading import Thread
import time

start = time.time()
def now():
return time.time() - start

class test_lock_process(object):
def __init__(self, lock, id, queue):
self.lock = lock
self.id = id
self.queue = queue
self.read_lock()

def read_lock(self):
for i in xrange(2):
self.lock.acquire()
self.queue.put('%9.6f [proc%d] Got lock' % (now(),
self.id))
time.sleep(.2)
self.queue.put('%9.6f [proc%d] Released lock' % (now(),
self.id))
self.lock.release()

def test_lock(processes=10, lock=Lock(), process=True, queue=None):
print_result = False
if queue == None:
print_result = True
queue = Queue()

threads = []
for i in xrange(processes):
if process: threads.append(Process(target=test_lock_process,
args=(lock,i,queue,)))
else: threads.append(Thread(target=test_lock_process,
args=(lock,i,queue,)))

for t in threads:
t.start()

for t in threads:
t.join()

if print_result:
try:
while True: print queue.get(block=False)
except Empty:
pass

if __name__ == "__main__":
#test_lock(processes=5, process=True)
test_lock(processes=5)

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: initialising a class by name

2009-01-14 Thread Nick Craig-Wood
Krishnakant  wrote:
>  I liked this idea of dispatchTable.
>  is it possible to say some thing like 
>  inst = dispatchTable{"ham"}
>  according to me, inst will become the instance of class ham.
>  Another thing to note is that all the classes are in different modules.
>  So where do I create the dict of classes mapped with the name?

You could use a metaclass which will get round the problem, provided
all the classes have a common baseclass, eg

class BaseThing(object):
registry = {}
class __metaclass__(type):
def __init__(cls, name, bases, dict):
cls.registry[cls.__name__] = cls

class ThingOne(BaseThing):
pass

class ThingTwo(BaseThing):
pass

class ThingThree(BaseThing):
pass

print BaseThing.registry["ThingOne"]
print BaseThing.registry["ThingTwo"]
print BaseThing.registry["ThingThree"]

Which prints





-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: ctype problem

2009-01-14 Thread Nick Craig-Wood
Grimson  wrote:
>  hello out there,
>  I have a problem with c-types.
>  I made a c-library, which expects a pointer to a self defined structure.
> 
>  let the funtion call myfunction(struct interface* iface)
> 
>  and the struct:
>  struct interface
>  {
>  int a;
>  int b;
>  char *c;
>  }
> 
>  the Python ctype port of this structur would be:
> 
>  class INTERFACE(Structure):
>  _fields_ = [("a" c_int),
>("b",c_int),
>("c", c_char)]
> 
>  in my python-struct a create a instance of INTERFACE
> 
>  myiface = INTERFACE()
>  myiface.a = ctypes.c_int(80)
>  myiface.b = ctypes.c_int(22)
>  ...
>  than I make a pointer onto it.
>  p_iface = ctypes.pointer(myiface)
>  and I tried it also with a reference
>  r_iface = ctypes.byref(myiface)
> 
>  but neither myclib.myfunction(p_iface) nor myclib.myfunction(r_iface)
>  works properly. The function is been called but it reads only zeros (0)
>  for each parameter (member in the struct).
> 
>  Where is my fault?

You didn't (or you didn't show) defining the argument types of the
function.

myclib = CDLL("myclib.so") # or whatever
myclib.myfunction.argtypes = [ POINTER(INTERFACE) ]
myclib.myfunction.restype = c_int # or whatever

If you do that then you should be able to pass in myiface directly or
byref(myiface).

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Python tricks

2009-01-13 Thread Nick Craig-Wood
Scott David Daniels  wrote:
>  RajNewbie wrote:
> > On Jan 12, 6:51 pm, Tim Chase  wrote:
>  [a perfectly fine reply which is how I'd solve it]
>  >> RajNewbie wrote:
> >>> ... The solution that I had in mind is:
> >>>while True:
> >>>  ...
> >>>  if : break
> >>>  if inifinte_loop(): raise infiinte_loop_exception
> >>>   Wherein infinite_loop is a generator, which returns true if i > 200
> >>>   def infinite_loop():
> >>>  i = 0
> >>>  while i < 200:
> >>>  i++
> >>>  yield False
> >>>  yield True
> >>> Could somebody let me know whether this is a good option?
> > ...
> > But, I still feel it would be much more aesthetically pleasing if I
> > can call a single procedure like
> > if infinite_loop() -> to do the same.
> > Is it somehow possible? - say by using static variables, iterators --
> > anything?
> 
>  Yes, it is possible.  After:
> 
>   def Fuse(count, exception):
>   for i in range(count):
>   yield None
>   raise exception
> 
>  You can do your loop as:
>   check_infinite = Fuse(200, ValueError('Infinite Loop')).next
>   while True:
>   ...
>   check_infinite()

Or related to the above and the original proposal

class InfiniteLoopError(Exception):
"""An 'infinite' loop has been detected"""

def infinite_loop(max=200):
for i in xrange(max):
yield i
raise InfiniteLoopError()

Use it like this

for i in infinite_loop():
if i > 10:
break
print "iteration", i

or

for i in infinite_loop(10):
print "iteration", i

>  but I agree with Tim that a for ... else loop for the limit is
>  clearer.

Probably yes

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Extending Python with C or C++

2009-01-09 Thread Nick Craig-Wood
Thomas Heller  wrote:
>  Nick Craig-Wood schrieb:
> > Thomas Heller  wrote:
> >>  Nick Craig-Wood schrieb:
> >> > Interesting - I didn't know about h2xml and xml2py before and I've
> >> > done lots of ctypes wrapping!  Something to help with the initial
> >> > drudge work of converting the structures would be very helpful.
> >> > 
> >> > ( http://pypi.python.org/pypi/ctypeslib/ )
> >> > 
> >> > I gave it a quick go and it worked fine.  I had to edit the XML in one
> >> > place to make it acceptable (removing a u from a hex number).
> >> 
> >>  If you are using a recent version of gccxml, then you should use the
> >>  ctypeslib-gccxml-0.9 branch of ctypeslib instead of the trunk. (I should
> >>  really merge the gccxml-0.9 branch into the trunk;-)
> >> 
> >>  If you are already using the branch and the XML file is not accepted,
> >>  then could you please provide a short C code snippet that reproduces
> >>  the problem so that I can fix it?
> > 
> > I'm using gccxml from debian testing 0.9.0+cvs20080525-1 which I guess
> > doesn't have the code from the branch in.
> 
>  I meant the branch in the repository where ctypeslib lives in.
>  Anyway, it doesn't matter anymore since I merged that branch
>  into the ctypeslib trunk.
> 
>  Now the problem with the 'u' suffix that you mentioned should be fixed, and
>  I also made a quick change so that the sized integer types from stdint.h are 
> generated
>  correctly. So, please update your ctypeslib installation and trey again;-)

I gave it a go and I can report success on both counts!

Well done and thank you.
-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Creating new instances of subclasses.

2009-01-08 Thread Nick Craig-Wood
J. Cliff Dyer  wrote:
>  I want to be able to create an object of a certain subclass, depending
>  on the argument given to the class constructor.
> 
>  I have three fields, and one might need to be a StringField, one an
>  IntegerField, and the last a ListField.  But I'd like my class to
>  delegate to the proper subclass automatically, so I can just do:
> 
> >>> f1 = Field('abc')
> >>> f2 = Field('123')
> >>> f3 = Field('D,E,F')
> >>> f1.data
>  'abc'
> >>> f2.data
>  123
> >>> f3.data
>  ['D','E','F']
> >>> type(f1)
> 
> >>> type(f2)
> 
> >>> type(f3)
> 
> 
>  I've come up with a solution, but I suspect there's something cleaner I
>  can do with the inheritance structure of __new__.  I don't like
>  explicitly leapfrogging over Field.__new__ to object.__new__.
> 
>  My attempt is below:
> 
>  def is_list(arg):
>  if ',' in arg:  return True
>  else:  return False
> 
>  def is_integer(arg):
>  try:  int(arg)
>  except ValueError:  return False
>  else:  return True
> 
>  class Field(object):
>  def __new__(cls, a):
>  if is_list(a):
>  return ListField(a)
>  elif is_integer(a):
>  return IntegerField(a)
>  else:
>  return StringField(a)
>  
>  def __init__(self, input):
>  super(Field, self).__init__(input)
>  self.data = input
> 
>  class IntegerField(Field):
>  def __new__(cls, a):
>  return object.__new__(cls, a)
>  def __init__(self, s):
>  super(IntegerField, self).__init__(s)
>  self.s = int(self.s)
>  
>  class ListField(Field):
>  def __new__(cls, a):
>  return object.__new__(cls, a)
>  def __init__(self, s):
>  super(ListField, self).__init__(s)
>  self.s = s.split(',')
> 
>  class StringField(Field):
>  def __new__(cls, a):
>  return object.__new__(cls, a)
> 
>  Is there a cleaner way to do this?  The main problem is that
>  Field.__new__ gets in the way of properly constructing the subclasses
>  once I've used it to select the proper subclass in the first place.

How about this minor modification?

# rest as above

class Field(object):
def __new__(cls, a):
if cls != Field:
return object.__new__(cls, a)
if is_list(a):
return ListField(a)
elif is_integer(a):
return IntegerField(a)
else:
return StringField(a)

def __init__(self, input):
super(Field, self).__init__(input)
self.data = input

class IntegerField(Field):
def __init__(self, s):
super(IntegerField, self).__init__(s)
self.s = int(s)

class ListField(Field):
def __init__(self, s):
super(ListField, self).__init__(s)
self.s = s.split(',')

class StringField(Field):
pass

Or you could go for the full metaclass self registration scheme like
this, which is actually less code since I delegated the "is this ok"
test to the subclass - failing it returns a ValueError.

class Field(object):
registry = []
class __metaclass__(type):
def __init__(cls, name, bases, dict):
cls.registry.append(cls)
def __new__(cls, a):
if cls != Field:
return object.__new__(cls, a)
for subcls in cls.registry:
if subcls == Field:
continue
try:
return subcls(a)
except ValueError:
pass
raise ValueError("Couldn't find subclass")
def __init__(self, input):
super(Field, self).__init__(input)
self.data = input

# Raise a ValueError in init if not suitable args for this subtype

class IntegerField(Field):
def __init__(self, s):
s = int(s)
super(IntegerField, self).__init__(s)
self.s = s

class ListField(Field):
def __init__(self, s):
if ',' not in s:
raise ValueError("Not a list")
super(ListField, self).__init__(s)
self.s = s.split(',')

class StringField(Field):
pass



-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Multiprocessing takes higher execution time

2009-01-08 Thread Nick Craig-Wood
Sibtey Mehdi  wrote:
> I use multiprocessing to compare more then one set of files.
> 
> For comparison each set of files (i.e. Old file1 Vs New file1)
> I create a process,
> 
> Process(target=compare, args=(oldFile, newFile)).start()
> 
> It takes 61 seconds execution time.
> 
> When I do the same comparison without implementing
> multiprocessing, it takes 52 seconds execution time.

> The oldProjects and newProjects will contains zip files
> i.e(oldxyz1.zip,oldxyz2.zip, newxyz2.zip,newxyz2.zip)
> it will unzip both the zip files and compare all the files between old
> and new (mdb files or txt files) and gives the result.
> I do this comparision for n number set of zip files and i assigne each
> set of zip files comparision to a process.

I had a brief look at the code and your use of multiprocessing looks
fine.

How many projects are you processing at once?  And how many MB of zip
files is it?  As reading zip files does lots of disk IO I would guess
it is disk limited rather than anything else, which explains why doing
many at once is actually slower (the disk has to do more seeks).

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Multiprocessing takes higher execution time

2009-01-07 Thread Nick Craig-Wood
Grant Edwards  wrote:
>  On 2009-01-07, Steve Holden  wrote:
> 
> >> I use multiprocessing to compare more then one set of files.
> >> 
> >> For comparison each set of files (i.e. Old file1 Vs New file1)
> >> I create a process,
> >> 
> >> Process(target=compare, args=(oldFile, newFile)).start()
> >> 
> >> It takes 61 seconds execution time.
> >> 
> >> When I do the same comparison without implementing
> >> multiprocessing, it takes 52 seconds execution time.
> 
> > My first suggestion would be: show us some code. We aren't
> > psychic, you know.
> 
>  I am!
> 
>  He's only got one processor, and he's just been bit by Amdahl's
>  law when P<1 and S<1.
> 
>  There you have a perfectly "psychic" answer: an educated guess
>  camoflaged in plausible-sounding but mostly-bullshit buzzwords.
>  A better psychic would have avoided making that one falsifiable
>  statement (he's only got one processor).

;-)

My guess would be that the job is IO bound rather than CPU bound, but
that is covered by Amdahl's Law too where P is approx 0, N
irrelevant...

Being IO bound explains why it takes longer with multiprocessing - it
causes more disk seeks to run an IO bound algorithm in parallel than
running it sequentially.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Extending Python with C or C++

2009-01-07 Thread Nick Craig-Wood
Thomas Heller  wrote:
>  Nick Craig-Wood schrieb:
> > Interesting - I didn't know about h2xml and xml2py before and I've
> > done lots of ctypes wrapping!  Something to help with the initial
> > drudge work of converting the structures would be very helpful.
> > 
> > ( http://pypi.python.org/pypi/ctypeslib/ )
> > 
> > I gave it a quick go and it worked fine.  I had to edit the XML in one
> > place to make it acceptable (removing a u from a hex number).
> 
>  If you are using a recent version of gccxml, then you should use the
>  ctypeslib-gccxml-0.9 branch of ctypeslib instead of the trunk. (I should
>  really merge the gccxml-0.9 branch into the trunk;-)
> 
>  If you are already using the branch and the XML file is not accepted,
>  then could you please provide a short C code snippet that reproduces
>  the problem so that I can fix it?

I'm using gccxml from debian testing 0.9.0+cvs20080525-1 which I guess
doesn't have the code from the branch in.

> > Here are my thoughts on the conversion :-
> > 
> > It converted an interface which looked like this (the inotify interface)
> > 
> > struct inotify_event {
> > int  wd;   /* Watch descriptor */
> > uint32_t mask; /* Mask of events */
> > uint32_t cookie;   /* Unique cookie associating related
> >   events (for rename(2)) */
> > uint32_t len;  /* Size of name field */
> > char name[];   /* Optional null-terminated name */
> > };
> > 
> > Into this
> > 
> > class inotify_event(Structure):
> > pass
> > inotify_event._fields_ = [
> > ('wd', c_int),
> > ('mask', uint32_t),
> > ('cookie', uint32_t),
> > ('len', uint32_t),
> > ('name', c_char * 0),
> > ]
> > 
> > Which is a very good start.  However it defined these which clearly
> > aren't portable
> > 
> > int32_t = c_int
> > uint32_t = c_uint
> > 
> > Whereas it should have been using the ctypes inbuilt types
> > 
> > c_int32
> > c_uint32
> 
>  IMO this would be difficult to achive automatically.  There are cases
>  wher c_int is the correct type, in other cases c_int32 is correct.

Yes it is almost impossible difficult to achieve automatically -
exactly how far do you want to unravel the twisty turny mess of
typedefs that make up uint32_t?  It is easy to change the generated
output since it defines the type in one place.

uint32_t and friends (stdint.h) are standardised in C99 so might it be
reasonable to put some special cases in for them, expecially since the
corresponding types already exist in ctypes?

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Extending Python with C or C++

2009-01-06 Thread Nick Craig-Wood
Ralf Schoenian  wrote:
>  Ryan wrote:
> > I've been using Python for many years now. It's a wonderful language
> > that I enjoy using everyday. I'm now interested in getting to know
> > more about the guts (C/C++) and extending it. But, extending python
> > still seems like a black art to me. Is there anymore docs or info on
> > extending it besides the standard sparse ones (http://www.python.org/
> > doc/2.5.2/ext/intro.html) that may give me more insight? Is there a
> > class available? How can I learn more about the guts of python? How
> > would one go about following an interest in contributing to the
> > development of python.
> 
>  It is not exactly what you are looking for but nevertheless I am 
>  thinking the  article "Automatic C Library Wrapping -- Ctypes from the 
>  Trenches" may be interesting for you. You can find it in the latest 
>  Python Papers issue or simply following the link: 
>  http://ojs.pythonpapers.org/index.php/tpp/article/view/71

Interesting - I didn't know about h2xml and xml2py before and I've
done lots of ctypes wrapping!  Something to help with the initial
drudge work of converting the structures would be very helpful.

( http://pypi.python.org/pypi/ctypeslib/ )

I gave it a quick go and it worked fine.  I had to edit the XML in one
place to make it acceptable (removing a u from a hex number).  The
output of xml2py was an excellent place to start for the conversion,
though I don't think I'd want to use an automated process like in the
paper above as its output needed tweaking.

...

Here are my thoughts on the conversion :-

It converted an interface which looked like this (the inotify interface)

struct inotify_event {
int  wd;   /* Watch descriptor */
uint32_t mask; /* Mask of events */
uint32_t cookie;   /* Unique cookie associating related
  events (for rename(2)) */
uint32_t len;  /* Size of name field */
char name[];   /* Optional null-terminated name */
};

Into this

class inotify_event(Structure):
pass
inotify_event._fields_ = [
('wd', c_int),
('mask', uint32_t),
('cookie', uint32_t),
('len', uint32_t),
('name', c_char * 0),
]

Which is a very good start.  However it defined these which clearly
aren't portable

int32_t = c_int
uint32_t = c_uint

Whereas it should have been using the ctypes inbuilt types

c_int32
c_uint32

Also I don't think c_char * 0 does anything sensible in ctypes,
c_byte * 0 is what is required plus a bit of casting.  This is a
non-standard GNU extension to C though.

All that said though, it looks like a great time saver.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: multiprocessing vs thread performance

2009-01-03 Thread Nick Craig-Wood
mk  wrote:
>  After reading http://www.python.org/dev/peps/pep-0371/ I was under 
>  impression that performance of multiprocessing package is similar to 
>  that of thread / threading. However, to familiarize myself with both 
>  packages I wrote my own test of spawning and returning 100,000 empty 
>  threads or processes (while maintaining at most 100 processes / threads 
>  active at any one time), respectively.
> 
>  The results I got are very different from the benchmark quoted in PEP 
>  371. On twin Xeon machine the threaded version executed in 5.54 secs, 
>  while multiprocessing version took over 222 secs to complete!
> 
>  Am I doing smth wrong in code below?

Yes!

The problem with your code is that you never start more than one
process at once in the multiprocessing example.  Just check ps when it
is running and you will see.

My conjecture is that this is due to the way fork() works under unix.
I think that when the parent forks it yields the CPU to the child.
Because you are giving the child effectively no work to do it returns
immediately, re-awakening the parent, thus serialising your jobs.

If you give the children some work to do you'll see a quite different
result.  I gave each child time.sleep(1) to do and cut down the total
number to 10,000.

$ ./test_multiprocessing.py
== Process 1000 working ==
== Process 2000 working ==
== Process 3000 working ==
== Process 4000 working ==
== Process 5000 working ==
== Process 6000 working ==
== Process 7000 working ==
== Process 8000 working ==
== Process 9000 working ==
== Process 1 working ==
=== Main thread waiting for all processes to finish ===
Total time: 101.382129192

$ ./test_threading.py
== Thread 1000 working ==
== Thread 2000 working ==
== Thread 3000 working ==
== Thread 4000 working ==
== Thread 5000 working ==
== Thread 6000 working ==
== Thread 7000 working ==
== Thread 8000 working ==
== Thread 9000 working ==
== Thread 1 working ==
Total time:  100.659118176

So almost identical results and as expected - we ran 10,000 sleep(1)s
in 100 seconds so we must have been running 100 simultaneously.

If you replace the "time.sleep(1)" with "for _ in xrange(100):
pass" you get this much more interesting answer on my dual core linux
laptop, showing nicely the effect of the contention on the python
global interpreter lock and how multiprocessing avoids it.

$ ./test_multiprocessing.py
== Process 1000 working ==
== Process 2000 working ==
== Process 3000 working ==
== Process 4000 working ==
== Process 5000 working ==
== Process 6000 working ==
== Process 7000 working ==
== Process 8000 working ==
== Process 9000 working ==
== Process 1 working ==
=== Main thread waiting for all processes to finish ===
Total time: 266.808327913

$ ./test_threading.py
== Thread 1000 working ==
== Thread 2000 working ==
== Thread 3000 working ==
== Thread 4000 working ==
== Thread 5000 working ==
== Thread 6000 working ==
== Thread 7000 working ==
== Thread 8000 working ==
== Thread 9000 working ==
== Thread 1 working ==
Total time:  834.81882

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: I always wonder ...

2008-12-24 Thread Craig Allen
this is one of the most subtle trolls I've ever read.

you sir, are a master!


On Dec 22, 7:53 am, s...@pobox.com wrote:
> ... shouldn't people who spend all their time trolling be doing something
> else: studying, working, writing patches which solve the problems they
> perceive to exist in the troll subject?  Is there some online troll game
> running where the players earn points for generating responses to their
> posts?
>
> --
> Skip Montanaro - s...@pobox.com -http://smontanaro.dyndns.org/

--
http://mail.python.org/mailman/listinfo/python-list


Re: Python is slow

2008-12-16 Thread Craig Allen
On Dec 14, 6:38 pm, cm_gui  wrote:
> hahaha, do you know how much money they are spending on hardware to
> make
> youtube.com fast???
>
> > By the way... I know of a very slow Python site called YouTube.com. In
> > fact, it is so slow that nobody ever uses it.

less than they'd spend to implement it in C
--
http://mail.python.org/mailman/listinfo/python-list


Re: How to modify a program while it's running?

2008-12-16 Thread Craig Allen
On Dec 16, 10:25 am, Joe Strout  wrote:
> Here's my situation: I'm making an AIM bot, but the AIM server will
> get annoyed if you log in too frequently (and then lock you out for a
> while).  So my usual build-a-little, test-a-little methodology doesn't
> work too well.
>
> So I'd like to restructure my app so that it can stay running and stay
> logged in, yet I can still update and reload at least most of the
> code.  But I'm not sure what's the best way to do this.  Should I move
> the reloadable code into its own module, and then when I give my bot a
> "reload" command, have it call reload on that module?  Will that work,
> and is there a better way?
>
> Thanks,
> - Joe

yes you want reload.  Design the high level part that knows how to log
in to be able to reload the stuff that changes.  My guess is that is
the best way, though I wouldn't be surprised if there are other
solutions.
--
http://mail.python.org/mailman/listinfo/python-list


Re: How do I manually uninstall setuptools (installed by egg)?

2008-12-12 Thread Nick Craig-Wood
David Cournapeau  wrote:
>  On Thu, Dec 11, 2008 at 10:30 PM, Nick Craig-Wood  
> wrote:
> > David Cournapeau  wrote:
> >>  On Wed, Dec 10, 2008 at 12:04 PM, Chris Rebert  wrote:
> >> > On Tue, Dec 9, 2008 at 6:49 PM,   wrote:
> >> >> On Ubuntu, I accidentally manually installed setuptools
> >> >> http://pypi.python.org/pypi/setuptools/0.6c9 (by running the .egg file
> >> >> as a shell script via sudo), and now realize I should just be using
> >> >> apt to take care of my system Python packages.
> >> >
> >> > Really, why? setuptools has more Python packages/programs available
> >> > and updates faster than Debian.
> >> > It's also likely that some of the Debian Python packages are installed
> >> > using setuptools anyway.
> >> > So, why do you think apt and not setuptools is The Right Way(tm)?
> >>
> >>  Setuptools is certainly not the right way to install packages
> >>  system-wide on debian, it is very likely to break the whole thing.
> >
> > It wouldn't be too difficult to make a .deb target which would collect
> > all the files that did get installed into a package.  It would be a
> > rather rough and ready package but would do the job.
> 
>  Depends what you mean by would do the job: rather rough certainly does
>  not mean "would do the job" for something as essential as a package
>  IMO.

Essentially a package has files in it in a fixed possition in the
filesystem.  The package manager's (dpkg at this level) job is to keep
track of those file and tell you about conflicts.

You can use alien to turn a tar.gz into a perfectly usable debian
package.  It won't have dependencies, or help or any of the other
things a package needs to be a proper package, but it satisfies my
basic needs that all software is installed as packages, so it can be
uninstalled cleanly.

> > The .deb would then be uninstallable in the usual (dpkg --purge) way.
> >
> > Did anyone think about that?
> 
>  Yes, there is stddeb which does that (make a .deb package from a
>  setuptools package).

Cool, I've not seen that before.  Probably because it isn't in debian!

> > easy_install can do that I think...
> 
>  Not without a lot of hoola, unfortunately; for example, it breaks
>  stow, so I have to use specialy scripts to circumvent the damn thing
>  and make it what I tell him to do. I never understood what's easy
>  about easy install: it is broken in so many ways, and has caused me so
>  many pains - even when I was not using - that I do not trust it at
>  all. I only use it to download packages (and even then it manage to
>  fail more than work), and always install them from setup.py afterwards
>  (at which step I of course also have to circumvent setuptools if the
>  package uses setuptools).

;-)

As a mostly linux developer I always install stuff in packages if
possible.

Failing that I'll use bdist_rpm then alien to convert to a deb which
works well enough I find.

However when I have to make my stuff work on Windows, I find
easy_install to be a fantastic timesaver as compared to looking for
the package on a web site, downloading it, unpacking it installing it
and then repeating for all the dependencies.

easy_install is a long way from being a proper package manager though
:-(

I guess it is aiming at the same territory as the cpan installer with
perl.  My experiences with that is that it works very well, but again
splatters untracked files all over your filesystem.  Debian comes with
dh-make-perl which can use CPAN directly to make .debs which works
quite well.  Something similar based on easy_install for python would
be ideal.  It looks like stdeb could become that one day.

-- 
Nick Craig-Wood  -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: [Python.NET] Where does the clr in IronPython look the dll

2008-12-12 Thread Craig Farrow


The dll needs to be on the Python path (sys.path). You can either add to
the path with sys.path.append("c:\") or put your dll in a folder in
the Python site-packages directory and add a .pth file (for Python.NET, 
but not IronPython

 -- it doesn't recognise the .pth files).

Regards,

Craig.

3/12/2008 9:20 p.m. dï, navneet khanna wrote:

Hello Everybody

I am trying to import dll with clr.AddReference("TCdll")
I am getting the following error.

Traceback (most recent call last):
  File "", line 1, in 
clr.AddReference("TCdll")
FileNotFoundException: Unable to find assembly 'TCdll'.
   at Python.Runtime.CLRModule.AddReference(String name)

I have loaded the dll in system32 folder. Have I loaded the dll at the 
right place or do I need to place it anywhere else?

Waiting for your replies.

Regards
Navneet


_
Python.NET mailing list - pythondot...@python.org
http://mail.python.org/mailman/listinfo/pythondotnet



--
http://mail.python.org/mailman/listinfo/python-list


Re: How do I manually uninstall setuptools (installed by egg)?

2008-12-11 Thread Nick Craig-Wood
David Cournapeau <[EMAIL PROTECTED]> wrote:
>  On Wed, Dec 10, 2008 at 12:04 PM, Chris Rebert <[EMAIL PROTECTED]> wrote:
> > On Tue, Dec 9, 2008 at 6:49 PM,  <[EMAIL PROTECTED]> wrote:
> >> On Ubuntu, I accidentally manually installed setuptools
> >> http://pypi.python.org/pypi/setuptools/0.6c9 (by running the .egg file
> >> as a shell script via sudo), and now realize I should just be using
> >> apt to take care of my system Python packages.
> >
> > Really, why? setuptools has more Python packages/programs available
> > and updates faster than Debian.
> > It's also likely that some of the Debian Python packages are installed
> > using setuptools anyway.
> > So, why do you think apt and not setuptools is The Right Way(tm)?
> 
>  Setuptools is certainly not the right way to install packages
>  system-wide on debian, it is very likely to break the whole thing.

It wouldn't be too difficult to make a .deb target which would collect
all the files that did get installed into a package.  It would be a
rather rough and ready package but would do the job.

The .deb would then be uninstallable in the usual (dpkg --purge) way.

Did anyone think about that?

There is a bdist_rpm target for distutils which I often use then use
alien to turn into a .deb.  easy_install is a lot easier though!

>  dpkg is a real package installer, with uninstallation feature, correct
>  dependency handling: if you start installing things with setuptools
>  there, dpkg cannot know anymore how to manage your system.

Agreed.

> That's why it is generally a very bad idea to install things which
> are not managed by dpkg in /usr - be it python or something else
> BTW. It is a much better practice to install from source into
> /usr/local, or your $HOME, etc... Anywhere which is not /usr.

easy_install can do that I think...

I find it odd that easy_install doesn't have
 a) a list what you installed with easy_install
 b) uninstall
in an otherwise excellent program.

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Learning Python now coming from Perl

2008-12-09 Thread Nick Craig-Wood
Roy Smith <[EMAIL PROTECTED]> wrote:
>  In article <[EMAIL PROTECTED]>,
>   Nick Craig-Wood <[EMAIL PROTECTED]> wrote:
>   
> > My favourite mistake when I made the transition was calling methods
> > without parentheses.  In perl it is common to call methods without
> > parentheses - in python this does absolutely nothing!  pychecker does
> > warn about it though.
> > 
> >   perl   -> $object->method
> >   python -> object.method()
> 
>  On the other hand, leaving out the parens returns the function itself, 
>  which you can then call later.  I've often used this to create data-driven 
>  logic.

I didn't say it wasn't useful, just that if you came from Perl like I
did, it is an easy mistake to make ;-)

>  For example, I'm currently working on some code that marshals objects of 
>  various types to a wire protocol.  I've got something like:
> 
>  encoders = {
> SM_INT: write_int,
> SM_SHORT: write_short,
> SM_FLOAT: write_float,
> # and so on
>  }
> 
>  class AnyVal:
> def __init__(self, type, value):
>self.type = type
>self.value = value
> 
>  def write_anyval(any):
> encoders[any.type](any.value)
> 
>  The fact that functions are objects which can be assigned and stored in 
>  containers makes this easy to do.

OO lore says whenever you see a type field in an instance you've gone
wrong - types should be represented by what sort of object you've got,
not by a type field.

Eg http://www.soberit.hut.fi/mmantyla/BadCodeSmellsTaxonomy.htm

"""The situation where switch statements or type codes are needed
should be handled by creating subclasses. """

Here is my first iteration (untested)

class AnyVal:
   def __init__(self, value):
  self.value = value
   def write(self):
  raise NotImplementedError()

class IntVal(AnyVal):
   def write(self):
  # write_int code

class ShortVal(AnyVal):
   def write(self):
  # write_short code

class FloatVal(AnyVal):
   def write(self):
  # write_float code

Then to write an AnyVal you just call any.write()

The initialisation of the AnyVals then becomes

from AnyVal(int_expression, SM_INT)
to IntVal(int_expression)

However, if the types of the expressions aren't known until run time,
then use a dict of class types

AnyValRegistry = {
   SM_INT: IntVal,
   SM_SHORT: ShortVal,
   SM_FLOAT: FloatVal,
   # and so on
}

And initialise AnyVal objects thus

  any = AnyValRegistry[type](value)

This smells of code duplication though and a real potential for a
mismatch between the AnyValRegistry and the actual classes.

I'd probably generalise this by putting the type code in the class and
use a __metaclass__ to autogenerate the AnyValRegistry dict which
would then become an attribute of AnyClass

Eg (slightly tested)

SM_INT=1
SM_SHORT=2
SM_FLOAT=3

class AnyVal(object):
   TYPE = None
   registry = {}
   class __metaclass__(type):
   def __init__(cls, name, bases, dict):
   cls.registry[cls.TYPE] = cls
   def __init__(self, value):
   self.value = value
   @classmethod
   def new(cls, type_code, value):
   """Factory function to generate the correct subclass of AnyVal by type 
code"""
   return cls.registry[type_code](value)
   def write(self):
   raise NotImplementedError()

class IntVal(AnyVal):
   TYPE = SM_INT
   def write(self):
  # write_int code
  print "int", self.value

class ShortVal(AnyVal):
   TYPE = SM_SHORT
   def write(self):
  # write_short code
  print "short", self.value

class FloatVal(AnyVal):
   TYPE = SM_FLOAT
   def write(self):
  # write_float code
  print "float", self.value

You then make new objects with any = AnyVal.new(type_code, value) and
write them with any.write()

Anyone can add a subclass of AnyVal and have it added to the
AnyVal.registry which is neat.

>>> any = AnyVal.new(SM_SHORT, 1)
>>> any
<__main__.ShortVal object at 0xb7e3776c>
>>> any.write()
short 1

>>> any = AnyVal.new(SM_FLOAT, 1.8)
>>> any
<__main__.FloatVal object at 0xb7e37a6c>
>>> any.write()
float 1.8

You could also override __new__ so you could write AnyVal(type_code,
value) to create the object of a new type.  I personally don't think
its is worth it - a factory function is nice and obvious and show
exactly what is going on.

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Learning Python now coming from Perl

2008-12-08 Thread Nick Craig-Wood
Bertilo Wennergren <[EMAIL PROTECTED]> wrote:
>  I'm planning to start learning Python now, using Python 3000.
>  I have no previous Python skills, but I now Perl pretty well.
>  I'm also well experienced with JavaScript.
> 
>  Any pointers and tips how I should go about getting into
>  Python?

Read "Dive Into Python" while following along with your keyboard.

( http://www.diveintopython.org/ free online or paper edition from
your favourite bookseller )

My favourite mistake when I made the transition was calling methods
without parentheses.  In perl it is common to call methods without
parentheses - in python this does absolutely nothing!  pychecker does
warn about it though.

  perl   -> $object->method
  python -> object.method()

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: How can I do this (from Perl) in Python? (closures)

2008-12-04 Thread Nick Craig-Wood
Duncan Booth <[EMAIL PROTECTED]> wrote:
>  [EMAIL PROTECTED] wrote:
> 
> > I just came across http://www.perl.com/pub/a/2002/05/29/closure.html
> > and wanted to try the "canonical example of closures" in Python. I
> > came up with the following, but it fails:
> > 
> > def make_counter(start_num):
> > start = start_num
> > def counter():
> > start += 1
> > return counter
> 
>  The other answers you got will work, but here's an alternative to 
>  consider: you can convert the function into a generator and then just 
>  move the variable inside.
> 
> >>> def make_counter(start_num):
>   def counter():
>   start = start_num
>   while 1:
>   yield start
>   start += 1
>   return counter().next

Interesting...

You can also write it without nested functions / closures

def counter(x):
while 1:
yield x
x += 1

def make_counter(start_num):
return counter(start_num).next

I expect the machinery is similar between generators and closures, but
with generators it is a lot more obvious exactly which version of
which variable you are using!

In fact you can always do what you can do with a closure with the
above technique I think...

def make_closure(*args, **kwargs):
# initialisation to local vars
def closure():
   # stuff, being careful with nonlocal args & kwargs
   return result

vs

def closure(*args, **kwargs):
   # initialisation to local vars
   while 1:
   # normal stuff using args and kwargs
   yield result

def make_closure(*args, **kwargs):
return closure(*args, **kwargs).next

I still prefer doing it explicitly with a class though ;-)

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: How can I do this (from Perl) in Python? (closures)

2008-12-04 Thread Nick Craig-Wood
[EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>  I just came across http://www.perl.com/pub/a/2002/05/29/closure.html
>  and wanted to try the "canonical example of closures" in Python. I
>  came up with the following, but it fails:
> 
>  def make_counter(start_num):
>  start = start_num
>  def counter():
>  start += 1
>  return counter
> 
>  from_ten = make_counter(10)
>  from_three = make_counter(3)
> 
>  print from_ten()   # 10
>  print from_ten()   # 11
>  print from_three() # 3
>  print from_ten()   # 12
>  print from_three() # 4
> 
>  The error message is: "UnboundLocalError: local variable 'start'
>  referenced before assignment". The same thing happens if I omit start
>  and just use start_num directly.
> 
>  How can I do it in Python?

With a class is the best way IMHO.

class make_counter(object):
def __init__(self, start_num):
self.x = start_num
def __call__(self):
x = self.x
self.x += 1
return x

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: How to instantiate in a lazy way?

2008-12-03 Thread Nick Craig-Wood
Slaunger <[EMAIL PROTECTED]> wrote:
>  On 3 Dec., 11:30, Nick Craig-Wood <[EMAIL PROTECTED]> wrote:
> > > ? ? ? ? ?cls = self.__class__
> > > ? ? ? ? ?if attr_name in cls.data_attr_names:
> >
> > self.data_attr_names should do instead of cls.data_attr_names unless
> > you are overriding it in the instance (which you don't appear to be).
> 
>  Yeah, I know. I just like the cls notation for code readability
>  because it tells you that it is a class attribute, which is not
>  instance- dependent.
> 
>  That may be legacy from my Java past, where I used to do it that
>  way.  I know perfectly well that self. would do it. i just find
>  that notation a little misleading

I quite like it... It looks in the instance, then in the class which I
find to be very elegant - you can set a default in the class and
override it on a per object or per subclass basis.

> > I think you want setattr() here - __dict__ is an implemetation detail
> > - classes with __slots__ for instance don't have a __dict__. ?I'd
> > probably do this
> 
>  Oh my, I did not know that. __slots__?? Something new I got to
>  understand.  But you are right. thanks!
> 
> >
> > ? ? ? ? ? ? ? ? ? ?for k, v in zip(("I1", "Q1", "I2", "Q2"), 
> > bytes_to_data(buf)):
> > ? ? ? ? ? ? ? ? ? ? ? ?setattr(self, k, v)
> > ? ? ? ? ? ? ? ? ? ?return object.__getattr__(self, attr_name)
> >
>  And perhaps even more readable (how I do it now, no need to call
>  __getattr__ for an attribute, whcih is already there):
>  ...
>  for attr, value in zip(cls.data_attr_names,
>  bytes_to_data(buf)):
>  setattr(self, attr, value)
> 
>  return getattr(self, attr_name)

I wrote the object.__getattr__ call to stop recursion troubles.  If
you are sure you've set the attribute then plain getattr() is OK I
guess...

>  In this process I have actaully managed to isolate all the
>  ...OnDemand stuff in an abstract PayloadOnDemand class
> 
>  I can now use this "decorator-like"/helper class to very easily
>  make an ...OnDemand variant of a class by just doing multiple
>  inheritance - no implementation:
> 
>  class PayloadBaconAndEggsOnDemand(PayloadOnDemand, PayloadBaconAndEggs): pass

You've reinvented a Mixin class!

  http://en.wikipedia.org/wiki/Mixin

It is a good technique.

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: How to instantiate in a lazy way?

2008-12-03 Thread Nick Craig-Wood
Slaunger <[EMAIL PROTECTED]> wrote:
>  Just wanted to show the end result in its actual implementation!
> 
>  I ended up *not* making a decorator, as I already had a good idea
>  about how to do it
>  using __getattr__
> 
>  class PayloadDualFrqIQOnDemand(PayloadDualFrqIQ):
>  """
>  This class has the same interface as its parent,
>  but unlike its parent, it is instantiated without
>  its payload parsed up in its instance attributes
>  Q1, I1, Q2 and I2. Instead it stores a reference to
>  the file object in which the Payload data can be
>  read, the file position and
>  the version of the payload data.
> 
>  On accessing one of the data attributes, the actual
>  payload data are read from the file, and the reference to
>  the file object is unbound.
>  The constructor signature is therefore different from its
>  parent as it takes the file object, position and version
>  as arguments instead of the actual data.
>  """
> 
>  @classmethod
>  def _unpack_from_file(cls, f, samples, ver):
>  bytes = samples * cls.bytes_per_sample
>  initial_pos = f.tell()
>  f.seek(initial_pos + bytes) #Skip over the payload
>  return cls(f, initial_pos, samples, ver)
> 
>  @classmethod
>  def unpack_from_ver3_file(cls, f, samples):
>  return cls._unpack_from_file(f, samples, ver=3)
> 
>  @classmethod
>  def unpack_from_ver4_file(cls, f, samples):
>  return cls._unpack_from_file(f, samples, ver=4)
> 
>  data_attr_names = frozenset(["Q1", "I1", "Q2", "I2"])
> 
>  def __init__(self, a_file, a_file_position, samples, a_version):
>  """
>  Returns an instance where the object knows where to
>  look for the payload but it will only be loaded on the
>  first attempt to read one of the data attributes
>  in a "normal" PayloadDualFrqIQ object.
>  """
>  self.f = a_file
>  self.file_position = a_file_position
>  self.samples = samples
>  self.ver = a_version
> 
>  def __getattr__(self, attr_name):
>  """
>  Checks if a request to read a non-existing data attribute
>  has an attribute corresponding to one of the data attributes
>  in a normal PayloadDualFrqIQ object.
> 
>  If true, the data attributes are created and bound to the
>  object using the file object instance, the file position
>  and the version.
> 
>  It is a prerequisite that the file object is still open.
>  The function leaves the file object at the file position
>  when it entered the method
> 
>  """
>  cls = self.__class__
>  if attr_name in cls.data_attr_names:

self.data_attr_names should do instead of cls.data_attr_names unless
you are overriding it in the instance (which you don't appear to be).

>  initial_pos = self.f.tell()
>  try:
>  bytes = self.samples * cls.bytes_per_sample
>  self.f.seek(self.file_position)
>  buf = self.f.read(bytes)
>  if self.ver == 3:
>  bytes_to_data = cls._v3_byte_str_to_data
>  elif self.ver == 4:
>  bytes_to_data = cls._v4_byte_str_to_data
>  else:
>  raise TermaNotImplemented, \
>  "Support for ver. %d not implemented." %
>  self.ver
>  I1, Q1, I2, Q2 = bytes_to_data(buf)
>  self.__dict__["I1"] = I1
>  self.__dict__["Q1"] = Q1
>  self.__dict__["I2"] = I2
>  self.__dict__["Q2"] = Q2
>  return self.__dict__[attr_name]

I think you want setattr() here - __dict__ is an implemetation detail
- classes with __slots__ for instance don't have a __dict__.  I'd
probably do this

   for k, v in zip(("I1", "Q1", "I2", "Q2"), 
bytes_to_data(buf)):
   setattr(self, k, v)
   return object.__getattr__(self, attr_name)

That then has duplicated a list of "I1", "Q1" etc which needs to be
factored out.

>  finally:
>  # Restore file position
>  self.f.seek(initial_pos)
>  # Unbind lazy attributes
>  del self.f
>  del self.ver
> 

Re: Do more imported objects affect performance

2008-12-03 Thread Nick Craig-Wood
On Tue, Dec 02, 2008 at 10:53:47PM -0500, Steve Holden wrote:
> Pardon me for intruding, but timings here are entirely the wrong focus
> for a Python newcomer. Given that imports are super-optimized (i.e. the
> code in the module is only performed once) such a small difference in
> timing is inconsequential, I would suggest.
> 
> As long as "from module import *" is only ever used with modules
> specifically designed to support it, the other forms can be used as
> required. Sure, there is a timing difference between
> 
>   import module
> ...
>   module.something()
> 
> and
> 
>   from module import something
> ...
>   something()
> 
> but that's hardly the point. Learning to write sound Python is *much*
> more important that learning to write fast Python, and often the two
> coincide anyway.
> 
> It was true when Kernighan and Plauger wrote it forty years ago and it's
> true now: "First, make it work. Then, *if it doesn't work fast enough*,
> make it work faster".

You are 100% right of course Steve.  I was just trying to answer the
specific question "which is faster" question which probably isn't
helpful for new Python programmers to focus on.

PS I enjoyed your book :-)

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: Confused about class relationships

2008-12-02 Thread Craig Allen
what you have is a totally acceptable factory system.  Not sure why
you are using a generator, but that's another matter.

I agree with the previous replies regarding inheritance... this is not
a case for inheritance.  You could, however, have Bar be a borg with
the Bar factory built in as a class method to the Bar class itself if
you want the whole job done by one component.  Questions to lead to or
away from that: will Engine every create things besides Bars?
--
http://mail.python.org/mailman/listinfo/python-list


Re: HELP!...Google SketchUp needs a Python API

2008-12-02 Thread Craig Allen
> Just remember thought that if you threat Python like a
> hammer, suddenly everything will look like a bail.
>

don't you mean if you use Python like a pitchfork?
--
http://mail.python.org/mailman/listinfo/python-list


Re: Do more imported objects affect performance

2008-12-02 Thread Nick Craig-Wood
Steven D'Aprano <[EMAIL PROTECTED]> wrote:
>  On Tue, 02 Dec 2008 11:12:31 +0000, Nick Craig-Wood wrote:
> 
> > I prefer the "from module import function".  That means that if "module"
> > doesn't supply "function" it raises an exception at compile time, not
> > run time when you try to run "module.function".
> 
>  Wanna bet?
> 
> 
> >>> def spam():
>  ... from math import harmonic_series
>  ... return harmonic_series()
>  ...
> >>> dis.dis(spam)
>2   0 LOAD_CONST   1 (-1)
>3 LOAD_CONST   2 (('harmonic_series',))
>6 IMPORT_NAME  0 (math)
>9 IMPORT_FROM  1 (harmonic_series)
>   12 STORE_FAST   0 (harmonic_series)
>   15 POP_TOP
> 
>3  16 LOAD_FAST0 (harmonic_series)
>   19 CALL_FUNCTION0
>   22 RETURN_VALUE
> >>> spam()
>  Traceback (most recent call last):
>File "", line 1, in 
>File "", line 2, in spam
>  ImportError: cannot import name harmonic_series
> 
> 
>  The same thing happens if the from...import is at the top level of the 
>  module, except that compilation is immediately followed by
>  execution.

You are technically right I am sure.

However the error happens when you import the module with the error
in, not when you run stuff from it which is the major difference.

$ echo -e "from os import sausage\n" > import_test.py
$ python
Python 2.5.2 (r252:60911, Sep 29 2008, 21:15:13)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> # This does produce an error
>>> import import_test
Traceback (most recent call last):
  File "", line 1, in 
  File "import_test.py", line 1, in 
from os import sausage
ImportError: cannot import name sausage
>>>
$ # This produces an error also
$ python import_test.py
Traceback (most recent call last):
  File "import_test.py", line 1, in 
from os import sausage
ImportError: cannot import name sausage
$  

Unlike

$ echo -e "import os\ndef f(): os.sausage\n" > import_test.py
$ python
Python 2.5.2 (r252:60911, Sep 29 2008, 21:15:13)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> # This doesn't produce an error
>>> import import_test
>>> # Until you actually call it
>>> import_test.f()
Traceback (most recent call last):
  File "", line 1, in 
  File "import_test.py", line 2, in f
def f(): os.sausage
AttributeError: 'module' object has no attribute 'sausage'
>>>
$ # No error here either
$ python import_test.py
$

> > It then becomes very easy to see which functions you use from any
> > given module too.
> 
>  If that's important to you. Personally, I find it more useful to know 
>  where a function is defined.

We can agree to differ there I'm sure ;-)

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


Re: How to instantiate in a lazy way?

2008-12-02 Thread Nick Craig-Wood
Slaunger <[EMAIL PROTECTED]> wrote:
>  On 2 Dec., 11:30, Nick Craig-Wood <[EMAIL PROTECTED]> wrote:
> 
> >
> > For 4 attributes I'd probably go with the __getattr__.
> >
>  OK, I'll do that!
> 
> > Or you could easily write your own decorator to cache the result...
> >
> > Eg http://code.activestate.com/recipes/363602/
> 
>  Cool. I never realized I could write my own decorators!
>  I've so far only used them for
>  @classmethod, @staticmethod and stuff like that.
>  User defined decorators are nice and fun to do as well.
>  I just hope it will be understandable
>  in four years also...

Actually that decorator does exactly what you want.  It replaces the
attribute with the actual data so after the first call there is no
overhead.  I'd use that I think!

> > > ?With the property methology you do the value check on each get, which
> > > ?does not look as "clean". The property methology is also a little less
> > > ?arcane I guess for less experienced Python programmers to understand
> > > ?when re-reading the code.
> >
> > Less magic is how I would put it. ?Magic is fun to write, but a pain
> > to come back to. ?Over the years I find I try to avoid magic more and
> > more in python.
> >
>  Ah, I see. I hope you do not consider user defined decorators
>  "magic" then? ;-)

Well the magic is contained in the decorator so it isn't splattered
throughout your code!

Decorators are part of the meta-programming toolbag along with
metaclasses.  Used sparingly they can be extrememly useful!

>  Again, thank you for your assistance, Nick!

No problem!

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


<    1   2   3   4   5   6   7   8   9   >