SIMD powered Python
Hi! Is there any I&D ongoing about using SIMD [1] instructions, like SSE [2], to speed up Python, especially regarding functional features, like list comprehension, map and reduce, etc.. ? Best regards, Hugo Ferreira -- [1] http://en.wikipedia.org/wiki/SIMD [2] http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions -- http://mail.python.org/mailman/listinfo/python-list
Re: Typed named groups in regular expression
Both Paddy (hackish) and McGuire (right tool for the job) ideas sound
very interesting ;-) I'll definitely research on them further.
Thanks for the support...
On 19 May 2007 04:39:58 -0700, Paul McGuire <[EMAIL PROTECTED]> wrote:
> On May 19, 12:32 am, Paddy <[EMAIL PROTECTED]> wrote:
> > On May 16, 6:58 pm, "Hugo Ferreira" <[EMAIL PROTECTED]> wrote:
> >
> > > Hi!
> >
> > > Is it possible to "automagically" coerce the named groups to python
> > > types? e.g.:
> >
> > > >>> type(re.match('(?P\d*)', '123').groupdict()['x'])
> >
> > >
> >
> > > But what I'm looking forward is for the type to be 'int'.
> >
> > > Cheers!
> >
> > > Hugo Ferreira
> >
> > If you do a ot of that sort of thing in many programs
> > then it might be worth your while to set up a framework
> > that does it. Something like adding an underscore
> > then the name of a type conversion function to all
> > group names, and creating a function to apply the
> > type convertion function to all named groups of a
> > match object.
> > - Paddy.
>
> pyparsing might just be this sort of framework, in that you can attach
> parse actions to elements within a grammar. At parse time, the parse
> action is called with the list of tokens currently matched.
>
> >>> from pyparsing import Regex
> >>> re = Regex( r"(\d*)" ).setResultsName("x")\
> ....setParseAction(lambda t:int(t[0]))
> >>> results = re.parseString("123")
> >>> print results.x
> 123
>
> -- Paul
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Typed named groups in regular expression
Hi!
Is it possible to "automagically" coerce the named groups to python types? e.g.:
>>> type(re.match('(?P\d*)', '123').groupdict()['x'])
But what I'm looking forward is for the type to be 'int'.
Cheers!
Hugo Ferreira
--
http://mail.python.org/mailman/listinfo/python-list
Python Dijkstra Shortest Path
While trying to optimize this: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/119466 ... and still have a fast edge lookup, I've done the following tweaks: class PathFind(object): def __init__(self): self.G = defaultdict(dict) self.E = defaultdict(dict) def addEdge(self, va, vb, cost, edge, way): if way == -1: (vb, va) = (va, vb) self.G[va][vb] = edge if not way: self.G[vb][va] = edge self.E[edge] = cost def findPath(self, start, end): def flatten(L): # Flatten linked list of form [0,[1,[2,[ while len(L) > 0: yield L[0] L = L[1] q = [(0, start, ())] # Heap of (cost, path_head, path_rest). visited = set() # Visited vertices. # Eliminate the dots pattern push, pop, add = heapq.heappush, heapq.heappop, visited.add G, E = self.G, self.E while True: (cost, v1, path) = pop(q) if v1 not in visited: add(v1) path = (v1, path) if v1 == end: return list(flatten(path))[::-1] for (v2, e) in G[v1].iteritems(): if v2 not in visited: push(q, (cost + E[e], v2, path)) def getEdges(self, path): edges = [] for ix in xrange(len(path) - 1): edges.append(self.G[path[ix]][path[ix + 1]]) return edges addEdge() is used to initialize the Graph. It takes a way param: if -1 then the vertex order is reversed; 0 means it is bidir; 1 vertex order is maintained. This version maintains two Dictionaries: one for pair_of_vertexes->edge and another for edge->cost. findPath() will find the path between two vertexes and getEdges(findPath()) will return this list of edges for that path. The "Eliminate the dots" pattern actually improved performance in about 10%. Still, I'm looking for some way to improve even more the performance, while maintaining the dijkstra intact (I don't want an A*). Psyco gave me about 20% of improvement. I wonder if anyone has any idea to make this faster (maybe map? list comprehension? some python trick?)... Profiling blames the heapq (eheh). On a my CoreDuo T2300, it takes 1.6seconds to find a path of 800 vertexes in an half a million mesh. Greetings! Hugo Ferreira -- http://mail.python.org/mailman/listinfo/python-list
Drawing Text on a Path
Hi everyone, I need to draw some curved text on an image. I've digged both PIL and aggdraw documentation for this kind of feature, but found nothing. However, AGG itself does support rendering text along a path, but I can't seem to figure out how to deal with the API/Wrapper differences... Could someone give me an hand here? Thanks in advance! Hugo Ferreira -- http://mail.python.org/mailman/listinfo/python-list
Re: Synchronous shutil.copyfile()
I'm banging my head over this one... sometimes it just works... *argh*
Here's a snip of my problem:
import win32com.client
import shutil
import time
engine = win32com.client.Dispatch("DAO.DBEngine.36")
table = 'STUFFTODELETE'
dstFilename = 'in.mdb'
workingfile = 'work.mdb'
shutil.copyfile(dstFilename, workingfile)
print 'Sleeping 5 after copy...'
time.sleep(5)
dstDb = engine.OpenDatabase(workingfile)
print 'Sleeping 5 after open...'
time.sleep(5)
dstDb.Execute('DELETE FROM ' + table)
At at *exactly* this line, I get the error:
C:\Python24\lib\site-packages\win32com\gen_py\00025E01-
--C000-0046x0x5x0.py510
com_error: (-2147352567, 'Exception occurred.', (0, 'DAO.Database', 'Not a
valid bookmark.', 'jeterr40.chm', 5003159, -2146825129), None)
Sometimes... Yes, the database is a %"#$"# MSAccess. This script is running
under windows.
Any ideas?
On 1/31/07, Hugo Ferreira <[EMAIL PROTECTED]> wrote:
Well.. Thx for the answers.
The only way I had to make it work was to use a time.sleep(10) after the
shutil.copyfile(). Since this is a night-run script, I can waste 10
seconds, but it still knocks me out "why" it happens...
Cheers!
Hugo Ferreira
On 30 Jan 2007 18:06:15 + (GMT), Matthew Woodcraft <[EMAIL PROTECTED]>
wrote:
>
> Hugo Ferreira < [EMAIL PROTECTED]> wrote:
> > I have a problem. I'm using calling shutil.copyfile() followed by
> > open(). The thing is that most of the times open() is called before
> > the actual file is copied. I don't have this problem when doing a
> > step-by-step debug, since I give enough time for the OS to copy the
> > file, but at run-time, it throws an exception.
> >
> > Is there anyway to force a sync copy of the file (make python wait for
> > the completion)?
>
> shutil.copyfile() closes both files before it returns, so I suspect
> this is an OS-level bug.
>
> The most likely culprits are buggy network filesystems and buggy
> on-access virus scanners.
>
> -M-
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Synchronous shutil.copyfile()
Well.. Thx for the answers. The only way I had to make it work was to use a time.sleep(10) after the shutil.copyfile(). Since this is a night-run script, I can waste 10 seconds, but it still knocks me out "why" it happens... Cheers! Hugo Ferreira On 30 Jan 2007 18:06:15 + (GMT), Matthew Woodcraft < [EMAIL PROTECTED]> wrote: Hugo Ferreira <[EMAIL PROTECTED]> wrote: > I have a problem. I'm using calling shutil.copyfile() followed by > open(). The thing is that most of the times open() is called before > the actual file is copied. I don't have this problem when doing a > step-by-step debug, since I give enough time for the OS to copy the > file, but at run-time, it throws an exception. > > Is there anyway to force a sync copy of the file (make python wait for > the completion)? shutil.copyfile() closes both files before it returns, so I suspect this is an OS-level bug. The most likely culprits are buggy network filesystems and buggy on-access virus scanners. -M- -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Synchronous shutil.copyfile()
Hi there, I have a problem. I'm using calling shutil.copyfile() followed by open(). The thing is that most of the times open() is called before the actual file is copied. I don't have this problem when doing a step-by-step debug, since I give enough time for the OS to copy the file, but at run-time, it throws an exception. Is there anyway to force a sync copy of the file (make python wait for the completion)? Thanks in advance! Hugo Ferreira -- http://mail.python.org/mailman/listinfo/python-list
re.sub and empty groups
Hi!
I'm trying to do a search-replace in places where some groups are
optional... Here's an example:
>> re.match(r"Image:([^\|]+)(?:\|(.*))?", "Image:ola").groups()
('ola', None)
>> re.match(r"Image:([^\|]+)(?:\|(.*))?", "Image:ola|").groups()
('ola', '')
>> re.match(r"Image:([^\|]+)(?:\|(.*))?", "Image:ola|ole").groups()
('ola', 'ole')
The second and third results are right, but not the first one, where
it should be equal to the second (i.e., it should be an empty string
instead of None). This is because I want to use re.sub() and when the
group is None, it blows up with a stack trace...
Maybe I'm not getting the essence of groups and non-grouping groups.
Someone care to explain (and, give the correct solution :)) ?
Thanks in advance,
Hugo Ferreira
--
GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85
--
http://mail.python.org/mailman/listinfo/python-list
Best memory analyzer?
Hi! I'm using the BGL bindings, but I think I'm having a giant memory leak. Thing is, I'm not sure if it is the bound C++ variables that are not being trashed, or if the leak is inside my program. What is the best way to debug this? Thanks! Hugo Ferreira -- GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85 -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way for inter-process communication in Python
There is another option that I thought while writing this... I can use the database for data communication. Like having a table with both in and out parameters. On the client-side, I fill the in parameters columns. Then I signal the external application which reads the parameters, and write the output. Which raises me the following question... How do I signal a python application under windows? (Is it possible to send something like a SIGHUP?) Cheers! On 12/4/06, Hugo Ferreira <[EMAIL PROTECTED]> wrote: > Hi everyone! > > Here's the current scenario: I have a program in Python that computes > something very fast (<1s), but it takes a considerable amount of time > to read the startup data (>90s). Since the startup data is pretty > static, I want this program to be resident and ready in memory all the > time. > > The client-side of this program is a function in PostgreSQL. For the > sake of simplicity, let's assume it is another program in Python that > will be asking the resident one for results on-demand. Let's also > assume that there will be dozens of concurrent requests. > > My problem is: what is the fastest, easiest way to accomplish this > inter-process communication? The data between them will be very small: > 1Kb max per request. I've thought about SOAP, sockets and named > pipes... But since I have no experience on any of them using Python, I > can't decide which way is better... > > Just a few considerations: Python version is 2.4. PostgreSQL version > is 8.2RC1, OS version is Windows Server 2003. > > Thanks in advance, > > Hugo Ferreira > > -- > GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85 > -- GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85 -- http://mail.python.org/mailman/listinfo/python-list
Best way for inter-process communication in Python
Hi everyone! Here's the current scenario: I have a program in Python that computes something very fast (<1s), but it takes a considerable amount of time to read the startup data (>90s). Since the startup data is pretty static, I want this program to be resident and ready in memory all the time. The client-side of this program is a function in PostgreSQL. For the sake of simplicity, let's assume it is another program in Python that will be asking the resident one for results on-demand. Let's also assume that there will be dozens of concurrent requests. My problem is: what is the fastest, easiest way to accomplish this inter-process communication? The data between them will be very small: 1Kb max per request. I've thought about SOAP, sockets and named pipes... But since I have no experience on any of them using Python, I can't decide which way is better... Just a few considerations: Python version is 2.4. PostgreSQL version is 8.2RC1, OS version is Windows Server 2003. Thanks in advance, Hugo Ferreira -- GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85 -- http://mail.python.org/mailman/listinfo/python-list
Re: PyParsing and Headaches
Chris,
Thanks for your quick answer. That changes a lot of stuff, and now I'm able
to do my parsing as I intended to.
Paul,
Thanks for your detailed explanation. One of the things I think is missing
from the documentation (or that I couldn't find easy) is the kind of
explanation you give about 'The Way of PyParsing'. For example, It took me a
while to understand that I could easily implement simple recursions using
OneOrMany(Group()). Or maybe things were out there and I didn't searched
enough...
Still, fwiw, congratulations for the library. PyParsing allowed me to do in
just a couple of hours, including learning about it's API (minus this little
inconvenient) what would have taken me a couple of days with, for example,
ANTLR (in fact, I've already put aside ANTLR more than once in the past for
a built-from-scratch parser).
Cheers,
Hugo Ferreira
On 11/22/06, Paul McGuire <[EMAIL PROTECTED]> wrote:
"Bytter" <[EMAIL PROTECTED]> wrote in message
news:[EMAIL PROTECTED]
> Hi,
>
> I'm trying to construct a parser, but I'm stuck with some basic
> stuff... For example, I want to match the following:
>
> letter = "A"..."Z" | "a"..."z"
> literal = letter+
> include_bool := "+" | "-"
> term = [include_bool] literal
>
> So I defined this as:
>
> literal = Word(alphas)
> include_bool = Optional(oneOf("+ -"))
> term = include_bool + literal
>
> The problem is that:
>
> term.parseString("+a") -> (['+', 'a'], {}) # OK
> term.parseString("+ a") -> (['+', 'a'], {}) # KO. It shouldn't
> recognize any token since I didn't said the SPACE was allowed between
> include_bool and literal.
>
As Chris pointed out in his post, the most direct way to fix this is to
use
Combine. Note that Combine does two things: it requires the expressions
to
be adjacent, and it combines the results into a single token. For
instance,
when defining the expression for a real number, something like:
realnum = Optional(oneOf("+ -")) + Word(nums) + "." + Word(nums)
Pyparsing would parse "3.14159" into the separate tokens ['', '3', '.',
'14159']. For this grammar, pyparsing would also accept "2. 23" as ['',
'2', '.', '23'], even though there is a space between the decimal point
and
"23". But by wrapping it inside Combine, as in:
realnum = Combine(Optional(oneOf("+ -")) + Word(nums) + "." + Word(nums))
we accomplish two things: pyparsing only matches if all the elements are
adjacent, with no whitespace or comments; and the matched token is
returned
as ['3.14159']. (Yes, I left off scientific notation, but it is an
extension of the same issue.)
Pyparsing in general does implicit whitespace skipping; it is part of the
zen of pyparsing, and distinguishes it from conventional regexps (although
I
think there is a new '?' switch for re's that puts '\s*'s between re terms
for you). This is to simplify the grammar definition, so that it doesn't
need to be littered with "optional whitespace or comments could go here"
expressions; instead, whitespace and comments (or "ignorables" in
pyparsing
terminology) are parsed over before every grammar expression. I
instituted
this out of recoil from a previous project, in which a co-developer
implemented a boolean parser by first tokenizing by whitespace, then
parsing
out the tokens. Unfortunately, this meant that "color=='blue' &&
size=='medium'" would not parse successfully, instead requiring "color ==
'blue' && size == 'medium'". It doesn't seem like much, but our support
guys got many calls asking why the boolean clauses weren't matching. I
decided that when I wrote a parser, "y=m*x+b" would be just as parseable
as
"y = m * x + b". For that matter, you'd be surprised where whitespace and
comments sneak in to people's source code: spaces after left parentheses
and
comments after semicolons, for example, are easily forgotten when spec'ing
out the syntax for a C "for" statement; whitespace inside HTML tags is
another unanticipated surprise.
So looking at your grammar, you say you don't want to have this be a
successful parse:
term.parseString("+ a") -> (['+', 'a'], {})
because, "It shouldn't recognize any token since I didn't said the SPACE
was
allowed between include_bool and literal." In fact, pyparsing allows
spaces
by default, that's why the given parse succeeds. I would turn this
question
around, and ask you in te
PyParsing and Headaches
Hi,
I'm trying to construct a parser, but I'm stuck with some basic stuff... For
example, I want to match the following:
letter = "A"..."Z" | "a"..."z"
literal = letter+
include_bool := "+" | "-"
term = [include_bool] literal
So I defined this as:
literal = Word(alphas)
include_bool = Optional(oneOf("+ -"))
term = include_bool + literal
The problem is that:
term.parseString("+a") -> (['+', 'a'], {}) # OK
term.parseString("+ a") -> (['+', 'a'], {}) # KO. It shouldn't recognize any
token since I didn't said the SPACE was allowed between include_bool and
literal.
Can anyone give me an hand here?
Cheers!
Hugo Ferreira
BTW, the following is the complete grammar I'm trying to implement with
pyparsing:
## L ::= expr | expr L
## expr ::= term | binary_expr
## binary_expr ::= term " " binary_op " " term
## binary_op ::= "*" | "OR" | "AND"
## include_bool ::= "+" | "-"
## term ::= ([include_bool] [modifier ":"] (literal | range)) | ("~"
literal)
## modifier ::= (letter | "_")+
## literal ::= word | quoted_words
## quoted_words ::= '"' word (" " word)* '"'
## word ::= (letter | digit | "_")+
## number ::= digit+
## range ::= number (".." | "...") number
## letter ::= "A"..."Z" | "a"..."z"
## digit ::= "0"..."9"
And this is where I got so far:
word = Word(nums + alphas + "_")
binary_op = oneOf("* and or", caseless=True).setResultsName("operator")
include_bool = oneOf("+ -")
literal = (word | quotedString).setResultsName("literal")
modifier = Word(alphas + "_")
rng = Word(nums) + (Literal("..") | Literal("...")) + Word(nums)
term = ((Optional(include_bool) + Optional(modifier + ":") + (literal |
rng)) | ("~" + literal)).setResultsName("Term")
binary_expr = (term + binary_op + term).setResultsName("binary")
expr = (binary_expr | term).setResultsName("Expr")
L = OneOrMore(expr)
--
GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85
--
http://mail.python.org/mailman/listinfo/python-list
Rendering Vector Graphics
Hi ppl,I need to render high-quality vector graphics with Python. I was thinking of something like 'cairo', though I need to run under win32 and can't find a pycairo package for it. Suggestions?Thanks, Hugo Ferreira-- GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85 -- http://mail.python.org/mailman/listinfo/python-list
Re: Proposal: [... for ... while cond(x)]
I actually like the proposal...If the argument to remove map, lambda and filter can be that list comprehension is more "readable", then why can't this one also use it?Which reminds me this discussion: http://awkly.org/archive/can-python-take-advantage-of-mapreduce/ Cheers!HugoOn 8/6/06, Slawomir Nowaczyk <[EMAIL PROTECTED]> wrote: On Sun, 06 Aug 2006 18:59:39 + (GMT)Duncan Booth < [EMAIL PROTECTED]> wrote:#> >> > I suggest a new extension of the list comprehension syntax:#> >> >#> >> > [x for x in xs while cond(x)]#> >> > #> >> > which would be equivalent to#> >> >#> >> > list(itertools.takewhile(cond, xs))#> >>#> >> What would this syntax offer that:#> >> #> >>[x for x in takewhile(cond, xs)]#> >>#> >> doesn't currently offer?#> >#> > The same thing that [f(x) for x in xs] offers that map(f, xs) doesn't,#> > and the same thing that [x for x in xs if f(x)] offers that filter(f, #> > xs) doesn't. It's more "pythonic". You can use an _expression_ for cond#> > instead of a lambda.#> >#> No, the list comprehension lets you write an _expression_ directly #> avoiding a function call, and it also allows you to add in a#> condition which can be used to filer the sequence.I am not sure if I understand you correctly, but... Does it?>>> a = [0,1,2,3,7,8,9] >>> [x for x in takewhile(lambda x: x in a, range(10))][0, 1, 2, 3]>>> [x for x in takewhile(x in a, range(10))]Traceback (most recent call last): File "", line 1, in ? TypeError: 'bool' object is not callableDid I miss something? Notice that using "if" gives different result:>>> [x for x in range(10) if x in a][0, 1, 2, 3, 7, 8, 9]#> Your proposal adds nothing. Well, I am not sure how useful the proposal really is, but it seems toadd *something* if it would allow for things like:[x for x in range(10) while x in a]-- Best wishes, Slawomir Nowaczyk ( [EMAIL PROTECTED] )Women who seek to be equal to men lack ambition.-- http://mail.python.org/mailman/listinfo/python-list-- GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85 -- http://mail.python.org/mailman/listinfo/python-list
Re: Type signature
Which is expecially true when using IDEs with auto-completion.Using VisualStudio/MonoDevelop and C# I rarely need to look at the documentation because I can quickly see what a method accept and returns. And when I need to pass flags or options, enums are much more neat and encapsulated. With Python I'm constantly looking at the documentation when surfing a library. I personally like the terse code and abstraction features of Python which is making me slowly writing more and more tools in it. Still, I have to agree that there are edges (like these) that must be sharpened out...On 7/24/06, paul kölle <[EMAIL PROTECTED]> wrote: Marc 'BlackJack' Rintsch wrote:> In <[EMAIL PROTECTED]>, Yacao Wang> wrote:>>> However, type signatures are not only a kind of information provided for >> the compiler, but also for the programmer, or more important, for the>> programmer. Without it, we have to "infer" the return type or required>> agument types of a function, and this can't be done without seeing the >> implementation of it,>> That's what documentation is meant for. If you are forced to look at the> implementation, the documentation is bad.I think the OP refers to reading the *code*, the documentation might not exist (yet). Sometimes I feel python is easier to write than to read and missing argument type declarations (just for documentation purposes)are IMHO one reason. Another are missing (optional) argument type checks at runtime. Something like WrongArgumentType exceptions insteadof rather unspecific AttributeError from deep inside the code would bevery convenient.Yes docstrings are nice but sometimes a simple: foo(int:param1, string:param2) is way better than:foo(param1, param2): """ @type param1: integer @type parame2: string """cheers Paul-- http://mail.python.org/mailman/listinfo/python-list-- GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85 -- http://mail.python.org/mailman/listinfo/python-list
