Re: mmap resizing macosx unix
For sake of documenting for list, I ended up opening file a second time with 'a', padding it to extend its size (previous size + additional bytes to accommodate the insertion), closing file, open file with 'r+', open a second mmap with with new size, moving text to new location, then inserting the replacement text. Not quite as smooth as doing a resize(), moving text and inserting if this had worked. Regards, David On 5-Apr-09, at 11:34 AM, Philip Semanchuk wrote: On Apr 5, 2009, at 10:28 AM, David Pratt wrote: Hi. I have been experimenting with mmap recently. I determined how to read and write properly from it and so search and replace on large files. The problem I am having is with replaces that are larger than the mmap. In this instance I need to * rewind * resize the mmap to accomodate the text * move some part of the text to a new location on the mmap so the new text does not overwrite the old * write the replacement text When I try to use resize it gives me the following error; SystemError: mmap: resizing not available--no mremap() Hi David, Based on experience with my posix_ipc module, ISTR that (a) resizing of the mmapp-ed segment happens via a call to ftruncate() (or os.truncate() in Python land) and (b) OS X only supports one call to this per segment; subsequent calls fail. Keep in mind that I was dealing with shared memory. A regular mmapped file might resize just fine so point (b) might be irrelevant for you. HTH Philip -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: mmap resizing macosx unix
Hi Phillip. I appreciate your reply. I think perhaps I will need to create a new mmap as a work around and write to locations of the second mmap based on my regex searches in the first. I should have said I am using 2.5.4 as well to be clear. I am wondering if I should recommend change to documentation for mmap if this is an issue for OSX. I have vmware installed so might try on CentOS to see what happens there. Many thanks Regards, David On 5-Apr-09, at 11:34 AM, Philip Semanchuk wrote: On Apr 5, 2009, at 10:28 AM, David Pratt wrote: Hi. I have been experimenting with mmap recently. I determined how to read and write properly from it and so search and replace on large files. The problem I am having is with replaces that are larger than the mmap. In this instance I need to * rewind * resize the mmap to accomodate the text * move some part of the text to a new location on the mmap so the new text does not overwrite the old * write the replacement text When I try to use resize it gives me the following error; SystemError: mmap: resizing not available--no mremap() Hi David, Based on experience with my posix_ipc module, ISTR that (a) resizing of the mmapp-ed segment happens via a call to ftruncate() (or os.truncate() in Python land) and (b) OS X only supports one call to this per segment; subsequent calls fail. Keep in mind that I was dealing with shared memory. A regular mmapped file might resize just fine so point (b) might be irrelevant for you. HTH Philip -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
mmap resizing macosx unix
Hi. I have been experimenting with mmap recently. I determined how to read and write properly from it and so search and replace on large files. The problem I am having is with replaces that are larger than the mmap. In this instance I need to * rewind * resize the mmap to accomodate the text * move some part of the text to a new location on the mmap so the new text does not overwrite the old * write the replacement text When I try to use resize it gives me the following error; SystemError: mmap: resizing not available--no mremap() I pass size to mmap to begin with based on my filesize size = os.path.getsize(filename) m = mmap.mmap(f.fileno(), size) I am attempting to give it new size by doing: m.resize(size + size_diff) size_diff is difference between original and what will be new file size after the insertion. Can someone provide some hint on resizing or its this a python bug? Many thanks Regards David -- http://mail.python.org/mailman/listinfo/python-list
mmap regex search replace
Hi. I have a circumstance where I have to search and replace a block
of text in a very large file. I have written some psuedo code to
locate the text and print the span of text to be removed and replaced
by new block. Can someone advise what to do to remove the text span
and insert with the new text. the match.span() provides a tuple of the
starting and ending position. Many thanks.
Regards,
David
import mmap
import re
text_to_insert = 'the block to insert'
pattern = re.compile(my regex here)
f = open('my_large_file.dat', 'r+')
try:
m = mmap.mmap(f.fileno(), 0)
try:
match = chart_re.search(m)
print match.span()
finally:
m.close()
finally:
f.close()
--
http://mail.python.org/mailman/listinfo/python-list
Re: Ordering attributes for dynamically generated class
Hi Aaron, this worked out fine. Using an ordered dict to subclass dict. Many thanks. David On Jan 18, 2009, at 11:57 AM, Aaron Brady wrote: On Jan 18, 9:52 am, David Pratt wrote: Hi list. I use 'type' to generate classes but have a need to order the attributes for the generated class. Of course a dict is not going to maintain ordering. Is there any way to dynamically generate a class with attributes in specific order? my_new_class = type( 'MyNewClass', tuple_of_bases, dict_of_attributes) Many thanks, David Just a thought, you can subclass 'dict' and assign an instance of it to the __dict__ member of your new instance. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Ordering attributes for dynamically generated class
Hi Aaron. Yeah, definitely sounds like a possibility. I was able to locate an ordered dict implementation that subclasses dict. This might work fine. Might be able to pass into type method directly since I think that dict passed into type is setting __dict__ I believe. Let you know if that works out. Many thanks. Regards, David On Jan 18, 2009, at 11:57 AM, Aaron Brady wrote: On Jan 18, 9:52 am, David Pratt wrote: Hi list. I use 'type' to generate classes but have a need to order the attributes for the generated class. Of course a dict is not going to maintain ordering. Is there any way to dynamically generate a class with attributes in specific order? my_new_class = type( 'MyNewClass', tuple_of_bases, dict_of_attributes) Many thanks, David Just a thought, you can subclass 'dict' and assign an instance of it to the __dict__ member of your new instance. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Ordering attributes for dynamically generated class
Hi list. I use 'type' to generate classes but have a need to order the attributes for the generated class. Of course a dict is not going to maintain ordering. Is there any way to dynamically generate a class with attributes in specific order? my_new_class = type( 'MyNewClass', tuple_of_bases, dict_of_attributes) Many thanks, David -- http://mail.python.org/mailman/listinfo/python-list
Re: 'new' module deprecation in python2.6
Hey Christian. Many thanks for explanation. Clears that up :-)
Regards,
David
On Nov 29, 2008, at 1:52 PM, Christian Heimes wrote:
David Pratt wrote:
Hi Mike. Many thanks for your reply and thank you for reference.
I have code that looks like the following so initially looking at
what will need to be done as it doesn't appear new will survive.
So first need to find way of translating this sort of thing using
types. I see there is a ClassType for types in Python 2.6 but it
does not exist in Python 3 so wonder where this is going? Is this
an oversight or maybe just not ready yet.
ClassType is the type of old style classes. Since old style classes
were removed the ClassType is also gone.
You can create new style classes with type:
>>> Name = type("Name", (object,), dict(spam="egg"))
>>> Name
>>> Name.spam
'egg'
Christian
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: 'new' module deprecation in python2.6
Rob. Sweet! Many thanks.
Regards,
David
On Nov 29, 2008, at 1:46 PM, Rob Williscroft wrote:
David Pratt wrote in news:mailman.4664.1227980181.3487.python-
[EMAIL PROTECTED] in comp.lang.python:
import new
class FirstBase(object):
foo = 'bar'
biz = 'baz'
class SecondBase(object):
bla = 'blu'
buz = 'brr'
attr = {
'fiz': 'An attribute', 'fuz': 'Another one'}
Test = new.classobj(
'Test', (FirstBase, SecondBase), attr)
Test = type(
'Test', (FirstBase, SecondBase), attr)
class MyNewClass(Test):
pass
a = MyNewClass()
print a.foo, a.buz, a.fiz, type(a)
print( ( a.foo, a.buz, a.fiz, type(a) ) )
py 3.0:
('bar', 'brr', 'An attribute', )
py 2.4
('bar', 'brr', 'An attribute', )
Rob.
--
http://www.victim-prime.dsl.pipex.com/
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: 'new' module deprecation in python2.6
Yeah, can just use types.ClassType instead of new.classobj, but still wonder what happens when we get to python 3. Regards, David On Nov 29, 2008, at 1:04 PM, Michael Crute wrote: On Sat, Nov 29, 2008 at 11:52 AM, David Pratt <[EMAIL PROTECTED]> wrote: Can someone tell me why 'new' has been deprecated in python 2.6 and provide direction for code that uses new for the future. I find new is invaluable for some forms of automation. I don't see a replacement for python 3 either. Many thanks. You might want to take a look at PEP 3108[1] which says: new * Just a rebinding of names from the 'types' module. * Can also call type built-in to get most types easily. * Docstring states the module is no longer useful as of revision 27241 (2002-06-15). [1] http://www.python.org/dev/peps/pep-3108/ -mike -- Michael E. Crute http://mike.crute.org God put me on this earth to accomplish a certain number of things. Right now I am so far behind that I will never die. --Bill Watterson -- http://mail.python.org/mailman/listinfo/python-list
Re: 'new' module deprecation in python2.6
Hi Mike. Many thanks for your reply and thank you for reference. I
have code that looks like the following so initially looking at what
will need to be done as it doesn't appear new will survive. So first
need to find way of translating this sort of thing using types. I see
there is a ClassType for types in Python 2.6 but it does not exist in
Python 3 so wonder where this is going? Is this an oversight or maybe
just not ready yet.
import new
class FirstBase(object):
foo = 'bar'
biz = 'baz'
class SecondBase(object):
bla = 'blu'
buz = 'brr'
attr = {
'fiz': 'An attribute', 'fuz': 'Another one'}
Test = new.classobj(
'Test', (FirstBase, SecondBase), attr)
class MyNewClass(Test):
pass
a = MyNewClass()
print a.foo, a.buz, a.fiz, type(a)
On Nov 29, 2008, at 1:04 PM, Michael Crute wrote:
On Sat, Nov 29, 2008 at 11:52 AM, David Pratt
<[EMAIL PROTECTED]> wrote:
Can someone tell me why 'new' has been deprecated in python 2.6
and provide
direction for code that uses new for the future.
I find new is invaluable for some forms of automation. I don't see a
replacement for python 3 either. Many thanks.
You might want to take a look at PEP 3108[1] which says:
new
* Just a rebinding of names from the 'types' module.
* Can also call type built-in to get most types easily.
* Docstring states the module is no longer useful as of revision
27241 (2002-06-15).
[1] http://www.python.org/dev/peps/pep-3108/
-mike
--
Michael E. Crute
http://mike.crute.org
God put me on this earth to accomplish a certain number of things.
Right now I am so far behind that I will never die. --Bill Watterson
--
http://mail.python.org/mailman/listinfo/python-list
'new' module deprecation in python2.6
Can someone tell me why 'new' has been deprecated in python 2.6 and provide direction for code that uses new for the future. I find new is invaluable for some forms of automation. I don't see a replacement for python 3 either. Many thanks. -- http://mail.python.org/mailman/listinfo/python-list
Conditionally subclassing based on Import
Hi, just want to conditionally base a class on another if it can be imported, otherwise base it on object. Does the following look ok for this? try: import foo.bar except ImportError: MyBase = foo.bar.Baz else: MyBase = object class Something(MyBase): -- http://mail.python.org/mailman/listinfo/python-list
Selective importing and package dependencies
Hi. I am in the midst of preparing a package to convert between various schemas including orms. The issue is I don't want django, slqalchemy, storm, rdflib etc. as hard dependencies of the package. Each module is a schema to schema conversion. As an example, I have imports for sqlalchemy with classes and methods that use them. from sqlalchemy.util import OrderedDict from sqlalchemy import types as rdbtype import sqlalchemy as sa I have my own ideas about how I might do this but looking for recommendations from others how they would handle this so the result would be: 1. no hard dependencies on any of these other packages 2. load the module without failure. 3. import the dependent package if available to perform the conversion Many thanks David -- http://mail.python.org/mailman/listinfo/python-list
Re: Help replacing os.system call with subprocess call
Hi Matt. My apologies, I was away a good part of the day and evening today. Here are the numbers for the different methods: original os.system call: 29.685759 s buffered stdout call:213.370982 s (with 1mb buffer) direct to stdout call:33.378756 s The second method worked great and is essentially equivalent in execution time to original call :-). This has got me smiling. Many thanks Matt for your help, particularly working through the second example that provided equivalent speed. Regards, David David Pratt wrote: > Hi Matt. Many thanks. Sorry I had not seen your second post. I'll give > this a try and time the completion to compare the differences and post > back later today to show os.system, buffered imput and using a file > directly for stdout. > > Regards, > David > > Matt Nordhoff wrote: >> David Pratt wrote: >>> Hi David and Matt. I appreciate your help which has got me moving >>> forward again so many thanks for your reply. I have been using >>> subprocess.Popen a fair bit but this was the first time I had to use >>> subprocess to capture large file output. The trouble I was having was >>> with the process would just hang. Chunking was the solution. I guess I >>> assumed this would be taken care of in the internals. >>> >>> Overall, I wish subprocess had some better documentation since it is >>> definitely not a drop in replacement for os.system. In other >>> circumstances I am using subprocess.call() for simple calls which works >>> fine. >>> >>> The speed of this solution is slower than os.system. Would a queue of >>> some kind be needed to speed this up? Has anyone implemented something >>> like this? Many thanks. >>> >>> Regards, >>> David >> Did you see my second message? That should help performance. If not, I'm >> totally out of my depth and have no idea at all. Sorry. >> >> (How much slower? 10%? 200%?) -- http://mail.python.org/mailman/listinfo/python-list
Re: Help replacing os.system call with subprocess call
Hi Matt. Many thanks. Sorry I had not seen your second post. I'll give this a try and time the completion to compare the differences and post back later today to show os.system, buffered imput and using a file directly for stdout. Regards, David Matt Nordhoff wrote: > David Pratt wrote: >> Hi David and Matt. I appreciate your help which has got me moving >> forward again so many thanks for your reply. I have been using >> subprocess.Popen a fair bit but this was the first time I had to use >> subprocess to capture large file output. The trouble I was having was >> with the process would just hang. Chunking was the solution. I guess I >> assumed this would be taken care of in the internals. >> >> Overall, I wish subprocess had some better documentation since it is >> definitely not a drop in replacement for os.system. In other >> circumstances I am using subprocess.call() for simple calls which works >> fine. >> >> The speed of this solution is slower than os.system. Would a queue of >> some kind be needed to speed this up? Has anyone implemented something >> like this? Many thanks. >> >> Regards, >> David > > Did you see my second message? That should help performance. If not, I'm > totally out of my depth and have no idea at all. Sorry. > > (How much slower? 10%? 200%?) -- http://mail.python.org/mailman/listinfo/python-list
Re: Help replacing os.system call with subprocess call
Hi David and Matt. I appreciate your help which has got me moving
forward again so many thanks for your reply. I have been using
subprocess.Popen a fair bit but this was the first time I had to use
subprocess to capture large file output. The trouble I was having was
with the process would just hang. Chunking was the solution. I guess I
assumed this would be taken care of in the internals.
Overall, I wish subprocess had some better documentation since it is
definitely not a drop in replacement for os.system. In other
circumstances I am using subprocess.call() for simple calls which works
fine.
The speed of this solution is slower than os.system. Would a queue of
some kind be needed to speed this up? Has anyone implemented something
like this? Many thanks.
Regards,
David
Matt Nordhoff wrote:
> David Pratt wrote:
>> Hi. I am trying to replace a system call with a subprocess call. I have
>> tried subprocess.Popen and subprocess.call with but have not been
>> successful. The command line would be:
>>
>> svnadmin dump /my/repository > svndump.db
>>
>> This is what I am using currently:
>>
>> os.system('svnadmin dump %s > %s' % (svn_dir,
>> os.path.join(backup_dir, 'svndump.db')))
>>
>> Many thanks.
>
> Try this:
>
> import os.path
> import subprocess
>
> p = subprocess.Popen(
> ['svnadmin', 'dump', svndir],
> stdout=subprocess.PIPE,
> )
>
> fh = open(os.path.join(backup_dir, 'svndump.db'), 'wb')
>
> while True:
> chunk = p.stdout.read(2**20) # 1 MB
> if not chunk:
> break
> fh.write(chunk)
>
> fh.close()
>
> It reads svnadmin's stdout in 1 MB chunks, in case it's large enough
> that reading the whole thing into RAM at once would be a bad idea.
>
> No error handling. For one, you might want to add a try...finally to
> ensure that fh will get closed. (Or if you have Python 2.5, use a with
> statement! :-) ) Also, Popen will raise an OSError if svnadmin can't be
> found or something. And this isn't even considering svnadmin erroring out...
>
> svnadmin's stderr will go to your stderr.
>
> I didn't test it, but I'm pretty sure it will work. (I spotted a syntax
> error while writing that though.) I don't have much experience with
> Popen's stdio objects, so it's possible you'd need to do something like
> call p.wait() to wait for it to exit before being able to read its stdout.
>
> It could be slower than the os.system version, since now Python is doing
> all of the I/O, instead of your shell, but I doubt that'll be a big problem.
>
> (Also, insert suggestion about using a good VCS. ;-) )
--
http://mail.python.org/mailman/listinfo/python-list
Help replacing os.system call with subprocess call
Hi. I am trying to replace a system call with a subprocess call. I have
tried subprocess.Popen and subprocess.call with but have not been
successful. The command line would be:
svnadmin dump /my/repository > svndump.db
This is what I am using currently:
os.system('svnadmin dump %s > %s' % (svn_dir,
os.path.join(backup_dir, 'svndump.db')))
Many thanks.
--
http://mail.python.org/mailman/listinfo/python-list
cProfile for python 2.4
I'd like to compile cProfile for python 2.4. Where can I get it to do this? I realize it is part of python 2.5. Many thanks. -- http://mail.python.org/mailman/listinfo/python-list
Python universal build, OSX 10.3.9 and undefined symbols when linking
Hi. I am on a PPC and have been getting the same undefined symbols traceback when linking applications after compiling. A few weeks back I posted on a problem to pythonmac-sig@python.org after attempting to build mod_python, today its pylucene - same problem. I initially shrugged off the problem with mod_python as likely an issue with the mac port. Crazy thing is that I built the same version of pylucene successfully just before upgrading to universal build of 2.4.4 python for mac without problems. So I recompiled the same version of pylucene I had previously built with a PPC only build of python to evaluate whether my thoughts were right. I had a hunch universal python was causing my trouble with mod_python and a newer build of pylucene and it appears I was right. I am currently using Mac OSX 10.3.9 on a PPC with universal build of 2.4.4. I should say that other than linking problem I am experiencing, the python functions as it should. From mod_python build: ld: Undefined symbols: _fstatvfs referenced from Python expected to be defined in libSystem _lchown referenced from Python expected to be defined in libSystem _statvfs referenced from Python expected to be defined in libSystem apxs:Error: Command failed with rc=65536 . make[1]: *** [mod_python.so] Error 1 make: *** [do_dso] Error 2 From pylucene build: ld: Undefined symbols: _fstatvfs referenced from Python expected to be defined in libSystem _lchown referenced from Python expected to be defined in libSystem _statvfs referenced from Python expected to be defined in libSystem gmake: *** [release/_PyLucene.so] Error 1 I have googled to see there have been others with this issue however they have tended to communicate about the problem on software lists or forums specific to the software they were attempting to build. I did not see this resolved in any case that I have read. I am hoping someone may be able to advise a possible solution. I am planning on upgrading to OSX 10.4 within a week since mac is not making the JDK 1.5.0 available for OSX 10.3 users - so pretty much being forced into upgrading in any case. If you had the same linking issue with the universal build of python it would be good to hear from you - better yet would be some way of solving this or at least understanding what may be going on. I have posted to pythonmac-sig@python.org in the interim but there is very little traffic on this list. In the meantime I thought there may other OSX 10.3.9 users out there who may have run into the same fate. Many thanks. Regards, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Collecting list of module dependencies
Hi skip. Many thanks for this. Exactly what I need :-) Regards, David [EMAIL PROTECTED] wrote: > David> Hi. Is anyone aware of any code to create a list of dependent > David> modules for a python module. > > modulefinder: http://docs.python.org/lib/module-modulefinder.html > > Added to Python in 2.3. > > Skip > -- http://mail.python.org/mailman/listinfo/python-list
Collecting list of module dependencies
Hi. Is anyone aware of any code to create a list of dependent modules for a python module. Ideally am looking for something with a method to create a unique list of imported modules that excludes imports from the module being analyzed. Many thanks. Regards, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Initializing an attribute that needs the object
Hi John. Thank you for the tips and the link. This is helpful. Many thanks. Regards David > A new-style class is one which inherits ultimately from the type that is > called "object". > > class NewStyleClass(object): > pass > > class OldStyleClass(): > pass > > Docs are a bit u ... > See this: http://www.python.org/doc/newstyle/ > > HTH, > John > -- http://mail.python.org/mailman/listinfo/python-list
Re: Initializing an attribute that needs the object
Hi Bruno. This is certainly what I was missing. Thank you. I am afraid I am behind the times with use of object. Will I only use object when I am not subclassing? Where will I find a document that provides info on the use of object in new style classes? Many thanks. Regards, David Bruno Desthuilliers wrote: > David Pratt a écrit : >> Hi. I want to have different handlers to do perform logic. The problem >> is the Handler requires an instance of the factory since it will use its >> own methods in conjunction with methods of the factory. >> >> Once I have got a Factory instance I can give it a new handler (see >> below). It would be more flexible if I could provide a handle in >> constructor - but how to do this when it requires the object itself. > > Hint : Python classes are objects too. > >> class Factory: > > Do yourself a favour : use new-style classes. > > class Factory(object): >def __init__(self, handler_class): > self.handler = handler_class(self) > > class SomeHandler(object): >def __init__(self, factory): > self.factory = factory > > f = Factory(SomeHandler) -- http://mail.python.org/mailman/listinfo/python-list
Re: Initializing an attribute that needs the object
My apologies. What I meant to write was this class Factory def __init__(self, handler): David Pratt wrote: > Hi Marco. Thanks for your reply. I am providing the handler with the > factory instance as I have shown. This is how my code currently works. > What I am trying to figure out is how to possibly provide the handler in > the constructor when it needs the factory instance. This would give me > some better flexibility. > > ie. > > class Factory > > def __init__(self, factory): > > At this point I don't have self. Would super help me? > > Regards, > David > > > Marco Giusti wrote: >> On Fri, Jun 02, 2006 at 06:15:28PM -0300, David Pratt wrote: >>> Hi. I want to have different handlers to do perform logic. The problem >>> is the Handler requires an instance of the factory since it will use its >>> own methods in conjunction with methods of the factory. >>> >>> Once I have got a Factory instance I can give it a new handler (see >>> below). It would be more flexible if I could provide a handle in >>> constructor - but how to do this when it requires the object itself. >>> Would I use a super for this sort of thing? Many thanks >> when __init__ is called the object already exists. >> >>> class Factory: >>> >>> def __init__(self): >>> self.some_handler = Handler(self) >>> >>> f = Factory() >>> f.some_handler = AnotherHandler(f) >> try this, should works: >> >> class Factory: >> >> def __init__(self): >> self._some_handler = AnotherHandler(self) >> >> maybe a class hierarchy is good for you >> >> ciao >> m. >> -- http://mail.python.org/mailman/listinfo/python-list
Re: Initializing an attribute that needs the object
Hi Marco. Thanks for your reply. I am providing the handler with the factory instance as I have shown. This is how my code currently works. What I am trying to figure out is how to possibly provide the handler in the constructor when it needs the factory instance. This would give me some better flexibility. ie. class Factory def __init__(self, factory): At this point I don't have self. Would super help me? Regards, David Marco Giusti wrote: > On Fri, Jun 02, 2006 at 06:15:28PM -0300, David Pratt wrote: >> Hi. I want to have different handlers to do perform logic. The problem >> is the Handler requires an instance of the factory since it will use its >> own methods in conjunction with methods of the factory. >> >> Once I have got a Factory instance I can give it a new handler (see >> below). It would be more flexible if I could provide a handle in >> constructor - but how to do this when it requires the object itself. >> Would I use a super for this sort of thing? Many thanks > > when __init__ is called the object already exists. > >> class Factory: >> >> def __init__(self): >> self.some_handler = Handler(self) >> >> f = Factory() >> f.some_handler = AnotherHandler(f) > > try this, should works: > > class Factory: > > def __init__(self): > self._some_handler = AnotherHandler(self) > > maybe a class hierarchy is good for you > > ciao > m. > -- http://mail.python.org/mailman/listinfo/python-list
Initializing an attribute that needs the object
Hi. I want to have different handlers to do perform logic. The problem is the Handler requires an instance of the factory since it will use its own methods in conjunction with methods of the factory. Once I have got a Factory instance I can give it a new handler (see below). It would be more flexible if I could provide a handle in constructor - but how to do this when it requires the object itself. Would I use a super for this sort of thing? Many thanks Regards, David class Factory: def __init__(self): self.some_handler = Handler(self) f = Factory() f.some_handler = AnotherHandler(f) -- http://mail.python.org/mailman/listinfo/python-list
Re: Package that imports with name of dependent package
Hi Peter. Thank you for this warning. I'll document this in the code. I plan on importing only from dependentpackage for portability. Also, much in the framework relies upon it. This approach is primarily for maintenance purposes and would also allow me to package the modified dependentpackage (with just the __init__.py ) along with mypackage if I plan to distribute it. It allows it to be a drop in replacement. I am hoping with packaging utilities, I can easily remove the dependentpackage if it is encountered in site-packages to replace it with my own. Regards, David Peter Otten wrote: > Peter Otten wrote: > >>> I'd appreciate hearing of what I can do in an __init__ file or what >>> other strategy could make this work. Many thanks. >> I think fixing the imports is the better long-term approach. But putting >> >> from pkgutil import extend_path >> import mypackage >> __path__ = extend_path(mypackage.__path__, __name__) >> >> into dependentpackage/__init__.py might work. > > One big caveat: If you are mixing both > > import mypackage.somemodule > > and > > import dependentpackage.somemodule > > in the same application, mypackage.somemodule and > dependentpackage.somemodule are *not* the same module instance. This may > have surprising effects when global variables in somemodule.py are > updated... > > Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: Package that imports with name of dependent package
Hi Peter. I'd like to fix the imports, but this would impact the portability of portions of the code that currently work with the existing package from the framework. This solution does the trick and allows me to create the package I want using a good amount of new material. I don't have to worry about adding to the original package each time a release comes out. I'll only have to monitor code changes for an impact on my classes, subclasses, etc. Still a pain, but a smaller one :-) Many thanks. Regards David Peter Otten wrote: > from pkgutil import extend_path > import mypackage > __path__ = extend_path(mypackage.__path__, __name__) > > into dependentpackage/__init__.py might work. > > Peter > > > -- http://mail.python.org/mailman/listinfo/python-list
Package that imports with name of dependent package
Hi. I have code that currently depends on a particular package of a framework. I have decided I want to create my own package because I have made many changes and it is getting too difficult to maintain each time I retrieve an updated version of the framework from svn. The problem is, that there are all sorts of imports to the dependent package throughout my code and I just want to replace this module with something that will provide a reference to my own package without changing the original imports. So it just needs to point to the new package so that the original imports in my code will continue to work. For example, here is a package structure. dependentpackage | + __init__.py + somemodule.py + somefolder | + __init__.py + somesubmodule.py + someotherfolder etc I simply want the dependentpackage to point to the new package leaving no more than an init file or whatever would have to be minimally required to make this work dependentpackage | + __init__.py mypackage | + __init__.py + somemodule.py + somefolder | + __init__.py + somesubmodule.py + someotherfolder etc I my code I still need to have this work: from dependentpackage.somemodule import something - but I want the package to be getting the code from the new module. I'd appreciate hearing of what I can do in an __init__ file or what other strategy could make this work. Many thanks. Regards, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Removing .DS_Store files from mac folders
Hi Ben. I hadn't realize that walk was just giving the file name so the
join did the job just great. Many thanks for helping me out with this.
Regards,
David
Ben Cartwright wrote:
> David Pratt wrote:
>
>>OSError: [Errno 2] No such file or directory: '.DS_Store'
>
>
>
> Ah. You didn't mention a traceback earlier, so I assumed the code was
> executing but you didn't see the file being removed.
>
>
>
>>>>for f in file_names:
>>>>current_file = os.path.basename(f)
>>>>print 'Current File: %s' % current_file
>>>>
>>>># Clean mac .DS_Store
>>>>if current_file == '.DS_Store':
>>>>print 'a DS_Store item encountered'
>>>>os.remove(f)
>
>
>
> How are you creating file_names? More importantly, does it contain a
> path (either absolute or relative to the current working directory)?
> If not, you need an os.path.join, e.g.:
>
> import os
> for root_path, dir_names, file_names in os.walk('.'):
> # file_names as generated by os.walk contains file
> # names only (no path)
> for f in file_names:
> if f == '.DS_Store':
> full_path = os.path.join(root_path, f)
> os.remove(full_path)
>
> --Ben
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: WxPython question re: embedded activex internet explorer and excel
Hi jojoba. I work with wxPython too but cannot answer this question.
There is an active and friendly community of wxPython users on
[EMAIL PROTECTED]
Robin Dunn, the lead person behind wxPython and others provide excellent
advice and support on this list.
Regards,
David
jojoba wrote:
> Hi
> I currently am using an activex wrapper in wxpython to embed Internet
> Explorer in a wxPanel:
>
> IEmodule=win32com.client.gencache.EnsureModule('{EAB22AC0-30C1-11CF-A7EB-C05BAE0B}',
> 0,1,1)
> InternetExplorerActiveXClass = MakeActiveXClass(IEmodule.WebBrowser,
> eventObj = self)
> self.WebBrowser = InternetExplorerActiveXClass(self,
> -1)
>
> (where self is a wx.Panel)
>
> That was ok.
> However, I am creating multiple of these activex webbrowsers and
> putting them in a wx.Notebook, one per page (where each page is a
> wx.Panel).
> On each page, i load a different .xls excel file:
>
> self.WebBrowser.Navigate2(excelFilePath)
>
> Great. All the files load fine.
> Ok, here's the weird part.
> When i choose among the different tabs for the wx.Notebook, I can see
> all the excel files fine. I can go into each and edit them too. But for
> some reason, i can't get focus away from the last-loaded excel file.
> That is, I can really only see the cursor flashing in a worksheet excel
> if and only if i am in the last-loaded excel file.
>
> I tried self.WebBrowser.Document.Application.Workbooks[0].Activate() to
> activate another workbook (representing a different .xls file) but i
> get a win32com error saying the "Workbook Activate method failed"
>
> Is there any way for me to get out of this? Does anyone have any idea
> how to bring other workbooks into focus using this activex scheme? Or
> am i approaching this from the completely wrong angle?
>
> thanks to anyone who can get past this madness,
> jojoba
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Removing .DS_Store files from mac folders
My apologies Ben. I should have included the traceback in my message.
The last line of the traceback I get from python when it gets to
os.remove is
OSError: [Errno 2] No such file or directory: '.DS_Store'
The traceback occurs immediately after printing:
Current File: .DS_Store
a DS_Store item encountered
so it seems that python can see the file to recognize it as the
current_file while walking the heirarchy but it cannot see the hidden
file to remove it with os.remove().
On mac recreating .DS_Store files, my original solution works to remove
them. In my code I cd to the folder and do the following which will find
and eliminate them all to any depth.
system('find . -name .DS_Store -exec rm {} \;')
I have been using this method for some time when I put together code in
a tarball since with mac the .DS_Store pollutes your files. Since I am
needing to do other things with the files, I though I would do remove
while walking the folders instead. I can always go back to this but I am
hoping someone can advise a way of deleting a hidden file. I am admin
on the mac so permissions is not an issue.
Regards,
David
Ben Cartwright wrote:
> David Pratt wrote:
>
>>Hi Ben. Sorry about the cut and paste job into my email. It is part of a
>>larger script. It is actually all tabbed. This will give you a better idea:
>>
>> for f in file_names:
>> current_file = os.path.basename(f)
>> print 'Current File: %s' % current_file
>>
>> # Clean mac .DS_Store
>> if current_file == '.DS_Store':
>> print 'a DS_Store item encountered'
>> os.remove(f)
>
>
>
> I'm no Mac expert, but could it be that OSX is recreating .DS_Store?
> Try putting this above your os.remove call:
>
> import os.stat
> print 'Last modified:', os.stat(f)[ST_MTIME]
>
> Then run your script a few times and see if the modified times are
> different.
>
> You might also try verifying that you get an exception when attempting
> to open the file right after removing it.
>
> --Ben
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Removing .DS_Store files from mac folders
Hi Ben. Sorry about the cut and paste job into my email. It is part of a larger script. It is actually all tabbed. This will give you a better idea: for f in file_names: current_file = os.path.basename(f) print 'Current File: %s' % current_file # Clean mac .DS_Store if current_file == '.DS_Store': print 'a DS_Store item encountered' os.remove(f) Ben Cartwright wrote: > David Pratt wrote: > >># Clean mac .DS_Store >> if current_file == '.DS_Store': >> print 'a DS_Store item encountered' >> os.remove(f) > > ... > >>I can't figure why >>remove is not removing. > > > > It looks like your indentation is off. From what you posted, the > "print" line is prepended with 9 spaces, while the "os.remove" line is > prepended with a single tab. Don't mix tabs and spaces. > > Also, shouldn't that be "os.remove(current_file)"? > > --Ben > -- http://mail.python.org/mailman/listinfo/python-list
Removing .DS_Store files from mac folders
Hi. I'm trying to clean files for packaging on mac using os.path.walk
and want to clean the .DS_Store files that are hidden from view but
could end up in code that I produce.
# Clean mac .DS_Store
if current_file == '.DS_Store':
print 'a DS_Store item encountered'
os.remove(f)
My code walks the folders properly and print prints the statement when a
.DS_Store file is encountered but the os.remove does not get rid of the
file. I have previously cd'd into the folder and can remove all by doing
which does work
system('find . -name .DS_Store -exec rm {} \;')
but I am walking the folders to do a few more things. I can't figure why
remove is not removing.
Regards,
David
--
http://mail.python.org/mailman/listinfo/python-list
Re: Recursive tree list from dictionary
Hi Bengt! I have been banging my head on this one all day! This is
brilliant (and recursive through each level which is exactly what I was
trying to work out)! Only part I needed to modify is
else: return title to else: return [title]
I tell you, you've made my day! I was getting a bit discouraged trying a
number of things but just could not get it to work for myself. Thought
I'd check the list once more to find that you had worked it out. Many
thanks Bengt, for this really super and efficient solution!
Regards,
David
> If you literally want "# Project" comments in the output, it will have
> to be a source code (string) output, e.g., the following generates and
> prints both:
>
> < david_pratt.py >--
> source_list = [{'title': 'Project', 'parent':'root'},
> {'title': 'Geometry', 'parent':'Geometries'},
> {'title': 'Soil', 'parent':'root'},
> {'title': 'Geometries', 'parent':'Project'},
> {'title': 'Verticals', 'parent':'Project'},
> {'title': 'Points', 'parent':'Geometry'},
> {'title': 'Layers', 'parent':'Geometry'},
> {'title': 'Water', 'parent':'Geometry'},
> {'title': 'Soiltypes', 'parent':'Soil'}
> ]
> children = {}
> for d in source_list:
> children.setdefault(d['parent'], []).append(d['title'])
>
> def get_progeny(title, d=children):
> if title in d: return [title] + [get_progeny(child) for child in d[title]]
> else: return title
>
> source = ['result_list = [']
> result_list = []
> for child in children['root']:
> source.append('# %s'%child)
> source.append('%r,'% get_progeny(child))
> result_list.append(get_progeny(child))
> source.append(']')
> result_list_source = '\n'.join(source)
> print result_list_source
> print result_list
>
>
> [18:20] C:\pywk\clp>py24 david_pratt.py
> result_list = [
> # Project
> ['Project', ['Geometries', ['Geometry', 'Points', 'Layers', 'Water']],
> 'Verticals'],
> # Soil
> ['Soil', 'Soiltypes'],
> ]
> [['Project', ['Geometries', ['Geometry', 'Points', 'Layers', 'Water']],
> 'Verticals'], ['Soil', '
> Soiltypes']]
>
> Regards,
> Bengt Richter
--
http://mail.python.org/mailman/listinfo/python-list
Re: Recursive tree list from dictionary
Hi Allan, Max, and bearophile
Many thanks for your replies to this. The number of levels can be deeper
than two for creating child, sibling relationships. This can lead to
futher nesting as shown in my sample result list (the result I am
attempting to acheive) which is reason that I believe this has to be
recursive to work. As an illustration, I have altered the source_dict to
show this by making Geometries the parent of Geometry. I should have
gone one more level to illustrate this properly - my apologies for not
doing this better.
source_list = [{'title': 'Project', 'parent':'root'},
{'title': 'Geometry', 'parent':'Geometries'},
{'title': 'Soil', 'parent':'root'},
{'title': 'Geometries', 'parent':'Project'},
{'title': 'Verticals', 'parent':'Project'},
{'title': 'Points', 'parent':'Geometry'},
{'title': 'Layers', 'parent':'Geometry'},
{'title': 'Water', 'parent':'Geometry'},
{'title': 'Soiltypes', 'parent':'Soil'}
]
Project and Soil in root,
Geometries (Geometries containing Geometry with its siblings) and
Verticals in Project,
Soiltypes in Soil
In this instance, I am looking for a result list that looks like this:
result_list = [
# Project
['Project', ['Geometries',['Geometry', ['Points',
'Layers','Water']],'Verticals']],
# Soil
['Soil', ['Soiltypes']]
]
I have not yet installed http://sourceforge.net/projects/pynetwork/ but
will also give this a try. What I need is something like elementree
minus the xml since I am working with list of dicts obtained from more
than one type of source.
Regards,
David
imd=dict()
for d in source_list:
>
> par=d['parent']
> v=d['title']
> try:
> imd[par].append(v)
> except:
> imd[par]=[v]
>
>
>
imd
>
> {'Project': ['Geometries', 'Verticals'], 'Geometry': ['Points',
> 'Layers', 'Water'], 'root': ['Project', 'Geometry', 'Soil'], 'Soil':
> ['Soiltypes']}
>
>
> max
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pythonic wrappers for SQL?
Hi Kenneth. Try SQLAlchemy. Regards, David Kenneth McDonald wrote: > I need to do some data manipulation, and SQLite is a nice little > product for it, except of course that I'd need to write SQL. Are > there any good libraries out there that let one write (basic) queries > in a Pythonic syntax, rather than directly in SQL? > > Thanks, > Ken -- http://mail.python.org/mailman/listinfo/python-list
Recursive tree list from dictionary
Hi. I am wanting to create a tree list result structure from a
dictionary to categorize results. The dictionary contains elements that
identify its parent. The levels of categorization is not fixed, so there
is a need for the code to be recursive to drill down to the lowest
level. I have contrived a small source and result list to illustrate:
source_list =[
{'title': 'Project', 'parent':'root'},
{'title': 'Geometry', 'parent':'root'},
{'title': 'Soil', 'parent':'root'},
{'title': 'Geometries', 'parent':'Project'},
{'title': 'Verticals', 'parent':'Project'},
{'title': 'Points', 'parent':'Geometry'},
{'title': 'Layers', 'parent':'Geometry'},
{'title': 'Water', 'parent':'Geometry'},
{'title': 'Soiltypes', 'parent':'Soil'}]
What I want to do is a tree list like this:
tree_result_list = [
# Project
['Project', ['Geometries','Verticals']],
# Geometry
['Geometry', ['Points', 'Layers', 'Water']],
# Soil
['Soil', ['Soiltypes']]
]
I have not been successful and am hoping someone can
advise a solution to accomplish this. Many thanks.
Regards
David
--
http://mail.python.org/mailman/listinfo/python-list
Re: Unicode Question
Hi Max. Many thanks for helping to realize where I was missing the point
and making this clearer.
Regards,
David
Max Erickson wrote:
> The encoding argument to unicode() is used to specify the encoding of the
> string that you want to translate into unicode. The interpreter stores
> unicode as unicode, it isn't encoded...
>
>
unicode('\xbe','cp1252')
>
> u'\xbe'
>
unicode('\xbe','cp1252').encode('utf-8')
>
> '\xc2\xbe'
>
>
>
> max
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Unicode Question
Hi Erik. Thank you for your reply. The advice I has helped clarify this for me. Regards, David Erik Max Francis wrote: > David Pratt wrote: > > >>This is not working for me. Can someone explain why. Many thanks. > > > Because '\xbe' isn't UTF-8 for the character you want, '\xc2\xbe' is, as > you just showed yourself in the code snippet. > -- http://mail.python.org/mailman/listinfo/python-list
Re: Unicode Question
Hi Martin. Many thanks for your reply. What I am reall after, the
following accomplishes.
>
> If you are looking for "at the same time", perhaps this is also
> interesting:
>
> py> unicode('\xbe', 'windows-1252').encode('utf-8')
> '\xc2\xbe'
>
Your answer really helped quite a bit to clarify this for me. I am using
sqlite3 so it is very happy to have utf-8 encoded unicode.
The examples you provided were the additional help I needed. Thank you.
Regards,
David
--
http://mail.python.org/mailman/listinfo/python-list
Unicode Question
Hi. I am working through some tutorials on unicode and am hoping that
someone can help explain this for me. I am on mac platform using python
2.4.1 at the moment. I am experimenting with unicode with the 3/4 symbol.
I want to prepare strings for db storage that come from normal Windows
machine (cp1252) so my understanding is to unicode and encode to utf-8
and to store properly. Since data will be used on the web I would not
have to change my encoding when extracting from the database. This first
example I believe simulates this with the 3/4 symbol. Here I want to
store '\xc2\xbe' in my database.
>>> tq = u'\xbe'
>>> tq_utf = tq.encode('utf8')
>>> tq, tq_utf
(u'\xbe', '\xc2\xbe')
To unicode withat a valiable, my understanding is that I can unicode and
encode at the same time
>>> tq = '\xbe'
>>> tq_utf = unicode(tq, 'utf-8')
Traceback (most recent call last):
File "", line 1, in ?
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbe in position 0:
unexpected code byte
This is not working for me. Can someone explain why. Many thanks.
Regards,
David
--
http://mail.python.org/mailman/listinfo/python-list
Datetime, pytz and strange offset
Hi. I am creating a couple of small methods to help me manage time from
UTC as standard but I am getting strange results.
If I start with a datetime of 2005-12-12 14:30:00 in timezone
'America/Halifax'
and I want to turn this into a UTC representation.
from datetime import datetime
from pytz.reference import UTC
import pytz
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
tz = pytz.timezone('America/Halifax')
dt = datetime(year=2005, month=12, day=11, hour=14, minute=30, second=0,
microsecond=0, tzinfo=tz)
print dt.strftime(fmt)
This is giving me a strange offset of -0414 and LMT timezone.
ie. 2005-12-12 14:30:00 LMT-0414
I am not sure where this is coming from
To get the utc equivalent using dt = dt.astimezone(UTC) I get this
strange result which is obviously also not correct:
# '2005-12-12 18:44:00 UTC+'
If I do datetime.now instead with the same timezone I get the correct
timezone and correct offset.
ie.
dt = datetime.now(tz=tz)
# '2005-12-12 21:37:17 AST-0400'
If I get now time from this as utc it is correct
dt = dt.astimezone(UTC)
# '2005-12-12 01:37:17 UTC+'
If I do a different timezone of 'America/Vancouver'
I get something appropriate for the 2:30 case
tz = pytz.timezone('America/Vancouver')
dt = datetime(year=2005, month=12, day=12, hour=14, minute=30, second=0,
microsecond=0, tzinfo=tz)
print dt.strftime(fmt)
# 2005-12-12 20:30:00 PST-0800
I am not sure why I am getting -0414 and LMT for 'America/Halifax'
timezone? Is this a bug in the pytz package? Many thanks.
Regards,
David
--
http://mail.python.org/mailman/listinfo/python-list
Re: Efficient lookup in list of dictionaries
This is a lot less clumsy and can easily be used in a class which is
very nice!
Many thanks bruno and Rob E for this good advice.
Regards,
David
On Monday, December 5, 2005, at 05:31 AM, bruno at modulix wrote:
> David Pratt wrote:
> (snip)
>> Can someone advise a more efficient lookup when using lists of
>> dictionaries. Many thanks.
>>
>>
>> TEST_CONSTANTS = [
>> {'color':'red', 'shape':'octagon'},
>> {'color':'yellow', 'shape':'triangle'},
>> {'color':'green', 'shape':'circle'}]
>
> COLOR_INDEX = dict([(item['color'], item) for item in TEST_CONSTANT])
> SHAPE_INDEX = dict([item['shape'], item) for item in TEST_CONSTANT])
>
> def getShapeForColor(color):
> return COLOR_INDEX.get(color, {'shape':None})['shape']
>
> def getColorForShape(shape):
> return SHAPE_INDEX.get(color, {'color': None})['color']
>
> This of course assume that there are no duplicate colors nor shapes.
>
> --
> bruno desthuilliers
> python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')])
> for
> p in '[EMAIL PROTECTED]'.split('@')])"
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Efficient lookup in list of dictionaries
Hi. I like working with lists of dictionaries since order is preserved
in a list when I want order and the dictionaries make it explicit what
I have got inside them. I find this combination very useful for storing
constants especially. Generally I find myself either needing to
retrieve the values of constants in an iterative way (as in my
contrived example below). Perhaps even more frequent is given one value
is to look up the matching value in a dict (contained in the list) and
then obtain the value of another element in the same dictionary.
Most of these lists are generally small so not normally a big deal but
I have one a list that contains about 100 or more dictionaries with
several elements and am thinking there is likely a more efficient way
of doing the lookup. For example if I have say 5000 insertions to do
into a database but have to check my constants since they effect what
is inserted (and it is doing a lookup each time before an insertion is
made), it is likely adding much to my processing time. At this point I
do not want store the constants themselves in a database table. I
usually use the lists by just importing them when needed.
Can someone advise a more efficient lookup when using lists of
dictionaries. Many thanks.
Regards
David
TEST_CONSTANTS = [
{'color':'red', 'shape':'octagon'},
{'color':'yellow', 'shape':'triangle'},
{'color':'green', 'shape':'circle'}]
def getShapeForColor(color):
shape = None
for test_constant in TEST_CONSTANTS:
if test_constant['color'] == color:
shape = test_constant['shape']
return shape
--
http://mail.python.org/mailman/listinfo/python-list
Re: Whitespace test after string.split
Hi Fredrik. Good to know. Many thanks for your replies. Regards David On Saturday, November 26, 2005, at 12:27 PM, Fredrik Lundh wrote: > David Pratt wrote: > >> Also thanks for heads up for changes with method. I am >> still using 2.3 but will move to 2.4 as soon as this is formally >> approved for use in Zope. > > note that the string.split function has been "outdated" since Python > 1.6 (released in 2000), and despite what the documentation implies, > it won't go away before 3.0. > > there's no reason not to use it in 2.3, but if you have lots of old > code > using the old syntax, there's no reason to change that just because you > want to upgrade to a newer Python release... > > > > > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Whitespace test after string.split
Hi Fredrik and Peter. Many thanks for this helpful advice :-) These
are very nice solutions and much better than what I had been
contemplating. Also thanks for heads up for changes with method. I am
still using 2.3 but will move to 2.4 as soon as this is formally
approved for use in Zope.
Regards,
David
On Saturday, November 26, 2005, at 11:42 AM, Fredrik Lundh wrote:
> Peter Otten wrote:
>
> [t.strip() for t in s.split(",") if t and not t.isspace()]
>> ['alpha', 'gamma', 'delta']
>
> footnote: this solution is faster than my filter version.
>
>
>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Whitespace test after string.split
Hi. I am splitting a string on a non whitespace character. One or more whitespace characters can be returned as items in the list. I do not want the items in the list that are only whitespace (can be one or more characters of whitespace) and plan to use string.strip on those items that are not only whitespace (to remove any whitespace from front or back of items). What kind of efficient test can I use to obtain only list items returned from the split that I am interested in, ignoring any list items that would only be comprised of one or more characters of whitespace (since whitespace can mean one or more spaces, tabs, and other characters) As a second question, I am seeing string split as deprecated in 2.4.2 manual. What is planned in future to split (strings or unicode)? Regards, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Dictionary of tuples from query question
Thanks Fredrik for your help. Really short and efficient - very nice!
Regards,
David
On Monday, November 14, 2005, at 12:12 PM, Fredrik Lundh wrote:
> I meant to write
>
> d = {}
> for index, record in enumerate(cursor.fetchall()):
> d[index+1] = tuple(record)
>
> which can be shorted to
>
> d = dict((k+1, tuple(v)) for k, v in enumerate(x))
>
> in recent versions of Python.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Dictionary of tuples from query question
Hi Fredrik. Many thanks for your reply and for the tuple tip. The
cursor.fetchall returns a list of lists in this instance with each list
in the main list containing the field values. With the tip, I
simplified my code to:
vlist_dict = {}
record_count = 0
for record in cursor.fetchall():
record_count += 1
vlist_dict[record_count] = tuple(record)
print vlist_dict
That's much better!
Regards
David
On Monday, November 14, 2005, at 07:58 AM, Fredrik Lundh wrote:
>
> but doesn't fetchall already returns tuples, btw? isn't something
> like
>
> d = {}
> for index, record in cursor.fetchall():
> d[index+1] = record
>
> an easier way to get the dictionary you want?
--
http://mail.python.org/mailman/listinfo/python-list
Dictionary of tuples from query question
Hi. I am interested in having results from a db query in the following form.
{1: (value0, value1, value2,), 2: (value0, value1, value2,), 3: (value0, value1, value2,)}
so I generate a dictionary of tuples (field values) dynamically so if I queried a table with five fields I would have five fields in my tuple and if I had 20 fields in my query, I would have 20 fields in my tuple, etc.
Note that the dict starts at 1 and not zero. From the code below, the structure I have currently is:
{1: {0:value0, 1:value1, 2:value2}, 2: {0:value0, 1:value1, 2:value2}, 3: {0:value0, 1:value1, 2:value2}}
My code so far. I guess my problem is how to generate a tuple dynamically when it is immutable?
Many thanks
David
cursor = db.cursor()
cursor.execute("""
SELECT * FROM countries;
""")
rows = cursor.fetchall()
vdict = {}
for row in range(len(rows)):
col_dict = {}
for col in range(len(rows[row])):
value = rows[row][col]
col_dict[col] = value
vdict[row + 1] = col_dict
print vdict
--
http://mail.python.org/mailman/listinfo/python-list
Re: sqlite3 decode error
Hi Jean-Paul for some really good advice. I'll take a look at the
project to see how this is handled. I was not aware of your wrapper
project for SQLite - so this is something new to look at too. I have
worked with SQLObject and also Django's db wrappers. In fact this
question has come out of an SQLObject implementation in RDFlib since it
is here I discovered this issue in the way this backend is behaving
with SQLite3 and I have got it working now. I am only starting to warm
to the idea of unicode throughout. For example. In the backend code
that I am trying to work with you have this.
_tokey is a helper to bring things into the relational database,
_fromkey is a helper when extracting data from the database.
Commenting out the .decode("UTF-8") and value = value.decode("UTF-8")
allowed me to get this working but I need to make this work with
unicode. My unicode experience is limited and I am confused about
writing unicode compatible replacements for things like:
return '<%s>' % ''.join(splituri(term.encode("UTF-8")))
def splituri(uri):
if uri.startswith('<') and uri.endswith('>'):
uri = uri[1:-1]
if uri.startswith('_'):
uid = ''.join(uri.split('_'))
return '_', uid
if '#' in uri:
ns, local = rsplit(uri, '#', 1)
return ns + '#', local
if '/' in uri:
ns, local = rsplit(uri, '/', 1)
return ns + '/', local
return NO_URI, uri
def _fromkey(key):
if key.startswith("<") and key.endswith(">"):
key = key[1:-1].decode("UTF-8") ## Fails here when data
extracted from database
if key.startswith("_"):
key = ''.join(splituri(key))
return BNode(key)
return URIRef(key)
elif key.startswith("_"):
return BNode(key)
else:
m = _literal.match(key)
if m:
d = m.groupdict()
value = d["value"]
value = unquote(value)
value = value.decode("UTF-8") ## Fails here when data
extracted from database
lang = d["lang"] or ''
datatype = d["datatype"]
return Literal(value, lang, datatype)
else:
msg = "Unknown Key Syntax: '%s'" % key
raise Exception(msg)
def _tokey(term):
if isinstance(term, URIRef):
term = term.encode("UTF-8")
if not '#' in term and not '/' in term:
term = '%s%s' % (NO_URI, term)
return '<%s>' % term
elif isinstance(term, BNode):
return '<%s>' % ''.join(splituri(term.encode("UTF-8")))
elif isinstance(term, Literal):
language = term.language
datatype = term.datatype
value = quote(term.encode("UTF-8"))
if language:
language = language.encode("UTF-8")
if datatype:
datatype = datatype.encode("UTF-8")
n3 = '"%s"@%s&<%s>' % (value, language, datatype)
else:
n3 = '"%s"@%s' % (value, language)
else:
if datatype:
datatype = datatype.encode("UTF-8")
n3 = '"%s"&<%s>' % (value, datatype)
else:
n3 = '"%s"' % value
return n3
else:
msg = "Unknown term Type for: %s" % term
raise Exception(msg)
In an unrelated question, it appears SQLite is also extremely flexible
about what types of data it can contain. When writing SQL in Postgres
I use timestamp type and can use this also in SQLite. With my work with
Django, the same information is mapped to datetime type. Would you be
inclined to recommend the use of one type over the other. If so, can
you explain the rationale for this choice. Many thanks.
Regards,
David
On Tuesday, November 8, 2005, at 04:49 PM, Jean-Paul Calderone wrote:
> On Tue, 08 Nov 2005 16:27:25 -0400, David Pratt
> <[EMAIL PROTECTED]> wrote:
>> Recently I have run into an issue with sqlite where I encode strings
>> going into sqlite3 as utf-8. I guess by default sqlite3 is converting
>> this to unicode since when I try to decode I get an attribute error
>> like this:
>>
>> AttributeError: 'unicode' object has no attribute 'decode'
>>
>> The code and data I am preparing is to work on postgres as well a
>> sqlite so there are a c
sqlite3 decode error
Recently I have run into an issue with sqlite where I encode strings
going into sqlite3 as utf-8. I guess by default sqlite3 is converting
this to unicode since when I try to decode I get an attribute error
like this:
AttributeError: 'unicode' object has no attribute 'decode'
The code and data I am preparing is to work on postgres as well a
sqlite so there are a couple of things I could do. I could always
store any data as unicode to any db, or test the data to determine
whether it is a string or unicode type when it comes out of the
database so I can deal with this possibility without errors. I will
likely take the first option but I looking for a simple test to
determine my object type.
if I do:
>>>type('maybe string or maybe unicode')
I get this:
>>>
I am looking for something that I can use in a comparison.
How do I get the type as a string for comparison so I can do something
like
if type(some_data) == 'unicode':
do some stuff
else:
do something else
Regards,
David
--
http://mail.python.org/mailman/listinfo/python-list
Re: Stripping ASCII codes when parsing
This is very nice :-) Thank you Tony. I think this will be the way to go. My concern ATM is where it will be best to unicode. The data after this will go into dict and a few processes and into database. Because input source if not explicit encoding, I will have to assume ISO-8859-1 I believe but could well be cp1252 for most part ( because it says no ASCII (0-30) but alright ASCII chars 128-254) and because most are Windows users. Am thinking to unicode after stripping these characters and validating text, then unicoding (utf-8) so it is unicode in dict. Then when I perform these other processes it should be uniform and then it will go into database as unicode. I think this should be ok. Regards, David On Monday, October 17, 2005, at 01:48 PM, Tony Nelson wrote: > In article <[EMAIL PROTECTED]>, > David Pratt <[EMAIL PROTECTED]> wrote: > >> I am working with a text format that advises to strip any ascii >> control >> characters (0 - 30) as part of parsing data and also the ascii pipe >> character (124) from the data. I think many of these characters are >> from a different time. Since I have never seen most of these >> characters >> in text I am not sure how these first 30 control characters are all >> represented (other than say tab (\t), newline(\n), line return(\r) ) >> so >> what should I do to remove these characters if they are ever >> encountered. Many thanks. > > Most of those characters are hard to see. > > Represent arbitrary characters in a string in hex: "\x00\x01\x02" or > with chr(n). > > If you just want to remove some characters, look into "".translate(). > > nullxlate = "".join([chr(n) for n in xrange(256)]) > delchars = nullxlate[:31] + chr(124) > outputstr = inputstr.translate(nullxlate, delchars) > ___ > _ > TonyN.:' > [EMAIL PROTECTED] > ' > <http://www.georgeanelson.com/> > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Stripping ASCII codes when parsing
Hi Steve. My plan is to parse the data removing the control characters and validate to data as records are being added to a dictionary. I am going to Unicode after this step but before it gets into storage (in which case I think the translate method could work well). The encoding itself is not explicit for this data except to say that it is ASCII and that besides not using chars 0-30, ASCII 128-254 is permitted. I am not certain whether I should assume cp1252 or ISO-8859-1. I can't say that everyone is using Windows although likely vast majority for sure. Would you think it safe to unicode before or after seeking out control characters and validating stage? My validations are relatively simple but to ensure that if I am expecting a date, integer, string etc the data is what it is supposed to be, (since next stage is database), unify whitespace, remove control characters, and check for SQL strings in the data to prevent any stupid things from happening if someone wanted to be malicious. Regards, David On Monday, October 17, 2005, at 12:49 PM, Steve Holden wrote: > David Pratt wrote: > [about ord(), chr() and stripping control characters] >> Many thanks Steve. This is good information. I think this should work >> fine. I was doing a string.replace in a cleanData() method with the >> following characters but don't know if that would have done it. This >> contains all the control characters that I really know about in normal >> use. ord(c) < 32 sounds like a much better way to go and >> comprehensive. >> So I guess instead of string.replace, I should do a... for char >> in ...and check evaluate each character, correct? - or is there a >> better way of eliminating these other that reading a string in >> character by character. >> >> '\a','\b','\e','\f','\n','\r','\t','\v','|' >> > > There are a number of different things you might want to try. One is > translate() which, given a string and a translate table, will perform > the translation all in one go. For example: > >>>> delchars = "".join(chr(i) for i in range(32)) + "|" >>>> print repr(delchars) > '\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12 > \x13\x14\ > x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f|' >>>> nultxfrm = "".join(chr(i) for i in range(256)) >>>> > > So delchars is a list of characters you want to remove, and nultxfrm is > a 256-character string where the nultxfrm[n] == chr(n) - this performs > no translation at all. So then > > s = s.translate(nultxfrm, delchars) > > will remove all the "illegal" characters from s. > > Note that I am sort-of cheating here, as this is only going to work for > 8-bit characters. Once Unicode enters the picture all bets are off. > > regards > Steve > -- > Steve Holden +44 150 684 7255 +1 800 494 3119 > Holden Web LLC www.holdenweb.com > PyCon TX 2006 www.python.org/pycon/ > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Stripping ASCII codes when parsing
Many thanks Steve. This is good information. I think this should work fine. I was doing a string.replace in a cleanData() method with the following characters but don't know if that would have done it. This contains all the control characters that I really know about in normal use. ord(c) < 32 sounds like a much better way to go and comprehensive. So I guess instead of string.replace, I should do a... for char in ...and check evaluate each character, correct? - or is there a better way of eliminating these other that reading a string in character by character. '\a','\b','\e','\f','\n','\r','\t','\v','|' Regards, David On Monday, October 17, 2005, at 06:04 AM, Steve Holden wrote: > David Pratt wrote: >> I am working with a text format that advises to strip any ascii >> control >> characters (0 - 30) as part of parsing data and also the ascii pipe >> character (124) from the data. I think many of these characters are >> from a different time. Since I have never seen most of these >> characters >> in text I am not sure how these first 30 control characters are all >> represented (other than say tab (\t), newline(\n), line return(\r) ) >> so >> what should I do to remove these characters if they are ever >> encountered. Many thanks. > > You will find the ord() function useful: control characters all have > ord(c) < 32. > > You can also use the chr() function to return a character whose ord() > is > a specific value, and you can use hex escapes to include arbitrary > control characters in string literals: > >myString = "\x00\x01\x02" > > regards > Steve > -- > Steve Holden +44 150 684 7255 +1 800 494 3119 > Holden Web LLC www.holdenweb.com > PyCon TX 2006 www.python.org/pycon/ > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Stripping ASCII codes when parsing
I am working with a text format that advises to strip any ascii control characters (0 - 30) as part of parsing data and also the ascii pipe character (124) from the data. I think many of these characters are from a different time. Since I have never seen most of these characters in text I am not sure how these first 30 control characters are all represented (other than say tab (\t), newline(\n), line return(\r) ) so what should I do to remove these characters if they are ever encountered. Many thanks. -- http://mail.python.org/mailman/listinfo/python-list
Re: CSV module and Mac excel format problem
Many thanks Andrew for this excellent piece of knowledge :-). 4 characters of code and everything is great! Regards, David On Friday, October 14, 2005, at 12:11 AM, Andrew McNamara wrote: >> Hi. I have had good success with CSV module but recently came across >> problem with reading excel from Mac Office. The trouble is with line >> endings. Instead of \r\n you have just \r and the file as a whole >> appears as a single line. CSV coughs and provides this exception: >> >> _csv.Error: newline inside string >> >> Saving as Windows (text) in Mac Office solves this but I don't >> necessarily want to force users to save this way just to avoid >> modifying my code. There is a lineterminator in the package. Adding >> parameter lineterminator='\r' did not correct the problem. > > Open the file in universal-newline mode - for example: > > sample = open(filename, 'rU') > > "lineterminator" is only used for output - we use the supplied iterator > (and whatever conventions it imposes) for input. > > -- > Andrew McNamara, Senior Developer, Object Craft > http://www.object-craft.com.au/ > -- http://mail.python.org/mailman/listinfo/python-list
CSV module and Mac excel format problem
Hi. I have had good success with CSV module but recently came across problem with reading excel from Mac Office. The trouble is with line endings. Instead of \r\n you have just \r and the file as a whole appears as a single line. CSV coughs and provides this exception: _csv.Error: newline inside string Saving as Windows (text) in Mac Office solves this but I don't necessarily want to force users to save this way just to avoid modifying my code. There is a lineterminator in the package. Adding parameter lineterminator='\r' did not correct the problem. The way I am reading tab delimited to dictionary: lines = sample.readlines() headers = csv.reader(lines, delimiter='\t').next() rows = csv.DictReader(lines, headers, delimiter='\t') for row in rows: print row Regards, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Python based unacceptable language filter
Hi. Thank you for the links. I am looking for something that would function in a similar way to Yahoo's filter for it's message boards. Perhaps I should have used the term profanity instead of unacceptable language. I am not concerned about correcting sentence structure or poor grammar. I realize screening profanity can be accomplished by simply looping over regular expressions from a database or dictionary to search the text to check against possibilities . I thought it possible that there may be something like this already in existence, perhaps already in a module since it is likely (despite how absurd) - that someone has developed a dictionary of profane word expressions I suspect. What's is perhaps more crazy, is that one has to consider including something like this in an application - but you have to conclude the Internet is what it is. Regards David From Yahoo: "The Profanity Filter allows you to control how you want to view messages with profanity in two ways. You can choose to view the messages with the profanity masked with italcized symbols (@$&% ), or you can have the messages containing profanity hidden entirely. You can also choose between a weak setting for exact word matches or a strong setting that will filter spelling variations." Well I know this thread is a On Sunday, October 2, 2005, at 10:45 PM, Nigel Rowe wrote: > David Pratt wrote: > >> Hi. Is anyone aware of any python based unacceptable language filter >> code to scan and detect bad language in text from uploads etc. >> >> Many thanks. >> David > > You might be able to adapt languagetool. > http://www.danielnaber.de/languagetool/features.html > > Later versions have been ported to Java, but the old python version of > languagetool is at http://tkltrans.sourceforge.net/#r03 > > His thesis paper is at > http://www.danielnaber.de/languagetool/download/ > style_and_grammar_checker.pdf > > Mind you, given the poor language skills of many native english > speakers > (not to mention those for whom english is a second language) relying on > automated filters to enforce 'good' language seems a trifle extreme. > This > post for example would probably not pass. -- http://mail.python.org/mailman/listinfo/python-list
Python based unacceptable language filter
Hi. Is anyone aware of any python based unacceptable language filter code to scan and detect bad language in text from uploads etc. Many thanks. David -- http://mail.python.org/mailman/listinfo/python-list
Re: Sniffing Text Files
Hi Steven. Thank you for your detailed response. The code will be
executed on a web server with limited memory so the desire to keep file
loading in check. I like the approach you have suggested to score to
give the best guess. It leaves it fairly modular in respect to how
detailed you want to be about adding statements specific to a
particular format (that would increase the certainty of choosing it
correctly). I wish I had more control over the files I may receive but
I have to assume the worse. File extensions are not always telling the
true situation and sometimes they can be left off. Mime types are not
always interpreted properly either and I am restricting these before
getting to a sniffing stage to eliminate certain types of files from
getting that far. I think what I might do is read the first x lines
with readlines(). I think a sample of up to the first 100 lines should
probably be good enough to generate a decent scores for the type.
Regards,
David
> def sniff(filename):
> """Return one of "xml", "csv", "txt" or "tkn", or "???"
> if it can't decide the file type.
> """
> fp = open(filename, "r")
> scores = {"xml": 0, "csv": 0, "txt": 0, "tkn": 0}
> for line in fp.readlines():
> if not line:
> continue
> if line[0] == "<":
> scores["xml"] += 1
> if '\t' in line:
> scores["txt"] += 1
> if ',' in line:
> scores["csv"] += 1
> if SOMETOKEN in line:
> scores["csv"] += 1
> # Pick the best guess:
> L = [(score, name) for (name, score) in scores.items()]
> L.sort()
> L.reverse()
> # L is now sorted from highest down to lowest by score.
> best_guess = L[0]
> second_best_guess = L[0]
> if best_guess[0] > 10*second_best_guess[0]:
> fp.close()
> return best_guess[1]
> fp.close()
> return "???"
--
http://mail.python.org/mailman/listinfo/python-list
Re: Sniffing Text Files
Thanks Mike this is really great! Regards, David On Friday, September 23, 2005, at 11:55 AM, Mike Meyer wrote: > David Pratt <[EMAIL PROTECTED]> writes: >> Thanks Mike for your reply. I am not aware of libmagic and will look >> to see what it provides. > > and ... > > Skip Montanaro <[EMAIL PROTECTED]> writes: >> You can also run the file(1) command and see what it says. I seem >> to recall someone asking about the equivalent to file(1) implemented >> in >> Python awhile back. > > libmagic is the smarts of the file command. As I said before, people > have done Python wrappers for it. It uses a text database describing > how to recognize a files type - see the magic(5) man page for details > on that. If you use libmagic, you'll probably want to provide your own > version of the databse, excerpted to include just the file types you > want to recognize. > > You can check on whether or not this will work for you by seeing what > the file command says about your sample data. > > -- > Mike Meyer <[EMAIL PROTECTED]> > http://www.mired.org/home/mwm/ > Independent WWW/Perforce/FreeBSD/Unix consultant, email for more > information. > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Sniffing Text Files
Hi Skip. Thank you for your reply. This is helpful and I am glad I put this to the list. There are some really good ideas that will help me come up with something good to use. Regards, David On Friday, September 23, 2005, at 11:14 AM, [EMAIL PROTECTED] wrote: > > David> I realize CSV module has a sniffer but it is something that > is > David> limited more or less to delimited files. > > Sure. How about: > > def sniff(fname): > if open(fname).read(4) == " return "xml" > else: > # assume csv - use its sniffer to generate a dialect > return d > > Of course, as the number of file formats grows, you'll need to expand > the > logic. You can also run the file(1) command and see what it says. I > seem > to recall someone asking about the equivalent to file(1) implemented in > Python awhile back. > > -- > Skip Montanaro > Katrina Benefit Concerts: http://www.musi-cal.com/katrina > [EMAIL PROTECTED] > -- http://mail.python.org/mailman/listinfo/python-list
Re: Sniffing Text Files
Thanks Mike for your reply. I am not aware of libmagic and will look to see what it provides. As far as your first suggestion, this is what I have been looking at - probably a combination regex and readlines or similar but trying to get a better sense of best sort of approach more or less. I can't rely on file extensions in this case so believing the content will be what the file extension indicates would not be so good. Mime types can be helpful but don't always give you the full story either - so the need to sniff in the first place so I can apply the right process to the file. As it stands I am filtering mime types to the importing process to attempt to limit the possibilities. Regards, David On Friday, September 23, 2005, at 02:01 AM, Mike Meyer wrote: > David Pratt <[EMAIL PROTECTED]> writes: > >> Hi. I have files that I will be importing in at least four different >> plain text formats, one of them being tab delimited format, a couple >> being token based uses pipes (but not delimited with pipes), another >> being xml. There will likely be others as well but the data needs to >> be extracted and rewritten to a single format. The files can be fairly >> large (several MB) so I do not want to read the whole file into >> memory. What approach would be recommended for sniffing the files for >> the different text formats. I realize CSV module has a sniffer but it >> is something that is limited more or less to delimited files. I have >> a couple of ideas on what I could do but I am interested in hearing >> from others on how they might handle something like this so I can >> determine the best approach to take. Many thanks. > > With GB memory machines being common, I wouldn't think twice about > slurping a couple of meg into RAM to examine. But if that's to much, > how about simply reading in the first bytes, and checking that > for the characters you want? should be large enough to reveal > what you need, but small enogh that your'e comfortable reading it > in. I'm not sure that there aren't funny interactions between read and > readline, so do be careful with that. > > Another approach to consider is libmagic. Google turns up a number of > links to Python wrappers for it. > >-- > Mike Meyer <[EMAIL PROTECTED]> > http://www.mired.org/home/mwm/ > Independent WWW/Perforce/FreeBSD/Unix consultant, email for more > information. > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Sniffing Text Files
Hi. I have files that I will be importing in at least four different plain text formats, one of them being tab delimited format, a couple being token based uses pipes (but not delimited with pipes), another being xml. There will likely be others as well but the data needs to be extracted and rewritten to a single format. The files can be fairly large (several MB) so I do not want to read the whole file into memory. What approach would be recommended for sniffing the files for the different text formats. I realize CSV module has a sniffer but it is something that is limited more or less to delimited files. I have a couple of ideas on what I could do but I am interested in hearing from others on how they might handle something like this so I can determine the best approach to take. Many thanks. Regards, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Example of signaling and creating a python daemon
Hi jepler! This is a nice example. I will study it so I can better see what each part is doing. I may have a couple of questions once I have done this because I have not yet daemonized a process and I want to be sure I understand before I attempt to run anything I create. Many thanks. David On Thursday, September 15, 2005, at 12:31 PM, [EMAIL PROTECTED] wrote: > Here's a program I use to control volume. Run one way, it waits for a > Unix signal and adjusts the volume according to the signal received. > Run another way, the PID of the daemon process is determined and a > signal is sent according to a commandline argument. -- http://mail.python.org/mailman/listinfo/python-list
Example of signaling and creating a python daemon
Hi. I am running a zope server. Zope runs 4 threads and I have a document processing method that can require minutes to run so I do not want to run out of threads. A solution to this is to run this process asynchronously. What I am hoping to do is send a signal to a python deamon to run a process and then go back to sleep. I am hoping to find a clear example of: * sending a signal from a python script to start a python daemon somewhere on the filesystem * a python daemon that utilizes the signal to wake up from sleep and run a method, then go back to sleep. I have no experience daemonizing a script so this help would be much appreciated. I have found a couple of scripts googling on creating a deemon but not work in this way. Regards, David -- http://mail.python.org/mailman/listinfo/python-list
setup install question
I have two versions of python running - 2.4.1 and 2.3.5. Python 2.4.1 is most current and will execute from 'python' at command line where 2.3.5 will execute from 'python2.3'. A recent problem I have is installing rdflib. It installed and works fine on 2.4.1. rdflib uses 'python setup install' for install from command line (fine for the 2.4.1 version) but if won't install with python2.3 setup install without generating a traceback. It is somehow looking at the 2.3 following python as parameters for the version of rdflib. Can I safely copy from 2.4.1 site-packages to 2.3.5's to make rdflib available to 2.3.5 python? Many thanks David -- http://mail.python.org/mailman/listinfo/python-list
Re: Splitting string into dictionary
Thanks George! You guys are great! I am always learning. Python is
awesome!!
On Friday, July 1, 2005, at 02:33 AM, George Sakkis wrote:
> "Robert Kern" wrote:
>
>> Ignore the last message.
>>
>> translations = [x.strip(" '") for x in line.split('|')]
>> d = dict(zip(translations[::2], translations[1::2]))
>
> Or in case you're working with a lot and/or huge records, the second
> line can be more efficiently written as:
>
> from itertools import izip, islice
> d = dict(izip(islice(translations,0,None,2),
> islice(translations,1,None,2)))
>
> George
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Splitting string into dictionary
Pretty amazing Devan! Great regex! Thank you. Regards, David On Friday, July 1, 2005, at 02:29 AM, Devan L wrote: > One line solution. > dict(re.findall(r"'(.+?)' \| '(.+?)'(?:\s\||$)",yourtexthere)) > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Splitting string into dictionary
Wow Robert that is incredible python magic! I am trying to figure out
what this is doing since my attempts were regex and some long string
splitting and collection.
Ok. So it is a list comprehension and then collection. What is zip
doing in the second line?
Regards
David
On Friday, July 1, 2005, at 02:11 AM, Robert Kern wrote:
> David Pratt wrote:
>> I have string text with language text records that looks like this:
>>
>> 'en' | 'the brown cow' | 'fr' | 'la vache brun'
>>
>> Two or more language records can exist in each string (example above
>> shows 2 - but could contain more)
>> The second vertical line character in the example above is the record
>> break in the pattern (between 'cow' and 'fr')
>>
>> What is the shortest route to getting this into a dictionary like:
>>
>> {'en':'the brown cow','fr':'la vache brun'}
>>
>> The language code is always 2 lower case letters.
>>
>> Many thanks.
>
> Ignore the last message.
>
> translations = [x.strip(" '") for x in line.split('|')]
> d = dict(zip(translations[::2], translations[1::2]))
>
> --
> Robert Kern
> [EMAIL PROTECTED]
>
> "In the fields of hell where the grass grows high
> Are the graves of dreams allowed to die."
>-- Richard Harter
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Splitting string into dictionary
I have string text with language text records that looks like this:
'en' | 'the brown cow' | 'fr' | 'la vache brun'
Two or more language records can exist in each string (example above
shows 2 - but could contain more)
The second vertical line character in the example above is the record
break in the pattern (between 'cow' and 'fr')
What is the shortest route to getting this into a dictionary like:
{'en':'the brown cow','fr':'la vache brun'}
The language code is always 2 lower case letters.
Many thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Python JavaScript solution to hold vertical scroll bar
Hi. I am putting together a database application on Zope. I have built a pager for my records (20 per page) but do not want the browser scroll bars to reset to the top of the browser each time the pager is advanced to the previous or next page. The normal behavior is fine for everything but the anchor tags I am using for the pager. Instead of resetting when a new page is sellected, I want the vertical scrollbar to maintain its position so when the link is clicked, the vertical scrollbar position is passed to the next page (so the page does not jerk to the top but stay in the same position). I am using css to style anchor tags as buttons and am currently passing the record start position for the page to the next page. ie https://mydomain.com/path/index_html?start:int=20";>2 I am assuming the best way of handling this would be to pass the current vertical scroll position as a parameter to a JavaScript. Does anyone have such a solution or other advice for solving this? Regards, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Iterate through a list calling functions
Cool! Many thanks George. Yes this is the way to go - objects. Much
better :-)
On Sunday, June 5, 2005, at 02:49 PM, George Sakkis wrote:
> David Pratt wrote:
>> Hi. I am creating methods for form validation. Each validator has its
>> own method and there quite a number of these. For each field, I want
>> to evaluate errors using one or more validators so I want to execute
>> the appropriate validator methods from those available. I am
>> iterating
>> over each validator using validateField method to gather my results.
>> It
>> works but it ugly and inefficient. Can someone advise whether there
>> is
>> a better way of doing this. I realize that the validator variable in
>> my iteration is only a string so question is how can I make the
>> validator string reference a function so I may be able to shorten
>> validateField to something similar to this (instead of my long list of
>> ifs which I am not very happy with):
>>
>> for validator in validators_list:
>> result = validator(name, value)
>> if type (result) in StringTypes:
>> results[name] = result
>>
>> Many thanks
>> David
>>
>> My current situation below:
>>
>> # A large list of validators
>> def isDecimal(name, value):
>> """ Test whether numeric value is a decimal """
>> result = validateRegex(name,
>> value,
>> r'^([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?$',
>> errmsg='is not a decimal number.',
>> ignore=None)
>> return result
>>
>> def isZipCode(name, value):
>> """ Tests if field value is a US Zip Code """
>> result = validateRegex(name,
>> value,
>> r'^(\d{5}|\d{9})$',
>> errmsg='is not a valid zip code.',
>> ignore=None)
>> return result
>>
>> ... more validators
>>
>> # Iterating over validators to gather field errors
>> def validateField(name, value, validators_list, range=None,
>> valid_values=None):
>> """ Validates field input """
>> results={}
>> for validator in validators_list:
>> if validator == 'isContainedIn':
>> result = isContainedIn(name, value)
>> if type (result) in StringTypes:
>> more...
>> if validator == 'isDate':
>> result = isDate(name, value)
>> if type (result) in StringTypes:
>> more...
>> if validator == 'isDecimal':
>> result = isDecimal(name, value)
>> if type (result) in StringTypes:
>> more...
>>
>> more validators ...
>
>
> That's a typical case for using an OO approach; just make a class for
> each validator and have a single polymorphic validate method (I would
> make validators __call__able instead of naming the method 'validate'):
>
> # Abstract Validator class; not strictly necessary but good for
> documentation
> class Validator(object):
> def __call__(self,field,value):
> '''Validate a value for this field.
> Return a string representation of value on success, or None on
> failure.
> '''
> raise NotImplementedError("Abstract method")
>
>
> class DecimalValidator(Validator):
> def __call__(self,name,value):
> '''Test whether numeric value is a decimal.'''
>
> class ZipCodeValidator(Validator):
> def __call__(self,name,value):
> '''Test if value is a US Zip Code.'''
>
>
> def validateField(name, value, validators):
> """ Validates field input """
> results = {}
> for validate in validators:
> result = validate(name,value)
> if result is not None:
> results[name] = result
> # XXX: if more than one validators succeed,
> # all but the last result will be overwritten
> return results
>
> # test
> validators = [DecimalValidator(), ZipCodeValidator()]
> print validateField("home ZIP", "94303", validators)
>
> Regards,
> George
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Iterate through a list calling functions
Hi Kent. Thank you for your reply. I gave this a go but get the
following traceback:
...
result = validator(name, value)
TypeError: 'str' object is not callable
Have put validators in list and iterate over it as in following:
validator_list =
[isContainedIn,isDate,isDecimal,isEmail,isEmpty,isInteger...
more validators]
results={}
for validator in validators_list:
result = validator(name, value)
if type (result) in StringTypes:
# do some stuff...
return results
Regards,
David
On Sunday, June 5, 2005, at 02:03 PM, Kent Johnson wrote:
> David Pratt wrote:
>> Hi. I am creating methods for form validation. Each validator has its
>> own method and there quite a number of these. For each field, I want
>> to
>> evaluate errors using one or more validators so I want to execute the
>> appropriate validator methods from those available. I am iterating
>> over
>> each validator using validateField method to gather my results. It
>> works
>> but it ugly and inefficient. Can someone advise whether there is a
>> better way of doing this. I realize that the validator variable in my
>> iteration is only a string so question is how can I make the validator
>> string reference a function so I may be able to shorten validateField
>> to
>> something similar to this (instead of my long list of ifs which I am
>> not
>> very happy with):
>>
>> for validator in validators_list:
>> result = validator(name, value)
>> if type (result) in StringTypes:
>> results[name] = result
>
> Actually you can do exactly that by putting references to the
> validator functions in your list instead of (string) name. For example
> if you have
> validators = [ 'isDecimal', 'isFoo', 'isBar' ]
>
> just change it to
> validators = [ isDecimal, isFoo, isBar ]
>
> and your loop above will work.
>
> Python makes data-driven programming easy :-)
> Kent
>
>>
>> Many thanks
>> David
>>
>> My current situation below:
>>
>> # A large list of validators
>> def isDecimal(name, value):
>> """ Test whether numeric value is a decimal """
>> result = validateRegex(name,
>> value,
>> r'^([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?$',
>> errmsg='is not a decimal number.',
>> ignore=None)
>> return result
>>
>> def isZipCode(name, value):
>> """ Tests if field value is a US Zip Code """
>> result = validateRegex(name,
>> value,
>> r'^(\d{5}|\d{9})$',
>> errmsg='is not a valid zip code.',
>> ignore=None)
>> return result
>>
>> ... more validators
>>
>> # Iterating over validators to gather field errors
>> def validateField(name, value, validators_list, range=None,
>> valid_values=None):
>> """ Validates field input """
>> results={}
>> for validator in validators_list:
>> if validator == 'isContainedIn':
>> result = isContainedIn(name, value)
>> if type (result) in StringTypes:
>> more...
>> if validator == 'isDate':
>> result = isDate(name, value)
>> if type (result) in StringTypes:
>> more...
>> if validator == 'isDecimal':
>> result = isDecimal(name, value)
>> if type (result) in StringTypes:
>> more...
>>
>> more validators ...
>>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Iterate through a list calling functions
Hi. I am creating methods for form validation. Each validator has its
own method and there quite a number of these. For each field, I want
to evaluate errors using one or more validators so I want to execute
the appropriate validator methods from those available. I am iterating
over each validator using validateField method to gather my results. It
works but it ugly and inefficient. Can someone advise whether there is
a better way of doing this. I realize that the validator variable in
my iteration is only a string so question is how can I make the
validator string reference a function so I may be able to shorten
validateField to something similar to this (instead of my long list of
ifs which I am not very happy with):
for validator in validators_list:
result = validator(name, value)
if type (result) in StringTypes:
results[name] = result
Many thanks
David
My current situation below:
# A large list of validators
def isDecimal(name, value):
""" Test whether numeric value is a decimal """
result = validateRegex(name,
value,
r'^([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?$',
errmsg='is not a decimal number.',
ignore=None)
return result
def isZipCode(name, value):
""" Tests if field value is a US Zip Code """
result = validateRegex(name,
value,
r'^(\d{5}|\d{9})$',
errmsg='is not a valid zip code.',
ignore=None)
return result
... more validators
# Iterating over validators to gather field errors
def validateField(name, value, validators_list, range=None,
valid_values=None):
""" Validates field input """
results={}
for validator in validators_list:
if validator == 'isContainedIn':
result = isContainedIn(name, value)
if type (result) in StringTypes:
more...
if validator == 'isDate':
result = isDate(name, value)
if type (result) in StringTypes:
more...
if validator == 'isDecimal':
result = isDecimal(name, value)
if type (result) in StringTypes:
more...
more validators ...
--
http://mail.python.org/mailman/listinfo/python-list
Re: Max files in unix folder from PIL process
Hi Jason. Many thanks your reply. This is good to know about ls - what did it do? Was it just slow or did the server or machine die? My images will be going into the path of a web server. This is unchartered territory for me and I don't know whether there will be speed and access problems or how the filesystem copes with this kind of volume. I am definitely planning to split the images into directories by size and that will at least divide the number by a factor of the various sizes (but on the higher end this could still be between 150 - 175 thousand images which is still a pretty big number. I don't know if this will be a problem or not or there is really anything to worry about at all - but it is better to obtain advice from those that have been there, done that - or are at least a bit more familiar with pushing limits on Unix resources than to wonder whether it will work. Regards, David On Monday, March 28, 2005, at 07:18 PM, Kane wrote: I ran into a similar situation with a massive directory of PIL generated images (around 10k). No problems on the filesystem/Python side of things but other tools (most noteably 'ls') don't cope very well.As it happens my data has natural groups so I broke the big dir into subdirs to sidestep the problem. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Max files in unix folder from PIL process
Hi. I am creating a python application that uses PIL to generate thumbnails and sized images. It is beginning to look the volume of images will be large. This has got me to thinking. Is there a number that Unix can handle in a single directory. I am using FreeBSD4.x at the moment. I am thinking the number could be as high 500,000 images in a single directory but more likely in the range of 6,000 to 30,000 for most. I did not want to store these in Postgres. Should this pose a problem on the filesystem? I realize less a python issue really but I though some one might have an idea on the list. Regards, David. -- http://mail.python.org/mailman/listinfo/python-list
pythonwx with database starting point
Hi, I am looking for some example code or open source project that involves pythonwx and a database. I am wanting to store images and data on the filesystem and transmit thumbnailed images and subsets of records to the web using xmlrpc preferably. I am wanting the application to be able to be run as an exe from a local Windows machine. I would appreciate links to any existing material that could assist me in getting started with some code. Many thanks. Regards, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Secondary list sorting comparison expression
Hi Raymond. I appreciate your reply. Yes, this is exactly what I was looking for. The syntax I had been trying to work out myself was not correct and not giving me the right thing. Many thanks for your help - this works just the way I wanted. David On Tuesday, March 8, 2005, at 03:46 PM, Raymond Hettinger wrote: Another way is to build out the sort_by_key function to handle multiple fields: def sort_by_key(list, i, j, k): list.sort(lambda a, b: cmp((a[i], a[j], a[k]), (b[i], b[j], b[k]))) In Py2.4, the key= option offers an alternative to cmp which is simpler and faster: def sort_by_key(list, i, j, k): list.sort(key = lambda a : (a[i], a[j], a[k])) -- http://mail.python.org/mailman/listinfo/python-list
Secondary list sorting comparison expression
I have been using the following for sorting a list of dictionaries.
This works but only provides sorting on a single key. I am wanting to
extend this with a better comparison expression so that it would sort
on one key as primary, a second key as secondary sort , and maybe even
a third as tertiary.
def sort_by_key(list, i):
list.sort(lambda a, b: cmp(a[i], b[i]))
test_list = [{'number': 1, 'written_number': 'one','animal':
'giraffe','price': 1.50}, {'number': 2, 'written_number':
'two','animal': 'elephant', 'price': 2.50}, {'number': 3,
'written_number': 'three','animal': 'zebra', 'price': 1.50}, {'number':
4, 'written_number': 'four','animal': 'hippopotamus', 'price': 1.50}]
sort_by_key(test_list, 'price')
This returns:
[{'price': 1.5, 'number': 1, 'animal': 'giraffe', 'written_number':
'one'}, {'price': 1.5, 'number': 3, 'animal': 'zebra',
'written_number': 'three'}, {'price': 1.5, 'number': 4, 'animal':
'hippopotamus', 'written_number': 'four'}, {'price': 2.5, 'number': 2,
'animal': 'elephant', 'written_number': 'two'}]
What I am looking for would sort on more than one key so, say 'price'
and 'animal' so that the list would be ordered with the three items at
1.50 each but records would be giraffe, hippopotamus and zebra. If I
wanted to sort on three indexes, say 'price', 'animal' and 'written
number' and say last two animals were both zebras that it would put
this in the correct order also.
I am have to say I have read a good deal about sorting today but have
not come across a solution and have seen only limited comparison
expressions to know what could work.
Regards,
David
--
http://mail.python.org/mailman/listinfo/python-list
