Re: [Tutor] UTF-8 filenames encountered in os.walk
On Wed, 2007-07-04 at 12:00 -0400, William O'Higgins Witteman wrote:
> On Wed, Jul 04, 2007 at 11:28:53AM -0400, Kent Johnson wrote:
>
> >FWIW, I'm pretty sure you are confusing Unicode strings and UTF-8
> >strings, they are not the same thing. A Unicode string uses 16 bits to
> >represent each character. It is a distinct data type from a 'regular'
> >string. Regular Python strings are byte strings with an implicit
> >encoding. One possible encoding is UTF-8 which uses one or more bytes to
> >represent each character.
> >
> >Some good reading on Unicode and utf-8:
> >http://www.joelonsoftware.com/articles/Unicode.html
> >http://effbot.org/zone/unicode-objects.htm
>
> The problem is that the Windows filesystem uses UTF-8 as the encoding
> for filenames, but os doesn't seem to have a UTF-8 mode, just an ascii
> mode and a Unicode mode.
Are you converting your utf-8 strings to unicode?
unicode_file_name = utf8_file_name.decode('UTF-8')
> >If you pass a unicode string (not utf-8) to os.walk(), the resulting
> >lists will also be unicode.
> >
> >Again, it would be helpful to see the code that is getting the error.
>
> The code is quite complex for not-relevant-to-this-problem reasons. The
> gist is that I walk the FS, get filenames, some of which get written to
> an XML file. If I leave the output alone I get errors on reading the
> XML file. If I try to change the output so that it is all Unicode, I
> get errors because my UTF-8 data sometimes looks like ascii, and I don't
> see a UTF-8-to-Unicode converter in the docs.
>
It is probably worth the effort to put together a simpler piece of code
that can illustrate the problem.
> >>I suspect that my program will have to make sure to recast all
> >>equivalent-to-ascii strings as UTF-8 while leaving the ones that are
> >>already extended alone.
> >
> >It is nonsense to talk about 'recasting' an ascii string as UTF-8; an
> >ascii string is *already* UTF-8 because the representation of the
> >characters is identical. OTOH it makes sense to talk about converting an
> >ascii string to a unicode string.
>
> Then what does mystring.encode("UTF-8") do?
It uses utf8 encoding rules to convert mystring FROM unicode to a
string. If mystring is *NOT* unicode but simply a string, it appears to
do a round trip decode and encode of the string. This allows you to
find encoding errors, but if there are no errors the result is the same
as what you started with.
The data in a file (streams of bytes) are encoded to represent unicode
characters. The stream must be decoded to recover the underlying
unicode. The unicode must be encoded when written to files. utf-8 is
just one of many possible encoding schemes.
--
Lloyd Kvam
Venix Corp
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] how can I compare a local directory or file with a remote one
On Thu, 2007-06-21 at 09:47 -0400, Danny Yoo wrote: > >> At first blush it sounds to me like you want rsync, as has been > >> pointed out. If on of the systems is running an OS that doesn't come > >> with rsync... I have used http://www.vdesmedt.com/~vds2212/rsync.html > >> to achieve the same result. > >> > > > > When I need to deal with Windows computers, I sorely feel the lack of > > ssh, rsync, and sshfs. > > > I know this is very Python-unspecific, but I might as well chime in: > there's a utility called "Unison" that's similar to rsync, and works on > Windows: > > http://www.cis.upenn.edu/~bcpierce/unison/ Unfortunately unison uses either ssh or an unsecured socket for communicating over a network. From their docs: Warning: The socket method is insecure: not only are the texts of your changes transmitted over the network in unprotected form, it is also possible for anyone in the world to connect to the server process and read out the contents of your filesystem! (Of course, to do this they must understand the protocol that Unison uses to communicate between client and server, but all they need for this is a copy of the Unison sources.) The socket method is provided only for expert users with specific needs; everyone else should use the ssh method. Just to get a bit more Python related, I've used the urllib2 libraries to make SSL connections to a DAV (also spelled WebDav) server. This is reasonably straightforward to program in Python, but the server side is clumsy to setup. -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] how can I compare a local directory or file with a remote one
On Wed, 2007-06-20 at 23:51 -0400, Reed O'Brien wrote: > On Jun 20, 2007, at 2:03 PM, Richard Querin wrote: > > > I'm interested in writing a quick script that would run a diff-type > > command that would compare a local directory to a remote one to > > identify the changes in the files within that directory. > > > > I was initially thinking that I would maybe use the linux diff command > > in conjunction with the wget command (or something similar) to create > > a local copy but that involves downloading files. Is there any way in > > python to do a similar thing but without having to download a copy of > > the remote files/directories? > > > > Any ideas? > > > > At first blush it sounds to me like you want rsync, as has been > pointed out. If on of the systems is running an OS that doesn't come > with rsync... I have used http://www.vdesmedt.com/~vds2212/rsync.html > to achieve the same result. > I downloaded the file from the web site, but was disappointed to see that there appears to be no support for any network transport. It simply operates on locally mounted directories. I only did a cursory scan of the source code. Did I miss something? Certainly the program provides useful example code for file comparisons, but it does not appear to directly support any kind of remote processing. When I need to deal with Windows computers, I sorely feel the lack of ssh, rsync, and sshfs. > ~r > ___ > Tutor maillist - Tutor@python.org > http://mail.python.org/mailman/listinfo/tutor -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] numpy memory muncher
On Sat, 2006-11-11 at 15:40 -0500, Joel Levine wrote: > I'm using, perhaps misusing numpy which is eating up the memory and, > eventually crashing my program. OK. that's a small enough piece of code to figure things out. One quick suggestion, for looping variables xrange avoids creating a real list. It provides a lazy evaluation for those cases where you never need the whole range-list to exist as an entity. > > Isolating it, the following piece of code continually eats memory. Is it my > program or what ...? > Thanks > Joel Levine > > Using Mac OSX 10.4.7 > Not clear on versions: Appears to be 0.9.8 with py2.4 > > - > > from numpy import * > > a=zeros((2000,100),float) > while True: > for row in range(2000): > for col in range(100): > x=a[row,col] > #if x!=0: print "?" > > > > > ___ > Tutor maillist - Tutor@python.org > http://mail.python.org/mailman/listinfo/tutor -- Lloyd Kvam Venix Corp. 1 Court Street, Suite 378 Lebanon, NH 03766-1358 voice: 603-653-8139 fax:320-210-3409 ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python processing of web html forms
On Thu, 2006-08-03 at 10:38 -0700, Jeff Peery wrote: > Hello, I want to use python to process information input into a HTML > form on my website. how is this typically performed? I saw cherrypy > and also quixote for web python stuff... I have no experience with > python on the web. Thanks. TurboGears provides a very nice tutorial. http://www.turbogears.org/docs/wiki20/ That should help get you started. > > jeff > > > > __ > Talk is cheap. Use Yahoo! Messenger to make PC-to-Phone calls. Great > rates starting at 1ยข/min. > ___ > Tutor maillist - Tutor@python.org > http://mail.python.org/mailman/listinfo/tutor -- Lloyd Kvam Venix Corp. 1 Court Street, Suite 378 Lebanon, NH 03766-1358 voice: 603-653-8139 fax:320-210-3409 ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Writing to a remote file
On Fri, 2006-06-16 at 14:47 +0100, kieran flanagan wrote: > Hi > > I want to run a script on one machine and log output messages to a > remote file located on another machine. Is there any easy method of > doing this ?. http://docs.python.org/lib/module-logging.html If the remote machine is a file server, you can use the logging module to write to a file on the remote machine Also the logging module can be used to easily log to a syslog handler. If this remote file is a "normal" log file managed by a syslog process then you should find the python part pretty easy. The syslog process still needs to be configured to accept your logging messages, but that should not be too difficult. Scanning the logging module docs, it looks like you can use it to write your own process to run on the remote machine to handle "logging messages". Then use the logging module on the local machine to send "logging messages" to the remote machine. > Thanks > Kieran > > -- > "Behind every great man, there is a great woman. Behind that woman is > Mr.T." > ___ > Tutor maillist - Tutor@python.org > http://mail.python.org/mailman/listinfo/tutor -- Lloyd Kvam Venix Corp. 1 Court Street, Suite 378 Lebanon, NH 03766-1358 voice: 603-653-8139 fax:320-210-3409 ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Building an SQL query (Gabriel Farrell)
The code to update the database should look something like:
the_cursor.execute( sql_cmd, data)
I am not using postgresql so I do not know the place-holder mark for
your module. You can get that from the module documentation.
So sql_cmd could be something like
"UPDATE my_table SET title= where
record_id="
If data gets passed as a tuple (some modules support dict), data is
("Lion's Mane", 12345)
For mysql the place-holder is %s, and the code would be
the_cursor.execute("UPDATE my_table SET title=%s where record_id=%s",
("Lion's Mane", 12345) )
> I'm just getting started with Python and PostgreSQL but I found that
> str(tuple(valueList)) wouldn't work for me because I had a few values
> with apostrophes. PostgreSQL needed 'Lion''s Mane' but Python was
> sending it "Lion's Mane", so I ended up writing this little function:
>
> def sqlNice(valueList):
> count = 1
> y = '('
> for x in valueList:
> x = x.replace("'", "''")
> y = y + "'" + x + "'"
> if count < len(valueList):
> y = y + ', '
> count = count + 1
> y = y + ')'
> y = y.replace("'NULL'", 'NULL')
> return y
>
> Does anyone see any major stumbling blocks in that?
>
> On a side note, I've struggled with PyGreSQL. At first I was using
> the pg module, but I switched to pgdb when insert() wasn't working for
> me and I thought I would have less trouble using something that's
> DB-API compliant. There seemed to be more documentation there, and I
> figured it's a good idea to go with the standard. However, it does
> seem like I'm covering ground I'm sure someone else has already
> crossed when I create these re functions to manipulate queries. For
> inserts, at least, it seems a Python dictionary should be able to do
> the job nicely.
>
> gabe
--
Lloyd Kvam
Venix Corp
--
Lloyd Kvam
Venix Corp
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
[Tutor] Are Empty "Placeholder" Functions possible?
An alternative to pass is raise NotImplementedError This has the advantage/disadvantage of providing a meaningful exception when you start testing and forget to provide some processing logic for a stub function. pass will simply return None when used for a function body. I nearly always use the NotImplementedError exception now in my code as the place holder. -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] brief description of URL's
A sample URL http://www.mysite.com/path-to/file?arg1=stuff&arg2=morestuff http: scheme of URL (specifies protocol or processing) //www.mysite.comis the host. the name of the computer That host computer could be offering many services, web sites, email, time, etc. Each service is given a number which is how you can specify the service you want to use. These numbers are ports. http 'listens' at port 80 smtp (email delivery) listens at port 25 pop (retrieving email from mailbox) is at port 110 https (SSL) is at port 443 If you want your software to connect to a different port number, that can be specified after the host. So if the www.mysite.com computer were running a Zope server that listened at port 8080, the URL to connect to Zope would be http://www.mysite.com:8080 A user name and password can precede the computer name. A @ is used to separate the computer name part from the user identification. http://myname:[EMAIL PROTECTED] The rules for URLs are specified here http://www.ietf.org/rfc/rfc1738.txt Google can help you find more. IANA now provides the registry for URL schemes along with pointers to documentation. http://www.iana.org/assignments/uri-schemes -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] Python Interest Group in New England
(From digest) > I'm aware of the Boston PIG, a smaller one in Amherst and have been > told that there is possibly a PIG in Manchester, NH. The Dartmouth Lake Sunapee Linux User Group http://dlslug.org/ also supports a python mail list. http://dlslug.org/mailing_lists.html The Python folks are scattered around the state. I'm in Lebanon, NH. Seven people in signed up for python.meetup in Lebanon, but we've never had more than 4 show up for a meeting. Sign up for the python-talk list and let us know you joined. http://www.gnhlug.org/ I think you'll find the local LUGs will welcome Python folks, even those using Macs and Windows. -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] TKinter and things over Linux
Learning to install packages from source tar balls is useful knowledge. However, most of the current Linux distributions have package managers that greatly simply installing software. apt-get and yum are available for Redhat style releases. They will download and install packages and figure out the dependency issues. (yum is written in Python so this is slightly on topic.) Sometimes the hardest part is getting the package manager working on an old release that did not include it. Fedora 3 will generally prove easier to manage than Redhat 9 simply because there are more people packaging code for the newer releases. You'll be able to rely on yum to install software and keep the system humming along. yum install MySQL-python will determine which packages are needed, and download and install them. -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] Talking to mssql?
I replied earlier from a digest entry (which starts a new thread) so you might have missed that reply. I included a URL for the mssql package which included the compile instructions for using freetds. Here is the link. Just follow the instructions and you should be able to compile cleanly. http://mail.python.org/pipermail/db-sig/2005-April/004403.html A new sybase module has been released. Note the instructions for installing with freetds. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] Talking to mssql
The freetds project talks to Sybase and mssql http://www.freetds.org/ http://mail.python.org/pipermail/db-sig/2005-April/004403.html A new sybase module has been released. Note the instructions for installing with freetds. -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] OT - SQL methodolgy
> > In the SQL books I've got, they always seem to have an optional select > > statement on the end of inserts/updates, and I was thinking maybe I > > could do it that way also, but I can't figure out a logical way of > > putting > > > > 'select primary_key from foo where primary_key value > every other > > primary_key value' > > > > select max(primary_key) from foo? select max will NOT work reliably when you have concurrent database inserts. You could obtain the number from someone else's insert. You need to use the function provided by the RDBMS that is tied to your connection/cursor so that you retrieve the primary_key that was assigned to *your* record. (I understood your request to be looking for the primary_key auto-assigned to your insert statement) -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] subclassing across multiple modules
You want a method in a base class to parse input and create instances of certain derived classes. Your sample code looks like: if some_condition_on_chunk_contents: node = Node1(chunk_contents) else: node = Node2(chunk_contents) I'd suggest changing the method to use a variable to determine the class. Following your pattern of coding: #Toolkit.py class Tree: node1 = Node1 node2 = Node2 ... class Node1(Node): ... #Application.py class Tree(Toolkit.Tree): node1 = Node1 node2 = Node2 ... class Nodes(Node,Toolkit.Node1): ... if some_condition_on_chunk_contents: node = self.node1(chunk_contents) else: node = self.node2(chunk_contents) This requires that the class constructors be similar enough so that they take the same parameters. It is also a manual effort to keep the node assignments in Tree in sync with the Node classes that you write. You also face a similar issue keeping Application.py and Toolkit.py in sync. -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] cataloging web Page Information
A minor addition to Kent's advice: urllib2 can be used to post form data to a web site. This is very convenient if the expected data format is stable. You will still need the urlencode from urllib to encode the data to be posted. -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] subclassing list -- slicing doesn't preserve type
> Message: 2 > Date: Tue, 22 Feb 2005 13:53:44 +0100 > From: [EMAIL PROTECTED] (Karl Pfl?sterer ) > Subject: Re: [Tutor] subclassing list -- slicing doesn't preserve type > To: tutor@python.org > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset=us-ascii > > On 22 Feb 2005, [EMAIL PROTECTED] wrote: > > > I'm trying to figure out how to subclass the list built-in. > You could do it e.g. like that: > > class Mylist (list): > def __init__(self, seq=None): > super(self.__class__, self).__init__(seq) > def __getslice__(self, start, stop): > return self.__class__(super(self.__class__, self).__getslice__(start, > stop)) > def __getitem__(self, key): > if isinstance(key, slice): > return self.__class__(super(self.__class__, > self).__getitem__(key)) > else: > return super(self.__class__, self).__getitem__(key) I've written code like this and then gotten burned. It's atractive in that you avoid hard-coding class names and it becomes simple boiler-plate that could be pasted anywhere. The problem comes when you create class Mybetterlist(Mylist): ... When a Mybetterclass instance provides the self, super(self.__class__, self) is Mylist. So you recurse into the same __getslice__ that you started with instead of getting Mylist's super class. I think (untested) that you really want return self.__class__(super(Mylist, self).() -- Lloyd Kvam Venix Corp ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
