Recommended exception for objects that can't be pickled

[Stefan Schwarzer]

Hi,

Recently, I got a request [1] to support pickling of
`FTPHost` instances in my `ftplib` library.

I explained in the ticket why I think it's a bad idea and
now want to make explicit that `FTPHost` objects can't be
pickled. The usual way to do this seems to be defining a
`__getstate__` method and raise an exception there.

Now the question is "which exception?" and my research left
me a bit confused. I didn't find a recommendation for this
on the web, not even in the Python documentation for the
`pickle` module [2]. The only hint is that the documentation
states:

"""
  The pickle module defines three exceptions:

  exception pickle.PickleError

  Common base class for the other pickling exceptions. It
  inherits Exception.

  exception pickle.PicklingError

  Error raised when an unpicklable object is encountered
  by Pickler. It inherits PickleError.

  Refer to What can be pickled and unpickled? to learn
  what kinds of objects can be pickled.

  exception pickle.UnpicklingError

  Error raised when there is a problem unpickling an
  object, such as a data corruption or a security
  violation. It inherits PickleError.

  Note that other exceptions may also be raised during
  unpickling, including (but not necessarily limited to)
  AttributeError, EOFError, ImportError, and IndexError.
"""

This sounds like unpicklable objects should raise a
`PicklingError`. Indeed, if I do this, `pickle.dumps`
gives me (my own) `PicklingError`:

  Python 3.3.2 (default, Nov  8 2013, 13:38:57)
  [GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import pickle
  >>> class X:
  ...   def __getstate__(self):
  ... raise pickle.PicklingError("can't pickle X objects")
  ...
  >>> x = X()
  >>> pickle.dumps(x)
  Traceback (most recent call last):
File "", line 1, in 
File "", line 3, in __getstate__
  _pickle.PicklingError: can't pickle X objects

What now confuses me is that most, if not all, objects from
the standard library that aren't picklable raise a
`TypeError` when I try to pickle them:

  >>> fobj = open("/etc/passwd")
  >>> pickle.dumps(fobj)
  Traceback (most recent call last):
File "", line 1, in 
  TypeError: cannot serialize '_io.TextIOWrapper' object

  >>> import socket
  >>> s = socket.socket()
  >>> pickle.dumps(s)
  Traceback (most recent call last):
File "", line 1, in 
File "/usr/lib64/python3.3/socket.py", line 116, in __getstate__
  raise TypeError("Cannot serialize socket object")
  TypeError: Cannot serialize socket object

So the documentation for the `pickle` module (to me) implies
I should raise a `PicklingError` while the standard library
usually seems to use a `TypeError`. When I grep through the
library files for `PicklingError`, I get very few hits, most
of them in `pickle.py`:

  $ find /usr/lib64/python3.3 -name "*.py" -exec grep -H PicklingError {} \;
  /usr/lib64/python3.3/site-packages/numpy/numarray/session.py:except 
(pickle.PicklingError, TypeError, SystemError):
  /usr/lib64/python3.3/pickle.py:__all__ = ["PickleError", "PicklingError", 
"UnpicklingError", "Pickler",
  /usr/lib64/python3.3/pickle.py:class PicklingError(PickleError):
  /usr/lib64/python3.3/pickle.py:raise 
PicklingError("Pickler.__init__() was not called by "
  /usr/lib64/python3.3/pickle.py:raise PicklingError("Can't 
pickle %r object: %r" %
  /usr/lib64/python3.3/pickle.py:raise PicklingError("%s must 
return string or tuple" % reduce)
  /usr/lib64/python3.3/pickle.py:raise PicklingError("Tuple 
returned by %s must have "
  /usr/lib64/python3.3/pickle.py:raise PicklingError("args from 
save_reduce() should be a tuple")
  /usr/lib64/python3.3/pickle.py:raise PicklingError("func from 
save_reduce() should be callable")
  /usr/lib64/python3.3/pickle.py:raise PicklingError(
  /usr/lib64/python3.3/pickle.py:raise PicklingError(
  /usr/lib64/python3.3/pickle.py:raise PicklingError(
  /usr/lib64/python3.3/pickle.py:raise PicklingError(
  /usr/lib64/python3.3/pickle.py:raise PicklingError(
  /usr/lib64/python3.3/idlelib/rpc.py:except pickle.PicklingError:

Which exception would you raise for an object that can't be
pickled and why?

[1] http://ftputil.sschwarzer.net/trac/ticket/75
[2] https://docs.python.org/3.4/library/pickle.html

Best regards,
Stefan
-- 
https://mail.python.org/mailman/listinfo/python-list

[ANN] ftputil 3.0a1 released

[Stefan Schwarzer]

ftputil 3.0a1 is now available from
http://ftputil.sschwarzer.net/download .

Changes since version 2.8
-

Note: This version of ftputil is _not_ backward-compatible
with earlier versions.See the links below for information
on adapting existing client code.

- This version adds Python 3 compatibility! :-)

  The same source is used for Python 2.x and Python 3.x.

  I had to change the API to find a good compromise for
  both Python versions.

- ftputil now requires at least Python 2.6.

- Remote file-like objects use the same semantics as Python's
  `io` module. (This is the same as for the built-in `open`
  function in Python 3.)

- `ftputil.ftp_error` was renamed to `ftputil.error`.

- For custom parsers, import `ftputil.parser` instead of
  `ftputil.stat`.

For more information please read
http://ftputil.sschwarzer.net/trac/wiki/WhatsNewInFtputil3.0
http://ftputil.sschwarzer.net/trac/wiki/PreReleaseDocumentation

What is ftputil?


ftputil is a high-level FTP client library for the Python programming
language. ftputil implements a virtual file system for accessing FTP
servers, that is, it can generate file-like objects for remote files.
The library supports many functions similar to those in the os,
os.path and shutil modules. ftputil has convenience functions for
conditional uploads and downloads, and handles FTP clients and servers
in different timezones.

License
---

ftputil is Open Source software, released under the revised BSD
license (see http://opensource.org/licenses/BSD-3-Clause ).

Stefan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Grammar question: Englisn and Python: qualified names

[Stefan Schwarzer]

Hi Chris,

On 2013-04-14 23:50, Chris Angelico wrote:
> Quirky question time!
> 
> When you read out a qualified name, eg collections.OrderedDict, do you
> read the qualifier ("collections dot ordered dict"), or do you elide
> it ("ordered dict")? I ask because it makes a difference to talking
> about just one of them:
> 
> ... or possibly a collections.OrderedDict...
> ... or possibly an collections.OrderedDict...
> 
> Written, the latter looks completely wrong; but if the name is read in
> its short form, with the "collections" part being implicit, then "an"
> is clearly correct! What do you think, experts and others?

I think if you _write_ "collections.OrderedDict", the
article you _write_ in front should match this. The phrase
"an collections.OrderedDict" looks odd to me, and if I read
it somewhere, it wouldn't cross my mind that the writer used
"an collections.OrderedDict" with the idea not to pronounce
"collections". ;-) In my opinion, this is too subtle.

On the other hand, when you _speak_ about the ordered dict,
use the article matching what you actually say.

Best regards,
Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: API design for Python 2 / 3 compatibility

[Stefan Schwarzer]

Terry, Ethan:

Thanks a lot for your excellent advice. :-)

On 2013-04-13 19:32, Terry Jan Reedy wrote:
> Approach 2 matches (or should match) io.open, which became
> builtin open in Python 3. I would simply document that
> ftp_host.open mimics io.open in the same way that
> ftp_host.chdir, etcetera, match os.chdir, etc. Your
> principle will remain intact.

I didn't know about `io.open` (or had forgotten it).

> Anyone writing *new* Py 2 code with any idea of ever
> running on Py 3 should be using io.open anyway. That is
> why it was backported. You might be able to reuse some io
> code or subclass some io classes for your implementation.

Since I use `socket.makefile` to create the underlying file
objects, I can use `BufferedReader`/`BufferedWriter` and
`TextIOWrapper` to supply buffering and encoding/decoding.

On 2013-04-13 20:03, Ethan Furman wrote:
> Approach 2, because it is much saner to deal with unicode
> inside the program, and only switch back to some kind of
> encoding when writing to files/pipes/etc.

Yes, this is a much saner design. I just was hesitant
because of the introduced backward incompatibility and
wanted to get other's opinions.

> Since you are going to support python 3 as well you can
> bump the major version number and note the backward
> incompatibility.

Actually I plan to increase the version number from 2.8 to
3.0 because of the Python 3 support and already intend to
change some module names that will be visible to client
code. So this is also a good opportunity to clean up the
file interface. :)

Best regards,
Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

API design for Python 2 / 3 compatibility

[Stefan Schwarzer]

Hello,

I'm currently changing the FTP client library ftputil [1]
so that the same code of the library works with Python
2 (2.6 and up) and Python 3. (At the moment the code is for
Python 2 only.) I've run into a API design issue where I
don't know which API I should offer ftputil users under
Python 2 and Python 3.

[1] http://ftputil.sschwarzer.net/

Some important background information: A key idea in ftputil
is that it uses the same APIs as the Python standard library
where possible. For example, with ftputil you can write code
like this:

  with ftputil.FTPHost(host, user, password) as ftp_host:
  # Like `os.path.isdir`, but works on the FTP server.
  if ftp_host.path.isdir("hello_dir"):
  # Like `os.chdir`, but works on the FTP server.
  ftp_host.chdir("hello_dir")
  # Like the `open` builtin, but opens a remote file.
  with ftp_host.open("new_file", "w") as fobj:
  # Like `file.write` and `file.close`
  fobj.write("Hello world!")
  fobj.close()

Since most of Python 2's and Python 3's filesystem-related
APIs accept either bytes and character strings (and return
the same type if they return anything string-like at all),
the design here is rather clear to me.

However, I have some difficulty with ftputil's counterpart
of the `open` builtin function when files are opened for
reading in text mode. Here are the approaches I've been
thinking of so far:

* Approach 1

  When opening remote text files for reading, ftputil will
  return byte strings from `read(line/s)` when run under
  Python 2 and unicode strings when run under Python 3.

Pro: Each of the Python versions has ftputil behavior
which matches the Python standard library behavior of
the respective Python version.

Con: Developers who want to use ftputil under Python 2
_and_ 3 have to program against two different APIs since
their code "inherits" ftputil's duality.

Con: Support for two different APIs will make the
ftputil code (at least a bit) more complicated than just
returning unicode strings under both Python versions.

* Approach 2

  When opening remote text files for reading, ftputil will
  always return unicode strings from `read(line/s)`,
  regardless of whether it runs under Python 2 or Python 3.

Pro: Uniform API, independent on underlying Python
version.

Pro: Supporting a single API will result in cleaner code
in ftputil than when supporting different APIs (see
above).

Con: This approach might break some code which expects
the returned strings under Python 2 to be byte strings.

Con: Developers who only use Python 2 might be confused
if ftputil returns unicode strings from `read(line/s)`
since this behavior doesn't match files opened with
`open` in Python 2.

Which approach do you recommend and why do you prefer that
approach? Are there other approaches I have overlooked? Do
you have other suggestions?

Best regards,
Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Naming future objects and their methods

[Stefan Schwarzer]

Hello,

Here's a summary of the messages which reached me in this
newsgroup and by e-mail. They are ordered by the time they
reached me at.

(According to the headers, each of the answers went to
either comp.lang.python or python-list@python.org . I
thought there was some kind of gateway, but in my reader I
only saw one answer in the newsgroup.)

Stefan Schwarzer wrote:
> I wrote a `Connection` class that can be found at [1]. A
> `Connection` object has a method `put_bytes(data)` which
> returns a "future" [2]. The data will be sent asynchronously
> by a thread attached to the connection object.
>
> The future object returned by `put_bytes` has a `was_sent`
> method which will return `True` once the sender thread has
> actually sent the data via a socket call. An example might
> look like
>
> put_result = connection.put_bytes(data)
> if put_result.was_sent(timeout=1.0):
> print "Data has been sent."
> else:
> print "Data hasn't been sent within one second."
>
> However, I'm not comfortable with the combination of the
> names of the future and its method. After all, not the
> `put_result` was sent, but the data that was the argument in
> the `put_bytes` call. Maybe `data_was_sent` is better than
> `was_sent`, but `put_result.data_was_sent()` doesn't feel
> right either.

1) Just name the future (`put_result` in my example)
   `future`. Register a callback function instead of asking
   the object returned by `put_bytes` for the result.

2) Rename `put_result` to `dataput` and `was_sent` to `sent`:

   dataput = connection.put_bytes(data)
   if dataput.sent(timeout=1.0):
   print "Data has been sent."
   else:
   print "Data hasn't been sent within one second."

   In my opinion, this gives a nice API (I'd spell `dataput`
   `data_put` though).

3) Similar to (2), but rename `dataput` to `packet`, so we get:

   packet = connection.put_bytes(data)
   if packet.sent(timeout=1.0):
   print "Data has been sent."
   else:
   print "Data hasn't been sent within one second."

4) Use variable names according to the goal/content of the
   values instead of their datatype.

   (`put_result` is somewhere in between; it's not named
   after the type `SendFuture`, but as I used a rather
   generic example, the purpose of the value isn't known,
   apart from that it was some "handle" to get the actual
   result from.)

I guess I'll go with (2). I like suggestion (3), too.
However, although in my opinion it's a nice name to read in
this context, it may put readers on the wrong track about
the kind of the data. I think `packet` reads as if this was
the content being sent or some kind of container for it.

Thank you for all the answers! :-)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Naming future objects and their methods

[Stefan Schwarzer]

Hello,

I wrote a `Connection` class that can be found at [1]. A
`Connection` object has a method `put_bytes(data)` which
returns a "future" [2]. The data will be sent asynchronously
by a thread attached to the connection object.

The future object returned by `put_bytes` has a `was_sent`
method which will return `True` once the sender thread has
actually sent the data via a socket call. An example might
look like

put_result = connection.put_bytes(data)
if put_result.was_sent(timeout=1.0):
print "Data has been sent."
else:
print "Data hasn't been sent within one second."

However, I'm not comfortable with the combination of the
names of the future and its method. After all, not the
`put_result` was sent, but the data that was the argument in
the `put_bytes` call. Maybe `data_was_sent` is better than
`was_sent`, but `put_result.data_was_sent()` doesn't feel
right either.

What do you think would be a "natural" way to name the
future returned by `put_bytes` and possibly the `was_sent`
method attached to it? Can you even come up with nice naming
rules for futures and their methods? :-)

I tried to find suggestions by using a search engine and
StackOverflow, but wasn't successful. PEP 3148 [3] describes
an API, but it's quite abstract (`Executor.submit`,
`Future.result` etc.).

[1] https://bitbucket.org/sschwarzer/connection/src/f705e612f764/connection.py
[2] http://en.wikipedia.org/wiki/Future_%28programming%29
[3] http://www.python.org/dev/peps/pep-3148/

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Gotcha's?

[Stefan Schwarzer]

Hi Miki,

On 2012-04-05 00:34, Miki Tebeka wrote:
> I'm going to give a "Python Gotcha's" talk at work.
> If you have an interesting/common "Gotcha" (warts/dark corners ...) please 
> share.
> 
> (Note that I want over http://wiki.python.org/moin/PythonWarts already).

I gave a somewhat similar talk a while ago:

http://sschwarzer.com/download/robust_python_programs_europython2010.pdf

The following is a German version of the talk slides, but
covers a bit more since there was a longer time slot. Even
if you don't know German, you'll most likely understand what
I'm talking about by reading the code. :-)

http://sschwarzer.com/download/robustere_python_programme_clt2010_print.pdf

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: using python ftp

[Stefan Schwarzer]

Hello Matt,

On 2010-12-23 01:03, Matt Funk wrote:
> i was wondering whether someone can point me whether the following
> already exists.
> 
> I want to connect to a server , download various files (for whose name i
> want to be able to use a wildcard), and store those files in a given
> location on the hard drive. If the file already exists i do not want to
> download it.
> [...]

You might want to check out ftputil:

http://ftputil.sschwarzer.net/
http://ftputil.sschwarzer.net/trac/wiki/Documentation

> Otherwise i was going to do it "by hand" using ftplib:
> 1) connect to server,
> 2) change to directory on server
> 3) get listing
> 4) match the file pattern i want to the listing
> 5) check if file already exists
> 6) download file if matched and doesn't exist
> 
> Can anyone offer any advice whether this already done somewhere?

ftputil will do most of these tasks easily. For step 4
you probably want to use Python's fnmatch module, see
http://docs.python.org/library/fnmatch.html .

If you have questions on ftputil, there's also a
mailing list:
http://ftputil.sschwarzer.net/trac/wiki/MailingList
(You need to be subscribed to the list to post, though.)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Tools for turning Python code into XMI?

[Stefan Schwarzer]

Hi Lawrence,

I missed your answer because I didn't expect someone to
respond after all this time. :-)

On 2010-10-30 04:07, Lawrence D'Oliveiro wrote:
>> I'm looking for a tool which can read Python files and write
>> a corresponding XMI file for import into UML tools.
> 
> UML ... isn’t that something more in vogue among the
> Java/DotNet corporate-code-cutter-drone crowd?

I don't know, that may well be. I still find it useful from
time to time. I don't mind using tools I find useful,
regardless who else uses them. :-)

> Specifically, you’re asking for something that can parse a
> dynamic language and generate a static structural
> description of that code. I don’t think it’s possible, in
> general.

The tools I described in my previous post go a great length
towards extracting the necessary information. But of course
it's not reliably possible to extract all information, for
example dynamically generated methods. But few Python
programs use this extensively, and a good tool will extract
most of the information interesting for me. I'm always
surprised about what Pylint finds out about my source code. :)

Given the existing open source tools, the problem rather
seems to be the effort of implementing XMI generation than
to get the information out of the Python code in the first
place.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How do I chain methods?

[Stefan Schwarzer]

Hi Steve and others,

On 2010-10-25 06:08, Steve Holden wrote:
> On 10/24/2010 11:42 PM, Chris Rebert wrote:
>> On Sun, Oct 24, 2010 at 4:11 PM, James Mills
>>  wrote:
>>> I don't agree but anyway... I've just not seen it commonly used
>>> amongst python programmers.
>>
>> If Python wanted to encourage method-chaining-style, then list.sort(),
>> list.reverse(), and several other built-in type methods would (ala
>> Ruby) return self rather than None. Since they don't, and since
>> "uncommon idiom" is a near-oxymoron, I think we can safely conclude
>> that method chaining isn't idiomatic in Python. Not that it doesn't
>> have specialized uses though (See asterisk note).
>>
> Yes, the Twisted guys use method chaining a lot - it's definitely
> idiomatic in that framework.

It's also used in the pstats module:
http://docs.python.org/library/profile.html#instant-user-s-manual

But I agree with others that it doesn't seem to be a
typical Python idiom.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: yield all entries of an iterable

[Stefan Schwarzer]

Hi Cameron,

On 2010-10-25 01:08, Cameron Simpson wrote:
> On 24Oct2010 20:58, Stefan Schwarzer  wrote:
> | On 2010-10-21 00:27, Sebastian wrote:
> | > Is there a simpler way to yield all elements of a sequence than this?
> | > for x in xs:
> | > yield x
> | 
> | Can you give an example where you would need this? Can't
> | you just iterate over the sequence?
> 
> The usual example is when the sequence comes from inside.
> Example, closely resembling some code from on of my projects:
> 
>   def leaves(N):
> if N.isleaf:
>   yield N
> for subN in N.subnodes:
>   for leaf in leaves(subN):
> yield leaf
> 
> which walks a tree structure returning leaf nodes.
> 
> The point is that you're calling leaves() on the subnode and yiled them
> directly to the outside. The caller may not even know there are "subnodes".

>From the question and the code snippet the OP gave I assumed
he meant that there already was a sequence (i. e. linear
structure) to begin with.

By the way, I think a well-known example of what you
describe is `os.walk`.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: yield all entries of an iterable

[Stefan Schwarzer]

Hi Sebastian,

On 2010-10-21 00:27, Sebastian wrote:
> Is there a simpler way to yield all elements of a sequence than this?
> for x in xs:
> yield x

Can you give an example where you would need this? Can't
you just iterate over the sequence? If you really need an
iterator, you can use `iter(sequence)`:

>>> my_list = [1, 2, 3]
>>> i = iter(my_list)
>>> i.next()
1
>>> i.next()
2
>>> i.next()
3
>>> i.next()
Traceback (most recent call last):
  File "", line 1, in 
StopIteration

This works at least with Python's built-in sequences (and
dictionaries and sets, but note that these don't have an
obvious order).

Stefan

-- 
http://mail.python.org/mailman/listinfo/python-list

Tools for turning Python code into XMI?

[Stefan Schwarzer]

Hello,

I'm looking for a tool which can read Python files and write
a corresponding XMI file for import into UML tools.

Ideally, the conversion tool should:

- be open source or freeware

- be available for Linux

- be a command line tool

- allow to specify exactly the Python files that should to
  be taken into account

- include operations, together with their parameters

- include attributes (in the UML sense) if possible (I know
  this isn't reliable)

Here are some programs I found via PyPI or search engine and
tried so far. All of them miss something I'd really like to
have.

- pyreverse (now included with Pylint):
  http://www.logilab.org/857

  I think this has the most potential. You can specify the
  files to import on the command line. There are many output
  formats for the information from the Python code, but XMI
  doesn't seem to be included. There are a few formats that
  could be parsed and turned into XMI (plain, plain-ext).
  Unfortunately, pyreverse doesn't include the parameters of
  operations.

- Umbrello:
  http://uml.sourceforge.net/

  Umbrello is a graphical KDE tool. You can specify either
  single files or whole directories including
  subdirectories. In my case, one of the subdirectories is
  for unit tests, which clutters the UML model. Attributes
  (again, in the UML sense) are seemingly not considered but
  operations. The parameters and return values of the latter,
  however, are all output as of type "string".

- Gaphor:
  http://gaphor.sourceforge.net/

  Gaphor is a GUI tool allowing to specify the files one by
  one which might be ok for a few files but surely
  becomes tedious if there are some more. The generated
  diagram, which can be exported to XMI, lacks the
  parameters of the operations.

Do you have any other suggestions for programs I should have
a look at? Does someone even has written such a tool and has
not published it yet? :)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: About __class__ of an int literal

[Stefan Schwarzer]

Hello Alex,

On 2010-09-28 11:27, AlexWalk wrote:
> In python 3.1.2(I'm using windows edition, 32bit),
> accessing __class__ of an int literal will raise a
> SyntaxException, while other literals will not. For
> example. 1.__class__ is an error, while 1.1.__class__ runs
> ok. 
> 
> I searched the python issue tracker but failed to find
> relevant reports. I wonder if  this is an unreported
> issue.

It's nothing serious; the Python parser gets a bit
confused and probably reads the dot as decimal point.
If you use (1).__class__, you get what you expected.

Stefan

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: creating python daemon ?

[Stefan Schwarzer]

Hi Daniel,

On 2010-09-23 07:30, vineet daniel wrote:
> On Sep 22, 2:20 pm, de...@web.de (Diez B. Roggisch) wrote:
>> vineet daniel  writes:
>>> On Sep 21, 9:47 pm, de...@web.de (Diez B. Roggisch) wrote:
 vineet daniel  writes:
> code that I am using is as follows :
>>
> #! /usr/bin/env python
> import sys,os
> pid = os.fork()
> os.chdir("/var/www/html/")
> os.setsid()
> os.umask(0)
> #this line will read apache log in real time as redirected from the
> CustomLog directive of Apache.
> log = sys.stdin.readlines()
> f = open('/var/www/logmongo.txt','a') # append log to this text file
> f.write(log)

I guess you'll need some loop to process the input from
Apache.

> There is a directive in Apache - CustomLog which can redirect logs to
> a script or another file by using Pipe. I googled about it and came to
> know that this works flawlessly with Perl as Apache treats Perl
> scripts as daemon, for Python to take input from Apache, it needs to
> work as daemon. Apache will then push its logs lines to this daemon as
> input. Perl takes this input as  but python's stdin is not
> working as expected it just takes the file name as given in CustomLog
> directive using Pipe i.e if I use CustomLog '|/var//script.py'

Are the _four_ ws intentional?

> combined, what script reads is just the file name, how and why I am
> not able to figure it out. And thats the reason I am here. I hope
> you've got what I am trying to do.

Do you have a line for the interpreter at the top of your
script, like

#!/usr/bin/python

Is the script executable (e. g. chmod +x script.py)?

Stefan

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: sequence multiplied by -1

[Stefan Schwarzer]

Hi Terry,

On 2010-09-25 19:24, Terry Reedy wrote:
> On 9/25/2010 4:22 AM, Yingjie Lan wrote:
> There is already a builtin reversed() function whose output can be 
> multiplied.

Seemingly, it can't:

  $ python
  Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
  [GCC 4.4.3] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> L = [1, 2, 3]
  >>> 3 * reversed(L)
  Traceback (most recent call last):
File "", line 1, in 
  TypeError: unsupported operand type(s) for *: 'int' and 'listreverseiterator'

  $ python3
  Python 3.1.2 (r312:79147, Apr 15 2010, 12:35:07)
  [GCC 4.4.3] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> L = [1, 2, 3]
  >>> 3 * reversed(L)
  Traceback (most recent call last):
File "", line 1, in 
  TypeError: unsupported operand type(s) for *: 'int' and 'list_reverseiterator'

You can convert the result of `reversed` to a list though:

  >>> 3 * list(reversed(L))
  [3, 2, 1, 3, 2, 1, 3, 2, 1]

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: sequence multiplied by -1

[Stefan Schwarzer]

Hi,

On 2010-09-25 15:54, Yingjie Lan wrote:
> The first one is simpler (4 chars v.s. 9 chars).

One thing is whether a certain form is shorter, another
thing to take into account is how often you need the
functionality.

> I thought it was also intuitive because if you multiply
> a vector by -1, you should get a vector 
> in the reversed direction. But, intuitiveness depends
> on who you are, what you do, etc

Sure ... in math, multiplying a vector by -1 gives a vector
with all its elements negated:

-1 * [1, 2, 3] -> [-1, -2, -3]

:-)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: sequence multiplied by -1

[Stefan Schwarzer]

Hi,

On 2010-09-25 14:11, Yingjie Lan wrote:
> Having more than one way of doing things sometimes is good.

In my opinion this _isn't_ a situation where it's good. :)

L[::-1]

is only marginally longer than

-1 * L

I think this small gain doesn't justify "violating" this
"Python Zen" rule (from `import this`):

There should be one-- and preferably only one --obvious way to do it.

One could argue that using L[::-1] isn't "obvious" but I
believe the rule refers to being obvious to people who have
used Python for a while. Besides that, multiplying a list
with a negative value to give a reverse list isn't so
intuitive either. :-)

On 2010-09-25 13:45, Thomas Jollans wrote:
> Multiplying a list by a negative integer should produce a list of negative 
> length, which does not exist. IMHO, the only correct behaviour would be to 
> raise an exception, though one could argue that there are practical benefits 
> for the operation to succeed for any integer operand.

I agree with raising an exception, probably a ValueError.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Design: Module interface vs. library interface

[Stefan Schwarzer]

Hello,

I maintain ftputil [1], an FTP client library.

Users of the library may use it like this:

| import ftputil
|
| with ftputil.FTPHost(server, userid, password) as ftp_host:
| # for example, make a remote file and read its content
| with ftp_host.open("remote_name", "rb") as remote_fobj:
| data = remote_fobj.read()

Even though "ftp files" are defined in module `ftp_file` [2]
via class `FTPFile`, users of the ftputil library should
never use `FTPFile` directly; it's up to `FTPHost.open` to
generate these file objects.

Now, from the "point of view" of the _module_ `ftp_file`,
I'd use

| # ftp_file.py
|
| # used outside this module
| __all__ = ['FTPFile']
|
| class FTPFile(object):
| ...

| # ftputil.py
|
| import ftp_file
|
| class FTPHost(object):
| ...
| def open(name, mode):
| ...
| return ftp_file.FTPFile(...)

On the other hand, from the point of view of the _library_
ftputil, `FTPFile` isn't part of the official API, so I may
as well write

| # ftp_file.py
|
| # be explicit, no user-public interface
| __all__ = []
|
| class FTPFile(object):
| ...

Which approach would you prefer and why? Or some other
approach? Would you use a different approach if the
library-internal class was named `_FTPFile` instead of
`FTPFile`?

[1] http://ftputil.sschwarzer.net
[2] http://ftputil.sschwarzer.net/trac/browser/ftp_file.py

Stefan

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: bool constructor is inconsistent?

[Stefan Schwarzer]

Hi Neal,

On 2010-09-10 20:23, Neal Becker wrote:
> IN [3]: bool('False')
> Out[3]: True

If you consider strings, only an empty string has a false
value. So the string 'False' which is non-empty, results in
a true boolean value.

For example, you can use

if my_string:
...

to execute some code if the string is not empty.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help needed - function apparently global cannot be called.

[Stefan Schwarzer]

Hi Ian,

On 2010-09-07 12:18, Ian Hobson wrote:
>  f = open('d:\logfile.txt','a')

Just a note: Using a backslash in a non-raw string will get
you in trouble as soon as the backslash is followed by a
character which makes a special character sequence, like "\n".
For example,

f = open('d:\nice_filename.txt', 'a')

will give surprising results. :-) Either double the
backslash, use a raw string, or, in the special case of
file system paths, possibly use a forward slash.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: doctest annoyance/puzzle

[Stefan Schwarzer]

Hi Steven,

On 2010-09-05 17:30, Steven D'Aprano wrote:
> I run the doctests with:
> 
> python2.6 -m doctest examples.txt
> 
> and the first example passes, but the second fails with NameError: 
> make_spam not defined.

I run my doctests by calling

doctest.testfile(filename)

for each file in a loop. This way, names and their
associated objects survice from one code block to the next.

I just read that the way you use doctest should behave the
same, according to the documentation. In case of a text file
I just tested, it does; all tests pass despite the text
snippets between the code blocks.

What do you get if you test your text file by explicitly
calling doctest.testfile?

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a Python equivalent to Perl's File::Listing::parse_dir

[Stefan Schwarzer]

Hi John,

On 2010-08-11 20:24, John Nagle wrote:
>Perl has a function which will take a remote directory page, in
> the form that most web sites return for a file directory, and
> parse it into a useful form:
> 
>   http://www.xav.com/perl/site/lib/File/Listing.html
> 
> This is especially useful for FTP sites.
> 
> Is there a Python equivalent of this?  I'm converting some
> old Perl code.
> 
> Even the Python FTP module doesn't have a directory parser.

I saw this posting only now. I hope it's not too late to
point you to ftputil, http://ftputil.sschwarzer.net . :-)
As the name implies, it's FTP-only for now, though.

If you have any questions regarding the library, please ask.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Saving (unusual) linux filenames

[Stefan Schwarzer]

Hi Grant,

On 2010-08-31 20:49, Grant Edwards wrote:
> How many filenames contain ","?

CVS repository files end with ,v . However, just let's agree
that nobody uses CVS anymore. :-)

> Not many, but the OP wants his
> program to be bulletproof.  Can't fault him for that.

What about using the csv (not CVS) module?

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Functions continuing to ru after returning something?

[Stefan Schwarzer]

Hi,

On 2010-08-31 02:05, Bradley Hintze wrote:
> I may be having a brain fart, but is it at all possible to have a
> function first return a value then continue its calculation. Like this
> simple example:
> 
> my_var = 5
> def my_function():
> return my_var
> my_var +=1
> 
> This obviously won't work as written but is there a cleaver way around this.

At least in CPython 2.6.5 the above code results in

Traceback (most recent call last):
  File "test.py", line 7, in 
my_function()
  File "test.py", line 3, in my_function
return my_var
UnboundLocalError: local variable 'my_var' referenced before assignment

as soon as the function is called.

If you want to have the global my_var modified, you need
a "global my_var" statement in the function body.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Confused: Newbie Function Calls

[Stefan Schwarzer]

Hi Pinku,

On 2010-08-11 21:35, Pinku Surana wrote:
> Even though I used the same name "x" for a local and global variable,
> they are actually completely different. When I call "fun(x)" it COPIES
> the global value of "x" into the local variable "x" in "fun". [...]

The global value isn't copied when calling the function.
Instead, the global and the local name both point to the
same object when the body of the function starts to run.

> def fun(x_local):
> y_local = 1
> x_local += y_local
> print x_local

Only after the assignment "x_local += y_local" x_local
points to a new object which is the result of the addition
of the previously "shared" object and y_local.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ftplib limitations?

[Stefan Schwarzer]

Hi Lawrence,

On 2010-08-28 01:49, Lawrence D'Oliveiro wrote:
>> Now it may be that the data connection, after having started
>> the transfer, works as it should, but the control connection
>> times out because the duration of the transfer is too long.
> 
> It might not be the fault of the FTP server. If you’re going through a 
> router doing NAT, that could be where the timeout is happening.

Good point, thanks! That may explain why it's a low-level
socket error instead of a 4xx timeout message from the
server which I would have expected.

If it's the router, the OP might try to change their router
settings to get rid of the problem.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Writing byte stream as jpeg format to disk

[Stefan Schwarzer]

Hi Navkirat,

On 2010-08-26 19:22, Navkirat Singh wrote:
> I am programming a webserver, I receive a jpeg file with
> the POST method.The file (.jpeg) is encoded in bytes, I
> parse the bytes by decoding them to a string. I wanted to
> know how i could write the file (now a string) as a jpeg
> image on disk. When I try to encode the same string to a
> bytes and write them in binary format to disk, the file is
> not recognized as jpeg. I would be grateful if someone
> could help me with this.

I guess you mean you "see" a byte string in your server and
want to write that to disk. Assuming the string you got is
the correct image data in the first place, you can, in
Python 2.x, write the string data to disk like this:

fobj = open("some_image.jpg", "wb")
fobj.write(byte_string)
fobj.close()

Note that you should use "wb" as mode to write as binary.
Otherwise you'll get automatic line ending conversion (at
least on Windows) which will give the result you describe.

If my answer doesn't help, you probably need to describe in
more detail what you're doing, including showing some real
code.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ftplib limitations?

[Stefan Schwarzer]

Hi durumdara,

On 2010-08-25 11:18, durumdara wrote:
> On aug. 25, 08:07, Stefan Schwarzer  wrote:
>>
>> The file is 2 GB in size and is fully transferred, without
>> blocking or an error message. The status message from the
>> server is '226-File successfully transferred\n226 31.760
>> seconds (measured here), 64.48 Mbytes per second', so this
>> looks ok, too.
>>
>> I think your problem is related to the FTP server or its
>> configuration.
>>
>> Have you been able to reproduce the problem?
> 
> Yes. I tried with saving the file, but I also got this error.
> but: Total COmmander CAN download the file, and ncftpget also can
> download it without problem...

I suppose they do the same as in my former suggestion:
"catching" the error and ignoring it. ;-)

After all, if I understood you correctly, you get the
complete file contents, so with ftplib the download
succeeds as well (in a way).

You might want to do something like (untested):

import os
import socket

import ftputil

def my_download(host, filename):
"""Some intelligent docstring."""
# Need timestamp to check if we actually have a new
#  file after the attempted download
try:
old_mtime = os.path.getmtime(filename)
except OSError:
old_mtime = 0.0
try:
host.download(filename, filename, 'b')
except socket.error:
is_rewritten = (os.path.getmtime(filename) != old_mtime)
# If you're sure that suffices as a test
is_complete = (host.path.getsize(filename) ==
   os.path.getsize(filename))
if is_rewritten and is_complete:
# Transfer presumably successful, ignore error
pass
else:
# Something else went wrong
raise

def main():
host = ftputil.FTPHost(...)
my_download(host, "large_file")
host.close()

If you don't want to use an external library, you can use
`ftplib.FTP`'s `retrbinary` and check the file size with
`ftplib.FTP.size`. This size command requires support for
the SIZE command on the server, whereas ftputil parses the
remote directory listing to extract the size and so doesn't
depend on SIZE support.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ftplib limitations?

[Stefan Schwarzer]

Hi durumdara,

On 2010-08-25 09:43, durumdara wrote:
>> I can imagine the error message (a full traceback if
>> possible) would help to say a bit more about the cause of
>> the problem and maybe what to do about it.
> 
> This was:
> 
> Filename: "Repositories 20100824_101805 (Teljes).zip" Size: 1530296127
> ..download: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
> Traceback (most recent call last):
>   File "C:\D\LocalBackup\ftpdown.py", line 31, in 
> ftp.retrbinary("retr " + s, CallBack)
>   File "C:\Python26\lib\ftplib.py", line 401, in retrbinary
> return self.voidresp()
>   File "C:\Python26\lib\ftplib.py", line 223, in voidresp
> resp = self.getresp()
>   File "C:\Python26\lib\ftplib.py", line 209, in getresp
> resp = self.getmultiline()
>   File "C:\Python26\lib\ftplib.py", line 195, in getmultiline
> line = self.getline()
>   File "C:\Python26\lib\ftplib.py", line 182, in getline
> line = self.file.readline()
>   File "C:\Python26\lib\socket.py", line 406, in readline
> data = self._sock.recv(self._rbufsize)
> socket.error: [Errno 10054] A lÚtez§ kapcsolatot a tßvoli ßllomßs
> kÚnyszerÝtette
> n bezßrta
> 
> So this message is meaning that the remote station forced close the
> existing connection.

The file transfer protocol uses two connections for data
transfers, a control connection to send commands and
responses, and a data connection for the data payload
itself.

Now it may be that the data connection, after having started
the transfer, works as it should, but the control connection
times out because the duration of the transfer is too long.
A hint at this is that the traceback above contains
`getline` and `readline` calls which strongly suggest that
this socket was involved in some text transfer (presumably
for a status message).

Most FTP servers are configured for a timeout of 5 or 10
minutes. If you find that the file transfers don't fail
reproducably for a certain size limit, it's probably not the
size of the file that causes the problem but some timing
issue (see above).

What to do about it? One approach is to try to get the
timeout value increased. Of course that depends on the
relation between you and the party running the server.
Another approach is to catch the exception and ignore it.
To make sure you only ignore timeout messages, you may want
to check the status code at the start of the error message
and re-raise the exception if it's not the status expected
for a timeout. Something along the lines of:

try:
# transer involving `retrbinary`
except socket.error, exc:
if str(exc).startswith("[Errno 10054] "):
pass
else:
raise

Note, however, that this is a rather brittle way to handle
the problem, as the status code or format of the error
message may depend on the platform your program runs on,
library versions, etc.

In any case you should close and re-open the FTP connection
after you got the error from the server.

> Now I'm trying with saving the file into temporary file, not hold in
> memory.

If my theory holds, that shouldn't make a difference. But
maybe my theory is wrong. :)

Could you do me a favor and try your download with ftputil
[1]? The code should be something like:

import ftputil

host = ftputil.FTPHost(server, userid, passwd)
for name in host.listdir(host.curdir):
host.download(name, name, 'b')
host.close()

There's neither a need nor - at the moment - a possibility
to specify a callback if you just want the download. (I'm
working on the callback support though!)

For finding the error, it's of course better to just use the
download command for the file that troubles you.

I'm the maintainer of ftputil and if you get the same or
similar error here, I may find a workaround for ftputil. As
it happens, someone reported a similar problem (_if_ it's
the same problem in your case) just a few days ago. [2]

[1] http://ftputil.sschwarzer.net
[2] http://www.mail-archive.com/ftpu...@codespeak.net/msg00141.html

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ftplib limitations?

[Stefan Schwarzer]

Hi durumdara,

On 2010-08-24 16:29, Stefan Schwarzer wrote:
>> I experienced some problem.
>> The server is Windows and FileZilla, the client is Win7 and Python2.6.
>> When I got a file with size 1 303 318 662 byte, python is halt on
>> "retrbinary" line everytime.
> 
> So if I understand correctly, the script works well on
> smaller files but not on the large one?

I just did an experiment in the interpreter which
corresponds to this script:

import ftplib

of = open("large_file", "wb")

def callback(data):
of.write(data)

ftp = ftplib.FTP("localhost", userid, passwd)
ftp.retrbinary("RETR large_file", callback)

of.close()
ftp.close()

The file is 2 GB in size and is fully transferred, without
blocking or an error message. The status message from the
server is '226-File successfully transferred\n226 31.760
seconds (measured here), 64.48 Mbytes per second', so this
looks ok, too.

I think your problem is related to the FTP server or its
configuration.

Have you been able to reproduce the problem?

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: equivalent of source in python?

[Stefan Schwarzer]

Hi Astan,

On 2010-08-24 21:18, Astan Chee wrote:
> I'm trying to convert my tcsh script to python and am stuck at one part, 
> particularly the part of the script that looks like this:
> 
> #!/bin/tcsh
> setenv LSFLOG /var/tmp/lsf_log
> source /etc/setup
> unalias cp
> umask 0
> env >> ${AFLOG}
> 
> What is the equivalent of doing this in python2.5?

I guess it doesn't make sense to "translate" this to Python
line by line. For example, you can't "source" shell code
into a Python program. The above commands don't look as if
they were the purpose of the program, but rather mostly some
preparation/setup before solving the actual problem.

Maybe it's more helpful to tell us what you want to achieve
in the end and we might be able to make suggestions on that.

How long is the shell script and what's your Python code so
far?

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ftplib limitations?

[Stefan Schwarzer]

Hi durumdara,

On 2010-08-24 11:21, durumdara wrote:
> def CallBack(Data):
> d['size'] = d['size'] + len(Data)
> d['buffer'].append(Data)
> percent = (d['size'] / float(fsize)) * 100
> percentp10 = int(percent/10)
> if percentp10 > d['lastpercentp10']:
> d['lastpercentp10'] = percentp10
> print str(percentp10 * 10) + "%",
> 
> ftp.retrbinary("retr " + s, CallBack)
> print ""
> print "..downloaded, joining"
> dbuffer = "".join(d['buffer'])
> [...]
> This code is login into a site, download and delete all files.
> 
> I experienced some problem.
> The server is Windows and FileZilla, the client is Win7 and Python2.6.
> When I got a file with size 1 303 318 662 byte, python is halt on
> "retrbinary" line everytime.

So if I understand correctly, the script works well on
smaller files but not on the large one?

> It down all of the file (100%) but the next line never reached.

_Which_ line is never reached? The `print` statement after
the `retrbinary` call?

> Some error I got, but this was in yesterday, I don't remember the text
> of the error.

Can't you reproduce the error by executing the script once
more? Can you copy the file to another server and see if the
problem shows up there, too?

I can imagine the error message (a full traceback if
possible) would help to say a bit more about the cause of
the problem and maybe what to do about it.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Open a command pipe for reading

[Stefan Schwarzer]

Hi Rodrick,

On 2010-08-17 18:40, Rodrick Brown wrote:
> I have a fairly large file 1-2GB in size that I need to
> process line by line but I first need to convert the file
> to text using a 3rd party tool that prints the records
> also line by line.
> 
> I've tried using Popen to do this with no luck. I'm trying
> to simulate 
> 
> /bin/foo myfile.dat 

Is foo the 3rd-party conversion tool you've mentioned?

It would be good to have a bit more context, e. g. an actual
snippet from your code.

> And as the records are being printed do some calculations. 
> 
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout 
> 
> for data in pipe.readlines():
> print data,
> 
> This operation blocks forever I'm guessing it's trying to
> process the entire file at once. 

If you use `readlines` on a file object it reads the whole
file at once and returns a list of the lines (including line
end characters, by the way). What you probably want is

for line in pipe:
print line,

which reads and prints the file contents line by line.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A question to experienced Pythoneers

[Stefan Schwarzer]

Hi Rony,

On 2010-08-20 10:16, Rony wrote:
> Here's the story :
> I've been hired by a company as a consultant to reorganise there
> development department.
> The actual situation is :
> The manager of the development has been fired, main reason (what they
> told me) is that they have big big troubles in keeping deadlines ! For
> there last product, for which they estimated 3 man years of
> development they had 9 months extra effort, and the product was
> delivered a year to late.

I guess the most important issue will be to find out what
was wrong with the development process and fix the problems,
i. e. don't repeat them. Keep in mind that "what was wrong"
can mean any number of reasons.

Until you haven't dealt with this, thinking about changing
the programming language in my opinion makes little sense.

> I basicaly have 'carte blanche' :)

Um, yes ... with the corresponding responsibilities and
risks. :-)

> So, my strategie would be :
> - Use Python and wxpython for the GUI
> - Develop critical routines in Pyrex or in C and make PYD's from it
> which will be imported by the main Python programm.
> - Distribute with py2exe on Win & py2app for Mac.

You plan to change a process (and people) used to developing
with C to developing with Python and the associated tools.
For a developer experienced with both C and Python, using
the latter usually will result in faster development. On the
other hand, switching to a tool you (i. e. an individual
developer) never used before adds some risk (-> uncertainty
in effort estimation).

If the project you're dealing with now is based on the
project you mentioned above, you better not start from
scratch in a different language. Very likely the existing
code will have subtle though important fixes for subtle
problems which were encountered during development. If you
start from scratch you risk losing these fixes and having to
re-discover the problems and fixes which can take a lot of
time. Even if you have all programmers of the old team
available, it's risky because they might have forgotten the
parts of the code or they may no longer be around when
you're about to implement the functionality of these
critical parts of the code.

If you haven't already, I recommend to read these books:

Steve McConnell
Rapid Development
http://www.amazon.com/x/dp/0072850604/

Tom DeMarco, Timothy Lister
Waltzing With Bears: Managing Risk on Software Projects
http://www.amazon.com/x/dp/0932633609/

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A question to experienced Pythoneers

[Stefan Schwarzer]

Hi Lawrence,

On 2010-08-20 13:11, Lawrence D'Oliveiro wrote:
> In message
> <8d1b76b7-1ba3-49c5-97cf-dc3837050...@y11g2000yqm.googlegroups.com>, Rony
> wrote:
>
>> The manager of the development has been fired, main reason (what they
>> told me) is that they have big big troubles in keeping deadlines ! For
>> there last product, for which they estimated 3 man years of
>> development they had 9 months extra effort, and the product was
>> delivered a year to late.
>
> 

I can't get the page; all I get is an error message:

"""
Not Found

The requested message, i4ku71$fd...@lust.ihug.co.nz, could not be found.
"""

Can you please give a part of the message to search for or
even better, an URL that works? :-)

Thanks,
Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: String substitution VS proper mysql escaping

[Stefan Schwarzer]

Hi Νίκος,

On 2010-08-19 09:10, Νίκος wrote:
> On 18 Αύγ, 12:50, Cameron Simpson  wrote:
>>
>> ("nikos",) is a single element tuple.
>> ["nikos"] is a single element list.
>> ["nikos",] is also a single element list, just written like the tuple.
> 
> It makes more sense if i:
> 
> "nikos" is just a string
> ("nikos") is a single element tuple
> ["nikos"] is also a single element list
> 
> After all () used to define tuples and [] usedd to define lists. Why
> commas?

Because parentheses are also used to group expressions.
For example,

>>> 2 * (1+2)
6

If it were as you would like, the result would have been

(3, 3)

So because  inside parentheses
already denotes an expression you have to add a comma to
distinguish a one-element tuple from an expression.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 79 chars or more?

[Stefan Schwarzer]

Hi Lie,

On 2010-08-18 12:02, Lie Ryan wrote:
> On 08/17/10 12:59, AK wrote:
>> On 08/16/2010 10:42 PM, James Mills wrote:
>>> My personal opinion (despite monitors being wider) is
>>> the horizontal scrolling isn't worth it. Stick to a 80-char width.
>>
>> But.. why horizontal scrolling, isn't autowrap much better than that?
> 
> Do you seriously use autowrapper when writing code?

I think he means wrapping on the screen, not actually
inserting line breaks.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Call Java Code from python

[Stefan Schwarzer]

Hi Bidda,

On 2010-08-18 09:19, Bidda Gowda wrote:
> I have a project coming up where i have to integrate our existing
> Python based web application with Java Programs. Basically i should be
> able to call Java programs which comes in the form of jars. Whats the
> best way to call these jars from python ?
> 
> I looked at jpype and tried with small program and it works. Are there
> any other tools which can do the same or if any how are they compared
> to jpype ?

Here are some slides from a talk by Andreas Schreiber,
"Mixing Python and Java", which might help:
http://www.slideshare.net/onyame/mixing-python-and-java

Personally, I've used JCC for accessing Lucene (a search
engine framework implemented in Java). It was a bit
rough in some places, but overall quite usable.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 79 chars or more?

[Stefan Schwarzer]

Hi Andrei,

On 2010-08-17 18:43, AK wrote:
> But let me ask you, would you really prefer to have:
> 
 self.expiration_date = translate_date(
   find(response, 'MPNExpirationDate').text,
   '%Y-%m-%d', '%m%d%Y')
> 
> (or the 4-line version of this above), even when it necessitates
> creation of a new function, rather than have this code on two lines?

Given that the reformatted code is three lines and the
former code two lines, I probably wouldn't change anything
but the formatting as shown. :)

> After all, I think it's a matter of balance between readability,
> expressiveness and succinctness. If I split a function in two, that
> still means that understanding the functionality of the code will
> require scrolling around and looking at the second function. I guess
> what I'm trying to say that we shouldn't just care about readability of
> lines but also readability of functions and blocks of functionality
> (that may include several functions that accomplish a single "task".)

I think you're right here; you should keep the overall
readability or (maintainability on the whole) in mind.

I agree with Neil that good refactoring can _improve_ the
understandability of the code, and it won't necessarily
require you to look up the code of the extracted
function or method.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 79 chars or more?

[Stefan Schwarzer]

On 2010-08-17 17:44, AK wrote:
> On 08/17/2010 10:28 AM, Stefan Schwarzer wrote:
>> I'd probably reformat this to
>>
>>self.expiration_date = translate_date(
>>  find(response, 'MPNExpirationDate').text,
>>  '%Y-%m-%d', '%m%d%Y')
>>
>> or even
>>
>>self.expiration_date = translate_date(
>>  find(response, 'MPNExpirationDate').text,
>>  '%Y-%m-%d',
>>  '%m%d%Y')
>>
>> for consistency.
>>
>> This not only limits the width but also makes the nesting of
>> the calls more visible.
> 
> Doesn't this create the problem of functions growing too long to fit in
> a screen? I think it's very useful to try to keep function size low
> enough so that you can view the whole function without having to scroll
> up and down. (even though that's not always possible) -ak

I think I'd extract some part of the function into a new
function then. In my opinion, the reasoning is similar to
the case, "Can't I use two spaces per indentation level?
That way I don't run so easily into the right margin if I
have more than five indentations in a function." ;-)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 79 chars or more?

[Stefan Schwarzer]

Hi Neil,

On 2010-08-17 14:42, Neil Cerutti wrote:
> On 2010-08-17, Michael Torrie  wrote:
>> In general if I find myself consistently going longer than 75
>> or 80 characters, I need to refactor my code to make it more
>> manageable.  If I have to scroll up five pages to find the
>> beginning of a block, that normally means my code could be
>> simplified and improved.
> 
> Looking through my code, the split-up lines almost always include
> string literals or elimination of meaningless temporary
> variables, e.g.:
> 
> self.expiration_date = translate_date(find(response,
> 'MPNExpirationDate').text, '%Y-%m-%d', '%m%d%Y')

I'd probably reformat this to

  self.expiration_date = translate_date(
find(response, 'MPNExpirationDate').text,
'%Y-%m-%d', '%m%d%Y')

or even

  self.expiration_date = translate_date(
find(response, 'MPNExpirationDate').text,
'%Y-%m-%d',
'%m%d%Y')

for consistency.

This not only limits the width but also makes the nesting of
the calls more visible.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Opposite of split

[Stefan Schwarzer]

Hi Alex,

On 2010-08-16 18:44, Alex van der Spek wrote:
> Anybody catches any other ways to improve my program (attached), you are 
> most welcome. Help me learn, that is one of the objectives of this 
> newsgroup, right? Or is it all about exchanging the next to impossible 
> solution to the never to happen unreal world problems?

I don't know what a concordance table is, and I haven't
looked a lot into your program, but anyway here are some
things I noticed at a glance:

| #! usr/bin/env python
| # Merge log files to autolog file
| import os
| import fileinput
| #top='C:\\Documents and Settings\\avanderspek\\My 
Documents\\CiDRAdata\\Syncrude\\CSL\\August2010'
| top='C:\\Users\\ZDoor\\Documents\\CiDRA\\Syncrude\CSL\\August2010'

If you have backslashes in strings, you might want to use
"raw strings". Instead of "c:\\Users\\ZDoor" you'd write
r"c:\Users\ZDoor" (notice the r in front of the string).

| i,j,k=0,0,0
| date={}

I suggest to use more spacing to make the code more
readable. Have a look at

http://www.python.org/dev/peps/pep-0008/

for more formatting (and other) tips.

| fps=0.3048
| tab='\t'
|
| bt='-999.25'+'\t''-999.25'+'\t''-999.25'+'\t''-999.25'+'\t'+'-999.25'

If these numbers are always the same, you should use
something like

NUMBER = "-999.25"
COLUMNS = 5
bt = "\t".join(COLUMNS * [NUMBER])

(with better naming, of course).

Why don't you use `tab` here?

I _highly_ recommend to use longer (unabbreviated) names.

| al='Status'+'\t'+'State'+'\t'+'-999.25'
|
| for root,dirs,files in os.walk(top):
| #Build a concordance table of days on which data was collected
| for name in files:
| ext=name.split('.',1)[1]

There's a function `splitext` in `os.path`.

| if ext=='txt':
| dat=name.split('_')[1].split('y')[1]
| if dat in date.keys():

You can just write `if dat in date` (in Python versions >=
2.2, I think).

| date[dat]+=1
| else:
| date[dat]=1
| print 'Concordance table of days:'
| print date
| print 'List of files processed:'
| #Build a list of same day filenames, 5 max for a profile meter,skip first 
and last days
| for f in sorted(date.keys())[2:-1]:
| logs=[]
| for name in files:
| ext=name.split('.')[1]
| if ext=='txt':
| dat=name.split('_')[1].split('y')[1]

I guess I'd move the parsing stuff (`x.split(s)[i]` etc.)
into small functions with meaningful names. After that I'd
probably notice there's much redundancy and refactor them. ;)

| if dat==f:
| logs.append(os.path.join(root,name))
| #Open the files and read line by line
| datsec=False
| lines=[[] for i in range(5)]

One thing to watch out for: The above is different from
`[[]] * 5` which uses the _same_ empty list for all entries.
Probably the semantics you chose is correct.

| fn=0
| for line in fileinput.input(logs):
| if line.split()[0]=='DataID':
| datsec=True
| ln=0
| if datsec:
| lines[fn].append(line.split())
| ln+=1
| if ln==10255:

This looks like a "magic number" and should be turned into a
constant.

| datsec=False
| fileinput.nextfile()
| fn+=1
| print fileinput.filename().rsplit('\\',1)[1]
| fileinput.close()
| aut='000_AutoLog'+f+'.log'
| out=os.path.join(root,aut)
| alf=open(out,'w')
| alf.write('Timestamp (mm/dd/ hh:mm:ss)VF 1VF 2VF 3
VF 4VF 5Q 1 Q 2 Q 3 Q 4 Q 5 Status  State   Metric  
Band Temperature 1  Band Temperature 2  Band Temperature 3  Band 
Temperature 4  Band Temperature 5  SPL 1   SPL 2   SPL 3   SPL 4   SPL 
5'+'\n')
| for wn in range(1,10255,1):

You don't need to write the step argument if it's 1.

| for i in range(5):
| lines[i][wn][2]=str(float(lines[i][wn][2])/fps)
| tp=lines[0][wn][0]+' '+lines[0][wn][1]
| vf=tab.join([lines[i][wn][2] for i in range(5)])
| vq=tab.join([lines[i][wn][3] for i in range(5)])
| vs=tab.join([lines[i][wn][4] for i in range(5)])
| #sf=tab.join([lines[i][wn][5] for i in range(5)])
| #sq=tab.join([lines[i][wn][6] for i in range(5)])
| #ss=tab.join([lines[i][wn][7] for i in range(5)])

Maybe use an extra function?

def choose_a_better_name():
return tab.join([lines[index][wn][2] for index in range(5)])

Moreover, the repetition of this line looks as if you wanted
to put the right hand sides of the assignments in a list,
instead of assigning to distinct names (`vf` etc.).

By the way, you use the number 5 a lot. I guess this should
be a constant, too.

| alf.write(tp+'\t'+vf+'\t'+vq+'\t'+al+'\t'+bt+'\t'+vs+'\n')

Re: Help to convert Number to String

[Stefan Schwarzer]

Hi Vamsi,

On 2010-08-13 22:50, Vamsi wrote:
> I am trying to count the number of lines in a file and insert  into
> the file but getting the error message "TypeError: must be string or
> read-only character buffer, not int", Could you please help me how to
> correct this?

Which Python version do you use?

For which statement exactly do you get the message?

> here is the code
> 
> lines1 = sum(1 for line in open('C:/test1.txt'))
> wfile = open('C:/test1.txt'', 'a')

You have two quotes here after the filename. These give a
SyntaxError here. Python sees two concatenated strings,
"C:/test1.txt" and ", " and stumbles over the a immediately
after the closing quote of the second string.

> wfile.write(str(lines1).zfill(9))
> wfile.close()

If I remove the redundant quote character and substitute
filenames appropriate for my system, everything works.

Maybe the code you included in this post isn't the same
which led to the error?

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How do I get number of files in a particular directory.

[Stefan Schwarzer]

On 2010-08-13 11:18, blur959 wrote:
> import os
> 
> directory = raw_input("Please input file directory. \n\n")
> s = raw_input("Please input a name to replace. \n\n")
> ext = raw_input("input file ext")
> 
> for files in os.listdir(directory):
> if ext in files:
> file_number = len(files)
> print file_number
> 
> 
> The result is:
> 13
> 13
> 13
> 6
> 15
> 8
> 10
> 10
> 8
> 7
> 5
> 
> where the result should be just 11. Can anyone help me? Thanks.

`os.listdir` returns a list of names. What you named
`files` should actually be `filename`. What you got
printed in the loop are the lengths of each filename.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: __class__ of what

[Stefan Schwarzer]

Hello Jean-Michel,

On 2010-08-12 16:06, Jean-Michel Pichavant wrote:
> Eric J. Van der Velden wrote:
> Should be
> 
> class C:
> n = 0
> def __init__(self):
>self.__class__.n+=1
>C.n+=1 # equivalent to this line (I prefer this one, more 
> readable, less refactor-friendly)

I think both lines have slightly different semantics if you
consider instantiating an object of a derived class. (The
following formatting is from IPython, thus the differences
to the "usual" interpreter output.)

Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)

>>> class C(object):
... n = 0
... def __init__(self):
... self.__class__.n += 1
...
>>> class D(C):
... pass
...
>>> d = D()
>>> D.n
1
>>> C.n
0

Here, the augmented assignment looks up self.__class__.n
which it doesn't find and so gets it from class C. The
actual assignment assigns to D.n, however.

On the other hand:

>>> class C(object):
... n = 0
... def __init__(self):
... C.n += 1
...
>>> class D(C):
... pass
...
>>> d = D()
>>> D.n
1
>>> C.n
1

Here, only C is changed. D doesn't get an own attribute n,
and after the instantiation of d, D.n looks up and gets n
from the base class C.

Curiously,

>>> dir(D)
['__class__',
 '__delattr__',
 '__dict__',
 '__doc__',
 '__format__',
 '__getattribute__',
 '__hash__',
 '__init__',
 '__module__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__weakref__',
 'n']

seems to contain n (is it supposed to be listed here?), but

>>> dict(D.__dict__)
{'__doc__': None, '__module__': '__main__'}

indicates that D doesn't contain n. But C does:

>>> dict(C.__dict__)
{'__dict__': ,
 '__doc__': None,
 '__init__': ,
 '__module__': '__main__',
 '__weakref__': ,
 'n': 1}

I hope everything I wrote above is valid. If not, please
correct me. :-)

Anyway, depending on what you want, either of the two
variants might be ok. In any case I'd include a meaningful
comment on why you actually wrote what you wrote. :-)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Weird Python behaviour

[Stefan Schwarzer]

Hi,

On 2010-08-10 17:01, Francesco Bochicchio wrote:
> There used to be a very nice (also graphic) explanationor this
> somewhere on the web, but my googling skills failed me this time,
> so instead I'll show you the concept using your own code:

Probably this isn't the page you're referring to, but I only
recently gave a beginners' talk at EuroPython:

http://sschwarzer.com/download/robust_python_programs_europython2010.pdf

The topic of identity and assignments starts on slide 7,
nice graphics start on slide 10. ;-)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python -Vs- Ruby: A regexp match to the death!

[Stefan Schwarzer]

On 2010-08-09 23:43, Stefan Schwarzer wrote:
> I got that traceback as soon as I typed in "%paste" and
> pressed enter, without pasting anything in the terminal.
> I had assumed it works like :paste in Vim, activating a

I meant ":set paste" of course.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python -Vs- Ruby: A regexp match to the death!

[Stefan Schwarzer]

Hi Robert,

On 2010-08-09 22:23, Robert Kern wrote:
> On 2010-08-09 06:42 , Stefan Schwarzer wrote:
>> Unfortunatey, when I enter
>>
>>In [2]: %paste
>>
>> at the prompt it gives me (before I pasted anything)
>>
>>In [2]: %paste
>>
>>   File "", line 1
>> http://pypi.python.org/pypi/ipython/0.10
>> ^
>>SyntaxError: invalid syntax
> 
> Yes, that's because you had that URL in your clipboard, not Python code. What 
> were you expecting to happen?

I got that traceback as soon as I typed in "%paste" and
pressed enter, without pasting anything in the terminal.
I had assumed it works like :paste in Vim, activating a
kind of "paste mode" where everything pasted into the
terminal is modified as the help text suggests.

Ok, I just noticed I should have actually _read_ the
help text, not just scanned it. ;-) Sorry for the
confusion.

Stefan



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python -Vs- Ruby: A regexp match to the death!

[Stefan Schwarzer]

Hi Steven,

On 2010-08-09 10:21, Steven D'Aprano wrote:
> And that it's quite finicky about blank lines between methods and inside 
> functions. Makes it hard to paste code directly into the interpreter.
> 
> And that pasting doesn't strip out any leading prompts. It needs a good 
> doctest mode.

ipython [1] should help here:

  IPython 0.10 -- An enhanced Interactive Python.
  ? -> Introduction and overview of IPython's features.
  %quickref -> Quick reference.
  help  -> Python's own help system.
  object?   -> Details about 'object'. ?object also works, ?? prints more.
  In [1]: %paste?
  Type:   Magic function
  Base Class: 
  String Form:>
  Namespace:  IPython internal
  File:   /usr/lib/pymodules/python2.6/IPython/Magic.py
  Definition: %paste(self, parameter_s='')
  Docstring:
  Allows you to paste & execute a pre-formatted code block from clipboard.

  The text is pulled directly from the clipboard without user
  intervention.

  The block is dedented prior to execution to enable execution of method
  definitions. '>' and '+' characters at the beginning of a line are
  ignored, to allow pasting directly from e-mails, diff files and
  doctests (the '...' continuation prompt is also stripped).  The
  executed block is also assigned to variable named 'pasted_block' for
  later editing with '%edit pasted_block'.

  You can also pass a variable name as an argument, e.g. '%paste foo'.
  This assigns the pasted block to variable 'foo' as string, without
  dedenting or executing it (preceding >>> and + is still stripped)

  '%paste -r' re-executes the block previously entered by cpaste.

  IPython statements (magics, shell escapes) are not supported (yet).

  See also
  
  cpaste: manually paste code into terminal until you mark its end.

Unfortunatey, when I enter

  In [2]: %paste

at the prompt it gives me (before I pasted anything)

  In [2]: %paste
  
 File "", line 1
   http://pypi.python.org/pypi/ipython/0.10
   ^
  SyntaxError: invalid syntax

So far, I couldn't find anything on the net on this.

[1] http://pypi.python.org/pypi/ipython

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: easy question on parsing python: "is not None"

[Stefan Schwarzer]

Hi Ben,

On 2010-08-08 01:16, Ben Finney wrote:
> Don't use strings for such values. The data isn't going to be used, so
> there's no sense using a semantically rich data type like a string.
> 
> Instead, use an ‘object’ instance; then, the only way to get a binding
> that will compare equal is to use the very object itself.
> 
> FORWARD = object()
> BACKWARD = object()

Yes, that's the obvious approach. :-) I had used strings in
the example with the reasoning that they could be readily
used in messages (from my post: "Using a string instead of
an `object()` has the advantage that it makes usage in error
messages easier.").

For the record, another approach I've seen is something like

FORWARD, BACKWARD, UNKNOWN = range(3)

This has the same problem as when using strings for the
constants but looking at the module contents it's a bit
more obvious that these are just "magic numbers", so
presumably the constant names should be used instead.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: default behavior

[Stefan Schwarzer]

On 2010-07-31 05:47, Steven D'Aprano wrote:
> On Fri, 30 Jul 2010 08:34:52 -0400, wheres pythonmonks wrote:
> It does re-use the same underlying data.
> 
> >>> from collections import defaultdict as dd
> >>> x = dd(list)
> >>> x[1].append(1)
> >>> x
> defaultdict(, {1: [1]})
> >>> y = dict(x)
> >>> x[1].append(42)
> >>> y
> {1: [1, 42]}

One thing to keep in mind: dict(some_defaultdict) doesn't
store a reference to the defaultdict; instead it makes a
shallow copy, so key/value pairs added _after_ the "cast"
aren't included in the new dict:

>>> y[2] = 17
>>> y
{1: [1, 42], 2: 17}
>>> x
defaultdict(, {1: [1, 42]})

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: easy question on parsing python: "is not None"

[Stefan Schwarzer]

Hi Steven,

On 2010-08-07 00:28, Steven D'Aprano wrote:
> On Fri, 06 Aug 2010 15:37:04 +0200, Stefan Schwarzer wrote:
>>> Plus, I believe the
>>> "==" operator will check if the variables point to the same object.
>>
>> No, that's what `is` is for.
> 
> Actually, yes, equality is implemented with a short-cut that checks for 
> identity first. That makes something like:
> [...]

Oops, I didn't realize that the OP had mentioned the
identity check as an optimization in case the objects are
the same. I thought he was confusing the operator with `is`.

> s = "abc"*1000*1000*10
> s == s
> 
> nice and quick, as Python can immediately recognise that a string is 
> always equal to itself without having to walk the entire string comparing 
> each character with itself.

Yes, that definitely makes sense. I guess I would have
implemented it this way as well. :)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: easy question on parsing python: "is not None"

[Stefan Schwarzer]

Hello Peter,

On 2010-08-06 19:20, Peter Pearson wrote:
> On Fri, 06 Aug 2010 15:37:04 +0200, Stefan Schwarzer wrote:
> [snip]
>> I can imagine a case where you might want to compare a
>> string with `is`:
>>
>> FORWARD = "forward"
>> BACKWARD = "backward"
>>
>> ...
>>
>> def func(direction=FORWARD):
>> if direction is FORWARD:
>> ...
>> elif direction is BACKWARD:
>> ...
>> else:
>> ...
>> [...]

> Hey, that's a cute example,

Thanks!

> but . . . what a trap!  Is it
> possible to document the use-the-object-not-the-string requirement
> loudly enough that people won't get caught?

That's a good question, actually that's the nagging problem
I also have with the approach.

In production code I might write

# Use _these_ objects to indicate directions, not string
# constants with the same value.
FORWARD = "forward"
BACKWARD = "backward"

However, that won't help if people import the module and see
the strings under "global data" and so assume they're "just
strings". Moreover, if you have to write such a comment for
every use of the idiom the conciseness of the approach
vanishes.

Another view at the matter would be to let clients of the
module find out their mistake by unit tests. But of course
that's somewhat doubtful as the intention should be to write
robust instead of bug-inviting code.

I wonder if there's a way to have both the "cuteness" of the
string constants _and_ more robustness.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: easy question on parsing python: "is not None"

[Stefan Schwarzer]

Hi DG,

On 2010-08-06 14:28, DG wrote:
> I've always thought of it as you don't compare strings with "is", you
> *should* use ==  The reasoning is that you don't know if that string
> instance is the only one in memory.  I've heard as an implementation
> detail, since strings are immutable, that Python will only create one
> actual instance and just assign new references to it (your first x is
> y example), but to compare equality it just makes sense to use "==",
> not to ask if it is the same object in memory.

Yes, you'd usually use == to compare strings. The use in the
post presumably was to show the behavior when you use `is`;
I guess it wasn't meant as an example for production code. :)

I can imagine a case where you might want to compare a
string with `is`:

FORWARD = "forward"
BACKWARD = "backward"

...

def func(direction=FORWARD):
if direction is FORWARD:
...
elif direction is BACKWARD:
...
else:
...

in case you expect people to specifically use the constants
you provided in the module. Here, the fact that FORWARD
actually is the string "forward" might be considered an
implementation detail. Using a string instead of an
`object()` has the advantage that it makes usage in error
messages easier.

Actually, I've never seen such a use, as far as I remember.
What do other people here think? Is the code above, which
compares strings with `is`, bad style, and if yes, why? How
would you write the code instead?

> Plus, I believe the
> "==" operator will check if the variables point to the same object.

No, that's what `is` is for.

> Using is/is not with None works well, because I believe there will
> always only be 1 None object.

Yes, and it avoids subtle bugs if someone overwrites `__eq__`
in some class:

>>> class AlwaysEqual(object):
... def __eq__(self, other):
... return True
...
>>> always_equal = AlwaysEqual()
>>> always_equal == None
True
>>> None == always_equal
True
>>> always_equal is None
False

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: simple integer subclass

[Stefan Schwarzer]

Hi Andreas,

On 2010-08-03 12:15, Andreas Pfrengle wrote:
> On 3 Aug., 03:22, Carl Banks  wrote:>
> Thinking about it, it might really be dangerous to coerce always to
> int1, since sometimes I might want a normal int as result (I can't
> tell yet for sure).

Yes, that way your problem may shift from inconvenient to
outright hairy. ;-)

> The application will be a browsergame, and most gamers start counting
> at 1, so they would probably wonder about a "level 0 item" ;-)
> If there didn't already exist lots of code, I would redesign the whole
> data-structure - I think that's "lessons learned" for the next project

What about

def _index_to_level(index):
return index + 1

with this or a different name? This admittedly is longer
than writing `something + 1` but in the latter case you
might wonder what the addition is for, i. e. if it's really
a level offset calculation or something else.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: __init__ as a lambda

[Stefan Schwarzer]

Hi Eric,

On 2010-08-04 21:58, Eric J. Van der Velden wrote:
> class C:
>  def __init__(self,name):self.name=name
> 
> I was wondering if I could make the __init__ a lambda function, but
> 
> class C:
>  __init__=lambda self,self.name:None
> 
> and then later,
> 
> C('Hello')
> 
> does not work; the first argument, self, is assigned all rigth, but
> you cannot write the second argument with a dot,  self.name .

The "problem" is that in a lambda function the part after
the colon has to be an expression. However, you have used
an assignment there which isn't an expression in Python but
a statement.

For example, you can use

f = lambda x: sys.stdout.write(str(x))

(sys.stdout.write(str(x)) is an expression)

but not

f = lambda x: print x

(print x  is a statement in Python versions < 3)

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

[ANN] Websourcebrowser 0.4a released

[Stefan Schwarzer]

Hello,

I'm happy to announce the release of Websourcebrowser 0.4a.

Websourcebrowser is a program intended to get a quick overview of a
project's source code. The program is controlled from a web browser
which displays a directory tree and a source code file side by side.

The homepage of the project is at

http://websourcebrowser.sschwarzer.net/

The download of version 0.4a is at

http://websourcebrowser.sschwarzer.net/download

*** Note that this is still an alpha release. ***

I use the program regularly to dive into new projects, but I consider
it in the alpha stage because it has been used and tested by only a
few other people than me.

On Windows, this version doesn't seem to work with IE 8.0 whereas
it works with Firefox 3.6.8.

I know the software could be improved a lot. For some ideas, look at

http://websourcebrowser.sschwarzer.net/trac/wiki/Ideas
http://websourcebrowser.sschwarzer.net/trac/browser/todo.txt

If I implement this alone, it'll probably take years, given that
Websourcebrowser is a spare time project (beside my other project,
ftputil). So it would be great to get more developers on board.

Of course, I'm thankful for feedback on the mailing list (subscription
required to avoid spam, sorry),

http://codespeak.net/mailman/listinfo/websourcebrowser

or via private e-mail. If you encounter problems, you may also file a
bug report at

http://websourcebrowser.sschwarzer.net/trac/newticket

*** You need to log in as user wsbuser with password wsb . ***

Stefan

-- 
http://mail.python.org/mailman/listinfo/python-list

[ANN] ftputil 2.2.3 released

[Stefan Schwarzer]

ftputil 2.2.3 is now available from
http://ftputil.sschwarzer.net/download .

Changes since version 2.2.2
---

This release fixes a bug in the ``makedirs`` call (report and fix by
Julian, whose last name I don't know ;-) ). Upgrading is recommended.

What is ftputil?


ftputil is a high-level FTP client library for the Python programming
language. ftputil implements a virtual file system for accessing FTP
servers, that is, it can generate file-like objects for remote files.
The library supports many functions similar to those in the os,
os.path and shutil modules. ftputil has convenience functions for
conditional uploads and downloads, and handles FTP clients and servers
in different timezones.

Read the documentation at
http://ftputil.sschwarzer.net/documentation .

License
---

ftputil is Open Source software, released under the revised BSD
license (see http://www.opensource.org/licenses/bsd-license.php ).

Stefan

-- 
Dr.-Ing. Stefan Schwarzer
SSchwarzer.com - Softwareentwicklung für Technik und Wissenschaft
http://sschwarzer.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Needed: FTP Upload Directory Tree script

[Stefan Schwarzer]

Hello Ian,

On 2007-06-10 10:01, IanC wrote:
> Does anyone know of a function or script to upload an entire
> subdirectory tree from a local file space to an FTP server?
> 
> The Python distribution comes with "ftpmirror.py", which performs
> a mirror download of a directory tree, but I need the "Upload"
> version of this.

I'm just working on an ftputil[1] extension[2] to copy/mirror
directory trees between local filesystems and remote FTP hosts.
It's in pre-alpha state (just in development), but maybe it
already does what you want. Did I note that this is an only
slightly tested module and might destroy your files? - That is,
use it at your own risk. I recommend you look at the source
code of ftp_sync.py before using it.

It works like this:

- Download[3] and install ftputil.

- Download the file[2] and put it into the site-packages/ftputil
  directory.

- Use the library like:

import ftputil
from ftputil import ftp_sync

source = ftp_sync.LocalHost()
target = ftputil.FTPHost(hostname, username, password)
syncer = ftp_sync.Syncer(source, target)
syncer.sync(source_directory, target_directory)

Note that the _contents_ of source_directory (_not_ the directory
itself) will be copied to the already-existing target_directory.

If you have further questions or feedback, please send e-mail
to the ftputil mailing list[4] or to me[5].

[1] http://ftputil.sschwarzer.net
[2] http://ftputil.sschwarzer.net/trac/browser/trunk/ftp_sync.py?format=txt
[3] http://ftputil.sschwarzer.net/trac/wiki/Download
[4] mailto:[EMAIL PROTECTED]
[5] mailto:[EMAIL PROTECTED]

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

[ANN] ftputil 2.2.1

[Stefan Schwarzer]

ftputil 2.2.1 is now available from
http://ftputil.sschwarzer.net/download .

Changes since version 2.2
-

This bugfix release checks and ignores status code 451 when FTPFiles
are closed (thanks go to Alexander Holyapin). Upgrading is recommended.

What is ftputil?


ftputil is a high-level FTP client library for the Python programming
language. ftputil implements a virtual file system for accessing FTP
servers, that is, it can generate file-like objects for remote files.
The library supports many functions similar to those in the os,
os.path and shutil modules. ftputil has convenience functions for
conditional uploads and downloads, and handles FTP clients and servers
in different timezones.

Read the documentation at
http://ftputil.sschwarzer.net/trac/wiki/Documentation .

License
---

ftputil is Open Source software, released under the revised BSD
license (see http://www.opensource.org/licenses/bsd-license.php ).

Stefan

-- 
Dr.-Ing. Stefan Schwarzer
SSchwarzer.com - Softwareentwicklung für Technik und Wissenschaft
http://sschwarzer.com
http://ftputil.sschwarzer.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: clarification on open file modes

[Stefan Schwarzer]

On 2007-01-05 03:46, tubby wrote:
> Is this the safest, most portable way to open files on any platform:
>
> fp = open(file_name, 'rb')
> fp.close()
>
> I understand that doing the following on Windows to a binary file (a
> jpeg or exe files for example) can cause file corruption, is that correct?
>
> fp = open(file_name, 'r')
> fp.close()

Rule of thumb: If the file is a text file (usually has a concept
of "text lines"), read and write it in text mode (without "b" in
the mode argument). So line endings will be converted, so that
your Python code can process the text data identically on all
platforms. On the other hand, if you process binary files (as you
mention, for example jpg or exe files), read and write them in
binary mode, so the data won't be corrupted.

> How can a simple open in read mode corrupt data???

You won't corrupt the data in the file, but you will kind of
corrupt the data that arrives in your program.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why less emphasis on private data?

[Stefan Schwarzer]

On 2007-01-07 01:54, BJörn Lindqvist wrote:
> Google for "python for consenting adults"
>
> Or ask yourself the opposite question. Why does C++ and C# prefer more
> private data? It is given that emphasizing private data
> (encapsulation) leads to more internal complexity and more lines of
> code because you have to write getters and setters and stuff. With
> that in mind, why do you think that data encapsulation makes code less
> error prone? Can you prove it? Or do you have anecdotal evidence of
> where data encapsulation saved your ass?
>
> IMHO, that data hiding is good, is one of those ideas that have been
> repeated so much that virtually everyone thinks it is true.  But
> Python proves that it isn't necessarily so.

I think attributes (callable or not) which relate to the
abstraction of the class should be "public" (special methods
or without leading underscore). Attributes that are there for a
specific implementation of the abstraction should be "private".

The internal implementation of a class is more-often changed
in incompatible ways than the abstraction, so distiguishing
between a public and a private interface will probably save
you from reworking the clients of a class if you prefer the
public interface. It will also make the client code easier to
understand.

Admittedly, there are special cases where you want to access
private attributes, e. g. debugging; that's ok.

In summary, the distinction between public and non-public
attributes IMHO makes sense, but I don't think that the
distinction should be enforced by the language as in C++
or Java.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to get file name on a remote server with ftplib?

[Stefan Schwarzer]

On 2007-01-05 16:10, [EMAIL PROTECTED] wrote:
> alex wrote:
>> My script is trying to get a file from a remote server, every day it
>> ftps from a directory. My code works perfect if I know the name of the
>> file in the remote directory.
>>
>> ftp.retrbinary('RETR ' + filename, handleDownload)
>>
>> The problem is that in the future the name of the file is going to be
>> aleatory. How can I get the file's name (only one file is in that
>> directory) and store it in a variable before executing ftp.retrbinary?
>>
>> Thanks a lot for your help,
>>
> You can use the nlst(dirname) method to retrieve a directory listing on
> the remote server. If the remote server doesn't support NLST command
> rhen you could do ftp.retrlines('LIST')  and parse the results to get a
> directory listing.

ftputil ( http://ftputil.sschwarzer.net/ ) could help with
this parsing. A session may look like

>>> import ftputil
>>> host = ftputil.FTPHost("ftp.domain.com", "user", "password")
>>> file_names = host.listdir(host.curdir)
>>> file_names
['wanted_filename']
>>> file_name = file_names[0]
>>> host.download(file_name, file_name, "b")
>>> host.close()

Ok, I admit this isn't a pure-ftplib solution. :-)

Stefan

-- 
Dr.-Ing. Stefan Schwarzer
SSchwarzer.com - Softwareentwicklung für Technik und Wissenschaft
http://sschwarzer.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: code optimization (calc PI)

[Stefan Schwarzer]

On 2007-01-03 16:50, mm wrote:
> More general, maybe there is a speed optimazation docu out there.

At least Alex Martellis "Python in a Nutshell" has a section on
optimization.

I presented this at the last EuroPython conference:
http://sschwarzer.com/download/optimization_europython2006.pdf

Stefan

-- 
Dr.-Ing. Stefan Schwarzer
SSchwarzer.com - Softwareentwicklung für Technik und Wissenschaft
http://sschwarzer.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: simple ftputil ssl client

[Stefan Schwarzer]

On 2006-12-31 19:27, Croteam wrote:
> I trying to make ftputil client that uses ssl security.First I was try
> to make that with M2Crypto,but
> when I use it, I get the error:
>
> Traceback (most recent call last):
>   File "", line 1, in -toplevel-
> import M2Crypto
>   File "C:\Python24\lib\site-packages\M2Crypto\__init__.py", line 15,
> in -toplevel-
> import m2
>   File "C:\Python24\Lib\site-packages\M2Crypto\m2.py", line 28, in
> -toplevel-
> from __m2crypto import *
> ImportError: DLL load failed with error code 182

It seems this problem has nothing to do with ftputil, so it might
be best to post the question on the failed import alone with a
subject like

ImportError for M2Crypto: "DLL load failed with error code 182"

or even, since a Google search implies the problem doesn't occur
only for M2Crypto,

ImportError: "DLL load failed with error code 182"

With the current subject, people may think "I don't know ftputil"
and don't read your posting at all.

You might ask the question with the above subject on the
Python-Crypto mailing list which seems to be the best place for
M2Crypto-related questions. You should tell the version of
M2Crypto and the operating system you use. By the way, have you
tried other M2Crypto versions?

Can you import the module when the current directory is the one
containing the DLL? To check, change to the directory and start
the interactive Python interpreter. (If you use an IDE or
something similar, as your traceback suggests, that program might
change the directory before the import statement runs.)
Additionally, you can try to modify sys.path to contain the path
of the DLL before invoking the import statement. See also my
response on the ftputil mailing list.

> ...so now I trying to make it with pyOpenSSL or twistedmatrix ssl,but I
> don't know how.If anybody
> have any idea or example how to I make it,please help me!!!

Now the part specific to ftputil ;-) ...

If you can't use M2Crypto, try to make sure that you have a class
which is compatible with ftplib.FTP. You can pass such a class as
session_factory to ftputil.FTPHost's constructor. If you don't
have such a class, you might be able to write it yourself.

Stefan

-- 
Dr.-Ing. Stefan Schwarzer
SSchwarzer.com - Softwareentwicklung für Technik und Wissenschaft
http://sschwarzer.com
http://www.sschwarzer.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to depress the output of an external module ?

[Stefan Schwarzer]

Hi Luis,

Luis Armendariz wrote:
> There's no need for savestdout. There's a backup copy in sys.__stdout__

Depending on the code that ran before the call to the
function run_without_stdout, sys.stdout may not be
the same as sys.__stdout__ . Of course, you also have
to be careful regarding threads which also use
sys.stdout, perhaps implicitly via print statements.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

[ANN] ftputil 2.2 released

[Stefan Schwarzer]

ftputil 2.2 is now available from
http://ftputil.sschwarzer.net/download .

Changes since version 2.1
-

- Results of stat calls (also indirect calls, i. e. listdir,
  isdir/isfile/islink, exists, getmtime etc.) are now cached and
  reused. This results in remarkable speedups for many use cases.
  Thanks to Evan Prodromou for his permission to add his lrucache
  module under ftputil's license.

- The current directory is also locally cached, resulting in further
  speedups.

- It's now possible to write and plug in custom parsers for directory
  formats which ftputil doesn't support natively.

- File-like objects generated via ``FTPHost.file`` now support the
  iterator protocol (for line in some_file: ...).

- The documentation has been updated accordingly. Read it under
  http://ftputil.sschwarzer.net/trac/wiki/Documentation .

Possible incompatibilities:

- This release requires at least Python 2.3. (Previous releases
  worked with Python versions from 2.1 up.)

- The method ``FTPHost.set_directory_format`` has been removed,
  since the directory format (Unix or MS) is set automatically. (The
  new method ``set_parser`` is a different animal since it takes
  a parser object to parse "foreign" formats, not a string.)

What is ftputil?


ftputil is a high-level FTP client library for the Python programming
language. ftputil implements a virtual file system for accessing FTP
servers, that is, it can generate file-like objects for remote files.
The library supports many functions similar to those in the os,
os.path and shutil modules. ftputil has convenience functions for
conditional uploads and downloads, and handles FTP clients and servers
in different timezones.

License
---

ftputil 2.2 is Open Source software, released under the revised BSD
license (see http://www.opensource.org/licenses/bsd-license.php ).

Stefan

-- 
Dr.-Ing. Stefan Schwarzer
SSchwarzer.com - Softwareentwicklung f??r Technik und Wissenschaft
http://sschwarzer.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: FTP over SSL

[Stefan Schwarzer]

On 2006-10-20 15:50, Tor Erik Soenvisen wrote:
> Anyone know about existing code supporting FTP over SSL?
> I guess pycurl http://pycurl.sourceforge.net/ could be used, but that
> requires knowledge of libcurl, which I haven't.

Possibly you could use M2Crypto [1]. It has a FTP_TLS class
in module ftpslib which is similar to ftplib.FTP. To use FTP
more easily, you can use ftputil [2] (I'm its author). You
should be able to use M2Crypto via ftputil [3].

[1] http://wiki.osafoundation.org/bin/view/Projects/MeTooCrypto#Downloads
[2] http://ftputil.sschwarzer.net
[3] 
http://ftputil.sschwarzer.net/trac/browser/branches/add_stat_caching/ftputil.txt?rev=622
line 893

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Data sticking around too long

[Stefan Schwarzer]

Hello Matimus,

On 2006-09-07 00:07, Matimus wrote:
> Someone correct me if I'm wrong (sometimes I get the terms mixed up)
> but I believe that what you are seeing is due to 'number' being an
> immutable type. This means that its value cannot be changed and thus
> each assignment is effectively creating a new instance if int.

The significant difference is not between mutable and immutable
objects, but between modifying an object in-place
(e. g. some_list.append(1)) and assigning an object to a name
(e. g. some_int = 1). If you do an assignment to a class
"variable", the binding to the previous object (if any) will be
removed and a new binding to the assigned object created.

The assignments Rob does with

c1.number = 1
c2.number = 2

create a new name "number" in each of the instances' namespaces
and assign the objects 1 and 2 to them, respectively. In contrast,
the list scanList is modified in-place with

c1.scanList.append("One")
c2.scanList.append("Two")

so no names are created in the instances' namespaces and both
operations modify the object in the class's namespace.

> I believe lists are considered immutable also

I guess you confuse this with tuples.

> but since it is a container
> type it behaves somewhat differently.

Again, this has nothing per se to do with being a container
though many containers are mutable objects and often are
modified in-place.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: programming is hard

[Stefan Schwarzer]

Hi placid:

On 2006-08-03 14:56, placid wrote:
> this site is contains code snippets for various languages and python
> does not have many code snippet posters.

Probably that's because there's a Python "snippet repository" at
http://aspn.activestate.com/ASPN/Python/Cookbook/  :-) It wouldn't
make sense to duplicate all this stuff.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Replace Whole Object Through Object Method

[Stefan Schwarzer]

Hi Bruno :)

On 2006-06-27 10:30, Bruno Desthuilliers wrote:
> FWIW, a good part of the canonical design patterns are mostly workaround
> the lack of flexibility in languages like C++ and Java.

May be, but that doesn't exclude that some patterns are also
useful in Python. :-)

> The Strategy
> pattern's simplest Python implementation is to dynamically replace a
> method on a per-object basis. The State pattern's simplest
> implementation in Python is to dynamically change the class of an object.

It may be the "simplest" but it might not be the most "natural"
in terms of the problem domain.

At the Informationsdienst Wissenschaft ( http://idw-online.de )
we had a similar design problem. There, a person can have many
roles, like subscriber, journalist, public info officer etc. We
found it most natural to implement this with the actor/role
pattern, as I V described it.

> Of course, one of the canonical solutions to the OP's problem is to use
> composition/delegation. But doing it the canonical way would imply
> creating instances of the role objects and passing them the instance as
> param so they can access it's attributes. It works, but it creates a
> cyclic dependancy between the object and it's roles, so you would have
> to use weakrefs.

You don't have to, unless both classes have a __del__ method and
the garbage collector thus can't decide which to collect/delete
first.

> Dynamically creating new classes with the appropriate
> bases and assigning them to the object's __class__ attribute is another
> way to achieve the same result, and it's perfectly legal.

It might be "legal" but it also may be confusing. I would use the
design that makes the software easiest to maintain in the long
run. (If you don't know what the long run is, consider the
"medium run" and refactor later. :) )

> Now I do agree that it can become tricky to manage correctly wrt/ mro
> rules !-)

See? ;-) Software shouldn't be tricky (or only as tricky as
necessary). I value clarity over cleverness.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ftp connection and commands (directroy size, etc)

[Stefan Schwarzer]

Hi Arne

On 2006-04-12 18:05, Arne wrote:
> I am working on Windows XP and I want to do the following:
> 
> 1. Connecting to a server using ftp
> 2. Getting the directory structure and the size of each directory in the 
> root
> 3. Getting the owner of a file
> 
> All these steps I want to do with python. What I already have is:
> [...]
> Please be so kind and post a little bit of a solution code

You should be able to do this with the ftputil library, at
http://ftputil.sschwarzer.net/ ;-) If you encounter any problems,
send a mail to the ftputil mailing list (see
http://ftputil.sschwarzer.net/trac/wiki/MailingList ) or to me.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ftp putting information in a variable

[Stefan Schwarzer]

Hi Arne,

On 2006-04-08 12:44, Arne wrote:
> I am looking for a way to put ftp returns in a variable.
>
> My OS is XP and I want to get the owner of a file. So I have to
>
> connect to ftp. But I am stacked with how I can receive this
> information and put it in a variable.

you can use a library to handle that. One of them is ftputil
( http://ftputil.sschwarzer.net/ ), which I know because I'm its
author. ;-) Surely, there are alternatives. You can search the
Python package index at http://www.python.org/pypi .

With ftputil, you would do

import ftputil
host = ftputil.FTPHost(hostname, user, password)
# st_uid is used by Python's os.(l)stat, but it's a string here,
# not an integer
user = host.stat(file_or_dir).st_uid
...
host.close()

This works only if the output of the FTP LIST command contains
the user information (which is often the case).

ftputil is pure Python and should work on OS X without problems.

If you think that installing/using an additional library is
overkill, you can extract the necessary parser code from the file
ftp_stat.py or write your own parser.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

[ANN] ftputil 2.1 released

[Stefan Schwarzer]

ftputil 2.1 is now available from
http://ftputil.sschwarzer.net/download .

Changes since version 2.0
-

- Added new methods to the FTPHost class, namely makedirs, walk,
  rmtree.

- The FTP server directory format ("Unix" vs. "Windows") is now set
  automatically (thanks to Andrew Ittner for testing it).

- Border cases like inaccessible login directories and whitespace in
  directory names, are now handled more gracefully (based on input
  from Valeriy Pogrebitskiy, Tommy Sundström and H. Y. Chu).

- The documentation was updated.

- A Russian translation of the documentation (currently slightly
  behind) was contributed by Anton Stepanov. It's also on the website
  at http://ftputil.sschwarzer.net/trac/wiki/RussianDocumentation .

- New website, http://ftputil.sschwarzer.net/ with wiki, issue tracker
  and Subversion repository (thanks to Trac!)

  Please enter not only bugs but also enhancement request into
  the issue tracker!

Possible incompatibilities:

- The exception hierarchy was changed slightly, which might break
  client code. See http://ftputil.sschwarzer.net/trac/changeset/489
  for the change details and the possibly necessary code changes.

- FTPHost.rmdir no longer removes non-empty directories. Use the new
  method FTPHost.rmtree for this.

What is ftputil?


ftputil is a high-level FTP client library for the Python programming
language. ftputil implements a virtual file system for accessing FTP
servers, that is, it can generate file-like objects for remote files.
The library supports many functions similar to those in the os,
os.path and shutil modules. ftputil has convenience functions for
conditional uploads and downloads, and handles FTP clients and servers
in different timezones.

License
---

ftputil 2.1 is Open Source software, released under the revised BSD
license (see http://www.opensource.org/licenses/bsd-license.php ).

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Non-GUI source code browser with file system tree view

[Stefan Schwarzer]

Hello,

from time to time I want to inspect the source code of projects
on remote computers.(*) I've googled for one or two hours but
didn't find anything helpful. :-/ I'm looking for something like
IDLE's path browser - i. e. a tree view and file view
side-by-side - but with the following differences:

- doesn't need an X connection to the remote computer where the
  sources are

- must also work with other programming languages (ideally with
  syntax highlighting)

- consequently, the tree view will not be based on a module
  search path but on a file system directory

Ideally, but not necessesarily, this browser should be written in
Python. If possible, it should be open source software. It _must_
run on GNU/Linux and, if possible, on Mac OS X (a local X server
is ok if the mentioned browser is a curses-based program).
(Another approach might be a small web application server in the
spirit of "pydoc -p " which could connect to a local port
forwarded with ssh.)

Does anyone know of a source code browser which meets the
requirements listed above or links that could help me? Many
thanks in advance. :-)

(*) Copying the files to the local host is probably rather
impractical because the files sometimes change very frequently.
Developing only locally is impractical for some projects because
the remote development server has some infrastructure that I
can't reproduce locally or only with a lot of work.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: i have error then use ftplib

[Stefan Schwarzer]

Hello nazarianin,

On 2006-03-30 09:35, 5>=>2 ;5:A59 wrote:
> from ftplib import FTP
> def handleDownload(block):
> file.write(block)
> print "."
>
> file = open('1', 'wb')
> ftp = FTP('ftp.utk.ru')
> ftp.set_pasv(1)
> ftp.login()
> ftp.retrlines('LIST')
> [...]

> and have this error message.
>
> Traceback (most recent call last):
>   File "ftp.py", line 10, in ?
> ftp.retrlines('LIST')
> [...]
> socket.error: (10065, 'No route to host')
>
> Programs that not use Python connect to the server ok.
> Where I do mistake?

Are you sure that you don't need user and password arguments in
.login()? The documentation on ftplib says:

login([user[, passwd[, acct]]])
Log in as the given user. The passwd and acct parameters are
optional and default to the empty string. If no user is
specified, it defaults to 'anonymous'. If user is
'anonymous', the default passwd is 'anonymous@'. This
function should be called only once for each instance, after
a connection has been established; it should not be called at
all if a host and user were given when the instance was
created. Most FTP commands are only allowed after the client
has logged in.

Perhaps the server wasn't satisfied with your credentials and
closed the connection between the .login() and .retrlines()
calls?

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Avoiding FTP server timeout for files based on sockets (long posting)

[Stefan Schwarzer]

Hi all!

For my FTP library module ftputil [1], some users have asked
for a way to avoid server timeouts (FTP status code 421). But I
haven't found out yet how I can do this in all cases.

I try to explain the problem in more detail. The following is
rather special and probably not so easy to understand, but I'll
do my best. Please ask if you need more information.

ftputil has an FTPHost class (defined in [2]), which can be
instantiated like

# very similar to ftplib.FTP
host = ftputil.FTPHost(ftphost, user, password)

You can use an FTPHost instance to get file(-like) objects from it:

read_from_this_file = host.file("remote_file", "r")
write_to_this_file = host.file("another_remote_file", "wb")

In the background, each call of the FTPHost.file method opens
another connection to the FTP server by using the login data from
the FTPHost instantiation. The return value of each call is a
_FTPFile object (defined in [3]) which wraps a file object
returned by socket.makefile.

My current FTPHost.keep_alive is roughly defined as

# in FTPHost
def keep_alive(self):
# just prevent loss of the connection, so discard the result
self.getcwd()
# refresh also connections of associated file-like objects
for host in self._children:
# host._file is an _FTPFile object (see [3])
host._file.keep_alive()

whereas in _FTPFile it's

# in _FTPFile
def keep_alive(self):
if self._readmode:
# read delegates to the file made from the data transfer
# socket, made with socket.makefile (see [4])
self.read(0)
else:
# write delegates to the file made from the data transfer
# socket, made with socket.makefile (see [4])
self.write("")
self.flush()

In fact, the read method call above on the data transfer channel
keeps the connection open but the call to the write method can't
avoid a timeout from the FTP server (however, I notice this only
when I call _FTPFile.close(), so it seems that no data is sent
until the _FTPFile.close call).

An approach which seems feasible at first is to call pwd() on the
FTP session (an ftplib.FTP class) on which the _FTPFile builds
(similar to the FTPHost.getcwd() call above). Unfortunately, this
doesn't work because as soon as the file is opened, a STOR
command has been sent to the FTP server and it seems I can't send
another FTP command until the data transfer is finished (by calling
_FTPFile.close; see _FTPFile._open in [3] for details of making
the connection).

So to re-phrase my question: How can I keep the connection - for
writing of remote files - from being closed by the FTP server
without requiring the user of the ftputil library to explicitly
send data with _FTPFile.write?

Stefan

[1] http://ftputil.sschwarzer.net/
[2] http://ftputil.sschwarzer.net/trac/browser/trunk/ftputil.py
[3] http://ftputil.sschwarzer.net/trac/browser/trunk/ftp_file.py
[4] http://docs.python.org/lib/socket-objects.html#l2h-2660
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problems with python and threads under Freebsd

[Stefan Schwarzer]

snacktime wrote:
> I am consistantly getting seg faults and bus errors when running a
> multi threaded python application under 5.3-release-p2.  Python was
> built from the port(python 2.4) with threading enabled.

I had similar problems some time ago. The solution/workaround was to remove
the threading library-related mappings from /etc/libmap.conf . Perhaps you
need to do that or the opposite, i. e. add a mapping.

What do you get if you run the threading tests
in /usr/local/lib/python2.4/test/test_thread*.py ? In my case,
test_threaded_import.py failed before the libmap.conf change.

If the libmap.conf change doesn't help, you could try to rebuild the base
system and kernel and after that the Python port. Also, reading the
UPDATING files for the base system and the ports tree may give you a hint.

Stefan

-- 
http://mail.python.org/mailman/listinfo/python-list