Re: [RELEASE] mpdecimal-2.5.0

[Antoine Pitrou]

Hi Stefan

On Mon, 29 Jun 2020 15:27:01 +0200
Stefan Krah  wrote:
> Hi,
> 
> I've released mpdecimal-2.5.0:
> 
>http://www.bytereef.org/mpdecimal/index.html
> 
> 15417edc8e12a57d1d9d75fa7e3f22b158a3b98f44db9d694cfd2acde8dfa0ca  
> mpdecimal-2.5.0.tar.gz
> 
> Starting with Python 3.9, this version should be used for an external 
> libmpdec.
> 
> Python versions 3.7 and 3.8 should use the previous version mpdecimal-2.4.2.

Speaking of which, is there a plan to incorporate a C API to the
_decimal module?

Currently, there's no reliable way AFAIK, from C or C++, to take a
PyObject* and acccess the underlying `mpd_t*` directly, for fast
conversions.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: AssertionError (3.X only) when calling Py_Finalize with threads

[Antoine Pitrou]

Hi Tom,

Tom Kent  gmail.com> writes:
> 
> I'm getting an error output when I call the C-API's Py_Finalize() from a
different C-thread than I made a
> python call on.

Can you please post a bug on https://bugs.python.org ?
Be sure to upload your example there.

Thank you

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Antoine Pitrou]

Christian Heimes  python.org> writes:
> 
> The article doesn't state if the writer is referring to virtual memory
> or resident set size.

Actually the article mentions the following recipe:

  resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

which means the author is probably looking at resident set size.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "High water" Memory fragmentation still a thing?

[Antoine Pitrou]

Hello,

Croepha  gmail.com> writes:
> 
> Question:
> 
> Does python in general terms (apart from extensions or gc manipulation),
exhibit a "high water" type leak of allocated memory in recent python
versions (2.7+)?

It is not a leak. It is a quite common pattern of memory fragmentation.
The article is wrong in assuming that Python doesn't return the memory to
the OS. Python does return its empty memory pools to the OS, however the OS
itself may not be able to release that memory, because of heap fragmentation.

As the article mentions, this was improved (mostly fixed?) in 3.3.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newer Debian versions of python on older Debian distros?

[Antoine Pitrou]

Hi,

Chris Angelico  gmail.com> writes:
> 
> On Tue, Sep 9, 2014 at 5:04 AM, Travis Griggs 
gmail.com> wrote:
> > Does anyone have experience with using newer versions of python debian
packages (in particular, python3
> and python3-bson-ext from ‘testing’) on older stable versions (‘wheezy’ in
this case)? If
> someone’s figured out how to do this easily, I’d love to hear the recipe!
> >
> 
> I don't know about grabbing from testing, because that's fraught with
> peril... but there's a pretty easy way to get a newer Python: build
> from source. Basically, the standard incantation: [...]

I know the OP solved their problem now, but for the record, you can also get
binaries of recent Pythons (and other packages) by using conda:
http://conda.pydata.org/miniconda.html#miniconda

This assumes you don't mind having a local install rather than a system install
of Python (something akin to a virtual environment).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: weakref, memory management and execution slow down in PyQt4

[Antoine Pitrou]

kjs  riseup.net> writes:
> 
> I have come to believe that the growing number of weakrefs is slowing
> down execution. Is my analysis misguided? How can I introspect further?
> If the slowdown can be attributed to weakref escalation, what are some
> next steps?

The way to analyze this is to build some gradually smaller subsets of your
application until you can isolate what is causing the growth in number of
objects (if any). I would suggest first remove the GUI and replace it with
some dummy functions, to stress your core logic.

Note that "top" isn't a very reliable tool, as memory fragmentation and
other factors can cause your process' visible size to grow even though
Python's memory consumption may be stable. There are dedicated Python tools
for finer analysis, such as tracemalloc, which is standard on 3.4 and available
as a backport for older versions:

https://docs.python.org/3/library/tracemalloc.html
http://pytracemalloc.readthedocs.org/

But regardless of such tools, the approach above (try to decompose your
workload into separate parts until your find the culprit) is highly recommended.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: __qualname__ in python 3.3

[Antoine Pitrou]

Hi,

ISE Development  gmail.com> writes:
> 'code' object 'function' object
>   
> co_name: test __qualname__: test
> co_name: T__qualname__: T
> co_name: method   __qualname__: test..T.method
> 
> The second call corresponds to the class definition and not the call to the 
> constructor (which is in fact a call to 'object.__init__', a C function 
> hence not traced as a 'call' event - I checked this by disassembling the 
> code object).

There's nothing wrong here. That's just the way things are implemented
internally. This may change in the future without prior notice, so
you shouldn't rely on it.

If you want to dig more, you have to look at how the hidden function ("T")
works:

>>> def f():
...   class T: pass
... 
>>> f.__code__.co_consts
(None, ", line 2>, 'T')
>>> dis.dis(f.__code__.co_consts[1])
  2   0 LOAD_NAME0 (__name__)
  3 STORE_NAME   1 (__module__)
  6 LOAD_CONST   0 ('f..T')
  9 STORE_NAME   2 (__qualname__)
 12 LOAD_CONST   1 (None)
 15 RETURN_VALUE

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

[ANN] pathlib 1.0.1

[Antoine Pitrou]

Hello,

I am announcing the release of pathlib 1.0.1.  This version makes pathlib
Python 2.6-compatible. Note that 2.6 compatibility may not have been as
well tested as more recent Python versions (especially on non-Unix 
platforms).

As a reminder, the standalone (PyPI) version of pathlib will not receive
any new features or general improvements. New developments and regular
bug fixes will happen mostly in the Python standard library, and be
publicly available in official Python releases.

See https://pypi.python.org/pypi/pathlib

Overview


pathlib offers a set of classes to handle filesystem paths.  It offers the
following advantages over using string objects:

* No more cumbersome use of os and os.path functions.  Everything can be
  done easily through operators, attribute accesses, and method calls.

* Embodies the semantics of different path types.  For example, comparing
  Windows paths ignores casing.

* Well-defined semantics, eliminating any warts or ambiguities (forward vs.
  backward slashes, etc.).

Requirements


Python 3.2 or later is recommended, but pathlib is also usable with Python
2.6 and 2.7.

Install
---

In Python 3.4, pathlib is now part of the standard library.  For Python 3.3
and earlier, ``easy_install pathlib`` or ``pip install pathlib`` should do
the trick.

Changelog for version 1.0.1
---

- Pull request #4: Python 2.6 compatibility by eevee.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Everything you did not want to know about Unicode in Python 3

[Antoine Pitrou]

Terry Reedy  udel.edu> writes:
> 
> On 5/13/2014 8:53 PM, Ethan Furman wrote:
> > On 05/13/2014 05:10 PM, Steven D'Aprano wrote:
> >> On Tue, 13 May 2014 10:08:42 -0600, Ian Kelly wrote:
> >>
> >>> Because Python 3 presents stdin and stdout as text streams however, it
> >>> makes them more difficult to use with binary data, which is why Armin
> >>> sets up all that extra code to make sure his file objects are binary.
> >>
> >> What surprises me is how hard that is. Surely there's a simpler way to
> >> open stdin and stdout in binary mode? If not, there ought to be.
> >
> > Somebody already posted this:
> >
> > https://docs.python.org/3/library/sys.html#sys.stdin
> >
> > which talks about .detach().
> 
> I sent a message to Armin about this.

And the documentation has now been fixed:
http://bugs.python.org/issue21364

So something *can* come out of a python-list rantfest, it seems.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: imaplib: how to specify SSL/TLS protocol version?

[Antoine Pitrou]

Grant Edwards  invalid.invalid> writes:
> 
> Experiments show that when calling ssl.wrap_socket() I have to specify
> ssl_version=PROTOCOL_TLSv1 to avoid the above error.
> 
> How do I tell imaplib to use TLS1 instead of SSL3?

Use Python 3 and pass the ssl_context parameter to IMAP_SSL:
https://docs.python.org/3.3/library/imaplib.html#imaplib.IMAP4_SSL

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [ANN] pathlib 1.0

[Antoine Pitrou]

Oh, and of course it is published on PyPI:
https://pypi.python.org/pypi/pathlib/

Regards

Antoine.

-- 
https://mail.python.org/mailman/listinfo/python-list

[ANN] pathlib 1.0

[Antoine Pitrou]

Hello,

I am announcing the release of pathlib 1.0.  This version brings pathlib
up to date with the official Python 3.4 release, and also fixes a couple of 
2.7-specific issues.  Detailed changelog can be found further below.

In the future, I expect the standalone (PyPI) version of pathlib to receive
little to no updates, except if severe issues are found. New developments
and regular bug fixes will happen mostly in the Python standard library,
and be publicly available in official Python releases.

Overview


pathlib offers a set of classes to handle filesystem paths.  It offers the
following advantages over using string objects:

* No more cumbersome use of os and os.path functions.  Everything can be
  done easily through operators, attribute accesses, and method calls.

* Embodies the semantics of different path types.  For example, comparing
  Windows paths ignores casing.

* Well-defined semantics, eliminating any warts or ambiguities (forward vs.
  backward slashes, etc.).

Requirements


Python 3.2 or later is recommended, but pathlib is also usable with Python 2.7.

Install
---

In Python 3.4, pathlib is now part of the standard library.  For Python 3.3
and earlier, ``easy_install pathlib`` or ``pip install pathlib`` should do
the trick.

Changelog for version 1.0
-

- Python issue #20765: Add missing documentation for PurePath.with_name()
  and PurePath.with_suffix().
- Fix test_mkdir_parents when the working directory has additional bits
  set (such as the setgid or sticky bits).
- Python issue #20111: pathlib.Path.with_suffix() now sanity checks the
  given suffix.
- Python issue #19918: Fix PurePath.relative_to() under Windows.
- Python issue #19921: When Path.mkdir() is called with parents=True, any
  missing parent is created with the default permissions, ignoring the mode
  argument (mimicking the POSIX "mkdir -p" command).
- Python issue #19887: Improve the Path.resolve() algorithm to support
  certain symlink chains.
- Make pathlib usable under Python 2.7 with unicode pathnames (only pure
  ASCII, though).
- Issue #21: fix TypeError under Python 2.7 when using new division.
- Add tox support for easier testing.


Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Thread._stop() behavior changed in Python 3.4

[Antoine Pitrou]

Hi,

Felix Yan  gmail.com> writes:
> 
> A minimized snippet to reproduce:
> 
> #!/usr/bin/python
> import threading
> def stale():
> import time
> time.sleep(1000)
> t = threading.Thread(target=stale)
> t.start()
> t._stop()
> 
> This works correctly with Python 3.3, the program exits immediately after 
> t._stop() called, and no exception was raised.

Basically what you are doing is abusing a private method because you want
to make the thread daemonic after it was started (a daemonic thread is
not waited for at interpreter exit). Please do note one thing: the _stop()
method does *not* actually stop the thread; it just marks it stopped, but
the underlying OS thread continues to run (and may indeed continue to
execute Python code until the interpreter exits).

So the obvious "solution" here is to mark the thread daemonic before 
starting it.

A possible related improvement would be to relax the contraints on 
Thread.daemon to allow setting the flag on a running thread?

That said, daemon threads (or abuse of the _stop() method as you did) can
lead to instabilities and oddities as some code will continue executing while 
the interpreter starts shutting down. This has been improved but perhaps
not totally solved in recent interpreter versions. A fully correct solution
would involve gracefully telling the thread to shut down, via a boolean
flag, an Event, a file descriptor or any other means.

(if you are interested in this, please open a new issue at 
http://bugs.python.org)

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What does gc.get_objects() return?

[Antoine Pitrou]

Chris Angelico  gmail.com> writes:
> 
> It's not strictly an implementation detail, beyond that there are
> certain optimizations. For instance...
> 
> >   For CPython 3.4 I guess strings and other atomic types such as ints are
> > not, as well as raw object() instances. Custom class instances on the other
> > hand seem to be under GC control.
> 
> ... strings and ints should never be listed, and custom objects should
> always be listed, but I'd say the non-tracking of object() would be an
> implementation-specific optimization.

These are all implementation details, tied to the fact that the primary
object reclaim mechanism in CPython is reference counting. Other 
implementations may use a full GC and gc.get_objects() may then also
return strings and other "atomic" objects (but the implementation may
also elicit to hack get_objects() in order to closely mimick CPython).

All in all, though, gc.get_objects() is an expensive function call (it
will walk the entire graph of objects tracked by the GC, which can be very
large in non-trivial applications), so it's really only useful for
debugging (and, I'd add, for low-level debugging). In most situations,
gc.get_objects() is certainly the wrong tool to use.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: which async framework?

[Antoine Pitrou]

Sturla Molden  gmail.com> writes:
> 
> Antoine Pitrou  pitrou.net> wrote:
> 
> > Yes, why use a library when you can rewrite it all yourself?
> 
> This assumes something equivalent to the library will have to be written.
> But if it can be replaced with something very minimalistic it is just
> bloat. I would also like to respond that the select module and pywin32 are
> libraries.

Lower-level ones, though. Just like C malloc() is lower-level than
Python's memory management.

This is the usual assumption that high-level libraries are made of useless
cruft piled up by careless programmers. But there are actual reasons 
why these frameworks have a significant amount of code, and people who 
decide to ignore those reasons are simply bound to reimplement 
non-trivial parts of those frameworks in less careful and less tested
ways (and they have to maintain it themselves afterwards).

What irks me with your response is that you phrased it as though writing
a good event loop was an almost trivial thing to do, which it is not
once you start considering multiple use cases and constraints. It is
quite irresponsible to suggest people don't need the power of network
programming frameworks. I would personally distrust a programmer who
chooses to reimplement their own event loop, except if they happen to
have a very brilliant track record.

> Another thing is that co-routines and "yield from" statements just
> makes it hard to follow the logic of the program. 

That's an optional part of Tornado and asyncio, though. It is very
reasonable to use Tornado or asyncio and still code in purely
callback-driven style (even though Guido himself prefers the coroutine
style).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: which async framework?

[Antoine Pitrou]

Sturla Molden  gmail.com> writes:
> 
> Chris Withers  simplistix.co.uk> wrote:
> > Hi All,
> > 
> > I see python now has a plethora of async frameworks and I need to try 
> > and pick one to use from:
> > 
> > - asyncio/tulip
> > - tornado
> > - twisted
> 
> I'd go for using iocp, epoll and kqueue/kevent directly. Why bother to
> learn a framework? You will find epoll and kqueue/kevent in the select
> module and iocp in pywin32.

Yes, why use a library when you can rewrite it all yourself?
Actually, you should probably issue system calls to the kernel directly,
the libc is overrated (as is portability, I suppose).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: which async framework?

[Antoine Pitrou]

Chris Withers  simplistix.co.uk> writes:
> 
> The protocols are all financial (do we really not have a pure-python FIX 
> library?!) but none are likely to have existing python implementations.

If you are mostly writing protocol implementations (aka parsers and
serializers), then you should consider writing them in a way that's
framework-agnostic.

See as an example:
https://pypi.python.org/pypi/obelus/0.1

(if you really want to settle on a single framework and don't mind
supporting old Python versions, then I recommend asyncio)

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLite + FTS (full text search)

[Antoine Pitrou]

Hi,

Mark Summerfield  qtrac.plus.com> writes:
> 
> My guess is that on Debian, the packagers install a full SQLite 3 and the
Python package uses that. But on
> Windows I think the Python packagers bundle their own SQLite (quite
rightly since it might not already be installed).
> 
> I'd like the Windows binary to include SQLite 3 with FTS4 support, but I
don't know how much work that
> involves or if it would make the Python .msi file too big?

You can create a feature request on http://bugs.python.org

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "More About Unicode in Python 2 and 3"

[Antoine Pitrou]

Terry Reedy  udel.edu> writes:
> 
> On 1/6/2014 11:29 AM, Antoine Pitrou wrote:
> 
> > People don't use? According to available figures, there are more
downloads of
> > Python 3 than downloads of Python 2 (Windows installers, mostly):
> > http://www.python.org/webstats/
> 
> While I would like the claim to be true, I do not see 2 versus 3 
> downloads on that page. Did you mean another link?

Just click on a recent month, scroll down to the "Total URLs By kB"
table, and compute the sum of the largest numbers for each Python
version.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "More About Unicode in Python 2 and 3"

[Antoine Pitrou]

Mark Lawrence  yahoo.co.uk> writes:
> [...]
> 
> And as I started this thread, I'll say what I please, throwing my toys 
> out of my pram in just the same way that your pal Armin is currently doing.

I'll join Ned here: please stop it. You are doing a disservice to
everyone.

Thanks in advance

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "More About Unicode in Python 2 and 3"

[Antoine Pitrou]

Ned Batchelder  nedbatchelder.com> writes:
> 
> 
> I never said they were the whole community, of course. But they are not 
> outliers either. By your own statistics above, 23% of respondents think 
> Python 3 was a mistake.  Armin and Kenneth are just two very visible 
> people.

Indeed, they are two very visible people.

> I'm not creating rock stars.  I'm acknowledging that these two people 
> are listened to by many others.  It sounds like part of your effort to 
> avoid rockstars is to ignore any one person's specific feedback?  I must 
> be misunderstanding what you mean.

I am not trying to ignore "any one person's specific feedback". I am 
ignoring your claim that we should give Armin's blog posts an
extraordinary importance because he is "revered".

Speaking of which, posting blog articles is not the preferred way to
give feedback. There are ample community resources for that. I am
irritated that we are apparently supposed to be monitoring blog posts, 
Twitter feeds and whatnot for any sign of dissent, and immediately react
to a criticism that wasn't even voiced directly to us.

> You are being given detailed specific feedback from intelligent 
> dedicated customers that many people listen to,

Could you please stop talking about customers? We are not selling
Python to anyone (*). Writing open source software as a volunteer is 
not supposed to be a sacrificial activity where we will bow with
extreme diligence to the community's every outburst. Please try to
respect us.

((*) Wikipedia: "A customer (sometimes known as a client, buyer, or 
purchaser) is the recipient of a good, service, product, or idea,
obtained from a seller, vendor, or supplier for a monetary or other 
valuable consideration")

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "More About Unicode in Python 2 and 3"

[Antoine Pitrou]

Chris Angelico  gmail.com> writes:
> 
> On Tue, Jan 7, 2014 at 3:29 AM, Antoine Pitrou  pitrou.net>
wrote:
> > People don't use? According to available figures, there are more
downloads of
> > Python 3 than downloads of Python 2 (Windows installers, mostly):
> > http://www.python.org/webstats/
> >
> 
> Unfortunately, that has a massive inherent bias, because there are
> Python builds available in most Linux distributions - and stats from
> those (like Debian's popcon) will be nearly as useless, because a lot
> of them will install one or the other (probably 2.x) without waiting
> for the user (so either they'll skew in favour of the one installed,
> or in favour of the one NOT installed, because that's the only one
> that'll be explicitly requested). It's probably fairly accurate for
> Windows stats, though, since most people who want Python on Windows
> are going to come to python.org for an installer.

Agreed, but it's enough to rebut the claim that "people don't use
Python 3". More than one million Python 3.3 downloads per month under
Windows is a very respectable number (no 2.x release seems to reach
that level).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "More About Unicode in Python 2 and 3"

[Antoine Pitrou]

Ned Batchelder  nedbatchelder.com> writes:
> 
> You can look through his problems and decide that he's "wrong," or that 
> he's "ranting," but that doesn't change the fact that Python 3 is 
> encountering friction.  What happens when a significant fraction of your 
> customers are "wrong"?

Well, yes, there is some friction and this is quite expectable, when
shipping incompatible changes. Other pieces of software have undergone a
similar process (e.g. Apache 1.x -> Apache 2.x).

(the alternative is to maintain a piece of software that sticks with obsolete
conventions, e.g. emacs)

> Core developers: I thank you for the countless hours you have devoted to 
> building all of the versions of Python.  I'm sure in many ways it's a 
> thankless task.  But you have a problem.  What's the point in being 
> right if you end up with a product that people don't use?

People don't use? According to available figures, there are more downloads of
Python 3 than downloads of Python 2 (Windows installers, mostly):
http://www.python.org/webstats/

The number of Python 3-compatible packages has been showing a constant and
healthy increase for years:
http://dev.pocoo.org/~gbrandl/py3.html

And Dan's survey shows 77% of respondents think Python 3 wasn't a mistake:
https://wiki.python.org/moin/2.x-vs-3.x-survey

> Maybe there are core developers who are trying hard to solve the 
> problems Kenneth and Armin are facing.  It would be great if that work 
> was more visible.  I don't see it, and apparently Armin doesn't either.

While this is being discussed:
https://mail.python.org/pipermail/python-dev/2014-January/130923.html

I would still point out that "Kenneth and Armin" are not the whole Python
community. Your whole argument seems to be that a couple "revered" (!!)
individuals should see their complaints taken for granted. I am opposed to 
rockstarizing the community.

Their contribution is always welcome, of course.

(as for network programming, the people working on and with asyncio don't
seem to find Python 3 terrible)

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "More About Unicode in Python 2 and 3"

[Antoine Pitrou]

On Sun, 05 Jan 2014 08:22:38 -0500
Ned Batchelder  wrote:
> On 1/5/14 8:14 AM, Mark Lawrence wrote:
> > http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/
> >
> > Please don't shoot the messenger :)
> >
> 
> With all of the talk about py 2 vs. py3 these days, this is the blog 
> post that I think deserves the most real attention.  I haven't had to do 
> the kind of coding that Armin is talking about, but I've heard more than 
> one person talk about the difficulty of it in Python 3.
> 
> If anyone wants Python 3 uptake improved, the best thing would be to 
> either explain to Armin how he missed the easy way to do what he wants 
> (seems unlikely), or advocate to the core devs why they should change 
> things to improve this situation.

Sometimes the best way to "advocate to the core devs" is to do part of
the work, though.

There are several people arguing for %-formatting or .format() on
bytes, but that still lacks a clear description of which formatting
codes would be supported, with which semantics.
(see e.g. http://bugs.python.org/issue3982)

As for the rest of Armin's rant, well, it's a rant. "In some cases
Python 3 is a bit less practical than Python 2" doesn't equate to
"Python 3 is broken and 2.8 should be released instead".

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Blog "about python 3"

[Antoine Pitrou]

Hi,

Robin Becker  reportlab.com> writes:
> 
> For fairly sensible reasons we changed the internal default to use unicode 
> rather than bytes. After doing all that and making the tests compatible
etc etc 
> I have a version which runs in both and passes all its tests. However, for 
> whatever reason the python 3.3 version runs slower
> 
> 2.7 Ran 223 tests in 66.578s
> 
> 3.3 Ran 223 tests in 75.703s

Running a test suite is a completely broken benchmarking methodology.
You should isolate workloads you are interested in and write a benchmark
simulating them.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Blog "about python 3"

[Antoine Pitrou]

Steven D'Aprano  pearwood.info> writes:
> 
> I expect that as excuses for not migrating get fewer, and the deadline for
> Python 2.7 end-of-life starts to loom closer, more and more haters^W
> Concerned People will whine about the lack of version 2.8 and ask for
> *somebody else* to fork Python.
> 
> I find it, hmmm, interesting, that so many of these Concerned People who say
> that they're worried about splitting the Python community[1] end up
> suggesting that we *split the community* into those who have moved forward
> to Python 3 and those who won't.

Indeed. This would be extremely destructive (not to mention alienating the
people doing *actual* maintenance and enhancements on Python-and-its-stdlib,
of which at least 95% are committed to the original plan for 3.x to slowly
supercede 2.x).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: pathlib 0.97 released

[Antoine Pitrou]

Hello,

As you may know, pathlib has recently been accepted for inclusion into
the Python 3.4 standard library.  You can view the new module's
documentation here: http://docs.python.org/dev/library/pathlib.html

As part of the inclusion process, many API changes were done to the
original pathlib API (*) so as to satisfy the python-dev community's 
comments and complaints. Yet, the PyPI version of pathlib (0.8) was still 
featuring the legacy API.

(*) INCLUDING COMPATIBILITY-BREAKING CHANGES

I am now releasing version 0.97 of pathlib, which brings the standalone
PyPI module up-to-date with the standard library version. The version
number reflects this. It isn't yet "officially" stable, since Python 3.4
hasn't been released yet; I plan to do a standalone 1.0 release by the
time Python 3.4 is released.

If you are not under Python 3.4:

Install
---

Download from https://pypi.python.org/pypi/pathlib/
or use `pip` or `easy_install`.
(be sure to *force* the upgrade if pathlib 0.8 or earlier is already
installed!)

Documentation
-

Standalone documentation is at https://pathlib.readthedocs.org/

Changes from 0.8
-

The API changes are too long to list; if you are already a pathlib user,
I would recommend you read the documentation and find out what has changed 
for you (your code will likely break loudly, anyway!).


Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Self-defence

[Antoine Pitrou]

Zero Piraeus  etiol.net> writes:
> 
> I don't believe that killfiles are a sufficient response in this
> situation.
> 
> I can, of course, stop Nikos' posts reaching me, and without too much
> hassle also stop replies to his posts reaching me. He would, however,
> continue to pollute the list in public, and his posts, whether replied
> to or not at the volume he's now sending them, would continue to damage
> the reputation of the list and, ultimately, I think possibly kill it.

As an occasional reader (and even more occasional poster), I agree with
this. As far as I'm concerned, if I were the list owner and if it were 
easy to ban him, I would already have banned him for a long time. I 
believe it is not desirable to let poisonous posters do their thing,
even if "they might improve". The time spent trying to improve a troll
is time not spent caring about and helping other, more benevolent posters, 
which makes it really harmful to the community (not only pointless) trying
to improve a troll.

As for the "but banning him would impede free expression" argument,
I'd say that people who want a "perfectly free" discussion space can
always open their own. We'll see how well it fares in practice, but
I'm not holding my hopes very high :-)

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

[OT] Re: Python Front-end to GCC

[Antoine Pitrou]

Steven D'Aprano  pearwood.info> writes:
> 
> On Fri, 25 Oct 2013 21:36:42 +0100, Mark Lawrence wrote:
> 
> > Mind you, the thought of a bot with a Ph.D. is mind boggling.
> 
> You can buy degrees on the Internet quite cheaply:
> 
> http://en.wikipedia.org/wiki/List_of_animals_with_fraudulent_diplomas
> 
> PhD's are more expensive, which leads me to think that Mark Jenssen is 
> being a tad flexible with the truth when he claims to have one.

He could be an upper-class twit. He would certainly have his chance in
the yearly competition ;-)

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.x ever become the actual standard?

[Antoine Pitrou]

>  gmail.com> writes:
> 
> I am starting to have doubts as to whether Python 3.x will ever be
actually adopted by the Python community at
> large as their standard.

We're planning to start the switch on 25th December 2013, 14h UTC.
It should be finished at most 48 hours later. You should expect some 
intermittent problems during the first few hours, but at the end
all uses of Twisted will be replaced with Tornado and asyncio (and
camelCase methods will have ceased to be).

By the way, if you want to join us, one week later we'll also switch
the Internet to IPv6 (except Germany).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Front-end to GCC

[Antoine Pitrou]

Steven D'Aprano  pearwood.info> writes:
> 
> On Tue, 22 Oct 2013 08:55:15 +, Antoine Pitrou wrote:
> 
> > If you don't implement exec() and eval() then people won't be able to
> > use namedtuples, which are a common datatype factory.
> 
> Philip could always supply his own implementation of namedtuple that 
> doesn't use exec.
> 
> But either way, if he doesn't implement eval and exec, what he has is not 
> Python, but a subset of Python. Perhaps an interesting and useful subset.

If you go that way, we already have Cython (which is both a subset and
superset of Python, although I don't know if it's still a strict subset these
days).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Front-end to GCC

[Antoine Pitrou]

Philip Herron  googlemail.com> writes:
> 
> Its interesting a few things come up what about:
> 
> exec and eval. I didn't really have a good answer for this at my talk at
PYCon IE 2013 but i am going to say no. I am
> not going to implement these. Partly because eval and exec at least to me
are mostly from developing
> interpreters as a debugging exercise so the test doesn't have to invoke
the program properly and feed in
> strings to interpret at least thats what i have done in the past with an
virtual machine i wrote before gccpy.

If you don't implement exec() and eval() then people won't be able to use
namedtuples, which are a common datatype factory.

As for the rest: well, good luck writing an AOT compiler producing
interesting results on average *pure* Python code. It's already been tried
a number of times, and has generally failed. Cython mitigates the issue by
exposing a superset of Python (including type hints, etc.).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tab completion in Python3.4

[Antoine Pitrou]

Steven D'Aprano  pearwood.info> writes:
> 
> I don't consider either of these solutions to be satisfactory. If you 
> agree, I urge you to try it out for yourself, and then leave a comment on 
> the bug tracker asking for tab completion to still insert tabs at the 
> beginning of the line:

Such a change is much more likely to happen if someone actually proposes
a patch for it, rather than merely "ask for it". I don't think anyone is
ideologically opposed to it.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: Obelus 0.1 -- Asterisk AMI / AGI implementation

[Antoine Pitrou]

Hello,

I'm pleased to announce the first release of Obelus, a MIT-licensed
library to interact with Asterisk using the AMI and AGI protocols.

This is version 0.1, and as such some APIs are a bit draftish and not 
guaranteed to be stable accross future releases. Also, documentation is
far from exhaustive.

Quick links
---

* Project page: https://pypi.python.org/pypi/obelus/
* Source code, issue tracker: https://bitbucket.org/optiflowsrd/obelus
* Documentation (incomplete): https://obelus.readthedocs.org

Features


* Python 2 and Python 3 support.
* AMI, FastAGI and Async AGI support.
* Event-driven API friendly towards non-blocking ("async") network
  programming styles.
* :pep:`3156`-style protocol implementations.
* Framework-agnostic.
* Adapters for the `Tornado`_, `Twisted`_, `Tulip`_ network programming
  frameworks.
* Unit-tested.

Requirements


* Python 2.7, 3.2 or later.


Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [python-committers] [RELEASED] Python 3.4.0a2

[Antoine Pitrou]

Le Mon, 9 Sep 2013 14:30:50 +0200,
Victor Stinner  a écrit :
> 2013/9/9 Larry Hastings :
> > Python 3.4 includes a range of improvements of the 3.x series,
> > including hundreds of small improvements and bug fixes.  Major new
> > features and changes in the 3.4 release series so far include:
> >
> > * PEP 446, changing file descriptors to not be inherited by default
> >in subprocesses
> 
> The title of the PEP is "Make newly created file descriptors
> non-inheritable". It has an impact on all functions creating files and
> sockets not only the subprocess module.

I don't think Larry's description is wrong. "Non-inheritable" is a
shorthand for "non-inheritable in subprocesses" with "subprocesses"
taken in the general sense (i.e. not only created with the subprocess
module).

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [RELEASED] Python 3.4.0a2

[Antoine Pitrou]

Le Mon, 9 Sep 2013 08:16:06 -0400,
Brett Cannon  a écrit :
> On Mon, Sep 9, 2013 at 8:02 AM, Larry Hastings 
> wrote:
> 
> >
> > On behalf of the Python development team, I'm chuffed to announce
> > the second alpha release of Python 3.4.
> >
> > This is a preview release, and its use is not recommended for
> > production settings.
> >
> > Python 3.4 includes a range of improvements of the 3.x series,
> > including hundreds of small improvements and bug fixes.  Major new
> > features and changes in the 3.4 release series so far include:
> >
> > * PEP 435, a standardized "enum" module
> > * PEP 442, improved semantics for object finalization
> > * PEP 443, adding single-dispatch generic functions to the standard
> > library
> > * PEP 445, a new C API for implementing custom memory allocators
> > * PEP 446, changing file descriptors to not be inherited by default
> >in subprocesses
> > * PEP 447, a new magic method for metaclasses (``__typelookup__``)
> > * PEP 448, making automatic sequence unpacking more general
> >
> 
> Those last two PEPs are still in draft form and not accepted nor have
> any committed code yet.

Unless Larry enthusiastically sneaked them into the release.

Regards

Antoine.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pair of filenos read/write each other?

[Antoine Pitrou]

Jack Bates  nottheoilrig.com> writes:
> > 
> > An alternative is to use multiprocessing.Pipe():
> > http://docs.python.org/3.3/library/multiprocessing.html#multiprocessing.Pipe
> > 
> > In any case, Python doesn't lack facilities for doing what you want.
> 
> Thank you for your help, I need to satisfy an interface that requires a single
> file descriptor number that can be both read from and written to. Is it
> possible with any of the solutions you pointed out to get a single file
> descriptor number for each end?

Yes, it is what multiprocessing.Pipe() provides:
http://docs.python.org/3.3/library/multiprocessing.html#multiprocessing.Connection.fileno

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pair of filenos read/write each other?

[Antoine Pitrou]

Nobody  nowhere.com> writes:
> 
> On Tue, 13 Aug 2013 16:10:41 -0700, Jack Bates wrote:
> 
> > Is there anything like os.pipe() where you can read/write both ends?
> 
> There's socket.socketpair(), but it's only available on Unix.
> 
> Windows doesn't have AF_UNIX sockets, and anonymous pipes (like the ones
> created by os.pipe()) aren't bidirectional.

I'm not sure I understand the problem: you can just create two pair of pipes
using os.pipe().
If that's too low-level, you can wrap the fds using BufferedRWPair:
http://docs.python.org/3.3/library/io.html#io.BufferedRWPair

(actual incantation would be:
 r1, w1 = os.pipe()
 r2, w2 = os.pipe()

 end1 = io.BufferedRWPair(io.FileIO(r1, 'r'), io.FileIO(w2, 'w'))
 end2 = io.BufferedRWPair(io.FileIO(r2, 'r'), io.FileIO(w1, 'w'))

 end1.write(b"foo")
 end1.flush()
 end2.read(3)  # -> return b"foo"
)

An alternative is to use multiprocessing.Pipe():
http://docs.python.org/3.3/library/multiprocessing.html#multiprocessing.Pipe

In any case, Python doesn't lack facilities for doing what you want.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with psycopg2, bytea, and memoryview

[Antoine Pitrou]

Frank Millman  chagford.com> writes:
> 
> Thanks for that, Antoine. It is an improvement over tobytes(), but i am 
> afraid it is still not ideal for my purposes.

I would suggest asking the psycopg2 project why they made this choice, and
if they would reconsider. Returning a memoryview doesn't make much sense IMHO.

For example, the standard sqlite3 module returns bytes for BLOB columns,
and str for TEXT columns:
http://docs.python.org/3.4/library/sqlite3.html#introduction

> Can anyone explain *why* the results do not compare equal? If I understood 
> the problem, I might be able to find a workaround.

Well, under recent Python versions, they should compare equal:

Python 3.2.3 (default, Oct 19 2012, 19:53:16) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> memoryview(b"abc") == b"abc"
True


Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with psycopg2, bytea, and memoryview

[Antoine Pitrou]

Frank Millman  chagford.com> writes:
> 
> I have some binary data (a gzipped xml object) that I want to store in a 
> database. For PostgreSQL I use a column with datatype 'bytea', which is 
> their recommended way of storing binary strings.
> 
> I use psycopg2 to access the database. It returns binary data in the form of 
> a python 'memoryview'.
> 
[...]
> 
> Using MS SQL Server and pyodbc, it returns a byte string, not a memoryview, 
> and it does compare equal with the original.
> 
> I can hack my program to use tobytes(), but it would add complication, and 
> it would be database-specific. I would prefer a cleaner solution.

Just cast the result to bytes (`bytes(row[1])`). It will work both with bytes
and memoryview objcts.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MyOpenID.com no longer supported

[Antoine Pitrou]

Le Mon, 29 Jul 2013 00:55:53 -0700,
Ethan Furman  a écrit :
> Excerpt from http://meta.stackoverflow.com/q/190442/176681:
> 
> Janrain no longer actively supports MyOpenID, and announced on
> Twitter that their users should proceed with caution.
> 
> This decision was made by Janrain, [snip]
> 
> I know the Python bug tracker allows MyOpenID logins; if that is your
> only login method you should add another that doesn't depend on
> MyOpenID.

I don't understand. The tracker allows any (sufficently compliant)
OpenID provider, not just "MyOpenID" or
"MyCorporateOpenIDWithTrademarks".

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why on CentOS, python consumes too much memory ?

[Antoine Pitrou]

 gmail.com> writes:
> 
> Hi:
> 
>Previously, we found that our python scripts consume too much memory.
So I use 
python's resource module to
> restrict RLIMIT_AS's soft limit and hard limit to 200M.
> On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 
6.2(x86_64)+python2.6.6, it reports memory
> error(exceeding 200M).

Take a look at http://www.selenic.com/smem/ for accurate measurement of
actual memory consumption under Linux. Virtual memory size is generally
useless for this purpose.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Stack Overflow moderator “animuson”

[Antoine Pitrou]

Joshua Landau  landau.ws> writes:
> 
> > The same with Unicode. We hate French people,
> 
> And for good damn reason too. They're ruining our language, á mon avis.

We do!


Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ssl handshake operation timed out on Python 3.3.2

[Antoine Pitrou]

Benedict Verheyen  gmail.com> writes:
> 
> Hi,
> 
> for a project, I need to post data to some aspx pages.
> The aspx pages are hosted by another company.
> I develop on a virtual Debian Wheezy (Virtual box) running on Windows.
> I couldn't get the code to run either on Windows nor Linux.
> 
> On my testserver (also a Debian Linux Wheezy) however, the code works with 
> Python 2.7.3 but not with Python 3.3.2. Another difference is that the 
> testserver has a direct ip, so nothing in between the machine and the 
> internet. I checked the firewall, the traffic passes without a problem.
> 
> The error I get is that the SSL handshake operation timed out.
> I tried some code that specifies the ssl protocol but the results are the 
> same.

This may be a IIS-specific problem.
Take a look at
http://stackoverflow.com/questions/16365483/iis-7-5-mercurial-setup-ignoring-maxallowedcontentlength
http://bz.selenic.com/show_bug.cgi?id=3905
http://bugs.python.org/issue17948

Otherwise, dumping network traffic with Wireshark could give you some hints
at to what is different between the SSL handshakes in the two setups.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why is regex so slow?

[Antoine Pitrou]

Roy Smith  panix.com> writes:
> 
> Every line which contains 'ENQ' also matches the full regex (61425
> lines match, out of 2.1 million total).  I don't understand why the
> first way is so much slower.

One invokes a fast special-purpose substring searching routine (the
str.__contains__ operator), the other a generic matching engine able to
process complex patterns. It's hardly a surprise for the specialized routine
to be faster. That's like saying "I don't understand why my CPU is slower
than my GPU at calculating 3D structures".

That said, there may be ways to improve the regex engine to deal with such
special cases in a speedier way. But there will still be some overhead related
to the fact that you are invoking a powerful generic engine rather than a
lean and mean specialized routine.

(to be fair, on CPython there's also the fact that operators are faster
than method calls, so some overhead is added by that too)

> Once the regex is compiled, you should have a state machine pattern
> matcher.  It should be O(n) in the length of the input to figure out
> that it doesn't match as far as "ENQ".  And that's exactly how long it
> should take for "if 'ENQ' not in line" to run as well.

You should read again on the O(...) notation. It's an asymptotic complexity,
it tells you nothing about the exact function values at different data points.
So you can have two O(n) routines, one of which always twice faster than the
other.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: usage of os.posix_fadvise

[Antoine Pitrou]

Hi,

Wolfgang Maier  biologie.uni-freiburg.de> writes:
> 
> Dear all,
> I was just experimenting for the first time with os.posix_fadvise(), which
> is new in Python3.3 . I'm reading from a really huge file (several GB) and I
> want to use the data only once, so I don't want OS-level page caching. I
> tried os.posix_fadvise with the os.POSIX_FADV_NOREUSE and with the
> os.POSIX_FADV_DONTNEED flags, but neither seemed to have any effect on the
> caching behaviour of Ubuntu (still uses all available memory to page cache
> my I/O).
> Specifically, I was trying this:
> 
> import os
> fd = os.open('myfile', os.O_RDONLY)
> # wasn't sure about the len parameter in fadvise,
> # so thought I just use it on the first 4GB
> os.posix_fadvise(fd, 0, 40, os.POSIX_FADV_NOREUSE) # or DONTNEED

The Linux version of "man posix_fadvise" probably holds the answer:

"In kernels before 2.6.18, POSIX_FADV_NOREUSE had the same semantics
as POSIX_FADV_WILLNEED.  This was probably a bug; since kernel
2.6.18, this flag is a no-op."

"POSIX_FADV_DONTNEED attempts to free cached pages associated with the
specified region.  This is useful, for example, while streaming large
files.  A program may periodically request the kernel to free cached
data that has already been used, so that more useful cached pages  are
not discarded instead."

So, in summary:

- POSIX_FADV_NOREUSE doesn't do anything on (modern) Linux kernels
- POSIX_FADV_DONTNEED must be called *after* you are done with a range of
  data, not before you read it (note that I haven't tested to confirm it :-))

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ordered dictionaries compared

[Antoine Pitrou]

Dan Stromberg  gmail.com> writes:
> 
> What kind of ordered dictionaries?  Sorted by key.

Calling them "sorted dictionaries" avoids any confusions with Python's
standard OrderedDict class:
http://docs.python.org/3.3/library/collections.html#ordereddict-objects

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The node.js Community is Quietly Changing the Face of Open Source

[Antoine Pitrou]

rusi  gmail.com> writes:
> 
> Just what I said: ecosystem matters.  We may or may not argue about
> "more than language", but it surely matters. Some examples:
> 
> 1. In the link that Roderick originally posted there is a long comment
> that adds perl to the languages the author discussed. As a language
> perl is… um well… its perl.  Yet when perl wins its because CPAN
> wins.
> 
> 2. Haskell as a language is very well designed. However its package
> system -- cabal+hackage -- is completely broken.

I think you are deluded. Haskell may very well designed from a language
theoretist's
point of view, but I suspect most average programmers would find it a hell
to code in.

Regards

Antoine.



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Unicode issue with Python v3.3

[Antoine Pitrou]

rusi  gmail.com> writes:
> 
> Hmm I see some cut-paste goofup on my part.
> I was meaning to juxtapose this thread where we put up with inordinate
> amount of nonsense from OP
> along with the recent thread in which a newcomer who thinks he has
> found a bug in pdb is made fun of.
> 
> Then thought better of it and deleted the stuff.
> However I did not do a good delete-job so I better now say what I
> avoided saying:
> 
> If those who habitually post rubbish are given much of our time and
> effort,
> whereas newcomers and first-timers are treated rudely, the list begins
> to smell like a club of old farts.

+1. If you think you have something intelligent to say to jmfauth,
you might as well start a private discussion with him.

As far as I'm concerned, python-list is *already* of club of old
farts. Many regular posters are more interested in "being right on the
Internet" rather than helping people out.

(this is where the StackOverflow mechanics probably work better, sadly)

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: standalone vs embedded interpreter

[Antoine Pitrou]

Nick Gnedin  gmail.com> writes:
> I expect it to behave the same way as if I was running it as a 
> standalone program. On Windows this is indeed the case, but on my Linux 
> box (Python 3.3.1 (default, Apr  8 2013, 22:33:31) [GCC 4.1.2 20080704 
> (Red Hat 4.1.2-51)]) I get a different behavior in handling console 
> input. A standalone interpreter cycles though the input history when I 
> use up and down arrows - say, I type this code:
> 
>  >>> 1
> 1
>  >>> a=4
>  >>> a
> 4
> 
> If I now press an  key in a standalone interpreter, I get 'a' placed 
> at the prompt (my previous command). However, in an embedded code I get
> 
>  >>> ^[[A
> 
> put at the prompt - does anyone know what I am doing wrong?

Are stdin and stdout both interactive? That is, are isatty(fileno(stdin))
and isatty(fileno(stdout)) both true?

If you want to debug, take a look at PyOS_Readline() in Parser/myreadline.c.
It probably holds the answer to your question.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speeding up Python's exit

[Antoine Pitrou]

Dave Angel  davea.name> writes:
> 
> Note he didn't say the python buffers would be flushed.  It's the OS 
> buffers that are flushed.

Now please read my message again. The OS buffers are *not* flushed according
to POSIX.



-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: pathlib 0.8

[Antoine Pitrou]

pathlib 0.8 has been released at https://pypi.python.org/pypi/pathlib/


Changes
---

- Add PurePath.name and PurePath.anchor.
- Add Path.owner and Path.group.
- Add Path.replace().
- Add Path.as_uri().
- Issue #10: when creating a file with Path.open(), don't set the executable
  bit.
- Issue #11: fix comparisons with non-Path objects.

What is pathlib?


pathlib offers a set of classes to handle filesystem paths.  It offers the
following advantages over using string objects:

* No more cumbersome use of os and os.path functions.  Everything can be
  done easily through operators, attribute accesses, and method calls.

* Embodies the semantics of different path types.  For example, comparing
  Windows paths ignores casing.

* Well-defined semantics, eliminating any warts or ambiguities (forward vs.
  backward slashes, etc.).

Requirements


Python 3.2 or later is recommended, but pathlib is also usable with Python 2.7.


Documentation
-

The full documentation can be read at `Read the Docs
`_.


Contributing


The issue tracker and repository are hosted by `BitBucket
`_.


Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speeding up Python's exit

[Antoine Pitrou]

Grant Edwards  invalid.invalid> writes:
> 
> > I assume that the memory used by the Python process will be reclaimed
> > by the operating system, but other resources such as opened files may
> > not be.
> 
> All open files (including sockets, pipes, serial ports, etc) will be
> flushed (from an OS standpoint) and closed.

According to POSIX, no, open files will not be flushed:

“The _Exit() and _exit() functions shall not call functions registered with 
atexit() nor any registered signal handlers. Open streams shall not be flushed. 
Whether open streams are closed (without flushing) is implementation-defined.”

http://pubs.opengroup.org/onlinepubs/9699919799/functions/_exit.html

(under the hood, os._exit() calls C _exit())

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: IMAP4_SSL and OpenSSL compatibility

[Antoine Pitrou]

W. Martin Borgert  debian.org> writes:
> >
> > There is already the ssl_context option for that:
> > http://docs.python.org/3.3/library/imaplib.html#imaplib.IMAP4_SSL
> 
> Many thanks! Two more questions:
> 
>   1. Is there any plan to backport this Python >= 3.3 feature to
>  Python 2?

No, we don't backport new features to maintenance releases.

>   2. Would the following lines be correct for Python 3.3?
> 
>  >>> import imaplib
>  >>> IMAP4_SSL("192.168.1.1.", ssl_context =  
> SSLContext(ssl.PROTOCOL_SSLv3))

It should be, yes.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Speeding up Python's exit

[Antoine Pitrou]

Steven D'Aprano  pearwood.info> writes:
> 
> I just quit an interactive session using Python 2.7 on Linux. It took in 
> excess of twelve minutes to exit, with the load average going well past 9 
> for much of that time.
> 
> I think the reason it took so long was that Python was garbage-collecting 
> a giant dict with 10 million entries, each one containing a list of the 
> form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it 
> took longer to dispose of the dict (12+ minutes) than it took to create 
> it in the first place (approx 3 minutes, with a maximum load of 4). 
> 
> Can anyone explain why this was so painfully slow, and what (if anything) 
> I can do to avoid it in the future?

You are basically asking people to guess where your performance problem
comes from, without even providing a snippet so that people can reproduce ;)

> I know there is a function os._exit which effectively kills the Python 
> interpreter dead immediately, without doing any cleanup. What are the 
> consequences of doing this? I assume that the memory used by the Python 
> process will be reclaimed by the operating system, but other resources 
> such as opened files may not be.

The OS always disposes of per-process resources when the process terminates
(except if the OS is buggy ;-)). However, file buffers will not be flushed,
atexit handlers and other destructors will not be called, database
transactions will be abandoned (rolled back), etc.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Writing to same file from two threads

[Antoine Pitrou]

Steven D'Aprano  pearwood.info> writes:
> 
> On Wed, 27 Feb 2013 13:26:18 +, Antoine Pitrou wrote:
> 
> > For the record, binary files are thread-safe in Python 3, but text files
> > are not.
> 
> Where is this documented please?

In the documentation, of course ;)
http://docs.python.org/3.3/library/io.html#multi-threading

Regards

Antoine.




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Writing to same file from two threads

[Antoine Pitrou]

Jens Thoms Toerring  toerring.de> writes:
> 
> Paul Rubin  nospam.invalid> wrote:
> > jt  toerring.de (Jens Thoms Toerring) writes:
> > > in garbled output (i.e. having some output from A inside a
> > > line written by B or vice versae) because the "main thread" or
> 
> > Yes they do get garbled like that.  Preferred Python style is put a
> > single thread in charge of all the i/o to that file, and communicate
> > with it by message passing through Queue objects.  That is safer than
> > directly using locks.
> 
> Thank you for confirmig my suspicion But you have induced
> another question: why is using a Queue safer than locking (not
> that I doubt that it might be more elegant etc.). Is it "safer"
> because it's less likely that one gets it wrong (e.g. by for-
> grtting to acquire the lock) or is there something inherently
> unsafe about locks?

For the record, binary files are thread-safe in Python 3, but text files
are not.
Locks are safe if you use them well. As you point out, if you forget
to acquire your lock, or if you devise a situation where there is a
deadlock between competing locks, you can have difficult to diagnose
issues. Queues have their internal locking all done for you.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Do you feel bad because of the Python docs?

[Antoine Pitrou]

Mitya Sirenef  lightbird.net> writes:
> I think the issue with python documentation is that it ignores the 95/5
> rule: 95% of people who land on a module's page are only looking for 5%
> of its information.

The 95/5 rule is generally a fallacy which ignores that the 5% which the
readers are expecting to learn about is not the same 5% from reader to reader.
(*)

Which means that in the end you would really want a diversity of HOWTOs
targeted at different usages of the stdlib. But it is a lot of work to
write *and* maintain.

(*) cf. http://www.joelonsoftware.com/items/2006/12/09.html

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Do you feel bad because of the Python docs?

[Antoine Pitrou]

Steven D'Aprano  pearwood.info> writes:
> 
> It is valuable to contrast and compare the PHP and Python docs:
> 
> http://php.net/manual/en/index.php
> http://www.python.org/doc/

I suppose you should compare it with http://docs.python.org/3/ instead.

> There's no doubt that one of PHP's strengths, perhaps its biggest 
> strength, is the good state of documentation.

My (probably outdated) experience with the PHP docs is that they are very
succinct and don't document failure cases or behavioral details at all.
Sure, if you only care about the big picture, they are good enough.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: IMAP4_SSL and OpenSSL compatibility

[Antoine Pitrou]

W. Martin Borgert  debian.org> writes:
> 
> When I add an ssl_version argument to the call to
> ssl.wrap_socket() in imaplib.IMAP4_SSL.open(), I can connect to
> the Exchange server without problems:
> 
> self.sslobj = ssl.wrap_socket(self.sock, self.keyfile, self.certfile,
>   ssl_version = ssl.PROTOCOL_SSLv3)
> 
> Would it make sense, to make this change in the Python standard
> library?

There is already the ssl_context option for that:
http://docs.python.org/3.3/library/imaplib.html#imaplib.IMAP4_SSL

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Thought of the day

[Antoine Pitrou]

Steven D'Aprano  pearwood.info> writes:
> 
> A programmer had a problem, and thought Now he has "I know, I'll solve 
> two it with threads!" problems.


Host: Last week the Royal Festival Hall saw the first performance of a new
logfile by one of the world's leading modern programmers, Steven
"Two threads" D'Aprano. Mr D'Aprano.

D'Aprano: Hello.

Host: May I just sidetrack for one moment. This -- what shall I call it --
nickname of yours...

D'Aprano: Ah yes.

Host: "Two threads". How did you come by it?

D'Aprano: Well, I don't use it myself, but some of my friends call me "Two
Threads".

Host: And do you in fact have two threads?

D'Aprano: No, I've only got one. I've had one for some time, but a few
years ago I said I was thinking of spawning another, and since then some
people have called me "Two Threads".

Host: In spite of the fact that you only have one.

D'Aprano: Yes.

Host: And are you still intending to spawn this second thread?

D'Aprano: (impatient) No!

Host: ...To bring you in line with your epithet?

D'Aprano: No.

Host: I see, I see. Well to return to your program.

D'Aprano: Ah yes.

Host: Did you write this logfile in the thread?

D'Aprano: (surprised) No!

Host: Have you written any of your recent files in this thread of yours?

D'Aprano: No, no, not at all. It's just an ordinary daemon thread.

Host: I see, I see. And you're thinking of spawning this second thread to
write in!

D'Aprano: No, no. Look. This thread business -- it doesn't really matter.
The threads aren't important. A few friends call me Two Threads and that's
all there is to it. I wish you'd ask me about the logfile. Everybody talks
about the threads. They've got it out of proportion -- I'm a programmer.
I'm going to get rid of the thread. I'm fed up with it!

Host: Then you'll be Steven "No Threads" D'Aprano, eh?


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Py3k Signal Handling

[Antoine Pitrou]

Jonathan Hayward  pobox.com> writes:
> 
> What needs changing here and how should I change it so that handle_signal()
> is called and then things keep on ticking?

So that it is called when exactly? Your message isn't clear as to what isn't 
working for you.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Memory profiling: Python 3.2

[Antoine Pitrou]

Andrew Robinson  r3dsolutions.com> writes:
> 
> When Python3.2 is running, is there an easy way within Python to capture 
> the *total* amount of heap space the program is actually using  (eg:real 
> memory)?

I'm not sure what you mean with "real memory" or how precise you want that 
measurement to be, but you could try http://www.selenic.com/smem/
(and, within Python, re-use smem's concepts, which shouldn't be too difficult)

> And how much of that heap space is allocated to variables ( 
> including re-capturable data not yet GC'd ) ?

Again, not sure what you mean with "allocated to variables" (global variables? 
local variables? everything?). As for "re-capturable data not yet GC'd", the
best way to figure out is to run gc.collect() :-)

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: deque and thread-safety

[Antoine Pitrou]

Hello,

Christophe Vandeplas  vandeplas.com> writes:
> 
> From the documentation I understand that deques are thread-safe:
> > Deques are a generalization of stacks and queues (the name is pronounced 
“deck”
> > and is short for “double-ended queue”). Deques support thread-safe, memory
> > efficient appends and pops from either side of the deque with approximately 
the
> > same O(1) performance in either direction.
> 
> It seems that appending to deques is indeed thread-safe, but not
> iterating over them.

Indeed, if you read the above sentence, deques only support "thread-safe [...] 
appends and pops". Other operations are not necessarily thread-safe.
Moreover, any operation which will call back into Python (such as iterating
several times) are *not* thread-safe.

Regardless, deques are not the right container for containment tests
("url in seen"). Like with lists or tuples, a containment test in a deque will 
be O(n). So if you want efficient containment tests, you should use a set or a 
dict.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Posix call (execve) breaks mercurial?

[Antoine Pitrou]

Hello,

Wayne Werner  waynewerner.com> writes:
> 
> So... curiouser and curiouser - it looks like it's not *actually* execve's 
> fault after all. I just compiled the code from the man page, tweaked it to 
> run 'hg root', and passed it a new environment. No problems. Well, then I 
> manually called the posix one from Python and thing worked fine. *Then* I 
> actually tried the above code, and *it* worked fine.
> 
> However I *still* get problems with the post-review code. So it looks like 
> when I get back to work on Monday I'll be looking to see  what the 
> difference in environment is there.

Your problem is a user question for the Mercurial mailing-list, not a Python
problem. See http://selenic.com/mailman/listinfo/mercurial

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Using Py_AddPendingCall

[Antoine Pitrou]

css322  gmail.com> writes:
> 
> (1) A worker thread calls Py_AddPendingCall and assigns a handler function.
> (2) When the Python interpreter runs, it calls the handler function whenever
it yields control to another thread

Not exactly. As the documentation says: "If successful, func will be called 
with the argument arg *at the earliest convenience*."
This is a deliberately vague wording to stress that the function will not be
called as early as you think. It *should* be called in a timely manner, but not
necessarily as soon as the thread switch happens.

Which begs the question:

> In this example, worker_thread is invoked in C as a pthread worker thread. 
> This
> loops infinitely, but the pending call handler is never invoked.

What is your main Python thread doing? Is it running Python code (*)? Or do you
have a main Python thread at all? The "main Python thread" is the one from which
Py_Initialize() was called.

(*) for example, running one of the following functions:
http://docs.python.org/dev/c-api/veryhigh.html

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The opener parameter of Python 3 open() built-in

[Antoine Pitrou]

Chris Angelico  gmail.com> writes:
> 
> On Wed, Sep 5, 2012 at 5:16 AM, Terry Reedy  udel.edu> wrote:
> > io.open depends on a function the returns an open file descriptor. opener
> > exposes that dependency so it can be replaced.
> 
> I skimmed the bug report comments but didn't find an answer to this:
> Why not just monkey-patch? When a module function calls on a support
> function and you want to change that support function's behaviour,
> isn't monkey-patching the most usual?

Monkey-patching globals is not thread-safe: other threads will see your
modification, which is risky and fragile.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Looking for an IPC solution

[Antoine Pitrou]

Laszlo Nagy  shopzeus.com> writes:

> 
> There are just so many IPC modules out there. I'm looking for a solution 
> for developing a new a multi-tier application. The core application will 
> be running on a single computer, so the IPC should be using shared 
> memory (or mmap) and have very short response times. But there will be a 
> tier that will hold application state for clients, and there will be 
> lots of clients. So that tier needs to go to different computers. E.g. 
> the same IPC should also be accessed over TCP/IP. Most messages will be 
> simple data structures, nothing complicated. The ability to run on PyPy 
> would, and also to run on both Windows and Linux would be a plus.

How about the standard multiprocessing module? It supports shared memory, remote
processes, and will most probably work under PyPy:
http://docs.python.org/library/multiprocessing.html

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Flexible string representation, unicode, typography, ...

[Antoine Pitrou]

 gmail.com> writes:
> 
> Pick up a random text and see the probability this
> text match the most optimized case 1 char / 1 byte,
> practically never.

Funny that you posted a text which does just that:
http://mail.python.org/pipermail/python-list/2012-August/629554.html

> In a funny way, this is what Python was doing and it
> performs better!

I honestly suggest you shut up until you have a clue.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Flexible string representation, unicode, typography, ...

[Antoine Pitrou]

Ramchandra Apte  gmail.com> writes:
> 
> The zen of python is simply a guideline

What's more, the Zen guides the language's design, not its implementation.
People who think CPython is a complicated implementation can take a look at 
PyPy 
:-)

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: SSLSocket.getpeercert() doesn't return issuer, serial number, etc

[Antoine Pitrou]

Hello,

Gustavo Baratto  gmail.com> writes:
> 
> SSL.Socket.getpeercert() doesn't return essential information present in the
> client certificate (issuer, serial number, not before, etc), and it looks it 
> is
> by design:

It does, in Python 3.2:
http://docs.python.org/py3k/library/ssl.html#client-side-operation

(although the getpeercert() doc should be updated to reflect this)

If some information is still lacking from the returned value, please open an 
issue at http://bugs.python.org

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Flushing buffer on file copy on linux

[Antoine Pitrou]

J  gmail.com> writes:
> 
> Now, the problem I have is that linux tends to buffer data writes to a
> device, and I want to work around that.  When run in normal non-stress
> mode, the program is slow enough that the linux buffers flush and put
> the file on disk before the hash occurs.  However, when run in stress
> mode, what I'm finding is that it appears that the files are possibly
> being hashed while still in the buffer, before being flushed to disk.

Your analysis is partly wrong. It is right that the files can be hashed from 
in-memory buffers; but even if you flush the buffers to disk using standard 
techniques (such as fsync()), those buffers still exist in memory, and 
therefore the file will still be hashed from memory (for obvious efficiency 
reasons).

I don't think there's a portable solution to get away entirely with the 
in-memory buffers, but under Linux you can write "1" to the special file 
/proc/sys/vm/drop_caches:

$ sudo sh -c "echo 1 > /proc/sys/vm/drop_caches"

Or, to quote the /proc man page:

   /proc/sys/vm/drop_caches (since Linux 2.6.16)
  Writing to this file  causes  the  kernel  to  drop  clean
  caches, dentries and inodes from memory, causing that mem‐
  ory to become free.

  To free pagecache, use echo 1 >  /proc/sys/vm/drop_caches;
  tofree   dentries   and   inodes,   use   echo   2   >
  /proc/sys/vm/drop_caches; to free pagecache, dentries  and
  inodes, use echo 3 > /proc/sys/vm/drop_caches.

  Because  this  is  a  nondestructive  operation  and dirty
  objects are not freeable,  the  user  should  run  sync(8)
  first.


Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net


-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: pathlib 0.7

[Antoine Pitrou]

Hello,

pathlib 0.7 has been released with the following changes:

- Add '**' (recursive) patterns to Path.glob().
- Fix openat() support after the API refactoring in Python 3.3 beta1.
- Add a *target_is_directory* argument to Path.symlink_to()

pathlib offers a set of classes to handle filesystem paths.  It offers the
following advantages over using string objects:

* No more cumbersome use of os and os.path functions.  Everything can be
  done easily through operators, attribute accesses, and method calls.

* Embodies the semantics of different path types.  For example, comparing
  Windows paths ignores casing.

* Well-defined semantics, eliminating any warts or ambiguities (forward vs.
  backward slashes, etc.).

PyPI download page at http://pypi.python.org/pypi/pathlib/
Documentation at http://pathlib.readthedocs.org/en/latest/
Code and issue tracker at https://bitbucket.org/pitrou/pathlib/

Regards

Antoine Pitrou.


-- 
Software development and contracting: http://pro.pitrou.net


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ssl: add msg_callback

[Antoine Pitrou]

Hi,

Thiébaud Weksteen  weksteen.fr> writes:
> 
> I wrote a patch for Python 3.2.3 to expose the function
> SSL_CTX_set_msg_callback in the module _ssl.
> 
[...]
> 
> Let me know your opinion on that. Does it worth being included?

Yes, it sounds useful. Your patch will have to be written against the default
branch, although I'm not sure it makes a difference here (see the devguide:
http://docs.python.org/devguide/ for more information).

Please open an issue at http://bugs.python.org and post your patch there.

Thanks

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: append in IMAP4 from imaplib very slow

[Antoine Pitrou]

Simon Pirschel  abusix.org> writes:
> 
> Hi,
> I'm currently experimenting with IMAP using Python 2.7.3 and IMAP4
> from imaplib. I noticed the performance to be very bad. I read 5000
> files from a directory and append them to an IMAP INBOX. The hole
> procedure of reading and appending is taking about 210 seconds.
> I set up the exact same code in Perl to check if there is a general
> IMAP server configuration issue, since CPU and I/O isn't the
> problem. The same amount of data on the same IMAP server is done
> after 7.9 seconds using Perl.
> The difference is huge and I tried to narrow the issue down by
> profiling the Python code.
> The profile results are, 206 seconds are spent in calling
> socket.recv.

This just means that most of the time is spent waiting for the server to
reply. Perhaps the Perl and Python IMAP libraries use different IMAP commands
for appending?

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Smallest/cheapest possible Python platform?

[Antoine Pitrou]

Roy Smith  panix.com> writes:
> 
> What's the smallest/cheapest/lowest-power hardware platform I can run 
> Python on today?  I'm looking for something to use as a hardware 
> controller in a battery-powered device and want to avoid writing in C 
> for this project.

It depends *which* Python. Complete Python implementations (CPython, PyPy,
IronPython, Jython) will have stronger requirements than minimal / incomplete
implementations.

As for CPython, it needs a C compiler, decent POSIX support, a 32-bit CPU at
least, and realistically you won't do much with at least 8 MB RAM.

We actually have a buildbot which regularly tests building and running of
CPython on an ARM machine:
http://www.python.org/dev/buildbot/all/buildslaves/warsaw-ubuntu-arm

It's a Cortex A8 with 1GB RAM, though, so I don't know if it's in your range
(but 1GB is not needed at all, except that it's nice when running the full
regression test suite).

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple TLS NPN negotiation not working on socket server

[Antoine Pitrou]

Alek Storm  gmail.com> writes:
> 
> Connecting with either Firefox 11 or Chrome (which both support NPN) causes
> this to print None, rather than a protocol name. What's going on?

Ok, I've just tried with Firefox 11. You have to go in "about:config" and set
"network.http.spdy.enabled" to true. Then the code snippet works (it prints
"spdy/2").
My OpenSSL version is 'OpenSSL 1.0.1-beta3 23 Feb 2012'.

I don't have Chrome to test, but perhaps there's a similar configuration option.

> Does the protocol matter? PROTOCOL_SSLv23 gives the same result, but
PROTOCOL_TLSv1
> makes it die with SSL3_GET_CLIENT_HELLO:wrong version number (strange, because
> both browsers ostensibly support TLS).

PROTOCOL_TLSv1 works here (with Firefox 11.0).

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: nested embedding of interpreter

[Antoine Pitrou]

Hello,

Eric Frederich  gmail.com> writes:
> 
> 1)Is calling Py_Initialize twice correct, or will I run into other problems
> down the road?

It's fine in practice (spurious calls are ignored).

> I am not sure if there is a mechanism to get something called at the end of
the 
> user's session with the program though, so is it a problem if I don't call
> Py_Finalize at the end?

Yes, it's a problem. If you don't call Py_Finalize, atexit handlers, object
destructors and the like will not be called. These include destructors of file
objects (including stdout): any buffering in a still-open file opened for
writing can be lost.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Getting involved

[Antoine Pitrou]

Hello,

Sophie Sperner  gmail.com> writes:
> 
> Let me ask here please. I'm a first-year PhD student in Ireland. My
> background is in mathematics, though I'm going to stream my career
> into programming with Python, Java and C++ languages. I have some good
> experience with C++ what allowed me to have a presentation in Europe
> quite recently - I participated in an open-source cloud computing
> project. I used Python a little bit in that project, though mainly it
> was in C++.
> 
> [...]
> 
> Could you please list me 2 or 3 projects in Python and/or Java which
> are currently active (vivid) and useful?

CPython itself - the reference and most used implementation - is written partly
in Python (most of the stdlib is) and partly in C. It is easy to find things to
do even without touching C. You could take a look:
http://docs.python.org/devguide/

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python, Solaris 10, and Mailman

[Antoine Pitrou]

On Fri, 21 Jan 2011 22:59:33 +
"McNutt Jr, William R"  wrote:
> I am attempting to install Mailman on a Sun Sunfire x4100 box running Solaris 
> ten. I keep running into brick walls that the Mailman group looks at, shrugs, 
> and says, that's a Python problem.
> 
> Has ANYBODY actually made this work?
> 
> Currently, I'm attempting to compile Python 2.4.4, which is the recommended 
> distro for Mailman,

2.4.4?? Do yourself a favour and choose a modern release.
2.4 hasn't been supported for years, and besides, the latest in that
branch in 2.4.6.
You can probably use whatever version of Python comes with Solaris, no
need to build your own.
Oh, and tell the Mailman guys that their recommendations are totally
obsolete.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: getdefaultencoding - how to change this?

[Antoine Pitrou]

On 20 Jan 2011 17:20:14 GMT
Helmut Jarausch  wrote:
> Thanks Robert,
> probably I wasn't too clear about my issue.
> I couldn't "print" any non-ascii character to my console although
> my console has an en_US.iso88591 locale.

Well, if you want a correct answer, you should paste actual code as
well as its result on your system, rather than start by asking how to
change getdefaultencoding (which you shouldn't do as explained by
others).

> I didn't modfiy anything in Python's source code.
> I noticed that my root account still had an ASCII locale.
> Now I have changed it such that the root account has the same locale
> as non-root accounts. Recompiling Python under such a root account
> has rectified things.

Recompiling Python doesn't change its behaviour wrt. locales and
encodings. All this is done at runtime.

> Now I can print non-ascii characters if they are
> properly encoded.

You can *always* print characters if they are properly encoded. What
you are asking is for Python to guess and do the encoding by itself,
which is a different matter (and a poorly supported one under 2.x;
Python 3 behaves much better in that regard).

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-8 question from Dive into Python 3

[Antoine Pitrou]

On Wed, 19 Jan 2011 19:18:49 + (UTC)
Tim Harig  wrote:
> On 2011-01-19, Antoine Pitrou  wrote:
> > On Wed, 19 Jan 2011 18:02:22 + (UTC)
> > Tim Harig  wrote:
> >> Converting to a fixed byte
> >> representation (UTF-32/UCS-4) or separating all of the bytes for each
> >> UTF-8 into 6 byte containers both make it possible to simply index the
> >> letters by a constant size.  You will note that Python does the
> >> former.
> >
> > Indeed, Python chose the wise option. Actually, I'd be curious of any
> > real-world software which successfully chose your proposed approach.
> 
> The point is basically the same.  I created an example because it
> was simpler to follow for demonstration purposes then an actual UTF-8
> conversion to any official multibyte format.  You obviously have no
> other purpose then to be contrary [...]

Right. You were the one who jumped in and tried to lecture everyone on
how UTF-8 was "big-endian", and now you are abandoning the one esoteric
argument you found in support of that.

> As soon as you start to convert to a multibyte format the endian issues
> occur.

Ok. Good luck with your "endian issues" which don't exist.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-8 question from Dive into Python 3

[Antoine Pitrou]

On Wed, 19 Jan 2011 18:02:22 + (UTC)
Tim Harig  wrote:
> On 2011-01-19, Antoine Pitrou  wrote:
> > On Wed, 19 Jan 2011 16:03:11 + (UTC)
> > Tim Harig  wrote:
> >> 
> >> For many operations, it is just much faster and simpler to use a single
> >> character based container opposed to having to process an entire byte
> >> stream to determine individual letters from the bytes or to having
> >> adaptive size containers to store the data.
> >
> > You *have* to "process the entire byte stream" in order to determine
> > boundaries of individual letters from the bytes if you want to use a
> > "character based container", regardless of the exact representation.
> 
> Right, but I only have to do that once.

You only have to decode once as well.

> If I leave the information as a
> simple UTF-8 stream,

That's not what we are talking about. We are talking about the supposed
benefits of your 6-byte representation scheme versus proper decoding
into fixed width code points.

> UTF-32/UCS-4 conversion is definitly supperior if you are actually
> doing any major but it adds the complexity and overhead of requiring
> the bit twiddling to make the conversions (once in, once again out).

"Bit twiddling" is not something processors are particularly bad at.
Actually, modern processors are much better at arithmetic and logic
than at recovering from mispredicted branches, which seems to suggest
that discovering boundaries probably eats most of the CPU cycles.

> Converting to a fixed byte
> representation (UTF-32/UCS-4) or separating all of the bytes for each
> UTF-8 into 6 byte containers both make it possible to simply index the
> letters by a constant size.  You will note that Python does the
> former.

Indeed, Python chose the wise option. Actually, I'd be curious of any
real-world software which successfully chose your proposed approach.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: __pycache__, one more good reason to stck with Python 2?

[Antoine Pitrou]

On Wed, 19 Jan 2011 08:30:12 -0800 (PST)
jmfauth  wrote:
> Yes, I can launch a pyc, when I have a single file.
> But what happens, if one of your cached .pyc file import
> a module with a name as defined in the parent directory?
> The machinery is broken. The parent dir is not in the
> sys.path.

Well, you don't have to launch a pyc file anyway. Put all your code in
some (pyc) modules on the standard Python path, and use a clear-text
script with some trivial code to invoke a function from the compiled
modules.

Otherwise you can customize sys.path a little from your script,
using __file__ and os.path.dirname. Nothing complicated AFAICT.

(by the way, the fact that pyc files are version-specific should
discourage any use outside of version-specific directories,
e.g. /usr/lib/pythonX.Y/site-packages)

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-8 question from Dive into Python 3

[Antoine Pitrou]

On Wed, 19 Jan 2011 16:03:11 + (UTC)
Tim Harig  wrote:
> 
> For many operations, it is just much faster and simpler to use a single
> character based container opposed to having to process an entire byte
> stream to determine individual letters from the bytes or to having
> adaptive size containers to store the data.

You *have* to "process the entire byte stream" in order to determine
boundaries of individual letters from the bytes if you want to use a
"character based container", regardless of the exact representation.
Once you do that it shouldn't be very costly to compute the actual code
points. So, "much faster" sounds a bit dubious to me; especially if you
factor in the cost of memory allocation, and the fact that a larger
container will fit less easily in a data cache.

> That said, and more importantly, many
> variable length byte streams may not have alternate representations as
> unicode does.

This whole thread is about UTF-8 (see title) so I'm not sure what kind
of relevance this is supposed to have.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: __pycache__, one more good reason to stck with Python 2?

[Antoine Pitrou]

On 19 Jan 2011 14:42:14 GMT
Steven D'Aprano  wrote:
> On Tue, 18 Jan 2011 00:58:14 -0800, jmfauth wrote:
> 
> > It is now practically impossible to launch a Python application via a
> > .pyc file. 
> 
> 
> When has that ever been possible?
> 
> 
> .pyc files are Python byte-code. You can't run them directly using Python 
> (except via the import machinery), you can't run them as a script, 
> they're not machine code. Unless you write a wrapper to import the file 
> as a module, you can't directly execute .pyc files.

It seems to work here:

$ echo "print 'foo'" > toto.py
$ python -m compileall -l .
Listing . ...
Compiling ./toto.py ...
$ rm toto.py
$ python toto.pyc
foo


But it still works under 3.2 even though the incantation is less pretty:

$ echo "import sys; print(sys.version)" > toto.py
$ __svn__/python -m compileall -l .
Listing . ...
Compiling ./toto.py ...
$ rm toto.py
$ __svn__/python __pycache__/toto.cpython-32.pyc
3.2rc1+ (py3k:88095M, Jan 18 2011, 17:12:15)
[GCC 4.4.3]


Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-8 question from Dive into Python 3

[Antoine Pitrou]

On Wed, 19 Jan 2011 14:00:13 + (UTC)
Tim Harig  wrote:
> 
> - Q: Can a UTF-8 data stream contain the BOM character (in UTF-8 form)? If
> - yes, then can I still assume the remaining UTF-8 bytes are in big-endian
> ^^
> - order?
>   ^^
> - 
> - A: Yes, UTF-8 can contain a BOM. However, it makes no difference as
>  ^^^ 
> - to the endianness of the byte stream. UTF-8 always has the same byte
> ^^
> - order.
>   ^^

Which certainly doesn't mean that byte order can be called "big
endian" for any recognized definition of the latter. Similarly, ASCII
test has its own order which certainly can't be characterized as either
"little endian" or "big endian".

> UTF-8 has no apparent endianess if you only store it as a byte stream.
> It does however have a byte order.  If you store it using multibytes
> (six bytes for all UTF-8 possibilites) , which is useful if you want
> to have one storage container for each letter as opposed to one for
> each byte(1)

That's a ridiculous proposition. Why would you waste so much space?
UTF-8 exists *precisely* so that you can save space with most scripts.
If you are ready to use 4+ bytes per character, just use UTF-32 which
has much nicer properties.

Bottom line: you are not describing UTF-8, only your own foolish
interpretation of it. UTF-8 does not have any endianness since it is a
byte stream and does not care about "machine words".

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-8 question from Dive into Python 3

[Antoine Pitrou]

On Wed, 19 Jan 2011 11:34:53 + (UTC)
Tim Harig  wrote:
> That is why the FAQ I linked to
> says yes to the fact that you can consider UTF-8 to always be in big-endian
> order.

It certainly doesn't. Read better.

> Essentially all byte based data is big-endian.

This is pure nonsense.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: move to end, in Python 3.2 Really?

[Antoine Pitrou]

On Tue, 18 Jan 2011 10:33:45 -0800 (PST)
rantingrick  wrote:
> 
> On Jan 18, 11:56 am, Antoine Pitrou  wrote:
> > On Tue, 18 Jan 2011 09:10:48 -0800 (PST)
> >
> > rantingrick  wrote:
> >
> > > Well don't get wrong i want to join in --not that i have all the
> > > solutions--
> >
> > Take a look athttp://docs.python.org/devguide/#contributing
> 
> Thanks for this link Antoine however i think you missed the point of
> my post.

You did say "I want to join in".

> What i would like to see is an forum where the "noob" to
> "average" python programmer can voice his/her opinion about the
> current state or future state of Pythons syntax, stdlib, goals and
> dreams, etc, al the while not fearing attack from all sides.

Well the only way for that to happen is to put it up yourself. Or to
gather some people around you to put it up together.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: move to end, in Python 3.2 Really?

[Antoine Pitrou]

On Tue, 18 Jan 2011 09:10:48 -0800 (PST)
rantingrick  wrote:
> 
> Well don't get wrong i want to join in --not that i have all the
> solutions--

Take a look at http://docs.python.org/devguide/#contributing


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: move to end, in Python 3.2 Really?

[Antoine Pitrou]

On Mon, 17 Jan 2011 21:20:48 -0800 (PST)
Raymond Hettinger  wrote:
> On Jan 17, 6:51 pm, nn  wrote:
> > ...But the api on this baffles me a bit:
> >
> > >>> d = OrderedDict.fromkeys('abcde')
> > >>> d.move_to_end('b', last=False)
> > >>> ''.join(d.keys)
> >
> > 'bacde'
> >
> > I understand that "end" could potentially mean either end, but would
> > "move_to_end" and "move_to_beginning" not have been clearer?
> 
> The default (and normal usage) is to move an item to the last
> position.
> So, od.move_to_end(k) becomes a fast equivalent to v=d.pop(k)
> followed by d[k]=v.
> 
> The less common usage of moving to the beginning is done with
> last=False.  This parallels the existing API for od.popitem():

Well I have to agree that moving to the beginning using move_to_end()
with a "last" argument looks completely bizarre and unexpected.
"Parallels popitem()" is not really convincing since popitem() doesn't
have "end" its name.

> Those were the design considerations.  Sorry you didn't like the
> result.

Design considerations? Where were they discussed?

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UTF-8 question from Dive into Python 3

[Antoine Pitrou]

On Mon, 17 Jan 2011 14:19:13 -0800 (PST)
carlo  wrote:
> Is it true UTF-8 does not have any "big-endian/little-endian" issue
> because of its encoding method?

Yes.

> And if it is true, why Mark (and
> everyone does) writes about UTF-8 with and without BOM some chapters
> later? What would be the BOM purpose then?

"BOM" in this case is a misnomer. For UTF-8, it is only used as a
marker (a magic number, if you like) to signal than a given text file
is UTF-8. The UTF-8 "BOM" does not say anything about byte order; and,
actually, it does not change with endianness.

(note that it is not required to put an UTF-8 "BOM" at the beginning of
text files; it is just a hint that some tools use when
generating/reading UTF-8)

> 2- If that were true, can you point me to some documentation about the
> math that, as Mark says, demonstrates this?

Math? UTF-8 is simply a byte-oriented (rather than word-oriented)
encoding. There is no math involved, it just works by construction.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 9 Month Python contract in Austin, TX

[Antoine Pitrou]

On Mon, 17 Jan 2011 11:08:52 -0800 (PST)
AlexLBasso  wrote:
> I am recruiting for a 9 month contract (with contract extension
> potential) for a company in North Austin.

Please post on the job board instead:
http://python.org/community/jobs/

Thank you

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python 3 and Unicode line breaking

[Antoine Pitrou]

On Fri, 14 Jan 2011 06:29:27 -0800 (PST)
leoboiko  wrote:
> 
> And it generally doesn’t try to pick good places to break lines
> at all, just making the assumption that 1 character = 1 column
> and that breaking on ASCII whitespaces/hyphens is enough.  We
> can’t really blame textwrap for that, it is a very simple module
> and Unicode line breaking gets complex fast (that’s why the
> consortium provides a ready-made algorithm).  It’s just that,
> with python3’s emphasis on Unicode support, I was surprised not
> to be able to find an UAX #14 implementation.  I thought someone
> would surely have written one and I simply couldn’t find, so I
> asked precisely that.

If you're willing to help on that matter (or some aspects of them,
textwrap-specific or not), you can open an issue on
http://bugs.python.org and propose a patch.

See also http://docs.python.org/devguide/#contributing if you need more
info on how to contribute.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python 3 and Unicode line breaking

[Antoine Pitrou]

On 14 Jan 2011 22:10:02 GMT
Steven D'Aprano  wrote:
> 
> This is good, helpful advice, and far more useful to the OP than just 
> ignoring his post. You have jumped to his defense (or rather, you have 
> jumped to criticise me) but I see that you haven't replied to his 
> question or given him any advice in how to solve his problem.

Simply because I have no elaborate answer to give, even in the light of
his/her recent precisions on the topic (and, actually, neither do you).
Asking for precisions is certainly fine; doing it in an agressive way
is not, especially when the original message doesn't look like the
usual blunt, impolite and typo-ridden "can you do my homework" message.

Also, I would expect someone familiar with the textwrap module's (lack
of) unicode capabilities would have been able to answer the first
message without even asking for precisions.

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python 3 and Unicode line breaking

[Antoine Pitrou]

Hey,

On 14 Jan 2011 16:07:12 GMT
Steven D'Aprano  wrote:
> 
> > I also see no reason to reply to a simple question with such
> > discourtesy, and cannot understand why someone would be so aggressive to
> > a stranger.
> 
> If you think my reply was aggressive and discourteous, you've got a lot 
> to learn about public forums.

Perhaps you've got to learn about politeness yourself! Just because
some people are jerks on internet forums (or in real life) doesn't mean
everyone should; this is quite a stupid and antisocial excuse actually.

You would never have reacted this way if the same question had been
phrased by a regular poster here (let alone on python-dev). Taking
cheap shots at newcomers is certainly not the best way to welcome
them.

Thank you

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating custom Python objects from C code

[Antoine Pitrou]

On Wed, 5 Jan 2011 11:27:02 -0500
Eric Frederich  wrote:
> I have read through all the documentation here:
> 
> http://docs.python.org/extending/newtypes.html
> 
> I have not seen any documentation anywhere else explaining how to
> create custom defined objects from C.
> I have this need to create custom objects from C and pass them as
> arguments to a function call.

What do you mean? Create instances of a type defined in Python code?

The C API is not very different from Python-land. When you want to
instantiate a type, just call that type (as a PyObject pointer) with the
right arguments (using PyObject_Call() and friends). Whether that type
has been defined in C or in Python does not make a difference.

> Question 2: How do I make C helper functions that are part of my
> extension available to other C projects in the same way that PyList_*,
> PyString_*, PyInt_* functions are available?
> Is it possible to have distutils make a .lib file for me?

I don't know. I'd say "probably" :S
Otherwise you can use the PyCapsule system, but that seems quite a bit
more involved: http://docs.python.org/c-api/capsule.html

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python comparison matrix

[Antoine Pitrou]

On Mon, 3 Jan 2011 16:17:00 -0800 (PST)
Alex Willmer  wrote:
> I've created a spreadsheet that compares the built ins, features and modules 
> of the CPython releases so far. For instance it shows: 

A couple of errors:
- BufferError is also in 3.x
- IndentationError is also in 3.x
- object is also in 3.x
- NotImplemented is not an exception type, it's a built-in singleton
  like None
- you forgot VMSError (only on VMS) :-)

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Nagios

[Antoine Pitrou]

On 31 Dec 2010 04:20:59 GMT
Steven D'Aprano  wrote:
> On Thu, 30 Dec 2010 23:04:33 -0500, Robert wrote:
> 
> > The
> > second way the Tcl community irks me is the "not invented here"
> > attitude. I like the syntax of Tcl and I like the community. They are
> > some good folks. Try asking "I want to build a Nagios clone in Tcl" type
> > question and invariably you get "Why? There is already Nagios?".
> 
> You're the one who wants to re-write Nagios in Tcl, the Tcl community are 
> perfectly happy using the existing Nagios instead of re-inventing the 
> wheel, and you accuse *them* of suffering from NIH syndrome.

Well, I don't know about Tcl but Nagios was re-written in Python:
http://www.shinken-monitoring.org/features/

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: issubclass(dict, Mapping)

[Antoine Pitrou]

On Wed, 22 Dec 2010 09:35:48 -0500
Adam Tauno Williams  wrote:
> 
> IMO, the "object model" isn't "leaky", it is simply "adhoc" and not
> really a "model" at all [write as many 800 page books as you want: if it
> walks like a zombie duck, smells like a zombie duck - it is still a
> zombie duck].  Performing introspection in Python is awful and a
> veritable land-mine of "implementation details".

Introspection is fine as long as you stick to officially promoted tools
such as isinstance(), issubclass(), dir() or the inspect module.
If you start looking inside the pants of the object model, you can have
surprises :)

Regards

Antoine.


-- 
http://mail.python.org/mailman/listinfo/python-list