Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Terry Reedy]

On 4/17/2011 1:32 AM, R. David Murray wrote:


As Brett said, people do come to depend on the details of the
implementation.  But IMO the PEP should be clarified to say that the
tests we are talking about should be tests *of the published API*.
That is, blackbox tests, not whitebox tests.


I think 100% *branch* coverage is barking up the wrong tree.
Better to say comprehensive *api* coverage. Bugs on the tracker 
generally come from not having that. (I am not saying 'all' to allow for 
bugs that happen from weird interactions or corner cases in spite of 
what could reasonably be called comprehensive._



I don't think the PEP is asking this either (or if it is I agree it
shouldn't be).  The way to get full branch coverage (and yes Exarkun is
right, this is about individual branches; see coverage.py --branch) is
to provide test cases that exercise the published API such that those
branches are taken.  If you can't do that, then what is that branch
of the Python code for?  If you can do that, how is the test case
testing an implementation detail?  It is testing the behavior of the
API.


Right.

--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Martin v. Löwis]

> Well, there was a 5x speedup demonstrated comparing simplejson to the
> standard library json module.

Can you kindly point to that demonstration?

> That sound like *very* worth pursuing (and
> crazy not to pursue). I've had json serialisation be the bottleneck in
> web applications generating several megabytes of json for some requests.

Hmm. I'd claim that the web application that needs to generate several
megabytes of json for something should be redesigned. I also wonder
whether the bottleneck was the *generation*, the transmission, or
the processing of the data on the receiving end.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Stefan Behnel]

Matt Billenstein, 17.04.2011 00:47:

On Sat, Apr 16, 2011 at 01:30:13PM +0200, Antoine Pitrou wrote:

On Sat, 16 Apr 2011 00:41:03 +
Matt Billenstein wrote:


Slightly less crude benchmark showing simplejson is quite a bit faster:

http://pastebin.com/g1WqUPwm

250ms vs 5.5s encoding and decoding an 11KB json object 1000 times...


This doesn't have much value if you don't say which version of Python
you ran json with. You should use 3.2, otherwise you might miss some
optimizations.


Yes, that was 2.6.5 -- 3.2 native json is comparable to simplejson here taking
about 330ms...


From the POV of CPython 3.2, is "native" Python or C?

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[R. David Murray]

On Sat, 16 Apr 2011 19:19:32 -0700, Raymond Hettinger 
 wrote:
> On Apr 16, 2011, at 2:45 PM, Brett Cannon wrote:
> 
> >
> >
> > On Sat, Apr 16, 2011 at 14:23, Stefan Krah  wrote:
> > Brett Cannon  wrote:
> > > In the grand python-dev tradition of "silence means acceptance", I 
> > > consider
> > > this PEP finalized and implicitly accepted.
> 
> I haven't seen any responses that said, yes this is a well thought-out 
> proposal
> that will actually benefit any of the various implementations.

In that case it may well be that the silence is because the other
implementations think the PEP is OK.  They certainly voted in favor of
the broad outline of it at the language summit.  Perhaps representatives
will speak up, or perhaps Brett will need to poll them proactively.

> Almost none of the concerns that have been raised has been addressed.  Does 
> the
> PEP only apply to purely algorithmic modules such as heapq or does it apply to
> anything written in C (like an xz compressor or for example)?  Does testing

Anything (new) written in C that can be also written in Python (and
usually is first, to at least prototype it).  If an XZ compressor is a
wrapper around an external library, that would be a different story.

> every branch in a given implementation now guarantee every implementation 
> detail
> or do we only promise the published API (historically, we've *always* done the
> latter)?

As Brett said, people do come to depend on the details of the
implementation.  But IMO the PEP should be clarified to say that the
tests we are talking about should be tests *of the published API*.
That is, blackbox tests, not whitebox tests.

> Is there going to be any guidance on the commonly encountered semantic
> differences between C modules and their Python counterparts (thread-safety,
> argument handling, tracebacks, all possible exceptions, monkey-patchable pure
> python classes versus hard-wired C types etc)?

Presumably we will need to develop such guidance.

> The PEP seems to be predicated on a notion that anything written in C is bad 
> and
> that all testing is good.  AFAICT, it doesn't provide any practical advice to
> someone pursuing a non-trivial project (such as decimal or threading).  The 
> PEP

Decimal already has a Python implementation with a very comprehensive
test suite (no, I don't know if it has 100% coverage).  My understanding
is that Stefan's code passes the Python test suite.  So I'm not sure
what the issue is, there.  Stefan?

Threading is an existing module, so it doesn't seem to me that the PEP
particularly applies to it.

> The PEP also makes some unsupported claims about saving labor.  My 
> understanding
> is the IronPython and Jython tend to re-implement modules using native
> constructs.  Even with PyPy, the usual pure python idioms aren't necessarily
> what is best for PyPy, so I expect some rewriting there also.  It seems the
> lion's share of the work in making other implementations has to do with
> interpreter details and whatnot -- I would be surprised if the likes of bisect
> or heapq took even one-tenth of one percent of the total development time for
> any of the other implementations.

That's an orthogonal issue.  Having *working* Python implementations of as
much of the stdlib as practical makes it easier to spin up a new Python
language implementation:  once you get the language working, you've got
all the bits of the stdlib that have Python versions.  *Then* you can
implement accelerators (and if you are CPython, you do that in C...)

> If you're saying that all implementation details (including internal branching
> logic) are now guaranteed behaviors, then I think this PEP has completely lost
> its way.  I don't know of any implementors asking for this.

I don't think the PEP is asking this either (or if it is I agree it
shouldn't be).  The way to get full branch coverage (and yes Exarkun is
right, this is about individual branches; see coverage.py --branch) is
to provide test cases that exercise the published API such that those
branches are taken.  If you can't do that, then what is that branch
of the Python code for?  If you can do that, how is the test case
testing an implementation detail?  It is testing the behavior of the
API.  The 100% branch coverage metric is just a measurable way to
improve test coverage.  As I've said before, it does not guarantee that all
important (API) test cases are covered, but it is one way to improve
that coverage that has a measure attached, and measures are helpful.

I personally have no problem with the 100% coverage being made a
recommendation in the PEP rather than a requirement.  It sounds like
that might be acceptable to Antoine.  Actually, I would also be fine with
saying "comprehensive" instead, with a note that 100% branch coverage is
a good way to head toward that goal, since a comprehensive test suite
should contain more tests than the minimum set needed to get to 100%
branch coverage.

A relevant story:  to achieve 100% bra

Re: [Python-Dev] python and super

[Mark Janssen]

Argh!  Sorry list.  I meant to discard the post that was just sent.

Please accept my humblest apologies...

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python and super

[Mark Janssen]

On Thu, Apr 14, 2011 at 7:09 AM, Ricardo Kirkner
 wrote:
> I recently stumbled upon an issue with a class in the mro chain not
> calling super, therefore breaking the chain (ie, further base classes
> along the chain didn't get called).
> I understand it is currently a requirement that all classes that are
> part of the mro chain behave and always call super. My question is,
> shouldn't/wouldn't it be better,
> if python took ownership of that part, and ensured all classes get
> called, even if some class misbehaved?

I get annoyed by this issue as well, in various forms.

It seems like such a discussion would have been resolved by now in the
multitude of OOP languages, but I have to say it is quite strange to
me that there is no distinction made between IS-A relationship and
HAS-A relationships with regard to the issue of Inheritence.  Python,
confusingly makes no syntactic distinction, and its semantic
distinction (through MRO and programmer conventions) seems quite
suboptimal and "special-cased".  --No fault of anyone's, perhaps it is
indeed an unresolved issue within Computer Science.

It should be clear that IS-A inheritance is really trying to say (or
should be) that the following set/class (of methods and attributes) is
a *super-set* of its "parent" (--See how the OO lexicon is already
confused and mixing metaphors?).  In this case, manually calling
super() is not only completely redundant but adds various confusions.

With regard to inheritence, I too would like to see automatic calls to
super classes in every case were there is a complete sClearly there is
utility in the notion of a set-theoretic containment


DISCARDING::  the points are moot and need finer granularity that only
the pangaia model can fix.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Brian Curtin]

On Sat, Apr 16, 2011 at 06:45, Gustavo Narea  wrote:

> Hello,
>
> On 15/04/11 13:30, Brian Curtin wrote:
> > To me, the fix *was* released.
>
> No, it wasn't. It was *committed* to the repository.
>

Yep, and that's enough for me. If you have a vulnerable system, you can now
patch it with an accepted fix.


>
> > Sure, no fancy installers were generated yet, but people who are
> > susceptible to this issue 1) now know about it, and 2) have a way to
> > patch their system *if needed*.
>
> Well, that's a long shot. I doubt the people/organizations affected are
> all aware.


Hence why this blog exists and why this post was made...

And I doubt they are all capable of patching their system or
> getting a patched Python from a trusted party.
>

Maybe that's where the post fell short. Should I have added a section with
an example of how to apply the patch to an example system like 2.6?


> Three weeks after this security vulnerability was *publicly* reported on
> bugs.python.org, and two days after it was semi-officially announced,
> I'm still waiting for security updates for my Ubuntu and Debian systems!
>
> I reckon if this had been handled differently (i.e., making new releases
> and communicating it via the relevant channels [1]), we wouldn't have
> the situation we have right now.


I don't really think there's a "situation" here, and I fail to see how the
development blog isn't one of the relevant channels.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[exarkun]

On 16 Apr, 11:03 pm, st...@pearwood.info wrote:

Brett Cannon wrote:
In the grand python-dev tradition of "silence means acceptance", I 
consider

this PEP finalized and implicitly accepted.


How long does that silence have to last?

I didn't notice a definition of what counts as "100% branch coverage". 
Apologies if I merely failed to notice it, but I think it should be 
explicitly defined.


Presumably it means that any time you have an explicit branch 
(if...elif...else, try...except...else, for...else, etc.) you need a 
test that goes down each branch. But it isn't clear to me whether it's 
sufficient to test each branch in isolation, or whether you need to 
test all combinations.


That is, if you have five branches, A or B, C or D, E or F, G or H, I 
or J, within a single code unit (function? something else?), is it 
sufficient to have at least one test that goes down each of A...J, or 
do you need to explicitly test each of:


A-C-E-G-I
A-C-E-G-J
A-C-E-H-I
A-C-E-H-J
A-C-F-G-I
...
B-D-F-H-J

(10 tests versus 32 tests).

If the latter, this could become impractical *very* fast. But if not, I 
don't see how we can claim 100% coverage when there are code paths 
which are never tested.


The mostly commonly used definition of branch coverage is that each 
outcome of each individual branch is executed, not that all possible 
combinations of all branches in a unit are executed.  I haven't heard 
anyone in this thread propose the latter, only the former.


"100% coverage" by itself is certainly ambiguous.


At the very least, I think you need to explicitly define what you mean 
by "100% branch coverage". Possibly this will assist in the 
disagreement between you and Antoine re "100% versus "comprehensive" 
coverage.


I suspect that everyone who has said "branch coverage" in this thread 
has intended the definition given above (and encourage anyone who meant 
something else to clarify their position).


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Raymond Hettinger]

On Apr 16, 2011, at 2:45 PM, Brett Cannon wrote:

> 
> 
> On Sat, Apr 16, 2011 at 14:23, Stefan Krah  wrote:
> Brett Cannon  wrote:
> > In the grand python-dev tradition of "silence means acceptance", I consider
> > this PEP finalized and implicitly accepted.

I haven't seen any responses that said, yes this is a well thought-out proposal 
that will actually benefit any of the various implementations.

Almost none of the concerns that have been raised has been addressed.  Does the 
PEP only apply to purely algorithmic modules such as heapq or does it apply to 
anything written in C (like an xz compressor or for example)?  Does testing 
every branch in a given implementation now guarantee every implementation 
detail or do we only promise the published API (historically, we've *always* 
done the latter)?  Is there going to be any guidance on the commonly 
encountered semantic differences between C modules and their Python 
counterparts (thread-safety, argument handling, tracebacks, all possible 
exceptions, monkey-patchable pure python classes versus hard-wired C types etc)?

The PEP seems to be predicated on a notion that anything written in C is bad 
and that all testing is good.  AFAICT, it doesn't provide any practical advice 
to someone pursuing a non-trivial project (such as decimal or threading).  The 
PEP mostly seems to be about discouraging any further work in C.  If that's the 
case, it should just come out and say it rather than tangentially introducing 
ambiguous testing requirements that don't make a lot of sense.

The PEP also makes some unsupported claims about saving labor.  My 
understanding is the IronPython and Jython tend to re-implement modules using 
native constructs.  Even with PyPy, the usual pure python idioms aren't 
necessarily what is best for PyPy, so I expect some rewriting there also.  It 
seems the lion's share of the work in making other implementations has to do 
with interpreter details and whatnot -- I would be surprised if the likes of 
bisect or heapq took even one-tenth of one percent of the total development 
time for any of the other implementations.


> 
> I did not really see an answer to these concerns:
> 
> http://mail.python.org/pipermail/python-dev/2011-April/110672.html
> 
> Antoine does seem sold on the 100% branch coverage requirement and views it 
> as pointless. I disagree. =)
> 
> As for the exception Stefan is saying may be granted, that is not in the PEP 
> so I consider it unimportant. If we really feel the desire to grant an 
> exception we can (since we can break any of our own rules that we 
> collectively choose to), but I'm assuming we won't.
>  
> http://mail.python.org/pipermail/python-dev/2011-April/110675.html
> 
> Raymond thinks that have a testing requirement conflates having 
> implementations match vs. APIs.

That is not an accurate restatement of my post.

> Well, as we all know, the stdlib ends up having its implementation details 
> relied upon constantly by people whether they mean to or not,  so making sure 
> that this is properly tested helps deal with this known reality.

If you're saying that all implementation details (including internal branching 
logic) are now guaranteed behaviors, then I think this PEP has completely lost 
its way.  I don't know of any implementors asking for this.


> This is a damned-if-you-do-damned-if-you-don't situation. The first draft of 
> this PEP said to be "semantically equivalent w/ divergence where technically 
> required", but I got pushback from being too wishy-washy w/ lack of concrete 
> details. So I introduce a concrete metric that some are accusing of being 
> inaccurate for the goals of the PEP. I'm screwed or I'm screwed. =) So I am 
> choosing to go with the one that has a side benefit of also increasing test 
> coverage.

Maybe that is just an indication that the proposal isn't mature yet.   To me, 
it doesn't seem well thought out and isn't realistic.  


> Now if people would actually support simply not accepting any more C modules 
> into the Python stdlib (this does not apply to CPython's stdlib), then I'm 
> all for that.
> I only went with the "accelerator modules are okay" route to help get 
> acceptance for the PEP. But if people are willing to go down a more stringent 
> route and say that any module which uses new C code is considered 
> CPython-specific and thus any acceptance of such modules will be damn hard to 
> accomplish as it will marginalize the value of the code, that's fine by me.

Is that what people want?   For example, do we want to accept a C version of 
decimal?  Without it, the decimal module is unusable for people with high 
volumes of data.  Do we want things like an xz compressor to be written in pure 
python and only in Python?  I don't think this benefits our users.

I'm not really clear what it is you're trying to get at.  For PyPy, IronPython, 
and Jython to succeed, does the CPython project need to come to a halt?  I 
don't think many people here r

Re: [Python-Dev] python and super

[Steven D'Aprano]

Michael Foord wrote:

On 15/04/2011 02:23, Steven D'Aprano wrote:

[...]
If we treat django's failure to use super as a bug, you want the 
Python language to work-around that bug so that:


What you say (that this particular circumstance could be treated as a 
bug in django) is true, however consider the "recently" introduced 
problem caused by object.__init__ not taking arguments. This makes it 
impossible to use super correctly in various circumstances.

[...]
It is impossible to inherit from both C and A and have all parent 
__init__ methods called correctly. Changing the semantics of super as 
described would fix this problem.


So you say. I don't have an an opinion on whether or not you are 
technically correct, but adding DWIM black-magic to super scares me. It 
scares me even if it were guaranteed to *only* apply to __init__, but if 
it applied to arbitrary methods, it frankly terrifies me.


If it were limited to only apply to __init__, there would be a constant 
stream of requests that we loosen the restriction and "make super just 
work" for all methods, despite the dangers of DWIM code.





--
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Matt Billenstein]

On Sat, Apr 16, 2011 at 01:30:13PM +0200, Antoine Pitrou wrote:
> On Sat, 16 Apr 2011 00:41:03 +
> Matt Billenstein  wrote:
> > 
> > Slightly less crude benchmark showing simplejson is quite a bit faster:
> > 
> > http://pastebin.com/g1WqUPwm
> > 
> > 250ms vs 5.5s encoding and decoding an 11KB json object 1000 times...
> 
> This doesn't have much value if you don't say which version of Python
> you ran json with. You should use 3.2, otherwise you might miss some
> optimizations.

Yes, that was 2.6.5 -- 3.2 native json is comparable to simplejson here taking
about 330ms...

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

On Sat, 16 Apr 2011 23:48:45 +0100
Michael Foord  wrote:

> On 16/04/2011 22:28, "Martin v. Löwis" wrote:
> > Am 16.04.2011 21:13, schrieb Vinay Sajip:
> >> Martin v. Löwis  v.loewis.de>  writes:
> >>
> >>> Does it actually need improvement?
> >> I can't actually say, but I assume it keeps changing for the better - 
> >> albeit
> >> slowly. I wasn't thinking of specific improvements, just the idea of 
> >> continuous
> >> improvement in general...
> > Hmm. I cannot believe in the notion of "continuous improvement"; I'd
> > guess that it is rather "continuous change".
> >
> > I can see three possible areas of improvment:
> > 1. Bugs: if there are any, they should clearly be fixed. However, JSON
> > is a simple format, so the implementation should be able to converge
> > to something fairly correct quickly.
> > 2. Performance: there is always room for performance improvements.
> > However, I strongly recommend to not bother unless a severe
> > bottleneck can be demonstrated.
> Well, there was a 5x speedup demonstrated comparing simplejson to the 
> standard library json module.

No.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Steven D'Aprano]

Brett Cannon wrote:

In the grand python-dev tradition of "silence means acceptance", I consider
this PEP finalized and implicitly accepted.


How long does that silence have to last?

I didn't notice a definition of what counts as "100% branch coverage". 
Apologies if I merely failed to notice it, but I think it should be 
explicitly defined.


Presumably it means that any time you have an explicit branch 
(if...elif...else, try...except...else, for...else, etc.) you need a 
test that goes down each branch. But it isn't clear to me whether it's 
sufficient to test each branch in isolation, or whether you need to test 
all combinations.


That is, if you have five branches, A or B, C or D, E or F, G or H, I or 
J, within a single code unit (function? something else?), is it 
sufficient to have at least one test that goes down each of A...J, or do 
you need to explicitly test each of:


A-C-E-G-I
A-C-E-G-J
A-C-E-H-I
A-C-E-H-J
A-C-F-G-I
...
B-D-F-H-J

(10 tests versus 32 tests).

If the latter, this could become impractical *very* fast. But if not, I 
don't see how we can claim 100% coverage when there are code paths which 
are never tested.


At the very least, I think you need to explicitly define what you mean 
by "100% branch coverage". Possibly this will assist in the disagreement 
between you and Antoine re "100% versus "comprehensive" coverage.




--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Michael Foord]

On 16/04/2011 22:28, "Martin v. Löwis" wrote:

Am 16.04.2011 21:13, schrieb Vinay Sajip:

Martin v. Löwis  v.loewis.de>  writes:


Does it actually need improvement?

I can't actually say, but I assume it keeps changing for the better - albeit
slowly. I wasn't thinking of specific improvements, just the idea of continuous
improvement in general...

Hmm. I cannot believe in the notion of "continuous improvement"; I'd
guess that it is rather "continuous change".

I can see three possible areas of improvment:
1. Bugs: if there are any, they should clearly be fixed. However, JSON
is a simple format, so the implementation should be able to converge
to something fairly correct quickly.
2. Performance: there is always room for performance improvements.
However, I strongly recommend to not bother unless a severe
bottleneck can be demonstrated.
Well, there was a 5x speedup demonstrated comparing simplejson to the 
standard library json module. That sound like *very* worth pursuing (and 
crazy not to pursue). I've had json serialisation be the bottleneck in 
web applications generating several megabytes of json for some requests.


All the best,

Michael Foord

3. API changes: people apparently want JSON to be more flexible wrt.
Python types that are not directly supported in JSON. I'd rather take
a conservative approach here, involving a lot of people before adding
an API feature or even an incompatibility.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Eric Snow]

On Sat, Apr 16, 2011 at 3:54 PM, Antoine Pitrou  wrote:

>
> Hmm, what's the difference between "the Python stdlib" and "CPython's
> stdlib"?
>
> I'm also not sure how you would enforce that anyway. If it means
> using ctypes to interface with system C libraries, I'm -10 on it :)
>
>
Sounds like Brett is talking about the distinction apparently discussed at
the language summit ("Standalone Standard Library"):

http://blog.python.org/2011/03/2011-language-summit-report.html

-eric



> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Vinay Sajip]

Martin v. Löwis  v.loewis.de> writes:

> I can see three possible areas of improvment:
> 1. Bugs: if there are any, they should clearly be fixed. However, JSON
>is a simple format, so the implementation should be able to converge
>to something fairly correct quickly.
> 2. Performance: there is always room for performance improvements.
>However, I strongly recommend to not bother unless a severe
>bottleneck can be demonstrated.
> 3. API changes: people apparently want JSON to be more flexible wrt.
>Python types that are not directly supported in JSON. I'd rather take
>a conservative approach here, involving a lot of people before adding
>an API feature or even an incompatibility.

I agree with all these points, though I was only thinking of Nos. 1 and 2. Over
a longer timeframe, improvements may also come with changes in the spec
(unlikely in the short and medium term, but you never know in the long term).

Regards,

Vinay Sajip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Antoine Pitrou]

On Sat, 16 Apr 2011 14:45:52 -0700
Brett Cannon  wrote:
> On Sat, Apr 16, 2011 at 14:23, Stefan Krah  wrote:
> 
> > Brett Cannon  wrote:
> > > In the grand python-dev tradition of "silence means acceptance", I
> > consider
> > > this PEP finalized and implicitly accepted.
> >
> > I did not really see an answer to these concerns:
> >
> > http://mail.python.org/pipermail/python-dev/2011-April/110672.html
> >
> 
> Antoine does seem sold on the 100% branch coverage requirement and views it
> as pointless.

Not really. I think this is an unreasonable requirement because of the
reasons I've stated in my previous messages :)
If you rephrase it to remove the "100% coverage" requirement and
replace it by something like "comprehensive coverage", then I'm ok.

> Now if people would actually support simply not accepting any more C modules
> into the Python stdlib (this does not apply to CPython's stdlib), then I'm
> all for that.

Hmm, what's the difference between "the Python stdlib" and "CPython's
stdlib"?

I'm also not sure how you would enforce that anyway. If it means
using ctypes to interface with system C libraries, I'm -10 on it :)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Brett Cannon]

On Sat, Apr 16, 2011 at 14:23, Stefan Krah  wrote:

> Brett Cannon  wrote:
> > In the grand python-dev tradition of "silence means acceptance", I
> consider
> > this PEP finalized and implicitly accepted.
>
> I did not really see an answer to these concerns:
>
> http://mail.python.org/pipermail/python-dev/2011-April/110672.html
>

Antoine does seem sold on the 100% branch coverage requirement and views it
as pointless. I disagree. =)

As for the exception Stefan is saying may be granted, that is not in the PEP
so I consider it unimportant. If we really feel the desire to grant an
exception we can (since we can break any of our own rules that we
collectively choose to), but I'm assuming we won't.


> http://mail.python.org/pipermail/python-dev/2011-April/110675.html
>

Raymond thinks that have a testing requirement conflates having
implementations match vs. APIs. Well, as we all know, the stdlib ends up
having its implementation details relied upon constantly by people whether
they mean to or not,  so making sure that this is properly tested helps deal
with this known reality.

This is a damned-if-you-do-damned-if-you-don't situation. The first draft of
this PEP said to be "semantically equivalent w/ divergence where technically
required", but I got pushback from being too wishy-washy w/ lack of concrete
details. So I introduce a concrete metric that some are accusing of being
inaccurate for the goals of the PEP. I'm screwed or I'm screwed. =) So I am
choosing to go with the one that has a side benefit of also increasing test
coverage.

Now if people would actually support simply not accepting any more C modules
into the Python stdlib (this does not apply to CPython's stdlib), then I'm
all for that. I only went with the "accelerator modules are okay" route to
help get acceptance for the PEP. But if people are willing to go down a more
stringent route and say that any module which uses new C code is considered
CPython-specific and thus any acceptance of such modules will be damn hard
to accomplish as it will marginalize the value of the code, that's fine by
me.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Martin v. Löwis]

Am 16.04.2011 21:13, schrieb Vinay Sajip:
> Martin v. Löwis  v.loewis.de> writes:
> 
>> Does it actually need improvement?
> 
> I can't actually say, but I assume it keeps changing for the better - albeit
> slowly. I wasn't thinking of specific improvements, just the idea of 
> continuous
> improvement in general...

Hmm. I cannot believe in the notion of "continuous improvement"; I'd
guess that it is rather "continuous change".

I can see three possible areas of improvment:
1. Bugs: if there are any, they should clearly be fixed. However, JSON
   is a simple format, so the implementation should be able to converge
   to something fairly correct quickly.
2. Performance: there is always room for performance improvements.
   However, I strongly recommend to not bother unless a severe
   bottleneck can be demonstrated.
3. API changes: people apparently want JSON to be more flexible wrt.
   Python types that are not directly supported in JSON. I'd rather take
   a conservative approach here, involving a lot of people before adding
   an API feature or even an incompatibility.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Stefan Krah]

Brett Cannon  wrote:
> In the grand python-dev tradition of "silence means acceptance", I consider
> this PEP finalized and implicitly accepted.

I did not really see an answer to these concerns:

http://mail.python.org/pipermail/python-dev/2011-April/110672.html
http://mail.python.org/pipermail/python-dev/2011-April/110675.html



Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Brett Cannon]

In the grand python-dev tradition of "silence means acceptance", I consider
this PEP finalized and implicitly accepted.

On Tue, Apr 12, 2011 at 15:07, Brett Cannon  wrote:

> Here is the next draft of the PEP. I changed the semantics requirement to
> state that 100% branch coverage is required for any Python code that is
> being replaced by accelerated C code instead of the broad "must be
> semantically equivalent". Also tweaked wording here and there to make
> certain things more obvious.
>
> --
>
> PEP: 399
> Title: Pure Python/C Accelerator Module Compatibility Requirements
>
> Version: $Revision: 88219 $
> Last-Modified: $Date: 2011-01-27 13:47:00 -0800 (Thu, 27 Jan 2011) $
> Author: Brett Cannon 
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 04-Apr-2011
> Python-Version: 3.3
> Post-History: 04-Apr-2011, 12-Apr-2011
>
>
> Abstract
> 
>
> The Python standard library under CPython contains various instances
> of modules implemented in both pure Python and C (either entirely or
> partially). This PEP requires that in these instances that the
> C code *must* pass the test suite used for the pure Python code
> so as to act as much as a drop-in replacement as possible
> (C- and VM-specific tests are exempt). It is also required that new
>
> C-based modules lacking a pure Python equivalent implementation get
> special permissions to be added to the standard library.
>
>
> Rationale
> =
>
> Python has grown beyond the CPython virtual machine (VM). IronPython_,
> Jython_, and PyPy_ all currently being viable alternatives to the
> CPython VM. This VM ecosystem that has sprung up around the Python
> programming language has led to Python being used in many different
> areas where CPython cannot be used, e.g., Jython allowing Python to be
> used in Java applications.
>
> A problem all of the VMs other than CPython face is handling modules
> from the standard library that are implemented (to some extent) in C.
>
> Since they do not typically support the entire `C API of Python`_ they
> are unable to use the code used to create the module. Often times this
> leads these other VMs to either re-implement the modules in pure
> Python or in the programming language used to implement the VM
> (e.g., in C# for IronPython). This duplication of effort between
> CPython, PyPy, Jython, and IronPython is extremely unfortunate as
> implementing a module *at least* in pure Python would help mitigate
> this duplicate effort.
>
> The purpose of this PEP is to minimize this duplicate effort by
> mandating that all new modules added to Python's standard library
> *must* have a pure Python implementation _unless_ special dispensation
> is given. This makes sure that a module in the stdlib is available to
> all VMs and not just to CPython (pre-existing modules that do not meet
> this requirement are exempt, although there is nothing preventing
> someone from adding in a pure Python implementation retroactively).
>
>
> Re-implementing parts (or all) of a module in C (in the case
> of CPython) is still allowed for performance reasons, but any such
> accelerated code must pass the same test suite (sans VM- or C-specific
> tests) to verify semantics and prevent divergence. To accomplish this,
> the test suite for the module must have 100% branch coverage of the
> pure Python implementation before the acceleration code may be added.
>
> This is to prevent users from accidentally relying
> on semantics that are specific to the C code and are not reflected in
> the pure Python implementation that other VMs rely upon. For example,
> in CPython 3.2.0, ``heapq.heappop()`` does an explicit type
> check in its accelerated C code while the Python code uses duck
> typing::
>
>
> from test.support import import_fresh_module
>
> c_heapq = import_fresh_module('heapq', fresh=['_heapq'])
> py_heapq = import_fresh_module('heapq', blocked=['_heapq'])
>
>
> class Spam:
> """Tester class which defines no other magic methods but
> __len__()."""
> def __len__(self):
> return 0
>
>
> try:
> c_heapq.heappop(Spam())
> except TypeError:
> # Explicit type check failure: "heap argument must be a list"
>
> pass
>
> try:
> py_heapq.heappop(Spam())
> except AttributeError:
> # Duck typing failure: "'Foo' object has no attribute 'pop'"
>
> pass
>
> This kind of divergence is a problem for users as they unwittingly
> write code that is CPython-specific. This is also an issue for other
> VM teams as they have to deal with bug reports from users thinking
> that they incorrectly implemented the module when in fact it was
> caused by an untested case.
>
>
> Details
> ===
>
> Starting in Python 3.3, any modules added to the standard library must
> have a pure Python implementation. This rule can only be ignored if
> the Python development team grants a special exemption for the m

Re: [Python-Dev] Status of json (simplejson) in cpython

[Vinay Sajip]

Martin v. Löwis  v.loewis.de> writes:

> Does it actually need improvement?

I can't actually say, but I assume it keeps changing for the better - albeit
slowly. I wasn't thinking of specific improvements, just the idea of continuous
improvement in general...

Regards,

Vinay Sajip



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Martin v. Löwis]

> I agree, my suggestion is orthogonal to the question of who maintains stdlib
> json. But if the json module is languishing in comparison to simplejson, then
> bringing the code bases closer together may be worthwhile.

Right: *if* the module is languishing. But it's not. It just diverges.

> It may be that no-one is willing or able to serve as an effective maintainer
> of stdlib json, but assuming that Bob will continue to maintain and improve
> simplejson

Does it actually need improvement?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

On Sat, 16 Apr 2011 16:47:49 + (UTC)
Vinay Sajip  wrote:
> 
> > What you're proposing doesn't address the question of who is going to
> > do the ongoing maintenance.
> 
> I agree, my suggestion is orthogonal to the question of who maintains stdlib
> json.

No, that's not what I'm talking about. The json module *is* maintained
(take a look at "hg log"), even though it may be less active than
simplejson (but simplejson doesn't receive many changes either).

I am talking about maintenance of the "shared codebase" you are talking
about. Mandating a single codebase between two different languages
(Python 2 and Python 3) and two different libraries (json and
simplejson) comes at a high maintenance cost, and it's not obvious in
your proposal who will bear that cost in the long run (you?). It is not
a one-time cost, but an ongoing one.

> Bob has said he isn't interested in Python 3, but he has said that "if
> someone contributes the code to make simplejson work in Python 3 I'm willing
> to apply the patches run the tests against any future changes."

I can't speak for Bob, but this assumes the patches are not invasive
and don't degrade performance. It's not obvious that will be the case.

> Bob made a comment in passing that simplejson (Python) is about as fast as
> stdlib json (C extension), on 2.x.

I think Bob tested with an outdated version of the stdlib json module
(2.6 or 2.7, perhaps). In my latest measurements, the 3.2 json C module
is as fast as the C simplejson module, the only difference being in
parsing of numbers, which is addressed in
http://bugs.python.org/issue11856

> That may or may not prove to be the case on
> 3.x, but at least it is now possible to run simplejson on 3.x (Python only, so
> far) to make a comparison.

Feel free to share your numbers.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

> > I've contributed a couple of patches myself after they were integrated
> > to CPython (they are part of the performance improvements Bob is talking
> > about), but that was exceptional. Backporting a patch to another project
> > with a different directory structure, a slightly different code, etc. is
> > tedious and not very rewarding for us Python core developers, while we
> > could do other work on our limited free time.
> Sure, I can understand that, but wouldn't it be easier if the two
> versions were kept in better sync (mostly removing the "slightly
> different code" part)?

You are assuming that we intend to backport all our json patches to
simplejson. I can't speak for other people, but I'm personally not
interested in doing that work (even if you find an "easier" scheme than
the current one).

Also, as Raymond said, it's not much of an issue if json and simplejson
diverge. Bob said he had no interest in porting simplejson to 3.x, while
we don't have any interest in making non-bugfix changes to 2.x json. As
long as basic functionality is identical and compliance to the spec is
ensured, I think most common uses are covered by both libraries.

So, unless you manage to find a scheme where porting patches is almost
zero-cost (for either us or Bob), I don't think it will happen.

> I'm thinking of a relation along the lines of Michael Foord's
> unittest2 (except maybe inverted, in that unittest2 is a backport of a
> next version's unittest)

Well, the big difference here is that Michael maintains both the stdlib
version and the standalone project, meaning he's committed to avoid any
divergence between the two codebases.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Vinay Sajip]

Hi Antoine,

Antoine Pitrou  pitrou.net> writes:

> What you're proposing doesn't address the question of who is going to
> do the ongoing maintenance.

I agree, my suggestion is orthogonal to the question of who maintains stdlib
json. But if the json module is languishing in comparison to simplejson, then
bringing the code bases closer together may be worthwhile. I've just been
experimenting with the feasibility of getting simplejson running on Python
3.x, and at present I have it working in the sense of all tests passing on
3.2. 

Bob has said he isn't interested in Python 3, but he has said that "if
someone contributes the code to make simplejson work in Python 3 I'm willing
to apply the patches run the tests against any future changes."

I take this to mean that Bob is undertaking to keep the codebase working in
both 2.x and 3.x in the future (though I'm sure he'll correct me if I've got it
wrong). 

I'm also assuming Bob will be receptive to patches which are functional
improvements added in stdlib json in 3.x, as his comments seem to indicate
that this is the case.

ISTM that for some library maintainers who are invested in 2.x and who don't
have the time or inclination to manage separate 2.x and 3.x codebases, a
common codebase is the way to go. This certainly seems to be the case for pip
and virtualenv, which we recently got running under Python 3 using a common
codebase approach. Certainly, the amount of work required for ongoing
maintenance can be much less, and only a little discipline is needed when
adding new code.

Bob made a comment in passing that simplejson (Python) is about as fast as
stdlib json (C extension), on 2.x. That may or may not prove to be the case on
3.x, but at least it is now possible to run simplejson on 3.x (Python only, so
far) to make a comparison.

It may be that no-one is willing or able to serve as an effective maintainer
of stdlib json, but assuming that Bob will continue to maintain and improve
simplejson and if an automatic mechanism for converting from a 3.x-compatible
simplejson to json can be made to work, that could be a way forward.

It's obviously early days to see how things will pan out, but it seems worth
exploring the avenue a little further, if Bob is amenable to this approach.

Regards,

Vinay

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Xavier Morel]

On 2011-04-16, at 17:25 , Antoine Pitrou wrote:
> Le samedi 16 avril 2011 à 17:07 +0200, Xavier Morel a écrit :
>> On 2011-04-16, at 16:52 , Antoine Pitrou wrote:
>>> Le samedi 16 avril 2011 à 16:42 +0200, Dirkjan Ochtman a écrit :
 On Sat, Apr 16, 2011 at 16:19, Antoine Pitrou  wrote:
> What you're proposing doesn't address the question of who is going to
> do the ongoing maintenance. Bob apparently isn't interested in
> maintaining stdlib code, and python-dev members aren't interested in
> maintaining simplejson (assuming it would be at all possible). Since
> both groups of people want to work on separate codebases, I don't see
> how sharing a single codebase would be possible.
 
 From reading this thread, it seems to me like the proposal is that Bob
 maintains a simplejson for both 2.x and 3.x and that the current
 stdlib json is replaced by a (trivially changed) version of
 simplejson.
>>> 
>>> The thing is, we want to bring our own changes to the json module and
>>> its tests (and have already done so, although some have been backported
>>> to simplejson).
>> 
>> Depending on what those changes are, would it not be possible to apply
>> the vast majority of them to simplejson itself?
> 
> Sure, but the thing is, I don't *think* we are interested in backporting
> stuff to simplejson much more than Bob is interested in porting stuff to
> the json module.
I was mostly thinking it could work the other way around, really: simplejson 
seems to move slightly faster than the stdlib's json (though it's not a 
high-churn module either these days), so improvements (from Python and third 
parties alike) could be applied there first and then forward-ported, rather 
than the other way around.

> I've contributed a couple of patches myself after they were integrated
> to CPython (they are part of the performance improvements Bob is talking
> about), but that was exceptional. Backporting a patch to another project
> with a different directory structure, a slightly different code, etc. is
> tedious and not very rewarding for us Python core developers, while we
> could do other work on our limited free time.
Sure, I can understand that, but wouldn't it be easier if the two versions were 
kept in better sync (mostly removing the "slightly different code" part)?

>> Furthermore, now that python uses Mercurial, it should be possible (or
>> even easy) to use a versioned queue (via MQ) for the trivial
>> adaptation, and the temporary alterations (things which will likely be
>> merged back into simplejson but are not yet, stuff like that) should
>> it not?
> Perhaps, perhaps not. That would require someone motivated to put it in
> place, ensure that it doesn't get in the way, document it, etc.
> Honestly, I don't think maintaining a single stdlib module should
> require such an amount of logistics.

I don't think mercurial queues really amount to logistic, it takes a bit of 
learning but fundamentally they're not much work, and make synchronization with 
upstream packages much easier. Which would (I believe) benefit both projects 
and — ultimately — language users by avoiding too extreme differences (on both 
API/features and performances).

I'm thinking of a relation along the lines of Michael Foord's unittest2 (except 
maybe inverted, in that unittest2 is a backport of a next version's unittest)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

On Sat, 16 Apr 2011 18:04:53 +0200
Stefan Behnel  wrote:
> 
> Well, if that is not possible, then the CPython devs will have a hard time 
> maintaining the json accelerator module in the long run. I quickly skipped 
> through the github version in simplejson, and it truly is some complicated 
> piece of code. Not in the sense that the code is ununderstandable, it's 
> actually fairly straight forward string processing code, but it's so 
> extremely optimised and tailored and has so much code duplicated for the 
> bytes and unicode types (apparently following the copy+paste+adapt pattern) 
> that it will be pretty hard to adapt to future changes of CPython, 
> especially the upcoming PEP 393 implementation.

Well, first, the Python 3 version doesn't have the duplicated code
since it doesn't accept bytes input. Second, it's not that complicated,
and we have already brought improvements to it, meaning we know the
code ("we" is at least Raymond and I). For example, see
http://bugs.python.org/issue11856 for a pending patch.

> Maintaining this is clearly no fun.

No more than any optimized piece of C code, but no less either.
It's actually quite straightforward compared to other classes such as
TextIOWrapper.

PEP 393 will be a challenge for significant chunks of the interpreter
and extension modules; it's not a json-specific issue.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Bob Ippolito]

On Saturday, April 16, 2011, Antoine Pitrou  wrote:
> Le samedi 16 avril 2011 à 17:07 +0200, Xavier Morel a écrit :
>> On 2011-04-16, at 16:52 , Antoine Pitrou wrote:
>> > Le samedi 16 avril 2011 à 16:42 +0200, Dirkjan Ochtman a écrit :
>> >> On Sat, Apr 16, 2011 at 16:19, Antoine Pitrou  wrote:
>> >>> What you're proposing doesn't address the question of who is going to
>> >>> do the ongoing maintenance. Bob apparently isn't interested in
>> >>> maintaining stdlib code, and python-dev members aren't interested in
>> >>> maintaining simplejson (assuming it would be at all possible). Since
>> >>> both groups of people want to work on separate codebases, I don't see
>> >>> how sharing a single codebase would be possible.
>> >>
>> >> From reading this thread, it seems to me like the proposal is that Bob
>> >> maintains a simplejson for both 2.x and 3.x and that the current
>> >> stdlib json is replaced by a (trivially changed) version of
>> >> simplejson.
>> >
>> > The thing is, we want to bring our own changes to the json module and
>> > its tests (and have already done so, although some have been backported
>> > to simplejson).
>>
>> Depending on what those changes are, would it not be possible to apply
>> the vast majority of them to simplejson itself?
>
> Sure, but the thing is, I don't *think* we are interested in backporting
> stuff to simplejson much more than Bob is interested in porting stuff to
> the json module.

I've backported every useful patch (for 2.x) I noticed from json to
simplejson. Would be happy to apply any that I missed if anyone can
point these out.

> I've contributed a couple of patches myself after they were integrated
> to CPython (they are part of the performance improvements Bob is talking
> about), but that was exceptional. Backporting a patch to another project
> with a different directory structure, a slightly different code, etc. is
> tedious and not very rewarding for us Python core developers, while we
> could do other work on our limited free time.

That's exactly why I am not interested in stdlib maintenance myself, I
only use 2.x and that's frozen... so I can't maintain the version we
would actually use.

> Also, some types of work would be tedious to backport, for example if we
> refactor the tests to test both the C and Python implementations.

simplejson's test suite has tested both for quite some time.

>> Furthermore, now that python uses Mercurial, it should be possible (or
>> even easy) to use a versioned queue (via MQ) for the trivial
>> adaptation, and the temporary alterations (things which will likely be
>> merged back into simplejson but are not yet, stuff like that) should
>> it not?
>
> Perhaps, perhaps not. That would require someone motivated to put it in
> place, ensure that it doesn't get in the way, document it, etc.
> Honestly, I don't think maintaining a single stdlib module should
> require such an amount of logistics.

It certainly shouldn't, especially because neither of them changes very fast.

-bob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Stefan Behnel]

Antoine Pitrou, 16.04.2011 16:19:

On Sat, 16 Apr 2011 09:50:25 + (UTC)
Vinay Sajip wrote:


If it is generally considered desirable to maintain some synchrony between
simplejson and stdlib json, then since Bob has stated that he no interest in
Python 3, it may be better to:

1. Convert the simplejson codebase so that it runs on both Python 2 and 3
(without running 2to3 on it). Once this is done, if upstream accepts these
changes, ongoing maintenance will be fairly simple for upstream, and changes
only really need to consider exception and string/byte literal syntax, for the
most part.
2. Merge this new simplejson with stdlib json for 3.3.


What you're proposing doesn't address the question of who is going to
do the ongoing maintenance. Bob apparently isn't interested in
maintaining stdlib code, and python-dev members aren't interested in
maintaining simplejson (assuming it would be at all possible). Since
both groups of people want to work on separate codebases, I don't see
how sharing a single codebase would be possible.


Well, if that is not possible, then the CPython devs will have a hard time 
maintaining the json accelerator module in the long run. I quickly skipped 
through the github version in simplejson, and it truly is some complicated 
piece of code. Not in the sense that the code is ununderstandable, it's 
actually fairly straight forward string processing code, but it's so 
extremely optimised and tailored and has so much code duplicated for the 
bytes and unicode types (apparently following the copy+paste+adapt pattern) 
that it will be pretty hard to adapt to future changes of CPython, 
especially the upcoming PEP 393 implementation. Maintaining this is clearly 
no fun.


Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

Le samedi 16 avril 2011 à 17:07 +0200, Xavier Morel a écrit :
> On 2011-04-16, at 16:52 , Antoine Pitrou wrote:
> > Le samedi 16 avril 2011 à 16:42 +0200, Dirkjan Ochtman a écrit :
> >> On Sat, Apr 16, 2011 at 16:19, Antoine Pitrou  wrote:
> >>> What you're proposing doesn't address the question of who is going to
> >>> do the ongoing maintenance. Bob apparently isn't interested in
> >>> maintaining stdlib code, and python-dev members aren't interested in
> >>> maintaining simplejson (assuming it would be at all possible). Since
> >>> both groups of people want to work on separate codebases, I don't see
> >>> how sharing a single codebase would be possible.
> >> 
> >> From reading this thread, it seems to me like the proposal is that Bob
> >> maintains a simplejson for both 2.x and 3.x and that the current
> >> stdlib json is replaced by a (trivially changed) version of
> >> simplejson.
> > 
> > The thing is, we want to bring our own changes to the json module and
> > its tests (and have already done so, although some have been backported
> > to simplejson).
> 
> Depending on what those changes are, would it not be possible to apply
> the vast majority of them to simplejson itself?

Sure, but the thing is, I don't *think* we are interested in backporting
stuff to simplejson much more than Bob is interested in porting stuff to
the json module.

I've contributed a couple of patches myself after they were integrated
to CPython (they are part of the performance improvements Bob is talking
about), but that was exceptional. Backporting a patch to another project
with a different directory structure, a slightly different code, etc. is
tedious and not very rewarding for us Python core developers, while we
could do other work on our limited free time.

Also, some types of work would be tedious to backport, for example if we
refactor the tests to test both the C and Python implementations.

> Furthermore, now that python uses Mercurial, it should be possible (or
> even easy) to use a versioned queue (via MQ) for the trivial
> adaptation, and the temporary alterations (things which will likely be
> merged back into simplejson but are not yet, stuff like that) should
> it not?

Perhaps, perhaps not. That would require someone motivated to put it in
place, ensure that it doesn't get in the way, document it, etc.
Honestly, I don't think maintaining a single stdlib module should
require such an amount of logistics.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Xavier Morel]

On 2011-04-16, at 16:52 , Antoine Pitrou wrote:
> Le samedi 16 avril 2011 à 16:42 +0200, Dirkjan Ochtman a écrit :
>> On Sat, Apr 16, 2011 at 16:19, Antoine Pitrou  wrote:
>>> What you're proposing doesn't address the question of who is going to
>>> do the ongoing maintenance. Bob apparently isn't interested in
>>> maintaining stdlib code, and python-dev members aren't interested in
>>> maintaining simplejson (assuming it would be at all possible). Since
>>> both groups of people want to work on separate codebases, I don't see
>>> how sharing a single codebase would be possible.
>> 
>> From reading this thread, it seems to me like the proposal is that Bob
>> maintains a simplejson for both 2.x and 3.x and that the current
>> stdlib json is replaced by a (trivially changed) version of
>> simplejson.
> 
> The thing is, we want to bring our own changes to the json module and
> its tests (and have already done so, although some have been backported
> to simplejson).

Depending on what those changes are, would it not be possible to apply the vast 
majority of them to simplejson itself?

Furthermore, now that python uses Mercurial, it should be possible (or even 
easy) to use a versioned queue (via MQ) for the trivial adaptation, and the 
temporary alterations (things which will likely be merged back into simplejson 
but are not yet, stuff like that) should it not?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python Language Summit at EuroPython: 19th June

[Michael Foord]

Hello all,

This is an invite to all core-python developers, and developers of 
alternative implementations, to attend the Python Language Summit at 
EuroPython. The summit will be on June 19th and EuroPython this year 
will be held at the beautiful city of Florence in Italy.


http://ep2011.europython.eu/

If you are not a core-Python developer but would like to attend then 
please email me privately and I will let you know if spaces are 
available. If you are a core developer, or you have received a direct 
invitation, then please respond by private email to let me know if you 
are able to attend. A maybe is fine, you can always change your mind 
later. Attending for only part of the day is fine.


We expect the summit to run from 10am - 4pm with appropriate breaks.

Like previous language summits it is an opportunity to discuss topics 
like, Python 3 adoption, PEPs and changes for Python 3.3, the future of 
Python 2.7, documentation, package index, web site, etc.


If you have topics you'd like to discuss at the language summit please 
let me know.


Volunteers for taking notes at the language summit, for posting to 
Python-dev and the Python Insiders blog after the event, would be much 
appreciated.


All the best,

Michael Foord

N.B. Due to my impending doom (oops, I mean impending fatherhood) I am 
not yet 100% certain I will be able to attend. If I can't I will arrange 
for someone else to chair.


--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

Le samedi 16 avril 2011 à 16:42 +0200, Dirkjan Ochtman a écrit :
> On Sat, Apr 16, 2011 at 16:19, Antoine Pitrou  wrote:
> > What you're proposing doesn't address the question of who is going to
> > do the ongoing maintenance. Bob apparently isn't interested in
> > maintaining stdlib code, and python-dev members aren't interested in
> > maintaining simplejson (assuming it would be at all possible). Since
> > both groups of people want to work on separate codebases, I don't see
> > how sharing a single codebase would be possible.
> 
> From reading this thread, it seems to me like the proposal is that Bob
> maintains a simplejson for both 2.x and 3.x and that the current
> stdlib json is replaced by a (trivially changed) version of
> simplejson.

The thing is, we want to bring our own changes to the json module and
its tests (and have already done so, although some have been backported
to simplejson).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Dirkjan Ochtman]

On Sat, Apr 16, 2011 at 16:19, Antoine Pitrou  wrote:
> What you're proposing doesn't address the question of who is going to
> do the ongoing maintenance. Bob apparently isn't interested in
> maintaining stdlib code, and python-dev members aren't interested in
> maintaining simplejson (assuming it would be at all possible). Since
> both groups of people want to work on separate codebases, I don't see
> how sharing a single codebase would be possible.

>From reading this thread, it seems to me like the proposal is that Bob
maintains a simplejson for both 2.x and 3.x and that the current
stdlib json is replaced by a (trivially changed) version of
simplejson.

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Nick Coghlan]

On Sat, Apr 16, 2011 at 9:45 PM, Gustavo Narea  wrote:
> I reckon if this had been handled differently (i.e., making new releases
> and communicating it via the relevant channels [1]), we wouldn't have
> the situation we have right now.

Nope, we would have a situation where the security team were still
attempting to coordinate with the release managers to cut new source
releases and new binary releases, and not even releasing the source
level patches that *will* allow many, many people to fix the problem
on their own.

I don't agree that such a situation would be better than the status
quo (i.e. where both the problem and *how to fix it yourself* are
public knowledge).

The *exact* patches for all affected versions of Python are readily
available by checking the changesets linked from
http://bugs.python.org/issue11662#msg132517

> May I suggest that you adopt a policy for handling security issues like
> Django's?
> http://docs.djangoproject.com/en/1.3/internals/contributing/#reporting-security-issues

When the list of people potentially using the software is "anyone
running Linux or Mac OS X and an awful lot of people running Windows
or an embedded device", private pre-announcements simply aren't a
practical reality. Neither is "stopping all other development" when
most of the core development team aren't on the secur...@python.org
list and don't even know a security issue exists until it is announced
publicly. Take those two impractical steps out of the process, and
what you have *is* the python.org procedure for dealing with security
issues.

And when official python.org releases require coordination of
volunteers scattered around the planet, there is a harsh trade-off to
be made when it comes to deciding how long to wait before publishing
the information people need in order to fix the issue themselves.

Bumping the priority of the next round of python.org releases should
definitely be on the agenda, but the "rapid response" side of things
needs to come from the OS vendors with paid release engineers. Dealing
with security issues on behalf of their end users is one of the key
reasons they're getting paid for free software in the first place.

It may be worth asking the OS vendors whether or not they have
representatives that receive the secur...@python.org notifications,
and if not, why they haven't approached python-dev about receiving
such notifications.

> Cheers,
>
> [1] For example,
> ,
> , .

Agreed that an announcement should be made on those locations, with a
list of links to the exact changesets for each affected version.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

Hello Vinay,

On Sat, 16 Apr 2011 09:50:25 + (UTC)
Vinay Sajip  wrote:
> 
> If it is generally considered desirable to maintain some synchrony between
> simplejson and stdlib json, then since Bob has stated that he no interest in
> Python 3, it may be better to:
> 
> 1. Convert the simplejson codebase so that it runs on both Python 2 and 3
> (without running 2to3 on it). Once this is done, if upstream accepts these
> changes, ongoing maintenance will be fairly simple for upstream, and changes
> only really need to consider exception and string/byte literal syntax, for the
> most part.
> 2. Merge this new simplejson with stdlib json for 3.3.

What you're proposing doesn't address the question of who is going to
do the ongoing maintenance. Bob apparently isn't interested in
maintaining stdlib code, and python-dev members aren't interested in
maintaining simplejson (assuming it would be at all possible). Since
both groups of people want to work on separate codebases, I don't see
how sharing a single codebase would be possible.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Vinay Sajip]

Sandro Tosi  gmail.com> writes:

> Luckily, upstream is receptive for patches, so part of the job is to
> forward patches written for cpython not already in the upstream code.

Further to my earlier response to your post, I should mention that my fork of
simplejson at

https://github.com/vsajip/simplejson/

passes all 136 tests for Python 2.7 and 3.2 (not been able to test with 3.3a0
yet). No tests were skipped, though adjustments were made for binary/string
literals and for one case where sorting was applied to incompatible types in the
tests.

Test output is at https://gist.github.com/923019

Bob - If you're reading this, what would you say to having a look at my fork,
and comment on the feasibility of merging my changes back into your master? The
changes are fairly easy to understand, all tests pass, and it's a 2.x/3.x single
codebase, so maintenance should be easier than with multiple codebases.

Admittedly I haven't looked at the C code yet, but that's next on my list.

Regards,

Vinay Sajip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Gustavo Narea]

Hello,

On 15/04/11 13:30, Brian Curtin wrote:
> To me, the fix *was* released.

No, it wasn't. It was *committed* to the repository.

> Sure, no fancy installers were generated yet, but people who are
> susceptible to this issue 1) now know about it, and 2) have a way to
> patch their system *if needed*.

Well, that's a long shot. I doubt the people/organizations affected are
all aware. And I doubt they are all capable of patching their system or
getting a patched Python from a trusted party.

Three weeks after this security vulnerability was *publicly* reported on
bugs.python.org, and two days after it was semi-officially announced,
I'm still waiting for security updates for my Ubuntu and Debian systems!

I reckon if this had been handled differently (i.e., making new releases
and communicating it via the relevant channels [1]), we wouldn't have
the situation we have right now.

May I suggest that you adopt a policy for handling security issues like
Django's?
http://docs.djangoproject.com/en/1.3/internals/contributing/#reporting-security-issues

Cheers,

[1] For example,
,
, .

-- 
Gustavo Narea .
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

On Sat, 16 Apr 2011 00:41:03 +
Matt Billenstein  wrote:
> 
> Slightly less crude benchmark showing simplejson is quite a bit faster:
> 
> http://pastebin.com/g1WqUPwm
> 
> 250ms vs 5.5s encoding and decoding an 11KB json object 1000 times...

This doesn't have much value if you don't say which version of Python
you ran json with. You should use 3.2, otherwise you might miss some
optimizations.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Vinay Sajip]

Sandro Tosi  gmail.com> writes:

> The version we have in cpython of json is simplejson 2.0.9 highly
> patched (either because it was converted to py3k, and because of the
> normal flow of issues/bugfixes) while upstream have already released
> 2.1.13 .

I think you mean 2.1.3?
 
> Their 2 roads had diverged a lot, and since this blocks any further
> update of cpython's json from upstream, I'd like to close this gap.
> 
> This isn't exactly an easy task, and this email is more about a
> brainstorming on the ways we have to achieve the goal: being able to
> upgrade json to 2.1.13.
> 
> Luckily, upstream is receptive for patches, so part of the job is to
> forward patches written for cpython not already in the upstream code.
> 
> But how am I going to do this? let's do a brain-dump:
> 
> - the history goes back at changeset f686aced02a3 (May 2009, wow) when
> 2.0.9 was merged on trunk
> - I can navigate from that CS up to tip, and examine the diffs and see
> if they apply to 2.1.3 and prepare a set of patches to be forwarded
> - part of those diffs is about py3k conversion, that probably needs to
> be extended to other part of the upstream code not currently in
> cpython. For those "new" code parts, do you have some guides about
> porting a project to py3k? it would be my first time and other than
> building it and running it with python3 i don't know what to do :)
> - once (and if :) I reach the point where I've all the relevant
> patches applied on 2.1.3 what's the next step?

If it is generally considered desirable to maintain some synchrony between
simplejson and stdlib json, then since Bob has stated that he no interest in
Python 3, it may be better to:

1. Convert the simplejson codebase so that it runs on both Python 2 and 3
(without running 2to3 on it). Once this is done, if upstream accepts these
changes, ongoing maintenance will be fairly simple for upstream, and changes
only really need to consider exception and string/byte literal syntax, for the
most part.
2. Merge this new simplejson with stdlib json for 3.3.

I looked at step 1 a few weeks ago and have made some progress with it. I've
just forked simplejson on Github and posted my changes to my fork:

https://github.com/vsajip/simplejson

All 136 tests pass on Python 2.7 (just as a control/sanity check), and on Python
3.2, there are 4 failures and 12 errors - see complete results at

https://gist.github.com/923019

I haven't looked at the C extension yet, just the Python code. I believe most of
the test failures will be down to string literals in the tests which should be
bytes, e.g. test_unicode.py:test_ensure_ascii_false_bytestring_encoding.

So, it looks quite encouraging, and if you think my approach has merit, please
take a look at my fork, and give feedback/join in!

Note that I used the same approach when porting pip/virtualenv to Python 3,
which seems to have gone quite smoothly :-)

Regards,


Vinay Sajip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] http://docs.python.org/py3k/ pointing to 2.7

[Georg Brandl]

Am 15.04.2011 15:47, schrieb Victor Stinner:
> Le vendredi 15 avril 2011 à 13:34 +0200, Jesus Cea a écrit :
>> http://docs.python.org/py3k/ takes you to 2.7, by default.
>> 
>> Should we update it to point to 3.2?. If the point is to promote Python 3...
>> 
>> I would point it to 3.2, with a big "access to documentation to legacy
>> 2.7" (beside the small left column link). What do you think?.
> 
> http://docs.python.org/ points to 2.7 yes. I'm already reading 3.3 doc
> to develop with Python 2.5: I prefer the most recent doc, and the API is
> usually exactly the same. So I vote +1 to make 3.3 the default doc.
> Anyway, Python 2 is a dead language!

That is exactly the kind of comment that leads to FUD outside of python-dev.
Of course Python 2 is not dead as long as 2.7 is maintained.

As long as we're getting complaints from users that the examples in the
2.7 docs don't work with Python 2.5, there is no way making docs.python.org
show docs for 3.2 is the better choice.

> I don't like URL prefixes to indicate the version: "py3k/" for 3.2 (last
> Python 3 stable version), no prefix for 2.7, "release/2.6.6/" for 2.6,
> "dev/py3k/" for 3.3, etc.
> 
> Can't we keep it simple as:
>  - "2.6/" for 2.6
>  - "2.7/" for 2.7
>  - "3.1/" for 3.1
>  - "3.2/" for 3.2
>  - "3.3/" for 3.3

That we have.

>  - "2.x/" (or maybe just "2/"?) as a redirection to 2.7
>  - "3.x/" (or maybe just "3/"?) as a redirection to 3.3

That we could introduce, if it's really needed.  (We'd need to keep
the old URLs as well as redirections of course, introducing even
more ways to spell things.)

> http://docs.python.org/ may be a redirection (to
> http://docs.python.org/3.x/) instead of directly the doc. So it would be
> intuitive to replace 2 by 3 (or 3 by 2) in the URL.
> 
> 
> The http://www.python.org/doc/versions/ page uses other URLS:
> http://www.python.org/doc//

These are legacy URLs and are redirected, as you note.

> which are redirections to
> http://docs.python.org/release//. For example:
> http://www.python.org/doc/3.2/ is a redirection to
> http://docs.python.org/release/3.2/ (which give the same content than
> http://docs.python.org/py3k/ !).

No, they don't.  /release/version docs are those *released with* a specific
version, i.e. frozen at the point of the release. /py3k/ (and /version/)
docs are refreshed daily from the respective branch.

> On the left of http://docs.python.org/, you have links to other versions
> of the doc: 2.6, 3.2, 3.3... but not 3.1. 3.3 doc have links to 2.7 and
> 3.2 (not 2.6 or 3.1). If there is a rule to choose links on the left, I
> don't understand this rule.

3.1 is not really interesting for 2.7 users since 3.2 is out, while 2.6 is
not really interesting for 3.2 users.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com