Re: [Python-Dev] Status of json (simplejson) in cpython

[Stefan Behnel]

Antoine Pitrou, 16.04.2011 19:27:

On Sat, 16 Apr 2011 16:47:49 + (UTC)
Vinay Sajip wrote:

Bob made a comment in passing that simplejson (Python) is about as fast as
stdlib json (C extension), on 2.x.


I think Bob tested with an outdated version of the stdlib json module
(2.6 or 2.7, perhaps). In my latest measurements, the 3.2 json C module
is as fast as the C simplejson module, the only difference being in
parsing of numbers, which is addressed in
http://bugs.python.org/issue11856


Ok, but then, what's the purpose of having the old Python implementation in 
the stdlib? The other Python implementations certainly won't be happy with 
something that is way slower (algorithmically!) than the current version of 
the non-stdlib implementation. The fact that the CPython json maintainers 
are happy with the performance of the C implementation does not mean that 
the performance of the pure Python implementation can be ignored now.


Note: I don't personally care about this question since Cython does not 
suffer from this issue anyway. This is just the general question about the 
relation of the C module and the Python module in the stdlib. Functional 
compatibility is not necessarily enough.


Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Raymond Hettinger]

 In the grand python-dev tradition of "silence means acceptance", I consider
 this PEP finalized and implicitly accepted.
>> 
>> I haven't seen any responses that said, yes this is a well thought-out 
>> proposal
>> that will actually benefit any of the various implementations.
> 
> In that case it may well be that the silence is because the other
> implementations think the PEP is OK.  They certainly voted in favor of
> the broad outline of it at the language summit.  

Sounds like it was implicitly accepted even before it was written or any of the 
details were discussed.  

The big picture of "let's do something to make life easier for other 
implementations" is a worthy goal.  What that something should be is still a 
bit ambiguous.


>> every branch in a given implementation now guarantee every implementation 
>> detail
>> or do we only promise the published API (historically, we've *always* done 
>> the
>> latter)?
> 
> As Brett said, people do come to depend on the details of the
> implementation.  But IMO the PEP should be clarified to say that the
> tests we are talking about should be tests *of the published API*.
> That is, blackbox tests, not whitebox tests.

+1 That's an excellent suggestion.  Without that change, it seems like the PEP 
is overreaching.


>> Is there going to be any guidance on the commonly encountered semantic
>> differences between C modules and their Python counterparts (thread-safety,
>> argument handling, tracebacks, all possible exceptions, monkey-patchable pure
>> python classes versus hard-wired C types etc)?
> 
> Presumably we will need to develop such guidance.

+1 That would be very helpful.  Right now, the PEP doesn't address any of the 
commonly encountered differences.


> I personally have no problem with the 100% coverage being made a
> recommendation in the PEP rather than a requirement.  It sounds like
> that might be acceptable to Antoine.  Actually, I would also be fine with
> saying "comprehensive" instead, with a note that 100% branch coverage is
> a good way to head toward that goal, since a comprehensive test suite
> should contain more tests than the minimum set needed to get to 100%
> branch coverage.

+1 better test coverage is always a good thing (IMO).


Raymond
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Matt Billenstein]

On Sun, Apr 17, 2011 at 08:22:20AM +0200, Stefan Behnel wrote:
> Matt Billenstein, 17.04.2011 00:47:
> >On Sat, Apr 16, 2011 at 01:30:13PM +0200, Antoine Pitrou wrote:
> >>On Sat, 16 Apr 2011 00:41:03 +
> >>Matt Billenstein wrote:
> >>>
> >>>Slightly less crude benchmark showing simplejson is quite a bit faster:
> >>>
> >>>http://pastebin.com/g1WqUPwm
> >>>
> >>>250ms vs 5.5s encoding and decoding an 11KB json object 1000 times...
> >>
> >>This doesn't have much value if you don't say which version of Python
> >>you ran json with. You should use 3.2, otherwise you might miss some
> >>optimizations.
> >
> >Yes, that was 2.6.5 -- 3.2 native json is comparable to simplejson here 
> >taking
> >about 330ms...
> 
> From the POV of CPython 3.2, is "native" Python or C?

"Native" as in the version that ships with 3.2.

And actually I think my test with 2.6.5 wasn't using the C extension for some
reason so that 5.5s number isn't right -- a fresh build of 2.7.1 gives me a
runtime of around 350ms.

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibility Requirements

[Stefan Krah]

Brett Cannon  wrote:
> Since they do not typically support the entire `C API of Python`_ they
> are unable to use the code used to create the module. Often times this
> leads these other VMs to either re-implement the modules in pure
> Python or in the programming language used to implement the VM
> (e.g., in C# for IronPython). This duplication of effort between
> CPython, PyPy, Jython, and IronPython is extremely unfortunate as
> implementing a module *at least* in pure Python would help mitigate
> this duplicate effort.
> 
> The purpose of this PEP is to minimize this duplicate effort by
> mandating that all new modules added to Python's standard library
> *must* have a pure Python implementation _unless_ special dispensation
> is given. This makes sure that a module in the stdlib is available to
> all VMs and not just to CPython (pre-existing modules that do not meet
> this requirement are exempt, although there is nothing preventing
> someone from adding in a pure Python implementation retroactively).

I'm not sure that I understand the duplication of effort: If there
is a C module without a Python implementation in the stdlib, then
the PyPy, Jython, and IronPython developers are free to cooperate
and implement a single Python version. I would not consider this
a duplication of effort.

If, on the other hand, they choose to provide three individual
implementations in C#, Java and (?), then that is their own choice
and surely not the fault of the C module developer.


By contrast, this PEP puts a great burden on the developers of
new C modules. If this PEP is accepted, it is the C module developers
who will have to do duplicate work.

In my view, the PEP should have a clause that *active* participation
of PyPy, Jython, and IronPython developers is expected if they want
pure compatible Python versions to exist.



> Re-implementing parts (or all) of a module in C (in the case
> of CPython) is still allowed for performance reasons, but any such
> accelerated code must pass the same test suite (sans VM- or C-specific
> tests) to verify semantics and prevent divergence. To accomplish this,
> the test suite for the module must have 100% branch coverage of the
> pure Python implementation before the acceleration code may be added.

Raymond has pointed out that the PEP seems to discourage C modules. This
is one of the examples. Since implementing C modules takes a lot of time,
I'd appreciate to know if they are just tolerated or actually welcome.



> As an example, to write tests which exercise both the pure Python and
> C accelerated versions of a module, a basic idiom can be followed::
[cut]
> 
>     heap = Spam()
>     self.assertFalse(isinstance(heap,
>     collections.abc.MutableSequence))
>     with self.assertRaises(TypeError):
>     self.heapq.heappop(heap)

If all possible exceptions must match, then in the case of decimal the
PEP should give permission to change the published API of an existing
Python module (in this case decimal.py). Otherwise, I see no way of
accomplishing this goal.


It is possible to give many frivolous examples:

>>> from decimal import *
>>> 
>>> class C():
... def __init__(self):
... self.traps = 'invalid'
... 
>>> # No exception
... setcontext(C())
>>> 


>>> from cdecimal import *
>>> class C():
... def __init__(self):
... self.traps = 'invalid'
... 
>>> setcontext(C())
Traceback (most recent call last):
  File "", line 1, in 
TypeError: argument must be a context.
>>> 


In the case of duck typing, the only solution I see is to lock down the
types in decimal.py, thus changing the API. This is one of the things that
should be decided *before* the PEP is accepted.



Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Vinay Sajip]

Antoine Pitrou  pitrou.net> writes:

> Feel free to share your numbers.

I've now got my fork working on Python 3.2 with speedups. According to a
non-scientific simple test:

Python 2.7
==
Python version: 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24) 
[GCC 4.5.2]
11.21484375 KiB read
Timing simplejson:
0.271898984909
Timing stdlib json:
0.338716030121

Python 3.2
==
Python version: 3.2 (r32:88445, Mar 25 2011, 19:28:28) 
[GCC 4.5.2]
11.21484375 KiB read
Timing simplejson:
0.3150200843811035
Timing stdlib json:
0.32146596908569336

Based on this test script:

https://gist.github.com/923927

and the simplejson version here:

https://github.com/vsajip/simplejson/

Regards,

Vinay Sajip


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Vinay Sajip]

Stefan Behnel  behnel.de> writes:

> Well, if that is not possible, then the CPython devs will have a hard time 
> maintaining the json accelerator module in the long run. I quickly skipped 
> through the github version in simplejson, and it truly is some complicated 
> piece of code. Not in the sense that the code is ununderstandable, it's 
> actually fairly straight forward string processing code, but it's so 
> extremely optimised and tailored and has so much code duplicated for the 
> bytes and unicode types (apparently following the copy+paste+adapt pattern) 
> that it will be pretty hard to adapt to future changes of CPython, 
> especially the upcoming PEP 393 implementation. Maintaining this is clearly 
> no fun.

Do we even need this complexity in Python 3.x? The speedup code for 2.x is
taking different, parallel paths for str and unicode types, either of which
might be legitimately passed into JSON APIs in 2.x code. However, in Python 3.x,
ISTM we should not be passing in bytes to JSON APIs. So there'd be no equivalent
parallel paths for bytes for 3.x speedup code to worry about.

Anyway, some simple numbers posted by me elsewhere on this thread show
simplejson to be only around 2% faster. Talk of a 5x speedup appears to be
comparing non-speeded up vs. speeded up code, in which case the comparison isn't
valid.

Of course, people might find other workloads which show bigger disparity in
performance, or might find something in my 3.x port of simplejson which
invalidates my finding of a 2% difference.

Regards,

Vinay Sajip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibility Requirements

[Stefan Krah]

Brett Cannon  wrote:
> Now if people would actually support simply not accepting any more C modules
> into the Python stdlib (this does not apply to CPython's stdlib), then I'm all
> for that. I only went with the "accelerator modules are okay" route to help 
> get
> acceptance for the PEP. But if people are willing to go down a more stringent
> route and say that any module which uses new C code is considered
> CPython-specific and thus any acceptance of such modules will be damn hard to
> accomplish as it will marginalize the value of the code, that's fine by me.


Could you explain why C code marginalizes the value of the code? Most
people use CPython and they definitely want fast C modules. Also,
many people actually use CPython specifically for its C-API.

It has been suggested recently that wrapping the ICU library would be
desirable for Python. Should all such projects be discouraged because
it does not benefit PyPy, Jython and IronPython?


I find these projects very interesting and wish them well, but IMO the
reality is that CPython will continue to be the dominant player for
at least another 10 years.


Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibility Requirements

[Stefan Krah]

R. David Murray  wrote:
> > The PEP seems to be predicated on a notion that anything written in C is 
> > bad and
> > that all testing is good.  AFAICT, it doesn't provide any practical advice 
> > to
> > someone pursuing a non-trivial project (such as decimal or threading).  The 
> > PEP
> 
> Decimal already has a Python implementation with a very comprehensive
> test suite (no, I don't know if it has 100% coverage).  My understanding
> is that Stefan's code passes the Python test suite.  So I'm not sure
> what the issue is, there.  Stefan?

test_decimal.py does not have 100% coverage yet. cdecimal passes the tests,
but several decimal.py functions would have to perform type checking to
get identical exception behavior.

The current version of the joint unit tests is here:

http://hg.python.org/features/cdecimal/file/b00f8fa70126/Lib/test/decimal_tests.py


cdecimal specific behavior is guarded by HAVE_CDECIMAL, so it is
possible to grep for the differences.



As an aside, test_decimal.py constitutes at most 1% of the total tests.
The important tests (mathematical correctness and conformance to the
specification) are in two separate test suites, one of which runs
tests against decimal.py and the other against decNumber. These tests
can easily take a week to run, so they can't be part of the regression
tests.


Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

On Sun, 17 Apr 2011 09:21:32 +0200
Stefan Behnel  wrote:
> Antoine Pitrou, 16.04.2011 19:27:
> > On Sat, 16 Apr 2011 16:47:49 + (UTC)
> > Vinay Sajip wrote:
> >> Bob made a comment in passing that simplejson (Python) is about as fast as
> >> stdlib json (C extension), on 2.x.
> >
> > I think Bob tested with an outdated version of the stdlib json module
> > (2.6 or 2.7, perhaps). In my latest measurements, the 3.2 json C module
> > is as fast as the C simplejson module, the only difference being in
> > parsing of numbers, which is addressed in
> > http://bugs.python.org/issue11856
> 
> Ok, but then, what's the purpose of having the old Python implementation in 
> the stdlib? The other Python implementations certainly won't be happy with 
> something that is way slower (algorithmically!) than the current version of 
> the non-stdlib implementation.

Again, I don't think it's "way slower" since the code should be almost
identical (simplejson hasn't changed much in the last year). That's
assuming you measure performance on 3.2 or 3.3, not something older.

Besides, the primary selling point of the stdlib implementation is
that... it's the stdlib implementation. You have a json
serializer/deserializer by default without having to install any
third-party package. For most people that's probably sufficient; people
with specific needs *may* benefit from installing simplejson.

Also, the pure Python paths are still used if you customize some
parameters (I don't remember which ones exactly, you could take a look
at e.g. Lib/json/encoder.py if you are interested).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Antoine Pitrou]

On Sat, 16 Apr 2011 21:32:48 -0500
Brian Curtin  wrote:
> > Three weeks after this security vulnerability was *publicly* reported on
> > bugs.python.org, and two days after it was semi-officially announced,
> > I'm still waiting for security updates for my Ubuntu and Debian systems!
> >
> > I reckon if this had been handled differently (i.e., making new releases
> > and communicating it via the relevant channels [1]), we wouldn't have
> > the situation we have right now.
> 
> 
> I don't really think there's a "situation" here, and I fail to see how the
> development blog isn't one of the relevant channels.

If we want to make official announcements (like releases or security
warnings), I don't think the blog is appropriate. A separate
announcement channel (mailing-list or newsgroup) would be better, where
people can subscribe knowing they will only get a couple of e-mails a
year.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Fred Drake]

On Sun, Apr 17, 2011 at 7:48 AM, Antoine Pitrou  wrote:
> A separate announcement channel (mailing-list or newsgroup) would be better,
> where people can subscribe knowing they will only get a couple of e-mails a
> year.

Sounds like python-announce to me, with a matching entry on the front
of www.python.org.


  -Fred

-- 
Fred L. Drake, Jr.    
"Give me the luxuries of life and I will willingly do without the necessities."
   --Frank Lloyd Wright
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Antoine Pitrou]

On Sun, 17 Apr 2011 01:32:15 -0400
"R. David Murray"  wrote:
> 
> I personally have no problem with the 100% coverage being made a
> recommendation in the PEP rather than a requirement.  It sounds like
> that might be acceptable to Antoine.  Actually, I would also be fine with
> saying "comprehensive" instead, with a note that 100% branch coverage is
> a good way to head toward that goal, since a comprehensive test suite
> should contain more tests than the minimum set needed to get to 100%
> branch coverage.

If that's a recommendation then it's ok, although I would still prefer
we don't advocate such metrics. It's too easy for some people to get
obsessed about numeric measurements of "quality", leading them to
dubious workarounds and tricks (e.g. when using style-checking tools à
la pylint).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Jesse Noller]

On Sun, Apr 17, 2011 at 7:48 AM, Antoine Pitrou  wrote:
> On Sat, 16 Apr 2011 21:32:48 -0500
> Brian Curtin  wrote:
>> > Three weeks after this security vulnerability was *publicly* reported on
>> > bugs.python.org, and two days after it was semi-officially announced,
>> > I'm still waiting for security updates for my Ubuntu and Debian systems!
>> >
>> > I reckon if this had been handled differently (i.e., making new releases
>> > and communicating it via the relevant channels [1]), we wouldn't have
>> > the situation we have right now.
>>
>>
>> I don't really think there's a "situation" here, and I fail to see how the
>> development blog isn't one of the relevant channels.
>
> If we want to make official announcements (like releases or security
> warnings), I don't think the blog is appropriate. A separate
> announcement channel (mailing-list or newsgroup) would be better, where
> people can subscribe knowing they will only get a couple of e-mails a
> year.
>
> Regards
>
> Antoine.

And whose responsibility is it to email yet another mythical list? The
person posting the fix? The person who found and filed the CVE? The
release manager?

Brian *helped* us by raising awareness of the issue: At least now
there's a chance that one or more of the OS vendors *saw* that this
was an issue that was fixed.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Antoine Pitrou]

Le dimanche 17 avril 2011 à 09:30 -0400, Jesse Noller a écrit :
> >
> > If we want to make official announcements (like releases or security
> > warnings), I don't think the blog is appropriate. A separate
> > announcement channel (mailing-list or newsgroup) would be better, where
> > people can subscribe knowing they will only get a couple of e-mails a
> > year.
> >
> > Regards
> >
> > Antoine.
> 
> And whose responsibility is it to email yet another mythical list? The
> person posting the fix? The person who found and filed the CVE? The
> release manager?

Well, whose responsibility is it to make blog posts about security
issues? If you can answer this question then the other question
shouldn't be any more difficult to answer ;)

I don't think the people who may be interested in security announcements
want to monitor a generic development blog, since Python is far from the
only piece of software they rely on. /I/ certainly wouldn't want to.

Also, I think Gustavo's whole point is that if we don't have a
well-defined, deterministic procedure for security announcements and
releases, then it's just as though we didn't care about security at all.
Saying "look, we mentioned this one on our development blog" isn't
really reassuring for the target group of people.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Stefan Behnel]

Vinay Sajip, 17.04.2011 12:33:

Antoine Pitrou writes:

Feel free to share your numbers.


I've now got my fork working on Python 3.2 with speedups. According to a
non-scientific simple test:

Python 2.7
==
Python version: 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2]
11.21484375 KiB read
Timing simplejson:
0.271898984909
Timing stdlib json:
0.338716030121

Python 3.2
==
Python version: 3.2 (r32:88445, Mar 25 2011, 19:28:28)
[GCC 4.5.2]
11.21484375 KiB read
Timing simplejson:
0.3150200843811035
Timing stdlib json:
0.32146596908569336

Based on this test script:

https://gist.github.com/923927

and the simplejson version here:

https://github.com/vsajip/simplejson/


Is this using the C accelerated version in both cases? What about the pure 
Python versions? Could you provide numbers for both?


Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Antoine Pitrou]

On Sun, 17 Apr 2011 08:30:33 -0400
Fred Drake  wrote:
> On Sun, Apr 17, 2011 at 7:48 AM, Antoine Pitrou  wrote:
> > A separate announcement channel (mailing-list or newsgroup) would be better,
> > where people can subscribe knowing they will only get a couple of e-mails a
> > year.
> 
> Sounds like python-announce to me, with a matching entry on the front
> of www.python.org.

Looking at python-announce, it can receive an arbitrary amount of
announcements from third-party projects, or even call for papers for
random conferences. It's probably easy to (dis)miss an important message
in all the churn.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Paul Moore]

On 17 April 2011 06:32, R. David Murray  wrote:
> I don't think the PEP is asking this either (or if it is I agree it
> shouldn't be).  The way to get full branch coverage (and yes Exarkun is
> right, this is about individual branches; see coverage.py --branch)

One thing I'm definitely uncomfortable about is expressing the
requirement in a way that depends on a non-stdlib module
(coverage.py). Should coverage.py be added to the stdlib if we're
going to take test coverage as a measure? Hmm, maybe it goes without
saying, but does coverage.py work on Jython, IronPython, etc? (A quick
google search actually indicates that there might be some issues still
to be resolved...)

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Jesse Noller]

On Sun, Apr 17, 2011 at 9:42 AM, Antoine Pitrou  wrote:
> Le dimanche 17 avril 2011 à 09:30 -0400, Jesse Noller a écrit :
>> >
>> > If we want to make official announcements (like releases or security
>> > warnings), I don't think the blog is appropriate. A separate
>> > announcement channel (mailing-list or newsgroup) would be better, where
>> > people can subscribe knowing they will only get a couple of e-mails a
>> > year.
>> >
>> > Regards
>> >
>> > Antoine.
>>
>> And whose responsibility is it to email yet another mythical list? The
>> person posting the fix? The person who found and filed the CVE? The
>> release manager?
>
> Well, whose responsibility is it to make blog posts about security
> issues? If you can answer this question then the other question
> shouldn't be any more difficult to answer ;)
>
> I don't think the people who may be interested in security announcements
> want to monitor a generic development blog, since Python is far from the
> only piece of software they rely on. /I/ certainly wouldn't want to.
>
> Also, I think Gustavo's whole point is that if we don't have a
> well-defined, deterministic procedure for security announcements and
> releases, then it's just as though we didn't care about security at all.
> Saying "look, we mentioned this one on our development blog" isn't
> really reassuring for the target group of people.
>
> Regards
>
> Antoine.

I'm not arguing against us having a well defined, deterministic
procedure! We need one, for sure - I'm just defending Brian's actions
as perfectly rational and reasonable. Without his post, that CVE would
have been published, publicly available on other sites (CVE tracking
sites, and hence on the radar for people looking to exploit it), and
no one would be the wiser.

At least it got *some* attention this way. Is it the right thing to do
moving forward? Probably not - but do we have the people/person
willing to head up defining the policy and procedure, and do we have
the needed contacts in the OS vendors/3rd party distributors to notify
them rapidly in the case of fixing something like this?

A lag of several weeks from fixing a security issue to a source level
release from us that OS vendors can run with is too slow honestly.

jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Jacob Kaplan-Moss]

On Sat, Apr 16, 2011 at 9:23 AM, Nick Coghlan  wrote:
> On Sat, Apr 16, 2011 at 9:45 PM, Gustavo Narea  wrote:
>> May I suggest that you adopt a policy for handling security issues like
>> Django's?
>> http://docs.djangoproject.com/en/1.3/internals/contributing/#reporting-security-issues
>
> When the list of people potentially using the software is "anyone
> running Linux or Mac OS X and an awful lot of people running Windows
> or an embedded device", private pre-announcements simply aren't a
> practical reality. Neither is "stopping all other development" when
> most of the core development team aren't on the secur...@python.org
> list and don't even know a security issue exists until it is announced
> publicly. Take those two impractical steps out of the process, and
> what you have *is* the python.org procedure for dealing with security
> issues.

Just to fill in a bit of missing detail about our process since the
doc doesn't perfectly describe what happens:

* Our pre-announce list is *really* short. It consists of release
managers for various distributions that distribute packaged versions
of Django -- Ubuntu, RedHat, and the like. Yes it's a bit of
bookkeeping, but we feel it's really important to our users: not
everyone installs the Django package *we* put out, so we think it's
important to coordinate security releases with downstream distributors
so that users get a fixed version of Django regardless of how they're
installing Django in the first place.

* We don't really halt all development. I don't know why that's in
there, except maybe that it pre-dates there being more than a
couple-three committers. The point is just that we treat the security
issue as our most important issue at the moment and fix it as quickly
as possible.

I don't really have a point here as it pertains to python-dev, but I
thought it's important to clarify what Django *actually* does if it's
being discussed as a model.

Jacob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[R. David Murray]

On Sun, 17 Apr 2011 09:30:17 -0400, Jesse Noller  wrote:
> On Sun, Apr 17, 2011 at 7:48 AM, Antoine Pitrou  wrote:
> > On Sat, 16 Apr 2011 21:32:48 -0500 Brian Curtin  
> > wrote:
> >> > Three weeks after this security vulnerability was *publicly* reported on
> >> > bugs.python.org, and two days after it was semi-officially announced,
> >> > I'm still waiting for security updates for my Ubuntu and Debian systems!
> >> >
> >> > I reckon if this had been handled differently (i.e., making new releases
> >> > and communicating it via the relevant channels [1]), we wouldn't have
> >> > the situation we have right now.
> >>
> >> I don't really think there's a "situation" here, and I fail to see how the
> >> development blog isn't one of the relevant channels.
> >
> > If we want to make official announcements (like releases or security
> > warnings), I don't think the blog is appropriate. A separate
> > announcement channel (mailing-list or newsgroup) would be better, where
> > people can subscribe knowing they will only get a couple of e-mails a
> > year.
> 
> And whose responsibility is it to email yet another mythical list? The
> person posting the fix? The person who found and filed the CVE? The
> release manager?
> 
> Brian *helped* us by raising awareness of the issue: At least now
> there's a chance that one or more of the OS vendors *saw* that this
> was an issue that was fixed.

That fact that Brian helped publicize it is not really relevant to
Antoine's point.  The *obvious* answer to your question about whose
responsibility it is is: *the security team*.  Brian's blog post would
then have been much more like he envisioned it when he wrote it, a peek
inside the process, rather than appearing to be the primary announcement
as many seem to be perceiving it.

That's how distributions, at least, handle this.  There's a mailing list for
security related announcements on which only the "security officer" or
"security team" posts announcements, and security related announcements
*only*.  Then then the people responsible for security in any context
(a distribution, a security manager for a company, J Random User) can
subscribe to it and get *only* security announcements.  That allows them
to easily prioritize those announcements on receipt.

Python should have such a mailing list.

--
R. David Murray   http://www.bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Nick Coghlan]

On Mon, Apr 18, 2011 at 12:03 AM, Jacob Kaplan-Moss  wrote:
> Just to fill in a bit of missing detail about our process since the
> doc doesn't perfectly describe what happens:
>
> * Our pre-announce list is *really* short. It consists of release
> managers for various distributions that distribute packaged versions
> of Django -- Ubuntu, RedHat, and the like. Yes it's a bit of
> bookkeeping, but we feel it's really important to our users: not
> everyone installs the Django package *we* put out, so we think it's
> important to coordinate security releases with downstream distributors
> so that users get a fixed version of Django regardless of how they're
> installing Django in the first place.

I'd rather have Red Hat and Canonical reps *on* the
secur...@python.org list rather than a separate pre-announce list.

> * We don't really halt all development. I don't know why that's in
> there, except maybe that it pre-dates there being more than a
> couple-three committers. The point is just that we treat the security
> issue as our most important issue at the moment and fix it as quickly
> as possible.

That makes a lot more sense.

> I don't really have a point here as it pertains to python-dev, but I
> thought it's important to clarify what Django *actually* does if it's
> being discussed as a model.

I'd personally like to see a couple of adjustments to
http://www.python.org/news/security/:

1. Identify a specific point-of-contact for the security list, for
security-related questions that aren't actually security issues (e.g.
how would a core developer go about asking to join the PSRT?)
2. Specifically state on the security page where vulnerabilities and
fixes will be announced and the information those announcements will
contain (as a reference for the PSRT when responding to an issue, and
also to inform others of the expected procedure)

The current page does a decent job of describing how to report a
security issue, but doesn't describe anything beyond that.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python and super

[Michael Foord]

On 17 April 2011 02:48, Steven D'Aprano  wrote:

> Michael Foord wrote:
>
>> On 15/04/2011 02:23, Steven D'Aprano wrote:
>>
> [...]
>
>  If we treat django's failure to use super as a bug, you want the Python
>>> language to work-around that bug so that:
>>>
>>
>> What you say (that this particular circumstance could be treated as a bug
>> in django) is true, however consider the "recently" introduced problem
>> caused by object.__init__ not taking arguments. This makes it impossible to
>> use super correctly in various circumstances.
>>
> [...]
>
>  It is impossible to inherit from both C and A and have all parent __init__
>> methods called correctly. Changing the semantics of super as described would
>> fix this problem.
>>
>
> So you say. I don't have an an opinion on whether or not you are
> technically correct, but adding DWIM black-magic to super scares me.



Well, super is already pretty "magic" and what I'm suggesting is no more
magic than currently exists. I'm suggesting (but it won't happen - no-one
else is in favour :-) *extending* the existing algorithm in a predictable
and understandable way. The main advantage is that it allows methods to
express "don't call my parent class methods but don't halt the chain of
calling", which is currently not possible (so in that context I don't really
know what you mean by "DWIM black-magic"). I'm *not* suggesting full auto
calling.

All the best,

Michael




> It scares me even if it were guaranteed to *only* apply to __init__, but if
> it applied to arbitrary methods, it frankly terrifies me.
>
> If it were limited to only apply to __init__, there would be a constant
> stream of requests that we loosen the restriction and "make super just work"
> for all methods, despite the dangers of DWIM code.
>
>
>
>
>
> --
> Steven
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibility Requirements

[R. David Murray]

On Sun, 17 Apr 2011 12:14:51 +0200, Stefan Krah  wrote:
> I'm not sure that I understand the duplication of effort: If there
> is a C module without a Python implementation in the stdlib, then
> the PyPy, Jython, and IronPython developers are free to cooperate
> and implement a single Python version. I would not consider this
> a duplication of effort.

Yes, that's exactly what we are trying to encourage.  If the Python
standard library is seen as common property of all Python implementations,
then this is *much* more likely to happen.

> If, on the other hand, they choose to provide three individual
> implementations in C#, Java and (?), then that is their own choice
> and surely not the fault of the C module developer.

Right.

> By contrast, this PEP puts a great burden on the developers of
> new C modules. If this PEP is accepted, it is the C module developers
> who will have to do duplicate work.

This is true only because of the current "blessed" position of CPython
in the Python ecosystem.  If a separate Python stdlib is the common
property of all Python implementations, then the same double burden
would apply to, say, an IronPython developer writing a module in C#
and wanting it included in the stdlib.

> In my view, the PEP should have a clause that *active* participation
> of PyPy, Jython, and IronPython developers is expected if they want
> pure compatible Python versions to exist.

> > Re-implementing parts (or all) of a module in C (in the case
> > of CPython) is still allowed for performance reasons, but any such
> > accelerated code must pass the same test suite (sans VM- or C-specific
> > tests) to verify semantics and prevent divergence. To accomplish this,
> > the test suite for the module must have 100% branch coverage of the
> > pure Python implementation before the acceleration code may be added.
> 
> Raymond has pointed out that the PEP seems to discourage C modules. This
> is one of the examples. Since implementing C modules takes a lot of time,
> I'd appreciate to know if they are just tolerated or actually welcome.

I believe they are welcome, but that they are a CPython implementation
detail, and the PEP is trying to make that distinction clear.

One can also imagine a C module getting accepted to the stdblib
because everybody agrees that (a) it can't be implemented in Python and
(b) every Python implementation should support it.  In that case only
the test suite will be part of the implementation-independent part of
the stdlib.  I do think that such modules (and we already have several)
should have a higher bar to cross to get in to the stdlib than modules
that have a pure Python implementation.

> If all possible exceptions must match, then in the case of decimal the
> PEP should give permission to change the published API of an existing
> Python module (in this case decimal.py). Otherwise, I see no way of
> accomplishing this goal.

This may well be what needs to be done, both for CPython and for other
implementations.  When we agree that some test covers something that is
an implementation detail, the tests should be so marked.  Making changes
to the API and tests to accommodate specific Python implementations
(including CPython) will be the right thing to do in some cases.
Obviously these will have to be considered on a case by case basis.

The Python sdtlib and its tests is already the standard that other
implementations need to conform to.  The PEP is trying to lay out some
rules so that CPython has to conform on equal footing with the
other implementations.

> It is possible to give many frivolous examples:
> 
> >>> from decimal import *
> >>>
> 
> >>> class C():
> ... def __init__(self):
> ... self.traps = 'invalid'
> ...
> 
> >>> # No exception
> ... setcontext(C())
> >>> 
> 
> 
> 
> >>> from cdecimal import *
> >>> class C():
> ... def __init__(self):
> ... self.traps = 'invalid'
> ... 
> 
> >>> setcontext(C())
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: argument must be a context.
> >>>
>
> In the case of duck typing, the only solution I see is to lock down the
> types in decimal.py, thus changing the API. This is one of the things that
> should be decided *before* the PEP is accepted.

Here you perceive the burden we are currently placing on the other
implementations.  That's the world they live in *now*.  The PEP is asking
CPython to share this pain equally.

I agree that this is a concrete example that the PEP could address.
I myself don't know enough about decimal/cdecimal or the Python C API
to know why cdecimal can't duck type here, but it certainly sounds
like a good example to use to clarify the requirements being advocated
by the PEP.  I won't be surprised to find that the issues involved are
the same issues that an accelerator module for the other Python
implementations would face.

--
R. David Murray   http://www.bitdance.com
___

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[R. David Murray]

On Sun, 17 Apr 2011 00:30:22 -0700, Raymond Hettinger 
 wrote:
>  In the grand python-dev tradition of "silence means acceptance", I 
>  consider
>  this PEP finalized and implicitly accepted.
> >>
> >> I haven't seen any responses that said, yes this is a well thought-out 
> >> proposal
> >> that will actually benefit any of the various implementations.
> >
> > In that case it may well be that the silence is because the other
> > implementations think the PEP is OK.  They certainly voted in favor of
> > the broad outline of it at the language summit.
> 
> Sounds like it was implicitly accepted even before it was written or any of 
> the
> details were discussed.

No, just the principle that something along these lines would be good.
Any final decision of course requires the actual PEP to look at, which
was also acknowledged at the summit.  My point was that lack of comment
from the other implementations *might* indicate they liked how the PEP
turned out.  But it might also mean they aren't paying attention, which
would be bad...

> The big picture of "let's do something to make life easier for other
> implementations" is a worthy goal.  What that something should be is still a 
> bit
> ambiguous.

As I said in another email, I think the something that should be
done is to put CPython on equal footing implementation-pain-wise and
lets-make-this-work-wise with the other implementations.  The end result
will be better test coverage and clearer APIs in the stdlib.

--
R. David Murray   http://www.bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Michael Foord]

On 17/04/2011 00:16, Antoine Pitrou wrote:

On Sat, 16 Apr 2011 23:48:45 +0100
Michael Foord  wrote:


On 16/04/2011 22:28, "Martin v. Löwis" wrote:

Am 16.04.2011 21:13, schrieb Vinay Sajip:

Martin v. Löwis   v.loewis.de>   writes:


Does it actually need improvement?

I can't actually say, but I assume it keeps changing for the better - albeit
slowly. I wasn't thinking of specific improvements, just the idea of continuous
improvement in general...

Hmm. I cannot believe in the notion of "continuous improvement"; I'd
guess that it is rather "continuous change".

I can see three possible areas of improvment:
1. Bugs: if there are any, they should clearly be fixed. However, JSON
 is a simple format, so the implementation should be able to converge
 to something fairly correct quickly.
2. Performance: there is always room for performance improvements.
 However, I strongly recommend to not bother unless a severe
 bottleneck can be demonstrated.

Well, there was a 5x speedup demonstrated comparing simplejson to the
standard library json module.

No.


Yes.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Michael Foord]

On 17/04/2011 07:28, "Martin v. Löwis" wrote:

Well, there was a 5x speedup demonstrated comparing simplejson to the
standard library json module.

Can you kindly point to that demonstration?

Hmm... according to a later email in this thread it is 350ms vs 250ms 
for an 11kb sample. That's a nice speedup but not a 5x one. Bob Ippolito 
did claim that simplejson was faster than json for real world workloads 
and I see no reason not to believe him. :-)



That sound like *very* worth pursuing (and
crazy not to pursue). I've had json serialisation be the bottleneck in
web applications generating several megabytes of json for some requests.

Hmm. I'd claim that the web application that needs to generate several
megabytes of json for something should be redesigned.


It was displaying (including sorting) large amounts of information in 
tables through a web ui. The customer wanted all the information 
available in the ables, so all the data needed to be sent. We did 
filtering on the server side where possible to minimize the data sent, 
but it was ~10mb for many of the queries. We also cached the data on the 
client and only updated as needed.


We could have "redesigned" the customer requirements I suppose...


  I also wonder
whether the bottleneck was the *generation*,
The bottleneck was generation. I benchmarked and optimised. (We were 
using simplejson but I trimmed down the data sent to the absolute 
minimum needed by the client app rather than merely serialising all the 
source data from the django model objects - I didn't optimise within 
simplejson itself...)



the transmission, or
the processing of the data on the receiving end.

Processing was done in IronPython in Silverlight using the .NET 
de-serialization APIs which were dramatically faster than the Python 
handling on the other side.


All the best,

Michael


Regards,
Martin



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Michael Foord]

On 17/04/2011 17:05, Michael Foord wrote:

On 17/04/2011 00:16, Antoine Pitrou wrote:

On Sat, 16 Apr 2011 23:48:45 +0100
Michael Foord  wrote:


On 16/04/2011 22:28, "Martin v. Löwis" wrote:

Am 16.04.2011 21:13, schrieb Vinay Sajip:

Martin v. Löwis   v.loewis.de>   writes:


Does it actually need improvement?
I can't actually say, but I assume it keeps changing for the 
better - albeit
slowly. I wasn't thinking of specific improvements, just the idea 
of continuous

improvement in general...

Hmm. I cannot believe in the notion of "continuous improvement"; I'd
guess that it is rather "continuous change".

I can see three possible areas of improvment:
1. Bugs: if there are any, they should clearly be fixed. However, JSON
 is a simple format, so the implementation should be able to 
converge

 to something fairly correct quickly.
2. Performance: there is always room for performance improvements.
 However, I strongly recommend to not bother unless a severe
 bottleneck can be demonstrated.

Well, there was a 5x speedup demonstrated comparing simplejson to the
standard library json module.

No.


Yes.


Well, maybe not. :-)




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk






--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibility Requirements

[Stefan Krah]

R. David Murray  wrote:
[snip a lot]

Thank you, this cleared up many things.


> > In the case of duck typing, the only solution I see is to lock down the
> > types in decimal.py, thus changing the API. This is one of the things that
> > should be decided *before* the PEP is accepted.
> 
> Here you perceive the burden we are currently placing on the other
> implementations.  That's the world they live in *now*.  The PEP is asking
> CPython to share this pain equally.
> 
> I agree that this is a concrete example that the PEP could address.
> I myself don't know enough about decimal/cdecimal or the Python C API
> to know why cdecimal can't duck type here, but it certainly sounds
> like a good example to use to clarify the requirements being advocated
> by the PEP.  I won't be surprised to find that the issues involved are
> the same issues that an accelerator module for the other Python
> implementations would face.

The technical reason is that the context is a speed critical data structure,
so I'm doing some tricks to emulate the context flags and traps dictionaries.


But I actually prefer that the context is locked down. The context
settings are absolutely crucial for the correctness of the result.
Here is a mistake that I've made multiple times while trying something
out with decimal.py:

>>> from decimal import *
>>> c = getcontext()
# Meaning c.Emax and c.Emin:
>>> c.emax = 99
>>> c.emin = -99
# The operation silently uses the unchanged context:
>>> Decimal(2)**9
Decimal('4.995010465071922539720163822E+30102')
>>> 


cdecimal raises an AttributeError:

>>> from cdecimal import *
>>> c = getcontext()
>>> c.emax = 99
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'cdecimal.Context' object has no attribute 'emax'
>>> 


So, if one of the goals of the PEP is to clean up various APIs, I'm all
for it. My concern is though that the process will be very slow due to
lack of time and general reluctance to change APIs. And this is where
I see a potentially negative effect:

Is it worth to stall development over relatively minor issues? Will
these differences actually affect someone in practice? Will the
four Python implementations block each other?



Stefan Krah

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibility Requirements

[R. David Murray]

On Sun, 17 Apr 2011 19:17:11 +0200, Stefan Krah  wrote:
> R. David Murray  wrote:
> [snip a lot]
> 
> Thank you, this cleared up many things.

Heh.  Keep in mind that this is my viewpoint.  I *think* Brett agrees
with me.  I'm sure he'll speak up if he doesn't.

> The technical reason is that the context is a speed critical data structure,
> so I'm doing some tricks to emulate the context flags and traps dictionaries.

[snip]

Thanks, your explanation seems to me to make a good case for making the
decimal.py implementation less permissive.

> So, if one of the goals of the PEP is to clean up various APIs, I'm all
> for it. My concern is though that the process will be very slow due to
> lack of time and general reluctance to change APIs. And this is where
> I see a potentially negative effect:

Well, the general reluctance to change APIs is certainly an issue.
But since you are advocating cdecimal changing the API *anyway*, if it
is going to go in to CPython this would have to be addressed regardless.
So I don't see that the PEP affects the speed of that part of the process
from CPython's point of view.

> Is it worth to stall development over relatively minor issues? Will
> these differences actually affect someone in practice? Will the
> four Python implementations block each other?

In my vision it wouldn't stall development in any place it shouldn't
be stalled by our normal backward compatibility rules.  It would be a
bug in the bug tracker saying "the API of module X has some undesirable
characteristics that get in the way of implementing accelerators, can
we change it?"  Again, I don't see this as changing what the current
procedure should be anyway, just clarifying it and making it more likely
that we will *notice* the changes and deal with them proactively rather
than finding out about them after the accelerator is in the field, having
introduced a backward-incompatible change unintentionally.  (Note: I'm
sure that we will still accidentally do this anyway, I'm just hoping to
reduce the frequency of such occurrences).

--
R. David Murray   http://www.bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Releases for recent security vulnerability

[Stephen J. Turnbull]

Gustavo Narea writes:

 > Well, that's a long shot. I doubt the people/organizations affected are
 > all aware.

That's really not Python's responsibility.  That's theirs.  Caveats:
Python should have a single place where security patches are announced
*first*, before developer blogs and the like.  Python's documentation
should make it clear at the most important entry points that the
appropriate place to report possible security issues is
secur...@python.org, not the tracker.  In particular, the tracker's
top page (the one you get from http://bugs.python.org/) should make
that clear.

Ironically, Brian's blog entry outlines a plausible security policy,
but a quick Google didn't find it elsewhere on site:python.org.  Oops,
a different search did find it -- under News/Security Advisories.

The tracker suggestion submitted as
http://psf.upfronthosting.co.za/roundup/meta/issue393.

 > And I doubt they are all capable of patching their system or
 > getting a patched Python from a trusted party.

Then they shouldn't be running Python in a security context, should
they?  Seriously, if they want somebody else to take care of their
security issues for them, they should pay for it.  As in almost areas
of life, security is at best worth what you pay for it.

 > Three weeks after this security vulnerability was *publicly* reported on
 > bugs.python.org,

Again, that's an issue with the reporter not knowing the policy, not
the policy itself, which is to report to secur...@python.org,
cf. Brian's post and the Security Advisory page.  The caveats above
apply, though.

 > and two days after it was semi-officially announced,
 > I'm still waiting for security updates for my Ubuntu and Debian systems!

Yeah, well, so much for depending on Ubuntu and Debian.  There are
reasons why people pay for RHEL.

 > I reckon if this had been handled differently (i.e., making new releases
 > and communicating it via the relevant channels [1]), we wouldn't have
 > the situation we have right now.

Of course not.  So what?  The question is "what is the best way to
reduce risks?"  *It is not possible to reduce all risks
simultaneously.*  What you are saying is "please keep things obscure
until I'm up to date."

It seems to be a consensus in the security community that most
security holes *are* sufficiently obscure that announcing the problem
before announcing a fix is likely to increase the likelihood of black
hats finding exploits and successfully executing them more than it
increases the likelihood that (3rd party) white hats will find and
contribute a fix.  So the question is whether to rely on obscurity
after the fix is devised.

Now, once there is a fix, there's always hysteresis in implementation,
as you note with respect to Ubuntu and Debian.  If you don't announce
the fix once available, you are increasing risk to conscientious,
patch-capable admins dramatically compared to the case where you give
them the patch.  I don't see why your Ubuntu/Debian systems should
take precedence over systems of those who prefer to rely on self-
built-and-maintained systems.  (In fact, since I generally fall into
the latter category, may I suggest it should be the other way around?
)

 > May I suggest that you adopt a policy for handling security issues like
 > Django's?
 > http://docs.djangoproject.com/en/1.3/internals/contributing/#reporting-security-issues

I'm -1 on that, except to the extent that it corresponds to existing
policy.  Eg,

This will probably mean a new release of Django, but in some cases
it may simply be patches against current releases.

seems to apply to the current case.  I really don't think the policy
Django claims is appropriate for Python for the following reasons,
among others:

Halt all other development as long as is needed to develop a fix,
including patches against the current and two previous releases.

is nonsense.  Perhaps what this means is that the "long-time, highly
trusted Django developers" on the security@django list *volunteer* to
put other Django work aside until the security hole is patched.  But
certainly the "hundreds of contributors around the world" won't stop
their work -- they won't know about the moratorium since they're not
on the security list.
In the case of Python, it's not even possible to stop commits without
closing the repo, as not all committers are on the security list.

Even for the security crew, in many cases of security problems, it's
something simple and readily fixed like a buffer overflow or an URL
traversal issue that just needs a simple filter on input before
executing a risky procedure.  So who does the fixing, reviewing, etc
should be decided on a case-by-case basis based on the problem itself
and available expertise IMO.

And this

Pre-notify everyone we know to be running the affected version(s)
of Django.

seems positively counter-productive if they're serious about
"everyone".  Surely the black hats are running the affected versions
of Django to test 

Re: [Python-Dev] Releases for recent security vulnerability

[Stephen J. Turnbull]

Nick Coghlan writes:

 > I'd personally like to see a couple of adjustments to
 > http://www.python.org/news/security/:

For another thing, it needs to be more discoverable.

For yet another thing, it has two ancient entries on it.  Surely there
are more than that?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python and super

[Nikolay Zakharov]

16.04.2011 03:38, Greg Ewing пишет:

Michael Foord wrote:


consider the "recently" introduced problem caused by object.__init__

> not taking arguments. This makes it impossible to use super correctly
> in various circumstances.
>
> ...
>
It is impossible to inherit from both C and A and have all parent 
__init__ methods called correctly. Changing the semantics of super as 
described would fix this problem.


I don't see how, because auto-super-calling would eventually
end up trying to call object.__init__ with arguments and fail.

You might think to "fix" this by making a special case of
object.__init__ and refraining from calling it. But the same
problem arises in a more general way whenever some class in
the mix has a method with the right name but the wrong
signature, which is likely to happen if you try to mix
classes that weren't designed to be mixed together.

Michael's words are not about *auto-calling* but about *stopping 
prevention* of parent's method call by a class that unrelated to such 
parent. In the example above A is such a stopper that prevents calling 
of B.__init__ and B is a stopper for calling A.__init__ but A and B are 
completely unrelated to each other.


object.__init__ would not be called anyway (in this example) but the 
point is that nobody (at least among Michael and myself) going to 
*auto-call* object.__init__ with some automagically picked arguments.


--
Nikolay Zakharov
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Antoine Pitrou]

On Sun, 17 Apr 2011 17:09:17 +0100
Michael Foord  wrote:
> On 17/04/2011 07:28, "Martin v. Löwis" wrote:
> >> Well, there was a 5x speedup demonstrated comparing simplejson to the
> >> standard library json module.
> > Can you kindly point to that demonstration?
> >
> Hmm... according to a later email in this thread it is 350ms vs 250ms 
> for an 11kb sample. That's a nice speedup but not a 5x one.

That speedup is actually because of a slowdown in py3k, which should be
solved with http://bugs.python.org/issue11856

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Martin v. Löwis]

> Of course, people might find other workloads which show bigger disparity in
> performance, or might find something in my 3.x port of simplejson which
> invalidates my finding of a 2% difference.

Thanks a lot for doing this research, by the way.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [ANN] Python 2.5.6 Release Candidate 1

[Martin v. Löwis]

On behalf of the Python development team and the Python community, I'm
happy to announce the release candidate 1 of Python 2.5.6.

This is a source-only release that only includes security fixes. The
last full bug-fix release of Python 2.5 was Python 2.5.4. Users are
encouraged to upgrade to the latest release of Python 2.7 (which is
2.7.1 at this point).

This releases fixes issues with the urllib, urllib2, SimpleHTTPServer,
and audiop modules. See the release notes at the website (also
available as Misc/NEWS in the source distribution) for details of bugs
fixed.

For more information on Python 2.5.6, including download links for
various platforms, release notes, and known issues, please see:

http://www.python.org/2.5.6

Highlights of the previous major Python releases are available from
the Python 2.5 page, at

http://www.python.org/2.5/highlights.html

Enjoy this release,
Martin

Martin v. Loewis
mar...@v.loewis.de
Python Release Manager
(on behalf of the entire python-dev team)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python and super

[Greg Ewing]

Mark Janssen wrote:

I have to say it is quite strange to
me that there is no distinction made between IS-A relationship and
HAS-A relationships with regard to the issue of Inheritence.


I'm not sure what you mean by that. Inheritance is (or
should be) used only for is-a relationships. Misusing it
for has-a relationships leads to problems.


Python, confusingly makes no syntactic distinction,


Yes, it does, as long as you use composition instead of
inheritance for has-a relationships.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Vinay Sajip]

Stefan Behnel  behnel.de> writes:

> Is this using the C accelerated version in both cases? What about the pure 
> Python versions? Could you provide numbers for both?

What I posted earlier were C-accelerated timings. I'm not sure exactly how to
turn off the speedups for stdlib json. With some assumptions, as listed in this
script:

https://gist.github.com/924626

I get timings like this:

Python version: 3.2 (r32:88445, Mar 25 2011, 19:28:28) 
[GCC 4.5.2]
11.21484375 KiB read
Timing simplejson (with speedups):
0.31562185287475586
Timing stdlib json (with speedups):
0.31923389434814453
Timing simplejson (without speedups):
4.586531162261963
Timing stdlib json (without speedups):
2.5293829441070557

It's quite likely that I've failed to turn off the stdlib json speedups (though
I attempted to turn them off for both encoding and decoding), which would
explain the big disparity in the non-speedup case. Perhaps someone with more
familiarity with stdlib json speedup internals could take a look to see what
I've missed? I perhaps can't see the forest for the trees.

Regards,

Vinay Sajip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibility Requirements

[Nick Coghlan]

On Mon, Apr 18, 2011 at 3:50 AM, R. David Murray  wrote:
> Thanks, your explanation seems to me to make a good case for making the
> decimal.py implementation less permissive.

Indeed. Since the current handling of Context in decimal.py violates
"Errors should never pass silently, unless explicitly silenced", I
would personally support a proposal to lock down its __setattr__ to a
predefined set of attributes, have its __delattr__ always raise an
exception, and introduce a parent ABC that is used for an isinstance()
check in setcontext(). (The ABC could include an attribute check, so
only objects that failed to provide all the appropriate methods and
attributes would raise the TypeError).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Nick Coghlan]

On Mon, Apr 18, 2011 at 10:19 AM, Vinay Sajip  wrote:
> It's quite likely that I've failed to turn off the stdlib json speedups 
> (though
> I attempted to turn them off for both encoding and decoding), which would
> explain the big disparity in the non-speedup case. Perhaps someone with more
> familiarity with stdlib json speedup internals could take a look to see what
> I've missed? I perhaps can't see the forest for the trees.

Consider trying:

import sys
sys.modules["_json"] = 0 # Block the C extension
import json

in a fresh interpreter.

(This is the same dance test.support.import_fresh_module() uses
internally to get unaccelerated modules for testing purposes)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [ANN] Python 2.5.6 Release Candidate 1

[Leo Jay]

Hi,

I think the release date of 2.5.6c1 should be 17-Apr-2011, instead of
17-Apr-2010
http://www.python.org/download/releases/2.5.6/NEWS.txt

On Mon, Apr 18, 2011 at 5:57 AM, "Martin v. Löwis"  wrote:
>
> On behalf of the Python development team and the Python community, I'm
> happy to announce the release candidate 1 of Python 2.5.6.
>
> This is a source-only release that only includes security fixes. The
> last full bug-fix release of Python 2.5 was Python 2.5.4. Users are
> encouraged to upgrade to the latest release of Python 2.7 (which is
> 2.7.1 at this point).
>
> This releases fixes issues with the urllib, urllib2, SimpleHTTPServer,
> and audiop modules. See the release notes at the website (also
> available as Misc/NEWS in the source distribution) for details of bugs
> fixed.
>
> For more information on Python 2.5.6, including download links for
> various platforms, release notes, and known issues, please see:
>
>    http://www.python.org/2.5.6
>
> Highlights of the previous major Python releases are available from
> the Python 2.5 page, at
>
>    http://www.python.org/2.5/highlights.html
>
> Enjoy this release,
> Martin
>
> Martin v. Loewis
> mar...@v.loewis.de
> Python Release Manager
> (on behalf of the entire python-dev team)


--
Best Regards,
Leo Jay
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [ANN] Python 2.5.6 Release Candidate 1

[Martin v. Löwis]

> I think the release date of 2.5.6c1 should be 17-Apr-2011, instead of
> 17-Apr-2010
> http://www.python.org/download/releases/2.5.6/NEWS.txt

Thanks, fixed.

Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com