Re: What's so funny? WAS Re: rotor replacement

[Nick Craig-Wood]

Paul Rubin  wrote:
>  Actually and surprisingly, that's not really true.  Crypto algorithms
>  are pretty straightforward, so if you examine the code and check that
>  it passes a bunch of test vectors, you can be pretty sure it's
>  correct.

I was going to write pretty much the same thing.

If a security flaw is found in a block cipher (say) it won't be
because it has a buffer overflow etc, it will be because the algorithm
is flawed.  You can't patch up crypto algorithms, you have to throw
them away and start again (you can't have two incompatible versions of
DES for instance).

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

Skip Montanaro <[EMAIL PROTECTED]> writes:
> While it might be convenient to not have to distribute some third
> party library in addition to Python, there is a fundamental problem
> implementing a crypto algorithm from scratch for inclusion into
> Python.  There is always the problem that the new code has to be
> more rigorously tested than typical code 

Actually and surprisingly, that's not really true.  Crypto algorithms
are pretty straightforward, so if you examine the code and check that
it passes a bunch of test vectors, you can be pretty sure it's
correct.  It's much harder to check something like a compiler, which
has a much bigger state space, far more decision points, etc.  The
usual bugs in crypto apps (and there are lots of such bugs) are in how
the primitives are used, not in the primitives themselves.  

> and new bugs means a new distribution of Python, not just a
> replacement library.

Why would that be true of a crypto module and not true of, say, the socket
module?  If the socket module has a bug that allows a remote takeover of
the application, that's as bad as a crypto bug.

> A bug in code that is not security-related generally means something
> doesn't work and only rarely means a security hole has been opened
> on the computer.  A bug in security-related code more often means
> the latter as well. 

People often don't understand that that almost all code is
security-related.  Any code that touches data that came from the
internet is security related.  If the math library arctangent function
has a buffer overflow bug triggered by a certain input number, and
someone uses math.arctan in an image viewing program, then maybe a
specially concocted image designed to set off the bug can take over
the user's computer.  So even something like math.arctan is security
related.

> While I imagine the changes were fairly small, the guys involved are
> all very smart, and the code is fairly straightforward (little, if
> any, memory allocation going on), there is still the possibility
> that a bug lurks in either the incorporated code or in the changes
> to it.  How quickly could the Python community respond if a bug was
> found and fixed in the public domain SHA code?  How much harder
> would it be for people to adapt if they had to reinstall Python
> instead of just an external library?

If they're able to install external libraries, what stops them from
reinstalling a patched sha module?

The hazards of using a crypto module are sort of like the hazards of
using the threading module.  Unless you know what you're doing and are
very careful, it's easy to make an error.  But the resulting bugs
happen because the module did exactly what you asked for, not because
it did something different from what you asked for.

If you ever write code that uses the Internet, I highly recommend the
book "Security Engineering", by Ross Anderson.  It will give you some
idea of what you are up against.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Skip Montanaro]

Nick> I think one of the special things about Python is its batteries
Nick> included approach, and a crypto library would seem to be an
Nick> obvious battery to install since it doesn't (or needn't) depend on
Nick> any other library or application.

Obvious for some I suppose (I've never used any).  While it might be
convenient to not have to distribute some third party library in addition to
Python, there is a fundamental problem implementing a crypto algorithm from
scratch for inclusion into Python.  There is always the problem that the new
code has to be more rigorously tested than typical code and new bugs means a
new distribution of Python, not just a replacement library.  A bug in code
that is not security-related generally means something doesn't work and only
rarely means a security hole has been opened on the computer.  A bug in
security-related code more often means the latter as well.  I'd much rather
trust a widely-disseminated piece of crypto code that is simply wrapped by
Python than one that was written expressly written for Python (and that will
likely not be exercised much outside the Python community).

I realize the sha module is incorporated this way.  It has this comment:

 * This code for the SHA algorithm was noted as public domain. The original
 * headers are pasted below.
 *
 * Several changes have been made to make it more compatible with the
 * Python environment and desired interface.

While I imagine the changes were fairly small, the guys involved are all
very smart, and the code is fairly straightforward (little, if any, memory
allocation going on), there is still the possibility that a bug lurks in
either the incorporated code or in the changes to it.  How quickly could the
Python community respond if a bug was found and fixed in the public domain
SHA code?  How much harder would it be for people to adapt if they had to
reinstall Python instead of just an external library?

Skip

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> > Oh, ok.  Earlier you said you wanted user feedback before you could
> > conclude that there was reason to want an AES module at all.
> 
> I believe I never said that. I said that I wanted user feedback to
> determine whether *this* AES module (where this is either your
> from-scratch implementation, or any other specific implementation
> contributed) is desirable.

If that's what you're saying now, I'll accept it and not bother
looking for your other posts that came across much differently.

> > I had thought there were insurmountable
> > obstacles of a nontechnical nature, mainly caused by legal issues,..
> 
> These obstacles are indeed real. But I believe they are not
> unsurmountable. For example, there is the valid complaint that,
> in order to export the code from SF, we need to follow U.S.
> export laws. 10 years ago, these would have been unsurmountable.

Well, yes, 10 years ago, SF didn't exist .  But there was an ftp
site run by Michael Johnson that had some special server side checks
that made sure the client was in the USA.  That was considered good
enough to comply with the export regs, and another guy and I
distributed crypto code (a program that let you use your PC as an
encrypted voice phone) through that site in 1995.

Of course, every time my co-author and I released a new version
through the controlled ftp site, within a day or so the code somehow
managed to show up on another ftp site in Italy with worldwide access.
We (the authors) always managed to be shocked when that happened.  But
we had nothing to do with it, so it wasn't our problem.

> Today, it is still somewhat painful to comply with these laws,
> but this is what the PSF is for, which can fill out the forms
> necessary to allow exporting this code from the U.S.A.

Well, complying is painful in the sense of being morally repugnant
(people shouldn't have to notify the government in order to exercise
their free speech rights), but the actual process is pretty easy in
terms of the work required.  Python should qualify for the TSU
exception which means you just need to send an email to the BXA,
without needing to fill out any forms.  I thought that's what you had
done for the rotor module.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

Nick Craig-Wood <[EMAIL PROTECTED]> writes:
> There is a PEP about this...
> 
>   API for Block Encryption Algorithms v1.0
>   http://www.python.org/peps/pep-0272.html

Yes, I know about that and have been in contact with its author.  He
and I are in agreement (or at least were in agreement some time ago)
that the proposed API of the new module is an improvement, at least
for a generic module.  PEP 272 seems to document the interface of
something that had been implemented for some particular application.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Nick Craig-Wood]

Paul Rubin  wrote:
>  "Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> > Apparently, people disagree on what precisely the API should be. E.g.
> > cryptkit has
> > 
> > obj = aes(key)
> > obj.encrypt(data)
> 
>  I don't disagree about the API.  The cryptkit way is better than ECB
>  example I gave, but the ECB example shows it's possible to do it in
>  one call.

There is a PEP about this...

  API for Block Encryption Algorithms v1.0
  http://www.python.org/peps/pep-0272.html

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Nick Craig-Wood]

Skip Montanaro <[EMAIL PROTECTED]> wrote:

>  Fine.  Go build a sumo distribution and track the normal CPython.
>  The problem isn't all that new.  (Take a look at scipy.org for one
>  take on that theme.  Of course Linux distros have been doing their
>  take on this forever.)

If I'm writing code just for fun. I'll be doing on Debian Linux, then
I can do

  apt-get install python-crypto

and I'm away. 

However if I'm writing code for work, it has to work on windows as
well, which introduces a whole extra barrier to using 3rd party
modules.  Something else to compile.  Something else to stick in the
installer.  Basically a whole heap of extra work.

I think one of the special things about Python is its batteries
included approach, and a crypto library would seem to be an obvious
battery to install since it doesn't (or needn't) depend on any other
library or application.

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

Paul Rubin wrote:
Oh, ok.  Earlier you said you wanted user feedback before you could
conclude that there was reason to want an AES module at all.
I believe I never said that. I said that I wanted user feedback to
determine whether *this* AES module (where this is either your
from-scratch implementation, or any other specific implementation
contributed) is desirable.
Hmm, this is very very interesting.  I am highly confident that all
the purely technical questions (i.e. everything about the API and the
code quality, etc.) can converge to a consensus-acceptable solution
without much hassle.  I had thought there were insurmountable
obstacles of a nontechnical nature, mainly caused by legal issues, and
that these are beyond any influence that I might have by writing or
releasing anything.
These obstacles are indeed real. But I believe they are not
unsurmountable. For example, there is the valid complaint that,
in order to export the code from SF, we need to follow U.S.
export laws. 10 years ago, these would have been unsurmountable.
Today, it is still somewhat painful to comply with these laws,
but this is what the PSF is for, which can fill out the forms
necessary to allow exporting this code from the U.S.A.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

Skip Montanaro <[EMAIL PROTECTED]> writes:
> >> * Quixote
> 
> Paul> Don't know what this is.
> 
> Web app framework.

I think Python should add a web app framework to its core, again since
it otherwise can't seriously begin to compete with PHP.  However,
there are lots of approaches so this is an example of where your
suggested process of letting a bunch of different implementations
circulate before choosing something is a good idea.

> >> * PyInline
> 
> Paul> Not sure what this is.
> 
> A module for inlining C code within a Python module.  Also see Weave from
> the scipy.org folks.  It was inspired by the Perl Inline::C module.

Hmm, sounds like it has the same issues as Pyrex.  I'm also not sure
why you'd want both PyInline and Pyrex.

> >> * PyGTK
> 
> Paul> wxPython might be a better choice.  
> 
> Doesn't matter.  At work they decreed GTK as the GUI platform long before I
> came along (they also use gtkmm for C++ apps).

Can't wxPython use GTK?

> It's still an example of a broadly useful package available outside
> the core distribution.

I'd say if access to GTK is widely important functionality, then the
core should provide it in some way (e.g. through wxPython) and that's
enough.  If your company wants some different (i.e. nonstandard, if
wxPython becomes the standard) form of access, then it can deal with
the consequences of not following standards.

> Paul> 2. Isn't xmlrpclib written in Python?  
> Yes.  The implementation language is just a detail.

I think it's more than a detail.  If an external module is written in
Python, I can download it from wherever and include it with my own app
that I send to an end user.  I do the work so the end user doesn't
have to.  If it's written in C, then the end user has to deal with it.

> Paul> See, as Python improved, those things went into the core.
> 
> Sure, than that's what Martin has been trying to tell you about your AES
> proposal.  Put it out there, refine it, and get it into the core when it's
> mature.

What kind of refinements are you envisioning?  This isn't a web
application framework we're talking about.  It's more like the sha
module.

> Paul> Could you use sigalarm instead?
> 
> I suppose.  That's not the point though.  I'm not married to the concept as
> you seem to be that something has to be in the core distribution to be of
> use to me.  I'm perfectly happy incorporating solutions other people
> provide.

So aren't you happier when the other person provides you with a
solution that installs with one command, instead of a solution that
requires you to download N different modules from who knows where, and
install them separately, all while hoping that they haven't been
tampered with?  If I'm trying to provide someone else with a solution,
I'd rather use sigalarm than make the end-user download an extra
module, because I think they'll be happier that way.

> Paul> What happens if you send your Python program to a
> Paul> nonprogrammer friend who has just a vanilla Python installation?
> 
> I figure out some other packaging solution.  In my world most of the
> software I write is for my employer, so this is not a problem I face very
> often.  People use freeze, py2exe, py2app or other packaging solutions to
> solve most/all of these problems.

Only those people who think that a cross-platform application is one
that works on both XP Home and XP Pro.  That does simplify some
things.  Life in a cult is often indeed simpler than life in the real
world .

> Actually, there were at least two fairly mature implementations of
> CSV modules out there before the PEP was a twinkle in anyone's eye.
> The authors of those modules got together and wrote the current PEP
> and module from scratch based upon their collective experience.

Yes, CSV is complicated and benefits from that process just like
web app frameworks do.  Let's pick another example, the hmac module
that appeared in Python 2.2.  It implements the RFC 2104 HMAC algorithm.

Where are the two mature implementations that circulated before the
hmac module was added?  Where were the authors pooling their
collective wisdom?  Where was the year of user feedback?  The answer
is, nothing like that was needed.  HMAC is simple enough for a module
author to read RFC 2104 and implement what it says, run some tests,
and declare the module good to go.

> I think the effort of having a couple versions out in the field
> followed by joint effort to produce something worthy of inclusion in
> the core is an excellent demonstration of what Martin has been
> saying all along.

Martin is saying the opposite: that he doesn't understand the point of
writing a new module that synthesizes from experiences with old
modules, instead of just using one of the old modules.

I don't think there's a one-size-fits-all answer to any of these
questions.  You have to have your hands in the details of a specific
problem, to arrive at the best way to deal with that problem.
-- 
ht

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

Nick Craig-Wood <[EMAIL PROTECTED]> writes:
> I would hate to see a module which only implemented ECB.  Sure its the
> only operation necessary to build the others out of, but its the least
> secure mode of any block cipher.

It's intended as a building block for other modes.  Most applications
shouldn't use it directly.

> If you don't offer users a choice, then they'll use ECB and just that
> along with all its pitfalls, meanwhile thinking that they are secure
> because they are using AES/DES...

The documentation has to be written somewhat forcefully to tell users
what not to do.  I can help with that.  I've had to do that a lot,
supporting crypto packages in projects where the other programmers
haven't used crypto very much.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> mxCrypto is primarily unsuitable for the core because Marc-Andre Lemburg
> will never ever contribute it. He is very concerned about including
> crypto code with the Python distribution, so he certainly won't
> contribute his own.

Oh wait, I confused mxcrypto and m2crypto.  Sorry.  Anyway, the
technical considerations are similar.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> Apparently, people disagree on what precisely the API should be. E.g.
> cryptkit has
> 
> obj = aes(key)
> obj.encrypt(data)

I don't disagree about the API.  The cryptkit way is better than ECB
example I gave, but the ECB example shows it's possible to do it in
one call.

> I think I would prefer explicit encrypt/decrypt methods over a
> direction parameter. Whether or not selection of mode is a separate
> parameter, or a separate method, might be debatable

I prefer separate methods too, however if it was done with a direction
flag instead, it wouldn't really cause a problem.  As long as the
functionality is there, I can use it.

> I would trust my intuition more for a single function than for an
> entire API. In this specific proposal, I think I would trust my
> intuition and reject the ECB function because of the direction argument.

As an experienced user of a lot of these packages, I can tell you I've
seen it done both ways and I have a slight preference for separate
calls, but it really doesn't matter one way or the other and it's not
worth getting in a debate about it or having a committee design the
API and worry about such trivial issues.  

BTW, the main reason to reject the example ECB function is that
creating a key object ("key schedule") from a string can take
significant computation (sort of like compiling a regexp) so the ECB
function for some ciphers would have to cache the object like the
regexp module does.  Yuck.

The direction flag question would normally be between:

key = aes.key(key_data)
ciphertext = key(plaintext, "e")

or 
key = aes.key(key_data)
ciphertext = key.encrypt(plaintext)

FWIW, another way to do it, also sometimes preferable, is:

   key = aes.ecb(key_data, "e")  # e for encryption, or "d" for decryption
   ciphertext = key(plaintext)

I think the module I proposed did it this last way, but I haven't
looked at it in a while.

The point is that when faced with yet another crypto package, I don't
get in too concerned about which simple API variant it uses to do such
a basic operation.  I just care that the operation is available.  I
look in the docs to find that package's particular API for that
operation, and I do what the docs say.

I should make it clear that this module is like Python's low-level
"thread" module in that you have to know what you're doing in order to
use it directly without instantly getting in deep trouble.  Most
applications would instead use it indirectly through one or more
intermediate layers.  

> I fully understand what you desire - to include the module "as a
> battery". What makes this decision difficult is that you fail to
> understand that I don't want included batteries so much that I
> would accept empty or leaking batteries.

I do understand that, and the prospect of empty or leaking batteries
is vitally important to considering whether to include a battery
that's included, but for the purposes of an included-battery
discussion, the characteristics of NON-included batteries is not
relevant, given that we know they exist.

> >>http://sourceforge.net/projects/cryptkit/  ...> 
> > I've examined that module, I wouldn't consider it
> > ideal for the core (besides AES, it has some complicated additional
> > functions that aren't useful to most people)
> 
> Ok, that would be a problem. If this is a simple removal of functions
> that you'ld request (which functions?), 

OK.  First you have to decide whether you want a general crypto
toolkit, or just an AES module.  I've been concentrating on just an
AES module (or rather, a generic block cipher module with AES and DES)
since I figure that creates fewer areas of controversy, etc.  I think
it's too early to standardize a fancy toolkit.  Once there's block
ciphers, we can think about adding more stuff afterwards.

For that module, I'd say remove everything except AES and maybe
SHA256, and ask that DES be added.  SHA256 is possibly useful, but
isn't really part of an encryption package; it can be separated out
like the existing sha and md5 modules.  Also, it should be brought
into PEP 247 compliance if it's not already.

Rationale: I'd get rid of the entropy module now that os.urandom is
available.  Having the OS provide entropy is much better than trying
to do it in user code.  I'd get rid of the elliptic curve stuff unless
there's some widely used standard or protocol that needs that
particular implementation.  Otherwise, if I want ECC in a Python
program, I'd just do it on characteristic-p curves in pure Python
using Python longs.  (Bryan's package uses characteristic-2 curves
which means the arithmetic is all boolean operations, that are more
efficient on binary CPU's, especially small ones.  But that means the
module has to be written in C, because doing all those boolean
operations in Python is quite slow.  It would be like trying to do
multi-precision arithmetic in Python with Python ints instead of
longs).  Once there's a widel

Re: What's so funny? WAS Re: rotor replacement

[Skip Montanaro]

Paul> I've had this discussion here before, maybe not with you.  What I
Paul> really want is zero installations of anything.  

Fine.  Go build a sumo distribution and track the normal CPython.  The
problem isn't all that new.  (Take a look at scipy.org for one take on that
theme.  Of course Linux distros have been doing their take on this forever.)

>> If everyone adopted your position that it wasn't Python unless it had
>> been added to the core, we'd all be reinventing lots of wheels or
>> tackling much less challenging tasks, if we programmed in Python at
>> all.  Here's an incomplete list of stuff not in the core I have used
>> happily over the past several years to do my jobs using Python:

Paul> That makes no sense at all.  That list is a list of programs
Paul> written in the Python language.  They are Python programs, where
Paul> Python is an adjective.

No, many of them are just modules or programming frameworks.

>> * SpamBayes

Paul> I have the impression this is an application and not a module, 

Yes, you're correct.

>> * Quixote

Paul> Don't know what this is.

Web app framework.

>> * Docutils

Paul> Should be in the core if it's what I think it is.

Probably will be someday.

>> * MoinMoin

Paul> Application, should be separate.  Also, GPL'd, I think.  Can't be
Paul> distributed under PSF license.

Sure.

>> * Psyco

Paul> I think this isn't ready for prime time yet.  Should go into the
Paul> core once it is.

It's getting close for those of us with Intel chips in our boxes.

>> * PyInline

Paul> Not sure what this is.

A module for inlining C code within a Python module.  Also see Weave from
the scipy.org folks.  It was inspired by the Perl Inline::C module.

>> * PyGTK

Paul> wxPython might be a better choice.  

Doesn't matter.  At work they decreed GTK as the GUI platform long before I
came along (they also use gtkmm for C++ apps).  It's still an example of a
broadly useful package available outside the core distribution.

>> * xmlrpclib before it was in the core

Paul> 1. Did you really need this, instead of some more reasonable rpc
Paul>format?

Yes, for several years I used a homegrown RPC solution behind the Musi-Cal
website that was Python only.  Eventually Mojam (a Perl shop) bought
Musi-Cal (a Python shop).  I switched to XML-RPC with little effort.  At one
point we also had Java talking XML-RPC.

Paul> xdrlib has been in the core forever.

Sure.  But it's somewhat lower level than XML-RPC and isn't really an RPC
protocol.  It's just a marshalling protocol and is probably not as flexible
as XML-RPC at that.

Paul> 2. Isn't xmlrpclib written in Python?  

Yes.  The implementation language is just a detail.  I also use Fredrik
Lundh's sgmlop library to accelerate XML-RPC and play some other games when
I know I'm talking Python-to-Python (marshal my args, then XML-RPC the
result passing a single argument between the client and server).

>> * MAL's mx.DateTime before the core datetime module was available

Paul> See, as Python improved, those things went into the core.

Sure, than that's what Martin has been trying to tell you about your AES
proposal.  Put it out there, refine it, and get it into the core when it's
mature.

>> * timeout_socket before sockets supported timeouts

Paul> Could you use sigalarm instead?

I suppose.  That's not the point though.  I'm not married to the concept as
you seem to be that something has to be in the core distribution to be of
use to me.  I'm perfectly happy incorporating solutions other people
provide.  I believe you will find I am in the majority in this regard.

>> Many of those things I could never have written myself, either for
>> lack of time, lack of skill or both.  I'm grateful they were
>> available when I needed them and feel no qualms about using them even
>> though they are not distributed with Python proper.

Paul> Sure, it's fine if you have all those modules and you write a
Paul> Python program that uses, say, five of them.  External modules
Paul> aren't so bad when the developer and the end user are the same
Paul> person.  What happens if you send your Python program to a
Paul> nonprogrammer friend who has just a vanilla Python installation?

I figure out some other packaging solution.  In my world most of the
software I write is for my employer, so this is not a problem I face very
often.  People use freeze, py2exe, py2app or other packaging solutions to
solve most/all of these problems.

Paul> Now he has to download and install those five modules too.  You
Paul> send him the url's where you got the modules a year ago.  What are
Paul> the chances that the 5 url's even all still work, much less the
Paul> chance of him being able to install and run all 5 of the modules
Paul> without needing help?  What if the versions he downloads (fro

Re: What's so funny? WAS Re: rotor replacement

[Nick Craig-Wood]

Paul Rubin  wrote:
>  An AES or DES addition to an existing module that implements just one
>  call:
> ECB(key, data, direction)
>  would be a huge improvement over what we have now.  A more complete
>  crypto module would have some additional operations, but ECB is the
>  only one that's really essential.

I would hate to see a module which only implemented ECB.  Sure its the
only operation necessary to build the others out of, but its the least
secure mode of any block cipher.

If you don't offer users a choice, then they'll use ECB and just that
along with all its pitfalls, meanwhile thinking that they are secure
because they are using AES/DES...

For those people following along at home (I'm sure everyone who has
contributed to thread knows this already) I tried to find a simple
link to why ECB is bad, this PDF is the best I could come up with, via
Google's Cache.

http://www.google.com/search?q=cache:U5-RsbkSs0MJ:www.cs.chalmers.se/Cs/Grundutb/Kurser/krypto/lect04_4.pdf

-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

Paul Rubin wrote:
(And
actually: mxCrypto is the most capable of these packages and might be
the one with the most users, but it's completely unsuitable for the
core because of its size).
mxCrypto is primarily unsuitable for the core because Marc-Andre Lemburg
will never ever contribute it. He is very concerned about including
crypto code with the Python distribution, so he certainly won't
contribute his own.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

Paul Rubin wrote:
An AES or DES addition to an existing module that implements just one
call:
   ECB(key, data, direction)
would be a huge improvement over what we have now.
Apparently, people disagree on what precisely the API should be. E.g.
cryptkit has
obj = aes(key)
obj.encrypt(data)
I think I would prefer explicit encrypt/decrypt methods over a
direction parameter. Whether or not selection of mode is a separate
parameter, or a separate method, might be debatable - I'ld personally
prefer a separate method. However, we would have to ask users.
If you think a function like that could be added to some existing
module with less hassle than adding a new module, then I can write one
and submit it.
I would trust my intuition more for a single function than for an
entire API. In this specific proposal, I think I would trust my
intuition and reject the ECB function because of the direction
argument.
Come on, you're being deliberately obtuse, we've discussed this over
and over.  There are plenty of AES modules that people can get from
somewhere.  The topic is what it takes to have an AES module that
people don't NEED to get from anywhere, because they already have it
from having Python installed.  Do I have to keep repeating "batteries
included" until you understand what it means?
I fully understand what you desire - to include the module "as a
battery". What makes this decision difficult is that you fail to
understand that I don't want included batteries so much that I
would accept empty or leaking batteries.
http://sourceforge.net/projects/cryptkit/
Well, that code has been around for over a year, people are using it,
etc.  Are you saying you'll support its inclusion if Bryan offers to
contribute it?
*Now* you get it. Precisely that. I would ask the users what they
think about the API (shouldn't be too difficult because the module
does have users) and what they think about other aspects (performance,
stability, and so on).
I've examined that module, I wouldn't consider it
ideal for the core (besides AES, it has some complicated additional
functions that aren't useful to most people)
Ok, that would be a problem. If this is a simple removal of functions
that you'ld request (which functions?), I'ld try to collect opinions
on that specific issue, and ask Bryan whether he could accept
removal of these functions.
   So if the module was primarily written to be included in the core, I
   would initially reject it for that very reason. After one year or so
   in its life, and a recognizable user base, inclusion can be considered.
The context was new modules in general, not specifically an AES
module.  Since "considered" means "thought about", so you said
inclusion shouldn't even be thought about until the module is already
done.  That's completely in conflict with the idea of inviting anyone
to work on a new module, since inviting means that there's been some
thought.
I rarely invite people to work on new modules. For new modules, I
normally propose that they develop the module, and ship it to users
for some time.
I may have made exceptions to this rule in the past, e.g. when the
proposal is to simply wrap an existing C API in a Python module
(like shadow passwords). In this case, both the interface and
the implementation are straight-forward, and I expect no surprises.
For an AES module (or most other modules), I do expect surprises.
I would say there's an implied promise of something more than a code
review.  There's an implied statement that you agree that the proposed
new functionality is useful, which means the patch has a good chance
of being accepted to the stdlib if it's not too messy or cumbersome.
I have said many times that I am in favour of including an AES
implementation in the Python distribution, e.g. in
http://mail.python.org/pipermail/python-dev/2003-April/034963.html
What I cannot promise is to include *your* AES implementation,
not without getting user feedback first. The whole notion of
creating the module from scratch just to include it in the core
strikes me as odd - when there are so many AES implementations
out there already that have been proven useful to users.
So let me just ask you one final question: suppose I do all that
stuff.  The question: in your personal opinion, based on the best
information you have, what is the your own subjective estimate of the
probability?
Eventually, with hard work, I estimate the chances at, say, 90%. That
is, eventually, unless the code itself shows flaws, the module *will*
be included. However, initially, when first proposed, the chances are
rather like 10%. I.e. people will initially object. Decision processes
take their time, and valid concerns must be responded to. I personally
think that there is a good response to each concern, but it will take
time to find it. Before that, it will take time to find out what
precisely the concern is.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

Paul Rubin  writes:
> actually: mxCrypto is the most capable of these packages and might be
> the one with the most users, but it's completely unsuitable for the
> core because of its size).

Oops, I should say, mxCrypto itself isn't that large; the issue is
that it needs OpenSSL which is a big unwieldy program.  Having
mxCrypto in the core as an OpenSSL interface is a legitimate notion.
But there should be something that doesn't depend on OpenSSL.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

Skip Montanaro <[EMAIL PROTECTED]> writes:
> And one that deals with cryptography is likely to be even more complex.

No.  The AES module would have about the same complexity as the SHA module.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

Skip Montanaro <[EMAIL PROTECTED]> writes:
> What's your point?  That I have to download and perhaps install them to use
> them?  In that case, how are these two scenarios different:
> 
> * I have to download and build the MySQLdb package to talk to MySQL
>   servers from Python code
> 
> * I have to ensure that the readline library and include files are
>   installed on my system before the readline module (which is included
>   in the core distribution) can be built

The difference is that once Python is installed on your machine and
you can get a ">>>" prompt, you have readline available right away but
you have to download something to use MySQLdb.  Whoever took care of
your Python installation, and it may not have been you, also took care
of readline.  The past several OS distributions I've installed have
included Python and readline out of the box, so I never had to think
about readline.  The last time I used a Python instance that didn't
come with the OS (on Windows XP at work), the IT department had
installed Python on my desktop before I started using it, so I still
didn't have to worry about readline.  But any module that doesn't
come in the distro, I have to download myself.

> I and many other people happily use external packages other people have
> written as well as make stuff available.  My guess is that you do as well.

No, I don't.  I do use them sometimes but I'm unhappy about them.  If
I can write something using a core module instead of an external
module, I prefer to use the core module.  So I'll generally use dbm
instead of MySQL unless I really need MySQL, which I haven't yet in
Python (I've used MySQL with Perl dbi, but Perl, you know, shudder).

Also, external module installation scripts often don't work properly,
so I end up having to wrestle the code to get it installed.  And if a
geek like me has such trouble installing external modules, what hope
does a normal end-user have?  Maybe if you're using Windows, that
stuff has been debugged better, but I've had poor results under
GNU/Linux.

I've had this discussion here before, maybe not with you.  What I
really want is zero installations of anything.  I just want to go to
the store and buy a new computer and have a complete OS install with
full sources and a full set of applications including Python already
installed when I first power it up.  My practical approximation is to
buy a new computer, immediately reformat the HD to remove the icky
Redmond virus, and then install a GNU/Linux distro that includes
Python (and readline).  If Python really aims for world domination,
that means it has to shoot for being preinstalled on almost every new
computer the way Windows is now.  And all the interesting modules
should be there, maybe in a "contrib" directory that gets little or no
maintenance priority from the core team.

> If everyone adopted your position that it wasn't Python unless it
> had been added to the core, we'd all be reinventing lots of wheels
> or tackling much less challenging tasks, if we programmed in Python
> at all.  Here's an incomplete list of stuff not in the core I have
> used happily over the past several years to do my jobs using Python:

That makes no sense at all.  That list is a list of programs written
in the Python language.  They are Python programs, where Python is an
adjective.  Python, the noun referring to a piece of software,
generally means the stuff in the Python distro.  That doesn't stop
programs outside the distro from being useful.  Mediawiki is a PHP
program.  That doesn't mean Mediawiki is part of PHP.

> * MySQLdb, Sqlite, pycopg, sybase-python - all database modules

These should all be in the core if Python wants to be a serious
competitor to PHP, which comes with interfaces for those db's and
several additional ones besides.  That these modules are missing are a
significant library deficiency.

> * CSV, Object Craft's csv, DSV - csv modules predating csv in the core

That's fixed now, csv is in the core.

> * SpamBayes

I have the impression this is an application and not a module, or
anyway is written mainly to support one application.  Should be
separate.  Also, it's written in Python(?) rather than C, which means
the installation headaches from not being in the core aren't so bad.

> * Quixote

Don't know what this is.

> * Docutils

Should be in the core if it's what I think it is.

> * MoinMoin

Application, should be separate.  Also, GPL'd, I think.  Can't be
distributed under PSF license.  

> * Pyrex

Sort of problematic, would be interesting to have something like this
in the core but maybe Pyrex as it currently stands isn't the answer.

I have the impression that PyPy is going to depend on Pyrex in a
fundamental way, so it will have to be in the core when we dump CPython.

> * Psyco

I think this isn't ready for prime time yet.  Should go into the core
once it is.

> * PyInline

Not sure what this is.

> * PyGTK

wxPython 

Re: What's so funny? WAS Re: rotor replacement

[Skip Montanaro]

>> What matters is the code complexity, not whether something is in a
>> separate module or not.

Martin> A module *is* typically more complex than a single function. 

And one that deals with cryptography is likely to be even more complex.

Skip
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Skip Montanaro]

>> http://www.python.org/pypi
>> THIS IS ALL PYTHON.

Paul> No.  Those are programs people have written in Python or as Python
Paul> extensions.

What's your point?  That I have to download and perhaps install them to use
them?  In that case, how are these two scenarios different:

* I have to download and build the MySQLdb package to talk to MySQL
  servers from Python code

* I have to ensure that the readline library and include files are
  installed on my system before the readline module (which is included
  in the core distribution) can be built

I and many other people happily use external packages other people have
written as well as make stuff available.  My guess is that you do as well.
If everyone adopted your position that it wasn't Python unless it had been
added to the core, we'd all be reinventing lots of wheels or tackling much
less challenging tasks, if we programmed in Python at all.  Here's an
incomplete list of stuff not in the core I have used happily over the past
several years to do my jobs using Python:

* MySQLdb, Sqlite, pycopg, sybase-python - all database modules
* CSV, Object Craft's csv, DSV - csv modules predating csv in the core
* SpamBayes
* Quixote
* Docutils
* MoinMoin
* Pyrex
* Psyco
* PyInline
* PyGTK
* xmlrpclib before it was in the core
* MAL's mx.DateTime before the core datetime module was available
* timeout_socket before sockets supported timeouts

Many of those things I could never have written myself, either for lack of
time, lack of skill or both.  I'm grateful they were available when I needed
them and feel no qualms about using them even though they are not
distributed with Python proper.

Notice another interesting feature of several of those items: csv,
xmlrpclib, mx.DateTime, timeout_socket.  They were all modules I used that
eventually wound up in the core in some fashion.  They didn't go in the core
first, then demonstrate their usefulness.  It was the other way around.

Not everything that is useful belongs in the core distribution.  I think you
are confusing "batteries included" with "everything, including the kitchen
sink".

Skip
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> Indeed, if it was a single new function to an existing module, I would
> not require that this be delivered to users first. It is entire new
> libraries that I worry about.

Why is it different if a single new function is added to an existing
module, or if the single new function has the boilerplate of a new
module wrapped around it?

Look at the sha and md5 modules.  They are very similar in both
interface and implementation.  The only internal function that's
really different is the update operation; they actually might have
been combined into one module that did the other operations with the
same code.  But, it's also reasonable to have them as separate
modules.  If users start needing sha256, it could be done the same
way, one new update operation and the rest boilerplate, but in
practice it would probably be a separate module.

Are you saying if there was user demand for an sha256 module and
someone wrote one, you'd still require a year of separate distribution?

> A module *is* typically more complex than a single function. If
> your new module has only a single new function, we should discuss
> whether it really needs to be a separate module.

I previously had the mistaken belief that urandom was a new module
rather than a function inserted into an existing module.  Note that
the urandom's implementation is not ultra-trivial.

An AES or DES addition to an existing module that implements just one
call:
   ECB(key, data, direction)
would be a huge improvement over what we have now.  A more complete
crypto module would have some additional operations, but ECB is the
only one that's really essential.  I already have a pure-Python module
that does all the other operations using ECB as a subroutine.
It's speed isn't great but it's useable in some real applications.
It's only the ECB operation that's intolerably slow in Python.

If you think a function like that could be added to some existing
module with less hassle than adding a new module, then I can write one
and submit it.

> > Well, if he indicates that it's not a policy and that the question is
> > still open, then I could see getting interested again in writing an
> > AES module.  At the moment I continue to see his python-dev post as
> > quite discouraging.
> 
> And, again, I consider this perfectly fine. This would be a volunteer
> effort, and volunteers are free to work on whatever they please.

Well, volunteers are more likely to work on modules that are mentioned
as being welcome by the developers, than modules affected by explicit
prior developers' public decisions that cast a chill over the hope of
ever getting such a module accepted.

> Furthermore, people who want an AES module for Python could get one from

Come on, you're being deliberately obtuse, we've discussed this over
and over.  There are plenty of AES modules that people can get from
somewhere.  The topic is what it takes to have an AES module that
people don't NEED to get from anywhere, because they already have it
from having Python installed.  Do I have to keep repeating "batteries
included" until you understand what it means?

> http://sourceforge.net/projects/cryptkit/
> 
> Maybe Bryan Mongeau will contribute this code to Python some day.

Well, that code has been around for over a year, people are using it,
etc.  Are you saying you'll support its inclusion if Bryan offers to
contribute it?  I've examined that module, I wouldn't consider it
ideal for the core (besides AES, it has some complicated additional
functions that aren't useful to most people), but it would certainly
take care of my AES needs (it's apparently missing DES though).

> I did not say that. I said we don't normally invite people to work
> on anything - I said not that we *should* not invite them.

I would say that inviting people to work on a module for the stdlib
means the developers have thought about whether such a module would be
useful and worth including, and are indicating that they're favorable
to the idea.  However, you wrote:

   In Message-ID: <[EMAIL PROTECTED]>
   So if the module was primarily written to be included in the core, I
   would initially reject it for that very reason. After one year or so
   in its life, and a recognizable user base, inclusion can be considered.

The context was new modules in general, not specifically an AES
module.  Since "considered" means "thought about", so you said
inclusion shouldn't even be thought about until the module is already
done.  That's completely in conflict with the idea of inviting anyone
to work on a new module, since inviting means that there's been some
thought.

> Now that you mention it, I find that there is an important exception
> from my factual statement: I do regularly ask people reporting bugs
> or requesting features to work fixing the bugs or implementing the
> features. It is perfectly fine if they say "no" then. If they say
> yes, there is an implied promise that I'll review their 

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

Paul Rubin wrote:
Oops, sorry, it's in the os module:
   http://docs.python.org/lib/os-miscfunc.html
The difference is simply a matter of the packaging.
No, it's not. It also is a matter of code size, and impact. Small
additions can be reviewed and studied more easily, and need to be
tested on less users. A new module is on a larger scale than
a mere new function.
> Unless you're saying that if I
wanted to add AES to the string module (so you could say 
'spam and sausage'.aes_encrypt('swordfish banana')) instead of writing a
separate module, then we wouldn't need this discussion.
Indeed, if it was a single new function to an existing module, I would
not require that this be delivered to users first. It is entire new
libraries that I worry about.
If you would propose a change to the string module to add an aes_encrypt
function, I would immediately reject that patch, of course, because that
function does not belong to the string module.
What matters is the code complexity, not whether
something is in a separate module or not.
A module *is* typically more complex than a single function. If
your new module has only a single new function, we should discuss
whether it really needs to be a separate module.
Well, if he indicates that it's not a policy and that the question is
still open, then I could see getting interested again in writing an
AES module.  At the moment I continue to see his python-dev post as
quite discouraging.
And, again, I consider this perfectly fine. This would be a volunteer
effort, and volunteers are free to work on whatever they please.
Furthermore, people who want an AES module for Python could get one
from
http://sourceforge.net/projects/cryptkit/
Maybe Bryan Mongeau will contribute this code to Python some day.
Not true.  For example, you once invited me to work on an ancillary
message feature for the socket module (SF bug 814689), and so it's
been on my want-to-do-one-of-these-days list since then.  I think it's
reasonable for me to have taken your message there as an expression of
interest, sufficient to get me to want to work on it.  So it's bogus
to say the Python developers should avoid expressing interest in
something that hasn't already been written.
I did not say that. I said we don't normally invite people to work
on anything - I said not that we *should* not invite them. Now that
you mention it, I find that there is an important exception from my
factual statement: I do regularly ask people reporting bugs
or requesting features to work fixing the bugs or implementing the
features. It is perfectly fine if they say "no" then. If they say
yes, there is an implied promise that I'll review their code when
they are done.
As it appears to be clear that you are not going to implement
an AES module in the foreseeable future, and as it also seems
to be clear that you cannot talk me into changing my views on
how Python should be developed, I think further discussing this
entire thing is pointless.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> > Let's see, the urandom module was recently released in 2.4, I think
> > initially at my urging.
> 
> There is no urandom module in Python 2.4.

Oops, sorry, it's in the os module:

   http://docs.python.org/lib/os-miscfunc.html

The difference is simply a matter of the packaging.  It's important
functionality and it was added after an amount of review and testing
that was deemed to be reasonable.  Unless you're saying that if I
wanted to add AES to the string module (so you could say 
'spam and sausage'.aes_encrypt('swordfish banana')) instead of writing a
separate module, then we wouldn't need this discussion.  (Putting it
in the string module is not a serious proposal but it wouldn't be
completely insane.)  What matters is the code complexity, not whether
something is in a separate module or not.

> > If Guido says "no crypto", is that something other than a
> > policy?  And are you speaking for him when you say that?  
> 
> If he had said such a thing in general, it would be a policy
> (although it best would be documented somewhere). I don't think
> he said that, in general, and with the intent of stating a policy.

Well, if he indicates that it's not a policy and that the question is
still open, then I could see getting interested again in writing an
AES module.  At the moment I continue to see his python-dev post as
quite discouraging.

> We don't normally invite people to work on anything. People pick the
> things they work on themselves.

Not true.  For example, you once invited me to work on an ancillary
message feature for the socket module (SF bug 814689), and so it's
been on my want-to-do-one-of-these-days list since then.  I think it's
reasonable for me to have taken your message there as an expression of
interest, sufficient to get me to want to work on it.  So it's bogus
to say the Python developers should avoid expressing interest in
something that hasn't already been written.

> > IETF often decides and announces that a feature is good long before
> > any such details are decided or agreed on.
> 
> No. They decided that spam machinery in DNS would be a good thing,
> people started to work on it, and then they decided that it is not
> such a good thing, after all, because it causes too many problems.

There's no contradiction.  IETF decided something based on some
reasonable beliefs that later turned out to be wrong, so they had to
undo the original decision because the original expectations couldn't
be met.  That happens sometimes.  Everyone has worked on projects that
ended up failing and getting cancelled for reasons like that.  There
are few ironclad guarantees of anything, only reasonable beliefs.  But
real-world organizations are not afraid to start projects based on
reasonable beliefs regardless.  Otherwise, few things would ever get
done.

> It does matter for me, yet Python is still more than the core.
> 
> You might be ignoring that, but it surely is more to the many
> developers which create Python libraries and distribute them
> themselves, see
> 
> http://www.python.org/pypi
> THIS IS ALL PYTHON.

No.  Those are programs people have written in Python or as Python
extensions.  I've written programs like that myself.  I don't consider
them to be part of Python.  They are separate programs.  If one of
them gets accepted into the distro (as this thread is about), then it
becomes part of Python.  See :

1.1.6   How do I obtain a copy of the Python source?

The latest Python source distribution is always available from
python.org, at http://www.python.org/download/. The latest development
sources can be obtained via anonymous CVS from SourceForge, at
http://www.sourceforge.net/projects/python.

Python is what you get from following the download instructions and
installing the result.  In ,
where it asks "why is Python installed on my machine?", that is what
they are referring to by "Python".

> > Obviously those require a different type of consideration.  I'm
> > talking about patches where there's a core developer with an interest.
> 
> I though you were talking about the AES module...

No, when this started you were talking about modules in general.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

Paul Rubin wrote:
Let's see, the urandom module was recently released in 2.4, I think
initially at my urging.  
There is no urandom module in Python 2.4.
If you can't speak for others, how can you say there's no policy in
force?
I should say I'm not aware of a policy.
If Guido says "no crypto", is that something other than a
policy?  And are you speaking for him when you say that?  
If he had said such a thing in general, it would be a policy
(although it best would be documented somewhere). I don't think
he said that, in general, and with the intent of stating a policy.
That's much different than merely deciding
that a feature is good and inviting people to work on it.
We don't normally invite people to work on anything. People
pick the things they work on themselves.
IETF often decides and announces that a feature is good long before
any such details are decided or agreed on.  For example, they decided
that IP connection encryption was a good feature, and appointed a
working group a general mandate to go figure out the details (IPSEC)
and write a spec.  See?
No. They decided that spam machinery in DNS would be a good thing,
people started to work on it, and then they decided that it is not
such a good thing, after all, because it causes too many problems.
So the decision "this is a good thing" is no guarantee for
"if it is done, we will publish it as a standard". They might
start a process, people might get nearly through it, and then
the process stucks or is given up entirely. Happens all the time.
I realize that the difference might
not matter to you, but it does matter to a heck of a lot of other
users.
It does matter for me, yet Python is still more than the core.
You might be ignoring that, but it surely is more to the many
developers which create Python libraries and distribute them
themselves, see
http://www.python.org/pypi
THIS IS ALL PYTHON.
Obviously those require a different type of consideration.  I'm
talking about patches where there's a core developer with an interest.
I though you were talking about the AES module...
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Terry Hancock]

On Tuesday 25 January 2005 01:51 am, Fredrik Lundh wrote:
> ah, you're the guy who flamed me and called me names because PIL didn't
> fit some hypothetical GNU-inspired definition of "open source software."  it's
> always sad when people have to attack those who don't share their religion,
> but it's not that much I can do about that.

That is a complete misrepresentation of the post in question.  It was in
no manner of interpretation a "flame".  You simply took offense out of thin
air.  And there's not much *I* can do about *that*.

I figured that my intent to contribute to your project was sufficient to
indicate my good intentions. 

> >  So, naturally, I just dropped it.
> 
> which means that you have no idea how easy or hard it is to install PIL today
> (hint: things have changed)

Oh?  In which version?

Are you saying that your installation procedure now honors the "--with-jpeg=" 
and
"--with-png=" flags to configure?   Because although you had these flags, in 
the last
version I installed from source, they did not work -- it was still necessary to 
edit
the Makefile.

That was either 1.1.3 or 1.1.4, and I'm pretty sure it was the latter.

Now if you've actually fixed this, I look forward to trying it out the next
time I have to do a complete from-source installation.  I'll say "thank you",
just in case.

Cheers,
Terry

--
--
Terry Hancock ( hancock at anansispaceworks.com )
Anansi Spaceworks  http://www.anansispaceworks.com

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> And I still stand by those blanket statements. Any new module (i.e.
> specific code) should see real users for some time before it can
> be incorporated into Python.

Let's see, the urandom module was recently released in 2.4, I think
initially at my urging.  I believe it was written just for 2.4,
combining a Windows function used only (as far as I know) by its
author, and a wrapper to read /dev/urandom on Posix systems.  It
didn't have any wide distribution or use before incorporation, and
didn't need it.  It was simple enough that a reasonable code review
and some testing on the relevant platforms was enough to make make
sure it worked properly.

> > I would say the first thing I'd request would be your sense of how
> > plausible it is that the current no-crypto policy might be relaxed.
> 
> There is no such policy in force. Different people have different
> opinions, and those opinions change over time. I cannot speak for others.

If you can't speak for others, how can you say there's no policy in
force?  If Guido says "no crypto", is that something other than a
policy?  And are you speaking for him when you say that?  

> proposed feature is good, and I said that, for an example, IETF even
> requires *two* implementations before deciding that the feature is good.

No, that's completely wrong.  IETF requires two implementations before
publishing a specification--a precise description--as a draft
standard, which means agreeing on how the feature should work at the
finest level of detail.  That's much different than merely deciding
that a feature is good and inviting people to work on it.

IETF often decides and announces that a feature is good long before
any such details are decided or agreed on.  For example, they decided
that IP connection encryption was a good feature, and appointed a
working group a general mandate to go figure out the details (IPSEC)
and write a spec.  See?  IETF announced in advance that IPSEC was a
good feature, and the working group members knew of that decision
before they committed the resources needed to write the draft.  And
when they started, there were zero actual IPSEC implementations,
because nobody knew what precise characteristics IPSEC would
eventually have.  The implementations got written in the process of
developing the spec and presenting it as a standard.

The two-implementation policy means there has to be two
implementations before the draft is finally declared complete, not
before the IETF can announce that it wants a standard to be written
and intends to use whatever the working group comes up with once the
draft and two implementations are done.  That is what happens in any
normal professional process.  Nobody commits resources to develop
something (other than for experimentation) unless they think the end
result will actually be used.  Are you really having trouble
understanding this?

> >>That's not true. PyCrypto does have AES support.
> > That's not in any Python distro that I know of.
> 
> Sure, but "Python" is more than the Python core.

Tell me again what "batteries included" means.  I personally interpret
"included" as meaning "included".  That means when I install Python,
the module is right there in the distro.  So if it's not in the
distro, Python itself doesn't have it.  "Batteries included" is an
expression of Pythonic philosophy that says: putting components that
end-users need in the distro is superior to making the users get them
from somewhere else separately.  I realize that the difference might
not matter to you, but it does matter to a heck of a lot of other
users.

> With that attitude, the patches tracker on SF would grow unbounded,
> because we *continuously* get patches that no core developer has
> any personal interest in.

Obviously those require a different type of consideration.  I'm
talking about patches where there's a core developer with an interest.
Let's say the cmath module doesn't exist and someone submits it.  It
computes functions like cos(z) for complex arguments.  If Tim, the
numerics expert, then says "this is worth having and I've examined the
code and it looks good, let's put it in", I'd say that takes care of
it, and I think that's what typically happens with real modules.  If
nobody on the core team has an informed opinion, then it has to be
decided some slower and more formal way.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

Paul Rubin wrote:
I can say that assuming I know what so-and-so is. For the specific
case of AES, I would say "I don't think the Python lib necessarily
needs to have an AES module, but I would not object if it had one"

Well, ok, you're changing your tune a little bit now, and getting more
reasonable.  Before, you were making blanket statements of any modules
written for the Python stdlib.  Now you're limiting it to AES and basing
it on some AES-specific things.
And I still stand by those blanket statements. Any new module (i.e.
specific code) should see real users for some time before it can
be incorporated into Python.
The question whether I would like to see a certain *feature*
in Python (whether as a separate module, a language feature, or
otherwise) is an entirely different question. That I would not
object to an AES code in general doesn't imply that I will
agree to any AES module that somebody contributes.
I would say the first thing I'd request would be your sense of how
plausible it is that the current no-crypto policy might be relaxed.
There is no such policy in force. Different people have different
opinions, and those opinions change over time. I cannot speak for
others.
Again OK.  I had thought from earlier discussions that you were a
crypto developer and familiar with this stuff; no problem if you're
not.  However, in that case, you probably wouldn't be using the module
if it got included.  I think the best policy if you don't feel
comfortable supporting including some module (whether it's crypto or
something else) that you're not personally going to use, is don't
support inclusion, but also don't obstruct it.
Thanks for the advise, but I'll have my own policies. I have included
several modules in the past which I'm not personally using.
In addition, for any new module, there is one primary requirement
for acceptance that cannot be fulfilled in code quality: the
contributor should promise to support the module in a foreseeable
future (i.e. a couple of years).

That's not too unreasonable, though I don't think I've heard it
mentioned as a requirement before.
See PEP 2.
I've never heard of any requirement before that there be two separate
implementations of every Python stdlib module, by the way.  That would
be silly.
And I did not suggest such a thing. You said you never heard about
a process which requires an implementation before deciding that the
proposed feature is good, and I said that, for an example, IETF even
requires *two* implementations before deciding that the feature is
good.
However, the result of my not writing an AES module is that Python
doesn't have an AES module.
That's not true. PyCrypto does have AES support.
That's not in any Python distro that I know of.  
Sure, but "Python" is more than the Python core.
I myself would probably never support including any
such module no matter how long it was distributed, but rather would
just defer the whole question to people experienced with such modules
and trust their sense of what the acceptance criteria should be for a
specific module.  That is, I'd abstain from any decision.
With that attitude, the patches tracker on SF would grow unbounded,
because we *continuously* get patches that no core developer has
any personal interest in. I'll try to look at *all* patches, whenever
I can, and in the process, I have to learn about technologies which
I've never used before (primarily operating systems, but also
APIs and protocols).
Rejecting all these patches would be unfair to the contributors.
Your answer is more like "make enough for 500 people and bring it to
the conference site and only then will we even THINK about whether to
serve it for breakfast.
I cannot see these as the same thing.
I'm sure that's true of lots of modules.  If you're the sole maintainer
of the Python distro, then you have to evaluate every module that anyone
submits.  But since you're part of a reasonable sized group, I think
it's best to examine the modules you care about, but don't feel that
you have to have your hands in everything.
Thanks again for the advise, but this is not how Python is being
maintained.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> > I don't see why you can't make up your mind enough to issue simple
> > statements like "the Python lib should have a module that does
> > so-and-so
> 
> I can say that assuming I know what so-and-so is. For the specific
> case of AES, I would say "I don't think the Python lib necessarily
> needs to have an AES module, but I would not object if it had one"

Well, ok, you're changing your tune a little bit now, and getting more
reasonable.  Before, you were making blanket statements of any modules
written for the Python stdlib.  Now you're limiting it to AES and basing
it on some AES-specific things.

> (the latter part in consideration of consequences that inclusion
>   of crypto code might have).

I would say the first thing I'd request would be your sense of how
plausible it is that the current no-crypto policy might be relaxed.
And I think this is about totally up to Guido, since (reading between
the lines of those python-dev messages) he's concerned about the Dutch
crypto restrictions, which makes some sense because he's Dutch and
still has to deal with the Dutch government from time to time, while
you or I might not care what the Dutch government thinks.

I was about to mention that Rop Gongrijp has been operating a crypto
company in Holland (www.nah6.com) but I now see that it's moved to
Germany, and I wonder if those Dutch restrictions might be the reason.
I don't know any details though.

> > and it should meet such-and-such requirements
> 
> I can only say such things if I know such-and-such in detail
> to specify requirements. For the specific case of AES, I don't
> know enough about it to specify requirements. I will have to
> trust others (and by that, I mean *multiple* others)

OK.  If it helps, I can tell you that the Python crypto crowd hangs
out on a mailing list called python-crypto and has discussed this
module at some length.

> > so if someone submits one that meets the requirements and passes
> > code review and testing and doesn't have unexpected issues or
> > otherwise fail to meet reasonable expectations, we'll use it".
> 
> Because I cannot specify requirements, I cannot make such a promise.

Again OK.  I had thought from earlier discussions that you were a
crypto developer and familiar with this stuff; no problem if you're
not.  However, in that case, you probably wouldn't be using the module
if it got included.  I think the best policy if you don't feel
comfortable supporting including some module (whether it's crypto or
something else) that you're not personally going to use, is don't
support inclusion, but also don't obstruct it.  If there's some other
stdlib maintainers who are knowledgeable about that kind of module and
have an opinion about it, then go along with them unless there's a
clear reason not to.  Like, if I were a stdlib maintainer and somebody
submitted a fancy multigrid PDE solver, that's outside my expertise
and I'd go along with whatever Tim recommended.

> In addition, for any new module, there is one primary requirement
> for acceptance that cannot be fulfilled in code quality: the
> contributor should promise to support the module in a foreseeable
> future (i.e. a couple of years).

That's not too unreasonable, though I don't think I've heard it
mentioned as a requirement before.

> You are talking about such a thing. I don't know enough about the
> functionality to specify what an obvious interface is, or to
> recognize one if I see it.

The thing to do then is just defer the whole issue to someone on the
core team who uses that type of function.  Only if no one on the team
has an opinion do you start having to look for stuff like wide use of
the module in the outside community.

> > I don't know what OMG is, but there is no IETF requirement that any
> > implementations be available in any particular language.
> 
> See RFC 2026, section 4.1.2. Two independent implementations
> are required for the document to advance to draft (!) standard.

That says nothing about the implementation languages.  Both might be
in Python, or one might be in VHDL and the other in Ada, or whatever.

This particular module we're discussing is a straightforward
implementation of AES, DES, and FIPS 81, for which there are thousands
of implementations in both hardware and software.  For the specific
Python API, there's a Python implementation and the proposed new
module is a C implementation that does the exact same thing only
faster.  So that's two implementations, though not independent in the
IETF sense, because they'd both be written by the same person.

I've never heard of any requirement before that there be two separate
implementations of every Python stdlib module, by the way.  That would
be silly.

> > However, the result of my not writing an AES module is that Python
> > doesn't have an AES module.
> 
> That's not true. PyCrypto does have AES support.

That's not in any Python distro that I know of.  I'm talking about
module

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

[EMAIL PROTECTED] wrote:
I don't see why you can't make up your mind enough to issue simple
statements like "the Python lib should have a module that does
so-and-so
I can say that assuming I know what so-and-so is. For the specific
case of AES, I would say "I don't think the Python lib necessarily
needs to have an AES module, but I would not object if it had one"
(the latter part in consideration of consequences that inclusion
 of crypto code might have).
and it should meet such-and-such requirements
I can only say such things if I know such-and-such in detail
to specify requirements. For the specific case of AES, I don't
know enough about it to specify requirements. I will have to
trust others (and by that, I mean *multiple* others)
so if
someone submits one that meets the requirements and passes code review
and testing and doesn't have unexpected issues or otherwise fail to
meet reasonable expectations, we'll use it".
Because I cannot specify requirements, I cannot make such a promise.
In addition, for any new module, there is one primary requirement
for acceptance that cannot be fulfilled in code quality: the
contributor should promise to support the module in a foreseeable
future (i.e. a couple of years).
Again, we're talking about straightforward modules whose basic
interface needs are obvious.  
You are talking about such a thing. I don't know enough about
the functionality to specify what an obvious interface is, or
to recognize one if I see it.
I don't know what OMG is, but there is no IETF requirement that any
implementations be available in any particular language.
See RFC 2026, section 4.1.2. Two independent implementations
are required for the document to advance to draft (!) standard.

However, the result of my not writing an AES module is that Python
doesn't have an AES module.  
That's not true. PyCrypto does have AES support.
No, it's three steps
1. decide that you want to do it
2. do it
3. decide whether you are pleased with the result, and only
   use it if you are
IOW, there should not be a blanket guarantee to use it after step 1.

But, it's completely normal to say before step 1 that "if the result
of step 2 does so-and-so, then I'll be pleased in step 3",
That's what I'm saying: If you distribute the module to users for
a year, and users express interest and support for your choice of
API, I'll support inclusion of the module. "do it" involves more than
just writing the code.
You and
Frederik seem to think there's something inappropriate or
self-inflated about wanting that expectation before committing to do a
pile of work that's primarily for other people's benefit.
It's very easy. If you are primarily do it for other people's
benefit, and if you don't find any satisfaction in the process
of doing it - THEN DON'T. I really mean that; this is how
free software works. People *volunteer* to do things. If they
don't volunteer - that's perfectly fine.
I think
your stated attitude is completely bizarre, that you can't really
believe anything so silly, so you're really just acting bureaucratic,
looking for excuses to say no instead of yes to worthwhile proposals.
As I said above - for the specific feature in question, I don't
care enough for the feature itself. Python will be just as useful
to me with the feature as it is without.
What I do care about is the effort that I will need to continue
maintaining Python. I don't want to have to maintain an ill-designed,
buggy module with no active maintainer, and I don't want to tell
people that I had to rip the module out just because it doesn't
work at all.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

Skip Montanaro <[EMAIL PROTECTED]> writes:
> >> As long as we are discussing cryptography, what's wrong with
> >> m2crypto?  Or, what about Andrew Kuchling's crypto toolkit?
> 
> Lucas> Umm, is it just me or did we just discuss the legal issues of
> Lucas> that??
> 
> You may have.  Whether or not there are legal issues with them is of little
> relevance to the point I was making.  Anything Paul writes would probably
> have the same legal entanglements.

Andrew's toolkit does incorporate some patented algorithms, but those
aren't critical and could be removed from a stdlib version.

> I was simply pointing out that maybe, just maybe, there are already
> suitable candidates from a technical standpoint and that he doesn't
> need to write anything.

There really don't appear to be any that are both technically
suitable, and that the authors are willing to contribute to PSF.  I'd
be delighted to be wrong.  Of course that still leaves the legal
issue.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

Skip Montanaro <[EMAIL PROTECTED]> writes:
> phr> It is not a whizbang module.  It is a stripped-down, basic
> phr> implementation of a well-accepted set of standards that are being
> phr> used in thousands of other applications in other languages.
> 
> Then there should be a library already out there already.  All you
> should need to do is wrap it (manually, with SWIG, whatever).

I'm currently using something wrapped with SWIG, but my understanding
is that core modules are not supposed to depend on SWIG.  So a core
module will likely use some existing primitives wrapped by hand.  That
is what I've offered to do, along with providing a somewhat more
Pythonic interface (though still a straightforward one) than directly
wrapping a C library intended for use from C applications.

> phr> There is demand for it.  Look at how this thread started: some
> phr> crypto user noticed that rotor was gone and wanted to know what to
> phr> do instead.
> 
> Yes, and putting rotor back would be the wrong thing to do.

Correct.  The right thing is to replace rotor with something
reasonable that follows standards.

> phr> The issue of whether there's enough desire for a crypto module to
> phr> warrant including one in the stdlib was decided a long time ago.
> phr> The proof of that somebody decided to accept the rotor module into
> phr> the distro.
> 
> No, rotor was added in Python's early days (July 1992).  Times have changed.

I don't see that.  There's surely even more demand for crypto now than
there was in 1992.

> As long as we are discussing cryptography, what's wrong with m2crypto?
> 
> http://sandbox.rulemaker.net/ngps/m2/ 

It's a good package but it's pretty heavyweight.  It depends on both
SWIG and OpenSSL.  I think it's still under development--there's an
occasional flurry of messages about it on python-crypto, but I haven't
been following it closely.  I'd have a hard time making a case for
accepting it into the core given the difficulty I'm having making the
case for something as simple as a block cipher wrapper.
m2crypto+OpenSSL is at least 100 times as much code as the module I've
proposed.  I think the Python lib should someday have its own
pure-Python SSL/TLS implementation sort of like the one Java has.  But
if m2crypto went into the lib, I'd use it.

> Why not incorporate it into the standard distribution?

I don't have the authority to incorporate anything into the standard
distribution.  All I can do is offer to contribute stuff that I write,
and let the distro maintainers decide whether to incorporate it.

I don't have the authority to offer m2crypto, since I'm not the author.
Only the authors can do that.  They haven't done so, as far as I know.

> Or, what about Andrew Kuchling's crypto toolkit?
> 
> http://www.amk.ca/python/code/crypto.html

This is perhaps more suitable than m2crypto but as far as I know,
Andrew hasn't offered to contribute it.  Whatever his reasons are, I
have to respect them.  I think it has more stuff than a core crypto
module really needs (e.g. numerous semi-obsolete algorithms that
aren't in widespread use so aren't needed for interoperability) but
the extra stuff doesn't really get in the way.  If it were in the
core, it would completely fulfill my desires and I would be
transported with ecstacy.  But I've never seen any indication that
it's headed there.

> I believe both have been around awhile.  If crypto-in-the-core is
> really what's needed why not see if one of them is ready to go?

I don't think m2crypto is the answer, though maybe I'm wrong.  And if
Andrew's package is the answer, he would have submitted it already.

> phr> Have you ever used a crypto library in a serious way?  
> 
> Nope, never directly.  Don't make this about me.  I'm interested in the
> Python development process and how you'd like to turn it on its head.

Either you're the one turning the actual process on its head, or else
it's already on its head and needs to be turned rightside up.  Either
way, the existing process has certainly been a total failure so far at
producing good crypto support in the lib.

> phr> It's already the category king, because there are precisely zero
> phr> other entrants in the category.  
> 
> See my above references.  Note, I don't use crypto at all, yet I was aware
> of both of these (no Googling required).

The authors have not offered to contribute them, so they're not in the
category.  The category consists of crypto modules that have actually
been offered.  As I keep saying, I'd love it if someone else offered
one.  I'm not eager for this headache.  I just saw that somebody
really ought to do it, and nobody was doing it, so I decided I was
elected.

> My guess would be they are substantially more mature than your
> proposed module.

This may sound a little strange, since my module is not yet written,
but if we take maturity to mean lower future maintenance needs, then I
wouldn't say m2crypto is precisel

Re: What's so funny? WAS Re: rotor replacement

[Skip Montanaro]

>> As long as we are discussing cryptography, what's wrong with m2crypto?
>> Or, what about Andrew Kuchling's crypto toolkit?

Lucas> Umm, is it just me or did we just discuss the legal issues of
Lucas> that??

You may have.  Whether or not there are legal issues with them is of little
relevance to the point I was making.  Anything Paul writes would probably
have the same legal entanglements.  I was simply pointing out that maybe,
just maybe, there are already suitable candidates from a technical
standpoint and that he doesn't need to write anything.

Skip

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Skip Montanaro]

>> Finally, what if, saints be preserved, your whizbang new module is

phr> It is not a whizbang module.  It is a stripped-down, basic
phr> implementation of a well-accepted set of standards that are being
phr> used in thousands of other applications in other languages.

Then there should be a library already out there already.  All you should
need to do is wrap it (manually, with SWIG, whatever).

>> included in the core distribution and it's just not as popular as was
>> first thought?

phr> There is demand for it.  Look at how this thread started: some
phr> crypto user noticed that rotor was gone and wanted to know what to
phr> do instead.

Yes, and putting rotor back would be the wrong thing to do.

phr> The issue of whether there's enough desire for a crypto module to
phr> warrant including one in the stdlib was decided a long time ago.
phr> The proof of that somebody decided to accept the rotor module into
phr> the distro.

No, rotor was added in Python's early days (July 1992).  Times have changed.

As long as we are discussing cryptography, what's wrong with m2crypto?

http://sandbox.rulemaker.net/ngps/m2/ 

Why not incorporate it into the standard distribution?

Or, what about Andrew Kuchling's crypto toolkit?

http://www.amk.ca/python/code/crypto.html

I believe both have been around awhile.  If crypto-in-the-core is really
what's needed why not see if one of them is ready to go?  

phr> The rotor module is gone (and good riddance).  That's how this
phr> thread started, remember?  It shows that bogus modules can be
phr> removed.

Yeah, but it was there for over 10 years.

phr> Have you ever used a crypto library in a serious way?  

Nope, never directly.  Don't make this about me.  I'm interested in the
Python development process and how you'd like to turn it on its head.

>> When it's the category king and there is substantial community
>> support for inclusion,

phr> It's already the category king, because there are precisely zero
phr> other entrants in the category.  

See my above references.  Note, I don't use crypto at all, yet I was aware
of both of these (no Googling required).  My guess would be they are
substantially more mature than your proposed module.

phr> I read those as saying that no crypto module will be considered for
phr> inclusion whether or not it's the category king, because any such
phr> module might conflict with crypto regulations in some countries.

That may be a problem, sure.  I'm not sure how the discussion here changes
that.  That's just life as we know it.

phr> So tell me again what to do after writing and releasing a C module.
phr> There's just no reason to write one, if the module can't go into
phr> the stdlib.  

Why in the heck is inclusion in the standard library a requirement for you
to write this thing?  If it's useful to you, write it and get on with your
life.

>> Python is popular in part because of its fairly conservative
>> development strategy.  That goes for the libraries as well as the
>> language itself.

phr> Tell me again how rotor got into the distribution.

Okay.  It was 1992.  Bill Clinton had recently been elected president.  It
was pretty much pre-WWW as we know it.  Definitely pre-comp.lang.python and
pre-Google (heck, pre-Yahoo, pre-Win98 and pre-Mac OSX as well).  Pre-string
methods.  Pre-triple-quoted strings.  Pre-spammers.  Pre-DSL.  Pre-lots of
stuff.  There was an Emacs python-mode and s, both thanks to Tim.
People opined about Python's performance, just as they do today.  Python's
version number was around 0.9.4, definitel < 1.0.  Guido was the only person
with direct repository access.  Including something in the distribution was
probably the only convenient way to make new modules available.  If nothing
else, the rotor module (along with anything else included in the
distribution back then) may have been a good exercise in and demonstration
of writing extension modules, so it probably served a useful non-crypto
purpose.  Python's user community was probably a few hundred people.  Guido
likely had no thoughts of world domination with the little language that
could.  Times have changed.  You seem think there was a PEP process and
distutils and Wikis.  I suspect some of the algorithms one might include in
a robust crypto toolkit today weren't even invented in 1992.

So throw away the rotor crutch and put your money where your mouth is.

Skip
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Lucas Raab]

As long as we are discussing cryptography, what's wrong with m2crypto?
http://sandbox.rulemaker.net/ngps/m2/ 

Why not incorporate it into the standard distribution?
Or, what about Andrew Kuchling's crypto toolkit?
http://www.amk.ca/python/code/crypto.html

Umm, is it just me or did we just discuss the legal issues of that??
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

Jeremy Bowers <[EMAIL PROTECTED]> writes:
> The policy has been laid out, multiple times, by multiple people now. The
> answer is, you are not going to get any such indication that will satisfy
> you.

Actually I already got an indication that satisfied me, from Guido and
Andrew, although it was later withdrawn for nontechnical reasons (i.e.
legal restrictions) that I don't feel entitled to argue against very
much.  I do, however, believe that what Frederik and Martin are saying
is bogus on technical grounds, and furthermore, they're just being
busybodies, since they've had no involvement at all in what's happened
already with that module, and their picture of the Python development
process doesn't have much resemblance to how the current Python distro
got to be the way it is.

> Note that I am not a Python contributor of any kind. Also note that I
> figured this out about two days ago. You can wheedle these guys all you
> want, 

I have not asked them for anything, since they are not in a position
to give it.

>but they are too experienced for you to extract a promise from them.

Actually, one of them already gave a promise that he couldn't keep
even if he wanted to.  He said that if I wrote a crypto module and got
it tested enough, it would be considered for inclusion (i.e. based on
its technical merits).  But, he doesn't get to decide that.  Current
policy per per Guido seems to be that because of the legal stuff,
there will be no crypto module in the stdlib regardless of its merits.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Jeremy Bowers]

On Thu, 27 Jan 2005 04:04:38 +, phr wrote:

> Skip Montanaro <[EMAIL PROTECTED]> writes:
>> Because good requirements specification is difficult and testing improves
>> the breed.  Better to have the major API changes and bugs taken care of, and
>> to have its popularity demonstrated *before* it gets into the Python
>> distribution.  The best way to do something isn't always obvious at the
>> outset.
> 
> That is sometimes true, but not always.  Sometimes the best way is
> obvious.

With all due respect, you aren't arguing very well. Your posts are, like
this quote, more reactionary than thought-out. 

You need to stop arguing for the sake of argument, read more carefully,
and think more about your global goals in this conversation than your
local ones. I'm 99% certain that you've already gotten all the answers
you're going to get and all the answers you need in this thread, but
you're so busy arguing the local points like this, you haven't noticed. 

(Either that, or it's the "you haven't answered me because you haven't
given the exact answer I want" bit, which I find quite tiresome; there's
another list where there are these three teenagers that don't even realize
they've defined "answer" that way and keep bitching that they aren't being
"answered" when the "answer" is, basically, "it's nice that you feel that
way." But I digress.)

> I don't need your permission to do that.  What I'm looking for is an
> indication that it's worth my while, before I spend the effort.

The policy has been laid out, multiple times, by multiple people now. The
answer is, you are not going to get any such indication that will satisfy
you. Note that I am not a Python contributor of any kind. Also note that I
figured this out about two days ago. You can wheedle these guys all you
want, but they are too experienced for you to extract a promise from them.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

Skip Montanaro <[EMAIL PROTECTED]> writes:
> Because good requirements specification is difficult and testing improves
> the breed.  Better to have the major API changes and bugs taken care of, and
> to have its popularity demonstrated *before* it gets into the Python
> distribution.  The best way to do something isn't always obvious at the
> outset.

That is sometimes true, but not always.  Sometimes the best way is
obvious.

> If multiple solutions exist in the community, everyone benefits.
> The best survive.  The worst don't.

What's needed here "good enough", not necessarily "the best
conceivable", a level of perfection which is perhaps unattainable.

> If one implementation is prematurely chosen for inclusion in the
> Python distribution, it stifles the vetting process.

There have been exactly zero alternative implementations proposed.

> Finally, what if, saints be preserved, your whizbang new module is

It is not a whizbang module.  It is a stripped-down, basic
implementation of a well-accepted set of standards that are being used
in thousands of other applications in other languages.

> included in the core distribution and it's just not as popular as
> was first thought?

There is demand for it.  Look at how this thread started: some crypto
user noticed that rotor was gone and wanted to know what to do instead.

The issue of whether there's enough desire for a crypto module to
warrant including one in the stdlib was decided a long time ago.  The
proof of that somebody decided to accept the rotor module into the
distro.

That decision, that enough users want crypto to make having a stdlib
module worthwhile, has never been reversed as far as I know.  Rather,
the specific module (rotor) was withdrawn because it turned out not to
do the job on technical grounds.  And the decision not to replace it
was based on new legal and political concerns that outweighed the
technical side, not on thinking that there wouldn't be enough users.

> It just winds up as a weight around the maintainers' necks.  Once
> included in the distribution it's difficult to remove.

The rotor module is gone (and good riddance).  That's how this thread
started, remember?  It shows that bogus modules can be removed.  

Have you ever used a crypto library in a serious way?  Which ones have
you used?  The python-crypto folks (including myself) have used quite
a few of them, and after a while one realizes that the usable ones all
do pretty much the same stuff.  The one we're talking about does about
the same stuff that the rest of them do.  It does the same stuff
listed in PEP 272, although it's organized a little bit differently in
order to make supplying multiple cipher primitives more convenient,
which the application that drove PEP 272 apparently didn't need to do.
Andrew (the author of PEP 272) has said that the different
organization is fine.  The different organization is similar to what's
done internally in OpenSSL, in Peter Gutmann's cryptlib, in GnuPG, and
various other successful programs.  It's not radical or experimental.
It's old hat and low risk.  It provides functionality that's missing
from the stdlib.

> Even rexec and Bastion, which are both known to be inadequate from a
> security standpoint, are still in the distribution.

They should be removed pronto, just as rotor was removed.  Some other
ones should be removed too, like SmartCookie.

> So, feel free to implement whatever it is you propose.

I don't need your permission to do that.  What I'm looking for is an
indication that it's worth my while, before I spend the effort.

> Register it with PyPI, announce it on comp.lang.python.announce, etc.

I don't know how to do that, but the folks who care about it are
mostly on the python-crypto mailing list, and the module has been
discussed at length there and has been well-received.

> Support it for awhile.  Run it through a couple revisions to fix API
> warts and bugs.  

I already did that with the Python implementation, and did in fact
revise the API somewhat based on reviewer feedback.  The C
implementation would have the exact same API that the reviewers seem
to be happy with.

> When it's the category king and there is substantial community
> support for inclusion,

It's already the category king, because there are precisely zero other
entrants in the category.  If someone else wants to do one and can get
it into the stdlib, I will gladly use theirs and stop worrying about
mine.

> it will be considered.

Not according to how I currently interpret Guido's posts to python-dev
about it.  I read those as saying that no crypto module will be
considered for inclusion whether or not it's the category king,
because any such module might conflict with crypto regulations in some
countries.  No matter how much community support or technical
perfection it has, it's just won't be included, period.  So tell me
again what to do after writing and releasing a C module.  There's just
no reason to write one, if the module can'

Re: What's so funny? WAS Re: rotor replacement

[Skip Montanaro]

phr> I don't see why you can't make up your mind enough to issue simple
phr> statements like "the Python lib should have a module that does
phr> so-and-so, and it should meet such-and-such requirements, so if
phr> someone submits one that meets the requirements and passes code
phr> review and testing and doesn't have unexpected issues or otherwise
phr> fail to meet reasonable expectations, we'll use it".

Because good requirements specification is difficult and testing improves
the breed.  Better to have the major API changes and bugs taken care of, and
to have its popularity demonstrated *before* it gets into the Python
distribution.  The best way to do something isn't always obvious at the
outset.  If multiple solutions exist in the community, everyone benefits.
The best survive.  The worst don't.  If one implementation is prematurely
chosen for inclusion in the Python distribution, it stifles the vetting
process.  Finally, what if, saints be preserved, your whizbang new module is
included in the core distribution and it's just not as popular as was first
thought?  It just winds up as a weight around the maintainers' necks.  Once
included in the distribution it's difficult to remove.  Even rexec and
Bastion, which are both known to be inadequate from a security standpoint,
are still in the distribution.

So, feel free to implement whatever it is you propose.  Register it with
PyPI, announce it on comp.lang.python.announce, etc.  Support it for awhile.
Run it through a couple revisions to fix API warts and bugs.  When it's the
category king and there is substantial community support for inclusion, it
will be considered.

Python is popular in part because of its fairly conservative development
strategy.  That goes for the libraries as well as the language itself.

Skip
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> > That it's not appropriate for the
> > distro maintainers to look at the spec and the reference (pure Python)
> > implementatation and say "yes, we want this, go write the C version
> > and we'll include it after it's had some testing".
> 
> I know that I'm not going to give a blanket promise to include some
> code in the future. I can give blanket promises to *review* such
> code. Actually, assuming I find the time, I will review *any*
> code that is contributed.

I don't see why you can't make up your mind enough to issue simple
statements like "the Python lib should have a module that does
so-and-so, and it should meet such-and-such requirements, so if
someone submits one that meets the requirements and passes code review
and testing and doesn't have unexpected issues or otherwise fail to
meet reasonable expectations, we'll use it".

> Sure. That is because O'Reilly is willing to take a financial risk
> of failure of a book product. I'm not willing to take similar risks
> for Python code (risk of having to redesign the interface, 

Again, we're talking about straightforward modules whose basic
interface needs are obvious.  And interfaces in stdlib do get extended
from version to version all the time, if users turn out to need
additional features beyond the obvious basics.

> fix bugs in the implementation, 

Obviously there must be testing and review before inclusion.
Acceptance is contingent on the module passing tests and review.

> or have the submitter run away after the contribution).

There is no way to know in advance whether that's going to happen.  A
lot of work on the ANSI X9 crypto standards came to a screeching halt
a few years ago when one of the more prolific contributors tripped
over his vacuum cleaner cord, fell down the stairs, and was killed.
So if you have to be absolutely sure that the submitter will always be
around, you can never accept anything.  I think you mostly have to go
by how maintainable the code looks and how much maintainance you think
it will actually need and how many people you think are around who can
take care of it when needed.  And I do believe that experienced
programmers are capable of making reasonable judgements about those
questions, so they should not refuse to ever make such judgements.

> > Similarly, look at how the whole PEP process works.  There are lots of
> > times when a PEP has been accepted before any working code is
> > distributed.
> 
> Indeed. For new language features, it is more difficult to try them out
> in advance than for new library API.

I don't see why that should be.  Nothing stops anyone from
implementing and testing a language feature before standardizing it.
It will get even easier with PyPy, which is perhaps a reason to ban
further language changes until PyPy is deployed.

> Taking it to the extreme misses the point. I'm asking for field
> testing for new modules - not for all changes.

"Field testing" to most people means testing that a specific
implementation is reliable enough for inclusion.  It is the final step
in a normal process of declaring a feature ready for deployment, not
the initial step:

  1) think about whether you want the feature
  2) decide you want it
  3) implement
  4) field test (this naturally recognizes the possibility of
 reversing step 2, if something unexpectedly goes wrong in testing
 that's not easily repaired, but step 2 declares the basic
 intention for what should happen after a successful test).
  5) deploy

You wanted much more than for step 4 to always happen before step 5,
which is reasonable.  You claimed step 4 should always happen before
step 1, which is silly.

> > That's bizarre and abnormal as a development process.  What kind of
> > development process in industry doesn't decide whether to include a
> > feature, until after the feature is completely implemented at a
> > production scale?
> 
> Any high-quality standardization process. Standards in IETF and OMG
> are only accepted after implementations have been available.

I don't know what OMG is, but there is no IETF requirement that any
implementations be available in any particular language.  There are
also plenty of instances where the IETF decides that it wants
something to be standardized (e.g. IPSEC) so it invites a working
group to develop a specification.  The WG members then spend a lot of
time in meetings and discussions reaching a consensus on what the spec
should say.  They are willing to spend that time because the IETF has
already given them reasonable expectation that their end result will
actually be used.  The IETF doesn't say "go develop a complete
standard and implementation and put it in the field for a year before
we [the IETF] will even think about whether we want to standardize
it".  They are capable of announcing in advance that they want to
standardize something.  I don't see why the Python folks have to be
incapable of ever doing the same.

For the mod

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

[EMAIL PROTECTED] wrote:
That it's not appropriate for the
distro maintainers to look at the spec and the reference (pure Python)
implementatation and say "yes, we want this, go write the C version
and we'll include it after it's had some testing".  
I know that I'm not going to give a blanket promise to include some
code in the future. I can give blanket promises to *review* such
code. Actually, assuming I find the time, I will review *any*
code that is contributed.
When the inestimable Alexbot went to O'Reilly to pitch "Python in a
Nutshell", I'm sure they didn't tell him "go write the whole book from
cover to cover, circulate it for at least a year and get it thoroughly
reviewed, and if enough people recommend it, then and only then will
we begin to think about distributing it ourselves".
Sure. That is because O'Reilly is willing to take a financial risk
of failure of a book product. I'm not willing to take similar risks
for Python code (risk of having to redesign the interface, fix bugs
in the implementation, or have the submitter run away after the
contribution).
Similarly, look at how the whole PEP process works.  There are lots of
times when a PEP has been accepted before any working code is
distributed.
Indeed. For new language features, it is more difficult to try them out
in advance than for new library API. As a result, flaws of the new 
feature are often only fully understood years after the new feature
first gets released. For an example, consider the buffer interface.

To take your view to an extreme, no project should even have a task
list of desired features that people are invited to implement.
Taking it to the extreme misses the point. I'm asking for field testing
for new modules - not for all changes.
That's bizarre and abnormal as a development process.  What kind of
development process in industry doesn't decide whether to include a
feature, until after the feature is completely implemented at a
production scale?
Any high-quality standardization process. Standards in IETF and OMG
are only accepted after implementations have been available.
You seem to have the attitude that since volunteer development effort
doesn't consume actual PSF funds, the volunteer effort is worth
nothing and can be expended arbitrarily.  The volunteers may not feel
that way.
The volunteers are free to work on whatever they please. If you chose
not to write an AES module - that's fine with me. Still, people do
contribute to Python, and they do so without asking for permission
first. Typically, they develop the code because it solves their own
needs - then it doesn't really matter whether it also solves the
needs of others.
In either case, the user would best use the pre-compiled binary that
somebody else provided for the platform. 

Who's the "somebody else"?
Some user. I find that users contribute binaries for all kinds of
platforms for the code I publish. This is how open source works.
Do you do that
with your own modules, and still say that it's easy?
I publish or link to binaries of my code that others have created,
and find no problems in doing so.
Unless your requirement is different than what you say it is, I do see
what it is, and I'm saying it's better to do what normal projects do
and what Python has done in the past.  That is, it's perfectly ok to
decide to do something and then do it, rather than insisting,
bizarrely, that it always be the other way around.
No, it's three steps
1. decide that you want to do it
2. do it
3. decide whether you are pleased with the result, and only
   use it if you are
IOW, there should not be a blanket guarantee to use it after
step 1.
I think that question isn't the right one.  We need to ask how many
users the sha module was required to have, before Greg and Andrew
could have reasonable confidence that the sha module would go into the
core once it was tested enough and shown to be reliable.
They did not have *any* guarantee until they asked. I guess when they
asked it was accepted immediately.
I suspect
they were able to have that confidence long before testing was
complete, and maybe before implementation even started.
I'm pretty certain that you are wrong with that assumption.
Again, we would have to ask - but I would not be surprised if
AMK started implementing the module without even *considering*
a later inclusion in the Python core at that time. He has done
so on many occasions (include PyXML, which I inherited from
him).
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> There isn't. The interface might be beautifully designed, and you might
> claim it is, and I would *still* require that the module gets field
> testing before being incorporated into Python.

Yes, of course, that's completely reasonable.

> If other people start attesting that the module is beatifully
> designed, and should be included in the Python core - *then* it is
> worth looking into inclusion.

You said something more than that.  You said it's not worth even
thinking about inclusion until the finished module is actually in
distribution outside the core.  That it's not appropriate for the
distro maintainers to look at the spec and the reference (pure Python)
implementatation and say "yes, we want this, go write the C version
and we'll include it after it's had some testing".  I think that's
silly.  (Anyway I don't think "beautifully designed" needs to be a
requirement, and I don't claim to have done a beautiful design.  "Gets
the job done straightforwardly" is all that's needed and is all that
I've aimed for).

When the inestimable Alexbot went to O'Reilly to pitch "Python in a
Nutshell", I'm sure they didn't tell him "go write the whole book from
cover to cover, circulate it for at least a year and get it thoroughly
reviewed, and if enough people recommend it, then and only then will
we begin to think about distributing it ourselves".  More normally,
he'd have given them a proposal with an outline and maybe a sample
chapter and they used that to decide to give him a go-ahead.  Usually
at that point, they'd even also pay him some money in advance to do
the rest of the work with, and promise to pay him a cancellation fee
if they changed their mind about the book.

I certainly never asked for anything like that from the Python guys
about the crypto module.  I did ask for, and receive, a "this looks
good, we want it".  I later heard that they backed off because of
legal concerns, so I put the task aside.

Similarly, look at how the whole PEP process works.  There are lots of
times when a PEP has been accepted before any working code is
distributed.

To take your view to an extreme, no project should even have a task
list of desired features that people are invited to implement.
There's been numerous times when I've written something for the GNU
project because it's been on a list like that, and it's been accepted
after checking that it does what it's supposed to.

> > I'm happy to have that kind of testing (and I requested it), given
> > that the goal is inclusion in the core, and the core developers have
> > told me (as they did) that the proposal looks good and they'd like to
> > have the module, so I can reasonably expect it to go into the core if
> > it meets its technical expectations.
> 
> Not if I have a say in it. *Any* new module should see out-of-the-core
> distribution first (unless there is BDFL pronouncement to include it,
> of course).
> 
> This really is a matter of development process, not of technical
> quality.

That's bizarre and abnormal as a development process.  What kind of
development process in industry doesn't decide whether to include a
feature, until after the feature is completely implemented at a
production scale?  They figure out what the features should be, and
write them into a specification.  The specification defines what
features will be included.  Then they implement the features.  (I'm
talking specificially about straightforward, well-defined features,
not blue-sky or experimental concepts).

You see, they decide in advance what features they're going to
include, because development consumes resources and they don't want to
burn resources to develop something unless they're pretty sure they're
going to use the result once it's done.  (Of course it doesn't always
work out that way.  Everyone has worked on projects that have gotten
cancelled before delivery.  However that by definition means that
something has gone wrong.  No engineer likes it when that happens.)

You seem to have the attitude that since volunteer development effort
doesn't consume actual PSF funds, the volunteer effort is worth
nothing and can be expended arbitrarily.  The volunteers may not feel
that way.

> > If the developers instead say (as they seemed to somewhat later) that
> > because of legal/political concerns, there's no way the module can
> > possibly go into the core no matter how good it is technically, then
> > my motivation for writing the module dries up quite a bit.
> 
> I personally would not say that, although I can imagine that some people
> do say that, and I would also defend an inclusion, and push compliance
> the BXA requirements so we can legally export Python out of the U.S.A.

The BXA notification on the US side is pretty easy to handle, as you
know.  If you look at those Python-dev messages, the concern was about
importing Python-with-crypto into countries like Holland, not about
exporting it from the USA.  Those concerns are

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

[EMAIL PROTECTED] wrote:
I hadn't thought there was any controversy over the technical side of
this.
There isn't. The interface might be beautifully designed, and you might
claim it is, and I would *still* require that the module gets field
testing before being incorporated into Python. If other people start
attesting that the module is beatifully designed, and should be included
in the Python core - *then* it is worth looking into inclusion.
I'm happy to have that kind of testing (and I requested it), given
that the goal is inclusion in the core, and the core developers have
told me (as they did) that the proposal looks good and they'd like to
have the module, so I can reasonably expect it to go into the core if
it meets its technical expectations.
Not if I have a say in it. *Any* new module should see out-of-the-core
distribution first (unless there is BDFL pronouncement to include it,
of course).
This really is a matter of development process, not of technical
quality.
If the developers instead say (as they seemed to somewhat later) that
because of legal/political concerns, there's no way the module can
possibly go into the core no matter how good it is technically, then
my motivation for writing the module dries up quite a bit.
I personally would not say that, although I can imagine that some people
do say that, and I would also defend an inclusion, and push compliance
the BXA requirements so we can legally export Python out of the
U.S.A.
Evidently not always.  And how would the CGI user create a binary
anyway, even given a way to install it, if the web hosting service is
using a platform that the CGI user doesn't have a compiler for?  Think
of a Mac user whose web host runs Windows, or vice versa.
In either case, the user would best use the pre-compiled binary that
somebody else provided for the platform. Actually, the Windows user
using an OS X CGI server can probably just invoke the gcc which is
on the target system, anyway.
See, this is the critical point: "commonly-used functions", not
"functions I believe would be commonly used". You must have
*existing* users for a function to be commonly-used.

You're going around in circles.
No, I'm merely repeating myself, and rephrasing each time.
I have to, because apparently you don't see what my requirement
is.
They have few Python users because the functions aren't available in
Python.  To fix that, they must be added to Python.  How many users
do you think the Python sha module had before it went into Python?
The original source code of the SHA-1 implementation is the NIST code
(Gutmann, then Hollerbach), so I guess that had hundreds of users before
the module was contributed to Python. The module itself (including the
API) was written by Greg Stein and Andrew Kuchling. I believe (without
being able to verify) that they distributed this module for quite some
time, before contributing it to Python. We would have to ask them
how many users they had until they felt confident to contribute the
code.
Regards,
Martin
Andrew Kuchling
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Fuzzyman]

I've already downloaded p3 - thanks :-)

I wonder how long it took (in reality) an average hacker to break the
algorithm used by rotor ?
Regards,

Fuzzy
http://www.voidspace.org.uk/python/index.shtml

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Robert Kern]

Fredrik Lundh wrote:
<[EMAIL PROTECTED]> wrote:

The likely-best-known Python application in the world (probably more
people have heard of it than have heard of Python) originally had
crypto

and what Python application is that?  I can think of quite a few applications
written in Python that are widely known, but none of these are distributed as
Python code to end users.
BitTorrent, probably.
I believe that a very high percentage of users get the BitTorrent apps 
as standalone executables, not as a Python package where one has to go 
download Python first. The ones that do get it as Python source probably 
get it from their Linux/FreeBSD/other distribution where such 
dependencies are also taken care of for them.

--
Robert Kern
[EMAIL PROTECTED]
"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Fredrik Lundh]

<[EMAIL PROTECTED]> wrote:

> The likely-best-known Python application in the world (probably more
> people have heard of it than have heard of Python) originally had
> crypto

and what Python application is that?  I can think of quite a few applications
written in Python that are widely known, but none of these are distributed as
Python code to end users.

> I do know that its author wanted an AES module in the core
> to use for that application, in order to not have to distribute a C
> extension

 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Fredrik Lundh]

Terry Hancock wrote:

> And, well, I'm sorry Mr. Lundh, but your PIL module actually is something
> of a pain to install still.  The OP is right about that.  Usually worth it, 
> but I don't
> like the fact that that pain is being passed on to the end-user of my 
> software.
>
> The fact that you got all snide and refused to work with me when I asked
> for a copy of the autoconf sources to try to fix the problem didn't make me
> too happy, either.

ah, you're the guy who flamed me and called me names because PIL didn't
fit some hypothetical GNU-inspired definition of "open source software."  it's
always sad when people have to attack those who don't share their religion,
but it's not that much I can do about that.

>  So, naturally, I just dropped it.

which means that you have no idea how easy or hard it is to install PIL today
(hint: things have changed)

 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

[Again I'm having news server trouble and made a previous attempt to
post this, so sorry if you see it twice.  This version is edited
somewhat from the previous.]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> This is not possible - whether the module is included in Python or not.
> People *will* have to download something besides your application,
> namely Python - and in the version your require, too. If they cannot
> compile Python themselves, they will have to find somebody who did that
> for them.

Sorry, the presumption is that they already have Python installed.

> > So, if your modules are generally useful, I hope you
> > will try to get them into the core.
> 
> Right - the critical condition is "if the modules are generally
> useful". I cannot possibly know whether they are generally useful
> until a general public has commented that they are.

There is considerable demand for a crypto module.  Note how this thread
started: someone who had been using the rotor module complained that it's
now missing.

> Hmm. Most applications don't have any crypto needs. Those that do
> typically need TLS, or secure password hashing (crypt(3) specifically).
> So I somehow doubt that an AES module would be generally useful.

I believe it's feasible to do TLS in pure Python with acceptable
performance, given a DES module and the existing sha module.  I think
Trevor Perrin has already written something like that.  It's feasible
because outside of bulk crypto operations using DES and SHA, TLS's
execution time is dominated by public-key math calculations, which are
acceptably fast using Python's built-in longs.  The rest of the stuff
like certificate parsing happens just once per session and doing it in
Python isn't too awful.  On the other hand, it's impossible to do a
general purpose TLS stack acceptably without a DES module.  According
to RFC 2246 sec. 9, supporting TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA is
mandatory unless the application profile specifies otherwise.  SHA is
already in the core and DHE/DSS can be done with Python longs, so DES
is the missing critical piece.

Also, since DES is in the process of being decertified and AES is its
replacement, supporting DES without supporting AES is silly.  GnuPG
for example uses AES by default, so writing an interoperating Python
application (I have some parts of one working) needs AES.

If by crypt(3) you mean the old thing of DES with the modified E
table, I don't think it's used much any more.  I don't see any reason
against adding it if people want it, but it's not an encryption
function, it's an authentication function, and shouldn't be subject to
the legal concerns of a general purpose encryption module.

I do think it would be nice to have TLS in the core someday, just like
it's already in the Java distro (JSSE).  However, that's much more
ambitious and anyway it would rely on the block cipher module, so the
block cipher module is needed anyway.  Meanwhile, a pure-Python TLS
implementation outside the core is ok as a workaround.  Getting more
pure Python modules into the core is in general less valuable than
getting more C modules into the core.

> For an AES module with an interface nobody has ever used but yourself,
> it would be very difficult to argue that the module is generally useful.

Eh?  Lots of people have asked for general purpose crypto modules, and
the interface follows the approach used by any number of existing
libraries.  It just does exposes the simplest possible interface to
the block cipher and then implements the FIPS operating modes in a
natural and obvious way.  I've used a number of other such libraries
and am pretty familiar with what they need to do.  The main thing it
tries to do differently from PEP 272 is separate the block ciphers
from the FIPS modes, so that each new cipher module doesn't have to
implement all the different modes.

I hadn't thought there was any controversy over the technical side of
this.  There's the legal/political issue which I can gently suggest is
overblown, but since I'm not the one who would have to take the heat
if some government bigwig somewhere flipped their lid over crypto in
the Python distro, I don't feel entitled to be very vociferous about
this aspect.

> I'm primarily looking for the user testing. I require that this happens
> *outside* the Python distribution. Make a distutils package, and report
> the user feedback. Then propose inclusion into the core.

I'm happy to have that kind of testing (and I requested it), given
that the goal is inclusion in the core, and the core developers have
told me (as they did) that the proposal looks good and they'd like to
have the module, so I can reasonably expect it to go into the core if
it meets its technical expectations.

If the developers instead say (as they seemed to somewhat later) that
because of legal/political concerns, there's no way the module can
possibly go into the core no matter how good it is technically, then
my motivation for writing the mo

Re: What's so funny? WAS Re: rotor replacement

[phr]

James Stroud <[EMAIL PROTECTED]> writes:
> Applications that lack features force users to accept a limited feature
> set or they use an alternative program with other limitations. Putting
> the possibility for cryptographic storage increases the utility of any
> application that stores data, and it could be done without much work if
> it were properly included in the core distribution. I have found it
> amazing how few programs include encryption as an option. I believe this
> is because its hard for programmers to include it and/or they falsely
> reason that "if I don't need it, the user doesn't", which is up there
> with "if I can't see them, then they can't see me" in terms of bad logic.

The likely-best-known Python application in the world (probably more
people have heard of it than have heard of Python) originally had
crypto (an AES module distributed as a C extension with the beta test
versions of the app) but the crypto was brutally ripped out of the
application for the final release (you can still see scar tissue in
the code where the crypto was, i.e., there's still a function called
something like "encrypt_packet" which no longer actually encrypts the
packet).  I haven't found out exactly why the crypto was removed so I
can't be certain that it's because of cross-platform installation
headaches.  I do know that its author wanted an AES module in the core
to use for that application, in order to not have to distribute a C
extension, and he did some of the early work with me on writing such a
module to submit for that purpose.  The module I ended up proposing is
an offshoot of that effort.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[James Stroud]

I was purposefully making an illogical statement to illustrate the lapse
in reason of Martin's statement. Users have crypto needs, not
applications. Applications are presumably not anthropomorphic enough to
have needs--hence the lack of logic.

However, I am not an application (as far as you or I know) and thus can
not truly speak to the needs of applications in general, if indeed they
have any.

Applications that lack features force users to accept a limited feature
set or they use an alternative program with other limitations. Putting
the possibility for cryptographic storage increases the utility of any
application that stores data, and it could be done without much work if
it were properly included in the core distribution. I have found it
amazing how few programs include encryption as an option. I believe this
is because its hard for programmers to include it and/or they falsely
reason that "if I don't need it, the user doesn't", which is up there
with "if I can't see them, then they can't see me" in terms of bad
logic.

James

On Mon, 2005-01-24 at 17:17, James Stroud wrote:
> I was purposefully making an illogical statement to illustrate the lapse
> in reason of Martin's statement. Users have crypto needs, not
> applications. Applications are presumably not anthropomorphic enough to
> have needs--hence the lack of logic.
> 
> However, I am not an application (as far as you or I know) and thus can
> not truly speak to the needs of applications in general, if indeed they
> have any.
> 
> Applications that lack features force users to accept a limited feature
> set or they use an alternative program with other limitations. Putting
> the possibility for cryptographic storage increases the utility of any
> application that stores data, and it could be done without much work if
> it were properly included in the core distribution. I have found it
> amazing how few programs include encryption as an option. I believe this
> is because its hard for programmers to include it and/or they falsely
> reason that "if I don't need it, the user doesn't", which is up there
> with "if I can't see them, then they can't see me" in terms of bad
> logic.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Terry Hancock]

On Monday 24 January 2005 03:40 pm, Fredrik Lundh wrote:
> have you tried this, or are you just making things up to be able to continue
> the argument?  (hint: it doesn't work; python portability means that it's 
> fairly
> easy to write programs that run on multiple platforms, not that they will run
> on all available platforms without porting and testing).

I don't know, they come pretty close. Problems with portability in pure-python
modules have been almost non-existent for me.  I almost never test on the
Windows platform, but I know some of my code is being used by Windows
users.  And I guess I won't start, really, because I haven't heard any 
complaints
about it breaking.

I *did* have a few problems with Zope packages running on FreeBSD, bizarrely
enough, given that it's in the "POSIX" family of OSs, and therefore seems very
similar on the surface.

And, well, I'm sorry Mr. Lundh, but your PIL module actually is something
of a pain to install still.  The OP is right about that.  Usually worth it, but 
I don't
like the fact that that pain is being passed on to the end-user of my software.

The fact that you got all snide and refused to work with me when I asked
for a copy of the autoconf sources to try to fix the problem didn't make me
too happy, either.  So, naturally, I just dropped it. I'm in no position to
pick it up now -- just don't need it badly enough now (AFAIK it's unchanged
still -- lately I've been using the Debian package though, so I don't have to
worry about it).

I know that you don't want PIL to go into the core, of course, but I'm pretty
sure that problem would've needed fixing, if it were to be introduced.

Small problem, trivial to fix, but that's what bothers me about it.

Terry

--
Terry Hancock ( hancock at anansispaceworks.com )
Anansi Spaceworks  http://www.anansispaceworks.com

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Peter Hansen]

James Stroud wrote:
Martin v. Löwi said
Hmm. Most applications don't have any crypto needs.
Any program where one stores data would have crypto needs.
James, you must have mistyped the above, or your logic
is quite flawed.
I have written dozens of programs which store data, and
only a couple have had any crypto needs.  This alone
invalidates your statement as it stands.
What did you really mean to say?  Maybe "could" instead
of "would"?  That would make the statement true, yet
somewhat pointless, and it doesn't contradict what
Martin wrote in either case.
-Peter
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[James Stroud]

Martin v. Löwi said
> Hmm. Most applications don't have any crypto needs.

Any program where one stores data would have crypto needs.

Here are some examples: Database, wordprocessor, spreadsheet, address
book, mail program, (should I go on?). What would be the alternative to
encryption to satisfy the functionality encryption
provides...memorization? But I believe this may be a case of things
being so obvious people forget their importance (e.g. breathing).

I have written a class that goes something like this (highly simplified)

class encryptFilter:
   def __init__(acipher):
 self.mycipher = acipher
   def open(self, filename):
 self.myfile = open(filename,"w")
   def write(data):
 encrypted_data = self.encrypt(data)
 self.myfile.write(encrypted_data)
   etc...

Now what if acipher and this class could be made from part of the core
distro? Any application could have the option of encryption with only a
few lines of code:

import std_crypt

# other code here

if get_user_pref("do_encryption"):
  password = my_get_password()
  acipher = std_crypt.AES(password)
  outfile = std_crypt.encryptFilter(acipher)
  outfile.open(filename)
else:
  outfile = open(filename,"w")

# all other code doesn't change...
outfile.write(buncha_data)


This could be even more simpler, but that's the general idea. The
usefulness would be tremendous.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

[EMAIL PROTECTED] wrote:
Maybe we're not thinking about the same problems.  Say I'm an app
writer and I want to use one of your modules.  My development
environment is GNU/Linux, and I want to ship a self-contained app that
anyone can run without having to download additional components.  That
includes people wanting to run my app on Windows, Macintosh, Amiga,
and various other Python target platforms that I don't have compilers
for.  How do I do it?
This is not possible - whether the module is included in Python or not.
People *will* have to download something besides your application,
namely Python - and in the version your require, too. If they cannot
compile Python themselves, they will have to find somebody who did that
for them.
This is very easy if you are using Windows, Debian, or Mac OS X.
It is very difficult if you are using an Amiga, or any of the other
Python target platforms that you don't have compilers for. HP-UX
users of your application will have a hard time to get it running
even if the module was available as a built-in.
This does not change much if the module is available separately,
and it does not change much whether the separate module is written
in Python or in C: Windows, Debian, and OS X users will still
find precompiled binaries for the module, and all others will still
have to build it themselves. This is very easy, since all they have
to do is to use distutils.
So, if your modules are generally useful, I hope you
will try to get them into the core.
Right - the critical condition is "if the modules are generally
useful". I cannot possibly know whether they are generally useful
until a general public has commented that they are.
I can't speak for the authors but I'd personally say that none of
those old modules are really suitable for the core on technical
grounds.
That is my impression, too.
The module I'm discussing would simply implement the FIPS standard
block ciphers (AES and DES) and the FIPS operating modes.  These are
the basic building blocks that I feel are missing from the core.  They
should be enough to fill the crypto needs of most applications.
Hmm. Most applications don't have any crypto needs. Those that do
typically need TLS, or secure password hashing (crypt(3) specifically).
So I somehow doubt that an AES module would be generally useful.
So if the module was primarily written to be included in the core, I
would initially reject it for that very reason. After one year or so
in its life, and a recognizable user base, inclusion can be
considered.

That makes no sense; improving the core is a perfectly valid reason to
write something.  If someone wrote a straightforward patch that made
the Python interpreter 3x faster, would you still want to wait a year
before thinking about including it?
I would have to apply the "generally useful" test. For an 3x speed
improvement, about 100% of the Python users would find that useful.
For an AES module with an interface nobody has ever used but yourself,
it would be very difficult to argue that the module is generally useful.
If the module does not serve a need that people have, in general,
I don't see why it should be included (and yes, I wish a few of the
modules that were included recently wouldn't have been added so
quickly).
I certainly believe that every
contribution needs code review and user testing before entering wide
release, but that doesn't seem to be what you're getting at.
I'm primarily looking for the user testing. I require that this happens
*outside* the Python distribution. Make a distutils package, and report
the user feedback. Then propose inclusion into the core.
I release this "app" and presto, every Python user on every OS
platform can run it with no fuss.  That includes CGI authors who
aren't allowed to install C modules at all, without getting their
hosting service to do it.  How do I do the same thing if there's no
built-in sha module and all I have is sha.c?
It very much depends on the details of the target system. Yes,
it is a bit more tricky to install the additional library - but
that is the case with any additional library, whether it is C
or not.
For the CGI user who is not allowed to install binaries into
/usr/local, she can still install the module in her home
directory, no?
To me, the whole Python notion of "batteries included" expresses
precisely the idea that I'm trying to describe.  Commonly-used
functions belong in the core, because making people download or
install them separately is a pain in the neck.
See, this is the critical point: "commonly-used functions", not
"functions I believe would be commonly used". You must have
*existing* users for a function to be commonly-used.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

"Fredrik Lundh" <[EMAIL PROTECTED]> writes:
> please answer the question: have you done this?  what kind of programs
> have you successfully delivered as "self-contained apps" for use on arbi-
> trary platforms?

Here's a simple one:

   import sha
   name = raw_input('Enter your name: ')
   print 'Your name hashes to:', sha.new(name).hexdigest()

Maybe the sha module isn't in fact available on all platforms, but
it's definitely available on more platforms than I personally have
compilers for.  Maybe some more complicated app will hit some
non-sha-related platform dependency and present a porting problem.
But it's worse if it has additional porting problems caused by the sha
module.  Making the sha module available on multiple platforms is a
good thing regardless of what the rest of the app does.  Doing that
means one less porting problem to worry about.  So your question is an
irrelevant misdirection.

Anyway, I've written Python code that that's run under Linux and
Windows and the Macintosh without needing porting.  That's not the
same as testing on every platform Python runs on, but if one of those
platforms fails other than from using a builtin module that's
documented as being platform specific, especially if the module is a
pure mathematically-defined algorithm like sha and not an OS-interfacing
module like threading or Tkinter, then I'd say the portability bug is
in Python or its library, not in my app.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Fredrik Lundh]

<[EMAIL PROTECTED]> wrote

> As an app writer I want to publish code that runs on multiple
> platforms without needing special attention from the end user, and
> preferably without needing special platform-specific attention from
> me.

please answer the question: have you done this?  what kind of programs
have you successfully delivered as "self-contained apps" for use on arbi-
trary platforms?

 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

"Fredrik Lundh" <[EMAIL PROTECTED]> writes:
> have you tried this, or are you just making things up to be able to
> continue the argument?  (hint: it doesn't work; python portability
> means that it's fairly easy to write programs that run on multiple
> platforms, not that they will run on all available platforms without
> porting and testing).

As an app writer I want to publish code that runs on multiple
platforms without needing special attention from the end user, and
preferably without needing special platform-specific attention from
me.  If you want to use the words "Python portability" to mean
something weaker than that, that's ok, but I'm trying to accomplish
something better.  I've written lots of Python code that uses the sha
module without my needing to worry about the target platform, so that
shows the goal can be met.  I'm aiming for an aes/des module that's as
convenient for app writers and end users as the sha module is.

So now I know what you think "Python portability" means.  What do you
think "batteries included" means?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Fredrik Lundh]

<[EMAIL PROTECTED]> wrote:

> Maybe we're not thinking about the same problems.  Say I'm an app
> writer and I want to use one of your modules.  My development
> environment is GNU/Linux, and I want to ship a self-contained app that
> anyone can run without having to download additional components.  That
> includes people wanting to run my app on Windows, Macintosh, Amiga,
> and various other Python target platforms that I don't have compilers
> for.  How do I do it?

have you tried this, or are you just making things up to be able to continue
the argument?  (hint: it doesn't work; python portability means that it's fairly
easy to write programs that run on multiple platforms, not that they will run
on all available platforms without porting and testing).

 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[phr]

[Note: this is a 2nd attempt at posting reply to Martin's message,
since the first one didn't reach the server.  It's a rewrite from memory
but says about the same thing as the other attempt.  --Paul]

"Martin v. Löwis" <[EMAIL PROTECTED]> writes:
> Paul Rubin wrote:
> > If he understood how Python is actually used, he'd understand that any
> > C module is a lot more useful in the core than out of it.
> 
> This is non-sense. I have been distributing C modules outside the
> core for quite some time now, and I found that the modules are quite
> useful. distutils makes it really easy for anybody to use them.

Maybe we're not thinking about the same problems.  Say I'm an app
writer and I want to use one of your modules.  My development
environment is GNU/Linux, and I want to ship a self-contained app that
anyone can run without having to download additional components.  That
includes people wanting to run my app on Windows, Macintosh, Amiga,
and various other Python target platforms that I don't have compilers
for.  How do I do it?

I'm sure your modules are excellent but as an app writer, I feel I
must try to avoid needing them, in favor of modules that are already
in the core, even if it means more work for me or worse functionality
in my app, just for the sake of reducing installation headaches for my
target audience.  So, if your modules are generally useful, I hope you
will try to get them into the core.

> > There are already tons of 3rd party crypto modules outside the
> > core, and the module I was writing wouldn't add anything useful to those.
> 
> Why do you think these are not part of the core? It's not because
> they contain crypto code, or because they had been rejected. They
> are primarily not included in Python because they have not been
> contributed to Python, yet.

I can't speak for the authors but I'd personally say that none of
those old modules are really suitable for the core on technical
grounds.  They're too fancy and specialized, or they depend on
external libraries like OpenSSL, or they're written mostly to support
some specific application and don't present a simple, general-purpose
interface like a core module should have.  Maybe the authors felt the
same way and chose not to submit them.  Or maybe the authors had other
reasons, such as licensing priorities that conflict with the Python
license.

The module I'm discussing would simply implement the FIPS standard
block ciphers (AES and DES) and the FIPS operating modes.  These are
the basic building blocks that I feel are missing from the core.  They
should be enough to fill the crypto needs of most applications.

> If they were contributed, a number of things still would need to
> be considered, e.g. what is the size of the code, including libraries,
> is the contributor really the author, is the code likely going
> to be maintained, and so on. However, it never got that far.

The module we're talking about wouldn't have had any of those problems.

> I know that *I* am very opposed to any contribution of a larger
> module that has not seen any real users, yet. 

We're not talking about a "larger module", we're talking about a
module that implements the basic AES and DES block ciphers (similar to
how the sha module implements the basic SHA-1 hash algorithm) and
wraps them with some well-defined FIPS operating modes (similar to how
the hmac module wraps the sha module to compute RFC 2104 hmac
authentication codes).  This stuff isn't rocket science.  It's just
straightforward implementation of modes and algorithms that go back to
the 1970's, are the subject of national and international standards,
and are already in use in thousands of non-Python applications all
over the world.

Also, I've already released a pure-Python implementation of the
interface, suitable for application testing but too slow for real use.
I tried rather hard to design a convenient and general interface that
I feel was an improvement over PEP 272.  I don't know if anyone except
me is using it, but several people including Andrew (author of PEP
272) have written favorably about it based on reading the
specification.  The spec was designed from the beginning to fill the
needs of the core.  If I were writing it as a non-core module I would
have included more stuff.

> So if the module was primarily written to be included in the core, I
> would initially reject it for that very reason. After one year or so
> in its life, and a recognizable user base, inclusion can be
> considered.

That makes no sense; improving the core is a perfectly valid reason to
write something.  If someone wrote a straightforward patch that made
the Python interpreter 3x faster, would you still want to wait a year
before thinking about including it?  I certainly believe that every
contribution needs code review and user testing before entering wide
release, but that doesn't seem to be what you're getting at.

> I fail to see the problem you seem to have with C modules. They are
> very easy to

Re: What's so funny? WAS Re: rotor replacement

[phr]

"Fuzzyman" <[EMAIL PROTECTED]> writes:
> I also feel the lack of a standard cryptography module in the core...
> even a *basic* one. At least rotor provided that, before it was deprecated.

Rotor was insecure and really shouldn't have been used.  If you need
crypto in pure Python, try this:

  http://www.nightsong.com/phr/crypto/p3.py

It's a nonstandard algorithm, but so was rotor.  Its security should
be much better than rotor's, and its speed should be tolerable for
most purposes.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Fuzzyman]

I also feel the lack of a standard cryptography module in the core...
even a *basic* one. At least rotor provided that, before it was
deprecated. I (along with many other python users) write CGIs where the
only extension modules that I can use are either pure python, or ones
my web hoster is willing to install. Luckily my hoster will install
modules without too much fuss - others aren't so lucky. Cryptography is
a pretty 'standard' requirement these days.. and IMHO ought to be in
the 'standard library'.

Without commenting (or reading properly) on this exchange... it does
seem that many of ''s contributions have been very bad tempered
recently. Hmmm.
Regards,

Fuzzy
http://www.voidspace.org.uk/python/index.shtml

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

["Martin v. Löwis"]

Paul Rubin wrote:
If he understood how Python is actually used, he'd understand that any
C module is a lot more useful in the core than out of it.  
This is non-sense. I have been distributing C modules outside
the core for quite some time now, and I found that the modules
are quite useful. distutils makes it really easy for anybody
to use them.
There are already tons of
3rd party crypto modules outside the core, and the module I was
writing wouldn't add anything useful to those.
Why do you think these are not part of the core? It's not because
they contain crypto code, or because they had been rejected. They
are primarily not included in Python because they have not been
contributed to Python, yet.
If they were contributed, a number of things still would need to
be considered, e.g. what is the size of the code, including libraries,
is the contributor really the author, is the code likely going
to be maintained, and so on. However, it never got that far.
Where does Frederik get
off lecturing me about wanting to get a module into the core, when
Guido had invited me to do precisely that with that very module?
I know that *I* am very opposed to any contribution of a larger
module that has not seen any real users, yet. So if the module
was primarily written to be included in the core, I would initially
reject it for that very reason. After one year or so in its life,
and a recognizable user base, inclusion can be considered.
I did release a replacement for the rotor module that's written in
Python, which means it's reasonably useable without being in the core.
However, while its security should be ok, to provide reasonable
performance it had to use a nonstandard algorithm and therefore isn't
good for interoperating with anything.  To be both acceptably fast and
interoperable with other applications, a C module is needed.
I fail to see the problem you seem to have with C modules. They are
very easy to use if written properly.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: What's so funny? WAS Re: rotor replacement

[Paul Rubin]

Brian van den Broek <[EMAIL PROTECTED]> writes:
> no Python expert, just a hobbyist. But, I think I can take this one on:
> 
> Fredrik's contributed a lot to Python. The Standard Library book,
> several well know tools, and, I'd wager a finger, a fair bit of code
> in the standard lib. I don't think the community gives you a name like
> F-bot, but that you know a wee bit about how Python is actually used, etc.

If he understood how Python is actually used, he'd understand that any
C module is a lot more useful in the core than out of it.  

His lecture to me about the crypto module was bizarre and
inappropriate.  The whole purpose of that module was to fix a
deficiency of Python, namely the lack of a good crypto module in the
core, that people could use without having to install any downloaded C
modules.  If it doesn't go into the core, the deficiency isn't fixed,
so the module has failed in its purpose.  There are already tons of
3rd party crypto modules outside the core, and the module I was
writing wouldn't add anything useful to those.  It only provided
AES/DES and the basic FIPS modes, and was designed specifically to be
suitable for the core where the other modules were considered too
fancy.  Also, Guido had expressed interest in having it, before
changing heart because of the legal stuff.  Where does Frederik get
off lecturing me about wanting to get a module into the core, when
Guido had invited me to do precisely that with that very module?

Frederik has written some pretty good C modules himself, that are a
pain in the neck to use because they're not in the core, which means
that before they can be used they first have to be compiled and
installed.  If he wanted to maximize their usefulness, he'd be trying
to get them into the core.  I don't know what his goals are though.

I did release a replacement for the rotor module that's written in
Python, which means it's reasonably useable without being in the core.
However, while its security should be ok, to provide reasonable
performance it had to use a nonstandard algorithm and therefore isn't
good for interoperating with anything.  To be both acceptably fast and
interoperable with other applications, a C module is needed.
-- 
http://mail.python.org/mailman/listinfo/python-list