Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 01:40:57PM +, David Wilson wrote:
> On Tue, Apr 12, 2016 at 11:12:27PM +1000, Steven D'Aprano wrote:
> > I can think of one possible threat. Suppose that the locale library
> > has a bug, so that calling "aardvark".isdigit seg faults, potentially
> > executing arbitrary C code, but at the very least crashing the
> > application. Is that the sort of attack you're concerned by?
> 
> This thread already covered the need to address SEGV at length. For a
> truly evil user, almost any kind of crash is an opportunity to take
> control of the system, and a security solution ignoring this is no
> security solution at all.

Indeed.

> But that's not what's happening, instead a dead horse is being flogged
> over a hundred messages in our inboxes and IMHO it is excruciating to
> watch.

I don't think that is true at all, and I personally I have found this
thread very interesting. I apologise if others have not.

> > Even if the only thing we learn from Jon's experiment is a new set of
> > tricks for breaking out of the sandbox, that's still interesting, if
> > not useful.
> 
> Don't forget the worst case: a fundamentally broken security module
> heavily marketed to the naive using claims the core team couldn't break
> it.

I should point out that my module is called "unsafe.py", is titled
an "experiment", and prominently states in the README:

  Do not use this code for any purpose in the real world.

I will not be putting it up as an installable package, and as already
stated it was never my intention to suggest that it or anything like
it be included in the stdlib. I will however leave it on github for
anyone who wants to have a go at breaking into it in the future.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 11:03:11PM +1000, Chris Angelico wrote:
> One of the key differences with scripts in web browsers is that there
> *is* no "outer environment" to access.

If you think that then I think you considerably misunderstand how
modern browsers work.

> Remember what I said about the difference between Python-in-Python
> sandboxing and, say, Lua-in-Python? One tiny exploit in
> Python-in-Python and you suddenly gain access to the entire outer
> environment, and it's game over. One tiny exploit in Lua-in-Python
> and you have whatever that exploit gave you, nothing more.

Are you imagining the Lua-in-Python as being completely isolated from
the Python namespace then?

> In fact, if you're prepared to forfeit almost all of Python's power to
> achieve security, you probably should look into embedding a JavaScript
> or Lua engine in your Python code.

Yes, I have in fact already done this (JavaScript using SpiderMonkey).
It allows the JavaScript to access Python objects and methods directly
from JavaScript so it doesn't actually help, but I think I could put
limits on that (e.g. making things read-only) and unlike most of this
Python stuff, that could be made a solid rule with no clever ways
around it.

> I suspect re is next on the chopping block (it has a global cache -
> if the outer system uses a regular expression more than once, it
> would potentially be possible to mess with it in the cache, and then
> next time it gets used, the injected code gets run),

All you could do would be to give misleading results from the regular
expression methods, but yes that is a good point. I regret that
I added the import stuff at all now - it has just been a distraction
from my original point.

> [1] And if anyone mentions PHP, I will set him to work on the hardest
> PHP problem I know of - no, not securing it. I mean convincing end
> users that it's not necessary. Securing it is trivial by comparison.

Fortunately I have managed to exclude PHP completely these days from
any system I have anything to do with!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 10:45:06PM +1000, Chris Angelico wrote:
> On Tue, Apr 12, 2016 at 10:42 PM, Jon Ribbens
> <jon+python-...@unequivocal.co.uk> wrote:
> > That's not a vulnerability, and it's something I already explicitly
> > mentioned - if you can get a function to return an object's __dict__
> > then you win. The question is: can you do that?
> 
> The question is, rather: Can you prove that we cannot?

I refer you to the answer given previously. Can you prove you cannot
write code to escape JavaScript sandboxes? No? Then why have you not
disabled JavaScript in your browser?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 02:16:57PM +0200, Victor Stinner wrote:
> I read your code and the code of CPython. I found many issues.

Thanks for your efforts.

> Your "safe import" hides real functions with a proxy. Ok. But the code
> of modules is still run in the real namespace,

Yes, that was the intention.

> I found functools.update_wrapper(). I was very surprised because this
> function calls getattr() and setattr(), whereas your sandbox replaces
> these builtin functions.

Good point. It seems it was almost certainly foolish of me to add
'import' back in in response to peoples' comments while my original
concept was still being discussed.

> So here you have:
> ---
> import functools

Thanks, that was pretty clever. I've of course fixed it by reducing
the list of imports (a lot, since I had really audited them at all).
But you make a good point.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 02:31:19PM +0200, Victor Stinner wrote:
> Oh, I forgot to mention another vulnerability: you block access to
> attributes by replacing getattr and by analyzing the AST. Ok, but one
> more time, it's not enough. If you get access to obj.__dict__, you
> will likely get access to any attribute using obj_dict[attr] instead
> of obj.attr.

That's not a vulnerability, and it's something I already explicitly
mentioned - if you can get a function to return an object's __dict__
then you win. The question is: can you do that?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 02:05:06PM +0200, Victor Stinner wrote:
> 2016-04-12 13:10 GMT+02:00 Jon Ribbens <jon+python-...@unequivocal.co.uk>:
> > No, it's a matter of reducing the whitelist. I must admit that
> > I don't understand in what way this is not already clear. Look:
> >
> >   >>> len(unsafe._SAFE_MODULES)
> >   23
> 
> You don't understand that even if the visible "Python scope", "Python
> namespace", or call it as you want (the code that is accessible from
> your sandbox) looks very tiny, the real effictive code is HUGE.

You are mistaken, I do understand that.

> In a few minutes, I found "{0.__class__}".format(obj) which is not a
> full escape of the sandbox, but it's just to give one example.

It's something I'd already thought of, and it's not an escape at all.

> > I could "mathematically prove" that there are no more security holes
> > in that list by reducing its length to zero.
> 
> You only see a very tiny portion of the real attack surface.

You've misunderstood my comment - I was saying that the security holes
from imported modules can be easily eliminated. That doesn't say
anything about security holes not from imported modules, of course.

> > The "minimum viable set" in my view would be: no builtins at all,
> > only allowing eval() not exec(), and disallowing yield [from],
> > lambdas and generator expressions.
> 
> IMHO it's a waste of time to try to reduce the great Python with
> battery included to a simple calculator to compute 1+2.

And in my opinion it isn't. There are plenty of use cases for such
a thing. Take a look at this for example:
https://developer.blender.org/D1862 

> It's very easy to implement your own calculator in pure Python, from
> the parser to the code to compute the operators. If you write yourself
> the whole code, it's much easier to control what is allowed and put
> limits. For example, with your own code, you can put limits on the
> maximum number, whereas your sandbox will kill your CPU and memory if
> you try 2**(2**100) (no builtin function required for this "exploit").

Yes, I'd already thought of that too, although if you allow functions
and methods to be called (which they are, in my minimal viable set
suggestion above) then I think perhaps you've not actually bought
yourself very much with all that work.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 01:38:09PM +0200, Maciej Fijalkowski wrote:
> Jon, let me reiterate. You asked people to break it (that's the title
> of the thread) and they did so almost immediately. Then you patched
> the thing and asked them to break it again and they did. Now the
> faulty assumption here is that this procedure, repeated enough times
> will produce a secure environment - this is not how security works,

That is not an accurate summary of what has happened so far,
nor am I making that assumption. You are misunderstanding the
purpose of the experiment - I am not sure how, as I have tried
to be quite clear.

The question is: with a minimal (or empty) set of builtins, and a
restriction on ast.Name and ast.Attribute nodes, can exec/eval be
made 'safe' so they cannot execute code outside the sandbox. The
answer appears to be "yes", if the restriction is "^f?_". (If you
additionally inject external objects to the namespace then they need
to be proxied and mro() prevented.)

> You can't do that just by asking on the mailing list and whacking
> all the examples.

If anyone had managed to find any more examples of holes in the
original featureset after the first couple then I would agree with
you, but they haven't. 

> As others pointed out, this particular approach (with maybe
> different details) has been tried again and again and again

This simply isn't true either. As far as I can see, only
RestrictedPython has tried anything remotely similar, and
to the best of my ability to determine, that project is not
considerd a failure.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 06:21:04AM -0400, Isaac Morland wrote:
> On Tue, 12 Apr 2016, Jon Ribbens wrote:
> >>This is still a massive game of whack-a-mole.
> >
> >No, it still isn't. If the names blacklist had to keep being extended
> >then you would be right, but that hasn't happened so far. Whitelists
> >by definition contain only a small, limited number of potential moles.
> >
> >The only thing you found above that even remotely approaches an
> >exploit is the decimal.getcontext() thing, and even that I don't
> >think you could use to do any code execution.
> 
> "I don't think"?
> 
> Where's the formal proof?

I disallowed the module completely, that's the proof.

> Without a proof, this is indeed just a game of whack-a-mole.

Almost no computer programs are ever "formally proved" to be secure.
None of those that run the global Internet are. I don't see why it
makes any sense to demand that my experiment be held to a massively
higher standard than the rest of the code everyone relies on every day.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 08:27:14PM +1000, Chris Angelico wrote:
> On Tue, Apr 12, 2016 at 8:06 PM, Jon Ribbens
> <jon+python-...@unequivocal.co.uk> wrote:
> > No, it still isn't. If the names blacklist had to keep being extended
> > then you would be right, but that hasn't happened so far. Whitelists
> > by definition contain only a small, limited number of potential moles.
> >
> > The only thing you found above that even remotely approaches an
> > exploit is the decimal.getcontext() thing, and even that I don't
> > think you could use to do any code execution.
> 
> decimal.getcontext is a simple and obvious example of a way that
> global mutable objects can be accessed across the boundary. There is
> no way to mathematically prove that there are no more, so it's still a
> matter of blacklisting.

No, it's a matter of reducing the whitelist. I must admit that
I don't understand in what way this is not already clear. Look:

  >>> len(unsafe._SAFE_MODULES)
  23

I could "mathematically prove" that there are no more security holes
in that list by reducing its length to zero. There are still plenty
of circumstances in which the experiment would be a useful tool even
with no modules allowed to be imported.

> I still think you need to work out a "minimum viable set" and set down
> some concrete rules: if any feature in this set has to be blacklisted
> in order to achieve security, the experiment has failed.

The "minimum viable set" in my view would be: no builtins at all,
only allowing eval() not exec(), and disallowing yield [from],
lambdas and generator expressions.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 06:57:37PM +1000, Chris Angelico wrote:
> And yes, you win if you get another module. Interestingly, you're
> allowed to import urllib.parse, but not urllib itself; but "import
> urllib.parse" makes urllib available - and, since modules inside
> modules are blacklisted, "urllib.parse" doesn't exist
> (AttributeError).

Yes, this is issue #3 on github. I'd need to spend a few minutes
thinking about how to make importing of submodules work out properly.

> You can access the decimal module, and call decimal.getcontext(). This
> returns the same default context object that the "outer" Python uses;

OK, decimal goes ;-)

> Even more curiously, you can "import fractions", but you don't get
> fractions.Fraction - though you *do* get fractions.Decimal.

That seems to be because Fraction inherits from numbers.Number,
which has a metaclass, so type(Fraction) is abc.ABCMeta not 'type'.
That's obviously not a security hole and may well be fixable.

> The sandbox code assumes that an attacker cannot create files in the
> current directory.

If the attacker can create such files then the system is already
compromised even if you're not using any sandboxing system, because
you won't be able to trust any normal imports from your own code.

> Setting LC_ALL and then working with calendar.LocaleTextCalendar()
> causes locale files to be read.

I don't think that has any obvious relevance. Doing "import enum"
causes "enum.py" to be read too, and that isn't a security hole.

> This is still a massive game of whack-a-mole.

No, it still isn't. If the names blacklist had to keep being extended
then you would be right, but that hasn't happened so far. Whitelists
by definition contain only a small, limited number of potential moles.

The only thing you found above that even remotely approaches an
exploit is the decimal.getcontext() thing, and even that I don't
think you could use to do any code execution.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 06:28:34PM +1000, Chris Angelico wrote:
> On Sat, Apr 9, 2016 at 12:18 AM, Jon Ribbens
> <jon+python-...@unequivocal.co.uk> wrote:
> > Anyway the code is at https://github.com/jribbens/unsafe
> > It requires Python 3.4 or later (it could probably be made to work on
> > Python 2.7 as well, but it would need some changes).
> 
> Rather annoying point: Your interactive mode allows no editing keys
> (readline etc), and also doesn't have underscore for "last result", as
> that's a forbidden name. :( Makes tinkering fiddly.

It's just a subclass of the stdlib class code.InteractiveConsole,
which seems not to offer those features unfortunately.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 01:08:36PM +1200, Greg Ewing wrote:
> Jon Ribbens wrote:
> >So far it looks like blocking "_*" and the frame object attributes
> >appears to be sufficient.
> 
> Even if your sandbox as it currently exists is secure, it's
> only an extremely restricted subset.

I'm not sure what you think the restrictions are, but yes a highly
restricted Python that was secure would be very useful sometimes.

> You seem to be assuming that if your technique works so far, then it
> can be extended to cover a larger subset, but I don't think that's
> certain.

No, I'm not assuming that.

> One problem that's been raised is how to prevent untrusted
> code from monkeypatching imported modules. Possibly that
> could be addressed by giving the untrusted code a copy of
> the module,

Yes, that's what it does.

> but I'm not entirely sure -- accidentally importing two copies of
> the same source file is a well-known source of bugs, after all.

I'm not sure what you mean by that.

> A related, but more difficult problem is that if we allow
> the untrusted code to import any pure-Python classes, it
> will be able to monkeypatch them. So it seems like it will
> need its own copy of those classes as well

Yes, that's also what it does.

> -- and having two copies of the same class around is *another* well
> known source of bugs.

I'm not sure what you mean by that either.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Tue, Apr 12, 2016 at 03:02:54AM +1000, Chris Angelico wrote:
> On Tue, Apr 12, 2016 at 2:53 AM, Jon Ribbens
> <jon+python-...@unequivocal.co.uk> wrote:
> > On Mon, Apr 11, 2016 at 04:04:21PM +0100, Paul Moore wrote:
> >> However, it's not at all clear (to me at least) what you *are* trying
> >> to do.
> >
> > I'm trying to see to what extent we can use ast node inspection to
> > remedy the failures of prior attempts at Python sandboxing. Is there
> > *any* extent to which Python can be sandboxed, or is even trying to
> > use it as a calculator function unfixably insecure?
> 
> It all depends on how much functionality you want. If all you need is
> a numeric expression evaluator, that's not too hard - disallow all
> forms of attribute access, etc, and just have simple numbers and
> operators. That's pretty useful, and safe.

By "calculator" I didn't necessarily mean to imply numeric-only,
sorry if I was unclear. Also perhaps I should have said "non-trivial",
inasmuch as if we restrict it that far then it would quite possibly be
simpler and quicker just to write the expression evaluator from scratch
and not use the Python interpreter at all.

> Alternatively, go completely the other way. Let people run whatever
> code they like... in an environment where it can't hurt anyone else.
> That's what PyPyJS does - don't bother looking for security holes in
> it, because all you're doing is attacking your own computer.

That's a very specific use case though: running client-side in the
user's browser.

> So before you can ask whether Python is unfixably insecure, you first
> have to decide what the minimum level of functionality is that you'll
> accept. Do you need basic arithmetic plus trignometric functions? Easy
> enough - disallow all attribute access and imports, and populate
> builtins with "from math import *". Need them to be able to assign
> variables and define functions? That's gonna be harder.

I think calling functions and accessing variables and attributes is
likely a minimum. Defining functions would be useful, and of course
defining classes would be another useful step further.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Mon, Apr 11, 2016 at 04:04:21PM +0100, Paul Moore wrote:
> However, it's not at all clear (to me at least) what you *are* trying
> to do.

I'm trying to see to what extent we can use ast node inspection to
remedy the failures of prior attempts at Python sandboxing. Is there
*any* extent to which Python can be sandboxed, or is even trying to
use it as a calculator function unfixably insecure?

> You're limiting the subset of Python that people can use,
> understood. And you're trying to ensure that people can't do "bad
> things". Again, understood. But what subset are you actually allowing,
> and what things are you trying to protect against? (For example, I
> can't calculate sin(1.2) using the math module - why is that not
> alllowed?

It wasn't allowed in the earlier version because I wasn't allowing
import at all, because this is just an experiment. As it happens,
I added 'import' yesterday so yes you can use math.sin.

> It feels at the moment as if I'm playing a game where I don't know the
> rules, and every time I think I scored a point, the rules are changed
> to retroactively disallow it.

The challenge is to show some code that will escape from the sandbox,
in a way that is not trivially fixable with a tiny patch, or in a way
that demonstrates that such a large number of tiny patches would be
required as to be unworkable.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Mon, Apr 11, 2016 at 08:35:11AM -0700, Nikolaus Rath wrote:
> On Apr 11 2016, Jon Ribbens <jon+python-...@unequivocal.co.uk> wrote:
> >> What I see is that you asked to break your sandbox, and less than 1
> >> hour later, a first vulnerability was found (exec called with two
> >> parameters). A few hours later, a second vulnerability was found
> >> (async generator and cr_frame).
> >
> > The former was just a stupid bug, it says nothing about the viability
> > of the methodology. The latter was a new feature in a Python version
> > later than I have ever used, and again does not imply anything much
> > about the viability.
> 
> It implies that new versions of Python may break your sandbox. That
> doesn't sound like a viable long-term solution.

That is obviously always going to be true of major new versions with
major new features, no matter what language we're talking about or
what method is being used to sandbox - unless the sandboxing were to
be built in to the language itself, which I have deliberately not
suggested.

But having said that, I already pointed out in the message you're
responding to that with the method I'm using now, coroutines would
not have been an issue even if I hadn't specifically fixed them.

> > I think now I've blocked the names of frame
> > object attributes it wouldn't be a vulnerability any more anyway.
> 
> It seems like you're playing whack-a-mole. 

Well, no, quite the opposite in fact. If that was true then I would
have given up already as the method having been proved useless.
So far it looks like blocking "_*" and the frame object attributes
appears to be sufficient.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Mon, Apr 11, 2016 at 11:40:05AM +0200, Victor Stinner wrote:
> 2016-04-10 18:43 GMT+02:00 Jon Ribbens <jon+python-...@unequivocal.co.uk>:
> > That's the opposite of my approach though - I'm starting small and
> > adding things, not starting with everything and removing stuff. Even
> > if what we end up with is an extremely restricted subset of Python,
> > there are still cases where that could be a useful tool to have.
> 
> You design rely on the assumption that CPython is only pure Python.

No it doesn't. Obviously I know CPython is written in C - the clue is
in the name. I'm not sure what you mean here. 

> It means that your protections like hiding builtin functions from the
> Python scope don't work. If an attacker gets access to a C function
> giving access to the hidden builtin, the game is over.

The former is only true if you assume the latter is possible.
Is there any reason to believe it is?

> It's hard to list all features of the C code which are indirectly
> accessible from the Python scope. Some examples: warnings and
> tracebacks. These features killed the pysandbox project because they
> open directly files on the filesystem, it's not possible to control
> these features from the Python scope.

I think what you're referring to is when they show context for errors,
for which they try and find the source code lines to display by
identifying the filename, and you can subvert that process by changing
__file__ and/or __name__. If so, you can't do that within my
experiment because you're not allowed to access either of those names.

> Another example which exposes a vulnerability of your sandbox:
> str.format() gets directly object attributes without the getattr()
> builtin function, so it's possible to escape your sandbox. Example:
> "{0.__class__}".format(obj) shows the type of an object.

Yes, I'd thought of that. However getting access to a string which
contains the name or a representation of an object is not at all the
same thing as getting access to the object itself. 

> Think also about the new f-string which allows arbitrary Python
> code: f"{code}".

Obviously I can't speak to features of future versions of Python.
I'd have to see the ast generated by an f-string to know if they
pose a problem or not, but I suspect they would compile to
expression nodes and hence be caught by the existing checks.

> > However on the other hand, nobody has tried before to do what I am
> > doing (static code analysis),
> 
> You're wrong.
> 
> Zope Security ("RestrictedPython") has a similar design. Analyzing AST
> is a common design to build a sanbox. But it's not safe.

Ah, I hadn't seen that one. Yes, they are doing something similar
(but also much more complex!) I don't know why you say this is
a "common design" though, that one is the only one that appears to
use it.

> What I see is that you asked to break your sandbox, and less than 1
> hour later, a first vulnerability was found (exec called with two
> parameters). A few hours later, a second vulnerability was found
> (async generator and cr_frame).

The former was just a stupid bug, it says nothing about the viability
of the methodology. The latter was a new feature in a Python version
later than I have ever used, and again does not imply anything much
about the viability. I think now I've blocked the names of frame
object attributes it wouldn't be a vulnerability any more anyway.

> By the way, are you sure that you fixed the vulnerability? You
> blacklisted "cb_frame", not cr_frame ;-)

Ah, thanks. As above, I think this doesn't actually make any
difference, but I've updated the code regardless.

> You should look closer, pysandbox is very close to you project.

I've just looked through it all again, and I don't understand why you
are saying that. It's nothing like my experiment. It's trying to alter
the global Python environment so that arbitrary code can be executed,
whereas I am not even trying to allow execution of arbitrary code and
am not altering the global environment.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Sun, Apr 10, 2016 at 02:08:16PM -0700, Nikolaus Rath wrote:
> On Apr 10 2016, Jon Ribbens <jon+python-...@unequivocal.co.uk> wrote:
> > On Sat, Apr 09, 2016 at 02:43:19PM +0200, Victor Stinner wrote:
> > That's the opposite of my approach though - I'm starting small and
> > adding things, not starting with everything and removing stuff.
> 
> That contradicts what you said in another mail:
> 
> On Apr 08 2016, Jon Ribbens <jon+python-...@unequivocal.co.uk> wrote:
> > Ah, I've not used Python 3.5, and I can't find any documentation on
> > this cr_frame business, but I've added cr_frame and f_back to the
> > disallowed attributes list.

No, you've just misunderstood my meaning. Obviously I'm not only
allowing access to whitelisted variable and property names, that
would be ridiculous ("your code may only use variables called
'foo', 'bar' and 'baz'...").

The point is that we can start with, say, only allowing expressions
and not statements, and a __builtins__ that contains literally
nothing. We can even limit ourselves to disallow, say, lambda and
yield and generator expressions if we like. Can this minimal
language be made "safe"? If so, we have already won something - the
ability to use "eval" as a powerful calculator function. Then, can
we allow statements? Can we allow user-defined classes? Can we allow
try/catch? etc.

With regard to names by the way, I suspect that disallowing just
anything starting "_" and the names of the properties of frame
objects would be good enough. Unless someone knows a way to get
to an object's __dict__ or its type without using vars() or type()
or underscore attributes...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Mon, Apr 11, 2016 at 12:07:48AM +0300, Serhiy Storchaka wrote:
> On 10.04.16 19:51, Jon Ribbens wrote:
> >On Sun, Apr 10, 2016 at 02:51:23PM +1000, Nick Coghlan wrote:
> >>On 9 April 2016 at 22:43, Victor Stinner <victor.stin...@gmail.com> wrote:
> >>>See pysandbox test suite for a lot of ways to escape a sandbox. CPython has
> >>>a list of know code to crash CPython (I don't recall the dieectory in
> >>>sources), even with the latest version of CPython.
> >>
> >>They're at https://hg.python.org/cpython/file/tip/Lib/test/crashers
> >
> >Thanks. I take your point that sandboxing Python requires CPython to
> >free of code execution bugs. However I will note that none of the
> >crashers in that directory will work inside my experiment (except
> >"infinite_loop_re.py", which isn't a crasher just a long loop).
> 
> Try following example:
> 
> it = iter([1])
> for i in range(100):
> it = filter(None, it)
> next(it)

That does indeed segfault. I guess you should report that as a bug!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Sun, Apr 10, 2016 at 02:51:23PM +1000, Nick Coghlan wrote:
> On 9 April 2016 at 22:43, Victor Stinner  wrote:
> > See pysandbox test suite for a lot of ways to escape a sandbox. CPython has
> > a list of know code to crash CPython (I don't recall the dieectory in
> > sources), even with the latest version of CPython.
> 
> They're at https://hg.python.org/cpython/file/tip/Lib/test/crashers

Thanks. I take your point that sandboxing Python requires CPython to
free of code execution bugs. However I will note that none of the
crashers in that directory will work inside my experiment (except
"infinite_loop_re.py", which isn't a crasher just a long loop).

> Even without those considerations though, there are system level
> denial of service attacks that untrusted code can perform without even
> trying to break out of the sandbox - the most naive is "while 1:
> pass", but there are more interesting ones like "from itertools import
> count; sum(count())", or even "sum(iter(int, 1))" and "list(iter(int,
> 1))".

Yes, of course. I have already explicitly noted that infinite loops
and memory exhausation are not preventable.

> Operating system level security sandboxes still aren't particularly
> easy to use correctly, but they're a lot more reliable than language
> runtime level sandboxes, can be used to defend against many more
> attack vectors, and even offer increased flexibility (e.g. "can write
> to these directories, but no others", "can read these files, but no
> others", "can contact these IP addresses, but no others").

I don't entirely trust operating system sandboxes either - I generally
assume that if someone can execute arbitrary code on my machine, then
they can do anything they want to that machine.

What I *might* trust, though, would be a "sandbox Python" that is
itself running inside an operating system sandbox...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Sat, Apr 09, 2016 at 02:43:19PM +0200, Victor Stinner wrote:
>Please don't loose time trying yet another sandbox inside CPython. It's
>just a waste of time. It's broken by design.
> 
>Please read my email about my attempt (pysandbox):
>https://lwn.net/Articles/574323/
> 
>And the LWN article:
>https://lwn.net/Articles/574215/
> 
>There are a lot of safe ways to run CPython inside a sandbox (and not rhe
>opposite).
> 
>I started as you, add more and more things to a blacklist, but it doesn't
>work.

That's the opposite of my approach though - I'm starting small and
adding things, not starting with everything and removing stuff. Even
if what we end up with is an extremely restricted subset of Python,
there are still cases where that could be a useful tool to have.

I've read your links above, and indeed everything I can find written
by anyone about historical attempts to sandbox Python. I'm aware that
others have tried and failed at this in the past, so it's certainly
true that there is room for suspicion that this simply cannot be done.

However on the other hand, nobody has tried before to do what I am
doing (static code analysis), so it's not necessarily a safe
assumption that the idea is doomed. For example, as far as I can see,
none of the methods used to break out of your pysandbox would work to
break out of my experiment.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Sat, Apr 09, 2016 at 02:20:49AM +1000, Chris Angelico wrote:
> On Sat, Apr 9, 2016 at 12:18 AM, Jon Ribbens
> <jon+python-...@unequivocal.co.uk> wrote:
> > Anyway the code is at https://github.com/jribbens/unsafe
> > It requires Python 3.4 or later (it could probably be made to work on
> > Python 2.7 as well, but it would need some changes).
> 
> Not being a security expert, I'm not the best one to try to break it
> maliciously; but I can break things accidentally. Pull request sent
> through. :)

Thanks, I've merged that in.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Fri, Apr 08, 2016 at 05:49:12PM +0200, Marcin Kościelnicki wrote:
> On 08/04/16 16:18, Jon Ribbens wrote:
> That one is trivially fixable, but here goes:
> 
> async def a():
> global c
> c = b.cr_frame.f_back.f_back.f_back
> 
> b = a()
> b.send(None)
> c.f_builtins['print']('broken')

Ah, I've not used Python 3.5, and I can't find any documentation on
this cr_frame business, but I've added cr_frame and f_back to the
disallowed attributes list.

> Also, if the point of giving me a subclass of datetime is to prevent access
> to the actual class, that can be circumvented:
> 
> >>> real_datetime = datetime.datetime.mro()[1]
> >>> real_datetime
> 
> 
> But I'm not sure what good that is.

It means you can alter the datetime class that is used by the
containing application, which is bad - you could lie to it about
what day it is for example ;-)

I've made it so instead of a direct subclass it now makes an
intermediate subclass which makes mro() return an empty list.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Fri, Apr 08, 2016 at 05:21:38PM +0200, Arthur Darcet wrote:
>If i'm not mistaken, this breaks out:
>> exec('open("out", "w").write("a")', {})
>because if the second argument of exec does not contain a __builtins__
>key, then a copy of the original builtins module is inserted:
>https://docs.python.org/3/library/functions.html#exec

Ah, that's a good point. I did think allowing eval/exec was a bit
ambitious. I've updated it to disallow passing namespace arguments to
them.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

On Fri, Apr 08, 2016 at 03:37:45PM +0100, Paul Moore wrote:
> On 8 April 2016 at 15:18, Jon Ribbens <jon+python-...@unequivocal.co.uk> 
> wrote:
> > I would be very interested to see if anyone can manage to break it.
> > Bugs which are trivially fixable are of course welcomed, but the real
> > question is: is this approach basically sound, or is it fundamentally
> > unworkable?
> 
> What are the limitations? It seems to even block "import" which seems
> over-zealous (no import math?)

The restrictions are:

  Of the builtins, __import__, compile, globals, input, locals,
  memoryview, open, print, type and vars are unavailable (and
  some of the exceptions, but mostly because they're irrelevant).

  You cannot access any name or attribute which starts with "_",
  or is called "gi_frame" or "gi_code".

  You cannot use the "with" statement (although it's possible it might
  be safe for me to add that back in if I also disallow access to
  attributes called "tb_frame").

Importing modules is fundamentally unsafe because the untrusted code
might alter the module, and the altered version would then be used by
the containing application. My code has a "_copy_module" function
which copies (some of) the contents of modules, so some sort of
import functionality of a white-list of modules could be added using
this, but there's no point in me going through working out which
modules are safe to white-list until I'm vaguely confident that my
approach isn't fundamentally broken in the first place.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Jon Ribbens]

I've made another attempt at Python sandboxing, which does something
which I've not seen tried before - using the 'ast' module to do static
analysis of the untrusted code before it's executed, to prevent most
of the sneaky tricks that have been used to break out of past attempts
at sandboxes.

In short, I'm turning Python's usual "gentleman's agreement" that you
should not access names and attributes that are indicated as private
by starting with an underscore into a rigidly enforced rule: try and
access anything starting with an underscore and your code will not be
run.

Anyway the code is at https://github.com/jribbens/unsafe
It requires Python 3.4 or later (it could probably be made to work on
Python 2.7 as well, but it would need some changes).

I would be very interested to see if anyone can manage to break it.
Bugs which are trivially fixable are of course welcomed, but the real
question is: is this approach basically sound, or is it fundamentally
unworkable?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What does a double coding cookie mean?

[Jon Ribbens]

On Tue, Mar 15, 2016 at 01:30:08PM -0700, Guido van Rossum wrote:
> I came across a file that had two different coding cookies -- one on
> the first line and one on the second. CPython uses the first, but mypy
> happens to use the second. I couldn't find anything in the spec or
> docs ruling out the second interpretation. Does anyone have a
> suggestion (apart from following CPython)?
> 
> Reference: https://github.com/python/mypy/issues/1281

If it helps, what 'vim' appears to do is to read the first 'n' lines
in order and then last 'n' lines in reverse order, stopping if the
second stage reaches a line already processed by the first stage.
So with 'modelines=5', the following file:

  /* vim: set ts=1: */
  /* vim: set ts=2: */
  /* vim: set ts=3: */
  /* vim: set ts=4: */
  /* vim: set sw=5 ts=5: */
  /* vim: set ts=6: */
  /* vim: set ts=7: */
  /* vim: set ts=8: */

sets sw=5 and ts=6.

Obviously CPython shouldn't be going through all that palaver!
But it would be a bit more vim-like to use the second line rather than
the first if both lines have the cookie.

Take that as you will - I'm not saying being 'vim-like' is an inherent
virtue ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Jon Ribbens]

On Mon, Jul 27, 2015 at 04:28:48PM -0700, Chris Barker wrote:
   The only other thing I found
   really weird about datetime is how Python 2 had no implementation of
   a UTC tzinfo class, despite this being utterly trivial -
 
Huh? it is either so trivial that there is no point -- simiply say that
your datetimes are UTC, and you are done.
Or it's not the least bit trivial -- the only difference between a UTC
datetime and a naive datetime is that one can be converted to (or
interact with) other time zones. Except that, as we know from this
conversation, is very, very non-trivial!

No, it has nothing to do with conversions. The difference between a
naive timezone and a UTC one is that the UTC one explicitly specifies
that it's UTC and not local time or some other assumed or unknown
timezone. This can make a big difference when passing datetime
objects to third-party libraries, such as database interfaces.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Jon Ribbens]

On Mon, Jul 27, 2015 at 01:04:03AM -0500, Tim Peters wrote:
 [Tim]
  The Python docs also are quite clear about that all arithmetic within
  a single timezone is naive.  That was intentional.  The _intended_
  way to do aware arithmetic was always to convert to UTC, do the
  arithmetic, then convert back.
 
 [Lennart]
  We can't explicitly implement incorrect timezone aware arithmetic and
  then expect people to not use it.
 
 Python didn't implement timezone-aware arithmetic at all within a
 single time zone.  Read what I wrote just above.  It implements naive
 arithmetic within a single time zone.

This usage of time zone is confusing. As far as I can tell, you
really mean UTC offset. A time zone would be something like
Europe/London, which has two different UTC offsets throughout
the year (not to mention other historical weirdnesses), whereas
arithmetic on a timezone-aware datetime is only going to work
so long as you don't cross any of the boundaries where the UTC
offset changes.

I agree with you about pretty much everything else about datetime,
just I find the terminology misleading. The only other thing I found
really weird about datetime is how Python 2 had no implementation of
a UTC tzinfo class, despite this being utterly trivial - but it's too
late to do anything about that now, of course.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None 1 raises TypeError

[Jon Ribbens]

On Mon, Feb 17, 2014 at 12:43:25PM +0100, M.-A. Lemburg wrote:
 This doesn't only apply to numeric comparisons. In Python 2 you
 can compare None with any kind of object and it always sorts first,

No you can't. See http://bugs.python.org/issue1673405 .

According to Tim Peters, the None is less than everything rule
never existed.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] SSL sockets and settimeout

[Jon Ribbens]

Am I correct in thinking that Python's newfangled socket.settimeout()
feature does not play well with SSL wrapped sockets? Would there be
any interest in making it so that it did?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] utf-8 encoding in checkins?

[Jon Ribbens]

On Sun, Mar 27, 2011 at 05:21:08PM +0200, Antoine Pitrou wrote:
 On Sun, 27 Mar 2011 10:06:49 -0500
 s...@pobox.com wrote:
  It seems that all checkin mails are utf-8-encoded.  This makes it
  challenging to view checkin mails if you have a text-based mail reader.  (I
  use VM within XEmacs.  XEmacs doesn't seem to support utf-8 out of the box.
  I believe you need to add MULE to it.)
 
 Hmm, are you sure? Software (especially mail-reading software) that
 doesn't support utf-8 in 2011 should be considered extremely broken.
 
 (I understand you may be a native English speaker, but do you never
 receive non-ASCII utf-8 e-mail at all? doesn't it get displayed
 properly?)

In my experience it is very difficult to get a whole system
configured, such that UTF-8 works reliably. In my case that would
mean, I think, Linux, glibc, mutt, screen, elinks, and SecureCRT would
all have to simultaneously be configured precisely correctly in order
to get it to work. I have certainly never managed it, anyway.

Mind you, I've never managed to get the -- button working reliably
either, but to be fair that's insanely complicated too.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] utf-8 encoding in checkins?

[Jon Ribbens]

On Sun, Mar 27, 2011 at 02:13:29PM -0400, Eugene Toder wrote:
  I'm not disputing that, and I understand that my current choice of mail
  reader limits me.  I was just asking if it would be possible (read: fairly
  easy) to only generate utf-8 when it was necessary.
 
 Isn't utf-8 itself same as ascii where no non-ascii symbols are used?

Yes, however some software will label outgoing stuff as UTF-8 even if
actually it contains no non-ASCII characters. Then, other software
will assume that it cannot understand the message (either because it
assumes the charset label is correct or because it doesn't know that
it can understand UTF-8 that contains no top-bit-set characters).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] utf-8 encoding in checkins?

[Jon Ribbens]

On Sun, Mar 27, 2011 at 02:21:25PM -0400, R. David Murray wrote:
  Mind you, I've never managed to get the -- button working reliably
  either, but to be fair that's insanely complicated too.
 
 No idea what that is.

Backspace key.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Distutils2 scripts

[Jon Ribbens]

On Fri, Oct 08, 2010 at 01:24:09PM +0200, Dirkjan Ochtman wrote:
 On Fri, Oct 8, 2010 at 09:05, Tarek Ziadé ziade.ta...@gmail.com wrote:
  The feedback I received for this is pretty clear: people want a single
  script that can be called directly. e.g.
 
  $ distutils2 depgraph
  $ distutils2 install
  $ distutils2 run command
 
  Fair enough, I can add that script in the project and get it installed
  when people install the project.
 
 It doesn't seem very nice to have a version in the script. Can we just
 call it distutils? Or py-dist?

pypackage?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Distutils2 scripts

[Jon Ribbens]

On Fri, Oct 08, 2010 at 11:04:35AM -0400, Toshio Kuratomi wrote:
 In the larger universe of programs, it might make for more intuitive
 remembering of the command to use a prefix (either py or python) though.
 
 python-setup  is a lot like python setup.py
 pysetup is shorter
 pyegg is even shorter :-)

I'd just like to say pypackage again.

pypackage install thingy
pypackage remove thingy
pypackage update thingy

etc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rework nntlib?

[Jon Ribbens]

On Tue, Sep 14, 2010 at 11:43:46AM -0500, s...@pobox.com wrote:
 We got rid of gopherlib a few years ago (deprecated in 2.5, presumably gone
 in 2.6).  I suspect the NNTP protocol has a greatly diminished user base as
 well, GMANE's presence notwithstanding.

NNTP is *very* considerably less dead than gopher.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Looking after the buildbots (in general)

[Jon Ribbens]

On Wed, Aug 04, 2010 at 07:26:06PM -0400, Barry Warsaw wrote:
 On Aug 04, 2010, at 06:39 PM, Steve Holden wrote:
 I'll see if I can get God to extend it for you.
 
 No need to involve the supernatural Steve!  Just approve that PSF grant I
 submitted so I can finish my (Python powered of course!) clone army.

Someone else got there ahead of you...
http://en.wikipedia.org/wiki/MASSIVE_%28software%29
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Jon Ribbens]

On Sun, May 23, 2010 at 03:16:27PM +0400, Andrew Svetlov wrote:
 Is there any reason to have Future .cancelled, .done, .running as methods?
 From my perspective they are really readonly properties.
 
 BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l'
 form: 'canceled'.

In English, only cancelled is correct. In American, either is correct.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urlparse.urlunsplit should be smarter about +

[Jon Ribbens]

On Sun, May 09, 2010 at 03:19:40PM -0600, David Abrahams wrote:
 Yes.  But the question is whether urlparse should really be so fragile
 that every hierarchical scheme needs to be explicitly registered.
 Surely ending with “+file” should be sufficient to have it recognized
 as a file-based scheme

How do you figure?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] argparse ambiguity handling

[Jon Ribbens]

On Wed, Apr 21, 2010 at 03:53:16AM -0400, Eric Smith wrote:
 I agree the new behavior is desirable. And I also think it should be the  
 default, although I feel less strongly about that.

 But since this behavior seems to be an accident of the implementation  
 (based on Steve's comment above), I think a test should be added for it  
 if it's accepted as a Good Thing (tm). Otherwise it's possible that it  
 will vanish as the implementation changes.

+1 to all of that
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] argparse ambiguity handling

[Jon Ribbens]

On Tue, Apr 20, 2010 at 03:27:53PM -0400, Neal Becker wrote:
 I have a preference to allow at least exact matches to succeed even in the 
 case of ambiguity - mainly because I accidentally created this already once, 
 and I feel it's better to at least work somewhat.  Not sure if there is any 
 more elegant solution.  OTOH, I feel this is somewhat inelegant, as it 
 appears to treat exact match as a special case.

Surely an exact match *is* a special case - it's an exact match!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ and bytecode-only

[Jon Ribbens]

On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote:
 Nope, sorry I should have been clearer.
 
 *creation* is the key here.  As per BDFL pronouncement, we'll support reading
 pyc-only modules just like we do today.  This is in PEP 3147.  We won't
 support creating them though.
 
 BTW, I'm actually starting to work on the implementation now, so if you want
 to play along:
 
 % bzr branch lp:~barry/python/pep3147

Sorry if I missed it, but why on earth is the bytecode directory
__pycache__ and not .pycache? (Or indeed anything else that starts
with a '.') Surely this is a classic ideal use case for a hidden
directory?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 389: argparse - new command line parsing module

[Jon Ribbens]

On Thu, Oct 01, 2009 at 09:58:59AM +0100, Paul Moore wrote:
 (Question - is it *ever* possible for a Unix program to have invalid
 file descriptors 0,1 and 2? At startup - I'm assuming anyone who does
 os.close(1) knows what they are doing!)

Yes, at startup you just have the file descriptors your parent process
gave you. This may or may not include 0, 1, and 2, and may or may not
include other descriptors too.

I seem to recall there have been security holes caused in the past by
people assuming 0-2 will exist.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 389: argparse - new command line parsing module

[Jon Ribbens]

On Mon, Sep 28, 2009 at 09:38:20AM +0100, Floris Bruynooghe wrote:
 On Mon, Sep 28, 2009 at 06:59:45AM +0300, Yuvgoog Greenle wrote:
  -1 for deprecating getopt. getopt is super-simple and especially useful for
  c programmers learning python.
  
  +1 for argparse.+1 for eventual deprecation of optparse - optparse and
  argparse have a very similar syntax and having both is just
  confusing. tsboapooowtdi
 
 +1 on all of this :-)
 
 It would be a shame to see getopt go but optparse - argparse seems
 logical.

+1 from me too - keep getopt, deprecate optparse.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue5434: datetime.monthdelta

[Jon Ribbens]

On Thu, Apr 16, 2009 at 12:10:36PM +0400, Oleg Broytmann wrote:
  This patch adds a monthdelta class and a monthmod function to the
  datetime module.  The monthdelta class is much like the existing
  timedelta class, except that it represents months offset from a date,
  rather than an exact period offset from a date.
 
I'd rather see the code merged with timedelta: timedelta(months=n).

Unfortunately, that's simply impossible. A timedelta is a fixed number
of seconds, and the time between one month and the next varies.

I am very much in favour of there being the ability to add months to
dates though. Obviously there is the question of what to do when you
move forward 1 month from the 31st January; in my opinion an optional
argument to specify different behaviours would be nice.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FreeBSD 7 amd64 and large memory tests

[Jon Ribbens]

On Wed, Sep 17, 2008 at 08:21:55AM -0700, Alex Martelli wrote:
 Unbelievable as this may seem, this crazy over-committing malloc
 behavior is by now a classic -- I first fought against it in 1990,
 when IBM released AIX 3 for its then-new RS/6000 line of workstations;
 in a later minor release they did provide a way to optionally switch
 this off, but, like on Linux, it's a system-wide switch, NOT
 per-process:-(.
 
 I concur with http://www.win.tue.nl/~aeb/linux/lk/lk-9.html (the best
 explanation I know of the subject, and recommended reading) which, on
 this subject, says Linux on the other hand is seriously broken.
 (just like AIX 3 was).  Sad to learn that BSD is now also broken in
 the same way:-(.

It's now now also broken, it has been that way for a very long time.
For example, see this message I wrote back in July 1999 complaining
about FreeBSD overcommit:
http://www.mail-archive.com/[EMAIL PROTECTED]/msg01056.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Consistent platform name for 64bit windows (was: distutils.util.get_platform() for Windows)

[Jon Ribbens]

On Wed, Mar 19, 2008 at 02:05:37AM +0900, [EMAIL PROTECTED] wrote:
 So, at the risk of painting a bike-shed, I'd like to propose that we adopt
 'AMD64' in distutils (needs a change), platform.py (needs a change to use
 sys.getwindowsversion() in preference to pywin32, if possible, anyway),
 and the Python banner (which already uses AMD64).

Debian uses 'amd64', it seems they chose this after a survey of what
existing systems used what names, and the consensus came out in favour
of amd64. http://lists.debian.org/debian-ctte/2004/06/msg00041.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jon Ribbens]

On Mon, Jan 28, 2008 at 08:07:21PM -0800, Guido van Rossum wrote:
 PS. There's something wrong with Raymond's mailer that creates a
 thread in gmail whenever he responds. I suspect it's not correctly
 adding an In-reply-to header. That makes the thread feel much more
 disconnected than most, because often the quoted text is in a
 different thread.

His mails don't have any indication they're a reply at all - not even
so much as a Re:  in the Subject. Even Outlook Express isn't *that*
broken; I suspect he's not actually using the 'reply' button in his
mailer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: per user site-packages directory

[Jon Ribbens]

On Mon, Jan 14, 2008 at 03:07:20PM -0500, Barry Warsaw wrote:
 I feel pretty strongly that ~/bin should *not* be used.  It makes  
 sense to me that ~/.local would mirror /usr/local.

It makes sense, but personally I have never heard before of ~/.local.
Whereas ~/bin is something I am quite familiar with.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Jon Ribbens]

On Tue, Oct 23, 2007 at 12:16:41PM +1300, Greg Ewing wrote:
 [EMAIL PROTECTED] wrote:
  This interface follows the completely stupid semantics of System V and
  IEEE Std 1003.1-1988 (``POSIX.1'') that require that all locks
  associated with a file for a given process are removed when any file
  descriptor for that file is closed by that process  Flock(2) is
  recommended for applications that want to ensure the integrity of their
  locks when using library routines or wish to pass locks to their
  children.
  
  I guess the BSD folks were a bit upset when they wrote that.
 
 That sounds more like something written by the GNU folks than
 the BSD folks.

It's from BSD. The earliest I can find it is 4.4BSD. It's still in the
latest versions of OpenBSD, NetBSD and FreeBSD.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Jon Ribbens]

On Tue, Oct 23, 2007 at 12:29:35PM +1300, Greg Ewing wrote:
 [EMAIL PROTECTED] wrote:
  Does fcntl.flock work over NFS and SMB and on Windows?
 
 I don't think file locking will ever work over NFS, since
 it's a stateless protocol by design, and locking would
 require maintaining state on the server.

You can do file locking over NFS, that's one of the reasons people
use fcntl. It uses an RPC side channel separate to the main NFS
protocol.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing the GIL (Me, not you!)

[Jon Ribbens]

On Thu, Sep 13, 2007 at 12:19:21PM +0200, André Malo wrote:
  Pardon my ignorance but why does Python do reference counting for truly
  global and static objects like None, True, False, small and cached
  integers, sys and other builtins? If I understand it correctly these
  objects are never garbaged collected (at least they shouldn't) until the
  interpreter exits. Wouldn't it decrease the overhead and increase speed
  when Py_INCREF and Py_DECREF are NOOPs for static and immutable objects?
 
 The check what kind of object you have takes time, too. Right now, just 
 counting up or down is most likely faster than that check on every refcount 
 operation.

To put it another way, would it actually matter if the reference
counts for such objects became hopelessly wrong due to non-atomic
adjustments?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing the GIL (Me, not you!)

[Jon Ribbens]

On Thu, Sep 13, 2007 at 01:15:39PM +0200, Martin v. Löwis wrote:
  To put it another way, would it actually matter if the reference
  counts for such objects became hopelessly wrong due to non-atomic
  adjustments?
 
 If they drop to zero (which may happen due to non-atomic adjustments),
 Python will try to release the static memory, which will crash the
 malloc implementation.

That could be avoided by a flag on the object which is checked in
free(). I'm just suggesting it as an alternative as it sounds like
it might be more efficient than either locking or avoiding having
reference counts on these objects (especially if the reference count
is initialised to MAX_INT/2 or whatever).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing the GIL (Me, not you!)

[Jon Ribbens]

On Thu, Sep 13, 2007 at 06:38:05PM -0500, [EMAIL PROTECTED] wrote:
 Hrvoje More precisely, Python will call the deallocator appropriate for
 Hrvoje the object type.  If that deallocator does nothing, the object
 Hrvoje continues to live.  Such objects could also start out with a
 Hrvoje refcount of sys.maxint or so to ensure that calls to the no-op
 Hrvoje deallocator are unlikely.
 
 Maybe sys.maxint/2?  You'd hate for the first incref to invoke the
 deallocator even if it was a no-op.  

I do believe I already suggested that ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pythreads and BSD descendants

[Jon Ribbens]

On Thu, Jul 26, 2007 at 04:08:37PM +, Cameron Laird wrote:
 Folklore that I remember so unreliably I avoid trying to repeat it here
 held that Python threading had problems on BSD and allied Unixes.  What's
 the status of this?  I suspect the answer is, Everything works, and the
 only real problem ever was that *signals* have different semantics under
 Linux and *BSD.  Anyone who can answer explicitly, though, would repre-
 sent a help to me.

This is just my personal opinion, but I suspect that this is perhaps
because people have *tried* threading more in Python than in many
other languages, because Python makes it particularly easy.

I've certainly had the experience that multithreaded stuff I have
tried has sometimes had problems under various OSes (Linux, Solaris,
OpenBSD, etc) due to operating system bugs with threading in general
rather than Python problems per se.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TLSAbruptCloseError

[Jon Ribbens]

On Tue, Jun 05, 2007 at 12:55:07PM -0400, Todd Hopfinger wrote:
I am using TLS Lite and J2ME SecureConnection for the purposes of
encrypting traffic to/from a Java Midlet client and a multithreaded Python
server. However, I encounter a TLSAbruptCloseError. I have tried to
determine the cause of the exception to no avail. I understand that it has
to do with close_notify alerts. My abbreviated code follows.

It may or may not be your specific problem, but Microsoft SSL servers
tend to just drop the TCP connection when they're done, rather than
do a proper SSL shutdown. This tends to make errors such as the above,
which you must then ignore.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The docs, reloaded

[Jon Ribbens]

On Tue, May 22, 2007 at 06:13:36PM +0200, Georg Brandl wrote:
 Finding the location that tells you how to suggest changes, and opening
 a new bug in the infamous SF tracker is not really something people do
 happily. A click here to suggest a change link that leads to a pseudo-
 edit-form, complete with preview facility, might prove more effective.

Indeed. I know my instinctive reaction to the Python docs is oh, this
is not something which the public can contribute to. Something more
like the PHP-style public annotations might be good - with an
appropriate moderation / voting system on the annotations it could
possibly be very good.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The docs, reloaded

[Jon Ribbens]

On Tue, May 22, 2007 at 01:19:36PM -0400, Steve Holden wrote:
  For example, this rocks! ::
  
  http://pydoc.gbrandl.de/search.html?q=os.patharea=default

 It would be more impressive if the search string returned hits ...

Also if it was not completely reliant on JavaScript...
(Maybe it's not finished yet?)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The docs, reloaded

[Jon Ribbens]

On Sat, May 19, 2007 at 07:14:09PM +0200, Georg Brandl wrote:
 For the impatient: the result can be seen at http://pydoc.gbrandl.de.

I think that looks great.

One comment I have, I don't know if it's relevant - it perhaps depends
on whether the Global Module Index is auto-generated or not. This is
the page I visit the most out of all the Python documentation, and
it's far too large and unwieldy. IMHO it would be much better if only
the top-level modules were shown here - having the single package
'distutils', for example, take up nearly 50 entries in the list is
almost certainly hindering a lot more people than it helps. It would
perhaps be better if such packages show up as one entry, which shows
the sub-modules when clicked on.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implicit String Concatenation and Octal Literals Was: PEP 30XZ: Simplified Parsing

[Jon Ribbens]

On Wed, May 02, 2007 at 10:23:39PM -0700, Raymond Hettinger wrote:
 Another way to look at it is to ask whether we would consider 
 adding implicit string concatenation if we didn't already have it.
 I think there would be a chorus of emails against it

Personally, I would have been irritated if it wasn't there. I'm used
to it from other languages, and it would seem like a gratuitous
incompatability if it wasn't supported.

I'm definitely against this proposal in its entirety.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python3k change to slicing

[Jon Ribbens]

Guido van Rossum [EMAIL PROTECTED] wrote:
 -1

Me too.

 While this may be theoretically preferable, I believe that in practice
 changing this would be a major pain for very little gain. I don't
 recall ever finding a bug related to this feature, and I believe it's
 occasionally useful.

I find it quite frequently useful.

 Here's something that would be much more cumbersome with your proposed
 change: suppose I have a string of unknown length and I want to get
 the first three characters, or less if it's not that long. Today I can
 write s[:3]. With your proposal I would have to write s[:min(3,
 len(s))].

... and that's exactly the sort of situation I find it useful ;-)

I certainly think it's easier to check that your result is the length
you wanted on the occasions you need to, than it would be to make the
replacement you show above everywhere else.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] deprecate commands.getstatus()

[Jon Ribbens]

\Martin v. Löwis\ [EMAIL PROTECTED] wrote:
  And do we even need os.fork(), os.exec*(), os.spawn*()?
 
 I don't know about about *os*.fork; I surely like to have posix.fork. 
 The posix module exposes many OS functions as-is. This has the
 advantage that their semantics are crystal-clear: they do whatever the
 system call does (which, ideally, is what POSIX specifies for it).
 So you can do systems programming in Python, and only need good
 knowledge of the underlying system calls (i.e. using Python as a
 better C).

I definitely agree. Removing the POSIX system call mappings would make
Python less useful and general-purpose.

Yes it's nice to have high-level utility functions like those in the
subprocess module, but I think it's very important for the low-level
functions to be there too when you need them.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of thread cancellation

[Jon Ribbens]

Nick Maclaren [EMAIL PROTECTED] wrote:
 Well, I have seen it hundreds of times on a dozen different Unices;
 it is very common.  You don't always SEE the stuck process - sometimes
 the 'kill -9' causes the pid to become invisible to ps etc., and
 just occasionally it can continue to use CPU until the system is
 rebooted.

You're describing something caused by a buggy operating system.
I have never seen any modern Unix exhibit any of the behaviours
you describe. I have seen such things in the 1990's though.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of thread cancellation

[Jon Ribbens]

Nick Maclaren [EMAIL PROTECTED] wrote:
 Sockets, terminals etc. are stateful devices, and killing a process
 can leave them in a very unclean state.  It is one of the most
 common causes of unkillable processes (the process can't go until
 its files do, and the socket is jammed).

Can you elaborate on this? You can get zombie entries in the process
table if nobody's called 'wait()' on them, and you can (extremely
rarely) get unkillable process in 'disk-wait' state (usually due to
hardware failure or a kernel bug, I suspect), but I've never heard
of a process on a Unix-like system being unkillable due to something
to do with sockets (or any other kind of file descriptor for that
matter). How could a socket be 'jammed'? What does that even mean?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thread safe SMTP module

[Jon Ribbens]

Gordon Messmer [EMAIL PROTECTED] wrote:
 After some discussion, Aahz suggested that I discuss the problem here, 
 on python-dev.  He seemed to think that the problem I saw may have been 
 an indication of a bug in python.  Could anyone take a look at that 
 thread and say whether it looks like a bug, or working with non-blocking 
 sockets was the right thing to do?

Well, not having to faff around with non-blocking IO is one of the
major advantages of threading, so if blocking IO is indeed the cause
of your problem then it certainly sounds like a bug in Python, or
possibly your operating system.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thread safe SMTP module

[Jon Ribbens]

Aahz [EMAIL PROTECTED] wrote:
 One small wrinkle (and the reason I suggested bringing this to
 python-dev): I suspect that the problem is not a bug, but simply the
 occasional failure of sockets.  When that happens in a threaded app
 without timeouts, eventually threads die (block forever).  But you
 apparently can't use timeouts with file-wrapped sockets, so if that's the
 problem, we need to pick one of:
 
 * switch to non-wrapped sockets 
 * switch to non-blocking I/O for smtplib
 * make file-wrapped sockets work with timeouts

In case it's helpful, I've put the file-like-sockets-with-timeouts
code that I wrote up here:

  http://www.unequivocal.co.uk/dl/socketutils.py

This is a pure-Python implementation that provides a file-like
interface to a socket, with timeouts, that definitely works with
threaded programs.

Note it was written in 2002, so any part of the Python 'file'
interface which was added since then is not in there.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thread safe SMTP module

[Jon Ribbens]

Gordon Messmer [EMAIL PROTECTED] wrote:
 Tonight I should have time to pull an old copy of the code out of CVS 
 and recreate the test script that I used.  Once I have, it should be a 
 matter of feeding a big list of email addresses to the script and 
 waiting a couple of minutes for the script to lock up.  The last time I 
 looked, the problem was very easy to observe.

Try it using the alternative makefile code I posted and see if it
makes a difference...
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Steven Bethard [EMAIL PROTECTED] wrote:
 I was trying to minimize the extent of this already too long thread,
 assuming you could go do the reading I referred you to if you were
 really interested in the answer.

There's nothing to be gained by being patronising.

 I still encourage you to read the reference, but for your sake,
 here's a brief example of temporal logic that would break::
 
 We know that:
 date(2006, 1, 1) *Includes* datetime(2006, 1, 1, 0, 0, 1)

Oh dear. You've fallen at the first hurdle. We do not know that.
You are assuming the answer you want and then using it as an axiom.
Your justification for this assumption was the whole point of my
question, but you have again failed to answer. I don't know how much
clearer I can make this, given that I have explicitly mentioned it
several times already.

 Hope that helps,

No, it doesn't.

Seriously, please try and understand this. I get that you wish Python
datetime was the ultra-complete temporal logic module you want it to
be. Good for you. I sympathise. If it was, I would agree with you. But
it isn't. It's a simple module for dealing with calendar dates and
times, not intervals in time. It makes pragmatic assumptions about
what people mean when they use date arithmetic. They might not be the
assumptions you want, but then it's probably not the module you want.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Collin Winter [EMAIL PROTECTED] wrote:
 Please find or write a package that makes the assumptions you want,
 since datetime clearly isn't the module you want.

Datetime clearly *is* the module I want. It already makes the
assumptions I want, I just want it to make them consistently.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] These csv test cases seem incorrect to me...

[Jon Ribbens]

Andrew McNamara [EMAIL PROTECTED] wrote:
 The point was to produce the same results as Excel. Sure, Excel probably
 doesn't generate crap like this itself, but 3rd parties do, and people
 complain if we don't parse it just like Excel (sigh).

The slight problem with copying Excel is that Excel won't parse its
own CSV output.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Phillip J. Eby [EMAIL PROTECTED] wrote:
 At 09:20 PM 3/9/2007 +, Jon Ribbens wrote:
 If you want the answer to be the entire of that day then you need
 to alter the datetime module so that, e.g. subtracting 2007-03-08
 from 2007-03-09 does not return one day as currently, but returns
 zero days instead (since of course there is no time between the end
 of one day and the start of the next day).
 
 Unless you decide that subtraction is between days' ends.  Or days' 
 beginnings.  Either suffices to produce a 1-day result in that case, and is 
 still consistent with viewing days as slices of time.

So you're deciding that a 'date' is 'the entire of that day', except
when you subtract two of them, when it suddenly means something else? ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Steven Bethard [EMAIL PROTECTED] wrote:
 Using the .date() is fine when the year/month/day doesn't match.  So
 the following are fine::
 datetime.datetime(2005, 1, 1, 0, 0, 0)  datetime.date(2006, 1, 1)
 datetime.datetime(2007, 1, 1, 0, 0, 0)  datetime.date(2006, 1, 1)
 It's *not* okay to say that a date() is less than, greater than or
 equal to a datetime() if the year/month/day *does* match.

Why not? That only makes sense if you decide that a Python 'date'
means 'the entire of that day'. It's not at all clear that that's
what a Python 'date' *does* mean. And, as I mentioned before, if
you do decide that then what Python currently does when you
subtract dates is broken.

I just found another case where 'date's pretend to have a time-part
of midnight - if you add a date to a timedelta. Add 23 hours to a
date and it's unchanged - add 24 and it moves forward a day. If a
'date' really is 'the entire of that day', then adding a timedelta
to it which is not an integer multiple of 1 day should either raise
an exception, or return some 'time duration' object that doesn't
currently exist.

 The correct temporal relation is During, but Python doesn't have a
 During operator. During is not the same as less-than, greater-than
 or equal-to, so all of these should be False::

I think you're assuming the Python datetime module has higher goals
than it does. It doesn't have any concept of a duration in time,
not unless you couple a 'datetime' with a 'timedelta' yourself.
And 'timedelta's are restricted to periods of time that are a fixed
number of seconds - i.e. there is no way to indicate a month
or a year (although that lack is a whole different argument ;-) )

Your argument is quite correct if you're considering some fancy
uber-complicated kitchen-sink-included all-encompassing temporal
logic package, but that's not what Python's datetime is, nor
frankly is what it should be.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Christian Heimes [EMAIL PROTECTED] wrote:
 Jon Ribbens schrieb:
  So you're deciding that a 'date' is 'the entire of that day', except
  when you subtract two of them, when it suddenly means something else? ;-)
 
 It makes kinda sense although it looks like a contradiction at first.
 
 The common linguistic usage of dates in English and German:
 On Saturday or On 2007-03-07 is a undefined time anytime on Saturday
 or the whole Saturday but Saturday and Sunday are a day apart. It feels
 right to me.

That doesn't help much. English is often very vague about times and
dates, but computer programs must be have a defined answer.

What do you feel next Tuesday plus 12 hours means? ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Christian Heimes [EMAIL PROTECTED] wrote:
  What do you feel next Tuesday plus 12 hours means? ;-)
 
 First thought: It's nonsense! Nobody would say that. ;)
 
 Second though: Tuesday noon (12h after the beginning of Tuesday)

I agree with you entirely. Your suggestions correspond to 'throw an
exception' and 'return a datetime'. Note that in the second case you
are agreeing with me that a date means midnight at the start of that
date ;-)

Python datetime disagrees with us both. It basically says 'add the
timedelta to midnight at the start of the date, then return just
the date part of the result'.

I assume there is some reason for this surprising behaviour. But this
is another example of where datetime is taking a pragmatic approach in
an ambiguous situation - exactly as I am suggesting in the situation
of comparing a date and a datetime.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Robert Brewer [EMAIL PROTECTED] wrote:
 One solution that just occurred to me -- and that
 skirts the issue of choosing an interpretation --
 is that, when comparing date and datetime objects,
 the datetime's .date() method is called and the
 result of that call is compared to the original
 date.
 
+1
 
...and it's a decision that can be made independently
of how to add or subtract dates.

I don't like it. It sounds suspiciously like when comparing integers
and floats, discard the non-integer portion of the float and then
compare as integers.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Steven Bethard [EMAIL PROTECTED] wrote:
 I don't see why the docs can't be explicit about what subtraction
 means given that there are a number of possible interpretations.

I don't see why the docs can't be explicit about what comparison
means given that there are a number of possible interpretations.

 You can draw the analogy here with integer addition::
 
  int(1) + int(.75)
 1
  int(1) + int(1.25)
 2

You're making my point for me. An integer (in the abstract sense)
*is* the same as a float with a zero fractional part. You're agreeing
that datetime treats dates as datetimes with a midnight time part.

 However, assuming that a date() is a datetime() with a time of
 midnight will clearly break that logic.

Could you please explain how? It doesn't look to me like it breaks
anything. I think you're assuming that date means the whole of
that day - if so then why do you think that assumption is valid?

Using your suggested analogy above, if you consider dates to be
ints and datetimes to be floats, then it seems to be that
everything makes sense and is logical and coherent.

 I still don't see why my more careful comparison would be bad for any
 of your code. Could you give an example where it would be bad for all
 the following to be False::
  date(2006, 1, 1)  datetime(2006, 1, 1, 0, 0, 0)
  date(2006, 1, 1)  datetime(2006, 1, 1, 0, 0, 0)
  date(2006, 1, 1) == datetime(2006, 1, 1, 0, 0, 0)

You need me to explain why a  b, a  b and a == b all being
false simultaneously is bad!? What would a != b return? What would
cmp(a, b) return?

 even though the following would be True::
  date(2006, 1, 1)  datetime(2006, 1, 2, 0, 0, 0)
  date(2006, 1, 1)  datetime(2005, 12, 31, 0, 0, 0)

Actually, your suggestion would be much better than the current
behaviour, for my uses, even though it's totally illogical.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

I see you snipped without response my request to back up your claim
that assuming that a date() is a datetime() with a time of midnight
will clearly break that logic.

Am I to assume you cannot back it up?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

\Martin v. Löwis\ [EMAIL PROTECTED] wrote:
 There are know problems comparing durations (e.g. is 30 days more
 or less than a month?). For time stamps, there is no issue. For
 calender dates, there are again problems, in particular with time
 zones.

Python durations (datetime.timedelta) do not support the concept of
months, so that's not an issue. The timedelta type only supports
durations that are a fixed number of seconds.

I don't see what the issue with time zones would be? Surely you just
convert to the same time zone and then compare?

 What I don't know (because I never used it) is whether the datetime
 module implements time stamps. If it does (i.e. no durations,
 no time zones, no issues with leap seconds), then supporting a
 total order seems reasonable.

It supports converting *from* timestamps, but not *to* timestamps,
which is the subject of a feature request I submitted the other day
(1673409) ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] datetime module enhancements

[Jon Ribbens]

Brett Cannon [EMAIL PROTECTED] wrote:
  Treat dates as if they have a time-part of midnight. This is my preferred
  solution, and it is already what the datetime module does, for example,
  when subtracting two dates.
 
 I personally like the current solution.  The proposal to just assume
 midnight goes against EIBTI in my mind.

Not in mine. The comparison of a date with a datetime throwing
an exception is astonishing in my mind.

The problem is, if you don't define a 'date' as I suggest (i.e.
midnight at the start of the day in question), then what *does*
a 'date' represent?

If you want the answer to be an unknown time on that day then
you need to alter the datetime module so that subtracting one
date from another throws an exception.

If you want the answer to be the entire of that day then you need
to alter the datetime module so that, e.g. subtracting 2007-03-08
from 2007-03-09 does not return one day as currently, but returns
zero days instead (since of course there is no time between the end
of one day and the start of the next day).

Obviously both of those alternatives are ludicrous (not to mention
they would break backwards-compatibility), so I would suggest the
only logical solution is to make the change I proposed ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com