[Python-Dev] Re: Recent PEP-8 change

[Giampaolo Rodola']

On Wed, Jul 1, 2020 at 2:34 AM Greg Ewing  wrote:
>
> Sorry for fanning the flames, but this whole thread is so over
> the top I'm finding it kind of entertaining.
>
> On 1/07/20 2:23 am, Piper Thunstrom wrote:
> > The grammarian movement, in general, was built on
> > elevating a very specific form of English over others. It specifically
> > was chosen to avoid "lower class" usages
>
> This argument seems to rest on the assumption that "lower
> class" equates to "non-white". This is an extremely US-centric
> idea. It could possibly even be described as exhibiting a
> "breathtaking level of ignorance"...
>
> > the
> > Elements of Style (And many works like it) are built on a system of
> > white supremacy.
>
> If that's true, then the entirety of Western culture is built
> on a system of white supremacy. That includes all our modern
> technology. It includes the Python programming language. We'd
> better stop recommending Python to people!
>
> > Each individual who likes
> > Elements of Style is not wrong for liking the book, you can keep it on
> > your shelf and no one will be angry.
>
> Okay, I'm confused. S&W is a symbol of white supremacy that
> shall never be recommended or mentioned in polite company, but
> it's all right to have one on your shelf, as long as you keep it
> to yourself... or something?
>
> You can't have it both ways.

Not if you keep the book between a rainbow and a unicorn.

-- 
Giampaolo - gmpy.dev
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CBHLMDEMEKQC5Z4P5OMMZJZ6X4BBTC4Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Recent PEP-8 change

[Giampaolo Rodola']

Mmm. Well, we said what we had to say. I think we both agree there's
no point in continuing. ;-)
Cheers.

On Tue, Jun 30, 2020 at 5:54 PM Piper Thunstrom  wrote:
>
> On Tue, Jun 30, 2020 at 11:36 AM Giampaolo Rodola'  wrote:
> > No we don't. Who are you to tell others what they should be aware of
> > or what they should fight for? And why should I join your battle that
> > I don't even agree with? Most of us probably had no idea what Elements
> > of Style was before this thread started, and *none* of us has ever
> > attributed any racial meaning to it in the 19 years since PEP-8 was
> > put in place. I still don't attribute any racial meaning to it,
> > despite all the twisted explanations which were given about the
> > correlation with white supremacy, which AFAIK are only shared by you
> > and the person who pushed this PR, who is also a newcomer. If anybody
> > can come up here for the first time, impose their world view on the
> > majority just "because it's oppressive", and do this sort of
> > pandemonium, then we can kiss Python goodbye. It also means there
> > virtually is no end to this in the long run, because anyone can come
> > up with any sort of funny theory in order to tilt the direction of the
> > language and deviate its culture from the outside.
> >
> > --
> > Giampaolo - gmpy.dev
>
> It's disheartening as hell to watch a core dev so clearly despise the
> idea of growing the Python community.
>
> You have open contempt of the code of conduct, despite the growing
> movement of underrepresented minorities stating that codes of conduct
> make them feel safer in their communities.
>
> You openly belittle the contributions of people you view as "newcomers".
>
> You uphold that grand tradition of technical people who decide that
> because they are ignorant to a specific kind of oppression, that
> oppression doesn't exist.
>
> My focus in Python is not specifically the advancement of the language
> as a language (though I care about that, otherwise why would I be
> subscribed to this list at all?). My focus
> and love for Python is as a community. And I care, specifically, that
> I see three threads that are leaving me breathless and sad for my
> community. I am imagining the pain and hurt
> these conversations will cause for people who are not as well
> positioned and not as comfortable in the community. We are going to
> lose FUTURE contributors because your vitriol
> and hate for people who care about oppression as it intersects the
> Python community.
>
> You may very well think that's the optimal way for a language to be.
>
> I do not. We must, as a community, examine our prejudices and aim to
> be welcoming or we're going to see a split in Python much worse than
> Py2 -> Py3.
>
> Piper Thunstrom
>
> My public key is available at https://keybase.io/pathunstrom
> Public key fingerprint: 8FF9 3F4E C447 55EC 4658 BDCC A57E A7A4 86D2 644F



--
Giampaolo - gmpy.dev
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YU3PRLKET4RZWTP3RPMHSJ4Z73GWBXPE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Recent PEP-8 change

[Giampaolo Rodola']

On Tue, Jun 30, 2020 at 4:39 PM Piper Thunstrom  wrote:
>
> > The original request for the change had absolutely no hint that the current 
> > text was racist in any way; then we find out that, apparently, we've been 
> > harboring white supremacist ideals by prescribing when to use apostrophes 
> > and commas?  That commit message (not the commit itself) took what should 
> > have been a simple change and turned into a platform for political 
> > grandstanding of the worst kind:
> >
> > - False, as far as I can tell (until given confirming examples from the S&W 
> > text)
> > - Only colored people are mentioned (and other /native English speakers/)
> > - Zero mention of non-native English speakers
>
> So, I think I can explain. (Not with references because I've lost most
> of them over the years, but bear with me.)
>
> The actual advice in The Elements of Style are mostly inoffensive when
> taken on their own, and out of context. The problem is that the
> Elements of Style (And many works like it) are built on a system of
> white supremacy. The grammarian movement, in general, was built on
> elevating a very specific form of English over others. It specifically
> was chosen to avoid "lower class" usages and things like AAVE (though
> that term would not exist for decades after the movement reached a
> furor).
>
> The commentary in the commit message is a plain and simple description
> of the effects of the grammarian movement to someone who has studied
> the topic.
>
> Strunk & White is just one possible edifice of that history. As
> mentioned already in this thread, it is not the name of the authors
> that is the problem, but the movement and history of Standard English
> that is the edifice of white supremacy. You cannot evaluate the book
> strictly outside of the context in which it was written and used and
> declare it's not white supremacist.
>
> In summary:
>
> The thing being objected to was the idea that we should choose
> Standard English as our basis for our language guide. Further, S&W, a
> classical work on how to write Standard English well, is a bad guide
> when discussing in light of that fact. Each individual who likes
> Elements of Style is not wrong for liking the book, you can keep it on
> your shelf and no one will be angry. But this argument about Standard
> English is propping up a hegemony that affects multiple axes of
> oppression, and we should be aware of that.

No we don't. Who are you to tell others what they should be aware of
or what they should fight for? And why should I join your battle that
I don't even agree with? Most of us probably had no idea what Elements
of Style was before this thread started, and *none* of us has ever
attributed any racial meaning to it in the 19 years since PEP-8 was
put in place. I still don't attribute any racial meaning to it,
despite all the twisted explanations which were given about the
correlation with white supremacy, which AFAIK are only shared by you
and the person who pushed this PR, who is also a newcomer. If anybody
can come up here for the first time, impose their world view on the
majority just "because it's oppressive", and do this sort of
pandemonium, then we can kiss Python goodbye. It also means there
virtually is no end to this in the long run, because anyone can come
up with any sort of funny theory in order to tilt the direction of the
language and deviate its culture from the outside.

--
Giampaolo - gmpy.dev
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/24G5JXOJTB7YM5BZWHMTLJHIZLBWXTPQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Giampaolo Rodola']

On Tue, Jun 30, 2020 at 3:16 PM Thomas Wouters  wrote:
>
>
>
> On Tue, Jun 30, 2020 at 2:36 PM Steven D'Aprano  wrote:
>>
>> It needs to be pointed out that Thomas Wouters was recently re-elected
>> to the PSF board. I think we need to know whether Thomas speaks for the
>> entire PSF board.Giampaolo feared this:
>>
>> "It's gonna happen again and again, until everybody gets in line, shuts
>> up or leaves due to exhaustion."
>>
>> and Thomas replied:
>>
>> "I'm not sure how much more clear python-dev and the PSF could have been
>> that this is true."
>>
>> So is it true? Do the core devs and the PSF have a policy of
>> pushing divisive political changes to silence and force out of the
>> community people who are guilty of thought-crime and holding the wrong
>> opinion on political matters?
>
>
> I wasn't speaking for the PSF or the Steering Council, nor was my intent to 
> "silence or force out people guilty of thought-crime and holding the wrong 
> opinion".
>
>>
>>
>> I think that Thomas' post violated the CoC:
>
>
> Please report all CoC violations to the CoC WG.

Please don't. As far as I'm concerned, me and Thomas are fine. Also,
as I said, I don't like CoCs nor appealing to them.

-- 
Giampaolo - gmpy.dev
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CP6ILO3OVZ6SLOHJUZKWDTCMOOCYQFTG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Giampaolo Rodola']

On Tue, Jun 30, 2020 at 11:53 AM Thomas Wouters  wrote:
> On Tue, Jun 30, 2020 at 4:28 AM Giampaolo Rodola'  wrote:
>>
>> This is not about the commit message. It’s way more than that. It's been 
>> going on non-stop and got increasingly worse since at least the preparation 
>> of the Python elections ~2 years ago. It is not normal what is going on 
>> here. People are scared. And it is pretty much guaranteed that this is not 
>> gonna be the last occurrence of it. On the horizon we have other 
>> language-related controversies like "whitelist" / "blacklist", renaming 
>> "master" to "main" in GIT, and who knows what else (maybe "whitespace"? or 
>> @property?).
>
>
> I don't have words for the irony of complaining about changing words while 
> objecting to the wording in a commit message.

They are two different things.
One thing is changing some words to make a concept more clear. I said
I agree with it.
Another thing is using that as an excuse to deliver a political
message. That I don't agree with.
I'm tired of it. I'm literally overwhelmed by it. I open Twitter and I
see politics. I open Facebook and I see politics. That's fine. But why
should I see politics also here? I don't know.

>> And every time that's gonna happen the motivation is gonna be about white 
>> supremacy/privilege/guilt etc. Because it's always about that, and we'll be 
>> having this discussion once again. On one hand Python gladly takes our 
>> patches and everything is smooth,
>
>
> I'm not sure who 'our' is in this sentence, but I'm certainly not glad Python 
> ever took any of your patches. No contribution to Python outweighs the harm 
> you're doing by espousing and advocating for these views. This kind of 
> sentiment scares away a lot of valuable contributors -- I know this because 
> *they have told me* --

I can say exactly the same thing. There are different people in this
thread and other threads who publicly said they are uncomfortable with
this situation. They don't espouse or advocate for any view in
particular. And me neither, because complaining about X doesn't
necessarily mean wanting to push for Y. I don't want to push for X nor
Y. And I don't want to be put in a situation where I am forced to
advocate for X or Y, or be vilified if I don't agree with X as it
happens here. It's just not the right place because it's too close to
the personal sphere (work, etc.). Some texted me privately exactly as
they did with you, because they are afraid of repercussions in that
regard (work). Others posted anonymously for the same reason. Does
that seem normal, sane or "welcoming" to you? Do you think it's
helping anybody anywhere? It is not. It is not me who's doing this *in
the least*. I'm merely calling it out.

>> on the other hand it wants us to not only accept "this" and be quiet, but 
>> also to take a stand and be an ally in the battle against the vocabulary "or 
>> else". So what's the point of contributing if the emotional distress and the 
>> risk that comes with it are so high?
>
>
> This is exactly why we want you to stop, yes. You're causing a lot of 
> emotional distress in people, and putting people at risk. You're even causing 
> it in people *in your purported demographic*, like me, let alone the people 
> you're trying to disadvantage. Stop it.

I'm not putting anybody at risk except myself.

>> In the previous discussion preceding this one where one PSF member left 
>> because it all got so political, somebody posted anonymously (and gently) 
>> for fear of repercussions. The same fear has been expressed in this thread. 
>> In the other thread it has even been suggested that "being silent re. > cause>" == "being complicit". I mean, are you serious? I explicitly avoided 
>> to comment on that because I didn't even know where to begin to explain how 
>> profoundly wrong that is on so many different levels. How irrespectful it is 
>> to ask people who just want to contribute some code here to take precise 
>> political sides or be damned if they don't. How unfair it is to do that 
>> especially towards old-time contributors. And now I even have to hope some 
>> moderator will be reasonable enough not to mark my emails as "white 
>> suprematism" (LOL) and send them through. This is just ridiculous. I've 
>> never been pro-CoC, but even if I were, this is what the enforcement part of 
>> the CoC dictates:
>>
>> > https://www.python.org/psf/conduct/enforcement/
>> > Reports that ar

[Python-Dev] Re: [Python-ideas] Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Giampaolo Rodola']

This is not about the commit message. It’s way more than that. It's been
going on non-stop and got increasingly worse since at least the preparation
of the Python elections ~2 years ago. It is not normal what is going on
here. People are scared. And it is pretty much guaranteed that this is not
gonna be the last occurrence of it. On the horizon we have other
language-related controversies like "whitelist" / "blacklist", renaming
"master" to "main" in GIT, and who knows what else (maybe "whitespace"? or
@property?). And every time that's gonna happen the motivation is gonna be
about white supremacy/privilege/guilt etc. Because it's always about that,
and we'll be having this discussion once again. On one hand Python gladly
takes our patches and everything is smooth, on the other hand it wants us
to not only accept "this" and be quiet, but also to take a stand and be an
ally in the battle against the vocabulary "or else". So what's the point of
contributing if the emotional distress and the risk that comes with it are
so high? In the previous discussion preceding this one where one PSF member
left because it all got so political, somebody posted anonymously (and
gently) for fear of repercussions. The same fear has been expressed in this
thread. In the other thread it has even been suggested that "being silent
re. " == "being complicit". I mean, are you serious? I
explicitly avoided to comment on that because I didn't even know where to
begin to explain how profoundly wrong that is on so many different levels.
How irrespectful it is to ask people who just want to contribute some code
here to take precise political sides or be damned if they don't. How unfair
it is to do that especially towards old-time contributors. And now I even
have to hope some moderator will be reasonable enough not to mark my emails
as "white suprematism" (LOL) and send them through. This is just
ridiculous. I've never been pro-CoC, but even if I were, this is what the
enforcement part of the CoC dictates:

> https://www.python.org/psf/conduct/enforcement/
> Reports that are not made in good faith (such as "reverse sexism" or
"reverse racism") may receive no response.

...so even the CoC won't help. So this is why this problem is more profound
than a simple commit message. It's gonna happen again and again, until
everybody gets in line, shuts up or leaves due to exhaustion.


On Mon, 29 Jun 2020 at 22:28, David Mertz  wrote:

> Can we simply revise the commit message to something neutral like "Removed
> specific reference to Strunk and White in favor of generic urge for
> language clarity."
>
> That's all the change actually was; there's no need for the other debate
> or broad political background.
>
> On Mon, Jun 29, 2020 at 3:28 PM Rhodri James  wrote:
>
>> On 29/06/2020 17:24, Abdur-Rahmaan Janhangeer wrote:
>> > Threads like these are meaningless, does not provide any learning
>> > value and is nowhere near the single vs double quote thread.
>>
>> I'm afraid I couldn't disagree more.  Since the PSF has seen fit to make
>> a political statement (re Black Lives Matter, and I don't particularly
>> disagree with either the statement or the choice of making it), threads
>> like these are both inevitable and necessary.  When such statements are
>> made, it is generally a good idea to be reasonably sure that the
>> community one is representing is broadly OK with that statement.  (I
>> speak in vague terms because you will never get 100% agreement from
>> anyone on anything!)
>>
>> The commit message that sparked this all was, quite unnecessarily, a
>> political statement.  The threads have demonstrated that it is not even
>> vaguely universally accepted, so it being in the PEPs repository (not
>> just a PR, it's there, public, and effectively representing you and me)
>> is a problem.  That it's still there now is pretty unacceptable in my
>> book.
>>
>> --
>> Rhodri James *-* Kynesim Ltd
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/HP2NJGITNURNCP4XOSUO7Y65IUUIX6KM/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> The dead increasingly dominate and strangle both the living and the
> not-yet born.  Vampiric capital and undead corporate persons abuse
> the lives and control the thoughts of homo faber. Ideas, once born,
> become abortifacients against new conceptions.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JOPTN4NMBEP3PBQDBGLEIR5GIKV37RLD/
> Code of Conduct: http://python

[Python-Dev] Re: [Python-ideas] Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Giampaolo Rodola']

On Mon, Jun 29, 2020 at 12:34 PM Nathaniel Smith  wrote:

> On Mon, Jun 29, 2020 at 2:31 AM Steve Holden  wrote:
> > The commit message used, however, reveals implementation details of the
> change which are irrelevant to the stated aim, which is making the
> documentation clear and concise. Use of such language is certainly
> regrettable, since it carries with it the implication that the Python
> developer community has somehow been wilfully sanctioning "relics of white
> supremacy" up until the change was made.
> >
> > There certainly is a place in tech for politics, as I have argued many
> times, and I am sure nobody wishes to continue to use language that might
> be offensive to readers. But I would suggest that the politics can safely
> be omitted from commit messages, since they can only properly be fully
> addressed in the conversation about the PR in advance. The wording of the
> commit message has the appearance (probably specious) of wanting to rub
> former misdeeds in the face of a largely innocent community, and that is
> the principal reason I found it distasteful and unnecessary.
>
> I just re-read the commit message, and I think you're being
> oversensitive and imagining things that aren't there. The actual
> commit message is written in a straightforward and factual way, and
> spends special effort on *absolving* the community of this kind of
> guilt.
>

"The community" has nothing to be absolved of, "Strunk & White" has nothing
to do with white supremacy and there is no guilt. If you feel guilty
because you're white then that's your problem. I don't feel guilty for
being white, the same way a black person should not feel guilty for being
black. And I have literally ZERO excuses to make to you or anybody else in
here because I'm white. Assuming guilt based on the color of your skin and
constantly attacking that specific group because of that is racist. It's
that simple. I find it astonishing how some people here don't seem to
realize that (or pretend not to).

And what's the goal anyway? Make us all feel guilty, create yet another
heated discussion, widen divisions, wait for the occasional folks who dare
to speak up against this vitriol and kick them out? And then what? What is
the plan here exactly? Don't you folks realize this is a technical forum?
Don't you understand how inappropriate it is to constantly bring up these
kinds of messages up here, and force people to either witness them silently
for fear of repercussions, or to engage in the discussion and risk paying
the consequences in terms of work / hiring / career / status / reputation
etc.? Because that's what happens, and we all know it. This is a very
public forum and we can all be traced back to here. There are professionals
here, people who go to conferences and/or make a living out of Python, who
pay the rent and support their family with it, and that don't want to be
put in this position.

It does not scale. It will never scale. Because whether we like it or not
we have to coexist together in this virtual space, including with people we
don't like. And this is why it is such a good idea to leave politics out of
the door and only stay focused on Python. We will still have different
opinions and occasional clashes, but as long as they are technical they
will be genuine, prolific and everything will be fine as it was before
"this" started (I've been reading this list for 12 years now). Discussing
politics, on the other hand, will only keep bringing conflict over and over
again. There's tons of proof of this already, and I can't envision a
different outcome in the long run. Because most of us are not OK with being
put against a wall and being blamed for "supremacy", "guilt", "privilege"
or whatever term you have in your jargon. I certainly am not. Furthermore,
that jargon makes no sense outside of the US and it's just ridiculous. I'm
European, am split between living here and in Asia, and I can guarantee you
that much.

Please, stop this.

-
Giampaolo - gmpy.dev 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TMLKHLDLDFE3JC3M6HOCUW4RIPQEXEIP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Map errno==ETIME to TimeoutError

[Giampaolo Rodola']

Sigh! I misread this discussion and thought the proposal was to add
TimeoutError which I forgot existed. Sorry for the noise and please
disregard my previous message.

On Mon, May 25, 2020 at 4:32 PM Giampaolo Rodola' 
wrote:

> I'm -1 because the concept of "timeout" is generic enough to be often
> implemented as a custom exception, which poses questions re.
> backward/forward compatibilty. E.g. in psutil I have "TimeoutExpired", also
> providing a "seconds" attribute. Also I've probably never seen ETIME /
> ETIMEDOUT happening, whereas AFAIU the point PEP 3151 was to create
> mappings for the most common errnos.
>
> On Sun, May 24, 2020 at 6:48 PM Eric V. Smith  wrote:
>
>> Does anyone have an opinion on https://bugs.python.org/issue39673? It
>> maps ETIME to TimeoutError, in addition to the already existing ETIMEDOUT.
>>
>> http://man7.org/linux/man-pages/man3/errno.3.html says:
>>
>>*ETIME   *Timer expired (POSIX.1 (XSI STREAMS option)).
>>
>>(POSIX.1 says "STREAM ioctl(2) 
>> <http://man7.org/linux/man-pages/man2/ioctl.2.html> timeout".)
>>
>>*ETIMEDOUT   *Connection timed out (POSIX.1-2001).
>>
>>
>> It seems like a reasonable change to me, but I'm not a subject matter
>> expert on STREAMS, or what other affect this might have.
>>
>> And if added to 3.10, should it be backported? I'd tend to say "no",
>> because of unknown impacts on existing code.
>>
>> Eric
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/C7KK6VSGPQKPA5IUCZ2MHH7QNLP2Q5QX/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> Giampaolo - http://grodola.blogspot.com
>
>

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BRVOG53QLBOM727IMOXBE7C7HEWYOBHI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Map errno==ETIME to TimeoutError

[Giampaolo Rodola']

Sure. Done.

On Mon, 25 May 2020 at 21:05, Eric V. Smith  wrote:

> Thanks, Giampaolo. Could you leave a comment on the issue?
>
> Erci
> On 5/25/2020 10:32 AM, Giampaolo Rodola' wrote:
>
> I'm -1 because the concept of "timeout" is generic enough to be often
> implemented as a custom exception, which poses questions re.
> backward/forward compatibilty. E.g. in psutil I have "TimeoutExpired", also
> providing a "seconds" attribute. Also I've probably never seen ETIME /
> ETIMEDOUT happening, whereas AFAIU the point PEP 3151 was to create
> mappings for the most common errnos.
>
> On Sun, May 24, 2020 at 6:48 PM Eric V. Smith  wrote:
>
>> Does anyone have an opinion on https://bugs.python.org/issue39673? It
>> maps ETIME to TimeoutError, in addition to the already existing ETIMEDOUT.
>>
>> http://man7.org/linux/man-pages/man3/errno.3.html says:
>>
>>*ETIME   *Timer expired (POSIX.1 (XSI STREAMS option)).
>>
>>(POSIX.1 says "STREAM ioctl(2) 
>> <http://man7.org/linux/man-pages/man2/ioctl.2.html> timeout".)
>>
>>*ETIMEDOUT   *Connection timed out (POSIX.1-2001).
>>
>>
>> It seems like a reasonable change to me, but I'm not a subject matter
>> expert on STREAMS, or what other affect this might have.
>>
>> And if added to 3.10, should it be backported? I'd tend to say "no",
>> because of unknown impacts on existing code.
>>
>> Eric
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/C7KK6VSGPQKPA5IUCZ2MHH7QNLP2Q5QX/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> Giampaolo - http://grodola.blogspot.com
>
> --
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DJXIGSGZ2G7OJYPWZWH265FDP3PP3U6B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Map errno==ETIME to TimeoutError

[Giampaolo Rodola']

I'm -1 because the concept of "timeout" is generic enough to be often
implemented as a custom exception, which poses questions re.
backward/forward compatibilty. E.g. in psutil I have "TimeoutExpired", also
providing a "seconds" attribute. Also I've probably never seen ETIME /
ETIMEDOUT happening, whereas AFAIU the point PEP 3151 was to create
mappings for the most common errnos.

On Sun, May 24, 2020 at 6:48 PM Eric V. Smith  wrote:

> Does anyone have an opinion on https://bugs.python.org/issue39673? It
> maps ETIME to TimeoutError, in addition to the already existing ETIMEDOUT.
>
> http://man7.org/linux/man-pages/man3/errno.3.html says:
>
>*ETIME   *Timer expired (POSIX.1 (XSI STREAMS option)).
>
>(POSIX.1 says "STREAM ioctl(2) 
>  timeout".)
>
>*ETIMEDOUT   *Connection timed out (POSIX.1-2001).
>
>
> It seems like a reasonable change to me, but I'm not a subject matter
> expert on STREAMS, or what other affect this might have.
>
> And if added to 3.10, should it be backported? I'd tend to say "no",
> because of unknown impacts on existing code.
>
> Eric
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/C7KK6VSGPQKPA5IUCZ2MHH7QNLP2Q5QX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B4Y2UZ3Z7ARHVP5JTP27UV2OH2PSDZ4P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Detect memory leaks in unit tests

[Giampaolo Rodola']

On Wed, May 13, 2020 at 9:17 AM Ammar Askar  wrote:

>  > Py_DECREF calls in the C code
>
> I think this part specifically is already covered through refleak
> checks:
> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py
>
> Since it can involve the repetition of tests many times, these aren't
> run on the CI though, they do get run on the refleak buildbots so
> issues get caught eventually:
> https://buildbot.python.org/all/#/builders?tags=%2Brefleak
>
> But again this is for PyObjects only. Were you able to find any memory
> leaks with your proof-of-concept? I don't think there's a lot of
> chances of someone missing a PyMem_Free call and there's not a lot of
> other manual memory management but I could be wrong. Anything found
> there could help motivate adding this a bit more.


Yeah, I agree it depends on how many PyMem_* occurrences are there, and it
probably makes more sense to cover those ones only. Under Modules/* I found:

- 24 occurrences for PyMem_RawMalloc
- 2 for PyMem_RawCalloc
- 106 for PyMem_Malloc
- 12 for PyMem_Calloc
- 39 for PyMem_New
- 5 for " = malloc("

I spent an hour covering around 20 of those and didn't find any leak. It's
a boring work. I will try to work on it over the next few weeks.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C7ZZRDPGIUS7Q6Q4AS4YFPD2OOF56JBO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Detect memory leaks in unit tests

[Giampaolo Rodola']

Hello there,
I would like to discuss a proposal regarding one aspect which AFAIK is
currently missing from cPython's test suite: the ability to detect memory
leaks of functions implemented in the C extension modules.
In psutil I use a test class/framework which calls a function many times,
and fails if the process memory increased after doing so. I do this in
order to quickly detect missing free() or Py_DECREF calls in the C code,
but I suppose there may be other use cases. Here's the class:
https://github.com/giampaolo/psutil/blob/913d4b1d6dcce88dea6ef9382b93883a04a66cd7/psutil/tests/__init__.py#L901

Detecting a memory leak is no easy task, and that's because the process
memory fluctuates. Sometimes it may increase (or even decrease!) even if
there's no leak, I suppose because of how the OS handles memory, the
Python's garbage collector, the fact that RSS is an approximation, and who
knows what else. In order to compensate fluctuations I did the following:
in case of failure (mem > 0 after calling fun() N times) I retry the test
for up to 5 times, increasing N (repetitions) each time, so I consider the
test a failure only if the memory keeps increasing across all runs. So for
instance, here's a legitimate failure:


psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_disk_partitions
...
Run #1: extra-mem=696.0K, per-call=3.5K, calls=200
Run #2: extra-mem=1.4M, per-call=3.5K, calls=400
Run #3: extra-mem=2.1M, per-call=3.5K, calls=600
Run #4: extra-mem=2.7M, per-call=3.5K, calls=800
Run #5: extra-mem=3.4M, per-call=3.5K, calls=1000
FAIL

If, on the other hand, the memory increased on one run (say 200 calls) but
decreased on the next run (say 400 calls), then it clearly means it's a
false positive, because memory consumption may be > 0 on the second run,
but if it's lower than the previous run with less repetitions, then it
cannot possibly represent a leak (just a fluctuation):


psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_net_connections
...
Run #1: extra-mem=568.0K, per-call=2.8K, calls=200
Run #2: extra-mem=24.0K, per-call=61.4B, calls=400
OK

This is the best I could come up with as a simple leak detection mechanism
to integrate with CI services, and keep more advanced tools like Valgrind
out of the picture (I just wanted to know if there's a leak, not to debug
the leak itself). In addition, since psutil is able to get the number of
fds (UNIX) and handles (Windows) opened by a process, I also run a separate
set of tests to make sure I didn't forget to call close(2) or CloseHandle()
in C.

Would something like this make sense to have in cPython? Here's a quick PoC
I put together just to show how this thing would look like in practice:
https://github.com/giampaolo/cpython/pull/2/files
A proper work in terms of API coverage would result being quite huge (test
all C modules), and ideally should also include cases where functions raise
an exception when being fed with an improper input. The biggest stopper
here is, of course, psutil, since it's a third party dep, but before
getting to that I wanted to see how this idea is perceived in general.

Cheers,

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NFHW4TP3ALY3CVRBVKRI4SRG7BOLZLJH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Dangerous default value for reuse_address in asyncio loop.create_datagram_endpoint()

[Giampaolo Rodola']

SO_REUSEADDR was controversial also for socket.create_server(). In the end
I concluded the best solution was to not expose a reuse_address parameter.
See:
https://github.com/python/cpython/blob/94e165096fd65e8237e60de570fb609604ab94c9/Lib/socket.py#L891-L899

It must be noted that right now also asyncio's create_server() method
allows passing reuse_address=True, which on Windows should probably be
turned into a no-op.

As for asyncio's create_datagram_endpoint() I partly agree with Antoine's
solution.
https://bugs.python.org/issue37228#msg357068
My course of action, though, would be the following:

* in 3.8: turn reuse_address parameter into a no-op, update doc
* in 3.9: raise error if reuse_address=True, update doc

Note: differently from TCP / create_server(), with UDP you can set
SO_REUSEADDR manually after calling create_datagram_endpoint() if you
really want to.

On Tue, Nov 19, 2019 at 4:48 AM  wrote:

> When creating UDP servers with asyncio's create_datagram_endpoint(), the
> default value for reuse_address = True, resulting in a dangerous (and
> currently incorrectly documented) situation. I have proposed changing the
> default value but understandably such a change for a core library function
> parameter is not to be taken lightly. Thus I put this up for discussion on
> the list.
>
> As background, when creating TCP servers on UNIX-like systems, it is
> almost boilerplate to set SO_REUSEADDR for all server sockets to make sure
> that a restarting server can immediately bind the socket again. Without the
> SO_REUSEADDR sockopt, the kernel will hold the addr:port in a TIME_WAIT
> state for a while, preventing reuse. Thus, when creating TCP servers with
> loop.create_server(), the parameter reuse_address has a very reasonable
> default value of True.
>
> However things are very different in UDP-land. The kernel does not hold
> UDP server ports in a waiting state so the SO_REUSEADDR sockopt was
> repurposed in Linux (and *BSD afaik) to allow multiple processes to bind
> the SAME addr:port for a UDP server. The kernel will then feed incoming UDP
> packets to all such processes in a semi-fair-roundrobin manner. This is
> very useful in some scenarios, for example I've used it myself in C++
> projects to allow UDP servers to be scaled easily and rolling upgrades to
> be implemented without separate load-balancing. But for this to be the
> default behaviour is quite dangerous.
>
> I discovered this default behaviour accidentally by having 2 separate
> Python programs (both doing SIP over UDP) accidentally configured to use
> the same UDP port. The result was that my 2 processes were indeed "sharing
> the load" - neither of them threw an exception at startup about the port
> being already in use and both started getting ~half of the incoming
> packets. So off to the docs I went and discovered that the documentation
> for create_datagram_endpoint() does not mention this behaviour at all,
> instead it mistakenly refers to the TCP protocol use of SO_REUSEADDR:
> "reuse_address tells the kernel to reuse a local socket in TIME_WAIT state,
> without waiting for its natural timeout to expire. If not specified will
> automatically be set to True on Unix."
>
> https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.create_datagram_endpoint
>
> What makes this default especially dangerous is,
> - Most people are not aware of this special mode that Linux allows for UDP
> sockets
> - Even if it was documented to be the default, many people would miss it
> unless a big warning was slapped on the docs
> - The problems are unlikely to appear in test scenarios and much more
> likely to pop up in production months or years after rolling out the code
> - If you have never used it on purpose, it is very confusing to debug,
> causing you to doubt your own and the kernel's sanity
> - The behaviour changes again if you happen to use a multicast address...
>
> Thus, my proposal is to change the default value for UDP to False or
> deprecate the function and introduce a new one as suggested by Yuri in my
> original bug report at:  https://bugs.python.org/issue37228
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TK2NTPWID7RBUUNUU5JYAZHR6FKEABRU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GSWPWB63DF4AT6I4R6O4MCJ5A52UZO3T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] Currently working on the release of Python 3.8.0rc1

[Giampaolo Rodola']

Hello Łukasz,
I consider this one critical enough to get into 3.8:
https://bugs.python.org/issue38319
Long story short: shutil.copyfile() and socket.sendfile() are broken on
32-bit platforms for files >= 2GiB.
shutil.copyfile() was modified by me in the 3.8 cycle so the bug only
affects 3.8 and 3.9.


On Mon, Sep 30, 2019 at 3:48 PM Łukasz Langa  wrote:

> Team,
> amazing job on getting us back on track over the weekend.
>
> *Thank you*
> All release blockers and deferred release blockers solved. And there was
> relatively little additional activity on the branch -- as expected at this
> point! Thank you for this, it will help get the release candidate out on
> time.
>
> *I'm working on cutting RC1 today*
> Hopefully all sanity checks, as well as building the source tarball and
> the binary installers for macOS and Windows will all work out fine and
> we'll be seeing RC1 out tonight.
>
> *RC2 and the date of 3.8.0 gold*
> If we manage to avoid the need for RC2, we will be able to release 3.8.0
> on October 14th. If we need an RC2, that will slip by a week. I hope we
> won't. Ideally RC1 should be identical codewise to 3.8.0.
>
> To help our chances, please avoid any source-related activity on the 3.8
> branch from now until the release of 3.8.0. Yes, that also counts for bug
> fixes unless they are critical. (Yeah, it might be a bit annoying but
> nobody wants to be the person who introduced a last minute regression in a
> major release.)
>
> Note I didn't say I forbid any activity. I have no power over you,
> actually. More importantly though, I trust your judgement if you assess
> some bug is bad enough the fix absolutely has to get into 3.8.0. Moreover,
> I specifically said source-related activity because...
>
> 3.8.0 is important. To some users it will be the first and the last
> release in the 3.8 series they will see.
>
> *We need you to focus on the docs now*
> Are all your changes properly documented?
> Did you notice other changes you know of to have insufficient
> documentation?
> Can you help with the "What's New" document?
>
> anxiously configure-&&-makingly y'rs
> - Ł
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-committ...@python.org/message/ACMKEQNGK4FVUIZ6TYD5H26OSPIO5GSN/
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VC7XNBK65YBNUHKXC7PNMJQCU5P5YJ4X/

[Python-Dev] Re: Announcing the new Python triage team on GitHub

[Giampaolo Rodola']

If the concern is pruning old/stale/invalid issues/PRs, I think the
proposal to add a specific "awaiting-close" (or similar) label should work.
These issues can periodically be handled by a core-dev who can click on the
label, then evaluate and bulk-close them in one shot if necessary. Also, it
seems to me that including "devguide" and "core-workflow" repos is
unnecessary since they are low-traffic administrative repos related to our
internal workflow, they are attended by a small portion of the core-dev
team only and labels don't play a central role as in the cpython repo.
Please let's keep in mind that this role implies higher responsibilities
than a usual contributor, and since the bar for being promoted is
considerably lower than a core-dev's, triager permissions should reflect
that and remain minimal. For instance, I don't think a triager should be
able to edit other users' comments or lock conversations, but I'm afraid
GitHub doesn't provide that level of granularity (correct?).

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2YYB6MAJ55U2U4KTKINFDSB4USJ3NMZY/

[Python-Dev] Re: Announcing the new Python triage team on GitHub

[Giampaolo Rodola']

On Wed, 21 Aug 2019 at 22:17, Raymond Hettinger 
wrote:

> Thanks for doing this.  I hope it encourages more participation.
>
> The capabilities of a triager mostly look good except for "closing PRs and
> issues".  This is a superpower that has traditionally been reserved for
> more senior developers because it grants the ability to shut-down the work
> of another aspiring contributor.  Marking someone else's suggestion as
> rejected is the most perilous and least fun aspect of core development.
> Submitters tend to expect their idea won't be rejected without a good deal
> of thought and expert consideration.   Our bar for becoming a triager is
> somewhat low, so I don't think it makes sense to give the authority to
> reject a PR or close an issue.


Agreed. Closing an issue basically means having the authority to reject a
proposal, which really is not different than accepting it. It's probably
the second highest responsibility after being able to merge a PR. The risk
is to potentially reject a good proposal without having the expertise and
authority to properly evaluate it, which is why we have the core-dev role.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/STV4WIJJ3LBTKQSOOWPLS6EAAUKWN75M/

Re: [Python-Dev] Python in next Windows 10 update

[Giampaolo Rodola']

On Wed, 22 May 2019 at 03:30, Steve Dower  wrote:

> Hi all
>
> Just sharing this here because I think it's important for us to be aware
> of it - I'm not trying to promote or sell anything here :) (Those who
> were at the language summit have seen this already.)
>
> In the next Windows 10 update that starts rolling out today, we
> (Microsoft) have added "python.exe" and "python3.exe" commands that are
> installed on PATH *by default* and will open the Microsoft Store at the
> page where we (Python core team) publish our build.
>
> This makes it a 1-2 click process to get from a clean machine to having
> a usable Python install ("python.exe" -> opens Store -> "Get it Free" ->
> "python.exe" now works!)
>
> The associated blog post:
>
>
> https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/
>
> Here are answers to a few questions that I assume will come up, at least
> from this audience that understands the issues better than most:
>
> * if someone had installed Python and put it on PATH with our installer,
> this new command *does not* interfere
> * if someone had manually modified their own PATH, they *may* see some
> interference (but we [Microsoft] decided this was an acceptable risk)
> * the Python 3.7 installed from the store will not auto-update to 3.8,
> but when 3.8 is released we (Microsoft) will update the redirect to
> point at it
> * if you pass arguments to the redirect command, it just exits with an
> error code - you only get the Store page if you run it without arguments
> * once the Store package is installed, the redirect command is replaced
> (this required a new feature in the OS). If you install with the regular
> installer and update PATH, or active a venv, it will add it *before* the
> redirect. So these scenarios should be all good.
>
> I'm happy to answer other questions here. The long-term contact for this
> integration is python (at) microsoft.com, which right now will come to me.
>
> And on a personal note, I'm very excited that we (Microsoft) got the
> approval to do this. Getting *anything* added to Windows is a big task,
> so it's a reflection of the popularity and support for Python that's
> growing within Microsoft that we were able to make this happen. That's
> due to every contributor, both to the core runtime and the ecosystem. I
> hope this will only help us improve the availability of Python for users
> and make it an easier choice for dev tasks in the future.
>
> Cheers,
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com



>
I am really glad this happened. I think that in a sense this could be
considered sort of historical.
-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Giampaolo Rodola']

On Tue, 21 May 2019 at 04:30, Antoine Pitrou  wrote:

>
> NNTP is still quite used (often through GMane, but probably not only) so
> I'd question the removal of nntplib.


I concur nntplib should be left alone. There are possibly even less used
network protocols such as telnet (tenetlib) which are not targeted by this
PEP but could have been by following the same logic. FTP is another one
that despite no longer popular, still has a constant user base (you’d be
surprised by the amount of traffic we got over FTP in the last company I
worked for). Overall, I think the bar for a module removal should be set
very high, especially for “standard” things such as these network
protocols, that despite being old are not likely to change. That means that
also the maintenance burden for python-dev will be low or close to none
after all.

It seems to me also spwd/crypt modules fall into this category (all UNIX
platforms have the shadow password db, which is nice to have in python out
of the box). In that case the removal appears to be more justified by the
security implications than them being not widely used though, so I would
use more caution and treat them differently (eg. opt for doc warning +
investigate a possible replacement). Also note that spwd could be used to
parse /etc/passwd, despite I admit its primary use case is password
checking. Certain users may even not care about the additional security
provided by PAM (eg. internal devop scripts).
-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Giampaolo Rodola']

On Tue, 21 May 2019 at 23:26, Christian Heimes  wrote:

> On 21/05/2019 18.08, Giampaolo Rodola' wrote:
> >
> >
> > On Tue, 21 May 2019 at 21:13, Christian Heimes  <mailto:christ...@python.org>> wrote:
> >
> > crypt
> > ~
> >
> > The `crypt <https://docs.python.org/3/library/crypt.html>`_ module
> implements
> > password hashing based on ``crypt(3)`` function from ``libcrypt`` or
> > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old,
> of poor
> > quality and insecure. Users are discouraged to use them.
> >
> > * The module is not available on Windows. Cross-platform application
> need
> >   an alternative implementation any way.
> > * Only DES encryption is guarenteed to be available. DES has an
> extremely
> >   limited key space of 2**56.
> > * MD5, salted SHA256, salted SHA512, and Blowfish are optional
> extension.
> >   SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the
> only
> >   algorithm that is still secure. However it's in glibc and
> therefore not
> >   commonly available on Linux.
> > * Depending on the platform, the ``crypt`` module is not thread
> safe. Only
> >   implementations with ``crypt_r(3)`` are thread safe.
> > * The module was never useful to interact with system user and
> password
> >   databases.
> >
> >
> > This is actually not true. Their main use case is to compare passwords
> against the shadowed password db:
> >
> https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L413
> > A quick search on searchcode.com <http://searchcode.com> shows both
> spwd and crypt modules are used. I am no security expert (and I wasn’t
> aware they are insecure until now, since the doc doesn’t mention it) but I
> would prefer seeing these 2 fixed or improved rather than bluntly removed.
>
> No, the statement is correct. I may have to explain this even further.
>
> The approach in pyftpdlib is the wrong and IMO deserves a CVE. The crypt()
> + spwd() approach is flawed on multiple levels. For example it bypasses
> account restriction, access control, and login session. It also requires
> you to run the service as root and with SELinux disabled or an unconfined
> context -- a bad combination. There is only one correct way to perform a
> credential check: use PAM.
>
> spwd can't be fixed. It could only be replaced with a completely different
> API that wraps PAM and Windows's authentication API.
>
> Christian
>
> PS: Authentication, authorization, and identity management are part of my
> day job at Red Hat.
>

Got it. I had no idea. Since you mentioned the CVE it looks like spwd/crypt
doc deserve a warning. This is probably out of the scope of the PEP, but I
wonder if the 3 third-party alternatives mentioned in the PEP are mature
enough and could be evaluated for stdlib inclusion (the part re. PAM /
password-checking at least). Perhaps spwd/crypt could be deprecated in 3.8
and the alternative added in 3.9 before the 3.10 removal.


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Giampaolo Rodola']

On Tue, 21 May 2019 at 21:13, Christian Heimes  wrote:

> crypt
> ~
>
> The `crypt `_ module
> implements
> password hashing based on ``crypt(3)`` function from ``libcrypt`` or
> ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor
> quality and insecure. Users are discouraged to use them.
>
> * The module is not available on Windows. Cross-platform application need
>   an alternative implementation any way.
> * Only DES encryption is guarenteed to be available. DES has an extremely
>   limited key space of 2**56.
> * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension.
>   SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only
>   algorithm that is still secure. However it's in glibc and therefore not
>   commonly available on Linux.
> * Depending on the platform, the ``crypt`` module is not thread safe. Only
>   implementations with ``crypt_r(3)`` are thread safe.
> * The module was never useful to interact with system user and password
>   databases.


This is actually not true. Their main use case is to compare passwords
against the shadowed password db:
https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L413
A quick search on searchcode.com shows both spwd and crypt modules are
used. I am no security expert (and I wasn’t aware they are insecure until
now, since the doc doesn’t mention it) but I would prefer seeing these 2
fixed or improved rather than bluntly removed.

> --
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Giampaolo Rodola']

On Tue, 21 May 2019 at 03:17, Christian Heimes  wrote:

spwd
> 
>
> The `spwd `_ module provides
> direct access to Unix shadow password database using non-standard APIs.
> In general it's a bad idea to use the spwd. The spwd circumvents system
> security policies, it does not use the PAM stack, and is
> only compatible with local user accounts.
>
> Module type
>   C extension
> Deprecated in
>   3.8
> To be removed in
>   3.10
> Substitute
>   **none**


I find this one useful and would be a bit sad to see it go. FWIW I use it
in pyftpdlib and I suppose there are other apps out there relying on UNIX
password db for authentication. The fact that it’s a C module is also an
incentive to leave it in the stdlib IMO (pure python modules can easily be
copied in the project instead of retrieving them from PYPI as a third party
dep - e.g. this is how I am likely going to replace asyncore/asynchat).

A big +1 to this PEP by the way.
-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove tempfile.mktemp()

[Giampaolo Rodola']

On Tue, Mar 19, 2019 at 6:21 PM Guido van Rossum  wrote:

>> On Tue, Mar 19, 2019 at 10:14 AM Giampaolo Rodola'  
>> wrote:
>> Technically you cannot make it 100% safe, only less likely to occur.
> That seems unnecessarily pedantic. A cryptographic random generator, like the 
> one in the secrets module, *is* safe enough for all practical purposes (with 
> a large margin).
> [...]
> Hm, the random sequence (implemented in tempfile._RandomNameSequence) is 
> currently derived from the random module, which is not cryptographically 
> secure. Maybe all we need to do is replace its source of randomness with one 
> derived from the secrets module. That seems a one-line change.

Seems to make sense to me. Currently the random string is 8 chars
long, meaning (if I'm not mistaken) max 40320 possible combinations:

>>> len(list(itertools.permutations('wuoa736m'
40320

We may use 9 chars and get to 362880 and increase "security".
Regardless, it seems to me the original deprecation warning for
mktemp() was not really justified and should be removed. IMO it
probably makes more sense to just clarify in the doc that
NamedTemporaryFile is the right / safe way to create a file with a
unique name and avoid the theoretical, rare name collision you'd get
by using open(mktemp(), 'w') instead. Also I'm not sure I understand
what the code sample provided in the doc aims to achieve:
https://docs.python.org/3/library/tempfile.html#tempfile.mktemp
The way I see it, the main (or "correct") use case for mktemp() is
when you explicitly want a file name which does not exist, in order to
exercise legitimate scenarios like these ones:

>>> self.assertRaises(FileNotFoundError, os.unlink, tempfile.mktemp())

>>> dst = tempfile.mktemp()
>>> os.rename(src, dst)
>>> assert not os.path.exists(src) and os.path.exists(dst)

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove tempfile.mktemp()

[Giampaolo Rodola']

On Tue, 19 Mar 2019 at 17:47, Sebastian Rittau  wrote:

> Am 19.03.19 um 17:23 schrieb Giampaolo Rodola':
> > @Sebastian
> >> If there are valid use cases for mktemp(), I recommend renaming
> >> it to mkname_unsafe() or something equally obvious.
> > I'm -1 about adding an alias (there should be one and preferably only
> > one way to do it). Also mkstemp() and mkdtemp() are somewhat poorly
> > named IMO, but I wouldn't add an alias for them either.
> >
> Just to clarify: I was not suggesting creating an alias, I was suggesting
> renaming the function, but keeping the old name for a normal
> deprecation cycle.
>
> But I had another thought: If I understand correctly, the exploitability
> of mktemp() relies on the fact that between determining whether the
> file exists and creation an attacker can create the file themselves.
> Couldn't this problem be solved by generating a filename of sufficient
> length using the secrets module? This way the filename should be
> "unguessable" and safe.


Technically you cannot make it 100% safe, only less likely to occur. And on
a second thought (I retract :)) since this could be used in real apps other
than tests (I was too focused on that) I think this should be a doc warning
after all, not info. Doc may suggest to use mode=x when creating the file,
in order to remove the security implications.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove tempfile.mktemp()

[Giampaolo Rodola']

On Tue, Mar 19, 2019 at 3:57 PM Antoine Pitrou  wrote:
>
>
> Le 19/03/2019 à 15:52, Guido van Rossum a écrit :
> >
> > If all you need is a random name, why not just use a random number
> > generator?
> > E.g. I see code like this:
> >
> > binascii.hexlify(os.urandom(8)).decode('ascii')
>
> mktemp() already does this, probably in a better way, including the
> notion of prefix, suffix, and parent directory.  Why should I have to
> reimplement it myself?

On top of that mktemp() tries exists() in a loop, diminishing the risk
of names collision.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove tempfile.mktemp()

[Giampaolo Rodola']

On Tue, Mar 19, 2019 at 2:06 PM Stéphane Wirtel  wrote:
>
> Hi,
>
> Context: raise a warning or remove tempfile.mktemp()
> BPO: https://bugs.python.org/issue36309
>
> Since 2.3, this function is deprecated in the documentation, just in the
> documentation. In the code, there is a commented RuntimeWarning.
> Commented by Guido in 2002, because the warning was too annoying (and I
> understand ;-)).
>
> So, in this BPO, we start to discuss about the future of this function
> and Serhiy proposed to discuss on the Python-dev mailing list.
>
> Question: Should we drop it or add a (Pending)DeprecationWarning?
>
> Suggestion and timeline:
>
> 3.8, we raise a PendingDeprecationWarning
> * update the code
> * update the documentation
> * update the tests
>   (check a PendingDeprecationWarning if sys.version_info == 3.8)
>
> 3.9, we change PendingDeprecationWarning to DeprecationWarning
>   (check DeprecationWarning if sys.version_info == 3.9)
>
> 3.9+, we drop tempfile.mktemp()
>
> What do you suggest?

I concur with others who think this should not be removed. It's used
in different stdlib and third party modules' test suites. I see
tempfile.mktemp() very similar to  test.support.find_unused_port() and
os.path.exists/isfile/isdir functions: they are all subject to race
conditions but still they are widely used and have reason to exist
(practicality beats purity). I suggest turning the doc deprecation
into a note:: or warning::, so that extra visibility is maintained.
Because the use case is legitimate and many fs-related APIs such as
this one are inevitably racy, I lean more towards a note:: rather than
a warning:: though, and we could probably do the same for os.path.is*
functions.

@Sebastian
> If there are valid use cases for mktemp(), I recommend renaming
> it to mkname_unsafe() or something equally obvious.

I'm -1 about adding an alias (there should be one and preferably only
one way to do it). Also mkstemp() and mkdtemp() are somewhat poorly
named IMO, but I wouldn't add an alias for them either.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (Licensing question) backport of shutil.copyfile() functionality

[Giampaolo Rodola']

On Tue, Mar 12, 2019 at 6:29 AM Terry Reedy  wrote:

> On 3/11/2019 10:54 PM, Inada Naoki wrote:
>
> >> Hello,
> >> some time ago I contributed a couple of patches to speedup
> shutil.copy*() functions:
> >> https://bugs.python.org/issue33671
> >> https://bugs.python.org/issue33695
>
> You retain copyright on the code you contributed.
> https://mail.python.org/mailman/listinfo/python-dev
>

I didn't write shutil.copytree()'s code though, and I'd need to copy it.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (Licensing question) backport of shutil.copyfile() functionality

[Giampaolo Rodola']

On Tue, Mar 12, 2019 at 3:01 AM Glenn Linderman 
wrote:

> On 3/11/2019 4:35 PM, Giampaolo Rodola' wrote:
>
> Hello,
> some time ago I contributed a couple of patches to speedup shutil.copy*()
> functions:
> https://bugs.python.org/issue33671
> https://bugs.python.org/issue33695
> I would like to backport both functionalities so that they can be used on
> Python 2.7 and <3.8 and put it on PYPI. In order to do so I will basically
> have to copy some parts of shutil module (copytree() function + the
> unit-tests I added in BPO-33671 and a couple of other things). Are there
> constraints regarding this in terms of license? Am I supposed to use GPL?
> (I was thinking about using MIT)
>
> Note: in this package called "zerocopy" I will probably want to expose
> other functionalities such as tee(), splice() and CopyFileEx and
> TransmitFile on Windows, so it's probably gonna be half a backport and half
> a brand new project.
>
> Thanks.
>
>
> Thanks for the contributions. I don't know about the licensing.
>
> I wonder if you should make two packages, though... one just exactly a
> backport of the shutil speedups, and the second containing the new
> functionalities.
>

That was my initial thought as well (a "backports.shutil_copy" module
targeting copy* functions only), but (especially after playing with this
today) I think I have something a bit more ambitious in mind. I'm currently
experimenting with different things which could be baked in a third-party
lib and possibly contributed back to Python later on:

1) on OSX we could use f/copyfile() syscall to copy file attrs/metadata;
that may be useful to speedup shutil.copystat() and shutil.copymode()
2) copytree() on OSX could take advantage of f/copyfile() +
COPYFILE_RECURSIVE (which is probably too platform-specific for inclusion)
3) on Linux we could use copy_file_range() as a replacement for
os.sendfile() in shutil.copyfile() (it's supposed to be faster)
4) on Linux ioctl() + FICLONE could be used to implement CoW (copy on
write) instantaneous copies, and could be added as shutil.cow_copyfile()
(haven't look into Windows yet)
5) I was thinking about backporting socket.socket.sendfile() as well, which
uses os.sendfile() on POSIX but not TransmitFile on Windows (asyncio does
though)
6)  another idea (but I'm not sure if it's possible, as I still have to dig
into that) is a new socket recvfile() function boosted up by tee() /
splice() on Linux, which maybe could be contributed back as
socket.socket.recvfile()

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] (Licensing question) backport of shutil.copyfile() functionality

[Giampaolo Rodola']

Hello,
some time ago I contributed a couple of patches to speedup shutil.copy*()
functions:
https://bugs.python.org/issue33671
https://bugs.python.org/issue33695
I would like to backport both functionalities so that they can be used on
Python 2.7 and <3.8 and put it on PYPI. In order to do so I will basically
have to copy some parts of shutil module (copytree() function + the
unit-tests I added in BPO-33671 and a couple of other things). Are there
constraints regarding this in terms of license? Am I supposed to use GPL?
(I was thinking about using MIT)

Note: in this package called "zerocopy" I will probably want to expose
other functionalities such as tee(), splice() and CopyFileEx and
TransmitFile on Windows, so it's probably gonna be half a backport and half
a brand new project.

Thanks.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEPs from non-core devs now need a sponsor

[Giampaolo Rodola']

On Wed, Mar 6, 2019 at 8:58 PM Abdur-Rahmaan Janhangeer <
arj.pyt...@gmail.com> wrote:

> i think that "should have at least a mentor guiding you" sounds a lot more
> better than
>
> a core developer needs to sign on to be a sponsor
>
> that sounds a lot more that without backing, you can't submit a pep, i
> guess the core devs wanted to make things easier but the sponsor thing etc
> put me off.
> for someone using py, ideas sometimes come but since i've not yet
> submitted a pep, when i see a change in the flow, i ask: will it be easier
> or more difficult to submit peps now? i really got the impression that now
> chances are slimmer.
>

Before submitting a PEP one (including core-devs) usually starts a
discussion on python-dev/ideas in order to start collecting feedback and,
most importantly, to figure out whether the idea deserves a PEP or not
(often times it doesn't). If the proposal is good it means somebody agreed
with you: those persons are likely gonna be the ones who'll likely sponsor
your PEP. If you can't find such a person immediately and the idea received
positive feedback I imagine you'll just be asked to write a proto-PEP
first, and if that is good enough somebody will eventually sponsor it and
possibly even help you. If you can't find any person then it probably means
it wasn't such a good idea, and that's also good because it will save you
from the trouble of writing the PEP in the first place (meeting the PEP
quality standards is not exactly a piece of cake). I doubt we'll end up in
a situation where a good proposal won't happen just because nobody is
willing to sponsor it.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asking for reversion

[Giampaolo Rodola']

On Sun, Feb 24, 2019 at 5:09 AM Davin Potts <
python+python_...@discontinuity.net> wrote:

> I have done what I was asked to do:  I added tests and docs in a new
> PR (GH-11816) as of Feb 10.
>
> Since that time, the API has matured thanks to thoughtful feedback
> from a number of active reviewers.  At present, we appear to have
> stabilized around an API and code that deserves to be exercised
> further.  To get that exercise and because there are currently no
> outstanding objections, I am merging the PR to get it into the second
> alpha.  There will undoubtedly be further revisions and adjustments.


Nice job! It wasn't easy to abstract such a low level interface.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding test.support.safe_rmpath()

[Giampaolo Rodola']

On Thu, Feb 14, 2019 at 6:48 PM Brett Cannon  wrote:

>
> With -j you can do parallel testing and I know I always run with that on.
> But TESTFN does *attempt *to account for that
> 
> by using the PID in the name.
>

Good to know, thanks. TESTFN aside, I was more interested in knowing if
there's interest in landing something like this in test.support:

def rmpath(path):
"Try to remove a path regardless of its type."
if os.path.isdir(path):
test.support.rmtree(path)
elif os.path.exists(path):
test.support.unlink(path)

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding test.support.safe_rmpath()

[Giampaolo Rodola']

On Thu, Feb 14, 2019 at 4:03 PM Tim Golden  wrote:

> On 14/02/2019 14:56, Giampaolo Rodola' wrote:
> >
> >
> > On Thu, Feb 14, 2019 at 3:25 PM Eric Snow  > <mailto:ericsnowcurren...@gmail.com>> wrote:
> >
> > On Thu, Feb 14, 2019, 02:47 Ronald Oussoren via Python-Dev
> > mailto:python-dev@python.org> wrote:
> >
> >
> > I usually use shutil.rmtree for tests that need to create
> > temporary files, and create a temporary directory for those
> > files (that is, use tempfile.mkdtemp in setUp() and use
> > shutil.rmtree in tearDown()). That way I don’t have to adjust
> > house-keeping code when I make changes to test code.
> >
> >
> > Same here.
> >
> > -eric
> >
> >
> > What I generally do is avoid relying on tempfile.mkdtemp() and always
> > use TESTFN instead. I think it's cleaner as a pradigm because it's an
> > incentive to not pollute the single unit tests with  `self.addCleanup()`
> > instructions (the whole cleanup logic is always supposed to occur in
> > setUp/tearDown):
>
> Must chime in here because I've been pushing (variously months & years
> ago) to move *away* from TESTFN because it generates numerous
> intermittent errors on my Windows setup. I've had several goes at
> starting to do that but a combination of my own lack of time plus some
> people's reluctance to go that route altogether has stalled the thing.
>
> I'm not sure I understand the difference in cleanup/teardown terms
> between using tempfile and using TESTFN. The objections I've seen from
> people (apart, obviously, from test churn) are to do with building up
> testing temp artefacts on a possibly low-sized disk.
>
> TJG
>

I suppose you mean the intermittent failures are usually due to "file is
already in use by another process" correct? test.support's unlink(),
rmdir() and rmtree() functions already implement a retry-with-timeout logic
in order to prevent this issue. I suppose when this issue may still occur,
though, is when the file/handle is held by another process, meaning that
the unit-test probably forgot to terminate()/wait() a subprocess or should
have used support.read_children(). In summary, my approach is more "strict"
because it implies that unit-tests always do a proper cleanup.
tempfile.mkdtemp() may prevent failures but it may hide a unit-test which
doesn't do a proper file/dir cleanup and should have been fixed instead.
The drawback in practical terms is that orphaned test files are left behind.

Extra: an argument in favor of using tempfile.mkdtemp() instead of TESTFN
is parallel testing, but I think we're not using it.


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding test.support.safe_rmpath()

[Giampaolo Rodola']

On Thu, Feb 14, 2019 at 3:25 PM Eric Snow 
wrote:

> On Thu, Feb 14, 2019, 02:47 Ronald Oussoren via Python-Dev <
> python-dev@python.org wrote:
>
>>
>> I usually use shutil.rmtree for tests that need to create temporary
>> files, and create a temporary directory for those files (that is, use
>> tempfile.mkdtemp in setUp() and use shutil.rmtree in tearDown()). That way
>> I don’t have to adjust house-keeping code when I make changes to test code.
>>
>
> Same here.
>
> -eric
>
>>
What I generally do is avoid relying on tempfile.mkdtemp() and always use
TESTFN instead. I think it's cleaner as a pradigm because it's an incentive
to not pollute the single unit tests with  `self.addCleanup()` instructions
(the whole cleanup logic is always supposed to occur in setUp/tearDown):


TESTFN = support.TESTFN
TESTFN2 = TESTFN + '2'

class FilesystemTest(unittest.TestCase):

def setUp(self):
remove_file_or_dir(TESTFN)
remove_file_or_dir(TESTFN2)
tearDown = setUp

def test_mkdir(self):
...
def test_listdir(self):
...
def test_rename(self):
...

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding test.support.safe_rmpath()

[Giampaolo Rodola']

On Wed, Feb 13, 2019 at 2:32 PM Victor Stinner  wrote:

> Bikeshedding: I suggest to remove "safe_" from the name, it's hard to
> guarantee that removal is "safe", especially on Windows where a
> removal can be blocked for many reasons.
>
> Victor
>

Agree. I actually meant "rmpath()" (which I used in my examples) but I
mispelled that in the mail title. =)


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding test.support.safe_rmpath()

[Giampaolo Rodola']

On Wed, Feb 13, 2019 at 2:27 PM Ronald Oussoren 
wrote:

>
>
> On 13 Feb 2019, at 13:24, Giampaolo Rodola'  wrote:
>
>
> Hello,
> after discovering os.makedirs() has no unit-tests (
> https://bugs.python.org/issue35982) I was thinking about working on a PR
> to increase the test coverage of fs-related os.* functions. In order to do
> so I think it would be useful to add a convenience function to "just delete
> something if it exists", regardless if it's a file, directory, directory
> tree, etc., and include it into test.support module.
>
>
> Something like shutil.rmtree() with ignore_errors=True?
>

shutil.rmtree() is about directories and can't be used against files.
support.rmpath() would take a path (meaning anything) and try to remove it.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Adding test.support.safe_rmpath()

[Giampaolo Rodola']

Hello,
after discovering os.makedirs() has no unit-tests (
https://bugs.python.org/issue35982) I was thinking about working on a PR to
increase the test coverage of fs-related os.* functions. In order to do so
I think it would be useful to add a convenience function to "just delete
something if it exists", regardless if it's a file, directory, directory
tree, etc., and include it into test.support module. Basically it would be
very similar to "rm -rf". I use something like this into psutil:
https://github.com/giampaolo/psutil/blob/3ea94c1b8589891a8d1a5781f0445cb5080b7c3e/psutil/tests/__init__.py#L696
I find this paradigm especially useful when testing functions involving two
files ("src" and "dst"). E.g. in case of os.renames() unit-tests I would
write something like this:


class RenamesTest(unittest.TestCase):
srcname = support.TESTFN
dstname = support.TESTFN + '2'

def setUp(self):
test.support.rmpath(self.srcname)
test.support.rmpath(self.dstname)
tearDown = setUp

def test_rename_file(self):
...
def test_rename_dir(self):
...
def test_rename_failure(self):
# both src and dst will not exist
...

With the current utilities included in test.support the setUp function
above would be written as such:

def setUp(self):
for path in (self.srcname, self.dstname):
if os.path.isdir(path):
test.support.rmtree(path)
elif os.path.exists(path):
test.support.unlink(path)

Extra: one may argue whether this utility could be included into shutil
module instead. The extra advantage of test.support.rmtree and
test.support.unlink though, is that on Windows they use a timeout, catching
"file is currently in use" exceptions for some time before giving up. That
IMO would probably make this utility function not palatable for inclusion
into shutil module, so test.support would probably be a better landing
place.

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asking for reversion

[Giampaolo Rodola']

On Wed, Feb 6, 2019 at 12:51 PM Giampaolo Rodola' 
wrote:

>
> Unless they are already there (I don't know) it would be good to have a
> full set of unit-tests for all the register()ed types and test them against
> SyncManager and SharedMemoryManager. That would give an idea on the real
> interchangeability of these 2 classes and would also help writing a
> comprehensive doc.
>

In order to speed up the alpha2 inclusion process I created a PR which
implements what said above:
https://github.com/python/cpython/pull/11772
https://bugs.python.org/issue35917
Apparently SharedMemoryManager works out of the box and presents no
differences with SyncManager, but the list type is not using ShareableList.
When I attempted to register it with "SharedMemoryManager.register('list',
list, ShareableList)" I got the following error:

Traceback (most recent call last):
  File "foo.py", line 137, in test_list
o = self.manager.list()
  File "/home/giampaolo/svn/cpython/Lib/multiprocessing/managers.py", line
702, in temp
proxy = proxytype(
TypeError: __init__() got an unexpected keyword argument 'manager'

I am not sure how to fix that (I'll leave it up to Davin). The tests as-is
are independent from PR-11772 so I suppose they can be reviewed/checked-in
regardless of the changes which will affect shared_memory.py.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asking for reversion

[Giampaolo Rodola']

Davin,
I am not familiar with the multiprocessing module, so take the following
with a big grain of salt. I took a look at the PR, then I got an idea of
how multiprocessing module is organized by reading the doc. Here's some
food for thought in terms of API reorganization.

SharedMemoryManager, SharedMemoryServer
---

It appears to me these are the 2 main public classes, and after reading the
doc it seems they really belong to "managers
"
(multiprocessing.managers namespace). Also:
* SharedMemoryManager is a subclass of multiprocessing.managers.SyncManager
* SharedMemoryServer is a subclass of multiprocessing.managers.Server
shared_memory.py could be renamed to _shared_memory.py and managers.py
could import and expose these 2 classes only.

Support APIs


These are objects which seem to be used in support of the 2 classes above,
but apparently are not meant to be public. As such they could simply live
in _shared_memory.py and not be exposed:

- shareable_wrap(): used only in SharedMemoryTracker.wrap()
- SharedMemoryTracker: used only by SharedMemoryServer
- SharedMemory, WindowsNamedSharedMemory, PosixSharedMemory: used by
shareable_wrap() and SharedMemoryTracker
- ShareableList: it appears this is not used, but by reading here

I have a doubt: shouldn't it be register()ed against SharedMemoryManager?

C extension module
--

- ExistentialError, Error - it appears these are not used
- PermissionsException, ExistentialException - I concur with Ronald
Oussoren's review: you could simply use PyErr_SetFromErrno() and let the
original OSError exception bubble up. Same for O_CREAT, O_EXCL, O_CREX,
O_TRUNC which are already exposed in the os module. I have a couple of
other minor nitpicks re. the code but I will comment on the PR.

Compatibility
-

I'm not sure if SyncManager and SharedMemoryManager are fully
interchangeable so I think the doc should clarify this. SyncManager handles
a certain set of types
.
It appears SharedMemoryManager is supposedly able to do the same except for
lists
.
Is my assumption correct? Also, multiprocessing.Manager() by default
returns a SyncManager. If we'll get to a point where SyncManager and
SharedMemoryManager are able to handle the same types it'd be good to
return SharedMemoryManager as the default, but it's probably safer to leave
it for later. Unless they are already there (I don't know) it would be good
to have a full set of unit-tests for all the register()ed types and test
them against SyncManager and SharedMemoryManager. That would give an idea
on the real interchangeability of these 2 classes and would also help
writing a comprehensive doc.

Hope this helps.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asking for reversion

[Giampaolo Rodola']

On Mon, Feb 4, 2019 at 4:21 AM Davin Potts <
python+python_...@discontinuity.net> wrote:

> I am attempting to do the right thing and am following the advice of other
> core devs in what I have done thus far.
>
> Borrowing heavily from what I've added to issue35813 just now:
>
> This work is the result of ~1.5 years of development effort, much of it
> accomplished at the last two core dev sprints.  The code behind it has been
> stable since September 2018 and tested as an independently installable
> package by multiple people.
>
> I was encouraged by Lukasz, Yury, and others to check in this code early,
> not waiting for tests and docs, in order to both solicit more feedback and
> provide for broader testing.  I understand that doing such a thing is not
> at all a novelty.
>

Actually it is a novelty (you should wait for review and approval). The
main problem I have with this PR is that it seems to introduce 8 brand new
APIs, but since there is no doc, docstrings or tests it's unclear which
ones are supposed to be used, how or whether they are supposed to supersede
or deprecate older (slower) ones involving inter process communication. The
introduction of new APIs in the stdlib is a sensitive topic because once
they get in they stay in, so a discussion should occur early on,
definitively not at alphaX stage. Don't mean to point fingers here, the
goal in itself (zero-copy, a topic I recently contributed to myself for the
shutil module) is certainly valuable, but I concur and think this change
should be reverted and post-poned for 3.9.

-- 
Giampaolo - http://grodola.blogspot.com
-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tests failing on Windows with TESTFN

[Giampaolo Rodola']

On Fri, Jul 27, 2018 at 4:48 PM Chris Jerdonek  wrote:
>
> On Fri, Jul 27, 2018 at 6:41 AM, Giampaolo Rodola'  wrote:
> >
> > Being TESTFN a global name it appears not suited for parallel testing.
>
> It was designed for parallel testing though:
>
> # Disambiguate TESTFN for parallel testing, while letting it remain a valid
> # module name.
> TESTFN = "{}_{}_tmp".format(TESTFN, os.getpid())
>
> https://github.com/python/cpython/blob/aee632dfbb0abbc0d2bcc988c43a736afd568c55/Lib/test/support/__init__.py#L807-L809

Oh, nice, I didn't notice that, sorry.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tests failing on Windows with TESTFN

[Giampaolo Rodola']

On Thu, Jul 26, 2018 at 12:16 AM Tim Golden  wrote:
>
> I'm just easing back into core development work by trying to get a
> stable testing environment for Python development on Windows.
>
> One problem is that certain tests use support.TESTFN (a local directory
> constructed from the pid) for output files etc. However this can cause
> issues on Windows when recreating the folders / files for multiple
> tests, especially when running in parallel.
>
> Here's an example on my laptop deliberately running 3 tests with -j0
> which I know will generate an error about one time in three:
>
> C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2
> test_importlib
>
> Running Debug|Win32 interpreter...
> Run tests in parallel using 6 child processes
> 0:00:23 [1/3/1] test_urllib2 failed
> test test_urllib2 failed -- Traceback (most recent call last):
>File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line
> 821, in test_file
>  f = open(TESTFN, "wb")
> PermissionError: [Errno 13] Permission denied: '@test_15564_tmp'
>
> Although these errors are both intermittent and fairly easily spotted,
> the effect is that I rarely get a clean test run when I'm applying a patch.
>
> I started to address this years ago but things stalled. I'm happy to
> pick this up again and have another go, but I wanted to ask first
> whether there was any objection to my converting tests to using tempfile
> functions which should avoid the problem?
>
> TJG

>From my experience open() raising PermissionDenied on Windows often
means "file is already in use" as I think is this case. The same issue
exists for directories: https://bugs.python.org/issue33240.

Being TESTFN a global name it appears not suited for parallel testing.
Windows makes this more noticeable than POSIX as it's more strict, but
accessing the same file from 2 unit tests is technically a problem
regardless of the platform. I don't think that means we should ditch
TESTFN in favor of tempfile.mktemp() though. Instead the file cleanup
functions (support.unlink() and support.rmtree()) may be more clever
and (important) they should always be used in setUp / tearDown. For
instance, the traceback you pasted refers to a test class which
doesn't do this.

In psutil I've had occasional Windows failures like this for years
then I got tired of it and came up with this:
https://github.com/giampaolo/psutil/blob/1b09b5fff78f705dfb42458726ff9789c26f6f21/psutil/tests/__init__.py#L686
...which basically aggressively retries os.unlink or shutil.rmtree for
1 sec in case of (any) error, and I haven't had this problem since
then.

I suppose test.support's unlink() and rmtree() can do something
similar, maybe just by using a better exception handling, and all unit
tests should use them in setUp / tearDown. I think this will diminish
the occasional failures on Windows, although not completely.

--
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Call for prudence about PEP-572

[Giampaolo Rodola']

Fair enough. I will. Sorry for the extra pressure at this particular stage.

On Mon, 9 Jul 2018 at 00:06, Guido van Rossum  wrote:

> Since you CC'ed me explicitly I feel compelled to respond. I have read
> your reasoning, and I simply don't agree with it. A few months ago I would
> have happily explained why (there is a reason) but given the endless debate
> we've already seen I am simply too tired for another long email. Please
> give me the benefit of the doubt. The sky is not falling.
>
> On Sun, Jul 8, 2018 at 2:27 PM Giampaolo Rodola' 
> wrote:
>
>> On Sun, Jul 8, 2018 at 7:24 PM Chris Angelico  wrote:
>> >
>> > On Mon, Jul 9, 2018 at 3:14 AM, Giampaolo Rodola' 
>> wrote:
>> > >
>> > >
>> > > On Sun, Jul 8, 2018 at 6:45 PM Steve Holden 
>> wrote:
>> > >>
>> > >> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' <
>> g.rod...@gmail.com>
>> > >> wrote:
>> > >>>
>> > >>> [...]
>> > >>> I find that (space between the parentheses of a function call
>> statement)
>> > >>> too unnatural as a place where to put an assignment. It is not even
>> > >>> "guarded" by a keyword like "if" or  "while" which can help as
>> indicators
>> > >>> that an assignment may occur. Also, I think it's way too easy to
>> confuse it
>> > >>> with a keyword argument:
>> > >>>
>> > >>> >>> foo(x = 1)  # keyword arg
>> > >>> >>> foo(x := 1)  # assignment + value passing
>> > >>> [...]
>> > >>
>> > >>
>> > >> But the PEP 8 spellings are
>> > >>
>> > >> foo(x=1)
>> > >>
>> > >> and
>> > >>
>> > >>f(x := 1).
>> > >>
>> > >> The extra spacing makes it obvious that this isn't a regular named
>> > >> argument.
>> > >
>> > >
>> > > What if the author of the code I'm reading didn't respect PEP-8? I
>> don't
>> > > think it's fair to invoke PEP-8 as a counter-measure to obviate a
>> syntax
>> > > which can clearly be mistaken with something else simply by omitting 2
>> > > spaces. Not to mention that I don't see why anyone would want to
>> declare a
>> > > variable in there in the first place.
>> > >
>> >
>> > It's not about why someone would want to assign inside a function
>> > call. It's about why it should be forbidden. Perhaps nobody has a good
>> > reason to use THlS_OBJECT as a variable name, and it's potentially
>> > very confusing; but should the grammar of Python forbid it? No.
>> > Because there is no value in forbidding it.
>>
>> I'll try to give some reasons on why I think it should be forbidden.
>> In order of importance, more or less:
>>
>> 1) Because of readability and because it's ambiguous. foo(x := 1) and
>> foo(x = 1) are visually too similar and mean 2 completely different
>> things. And no, I don't think PEP-8 compliant variants are much
>> better:
>> >>> foo(x := 1)
>> >>> foo(x=1)
>> Good luck explaining the difference between the two to a beginner
>> including the introduction of PEP-8 concepts and why spaces are so
>> important in this case. The fact that spaces are more important here
>> than in other places alone makes me think this is unlikely a good
>> idea.
>>
>> 2) It's an incentive to write more compact code at the expense of
>> readability. I see := being used in "if" and "while" statements as
>> different in this regard. They imply an indented code block will
>> follow and the variables being set in the "if/while" statement will be
>> used right after that, supposedly in the very next line and *in that
>> block only*. This is what
>> https://github.com/python/cpython/pull/8122/files is all about,
>> really. There is no block in this case hence AFAICT this syntax is
>> only meant for writing one-liners.
>>
>> 3) := assignments in {if, while, yield, assert} statements are more
>> clearly noticeable because delimited by such keywords whereas a
>> function call can appear anywhere in the code. The only way to easily
>> track assignments such as foo(x := 1) is via linting.
>>
>> 4) IMO this is t

Re: [Python-Dev] Call for prudence about PEP-572

[Giampaolo Rodola']

On Sun, Jul 8, 2018 at 7:24 PM Chris Angelico  wrote:
>
> On Mon, Jul 9, 2018 at 3:14 AM, Giampaolo Rodola'  wrote:
> >
> >
> > On Sun, Jul 8, 2018 at 6:45 PM Steve Holden  wrote:
> >>
> >> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' 
> >> wrote:
> >>>
> >>> [...]
> >>> I find that (space between the parentheses of a function call statement)
> >>> too unnatural as a place where to put an assignment. It is not even
> >>> "guarded" by a keyword like "if" or  "while" which can help as indicators
> >>> that an assignment may occur. Also, I think it's way too easy to confuse 
> >>> it
> >>> with a keyword argument:
> >>>
> >>> >>> foo(x = 1)  # keyword arg
> >>> >>> foo(x := 1)  # assignment + value passing
> >>> [...]
> >>
> >>
> >> But the PEP 8 spellings are
> >>
> >> foo(x=1)
> >>
> >> and
> >>
> >>f(x := 1).
> >>
> >> The extra spacing makes it obvious that this isn't a regular named
> >> argument.
> >
> >
> > What if the author of the code I'm reading didn't respect PEP-8? I don't
> > think it's fair to invoke PEP-8 as a counter-measure to obviate a syntax
> > which can clearly be mistaken with something else simply by omitting 2
> > spaces. Not to mention that I don't see why anyone would want to declare a
> > variable in there in the first place.
> >
>
> It's not about why someone would want to assign inside a function
> call. It's about why it should be forbidden. Perhaps nobody has a good
> reason to use THlS_OBJECT as a variable name, and it's potentially
> very confusing; but should the grammar of Python forbid it? No.
> Because there is no value in forbidding it.

I'll try to give some reasons on why I think it should be forbidden.
In order of importance, more or less:

1) Because of readability and because it's ambiguous. foo(x := 1) and
foo(x = 1) are visually too similar and mean 2 completely different
things. And no, I don't think PEP-8 compliant variants are much
better:
>>> foo(x := 1)
>>> foo(x=1)
Good luck explaining the difference between the two to a beginner
including the introduction of PEP-8 concepts and why spaces are so
important in this case. The fact that spaces are more important here
than in other places alone makes me think this is unlikely a good
idea.

2) It's an incentive to write more compact code at the expense of
readability. I see := being used in "if" and "while" statements as
different in this regard. They imply an indented code block will
follow and the variables being set in the "if/while" statement will be
used right after that, supposedly in the very next line and *in that
block only*. This is what
https://github.com/python/cpython/pull/8122/files is all about,
really. There is no block in this case hence AFAICT this syntax is
only meant for writing one-liners.

3) := assignments in {if, while, yield, assert} statements are more
clearly noticeable because delimited by such keywords whereas a
function call can appear anywhere in the code. The only way to easily
track assignments such as foo(x := 1) is via linting.

4) IMO this is the main bit of PEP-572 which can be subject to serious
abuse. The value of "explicit is better than implicit" is an advanced
concept. We can't expect it can be grasped by everybody. Many folks
may be tempted to write one-liners at the top of a function and even
feel good about it because a bunch of lines were saved. Potentially *a
lot* of lines can be saved so it even more advanced users may be
tempted to use it. After all the language allows it +  "it's brand new
syntax so it must be good".

5) It has no keyword argument correspondence. If foo(x := 1) is
allowed then why this one is not?
>>> foo(x=(x := 1))
(I don't think it should BTW: it's not pretty)

6) Differently from := usage in {if, while, comprehensions} no valid
use case which would justify its usage has been provided so far.
AFAICT the only valid use case is for writing one-liners.

7) Defining something in there just feels wrong (to me at least)

8) I'm pretty sure any linter would emit a warning by default so why
bother adding support for a brand new syntax which we already know it
would be discouraged anyway?

9) It goes against half of the Python Zen.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Call for prudence about PEP-572

[Giampaolo Rodola']

On Sun, Jul 8, 2018 at 6:45 PM Steve Holden  wrote:

> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' 
> wrote:
>
>> ​[...]
>> I find that (space between the parentheses of a function call statement)
>> too unnatural as a place where to put an assignment. It is not even
>> "guarded" by a keyword like "if" or  "while" which can help as indicators
>> that an assignment may occur. Also, I think it's way too easy to confuse it
>> with a keyword argument:
>>
>> >>> foo(x = 1)  # keyword arg
>> >>> foo(x := 1)  # assignment + value passing
>> ​[...]
>>
>
> ​But the PEP 8 spellings are​
>
> foo(x=1)
>
> and
>
>f(x := 1).
>
> The extra spacing makes it obvious that this isn't a regular named
> argument.
>

What if the author of the code I'm reading didn't respect PEP-8? I don't
think it's fair to invoke PEP-8 as a counter-measure to obviate a syntax
which can clearly be mistaken with something else simply by omitting 2
spaces. Not to mention that I don't see why anyone would want to declare a
variable in there in the first place.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Call for prudence about PEP-572

[Giampaolo Rodola']

On Sat, Jul 7, 2018 at 5:48 PM Guido van Rossum  wrote:

> Enforcing such restrictions in the grammar would actually be complicated,
> due to nesting -- but even if it wasn't, I wouldn't want to, as I don't
> want to limit future generations to only the benefits of the new construct
> that we can now come up with. Orthogonality is a good thing in my mind,
> else we might never have had nested functions or conditional imports.
>

Got it.

As to why you might want to use := in a function call, I could imagine
> writing
>
> if validate(name := re.search(pattern, line).group(1)):
> return name
>

I meant this case:

>>> foo(x := 1)   # execute "x = 1" AND "foo(1)"

I find that (space between the parentheses of a function call statement)
too unnatural as a place where to put an assignment. It is not even
"guarded" by a keyword like "if" or  "while" which can help as indicators
that an assignment may occur. Also, I think it's way too easy to confuse it
with a keyword argument:

>>> foo(x = 1)  # keyword arg
>>> foo(x := 1)  # assignment + value passing

As for "assert" what I'm concern about is the proliferation of things like
this:

class Foo:
def __init__(self):
assert self.x := fun1()
assert self.y := fun2()
assert self.z := fun3()

When I look at that my brain tells me that the main subject of the line is
"assert", not the assignment, but maybe it's just because I'm not used to
it. That aside there's the question of what to do when "python -O" switch
is used. With this in place "-O" would acquire a new meaning, as it would
disable "assert" statements AND assignments.


> On Sat, Jul 7, 2018 at 6:12 AM Giampaolo Rodola' 
> wrote:
>
>> Sorry in advance for opening yet another topic about PEP-572. With
>> PEP-572 being officially accepted I know debating its inclusion in the
>> language is a useless exercise at this point, but since it's still in
>> "draft" status I would like to express my opinion as I think this is a
>> feature which can potentially be abused fairly easily. FWIW I initially
>> found myself disliking the idea as a whole but
>> https://github.com/python/cpython/pull/8122 made me (and others)
>> reconsider it quite a bit (see:
>> https://twitter.com/grodola/status/1015251302350245888). PR-8122 clearly
>> shows an improvement in expressiveness and compactness (many folks argue
>> this is too much) but PEP-572 as it currently stands is too permissive
>> IMHO. My concern about "easily abusable ugly cases" still remains, and I
>> think they should be banned instead of just discouraged in the PEP or in
>> the doc. Since we spend way more time *reading* code rather than writing
>> it, as a "reader" I would expect a more prudent approach to the problem.
>>
>> Proposal
>> 
>>
>> 1) allow only one := assignment per line in "if" statements:
>> >>> if x := val1 and y := val2:   # SyntaxError or SyntaxWarning
>> >>> if x == val1 and y := val2:   # SyntaxError or SyntaxWarning
>> >>> if x := val1 and y == val2:   # SyntaxError or SyntaxWarning
>> >>> if x := val1:  # OK
>> >>> if (x := val1):  # OK
>>
>> 2) allow := in "while" statements, "if" statements and comprehensions
>> only:
>> >>> foo(x := 0)  # SyntaxError
>> >>> yield x := 3  # SyntaxError
>> >>> assert y := 3  # SyntaxError
>>
>> 3) (debatable) disallow := if the variable is already defined:
>> >>> x = 5
>> >>> if (x := val):  # SyntaxError or SyntaxWarning
>>
>> 4) ban "a = (b := c)", "x = a := (b := (c := d))" and similar (they're
>> just too ugly IMHO)
>>
>> Rationale 1
>> ===
>>
>> In visual terms assignments in Python have always occurred at the
>> BEGINNING of the line and always on the most LEFT side:
>>
>> >>> foo = fun1()
>> >>> bar = fun2()
>> >>> baz = fun3()
>>
>> That is where I naturally expect an assignment to be when reading code.
>> My main concern with PEP-572 is that an assignments can now occur at *any
>> point* in the line:
>>
>> >>> foo = fun1()
>> >>> bar = fun2()
>> >>> if foo == val1 and bar == val2 and baz := fun3():
>> ......
>>
>> That forces me to visually scan the whol

[Python-Dev] Call for prudence about PEP-572

[Giampaolo Rodola']

Sorry in advance for opening yet another topic about PEP-572. With PEP-572
being officially accepted I know debating its inclusion in the language is
a useless exercise at this point, but since it's still in "draft" status I
would like to express my opinion as I think this is a feature which can
potentially be abused fairly easily. FWIW I initially found myself
disliking the idea as a whole but
https://github.com/python/cpython/pull/8122 made me (and others) reconsider
it quite a bit (see: https://twitter.com/grodola/status/1015251302350245888).
PR-8122 clearly shows an improvement in expressiveness and compactness
(many folks argue this is too much) but PEP-572 as it currently stands is
too permissive IMHO. My concern about "easily abusable ugly cases" still
remains, and I think they should be banned instead of just discouraged in
the PEP or in the doc. Since we spend way more time *reading* code rather
than writing it, as a "reader" I would expect a more prudent approach to
the problem.

Proposal


1) allow only one := assignment per line in "if" statements:
>>> if x := val1 and y := val2:   # SyntaxError or SyntaxWarning
>>> if x == val1 and y := val2:   # SyntaxError or SyntaxWarning
>>> if x := val1 and y == val2:   # SyntaxError or SyntaxWarning
>>> if x := val1:  # OK
>>> if (x := val1):  # OK

2) allow := in "while" statements, "if" statements and comprehensions only:
>>> foo(x := 0)  # SyntaxError
>>> yield x := 3  # SyntaxError
>>> assert y := 3  # SyntaxError

3) (debatable) disallow := if the variable is already defined:
>>> x = 5
>>> if (x := val):  # SyntaxError or SyntaxWarning

4) ban "a = (b := c)", "x = a := (b := (c := d))" and similar (they're just
too ugly IMHO)

Rationale 1
===

In visual terms assignments in Python have always occurred at the BEGINNING
of the line and always on the most LEFT side:

>>> foo = fun1()
>>> bar = fun2()
>>> baz = fun3()

That is where I naturally expect an assignment to be when reading code. My
main concern with PEP-572 is that an assignments can now occur at *any
point* in the line:

>>> foo = fun1()
>>> bar = fun2()
>>> if foo == val1 and bar == val2 and baz := fun3():
......

That forces me to visually scan the whole line horizontally from left to
right 'till its end, looking for possible variables being set. I'm
concerned that I will miss := occurrences because visually they are very
similar to == unless parentheses are made mandatory:

>>> if foo == val1 and bar == val2 and (baz := fun3()):
......

Also, in case of multi-line conditionals I have to visually scan the
construct both horizontally AND vertically:

>>> if (foo == val1 and \
...bar == val2 and \
...baz := val3):
... ...

Again, that is not a place where I would expect to find or look for a
variable assignment. I know I wouldn't like to read or review a code which
does that and I suspect linters will likely end up wanting to emit a
warning in that case (see: https://github.com/PyCQA/pylint/issues/2246).
https://github.com/python/cpython/pull/8116/files avoids using multiple :=
per line and that's why the result appears readable enough IMO.

Rationale 2
===

PEP-572 states:

> The := operator may be used directly in a positional function call
argument

That means allowing:

>>> foo(x := 0)

I honestly don't see why anyone would want to call a function AND assign a
variable value at the same time (except in comprehensions). With this in
place I not only have to guard against "if" statements assigning values at
any point in the code, but also function calls, both horizontally and
vertically e.g.:

>>> foo(some_long_var_name, another_one, x := bar(),
y := fun())

To me this looks like the perfect example of where this functionality can
be abused. Also, I'm not clear what PEP-572 intend to do about "all other
places". E.g. should these cases be allowed? (IMO no)

>>> yield x := 3
>>> assert y := 3

--
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Giampaolo Rodola']

I personally use a GIT commit hook which runs flake8 against the *modified
py files only* and rejects the commit in case of non-compliance:
https://github.com/giampaolo/psutil/blob/master/.git-pre-commit
...I install it via make:
https://github.com/giampaolo/psutil/blob/ad4acae5489f86fc3bef645505b3873f156b4867/Makefile#L186
...and once the whole code base is in a good shape have Travis run flake8
against the whole code base on each commit.

When I join an existing project I try to enforce the same workflow and rely
on pep8ify (https://github.com/spulec/pep8ify) to adjust the existing code.
In order to not be too "strict" I may modify the GIT hook to emit a warning
instead of rejecting the commit.

On one hand I think it would be great to do the same for cPython. On the
other hand I see 2 big downsides:

- losing "git blame" history for A LOT of lines of code
- even if you're familiar with PEP8 it's sometimes hard to spot
non-compliant lines unless you integrate flake8 into your IDE (I do for
Sublime); putting such a constraint on contributing users is probably too
much and may discourage contributions.


On Sun, Feb 25, 2018 at 6:13 AM, Guido van Rossum  wrote:

> It's easy to only analyze the files in the diff (these linters don't do
> cross-file analysis anyways, typically) and it's possible to write a filter
> that only keeps warnings about lines that are changed, but I don't know of
> a standard solution for the latter (places where I worked where I've seen
> this always had their own custom implementation).
>
> On Sat, Feb 24, 2018 at 7:35 PM, Mariatta Wijaya <
> mariatta.wij...@gmail.com> wrote:
>
>> Can any of these said linters analyze only the diff in the PR, instead of
>> the entire CPython codebase?
>>
>> Mariatta Wijaya
>>
>> ᐧ
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
>> 40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/g.
> rodola%40gmail.com
>
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reorganize Python categories (Core, Library, ...)?

[Giampaolo Rodola']

On Wed, Oct 4, 2017 at 11:52 AM, Victor Stinner 
wrote:

> Hi,
>
> Python uses a few categories to group bugs (on bugs.python.org) and
> NEWS entries (in the Python changelog). List used by the blurb tool:
>
> #.. section: Security
> #.. section: Core and Builtins
> #.. section: Library
> #.. section: Documentation
> #.. section: Tests
> #.. section: Build
> #.. section: Windows
> #.. section: macOS
> #.. section: IDLE
> #.. section: Tools/Demos
> #.. section: C API
>
> My problem is that almost all changes go into "Library" category. When
> I read long changelogs, it's sometimes hard to identify quickly the
> context (ex: impacted modules) of a change.
>
> It's also hard to find open bugs of a specific module on
> bugs.python.org, since almost all bugs are in the very generic
> "Library" category. Using full text returns "false positives".
>
> I would prefer to see more specific categories like:
>
> * Buildbots: only issues specific to buildbots
> * Networking: socket, asyncio, asyncore, asynchat modules
> * Security: ssl module but also vulnerabilities in any other part of
> CPython -- we already added a Security category in NEWS/blurb
> * Parallelim: multiprocessing and concurrent.futures modules
>
> It's hard to find categories generic enough to not only contain a
> single item, but not contain too many items neither. Other ideas:
>
> * XML: xml.doc, xml.etree, xml.parsers, xml.sax modules
> * Import machinery: imp and importlib modules
> * Typing: abc and typing modules
>
> The best would be to have a mapping of a module name into a category,
> and make sure that all modules have a category. We might try to count
> the number of commits and NEWS entries of the last 12 months to decide
> if a category has the correct size.
>
> I don't think that we need a distinct categoy for each module. We can
> put many uncommon modules in a generic category.
>
> By the way, we need maybe also a new "module name" field in the bug
> tracker. But then comes the question of normalizing module names. For
> example, should "email.message" be normalized to "email"? Maybe store
> "email.message" but use "email" for search, display the module in the
> issue title, etc.
>
> Victor


Personally I've always dreamed about having *all* module names. That would
reflect experts.rst file:
https://github.com/python/devguide/blob/master/experts.rst



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553: Built-in debug()

[Giampaolo Rodola']

> It's a lot to type (27 characters).

True. Personally I have a shortcut in my IDE (Sublime) so I when I type
"pdb" -> TAB it auto completes it.

> Python linters (e.g. flake8 [1]) complain about this line because it
contains two statements. Breaking the idiom up into two lines further
complicates the use of the debugger,

I see this more as an advantage. After all a pdb line is "temporary" and
you never want to commit it. When you do it is by accident so you want it
to be more noticeable. I can configure my IDE to use flake8 and highlight
the pdb line for me so that I can immediately see it because it's not PEP-8
compliant. I can have a GIT commit hook which checks there's no
"pdb.set_trace()" in the files I'm committing:
https://github.com/giampaolo/psutil/blob/master/.git-pre-commit#L93-L96
Somehow I think debug() would make this a bit harder as it's more likely a
"debug()" line will pass unnoticed.
For this reason I would give a -1 to this proposal.

> It ties debugging directly to the choice of pdb. There might be other
debugging options, say if you're using an IDE or some other development
environment.

Personally I would find it helpful if there was a hook to choose the
default debugger to use on "pdb.set_trace()" via .pdbrc or PYTHONDEBUGGER
environment variable or something.
I tried (unsuccessfully) to run ipdb on "pdb.set_trace()", I gave up and
ended up emulating auto completion and commands history with this:
https://github.com/giampaolo/sysconf/blob/master/home/.pdbrc.py


On Wed, Sep 6, 2017 at 9:14 AM, Barry Warsaw  wrote:

> I’ve written a PEP proposing the addition of a new built-in function
> called debug().  Adding this to your code would invoke a debugger through
> the hook function sys.debughook().
>
> Like the existing sys.displayhook() and sys.excepthook(), you can change
> sys.debughook() to point to the debugger of your choice.  By default it
> invokes pdb.set_trace().
>
> With this PEP instead of:
>
> foo()
> import pdb; pdb.set_trace()
> bar()
>
> you can write:
>
> foo()
> debug()
> bar()
>
> and you would drop into the debugger after foo() but before bar().  More
> rationale and details are provided in the PEP:
>
> https://www.python.org/dev/peps/pep-0553/
>
> Unlike David, but like Larry, I have a prototype implementation:
>
> https://github.com/python/cpython/pull/3355
>
> Cheers,
> -Barry
>
> P.S. This came to me in a nightmare on Sunday night, and the more I
> explored the idea the more it frightened me.  I know exactly what I was
> dreaming about and the only way to make it all go away was to write this
> thing up.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/g.
> rodola%40gmail.com
>
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] pythonhosted.org doc upload no longer works

[Giampaolo Rodola']

I know pythonhosted.org was deprecated long ago in favor of readthedocs but
I kept postponing it and my doc for psutil is still hosted on
https://pythonhosted.org/psutil/.

http://pythonhosted.org/ web page still recommends this:

>

I took a look at:
https://pypi.python.org/pypi?:action=pkg_edit&name=psutil
...but the form to upload the doc in zip format is gone.

The only available option is the "destroy documentation" button.
I would like to at least upload a static HTML page redirecting to the new
doc which I will soon move on readthedocs. Is that possible?

Thanks in advance.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Need help to fix urllib(.parse) vulnerabilities

[Giampaolo Rodola']

On Sat, Jul 22, 2017 at 7:10 PM, Giampaolo Rodola' 
wrote:

>
>
> On Sat, Jul 22, 2017 at 6:38 PM, Victor Stinner 
> wrote:
>
>> Le 22 juil. 2017 8:04 AM, "Serhiy Storchaka"  a
>> écrit :
>>
>> I think the only reliable way of fixing the vulnerability is rejecting or
>> escaping (as specified in RFC 2640) CR and LF inside sent lines. Adding the
>> support of RFC 2640 is a new feature and can be added only in 3.7. And this
>> feature should be optional since not all servers support RFC 2640.
>> https://github.com/python/cpython/pull/1214 does the right thing.
>>
>>
>> In that case, I suggest to reject newlines in ftplib, and maybe add an
>> opt-in option to escape newlines.
>>
>> Java just rejected newlines, no? Or does Java allows to escape them?
>>
>> Victor
>>
>>
> OK, let's just reject \n then and be done with it. It's a rare use case
> after all.
> Java just rejects \n for all commands and does not support escaping (aka
> RFC 2640).
>

I've just merged the PR. There's the question whether to backport this to
older versions, considering there's a small chance this may break some
code/apps, but considering the chance is small and this a security fix I'd
probably be +0.5 for backporting it (2.7 + 3.x - not sure up 'till when).

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Need help to fix urllib(.parse) vulnerabilities

[Giampaolo Rodola']

On Sat, Jul 22, 2017 at 6:38 PM, Victor Stinner 
wrote:

> Le 22 juil. 2017 8:04 AM, "Serhiy Storchaka"  a
> écrit :
>
> I think the only reliable way of fixing the vulnerability is rejecting or
> escaping (as specified in RFC 2640) CR and LF inside sent lines. Adding the
> support of RFC 2640 is a new feature and can be added only in 3.7. And this
> feature should be optional since not all servers support RFC 2640.
> https://github.com/python/cpython/pull/1214 does the right thing.
>
>
> In that case, I suggest to reject newlines in ftplib, and maybe add an
> opt-in option to escape newlines.
>
> Java just rejected newlines, no? Or does Java allows to escape them?
>
> Victor
>
>
OK, let's just reject \n then and be done with it. It's a rare use case
after all.
Java just rejects \n for all commands and does not support escaping (aka
RFC 2640).


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Need help to fix urllib(.parse) vulnerabilities

[Giampaolo Rodola']

On Fri, Jul 21, 2017 at 12:45 PM, Victor Stinner 
wrote:

> 2017-07-21 12:02 GMT+02:00 Victor Stinner :
> > https://bugs.python.org/issue29606
> > http://python-security.readthedocs.io/vuln/urllib_
> ftp_protocol_stream_injection.html#urllib-ftp-protocol-stream-injection
> > => not fixed yet
>
> Ok, I more concrete problem. To fix the "urllib FTP" bug, we have to
> find a balance between security (reject any URL looking like an
> attempt to counter the security protections) and backward
> compatibility (accept filenames containing newlines).
>
> Maybe we need to only reject an URL which contains a newline in the
> "host" part, but accept them in the "path" part of the URL? The
> question is if the code splits correctly "host" and "path" parts when
> the URL contains a newline. My bet is that no, it behaves badly :-)
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/g.
> rodola%40gmail.com
>

It took me a while to understand the security implications of this
FTP-related bug, but I believe I got the gist of it here (I can elaborate
further if it's not clear):
https://github.com/python/cpython/pull/1214#issuecomment-298393169
My proposal is to fix ftplib.py and guard against malicious strings
involving the *PORT command only*. This way we fix the issue *and* maintain
backward compatibility by allowing users to specify "\n" in their paths and
username / password pairs. Java took a different approach and disallowed
"\n" completely.
To my understanding fixing ftplib would automatically mean fixing urllib as
well.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Impact of Namedtuple on startup time

[Giampaolo Rodola']

On Mon, Jul 17, 2017 at 11:24 PM, Tim Peters  wrote:

> [Giampaolo Rodola' ]
> > 
> > To be entirely honest, I'm not even sure why they need to be forcefully
> > declared upfront in the first place, instead of just having a first-class
> > function (builtin?) written in C:
> >
> > >>> ntuple(x=1, y=0)
> > (x=1, y=0)
> >
> > ...or even a literal as in:
> >
> > >>> (x=1, y=0)
> > (x=1, y=0)
>
> How do you propose that the resulting object T know that T.x is 1. T.y
> is 0, and T.z doesn't make sense?


I'm not sure I understand your concern. That's pretty much what
PyStructSequence already does.


> Declaring a namedtuple up front
> allows the _class_ to know that all of its instances map attribute "x"
> to index 0 and attribute "y" to index 1.  The instances know nothing
> about that on their own


Hence why I was talking about a "(lightweight) anonymous tuple with named
attributes". The primary use case for namedtuples is accessing values by
name (obj.x). Personally I've always considered the upfront module-level
declaration only an annoyance which unnecessarily pollutes the API and adds
extra overhead. I typically end up putting all namedtuples in a private
module:
https://github.com/giampaolo/psutil/blob/8b8da39e0c62432504fb5f67c418715aad35b291/psutil/_common.py#L156-L225
...then import them from elsewhere and make sure they are not exposed
publicly because the intermediate object returned by
collections.namedtuple() is basically useless for the end-user. Also
picking up a sensible name for the namedtuple is an annoyance and kinda
weird. Consider this:

from collections import namedtuple

Coordinates = namedtuple('coordinates', ['x', 'y'])

def get_coordinates():
return Coordinates(10, 20)

...vs. this:

def get_coordinates():
return ntuple(x=10, y=20)

...or this:

def get_coordinates():
return (x=10, y=20)

If your `ntuple()` returns an object implementing its own
> mapping, it loses a primary advantage (0 memory overhead) of
> namedtuples.
>

The extra memory overhead is a price I would be happy to pay considering
that collections.namedtuple is considerably slower than a plain tuple.
Other than the additional overhead on startup / import time, instantiation
is 4.5x slower than a plain tuple:

$ python3.7 -m timeit -s "from collections import namedtuple; nt =
namedtuple('xxx', ('x', 'y'))" "nt(1, 2)"
100 loops, best of 5: 313 nsec per loop

$ python3.7 -m timeit "tuple((1, 2))"
500 loops, best of 5: 68.4 nsec per loop

...and name access is 2x slower than index access:

$ python3.7 -m timeit -s "from collections import namedtuple; nt =
namedtuple('xxx', ('x', 'y')); x = nt(1, 2)" "x.x"
500 loops, best of 5: 41.9 nsec per loop

$ python3.7 -m timeit -s "from collections import namedtuple; nt =
namedtuple('xxx', ('x', 'y')); x = nt(1, 2)" "x[0]"
1000 loops, best of 5: 20.2 nsec per loop
$ python3.7 -m timeit -s "x = (1, 2)" "x[0]"
1000 loops, best of 5: 20.5 nsec per loop

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Impact of Namedtuple on startup time

[Giampaolo Rodola']

On Mon, Jul 17, 2017 at 11:07 PM, Petr Viktorin  wrote:

> On 07/17/2017 10:31 PM, Giampaolo Rodola' wrote:
>
>> I completely agree. I love namedtuples but I've never been too happy
>> about the additional overhead vs. plain tuples (both for creation and
>> attribute access times), to the point that I explicitly avoid to use them
>> in certain circumstances (e.g. a busy loop) and only for public end-user
>> APIs returning multiple values.
>>
>> To be entirely honest, I'm not even sure why they need to be forcefully
>> declared upfront in the first place, instead of just having a first-class
>> function (builtin?) written in C:
>>
>>  >>> ntuple(x=1, y=0)
>> (x=1, y=0)
>>
>> ...or even a literal as in:
>>
>>  >>> (x=1, y=0)
>> (x=1, y=0)
>>
>> Most of the times this is what I really want: quickly returning an
>> anonymous tuple with named attributes and nothing else, similarly to
>> os.times() & others. [...]
>>
>
> It seems that you want `types.SimpleNamespace(x=1, y=0)`.
>

That doesn't support indexing (obj[0]).

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Impact of Namedtuple on startup time

[Giampaolo Rodola']

I completely agree. I love namedtuples but I've never been too happy about
the additional overhead vs. plain tuples (both for creation and attribute
access times), to the point that I explicitly avoid to use them in certain
circumstances (e.g. a busy loop) and only for public end-user APIs
returning multiple values.

To be entirely honest, I'm not even sure why they need to be forcefully
declared upfront in the first place, instead of just having a first-class
function (builtin?) written in C:

>>> ntuple(x=1, y=0)
(x=1, y=0)

...or even a literal as in:

>>> (x=1, y=0)
(x=1, y=0)

Most of the times this is what I really want: quickly returning an
anonymous tuple with named attributes and nothing else, similarly to
os.times() & others. I believe that if something like this would exist we
would witness a big transition from tuple() to ntuple() for all those
functions returning more than 1 value. We witnessed a similar transition in
many parts of the stdlib when collections.namedtuple was first introduced,
but not everywhere, probably because declaring a namedtuple is more work,
it's more expensive, and it still feels like you're dealing with some kind
of too high-level second-class citizen with too much overhead and too many
sugar in terms of API (e.g. "verbose", "rename", "module" and "_source").

If something like this were to happen I expect collections.namedtuple to be
used only by those who want to subclass it in order to attach methods,
whereas the rest would stick and use ntuple() pretty much everywhere (both
in "private" and "public" functions).


On Mon, Jul 17, 2017 at 5:49 PM, Guido van Rossum  wrote:

> I am firmly with Antoine here. The cumulative startup time of large Python
> programs is a serious problem and namedtuple is one of the major
> contributors -- especially because it is so convenient that it is
> ubiquitous. The approach of generating source code and exec()ing it, is a
> cool demonstration of Python's expressive power, but it's always been my
> sense that whenever we encounter a popular idiom that uses exec() and
> eval(), we should augment the language (or the builtins) to avoid these
> calls -- that's for example how we ended up with getattr().
>
> One of the reasons to be wary of exec()/eval() other than the usual
> security concerns is that in some Python implementations they have a high
> overhead to initialize the parser and compiler. (Even in CPython it's not
> that fast.)
>
> Regarding the argument that it's easier to learn what namedtuple does if
> the generated source is available, while I don't feel this is important,
> supposedly it is important to Raymond. But surely there are other
> approaches possible that work just as well in an educational setting while
> being more efficient in production use. (E.g. the approach taken by
> itertools, where the docs show equivalent Python code.)
>
> Concluding, I think we should move on from the original implementation and
> optimize the heck out of namedtuple. The original has served us well. The
> world is constantly changing. Python should adapt to the (happy) fact that
> it's being used for systems larger than any of us could imagine 15 years
> ago.
>
> --Guido
>
> On Mon, Jul 17, 2017 at 7:59 AM, Raymond Hettinger <
> raymond.hettin...@gmail.com> wrote:
>
>>
>> > On Jul 17, 2017, at 6:31 AM, Antoine Pitrou  wrote:
>> >
>> >> I think I understand well enough to say something intelligent…
>> >>
>> >> While actual references to _source are likely rare (certainly I’ve
>> never
>> >> used it), my understanding is that the way namedtuple works is to
>> >> construct _source, and then exec it to create the class. Once that is
>> >> done, there is no significant saving to be had by throwing away the
>> >> constructed _source value.
>>
>> There are considerable benefits to namedtuple being able to generate and
>> match its own source.
>>
>> * It makes it is really easy for a user to generate the code, drop it
>> into another another module, and customize it.
>>
>> * It makes the named tuple factory function completely self-documenting.
>>
>> * The verbose/_source option teaches you exactly what named tuple does.
>> That makes the tool relatively easy to learn, understand, and debug.
>>
>> I really don't want to throw away these benefits to save a couple of
>> milliseconds.   As Nick Coghlan recently posted, "Speed isn't everything,
>> and it certainly isn't adequate justification for breaking public APIs that
>> have been around for years."
>>
>> FWIW, the template/exec implementation has had excellent benefits for
>> maintainability making it very easy to fix and update.  As other parts of
>> Python have changed (limitations on number of arguments, what is allowed as
>> an identifier, etc), it mostly automatically stays in sync with the rest of
>> the language.
>>
>> ISTM this issue is being pressed by micro-optimizers who are being very
>> aggressive and not responding to actual user needs (it is more an invented
>> issue than a rea

Re: [Python-Dev] Asking for feedback about fixing `ftplib' (issue25458)

[Giampaolo Rodola']

On Wed, Dec 28, 2016 at 3:57 PM, Ivan Pozdeev  wrote:

> On 25.12.2016 17:21, Giampaolo Rodola' wrote:
>
> From what I understand the problem should be fixed in urllib so that it
> always closes the FTP connection. I understand this is what happens in
> recent 3.x versions but not on 2.7.
>
> urllib in 2.7 should indeed be fixed to close FTP connections rather than
> leave them dangling, no question about that.
>

To me it looks like this is the only issue (currently tracked in
http://bugs.python.org/issue28931).


> I tried to fix urllib so that it's guaranteed to call voidresp(), and
> failed. Too complicated, effectively reimplementing a sizable part of FTP
> client logic is required, perhaps even monkey-patching ftplib to be able to
> set flags in the right places.
>

Why did you fail? Why urllib can't close() -> voidresp() the FTP session
once it's done with it?


> With simple commands (voidcmd) and self-contained transfer commands
> (retrlines/retrbinary), ftplib does incapsulate statefulness, by handling
> them atomically. But with transfercmd(), it returns control to the user
> halfway through.
>

That's by design. ftplib's transfercmd() just works like that: it's a low
level method returning a "data" socket and it's just up to you to cleanly
close the session (close() -> voidresp()) once you're done with it. Most of
the times you don't even want to use transfercmd() in the first place, as
you just use stor* and retr* methods.


> At this point, if ftplib doesn't itself keep track that the control
> connection is currently in invalid state for the next command, the user is
> forced to do that themselves instead.
>

Hence why I suggested a docfix.


> And guess what - urllib has to use ftplib through a wrapper class that
> does exactly that. That's definitely a "stock logic part" 'cuz it's an
> integral part of FTP rather than something specific to user logic.
>

That wrapper should just cleanly close the session.


> The other "stock logic part" currently missing is error handling during
> transfer. If the error's cause is local (like a local exception that
> results in closing the data socket prematurely, or the user closing it by
> hand, or an ABOR they sent midway), the error code from the end-of-transfer
> response should be ignored, otherwise, it shouldn't.
>

Absolutely not. Base ftplib should NOT take any deliberate decision such as
ignoring an error.
I can even come up with use cases: use ftplib to test FTP servers so that
they *do* reply with an error code in certain conditions. Here's a couple
examples in pyftpdlib:

https://github.com/giampaolo/pyftpdlib/blob/17bebe3714752b01e43ab60d2771308f4594bd99/pyftpdlib/test/test_functional.py#L1354-L1369

https://github.com/giampaolo/pyftpdlib/blob/17bebe3714752b01e43ab60d2771308f4594bd99/pyftpdlib/test/test_functional.py#L1973-L1980


> Other than making ftplib keep track of the session's internal state while
> the user has control and providing the custom error handling logic to them,
> another solution is to never release control to the user midway, i.e. ban
> transfercmd() entirely and only provide self-contained
> retr*()/dir()/whatever, but let the callback signal them to abort the
> transfer. That way, the state would be kept implicitly - in the execution
> pointer.
>

Banning transfercmd() means renaming it to _transfercmd() which is a no-no
for backward compatibility. Also, as I've shown above, transfercmd() does
have a use case which relies on it behaving like that (return control
midway).


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asking for feedback about fixing `ftplib' (issue25458)

[Giampaolo Rodola']

>From what I understand the problem should be fixed in urllib so that it
always closes the FTP connection. I understand this is what happens in
recent 3.x versions but not on 2.7.
As for ftplib, I don't like the solution of using a `transfer_in_progress`
flag in all commands. I see that as a cheap way to work around the fact
that the user may forget to call voidresp() and close the socket after
nt/transfercmd().
It probably makes sense to just update nt/transfercmd() doc instead and
make that clear.


On Mon, Dec 19, 2016 at 12:01 PM, Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

> I'm currently working on http://bugs.python.org/issue25458 .
> There are a few options there, and each one has drawbacks.
> So, I'd like to get some feedback on which way is prefereable before
> working towards any of them and/or other ideas should they arise.
>
> The problem and the options are summarized in
> http://bugs.python.org/issue25458#msg283073 and the message after that
> one.
>
> Apart from the options, I'd like to know if I must solve the 2nd problem
> (error handling), too, or it can be handled in a separate ticket.
>
> --
> Regards,
> Ivan
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/g.rodola%
> 40gmail.com
>



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pythonhosted.org upload no longer works

[Giampaolo Rodola']

Thanks Paul,
no I wasn't aware of that, and I will subscribe to distutils-sig from now
on.
I don't remember ever receiving a warrant about this decision but it's
entirely possible I may have forgot. =)
So long story short is that I will have to move the doc on readthedocs,
correct?
Does pythonhosted provide an automatic redirect or I'll have to upload a
static page which states the doc has been moved?

On Mon, Oct 10, 2016 at 4:41 PM, Paul Moore  wrote:

> On 10 October 2016 at 14:34, Giampaolo Rodola'  wrote:
> > This is what I've bumped into just now:
> >
> > python setup.py upload_sphinx --upload-dir=docs/_build/html
> > running upload_sphinx
> > Submitting documentation to https://upload.pypi.org/legacy/
> > Upload failed (410): Uploading documentation is no longer supported, we
> > recommend using https://readthedocs.org/.
> >
> > Personally I prefer pythonhosted over readthedocs as I can provide the
> same
> > style as official Python's doc (see https://pythonhosted.org/psutil/)
> and,
> > most importantly, because I can automatize doc upload just by running
> "make
> > doc-upload".
> > Is there a reason upload functionality has been disabled?
> > Is pythonhosted.org going to be dismissed or something?
>
> This was discussed on distutils-sig back in May 2015. The thread
> starts here: https://mail.python.org/pipermail/distutils-sig/2015-
> May/026327.html.
> One of the actions in the final proposal
> (https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html)
> included a step to contact projects that were using pythonhosted.org.
> Did you not get any contact?
>
> It might be worth picking this up on distutils-sig, as that's where
> the original discussion occurred. I've copied this reply to there, and
> suggest followups go to that list.
>
> Paul
>



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] pythonhosted.org upload no longer works

[Giampaolo Rodola']

This is what I've bumped into just now:

python setup.py upload_sphinx --upload-dir=docs/_build/html
running upload_sphinx
Submitting documentation to https://upload.pypi.org/legacy/
Upload failed (410): Uploading documentation is no longer supported, we
recommend using https://readthedocs.org/.

Personally I prefer pythonhosted over readthedocs as I can provide the same
style as official Python's doc (see https://pythonhosted.org/psutil/) and,
most importantly, because I can automatize doc upload just by running "make
doc-upload".
Is there a reason upload functionality has been disabled?
Is pythonhosted.org going to be dismissed or something?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rewrite @contextlib.contextmanager in C

[Giampaolo Rodola']

On Tue, Aug 9, 2016 at 3:30 PM, Nick Coghlan  wrote:

> On 9 August 2016 at 23:26, Nick Coghlan  wrote:
>
>> On 9 August 2016 at 06:18, Guido van Rossum  wrote:
>>
>>> I think Nick would be interested in understanding why this is the case.
>>> What does the decorator do that could be so expensive?
>>>
>>
>> Reviewing https://hg.python.org/cpython/file/default/Lib/contextlib.py
>> #l57, Chris's analysis seems plausible to me
>>
>
> Sorry Wolfgang - I missed that Chris was expanding on a comparison you
> initially made!
>
> Either way, I agree that aspect does make up the bulk of the difference in
> speed, so moving to C likely wouldn't help much. However, the speed
> difference relative to the simpler warppers is far less defensible - I
> think there are some opportunities for improvement there, especially around
> moving introspection work out of _GeneratorContextManager.__init__ and
> into the contextmanager decorator itself.
>

By moving the __doc__ introspection out of __init__ and by introducing
__slots__ I got from -4.37x to -3.16x (__slot__ speedup was about +0.3x).
Chris' SimplerContextManager solution is faster because it avoids the
factory function but that is necessary for supporting the decoration of
methods. Here's a PoC:


diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 7d94a57..45270dd 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -2,7 +2,7 @@
 import abc
 import sys
 from collections import deque
-from functools import wraps
+from functools import wraps, update_wrapper

 __all__ = ["contextmanager", "closing", "AbstractContextManager",
"ContextDecorator", "ExitStack", "redirect_stdout",
@@ -57,25 +57,18 @@ class ContextDecorator(object):
 class _GeneratorContextManager(ContextDecorator, AbstractContextManager):
 """Helper for @contextmanager decorator."""

+__slots__ = ['gen', 'funcak']
+
 def __init__(self, func, args, kwds):
 self.gen = func(*args, **kwds)
-self.func, self.args, self.kwds = func, args, kwds
-# Issue 19330: ensure context manager instances have good
docstrings
-doc = getattr(func, "__doc__", None)
-if doc is None:
-doc = type(self).__doc__
-self.__doc__ = doc
-# Unfortunately, this still doesn't provide good help output when
-# inspecting the created context manager instances, since pydoc
-# currently bypasses the instance docstring and shows the docstring
-# for the class instead.
-# See http://bugs.python.org/issue19404 for more details.
+self.funcak = func, args, kwds

 def _recreate_cm(self):
 # _GCM instances are one-shot context managers, so the
 # CM must be recreated each time a decorated function is
 # called
-return self.__class__(self.func, self.args, self.kwds)
+func, args, kwds = self.funcak
+return self.__class__(func, args, kwds)

 def __enter__(self):
 try:
@@ -157,6 +150,8 @@ def contextmanager(func):
 @wraps(func)
 def helper(*args, **kwds):
 return _GeneratorContextManager(func, args, kwds)
+
+update_wrapper(helper, func)
 return helper


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rewrite @contextlib.contextmanager in C

[Giampaolo Rodola']

On Mon, Aug 8, 2016 at 11:59 PM, Chris Angelico  wrote:

> On Tue, Aug 9, 2016 at 7:14 AM, Wolfgang Maier
>  wrote:
> > Right, I think a fairer comparison would be to:
> >
> > class ctx2:
> > def __enter__(self):
> > self.it = iter(self)
> > return next(self.it)
> >
> > def __exit__(self, *args):
> > try:
> > next(self.it)
> > except StopIteration:
> > pass
> >
> > def __iter__(self):
> > yield
> >
> > With this change alone the slowdown diminishes to ~ 1.7x for me. The
> rest is
> > probably the extra overhead for being able to pass exceptions raised
> inside
> > the with block back into the generator and such.
>
> I played around with a few other variants to see where the slowdown
> is. They all work out pretty much the same as the above; my two
> examples are both used the same way as contextlib.contextmanager is,
> but are restrictive on what you can do.
>
> import timeit
> import contextlib
> import functools
>
> class ctx1:
> def __enter__(self):
> pass
> def __exit__(self, *args):
> pass
>
> @contextlib.contextmanager
> def ctx2():
> yield
>
> class SimplerContextManager:
> """Like contextlib._GeneratorContextManager but way simpler.
>
> * Doesn't reinstantiate itself - just reinvokes the generator
> * Doesn't allow yielded objects (returns self)
> * Lacks a lot of error checking. USE ONLY AS DIRECTED.
> """
> def __init__(self, func):
> self.func = func
> functools.update_wrapper(self, func)
> def __call__(self, *a, **kw):
> self.gen = self.func(*a, **kw)
> return self
> def __enter__(self):
> next(self.gen)
> return self
> def __exit__(self, type, value, traceback):
> if type is None:
> try: next(self.gen)
> except StopIteration: return
> else: raise RuntimeError("generator didn't stop")
> try: self.gen.throw(type, value, traceback)
> except StopIteration: return True
> # Assume any instance of the same exception type is a proper
> reraise
> # This is way simpler than contextmanager normally does, and costs
> us
> # the ability to detect exception handlers that coincidentally
> raise
> # the same type of error (eg "except ZeroDivisionError:
> print(1/0)").
> except type: return False
>
> # Check that it actually behaves correctly
> @SimplerContextManager
> def ctxdemo():
> print("Before yield")
> try:
> yield 123
> except ZeroDivisionError:
> print("Absorbing 1/0")
> return
> finally:
> print("Finalizing")
> print("After yield (no exception)")
>
> with ctxdemo() as val:
> print("1/0 =", 1/0)
> with ctxdemo() as val:
> print("1/1 =", 1/1)
> #with ctxdemo() as val:
> #print("1/q =", 1/q)
>
> @SimplerContextManager
> def ctx3():
> yield
>
> class TooSimpleContextManager:
> """Now this time you've gone too far."""
> def __init__(self, func):
> self.func = func
> def __call__(self):
> self.gen = self.func()
> return self
> def __enter__(self):
> next(self.gen)
> def __exit__(self, type, value, traceback):
> try: next(self.gen)
> except StopIteration: pass
>
> @TooSimpleContextManager
> def ctx4():
> yield
>
> class ctx5:
> def __enter__(self):
> self.it = iter(self)
> return next(self.it)
>
> def __exit__(self, *args):
> try:
> next(self.it)
> except StopIteration:
> pass
>
> def __iter__(self):
> yield
>
> t1 = timeit.timeit("with ctx1(): pass", setup="from __main__ import ctx1")
> print("%.3f secs" % t1)
>
> for i in range(2, 6):
> t2 = timeit.timeit("with ctx2(): pass", setup="from __main__
> import ctx%d as ctx2"%i)
> print("%.3f secs" % t2)
> print("slowdown: -%.2fx" % (t2 / t1))
>
>
> My numbers are:
>
> 0.320 secs
> 1.354 secs
> slowdown: -4.23x
> 0.899 secs
> slowdown: -2.81x
> 0.831 secs
> slowdown: -2.60x
> 0.868 secs
> slowdown: -2.71x
>
> So compared to the tight custom-written context manager class, all the
> "pass it a generator function" varieties look pretty much the same.
> The existing contextmanager factory has several levels of indirection,
> and that's probably where the rest of the performance difference comes
> from, but there is some cost to the simplicity of the gen-func
> approach.
>
> My guess is that a C-implemented version could replace piles of
> error-handling code with simple pointer comparisons (hence my
> elimination of it), and may or may not be able to remove some of the
> indirection. I'd say it'd land about in the same range as the other
> examples here. Is that worth it?
>
> ChrisA
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://ma

Re: [Python-Dev] Rewrite @contextlib.contextmanager in C

[Giampaolo Rodola']

On Mon, Aug 8, 2016 at 10:07 PM, Yury Selivanov 
wrote:

>
>
> On 2016-08-08 3:33 PM, Giampaolo Rodola' wrote:
>
>> I wanted to give it a try rewriting this in C but since @contextmanager
>> has a lot of magic I wanted to ask first whether this 1) is technically
>> possible 2) is desirable.
>>
>
> It is definitely technologically possible.  However, the C implementation
> will be quite complex, and will require a lot of time to review and later
> maintain.  My question would be how critical is the performance of
> @contextmanager?  I'd say that unless it's used in a tight loop it can't
> affect the performance too much.
>

Yeah, the (my) use case is exactly that (tight loops).


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Rewrite @contextlib.contextmanager in C

[Giampaolo Rodola']

import timeit
import contextlib

@contextlib.contextmanager
def ctx1():
yield

class ctx2:
def __enter__(self):
pass
def __exit__(self, *args):
pass

t1 = timeit.timeit("with ctx1(): pass", setup="from __main__ import ctx1")
t2 = timeit.timeit("with ctx2(): pass", setup="from __main__ import ctx2")
print("%.3f secs" % t1)
print("%.3f secs" % t2)
print("slowdown: -%.2fx" % (t1 / t2))


...with Python 3.5:

1.938 secs
0.443 secs
slowdown: -4.37x

I wanted to give it a try rewriting this in C but since @contextmanager has
a lot of magic I wanted to ask first whether this 1) is technically
possible 2) is desirable.
Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Stop using timeit, use perf.timeit!

[Giampaolo Rodola']

On Fri, Jun 10, 2016 at 1:13 PM, Victor Stinner 
wrote:

> Hi,
>
> Last weeks, I made researchs on how to get stable and reliable
> benchmarks, especially for the corner case of microbenchmarks. The
> first result is a serie of article, here are the first three:
>
> https://haypo.github.io/journey-to-stable-benchmark-system.html
> https://haypo.github.io/journey-to-stable-benchmark-deadcode.html
> https://haypo.github.io/journey-to-stable-benchmark-average.html
>
> The second result is a new perf module which includes all "tricks"
> discovered in my research: compute average and standard deviation,
> spawn multiple worker child processes, automatically calibrate the
> number of outter-loop iterations, automatically pin worker processes
> to isolated CPUs, and more.
>
> The perf module allows to store benchmark results as JSON to analyze
> them in depth later. It helps to configure correctly a benchmark and
> check manually if it is reliable or not.
>
> The perf documentation also explains how to get stable and reliable
> benchmarks (ex: how to tune Linux to isolate CPUs).
>
> perf has 3 builtin CLI commands:
>
> * python -m perf: show and compare JSON results
> * python -m perf.timeit: new better and more reliable implementation of
> timeit
> * python -m metadata: display collected metadata
>
> Python 3 is recommended to get time.perf_counter(), use the new
> accurate statistics module, automatic CPU pinning (I will implement it
> on Python 2 later), etc. But Python 2.7 is also supported, fallbacks
> are implemented when needed.
>
> Example with the patched telco benchmark (benchmark for the decimal
> module) on a Linux with two isolated CPUs.
>
> First run the benchmark:
> ---
> $ python3 telco.py --json-file=telco.json
> .
> Average: 26.7 ms +- 0.2 ms
> ---
>
>
> Then show the JSON content to see all details:
> ---
> $ python3 -m perf -v show telco.json
> Metadata:
> - aslr: enabled
> - cpu_affinity: 2, 3
> - cpu_count: 4
> - cpu_model_name: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
> - hostname: smithers
> - loops: 10
> - platform: Linux-4.4.9-300.fc23.x86_64-x86_64-with-fedora-23-Twenty_Three
> - python_executable: /usr/bin/python3
> - python_implementation: cpython
> - python_version: 3.4.3
>
> Run 1/25: warmup (1): 26.9 ms; samples (3): 26.8 ms, 26.8 ms, 26.7 ms
> Run 2/25: warmup (1): 26.8 ms; samples (3): 26.7 ms, 26.7 ms, 26.7 ms
> Run 3/25: warmup (1): 26.9 ms; samples (3): 26.8 ms, 26.9 ms, 26.8 ms
> (...)
> Run 25/25: warmup (1): 26.8 ms; samples (3): 26.7 ms, 26.7 ms, 26.7 ms
>
> Average: 26.7 ms +- 0.2 ms (25 runs x 3 samples; 1 warmup)
> ---
>
> Note: benchmarks can be analyzed with Python 2.
>
> I'm posting my email to python-dev because providing timeit results is
> commonly requested in review of optimization patches.
>
> The next step is to patch the CPython benchmark suite to use the perf
> module. I already forked the repository and started to patch some
> benchmarks.
>
> If you are interested by Python performance in general, please join us
> on the speed mailing list!
> https://mail.python.org/mailman/listinfo/speed
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>

This is very interesting and also somewhat related to psutil. I wonder...
would increasing process priority help isolating benchmarks even more? By
this I mean "os.nice(-20)".
Extra: perhaps even IO priority:
https://pythonhosted.org/psutil/#psutil.Process.ionice ?


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancments - method name only

[Giampaolo Rodola']

On Fri, Apr 8, 2016 at 9:09 PM, Chris Angelico  wrote:

> On Sat, Apr 9, 2016 at 5:03 AM, Chris Barker 
> wrote:
> > On Fri, Apr 8, 2016 at 11:34 AM, Koos Zevenhoven 
> wrote:
> >>
> >> >
> >> > __pathstr__ # pathstring
> >> >
> >>
> >> Or perhaps __pathstring__ in case it may be or return byte strings.
> >
> >
> > I'm fine with __pathstring__ , but I thought it was already decided that
> it
> > would NOT return a bytestring!
>
> I sincerely hope that's been settled on. There's no reason to have
> this ever return anything other than a str. (Famous last words, I
> know.)
>
> ChrisA
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>

I'm kind of scared about this: scared to state and be 100% sure that bytes
won't *never ever* be returned.
As such I would call this __fspath__ or something, but I would definitively
avoid to use "str".

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python should be easily compilable on Windows with MinGW

[Giampaolo Rodola']

On Fri, Feb 26, 2016 at 6:55 PM, Alexander Walters 
wrote:

> No.
>
> Visual Studio is a solid compiler suit, mingw is a jenky mess, especially
> when you try and move to 64bit (where I don't think there is one true
> version of mingw).  I'm sorry that Visual Studio makes it very hard for you
> to contribute, but changing THE compiler of the distribution from the
> platform compiler, especially when we FINALLY got a stable abi with it, is
> going to be a non starter.
>
> Compiling on MinGW for your own edification is fine, but that's not the
> build platform for windows python, nor should it be. Contributions are, and
> should continue to be, tested against Visual Studio.
>
>
> On 2/26/2016 05:12, Mathieu Dupuy wrote:
>
>> Hi.
>> I am currently working on adding some functionality on a standard
>> library module (http://bugs.python.org/issue15873). The Python part
>> went fine, but now I have to do the C counterpart, and I have ran into
>> in several problems, which, stacked up, are a huge obstacle to easily
>> contribute further. Currently, despite I could work, I can't go
>> further
>> on my patch.
>>
>> I am currently working in very limited network, CPU and time
>> ressources* which are quite uncommon in the western world, but are
>> much less in the rest of the world. I have a 2GB/month mobile data
>> plan and a 100KB/s speed. For the C part of my patch, I should
>> download Visual Studio. The Express Edition 2015 is roughly 9GB. I
>> can't afford that.
>>
>> I downloaded Virtualbox and two Linux netinstall (Ubuntu 15.10 and
>> Fedora 23). Shortly, I couldn't get something working quickly and
>> simply (quickly = less than 2 hours, downloading time NOT included,
>> which is anyway way too already much). What went wrong and why it went
>> wrong could be a whole new thread and is outside of the scope of this
>> message.
>> Let me precise this : at my work I use many virtualbox instances
>> automatically fired and run in parallel to test new deployments and
>> run unittests. I like this tool,
>> but despite its simple look, it (most of the time) can not be used
>> simply by a profane. The concepts it requires you to understand are
>> not intuitive at first sight and there is *always* a thing that go
>> wrong (guest additions, mostly).(for example : Ubuntu and Virtualbox
>> shipped for a moment a broken version of mount.vboxsf, preventing
>> sharing folder to mount. Despite it's fixed, the broken releases
>> spread everywhere and you may encounter them a lot in various Ubuntu
>> and Virtualbox version. I downloaded the last versions of both and I
>> am yet infected. https://www.virtualbox.org/ticket/12879). I could do
>> whole new thread on why you can't ask newcomers to use Virtualbox
>> (currently, at least).
>>
>> I ran into is a whole patch set to make CPython compile on MinGW
>> (https://bugs.python.org/issue3871#msg199695). But it is not denying
>> it's very experimental, and I know I would again spent useless hours
>> trying to get it work rather than joyfully improving Python, and
>> that's exactly what I do not want to happen.
>>
>> Getting ready to contribute to CPython pure python modules from an
>> standard, average mr-everyone Windows PC for a beginner-to-medium
>> contributor only require few megabytes of internet and few minutes of his
>> time: getting a tarball of CPython sources (or cloning the github CPython
>> mirror)**, a basic text editor and msys-git. The step further, if doing
>> some -even basic- C code is required, implies downloading 9GB of Visual
>> Studio and countless hours for it to be ready to use.
>> I think downloading the whole Visual Studio suite is a huge stopper to
>> contribute further for an average medium-or-below-contributor.
>>
>> I think (and I must not be the only one since CPython is to be moved
>> to github), that barriers to contribute to CPython should be set to
>> the lowest.
>> Of course my situation is a bit special but I think it represents
>> daily struggle of a *lot* of non-western programmer (at least for
>> limited internet)(even here in Australia, landline limited internet
>> connections are very common).
>> It's not a big deal if the MinGW result build is twenty time slower or
>> if some of the most advanced modules can't be build. But everyone
>> programmer should be able to easily make some C hacks and get them to
>> work.
>>
>> Hoping you'll be receptive to my pleas,
>> Cheers
>>
>>
>> * I am currently picking fruits in the regional Australia. I live in a van
>> and have internet through with smartphone through an EDGE connection. I
>> can
>> plug the laptop in the farm but not in the van.
>> ** No fresh programmer use mercurial unless he has a gun pointed on his
>> head.
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com
>>
>
> _

Re: [Python-Dev] Thank you.

[Giampaolo Rodola']

On Sun, Feb 21, 2016 at 9:29 AM, Alexander Walters 
wrote:

> I don't know if it is appropriate for this list, or not.  I don't exactly
> care.  As much as I might disagree with some of you...
>
> Thank you.
>

>From time to time I also think about how deeply Python impacted my life.
Places I visited and lived in, people I got in contact with, the mere every
day life and afternoons spent writing code just for the heck of it If
it weren't for Python all of that would have been profoundly different and
most certainly not as good. I would like to say thanks as well. Thank you
Guido and thank you all core-devs. You don't change only code: you
literally change lives as well. And that is profound.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] congrats on 3.5! Alas, windows 7 users are having problems installing it

[Giampaolo Rodola']

Same here. I get a "0x80240017 error" during installation.

On Sun, Sep 13, 2015 at 10:41 PM, Laura Creighton  wrote:

> webmaster has already heard from 4 people who cannot install it.
> I sent them to the bug tracker or to python-list but they seem
> not to have gone either place.  Is there some guide I should be
> sending them to, 'how to debug installation problems'?
>
> Laura
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 475 accepted

[Giampaolo Rodola']

On Mon, Feb 2, 2015 at 10:46 PM, Victor Stinner 
wrote:

> 2015-02-02 22:11 GMT+01:00 Giampaolo Rodola' :
> > I may be chiming in a little late, however, I'd have a question: does
> this
> > affect non-blocking applications somehow?
> > How often should we expect to receive EINTR?
>
> Each time a program is interrupted by a signal while it was waiting
> for a sycall. Most syscalls are fast, especially in an application
> written with non blocking operations.
>
> For example, timeit says that os.getuid() takes 370 nanoseconds (on
> Linux). getpid() only takes 285 nanoseconds, but it looks to be cached
> in the libc: strace doesn't show me syscalls.
>
> Network syscalls are probably slower.
>
> haypo@selma$ ./python -Wignore -m timeit -s 'import socket;
> s,c=socket.socketpair()' 's.send(b"a"); c.recv(1)'
> 10 loops, best of 3: 2.26 usec per loop
>
> > Is it correct to state that in case many EINTR signals are sent
> > repeatedly a non-blocking framework such as asyncio may hang for "too
> long"?
>
> A syscall fails with EINTR each time it gets a signal. You are
> unlikely if the same syscall requires to be retried twice (executed 3
> times) because it got EINTR twice.
>
> I don't see why the kernel would make a syscall fails EINTR multiple times.
>
> asyncio doesn't process the signal immediatly. asyncio uses
> signal.set_wakeup_fd(). At the C level, the C signal handler writes
> the signal number into a pipe. At the Python level, the selector is
> awaken by the write. Later, asyncio executes the Python handler of the
> signal (if a Python signal handler was registered).
>
> A nice side effect of the PEP is to avoid to awake the application if
> it's not necessary. If no Python signal handler is registered, no byte
> is written into the pipe, the selector continues to wait for events.
>
> Victor
>

OK, thanks for clarifying, this is a very nice addition. One last thing:
should InterruptedError exception be deprecated? As far as I understand it
should never occur again, right?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 475 accepted

[Giampaolo Rodola']

On Mon, Feb 2, 2015 at 9:45 PM, Antoine Pitrou  wrote:

>
> Hello,
>
> I'm now accepting PEP 475 - "Retry system calls failing with EINTR".
> You can read it at https://www.python.org/dev/peps/pep-0475/
>
> The implementation is more or less ready at
> http://bugs.python.org/issue23285, so you can expect it to land in the
> main repo relatively soon.
>
> Regards
>
> Antoine.
>

I may be chiming in a little late, however, I'd have a question: does this
affect non-blocking applications somehow?
How often should we expect to receive EINTR? Is it correct to state that in
case many EINTR signals are sent repeatedly a non-blocking framework such
as asyncio may hang for "too long"?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[Giampaolo Rodola']

On Wed, Dec 10, 2014 at 5:59 PM, Bruno Cauet  wrote:

> Hi all,
> Last year a survey was conducted on python 2 and 3 usage.
> Here is the 2014 edition, slightly updated (from 9 to 11 questions).
> It should not take you more than 1 minute to fill. I would be pleased if
> you took that time.
>
> Here's the url: http://goo.gl/forms/tDTcm8UzB3
> I'll publish the results around the end of the year.
>
> Last year results: https://wiki.python.org/moin/2.x-vs-3.x-survey
>
> Thank you
> Bruno
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>

I still think the only *real* obstacle remains the lack of important
packages such as twisted, gevent and pika which haven't been ported yet.
With those ones ported switching to Python 3 *right now* is not only
possible and relatively easy, but also convenient.


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

[Giampaolo Rodola']

On Fri, Sep 26, 2014 at 8:01 PM, Steve Dower 
wrote:

> Hi all,
>
> (This is advance notice since people on this list will be interested.
> Official announcements are coming when setuptools makes their next release.)
>
> Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9).
> We've produced this package to help library developers build wheels for
> Windows, but also to help users unblock themselves when they need to build
> C extensions themselves.
>
> The Microsoft Visual C++ Compiler for Python 2.7 is available from:
> http://aka.ms/vcpython27
>
> This package contains all the tools and headers required to build C
> extension modules for Python 2.7 32-bit and 64-bit (note that some
> extension modules require 3rd party dependencies such as OpenSSL or libxml2
> that are not included). Other versions of Python built with Visual C++ 2008
> are also supported.
>
> You can install the package without requiring administrative privileges
> and, with the latest version of setuptools (when it releases), use tools
> such as pip, wheel, or a setup.py file to produce binaries on Windows.
>
> Unfortunately, this package isn't necessarily going to help with building
> CPython 2.7 itself, as the build process is complicated and Visual Studio
> 2008 is practically required. However, as most people aren't building
> CPython on Windows, this isn't a huge issue. This compiler package should
> be sufficient for most extension modules.
>
> I should also point out that VC9 is no longer supported by Microsoft. This
> means there won't be any improvements or bug fixes coming, and there's no
> official support offered. Feel free to contact me directly <
> steve.do...@microsoft.com> if there are issues with the package.
>
> Cheers,
> Steve
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>

This is awesome, thanks.
I guess Python 2.6 is not supported, isn't it?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bluetooth 4.0 support in "socket" module

[Giampaolo Rodola']

On Mon, Jul 14, 2014 at 3:57 PM, Tim Tisdall  wrote:

> I was interested in providing patches for the socket module to add
> Bluetooth 4.0 support.  I couldn't find any details on how to provide
> contributions to the Python project, though...  Is there some online
> documentation with guidelines on how to contribute?  Should I just provide
> a patch to this mailing list?
>
> Also, is there a method to test changes against all the different *nix
> variations?  Is Bluez the standard across the different *nix variations?
>
> -Tim
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>
>
Hello there,
you can take a look at:
https://docs.python.org/devguide/#contributing
Patches must be submitted on the Python bug tracker:
http://bugs.python.org/

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler

[Giampaolo Rodola']

On Sat, Jun 7, 2014 at 7:05 AM, Donald Stufft  wrote:

>
> I don’t particularly care too much though, I just think that bumping
> the compiler in a 2.7.Z release is a really bad idea and that either
> of the other two options are massively better.


+1


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 469: Restoring the iterkeys/values/items() methods

[Giampaolo Rodola']

On Sat, Apr 19, 2014 at 4:31 AM, Nick Coghlan  wrote:

> After spending some time talking to the folks at the PyCon Twisted
> sprints, they persuaded me that adding back the iterkeys/values/items
> methods for mapping objects would be a nice way to eliminate a key
> porting hassle for them (and likely others), without significantly
> increasing the complexity of Python 3.
>

I don't see this as a key porting hassle *at all* and I don't understand
why they think this would significantly help their porting (it wouldn't).
The only real barrier is the str/bytes conversion, really, and this is even
more true for projects massively centered around IO, such as Twisted and,
I'm sure, the main (only?) reason why Twisted hasn't been ported yet. They
will get much more benefit from additions such as PEP-461, which is of
great help for verbose protocols such as FTP, not this.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

[Giampaolo Rodola']

On Thu, Apr 17, 2014 at 8:17 PM, Antoine Pitrou  wrote:

> On Thu, 17 Apr 2014 18:09:22 +
> Brett Cannon  wrote:
> > >
> > >I would really love to have better startup times in production, but
> I
> > > would also really hate to lose the ability to hack around in stdlib
> > > sources during development just to get better startup performance.
> > >
> > >In general, what I really like about using Python for software
> > > development is the ability to open any stdlib file and easily go poking
> > > around using stuff like 'import pdb;pdb.set_trace()' or simple print
> > > statements. Researching mysterious behaviour is generally much much
> > > MUCH! easier (read: takes less hours/days/weeks) if it ends up leading
> > > into a stdlib Python module than if it takes you down into the bowels
> of
> > > some C module (think zipimport.c *grin*). Not to mention the effect
> that
> > > being able to quickly resolve a mystery by hacking on some Python
> > > internals leaves you feeling very satisfied, while having to entrench
> > > yourself in those internals for a long time just to find out you've
> made
> > > something foolish on your end leaves you feeling exhausted at best.
> > >
> >
> > Freezing modules does not affect the ability to use gdb. And as long as
> you
> > set the appropriate __file__ values then tracebacks will contain even the
> > file line and location.
>
> I sympathize with Jurko's opinion. Being able to poke inside stdlib
> source files makes Python more approachable. I'm sure several of us got
> into Python that way.
>
> Regards
>
> Antoine.


I also wouldn't want that to be the default but Martin also suggested a -Z
cmdline option which sounds like an interesting idea to me.
...Or maybe simply use the existent -O option, which doesn't really
optimize much AFAIK.


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: fix #21076: turn signal module constants into enums

[Giampaolo Rodola']

Sorry for the troubles. :(
I committed this because it worked on my local copy of Python 3.5 but after
I tried a brand new "hg clone" it didnt.


On Fri, Apr 4, 2014 at 4:43 PM, Brett Cannon  wrote:

>
>
> On Fri Apr 04 2014 at 10:34:06 AM, Victor Stinner <
> victor.stin...@gmail.com> wrote:
>
>> 2014-04-04 16:21 GMT+02:00 Brett Cannon :
>> > Fix is in rev c6e63bb132fb.
>>
>> Hum, this one was not enough for me. I also modified Modules/
>> Setup.config.in:
>>
>
> Wasn't for me either in the end as it failed when I did a distclean. The
> unfortunately bit me in the middle of trying to get a 3.4->default merge so
> I just tried to do the best I could. This might take a little while to
> clean up as the Windows 7 buildbot now builds -- which it has been doing
> for a while -- but claims it can't find _signal.sigpending which I can at
> least see on OS X so that bit of code my need tweaking.
>
> -Brett
>
>
>>
>> changeset:   90137:df5120efb86e
>> tag: tip
>> user:Victor Stinner 
>> date:Fri Apr 04 16:30:04 2014 +0200
>> files:   Modules/Setup.config.in
>> description:
>> Issue #21076: the C signal module has been renamed to _signal
>>
>>
>> diff -r c6e63bb132fb -r df5120efb86e Modules/Setup.config.in
>> --- a/Modules/Setup.config.in   Fri Apr 04 10:20:28 2014 -0400
>> +++ b/Modules/Setup.config.in   Fri Apr 04 16:30:04 2014 +0200
>> @@ -7,7 +7,7 @@
>>  @USE_THREAD_MODULE@_thread _threadmodule.c
>>
>>  # The signal module
>> -@USE_SIGNAL_MODULE@signal signalmodule.c
>> +@USE_SIGNAL_MODULE@_signal signalmodule.c
>>
>>  # The rest of the modules previously listed in this file are built
>>  # by the setup.py script in Python 2.1 and later.
>>
>> Victor
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Fwd: shutdown concurrent.futures.ProcessPoolExecutor

[Giampaolo Rodola']

This is a follow up of:
https://groups.google.com/forum/#!topic/python-tulip/91NCCqV4SFs
According to the information I collected so far it seems it's not possible
(or at least very hard) to cleanly shutdown a process pool and all its
workers in case of KeyboardInterrupt / SIGINT.
Literally, what I would like to do is this:

- intercept SIGINT form the main process and/or from each worker process
- send SIGINT to all workers
- wait for them to finish with a timeout
- finally exit

Whereas it exists a solution based on multiprocessing:
http://noswap.com/blog/python-multiprocessing-keyboardinterrupt
...concurrent.futures.ProcessPoolExecutor does not expose all the necessary
bits to emulate it.

By opening this thread I hope either to find a solution and document it or
if it turns out that it is simply not possible to do this via
concurrent.futures then discuss whether it's the case to expose more
ProcessPoolExecutor APIs, because right now the interface it offers is
pretty limited compared to multiprocessing.Pool:
http://docs.python.org/3.5/library/multiprocessing.html#multiprocessing.pool.Pool
In particular, it won't let you pass an initializer function to pass to
multiprocessing.Pool nor terminate() the process pool (only wait() for it).



-- Forwarded message --
From: Guido van Rossum 
Date: Sat, Mar 29, 2014 at 12:05 AM
Subject: Re: [python-tulip] How to cleanly shutdown the IO loop when using
run_in_executor()?
To: Giampaolo Rodola' 
Cc: python-tulip 


You're going to have to move the discussion to python-dev or the python
issue tracker then.


On Fri, Mar 28, 2014 at 4:02 PM, Giampaolo Rodola' wrote:

>
> On Wed, Mar 26, 2014 at 7:35 PM, Guido van Rossum wrote:
>
>> Another thing to investigate might be how the executor creates the
>> processes, and if anything happens to signals there.
>
>
> After some further digging this seems to be related with the problem at
> hand.
> According to this:
> http://noswap.com/blog/python-multiprocessing-keyboardinterrupt
> ...it appears a feasible solution is to prevent workers to ever receive
> KeyboardInterrupt and have the main process shutdown the pool via
> pool.terminate() and finally pool.join().
> Unfortunately concurrent.futures.ProcessPoolExecutor does not expose all
> the necessary bits to do that (multiprocessing.Pool(initializer=...)
> argument and terminate() method).
>
> I also suspect that in order to emulate the suggested solution the
> underlying Process instance should be daemonized, similarly to what
> multiprocessing.Pool does:
>
> http://hg.python.org/cpython/file/3567d7ebd382/Lib/multiprocessing/pool.py#l222
>
> I wonder whether concurrent.futures.ProcessPoolExecutor would be better
> off exposing more APIs in order to facilitate tasks like these.
>
>
> --
> Giampaolo - http://grodola.blogspot.com
>
>


-- 
--Guido van Rossum (python.org/~guido)



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Python 3.4.0

[Giampaolo Rodola']

The what's new looks truly amazing, with pathlib and asyncio being my
favourite additions.
Thanks for all the hard work.


On Mon, Mar 17, 2014 at 5:57 PM, Ryan Gonzalez  wrote:

> YES!!! +1 to the authors of the statistics and pathlib modules.
>
>
> On Mon, Mar 17, 2014 at 1:29 AM, Larry Hastings wrote:
>
>>
>> On behalf of the Python development team, I'm thrilled to announce
>> the official release of Python 3.4.
>>
>>
>> Python 3.4 includes a range of improvements of the 3.x series, including
>> hundreds of small improvements and bug fixes.  Major new features and
>> changes in the 3.4 release series include:
>>
>> * PEP 428, a "pathlib" module providing object-oriented filesystem paths
>> * PEP 435, a standardized "enum" module
>> * PEP 436, a build enhancement that will help generate introspection
>>information for builtins
>> * PEP 442, improved semantics for object finalization
>> * PEP 443, adding single-dispatch generic functions to the standard
>> library
>> * PEP 445, a new C API for implementing custom memory allocators
>> * PEP 446, changing file descriptors to not be inherited by default
>>in subprocesses
>> * PEP 450, a new "statistics" module
>> * PEP 451, standardizing module metadata for Python's module import system
>> * PEP 453, a bundled installer for the *pip* package manager
>> * PEP 454, a new "tracemalloc" module for tracing Python memory
>> allocations
>> * PEP 456, a new hash algorithm for Python strings and binary data
>> * PEP 3154, a new and improved protocol for pickled objects
>> * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O
>>
>>
>> To download Python 3.4.0 visit:
>>
>> http://www.python.org/download/releases/3.4.0/
>>
>>
>> This is a production release.  Please report any issues you notice to:
>>
>>  http://bugs.python.org/
>>
>>
>> Enjoy!
>>
>>
>> --
>> Larry Hastings, Release Manager
>> larry at hastings.org
>> (on behalf of the entire python-dev team and 3.4's contributors)
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> rymg19%40gmail.com
>>
>
>
>
> --
> Ryan
> If anybody ever asks me why I prefer C++ to C, my answer will be simple:
> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
> nul-terminated."
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Confirming status of new modules in 3.4

[Giampaolo Rodola']

On Sat, Mar 15, 2014 at 3:17 AM, Guido van Rossum  wrote:

> I don't think so. asyncio depends on selectors but not vice versa. The
> selectors module was not part of PEP 3156. It is solid and I don't see any
> reason why it should get a reprieve from the usual strict backwards
> compatibility standards.


One part which can be improved is that right now the selectors module
doesn't take advance of e/poll()'s modify() method: instead it just
unregister() and register() the fd every time, which is of course
considerably slower (there's also a TODO in the code about this).
I guess that can be fixed later in a safely manner.

Another concern I have is that we should probably rename self._epoll,
self._select, self._kqueue to just "self._poller": that would make
subclassing easier (see patch in issue http://bugs.python.org/issue18931)
and would provide a unified interface for those users who want to reference
the underlying poller object for some reason.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Confirming status of new modules in 3.4

[Giampaolo Rodola']

On Fri, Mar 14, 2014 at 9:25 PM, R. David Murray wrote:

> Not Provisional:
>
> selectors
>

Wouldn't it be wiser to consider this one provisional as well?


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Removing doc from pythonhosted.org

[Giampaolo Rodola']

I'm not sure whether this is the right mailing list where to post this.
However, it seems the pypi UI currently provides a way to upload doc at the
bottom of the page
https://pypi.python.org/pypi?%3Aaction=pkg_edit&name=PROJECT_NAME but
there's currently no way to remove it:
https://bitbucket.org/pypa/pypi/issue/24/allow-deleting-project-documentation-from
http://sourceforge.net/p/pypi/support-requests/294/
https://github.com/litl/rauth/issues/81
The only workaround as of
http://stackoverflow.com/questions/6521931/how-to-delete-documentation-from-pypiappears
to be uploading a new .zip file with a redirect.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 428 - pathlib - ready for approval

[Giampaolo Rodola']

On Tue, Nov 19, 2013 at 10:04 PM, Antoine Pitrou wrote:

>
> Hello,
>
> Guido has told me that he was ready to approve PEP 428 (pathlib) in its
> latest amended form.  Here is the last call for any comments or
> arguments against approval, before Guido marks the PEP accepted (or
> changes his mind :-)).
>
> Regards
>
> Antoine.
>


Isn't this redundant?

>>> Path.cwd()
PosixPath('/home/antoine/pathlib')

Probably this is just personal taste but I'd prefer the more explicit:

>>> Path(os.getcwd())
PosixPath('/home/antoine/pathlib')

I understand all the os.* replication (Path.rename, Path.stat etc.) but all
these methods assume you're dealing with an instantiated Path instance
whereas Path.cwd is the only one which doesn't.
Not a big deal anyway, it just catched my eye because it's different.
Other than that the module looks absolutely awesome and a big improvement
over os.path!


--- Giampaolo
https://code.google.com/p/pyftpdlib/
https://code.google.com/p/psutil/
https://code.google.com/p/pysendfile/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore().

[Giampaolo Rodola']

On Fri, Oct 11, 2013 at 11:00 AM, Antoine Pitrou wrote:

>
> Let me answer here to Nick's argument on the tracker (made last year,
> before the patch was committed):
>
> > As with many context managers, a key benefit here is in the priming
> > effect for readers. In this code:
> >
> > try:
> ># Whatever
> > except (A, B, C):
> >pass
> >
> > the reader doesn't know that (A, B, C) exceptions will be ignored
> > until the end. The with statement form makes it clear before you
> > start reading the code that certain exceptions won't propagate:
> >
> > with ignored(A, B, C):
> > # Whatever
>
> The problem with this argument is that it assumes a very specific idiom:
> i.e. writing long "try" blocks in the purpose of silencing exceptions.
>
> I'd like to make the following points:
>
> - when catching an exception, the common (and recommended) behaviour is
>   to do something else - not merely silencing it.  Silencing is not very
>   common in my experience, except in badly written code
>
> - when catching an exception, it is recommended for the "try" block to
>   be as slim as possible - so that you don't catch other unintended
>   exceptions by mistake. This is a point I already made in PEP 3151.
>   Many exception classes (OSError, KeyError, RuntimeError...) are
>   polysemic.
>
> The bottom line is that there shouldn't be any long "try" blocks
> followed by a single "except FooException: pass" clause in well-written
> code. The presence of such an idiom is a strong code smell.
>
> Therefore contextlib.ignore() seems aimed at making it easier to write
> bad code, not good code. I don't think it should exist in the stdlib.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>


I'm with Antoine here.
Despite "with ignore(OSError): os.remove" looks appealing to the eye I
think many people will end up thinking it's fine to write long "ignore"
blocks, because that's perfectly fine when using the "with" statement.
Point is this is actually a "try" block disguised as a "with", and "try"
blocks should usually be followed by very few indented lines (usually just
1) for a very good reason.

-1

--- Giampaolo
https://code.google.com/p/pyftpdlib/
https://code.google.com/p/psutil/
https://code.google.com/p/pysendfile/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Help requested for issue 9285 (profile.py)

[Giampaolo Rodola']

http://bugs.python.org/issue9285#msg182986
I'm stuck as I really have no clue what that error means.
Any help from someone experienced with profile.py code is welcome.

--- Giampaolo
https://code.google.com/p/pyftpdlib/
https://code.google.com/p/psutil/
https://code.google.com/p/pysendfile/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Should ftplib use UTF-8 instead of latin-1 encoding?

[Giampaolo Rodola']

Hi,
while attempting to port pyftpdlib [1] to Python 3 I have noticed that
ftplib differs from the previous 2.x version in that it uses latin-1
to encode everything it's sent over the FTP command channel, but by
reading RFC-2640 [2] it seems that UTF-8 should be preferred instead.
I'm far from being an expert of encodings, plus the RFC is quite hard
to understand, so sorry in advance if I have misunderstood the whole
thing.
Just wanted to put this up to people more qualified than me.


[1] http://code.google.com/p/pyftpdlib
[2] http://www.ietf.org/rfc/rfc2640.txt


--- Giampaolo
http://code.google.com/p/pyftpdlib
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python security team

[Giampaolo Rodola']

Yeah, right. Let's continue there.

--- Giampaolo
http://code.google.com/p/pyftpdlib



On 29 Set, 22:44, "Josiah Carlson" <[EMAIL PROTECTED]> wrote:
> On Mon, Sep 29, 2008 at 12:02 PM, Giampaolo Rodola' <[EMAIL PROTECTED]> wrote:
> > On 27 Set, 20:04, "Josiah Carlson" <[EMAIL PROTECTED]> wrote:
> >> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
>
> >> <[EMAIL PROTECTED]> wrote:
> >> > Second, I would like to help to fix all Python security issues. It looks 
> >> > like
> >> > Python community isn't very reactive (proactive?) about security. Eg. a 
> >> > DoS
> >> > was reported in smtpd server (integrated to Python)... 15 months ago. A 
> >> > patch
> >> > is available but it's not applied in Python trunk.
>
> >> The smtpd module is not meant to be used without modification.  It is
> >> the responsibility of the application writer to decide the limitations
> >> of the emails they want to allow sending, and subsequently handle the
> >> case where emails overrun that limit.
>
> > The issue does not concern the emails but the buffer used internally
> > to store the received raw data sent by client.
> > The user who wants to fix the issue (#1745035) should override the
> > collect_incoming_data method which is usually not meant to be
> > modified.
> > Moreover, there are two RFCs which state that extremely long lines
> > must be truncated and an error reply must be returned.
>
> We can and should discuss the specifics of this item in the bug report
> itself.  I should have replied there instead.
>
>  - Josiah
> ___
> Python-Dev mailing list
> [EMAIL PROTECTED]://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...-
>  Nascondi testo citato
>
> - Mostra testo citato -
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python security team

[Giampaolo Rodola']

On 27 Set, 20:04, "Josiah Carlson" <[EMAIL PROTECTED]> wrote:
> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
>
> <[EMAIL PROTECTED]> wrote:
> > Second, I would like to help to fix all Python security issues. It looks 
> > like
> > Python community isn't very reactive (proactive?) about security. Eg. a DoS
> > was reported in smtpd server (integrated to Python)... 15 months ago. A 
> > patch
> > is available but it's not applied in Python trunk.
>
> The smtpd module is not meant to be used without modification.  It is
> the responsibility of the application writer to decide the limitations
> of the emails they want to allow sending, and subsequently handle the
> case where emails overrun that limit.  

The issue does not concern the emails but the buffer used internally
to store the received raw data sent by client.
The user who wants to fix the issue (#1745035) should override the
collect_incoming_data method which is usually not meant to be
modified.
Moreover, there are two RFCs which state that extremely long lines
must be truncated and an error reply must be returned.

--- Giampaolo
http://code.google.com/p/pyftpdlib/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ssl module, non-blocking sockets and asyncore integration

[Giampaolo Rodola']

Some good news: I finally figured out how to modify asyncore to make
it properly handle the non-blocking ssl-handshake.
I provided a patch for test_ssl.py in issue 3899.
Bill, could you please review it?


--- Giampaolo
http://code.google.com/p/pyftpdlib/

On 18 Set, 00:49, "Giampaolo Rodola'" <[EMAIL PROTECTED]> wrote:
> Ok, here's some news, in case they can be of some interest.
> I managed to write an asyncore disptacher which seems to work.
> I used my test suite against it and 70 tests passed and 2 failed.
> The tests failed because at a certain point a call to do_handhsake
> results in an EOF exception, which is very strange since it is
> supposed to raise SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE only.
> I'll keep you updated in case I have some news.
>
> --- Exception ---
>
>   File "C:\python26\lib\ssl.py", line 293, in do_handshake
>     self._sslobj.do_handshake()
> SSLError: [Errno 8] _ssl.c:480: EOF occurred in violation of protocol
>
> --- SSL dispatcher 
>
> class SSLConnection(asyncore.dispatcher):
>
>     def __init__(self):
>         self.ssl_handshake_pending = False
>
>     def do_ssl_handshake(self):
>         try:
>             self.socket.do_handshake()
>         except ssl.SSLError, err:
>             if err.args[0] == ssl.SSL_ERROR_WANT_READ:
>                 self.ssl_handshake_pending = 'read'
>             elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
>                 self.ssl_handshake_pending = 'write'
>             else:
>                 raise
>         else:
>             self.ssl_handshake_pending = False
>
>     def handle_read_event(self):
>         if self.ssl_handshake_pending == 'read':
>             self.do_ssl_handshake()
> ##            if not self.ssl_handshake_pending:
> ##                asyncore.dispatcher.handle_read_event(self)
>         else:
>             asyncore.dispatcher.handle_read_event(self)
>
>     def handle_write_event(self):
>         if self.ssl_handshake_pending == 'write':
>             self.do_ssl_handshake()
> ##            if not self.ssl_handshake_pending:
> ##                asyncore.dispatcher.handle_write_event(self)
>         else:
>             asyncore.dispatcher.handle_write_event(self)
>
> --- Giampaolohttp://code.google.com/p/pyftpdlib/
> ___
> Python-Dev mailing list
> [EMAIL PROTECTED]://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ssl module, non-blocking sockets and asyncore integration

[Giampaolo Rodola']

Ok, here's some news, in case they can be of some interest.
I managed to write an asyncore disptacher which seems to work.
I used my test suite against it and 70 tests passed and 2 failed.
The tests failed because at a certain point a call to do_handhsake
results in an EOF exception, which is very strange since it is
supposed to raise SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE only.
I'll keep you updated in case I have some news.



--- Exception ---

  File "C:\python26\lib\ssl.py", line 293, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 8] _ssl.c:480: EOF occurred in violation of protocol



--- SSL dispatcher 

class SSLConnection(asyncore.dispatcher):

def __init__(self):
self.ssl_handshake_pending = False

def do_ssl_handshake(self):
try:
self.socket.do_handshake()
except ssl.SSLError, err:
if err.args[0] == ssl.SSL_ERROR_WANT_READ:
self.ssl_handshake_pending = 'read'
elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
self.ssl_handshake_pending = 'write'
else:
raise
else:
self.ssl_handshake_pending = False

def handle_read_event(self):
if self.ssl_handshake_pending == 'read':
self.do_ssl_handshake()
##if not self.ssl_handshake_pending:
##asyncore.dispatcher.handle_read_event(self)
else:
asyncore.dispatcher.handle_read_event(self)

def handle_write_event(self):
if self.ssl_handshake_pending == 'write':
self.do_ssl_handshake()
##if not self.ssl_handshake_pending:
##asyncore.dispatcher.handle_write_event(self)
else:
asyncore.dispatcher.handle_write_event(self)



--- Giampaolo
http://code.google.com/p/pyftpdlib/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ssl module, non-blocking sockets and asyncore integration

[Giampaolo Rodola']

Sorry, ignore my 2nd question, I see now that you already gave a very
clear answer in your first message.
I change my question: how am I supposed to know when the SSL hanshake
is completed? When pending() returns False?
If so I'd recommend to document the method.


--- Giampaolo
http://code.google.com/p/pyftpdlib/

On 17 Set, 03:24, "Giampaolo Rodola'" <[EMAIL PROTECTED]> wrote:
> I've tried to modify my existing asyncore-based code but I'm
> encountering a lot of different problems I didn't manage to fix.
> It seems that playing with the do_handshake_on_connect flag doesn't
> make any difference.
> I guess that without some kind of documentation describing how to deal
> with non-blocking "ssl-wrapped" sockets I won't get too far.
> I try to ask two questions in case the answers may help me in some
> way:
>
> 1 - What pending() method is supposed to do (it's not documented)?
> 2 - By reading ssl.py code I noticed that when do_handshake_on_connect
> flag is False the do_handshake() method is never called. Is it
> supposed to be manually called when dealing with non-blocking sockets?
>
> In the meanwhile I noticed something in the ssl.py code which seems to
> be wrong:
>
>     def recv (self, buflen=1024, flags=0):
>         if self._sslobj:
>             if flags != 0:
>                 raise ValueError(
>                     "non-zero flags not allowed in calls to sendall()
> on %s" %
>                     self.__class__)
>             while True:
>                 try:
>                     return self.read(buflen)
>                 except SSLError, x:
>                     if x.args[0] == SSL_ERROR_WANT_READ:
>                         continue
>                     else:
>                         raise x
>         else:
>             return socket.recv(self, buflen, flags)
>
> I don't know the low levels but that while statement which continues
> in case of SSL_ERROR_WANT_READ seems to be wrong (blocking), at least
> when dealing with non-blocking sockets. I think the proper way of
> doing recv() here is letting SSL_ERROR_WANT_READ propagate and let the
> upper application (e.g. asyncore) deal with it.
>
> Hope this helps,
>
> --- Giampaolohttp://code.google.com/p/pyftpdlib/downloads/list
>
> On 15 Set, 04:50, Bill Janssen <[EMAIL PROTECTED]> wrote:
>
>
>
> > Hi, Giampaolo.
>
> > If you look a bit further in Lib/test/test_ssl.py, you'll see a
> > non-blocking use of the "do_handshake" method.  Basically, the flag
> > "do_handshake_on_connect" says whether you want this to happen
> > automatically and blockingly (True), or whether you want to do it
> > yourself (False).  In the test suite, the function
> > "testNonBlockingHandshake" does the async client-side handshake; the
> > server side logic is just the same, only it would happen in the server's
> > "handle new connection" code -- you'd have to add a state variable, and
> > bind handlers for "read_event" and "write_event", which would consult
> > the state variable to see whether they had finished the handshake yet.
>
> > I just made the server do it automatically to make life easier.
>
> > The hard part isn't really doing the non-blocking, it's trying to figure
> > out how to use asyncore correctly, IMO.
>
> > Giampaolo Rodola' <[EMAIL PROTECTED]> wrote:
> > > I'm interested in using the ssl module with asyncore but since there's
> > > no real documentation about how using ssl module with non-blocking
>
> > If you'd like to contribute a doc patch, that would be great.
>
> > Here's what it current says for do_handshake_on_connect:
>
> >   The parameter do_handshake_on_connect specifies whether to do the SSL
> >   handshake automatically after doing a socket.connect(), or whether the
> >   application program will call it explicitly, by invoking the
> >   SSLSocket.do_handshake() method. Calling SSLSocket.do_handshake()
> >   explicitly gives the program control over the blocking behavior of the
> >   socket I/O involved in the handshake.
>
> > and here's what the docs for do_handshake() says:
>
> >   SSLSocket.do_handshake()¦ Perform a TLS/SSL handshake. If this is used
> >   with a non-blocking socket, it may raise SSLError with an arg[0] of
> >   SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, in which case it must be
> >   called again until it completes successfully. For example, to simulate
> >   the behavior of a blocking socket, one might write:
>
> >     while True:
>

Re: [Python-Dev] ssl module, non-blocking sockets and asyncore integration

[Giampaolo Rodola']

I've tried to modify my existing asyncore-based code but I'm
encountering a lot of different problems I didn't manage to fix.
It seems that playing with the do_handshake_on_connect flag doesn't
make any difference.
I guess that without some kind of documentation describing how to deal
with non-blocking "ssl-wrapped" sockets I won't get too far.
I try to ask two questions in case the answers may help me in some
way:

1 - What pending() method is supposed to do (it's not documented)?
2 - By reading ssl.py code I noticed that when do_handshake_on_connect
flag is False the do_handshake() method is never called. Is it
supposed to be manually called when dealing with non-blocking sockets?

In the meanwhile I noticed something in the ssl.py code which seems to
be wrong:

def recv (self, buflen=1024, flags=0):
if self._sslobj:
if flags != 0:
raise ValueError(
"non-zero flags not allowed in calls to sendall()
on %s" %
self.__class__)
while True:
try:
return self.read(buflen)
except SSLError, x:
if x.args[0] == SSL_ERROR_WANT_READ:
continue
else:
raise x
else:
return socket.recv(self, buflen, flags)

I don't know the low levels but that while statement which continues
in case of SSL_ERROR_WANT_READ seems to be wrong (blocking), at least
when dealing with non-blocking sockets. I think the proper way of
doing recv() here is letting SSL_ERROR_WANT_READ propagate and let the
upper application (e.g. asyncore) deal with it.


Hope this helps,

--- Giampaolo
http://code.google.com/p/pyftpdlib/downloads/list


On 15 Set, 04:50, Bill Janssen <[EMAIL PROTECTED]> wrote:
> Hi, Giampaolo.
>
> If you look a bit further in Lib/test/test_ssl.py, you'll see a
> non-blocking use of the "do_handshake" method.  Basically, the flag
> "do_handshake_on_connect" says whether you want this to happen
> automatically and blockingly (True), or whether you want to do it
> yourself (False).  In the test suite, the function
> "testNonBlockingHandshake" does the async client-side handshake; the
> server side logic is just the same, only it would happen in the server's
> "handle new connection" code -- you'd have to add a state variable, and
> bind handlers for "read_event" and "write_event", which would consult
> the state variable to see whether they had finished the handshake yet.
>
> I just made the server do it automatically to make life easier.
>
> The hard part isn't really doing the non-blocking, it's trying to figure
> out how to use asyncore correctly, IMO.
>
> Giampaolo Rodola' <[EMAIL PROTECTED]> wrote:
> > I'm interested in using the ssl module with asyncore but since there's
> > no real documentation about how using ssl module with non-blocking
>
> If you'd like to contribute a doc patch, that would be great.
>
> Here's what it current says for do_handshake_on_connect:
>
>   The parameter do_handshake_on_connect specifies whether to do the SSL
>   handshake automatically after doing a socket.connect(), or whether the
>   application program will call it explicitly, by invoking the
>   SSLSocket.do_handshake() method. Calling SSLSocket.do_handshake()
>   explicitly gives the program control over the blocking behavior of the
>   socket I/O involved in the handshake.
>
> and here's what the docs for do_handshake() says:
>
>   SSLSocket.do_handshake()¦ Perform a TLS/SSL handshake. If this is used
>   with a non-blocking socket, it may raise SSLError with an arg[0] of
>   SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, in which case it must be
>   called again until it completes successfully. For example, to simulate
>   the behavior of a blocking socket, one might write:
>
> while True:
> try:
> s.do_handshake()
> break
> except ssl.SSLError, err:
> if err.args[0] == ssl.SSL_ERROR_WANT_READ:
> select.select([s], [], [])
> elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
> select.select([], [s], [])
> else:
> raise
>
> Bill
> ___
> Python-Dev mailing list
> [EMAIL PROTECTED]://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Add python.exe to PATH environment variable

[Giampaolo Rodola']

Hi,
I've always found it strange that Python Windows installers never
managed to add the python executable to the PATH environment variable.
Are there plans for adding such a thing?

Thanks in advance

--- Giampaolo
http://code.google.com/p/pyftpdlib/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore patch

[Giampaolo Rodola']

On 10 Giu, 07:01, "Josiah Carlson" <[EMAIL PROTECTED]> wrote:
> On Mon, Jun 9, 2008 at 7:19 PM, Benjamin Peterson
>
> <[EMAIL PROTECTED]> wrote:
> > On Mon, Jun 9, 2008 at 8:42 PM, Josiah Carlson <[EMAIL PROTECTED]> wrote:
>
> >> Would it be ok if I committed the changes?  Neal, do you want to
> >> commit the changes if I post an updated patch with a blurb for the
> >> NEWS file?
>
> > You are the asyncore maintainer, correct? I believe it's pretty much
> > up to you, then. :)
>
> Indeed, but I didn't want to step on anyone's toes.
>
> It's committed in revision 64062 for anyone who cares.
>
>  - Josiah


I've started to test the new code by using the pyftpdlib test suite.
On Windows all tests passed but on Linux I get some "EBADF Bad file
descriptor" errors occurring when using poll() instead of select().
I'll try to look into them today and open a report if necessary.
In the meanwhile I noticed some minor bugs in asyncore.py. Take a look
at the patch below:


Index: Lib/asyncore.py
===
--- Lib/asyncore.py (revisione 64069)
+++ Lib/asyncore.py (copia locale)
@@ -228,7 +228,7 @@
 # passed be connected.
 try:
 self.addr = sock.getpeername()
-except socket.error:
+except socket.error, err:
 if err[0] == ENOTCONN:
 # To handle the case where we got an unconnected
 # socket.
@@ -424,7 +424,7 @@
 #check for errors
 err = self.socket.getsockopt(socket.SOL_SOCKET,
socket.SO_ERROR)
 if err != 0:
-raise socket.error(err, strerror(err))
+raise socket.error(err, _strerror(err))

 self.handle_connect_event()
 self.handle_write()


--- Giampaolo
http://code.google.com/p/pyftpdlib/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modules for 2.6 inclusion

[Giampaolo Rodola']

On 6 Giu, 13:27, Georg Brandl <[EMAIL PROTECTED]> wrote:

> - setuptools
>BDFL pronouncement for inclusion in 2.5:
>http://mail.python.org/pipermail/python-dev/2006-April/063964.html

I'd like to see more interest about this issue since it's a real shame
that the current distutils is not even able to solve dependencies and
has such a poor integration with Pypi.
Having a *valid* distribution tool like setuptools integrated with
Python would be the first step to have one of the most important
things Python is currently missing finally available: a centralized
archive of softwares like Perl CPAN or Ruby Gems.
IMHO this should be put on top of the "TODO" list and I'm honestly
surprised it raised so little clamor so far.


--- Giampaolo
http://code.google.com/p/pyftpdlib
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] socket.try_reuse_address()

[Giampaolo Rodola']

Maybe it would be better considering Windows CE systems too:

- if os.name == 'nt'
+ if os.name in ('nt', 'ce')

Moreover if the method is called "try_reuse_address" I'd expect that
"self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)" would be placed
inside a try/except block.

On 29 Apr, 15:58, Trent Nelson <[EMAIL PROTECTED]> wrote:
> Since the recent changes to networking-oriented tests (issue 2550, r62234 and 
> r62237), I think it's safe to say stability of the test suite on all the 
> non-Windows platforms has improved significantly in this area (I don't recall 
> seeing any socket errors in *nix buildbot logs since those commits).
>
> However, Windows buildbots are still periodically failing.  More 
> specifically, my Windows buildbots are still failing.  One thing that's 
> different about my buildbots is that two are being run at the same time for 
> both trunk and py3k -- one doing an x86 build, the other doing x64 build.
>
> Since the changes in the aforementioned revisions, the behaviour of my 
> buildbots has definitely improved -- they no longer completely wedge on 
> test_async(chat|core), mainly due to abolishing all use of SO_REUSEADDR as a 
> socket option in any network-oriented tests.
>
> However, periodically, they're still dying/failing in a variety of ways -- 
> see relevant log snippets at the end of this e-mail for examples.  I 
> attribute this to the fact that SO_REUSEADDR is still set as a socket option 
> in asyncore.py and SocketServer.py.  Basically, SO_REUSEADDR should *never* 
> be set on Windows for TCP/IP sockets.  Using asyncore.py as an example, here 
> are two ways we could handle this:
>
> 1. Hard code the Windows opt-out:
> --- asyncore.py (revision 62509)
> +++ asyncore.py (working copy)
> @@ -267,6 +267,8 @@
>
>      def set_reuse_addr(self):
>          # try to re-use a server port if possible
> +        if os.name == 'nt' and self.socket.socket_type != socket.SOCK_DGRAM:
> +            return
>          try:
>              self.socket.setsockopt(
>                  socket.SOL_SOCKET, socket.SO_REUSEADDR,
>
> 2. Introduce socket.try_reuse_address():
> --- asyncore.py (revision 62509)
> +++ asyncore.py (working copy)
> @@ -266,15 +266,7 @@
>          self.add_channel(map)
>
>      def set_reuse_addr(self):
> -        # try to re-use a server port if possible
> -        try:
> -            self.socket.setsockopt(
> -                socket.SOL_SOCKET, socket.SO_REUSEADDR,
> -                self.socket.getsockopt(socket.SOL_SOCKET,
> -                                       socket.SO_REUSEADDR) | 1
> -                )
> -        except socket.error:
> -            pass
> +        self.socket.try_reuse_address()
>
> With try_use_address implemented as follows:
>
> --- socket.py   (revision 62509)
> +++ socket.py   (working copy)
> @@ -197,6 +197,10 @@
>          Return a new socket object connected to the same system resource."""
>          return _socketobject(_sock=self._sock)
>
> +    def try_reuse_address(self):
> +        if not (os.name == 'nt' and self._sock.type != SOCK_DGRAM):
> +            self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
> +
>      def makefile(self, mode='r', bufsize=-1):
>          """makefile([mode[, bufsize]]) -> file object
>
> I prefer the latter as it's cleaner, easier to document and encapsulates what 
> we're trying to do relatively well.  The affected modules would be 
> asyncore.py, SocketServer.py and idlelib/rpc.py.  Thoughts?
>
> Regards,
>
>         Trent.
>
> 
> test_ftplib
>
> remoteFailed: [Failure instance: Traceback (failure with no frames): 
> twisted.internet.error.ConnectionLost: Connection to the other side was lost 
> in a non-clean fashion.
> ]
> 
>
> 
> test_asynchat
> test test_asynchat failed -- errors occurred; run in verbose mode for details
> [snip to bottom of log where test_asynchat is re-run]
> 1 test failed:
>     test_asynchat
> 33 tests skipped:
>     test__locale test_aepack test_applesingle test_cProfile
>     test_commands test_crypt test_curses test_dbm test_epoll
>     test_fcntl test_fork1 test_gdbm test_grp test_ioctl test_kqueue
>     test_macostools test_mhlib test_nis test_openpty test_ossaudiodev
>     test_pipes test_poll test_posix test_pty test_pwd test_resource
>     test_scriptpackages test_signal test_syslog test_threadsignals
>     test_wait3 test_wait4 test_zipfile64
> Those skips are all expected on win32.
> Re-running failed tests in verbose mode
> Re-running test 'test_asynchat' in verbose mode
> test_close_when_done (test.test_asynchat.TestAsynchat) ... ok
> test_empty_line (test.test_asynchat.TestAsynchat) ... ok
> test_line_terminator1 (test.test_asynchat.TestAsynchat) ... ok
> test_line_terminator2 (test.test_asynchat.TestAsynchat) ... ok
> test_line_terminator3 (test.test_asynchat.TestAsynchat) ... ok
> test_none_terminator (test.test_asynchat.TestAsynchat) ... ok
> test_numeric_terminator1 (test.test_asynchat.TestAsynchat) ... ok
> test_numeric_terminator2 (test.test_asynch

Re: [Python-Dev] fixing tests on windows

[Giampaolo Rodola']

On 1 Apr, 21:06, Tim Golden <[EMAIL PROTECTED]> wrote:
> Giampaolo Rodola' wrote:
>
> > On 1 Apr, 18:27, "Steven Bethard" <[EMAIL PROTECTED]> wrote:
> >> On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista
>
> >> <[EMAIL PROTECTED]> wrote:
> >>> 2008/4/1, Tim Golden <[EMAIL PROTECTED]>:
> >>>  >  If this is the thing to do, presumably test_support should
> >>>  >  grow a "remove_file" which does something of this sort?
> >>>  +1 (I was thinking exactly that).
> >> +1 here too. That looks like a great solution.  Of course, once it's
> >> in test_support, we need to fix *all* file removals in the test suite.
> >> ;-)
>
> >> Steve
>
> > Why not just modifying test_support.unlink() like this?
> > Surely more convenient than modifying the whole suite.
>
> > def unlink(filename):
> >     try:
> >         if os.name == 'nt':
> >             os.rename(filename, filename + ".deleted")
> >             filename = filename + ".deleted"
> >         os.unlink(filename)
> >     except OSError:
> >         pass
>
> Funnily enough, that's exactly what the patch I've
> put together does.

Sorry but maybe I misunderstood what you said above.
It seems to me you proposed to add a new "remove_file" function to
test_support while the solution I suggested was modifying the current
test_support.unlink() function in a similar manner you proposed and
have all tests use it wherever it is possible.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] fixing tests on windows

[Giampaolo Rodola']

On 1 Apr, 21:03, "Giampaolo Rodola'" <[EMAIL PROTECTED]> wrote:
> On 1 Apr, 18:27, "Steven Bethard" <[EMAIL PROTECTED]> wrote:
>
> > On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista
>
> > <[EMAIL PROTECTED]> wrote:
> > > 2008/4/1, Tim Golden <[EMAIL PROTECTED]>:
> > >  >  If this is the thing to do, presumably test_support should
> > >  >  grow a "remove_file" which does something of this sort?
>
> > >  +1 (I was thinking exactly that).
>
> > +1 here too. That looks like a great solution.  Of course, once it's
> > in test_support, we need to fix *all* file removals in the test suite.
> > ;-)
>
> > Steve
>
> Why not just modifying test_support.unlink() like this?
> Surely more convenient than modifying the whole suite.
>
> def unlink(filename):
>     try:
>         if os.name == 'nt':
>             os.rename(filename, filename + ".deleted")
>             filename = filename + ".deleted"
>         os.unlink(filename)
>     except OSError:
>         pass


Another solution, probably better:


def unlink(filename):
try:
os.unlink(filename)
except OSError:
if os.name == 'nt':
try:
os.rename(filename, filename + ".deleted")
os.unlink(filename + ".deleted")
except OSError:
pass
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] fixing tests on windows

[Giampaolo Rodola']

On 1 Apr, 18:27, "Steven Bethard" <[EMAIL PROTECTED]> wrote:
> On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista
>
> <[EMAIL PROTECTED]> wrote:
> > 2008/4/1, Tim Golden <[EMAIL PROTECTED]>:
> >  >  If this is the thing to do, presumably test_support should
> >  >  grow a "remove_file" which does something of this sort?
>
> >  +1 (I was thinking exactly that).
>
> +1 here too. That looks like a great solution.  Of course, once it's
> in test_support, we need to fix *all* file removals in the test suite.
> ;-)
>
> Steve

Why not just modifying test_support.unlink() like this?
Surely more convenient than modifying the whole suite.


def unlink(filename):
try:
if os.name == 'nt':
os.rename(filename, filename + ".deleted")
filename = filename + ".deleted"
os.unlink(filename)
except OSError:
pass
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com