[Python-Dev] Re: 1 PR, multiple issues

[Terry Reedy]

This was more of a core-mentorship question (tracker management) than a 
pydev question (Python development).  But since we are here...


On 6/10/2019 11:41 PM, Mariatta wrote:
Hmm, I personally would still consider the additional issues as 
duplicates. So I would make note in each of the duplicate issue saying 
"this is caused by the same problem as bpo-X", and close them, and 
attach the PR only to one bpo ticket.
To decide which one I would keep open, perhaps the oldest one, but maybe 
it depends. Perhaps whichever that seems to contain most detailed info?


Maybe other core devs have other preferences?


No me.  The general rule/convention has been to keep the oldest open and 
close others.  I put a note on the one kept open as to others closed.


If it is discovered after the fact that a patch for A fixed B, B can be 
closed as 'out-of-date' with a comment 'fixed by ...'


--
Terry Jan Reedy
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XKPMAEDXKUSOK6VPQXYRAP6JY4JYL37W/

[Python-Dev] Re: 1 PR, multiple issues

[Ned Deily]

On Jun 10, 2019, at 23:41, Mariatta  wrote:
> On Mon, Jun 10, 2019 at 8:19 PM Jeffrey Kintscher  
> wrote:
>> A hypothetical example would be two different issues whose descriptions 
>> appear to be for different behaviors, but are actually caused by the same 
>> underlying chunk of problematic code being called in different ways. It 
>> could be something trivial like a corner case not being handled.
> Hmm, I personally would still consider the additional issues as duplicates. 
> So I would make note in each of the duplicate issue saying "this is caused by 
> the same problem as bpo-X", and close them, and attach the PR only to one bpo 
> ticket.
> To decide which one I would keep open, perhaps the oldest one, but maybe it 
> depends. Perhaps whichever that seems to contain most detailed info? 

I agree.  If it seems better to solve everything in one PR, I would generally 
associate the PR with one issue and close the others as duplicates with 
explanations why.

--
  Ned Deily
  n...@python.org -- []
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LGOT7ID2QQC5FPY77YVX3MH3N3WZUEY2/

[Python-Dev] Re: 1 PR, multiple issues

[Mariatta]

Hmm, I personally would still consider the additional issues as duplicates.
So I would make note in each of the duplicate issue saying "this is caused
by the same problem as bpo-X", and close them, and attach the PR only to
one bpo ticket.
To decide which one I would keep open, perhaps the oldest one, but maybe it
depends. Perhaps whichever that seems to contain most detailed info?

Maybe other core devs have other preferences?

On Mon, Jun 10, 2019 at 8:19 PM Jeffrey Kintscher 
wrote:

> A hypothetical example would be two different issues whose descriptions
> appear to be for different behaviors, but are actually caused by the same
> underlying chunk of problematic code being called in different ways. It
> could be something trivial like a corner case not being handled.
>
> //Jeff
>
> ᐧ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6LYS2FPQXIYUAQKIATA2ZUAPNYR6OIAL/

[Python-Dev] Re: 1 PR, multiple issues

[Jeffrey Kintscher]

A hypothetical example would be two different issues whose descriptions 
appear to be for different behaviors, but are actually caused by the 
same underlying chunk of problematic code being called in different 
ways. It could be something trivial like a corner case not being handled.


//Jeff

On 6/10/19 7:57 PM, Mariatta wrote:
Usually we prefer 1 PR per 1 bpo ticket. The smaller the PR is, the 
easier it is for us to review, and less likely to cause conflict when 
it comes to auto-backporting.


When you say it resolves multiple issues, are these the same issue 
reported multiple times?

ᐧ

On Mon, Jun 10, 2019 at 7:52 PM Jeffrey Kintscher 
mailto:websur...@surf2c.net>> wrote:


When I create a PR that resolves multiple issues, should I add it
to each issue, or just to one issue and add relevant comments to
the others?

//Jeff

-- 
-

 From there to here, from here to there,
funny things are everywhere.
-- Theodore Geisel

___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/74FS2DW2L4MDJXRREAABXYRWNI6INMBA/



--
-
From there to here, from here to there,
funny things are everywhere.
   -- Theodore Geisel

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2UWQJWABSSJZD2YRJGMHZQ5WD64HV3UM/

[Python-Dev] Re: Enum and annotations

[Jonathan Goble]

On Mon, Jun 10, 2019 at 6:44 PM Steve Dower  wrote:
>
> I'd expect people coming from other languages to get it wrong this way too:
>
> class Status(Enum):
>  on
>  off
>
> Which will of course raise NameError and be just as opaque to the naive
> user as the AttributeError later on. I'm not sure we can do as much to
> help this case, but perhaps we can update __getattr__ to check
> __annotations__ on the class before failing and provide a clearer
> message? e.g. "AttributeError: 'on' was specified without assigning a
> value, did you use ':' instead of '='?" Or we could do this on
> construction, but that may rule out some interesting uses in the future
> if you have a need to delay specifying enum values.

As a side note, the aenum package on PyPI [1] supports this bare-name
notation with its AutoNumberEnum class, which uses some metaclass
magic to do it. aenum's code is complex, but an extremely bare-bones
version can be written in about 15 lines of code:

```
class AutoNumberDict(dict):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.count = 0

def __missing__(self, key):
if isinstance(key, str) and key.startswith('__') and key.endswith('__'):
raise KeyError(key)
self[key] = self.count
self.count += 1

class EnumType(type):
def __prepare__(name, bases):
return AutoNumberDict()

class Enum(metaclass=EnumType):
pass
```

Of course, there are some cases that this doesn't handle (such as
duplicate names or bit flags), but for a one-off script or a personal
project, it could be enough.

[1] https://pypi.org/project/aenum/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CJIWLLZ2JEN2TH4B33PNWXFBN4OAOSSD/

[Python-Dev] Re: 1 PR, multiple issues

[Mariatta]

Usually we prefer 1 PR per 1 bpo ticket. The smaller the PR is, the easier
it is for us to review, and less likely to cause conflict when it comes to
auto-backporting.

When you say it resolves multiple issues, are these the same issue reported
multiple times?
ᐧ

On Mon, Jun 10, 2019 at 7:52 PM Jeffrey Kintscher 
wrote:

> When I create a PR that resolves multiple issues, should I add it to each
> issue, or just to one issue and add relevant comments to the others?
>
> //Jeff
>
> --
> -
> From there to here, from here to there,
> funny things are everywhere.
>-- Theodore Geisel
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/74FS2DW2L4MDJXRREAABXYRWNI6INMBA/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/34H2ECYKINM4ATVJU5YLIBXP5CEMXU3G/

[Python-Dev] 1 PR, multiple issues

[Jeffrey Kintscher]

When I create a PR that resolves multiple issues, should I add it to 
each issue, or just to one issue and add relevant comments to the others?


//Jeff

--
-
From there to here, from here to there,
funny things are everywhere.
   -- Theodore Geisel

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/74FS2DW2L4MDJXRREAABXYRWNI6INMBA/

[Python-Dev] Re: Enum and annotations

[Steven D'Aprano]

On Mon, Jun 10, 2019 at 02:29:56PM -0700, Ethan Furman wrote:
> Greetings!
> 
> I saw my first annotation mix-up with regards to Enum today:
[...]

I don't think this is particular to Enums. I think this is going to be a 
stumbling block for any class.

py> class X:
... a: 1
... b: 2
...
py> X.__annotations__
{'a': 1, 'b': 2}
py> X.a
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: type object 'X' has no attribute 'a'

Even after 20 years, I occassionally mix up a:1 and a=1 syntax in dicts. 
It doesn't matter there, because its a syntax error. If I wrote more 
class attributes, I'd probably make this error occassionally too.

I think this is going to especially bite beginners, who tend to write 
more class attributes. (Especially those coming from other languages, 
where instance attributes are declared at class level.)


-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B43WEQFL5HVRT6KS5TKSNFEKUWYX4DEB/

[Python-Dev] Re: Enum and annotations

[Steve Dower]

On 10Jun2019 1429, Ethan Furman wrote:

I saw my first annotation mix-up with regards to Enum today:

   https://stackoverflow.com/q/56532591/208880

     #enum import:
     from enum import Enum

     # enum definition:
     class Status(Enum):
     on: 1
     off: 2


For those who don't follow links, the end result is that you get an 
AttributeError the first time you do "Status.on" or "Status.off".



My question for the group:

Is this worth "fixing" or should I leave it alone?  On the one hand it 
seems like an obvious error, but on the other I hate getting in the way 
of needs I haven't thought of.


I'd expect people coming from other languages to get it wrong this way too:

class Status(Enum):
on
off

Which will of course raise NameError and be just as opaque to the naive 
user as the AttributeError later on. I'm not sure we can do as much to 
help this case, but perhaps we can update __getattr__ to check 
__annotations__ on the class before failing and provide a clearer 
message? e.g. "AttributeError: 'on' was specified without assigning a 
value, did you use ':' instead of '='?" Or we could do this on 
construction, but that may rule out some interesting uses in the future 
if you have a need to delay specifying enum values.


I'm not particularly a fan of how it currently is.

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VV77CSOYTO2BQKT5E2PSDDWYUAB4EPPX/

[Python-Dev] Enum and annotations

[Ethan Furman]

Greetings!

I saw my first annotation mix-up with regards to Enum today:

  https://stackoverflow.com/q/56532591/208880

#enum import:
from enum import Enum

# enum definition:
class Status(Enum):
on: 1
off: 2

My question for the group:

Is this worth "fixing" or should I leave it alone?  On the one hand it seems 
like an obvious error, but on the other I hate getting in the way of needs I haven't 
thought of.

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KFGYWLRPBSTST2K4QW5HRULF6EM6Q5F6/

[Python-Dev] Re: obmalloc (was Have a big machine and spare time? Here's a possible Python bug.)

[Tim Peters]

[Neil Schemenauer ]
> I've done a little testing the pool overhead.  I have an application
> that uses many small dicts as holders of data.  The function:
>
> sys._debugmallocstats()
>
> is useful to get stats for the obmalloc pools.  Total data allocated
> by obmalloc is 262 MB.  At the 4*PAGE_SIZE pool size, the wasted
> space due to partly filled pools is only 0.18%.  For 16*PAGE_SIZE
> pools, 0.71%.
>
> I have a set of stats for another program.  In that case, total
> memory allocated by obmalloc is 14 MB.  For 4*PAGE_SIZE pools,
> wasted space is 0.78% of total.  At 16*PAGE_SIZE, it is 2.4%.

Definitely a tradeoff here:  increasing the number of pages per pool
is a pure win for objects of very popular size classes, but a pure
loss for objects of unpopular size classes.  New comments about
POOL_SIZE in the patch briefly hint at that.

> Based on that small set of data, using 4*PAGE_SIZE seems
> conservative.

Which is a nice way of saying "pulled out of Tim's ass" ;-)

> As I'm sure you realize, making pools bigger will
> waste actual memory, not just virtual address space because you
> write the arena pointer to each OS page.

Yes, but mostly no!  It remains the case that obmalloc neither writes
nor reads anything in an arena until a malloc/realloc call actually
needs to return a pointer to a never-before accessed page.  Writing
the arena index is NOT done to pages when the arena is allocated, or
even when a new pool is carved off an arena, but lazily, one page at a
time, lowest address to highest, as fresh pages actually need to be
returned to the caller.

So arena pointers aren't actually written more frequently than when
using pools of just one page, as is done now (which _still_ writes the
arena index into each page).

Subtlety:  currently, if you pass a nonsense address to
address_in_range that happens to be in one of the arena's pools,
address_in_range() says "yes".  However, after the patch, it may well
return "no" if the address is in one of the pool's pages that hasn't
yet been returned to a malloc/realloc caller (in which case the arena
index hasn't yet been stored at the start of the page).

I don't care about that, because it's 100% a logic error to pass an
address to free/realloc that wasn't obtained from a previous
malloc/realloc call.  So it's a change in undefined behavior.

> I want to do performance profiling using Linux perf.  That should
> show where the hotspot instructions in the obmalloc code.  Maybe
> that will be useful to you.

Would be good to know, yes!  But may also depend on the app.

> Another thought about address_in_range(): some operating systems
> allow you to allocate memory a specific alignments.  Or, you can
> even allocate a chunk of memory at a fixed memory location if you do
> the correct magic incantation.  I noticed that Go does that.  I
> imagine doing that has a bunch of associated challenges with it.
> However, if we could control the alignment and memory location of
> obmalloc arenas, we would not have the segv problem of
> address_in_range().  It's probably not worth going down that path
> due to the problems involved.

I'm unclear on how that could be exploited.  It's not addresses that
come from arenas that create segv headaches, it's addresses that come
from the system's malloc/realloc.

If we were, e.g., to pick a maximum amount of address space obmalloc
can use in advance, we could live with a single arena of that size,
and just check whether an address is within it.  All the problems come
from that address space may alternate, any number of times, between
addresses in obmalloc arenas and addresses from the system malloc's
internals.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YUWFWPKN67EOGF5F7QFGB7IFRH5K2PFW/

[Python-Dev] Re: obmalloc (was Have a big machine and spare time? Here's a possible Python bug.)

[Neil Schemenauer]

On 2019-06-09, Tim Peters wrote:
> And now there's a PR that removes obmalloc's limit on pool sizes, and,
> for a start, quadruples pool (and arena!) sizes on 64-bit boxes:

Neat.

> As the PR says,
> 
> """
> It would be great to get feedback from 64-bit apps that do massive
> amounts of small-object allocations and deallocations.
> """

I've done a little testing the pool overhead.  I have an application
that uses many small dicts as holders of data.  The function:

sys._debugmallocstats()

is useful to get stats for the obmalloc pools.  Total data allocated
by obmalloc is 262 MB.  At the 4*PAGE_SIZE pool size, the wasted
space due to partly filled pools is only 0.18%.  For 16*PAGE_SIZE
pools, 0.71%.

I have a set of stats for another program.  In that case, total
memory allocated by obmalloc is 14 MB.  For 4*PAGE_SIZE pools,
wasted space is 0.78% of total.  At 16*PAGE_SIZE, it is 2.4%.

Based on that small set of data, using 4*PAGE_SIZE seems
conservative.  As I'm sure you realize, making pools bigger will
waste actual memory, not just virtual address space because you
write the arena pointer to each OS page.

I want to do performance profiling using Linux perf.  That should
show where the hotspot instructions in the obmalloc code.  Maybe
that will be useful to you.

Another thought about address_in_range(): some operating systems
allow you to allocate memory a specific alignments.  Or, you can
even allocate a chunk of memory at a fixed memory location if you do
the correct magic incantation.  I noticed that Go does that.  I
imagine doing that has a bunch of associated challenges with it.
However, if we could control the alignment and memory location of
obmalloc arenas, we would not have the segv problem of
address_in_range().  It's probably not worth going down that path
due to the problems involved.

Regards,

  Neil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LJWXNQNZRVCCF36ALXKZVWYHINXTVMVB/

[Python-Dev] Re: Expected stability of PyCode_New() and types.CodeType() signatures

[Brett Cannon]

Neil Schemenauer wrote:
> On 2019-05-31, Simon Cross wrote:
> >   As the maintainer of Genshi, one the libraries
> > affected by the CodeType and
> >  similar changes, I thought I could add a users perspective to the
> >  discussion: [...]
> 
> Thanks.  I think this change to PyCode_New() could have been handled
> a bit better.  Couldn't we introduce a new CPP define that enables
> the revised API of PyCode_New()?  For 3.8 extensions, they would get
> the backwards compatible API (and a warning) unless they set the
> define. For 3.9, we would enable the new API by default.  That gives
> 3rd party extensions one release cycle to catch up to the change.
> Perhaps something similar could be done for CodeType called from
> within Python code.
> 
> In this case, it seems that introducing a new API like
> PyCode_NewEx() is not the way.  However, just changing an API like
> that is not very friendly to 3rd party extensions, even if we don't
> claim it is a stable API.  For me, change to PyCode_New() means that
> I can't test the 3.8.0b1 because the 3rd party extensions I rely on
> don't compile with it. Normally I try to test my application code
> with the latest alpha and beta releases.
> 
> It would be great if we had a system that did CI testing with the
> top PyPI modules.  E.g. pull the latest versions of the top 100 PyPI
> modules and test them with the latest CPython branch.  With that, at
> least we would know what the fallout would be from some incompatible
> CPython change.  Setting that system up would be a fair amounnt of
> work but I suspect the PSF could fund someone who puts together a
> plan to do it.  Such a system would be even more useful if we start
> moving stuff out of stdlib into PyPI.

This has been brought up multiple times before and no one every objects, it's 
just a time/effort problem nearly every time. For the PSF to fund it someone 
would have to write up a grant proposal and submit it.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DKJMV5ZTCMKYIYTCE6RGEA7YBMIE63MY/