Re: [Python-Dev] C99
On Thu, Oct 06, 2016 at 10:10:54PM -0700, Nathaniel Smith wrote: > The reason this matters is that the official vendor compiler on RHEL 5 > is gcc 4.1, but there's also a separately-distributed version of gcc > 4.8.2 that can target it. Where can I get that 4.8 for RHEL 5? I'm using Centos 5, which ought to be the same as RHEL 5, and the standard gcc is 4.1, with 4.4 available through yum. If 4.8 is available anywhere, I haven't been able to find it. And as far as I can see, 3.6 won't build under either 4.1 or 4.4 on Centos 5. > If a packager trying to build manylinux > wheels wants a more modern gcc, then it's reasonable to ask them to > get this patched gcc. But for an end-user who's just trying to build > CPython on their machine, you might or might not consider this an > acceptable request -- maybe CPython wants to work on default vendor > compiler to imposing that on users. > > And in practice this almost certainly doesn't matter -- the only > reason people jump through hoops to get gcc 4.8 is for its improved > C++ support. I just tried my c99 test file on CentOS 5's default gcc > 4.1 and it was fine. Can you try building Python 3.6? Because it fails for me, and the discussion here: http://bugs.python.org/issue28092 concluded that 4.1 is not supported and I'm right out of luck until I can upgrade. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optimizing list.sort() by checking type in advance
On Mon, Oct 10, 2016 at 09:16:32PM +, Elliot Gorokhovsky wrote: > Anyway, benchmarking technique aside, the point is that it it works well > for small lists (i.e. doesn't affect performance). You've been shown that there is something suspicious about your benchmarking technique, something that suggests that the timing results aren't trustworthy. Until you convince us that your timing results are reliable and trustworthy, you shouldn't be drawing *any* conclusions about your fastsort versus the standard sort. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] itertools predicates
On Wed, Nov 02, 2016 at 07:03:27PM -0300, Francisco Couzo wrote: > Some functions that take a predicate from itertools accept None as its > predicate: [...] > I'd be interested in writing a patch to make itertools more consistent if > there's a consensus. There's a line in PEP 8 about foolish consistency. Consistency just for the sake of consistency is not a virtue. I have no objection to adding support for None to individual itertools functions on the basis that it makes sense for that function. If you can suggest a reasonable and *useful* default predicate or key function to use for None, then sure, add it to the tracker as a feature enhancement. (And, of course, a patch would be appreciated.) But that will be because it is a useful enhancement, not because of "consistency". > I see two possible solutions: Before there can be a solution, there first has to be a problem that needs solving. "Lack of consistency" is not necessarily a problem. The intertools functions are quite different, they do different things with different APIs. The question here should not be "why don't these functions take None as an argument?", rather it should be "why should these functions take None as an argument?". -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Someons's put a "Python 2.8" on GitHub
On Fri, Dec 09, 2016 at 09:56:44PM -0800, Larry Hastings wrote: > > "Python 2.8 is a backwards-compatible Python interpreter with new > features from Python 3.x. It was produced by forking Python 2.7.12 and > backporting [...] > https://github.com/naftaliharris/python2.8 I seem to recall that when we discussed the future of Python 2.x, and the decision that 2.7 would be the final version and there would be no 2.8, we reached a consensus that if anyone did backport Python 3 features to a Python 2 fork, they should not call it Python 2.8 as that could mislead people into thinking it was officially supported. I think the project should be renamed to make it clear that its a fork, like Stackless. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Someons's put a "Python 2.8" on GitHub
On Mon, Dec 12, 2016 at 03:10:09AM -0600, Wes Turner wrote: > [Continuing to play devil's advocate for the sake of clarification] Clarification of *what* exactly? You don't seem to be asking any questions, just making statements. If you have a concrete, specific question, please ask it. If its a general question, you can ask it too, but don't be surprised if the answer is "it depends on the specific circumstances". -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory)
I haven't seen any response to the following alleged security vulnerability. I am not qualified to judge the merits of this, but it does seem worrying that (alledgedly) the Python security team hasn't responded for over 12 months. Is anyone able to comment? Thanks, Steve On Mon, Feb 20, 2017 at 09:01:21PM +, nos...@curso.re wrote: > Hello, > > I have just noticed that an FTP injection advisory has been made public > on the oss-security list. > > The author says that he an exploit exists but it won't be published > until the code is patched > > You may be already aware, but it would be good to understand what is the > position of the core developers about this. > > The advisory is linked below (with some excerpts in this message): > > http://blog.blindspotsecurity.com/2017/02/advisory-javapython-ftp-injections.html > >Protocol injection flaws like this have been an area of research of mine >for the past few couple of years and as it turns out, this FTP protocol >injection allows one to fool a victim's firewall into allowing TCP >connections from the Internet to the vulnerable host's system on any >"high" port (1024-65535). A nearly identical vulnerability exists in >Python's urllib2 and urllib libraries. In the case of Java, this attack >can be carried out against desktop users even if those desktop users do >not have the Java browser plugin enabled. >As of 2017-02-20, the vulnerabilities discussed here have not been patched >by the associated vendors, despite advance warning and ample time to do >so. >[...] >Python's built-in URL fetching library (urllib2 in Python 2 and urllib in >Python 3) is vulnerable to a nearly identical protocol stream injection, >but this injection appears to be limited to attacks via directory names >specified in the URL. >[...] >The Python security team was notified in January 2016. Information >provided included an outline of the possibility of FTP/firewall attacks. >Despite repeated follow-ups, there has been no apparent action on their >part. > > Best regards, > > -- Stefano > > P.S. > I am posting from gmane, I hope that this is OK. > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40pearwood.info > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Translated Python documentation
On Fri, Feb 24, 2017 at 06:01:59AM -0500, tritium-l...@sdamon.com wrote: > My gut splits the difference on this issue; I suggest an approach to > meet in the middle – a version of the docs written in simplified > English (Not quite Up Goer Five simplified, but simplified.) As an English speaker, my gut tells me that it would be much harder to write *accurate* simplified English technical documentation than to translate it into another language. You have all the difficulties of translation, plus you're working under a handicap of only using some (ill-defined?) subset of English. Wikipedia offers some evidence supporting my view: - the main English Wikipedia has 5 million articles, written by nearly 140K active users; - the Swedish Wikipedia is almost as big, 3M articles from only 3K active users; - but the Simple English Wikipedia has just 123K articles and 871 active users. That's fewer articles than Esperanto! https://meta.wikimedia.org/wiki/List_of_Wikipedias Nevertheless, I certainly wouldn't object if people wanted to try writing Simple English translations of the docs. But I don't think they would be as useful as translations into non-English. [...] > For any language you want to support other than English, you need a > translator who is A: a python expert, B: fluent in English, and C: > fluent in the target language. I disagree. You need a translator who is A: skilled at technical documentation, with B and C as above. They don't need to be a Python expert. We have plenty of Python experts that they can consult with and ask questions. But they need to know the right questions to ask: "Python attributes, they're kind of like C members, right? I would translate 'member' into Klingon as 'gham', which means 'arm or leg', so I can use the same word for attribute." > …And then you need another one to > check what was written. These are practical problems. There are > extant services to support this, they are expensive in either money or > time, and the docs produced usually lag behind English quite a bit. Is this a good use for some PSF funding? Would companies be willing to invest money in translating Python documentation? Just because we're Open Source, doesn't mean that everything we do has to be purely volunteer. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can I revoke PEP 103 (info about git)?
On Wed, Mar 08, 2017 at 04:30:41PM +0100, Oleg Broytman wrote: > On Wed, Mar 08, 2017 at 09:50:06AM -0500, Barry Warsaw > wrote: > > It's also okay to remove much of the content and just leave a placeholder. > > The historical record would of course always be available in the vcs. > >Thanks! That's what I've planned to do in case we don't remove PEPs. Why remove the content? In fact, since its just an informational PEP, why withdraw it? Some people find it too generic and not enough about Python -- okay. So what? Is PEP 103 actively harmful? -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Application support
Hello Ryan, Welcome! My response is below. On Sun, Mar 12, 2017 at 10:49:13AM -0700, Ryan James Kenneth Murray wrote: > To whom it may concern, > > I was about to use Markdown to verify or indicate any changes to the files > when I was directed to c python and pep, however I have reached my > abilities in understanding or contributing. I will have to broden my > knowledge and insight into python. That is why I am asking for help in > making sure that everything is satisfactory. As you can see I am concerned > about entering anymore data I'm afraid I cannot make head or tail of what you are talking about here. To be perfectly honest, your post sounds like something generated by a quite clever bot using a Markov chain to generate random text. It *almost* is meaningful, but not quite: there's a bunch of tech buzzwords (Markdown, C, Python, PEP) in some sentences which are grammatically correct but don't seem to mean anything. What files are you changing, what data are you talking about, and what is "everything" that needs to be satisfactory? How is this relevant to developing Python? If you are a bot, then you aren't welcome and somebody will soon remove you from the mailing list. (And I will feel silly for talking to you as if you were a person.) But in case you actually are a human being, I'm giving you the benefit of the doubt and allowing you the opportunity to say something that proves you are a human. Thank you. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue with _thread.interrupt_main (29926)
On Mon, Mar 27, 2017 at 10:33:44PM -0400, Terry Reedy wrote: > https://bugs.python.org/issue29926 was opened as an IDLE issue, which > means that most watching the new issues list would ignore it. But I > think it is an issue with _thread.interrupt_main (which IDLE calls in > respond to ^C) not interrupting time.sleep(n) in main thread*. I tested > on Windows, don't know yet about OP. Since there is no Expert's Index > listing for _thread (or threading), I am asking here for someone who > knows anything to take a look. > > * > >>> time.sleep(10) > > > <... remainder of 10 seconds pass> > KeyboardInterrupt I get similar behaviour under Linux. I don't have the debug print, but the KeyboardInterrupt doesn't interrupt the sleep until the 10 seconds are up. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sorting
On Thu, Apr 06, 2017 at 02:30:06AM +0200, Oleg Broytman wrote: >Sorry, let me pour a bit of cold water here. Recently I was busy > porting a few big and small (but complex) programs to Python 3 and was > sending thousands curses every day. str=>unicode is the biggest change > but not the most painful. The worst was when I spent few days hunting > for a subtle bug caused by absent of unbound methods. >Painful. :-( I'm curious about this. What do you mean? Python 3 has unbound methods, they're just the original, unwrapped function: py> class K: ... def method(self, arg): ... pass ... py> K.method The only(?) functional difference between unbound methods in Python 2 versus 3 is that in Python 2 they automatically do a type-check that `self` is an instance of the class. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mention aenum in the Enum docs?
On Tue, May 09, 2017 at 10:05:43AM -0700, Ethan Furman wrote: > A comment on a recent SO answer [1] wondered why my aenum library wasn't > mentioned in the docs to help guide people that needed/wanted more advanced > Enum options to it. I know that the std lib is where good modules go to die :-) Is the aenum module feature complete? Is it ready to ~~die~~ move to the std lib? If aenum is not going through active development, perhaps the missing features should be ported to enum and then there's no need to recommend a second module. > Pros: > - drop-in replacement for the stdlib Enum > - has many advanced features such as > - auto __init__ building > - multi-value members > - duplicate value but non-aliasing members > - etc. Those three advanced features sound useful. Is there a reason enum doesn't support them? -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 484 update proposal: annotating decorated declarations
On Tue, May 09, 2017 at 11:54:26PM +0400, Abdur-Rahmaan Janhangeer wrote: > I'm really new to the mailing list. Can someone just summarise the > preceding message in 5 to 10 lines like what it is, what type is it or when > does it happen It is an update to PEP 484, adding support for type-checking decorators. https://www.python.org/dev/peps/pep-0484/ I don't understand what you mean by "what type is it". When does it happen: if there are no strong or conclusive objections (and I don't think there have been) it may have already happened by now. If not, then soon. Depends on Guido's schedule. As for when it will actually be visible in Python, the original post already answered that: Guido hopes to add it to Python 3.6.2 and 3.5.4. I don't think this will have much visible impact unless you're using MyPy for type checking. By the way, the original post was less than fifty lines. That already is a summary. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Format strings, Unicode, and Py2.7: need clarification
On Wed, May 17, 2017 at 02:41:29PM -0700, Craig Rodrigues wrote:
> e = "{}".format(u"hi")
[...]
> type(e) == str
> The confusion for me is why is type(e) of type str, and not unicode?
I think that's one of the reasons why the Python 2.7 string model is (1)
convenient to those using purely ASCII, but (2) ultimately broken.
You can see why it's broken if you do this:
py> "{}".format(u"hiµ")
Traceback (most recent call last):
File "", line 1, in
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb5' in
position 2: ordinal not in range(128)
So it tries to encode the Unicode string to ASCII, and if that succeeds,
format returns a byte str. I'm not sure if that was a deliberate design
choice for format, or just a side-effect of it calling str() on its
arguments by default.
I'm not sure if I've answered your question or not. Are you looking for
justification of this misfeature, or an explanation of the historical
reasons why it exists, or something else?
(If you're looking for the same behaviour in Python 3 and 2.7, probably
the best thing you can do is just religiously use unicode strings u'' in
both. You might try:
from __future__ import unicode_literals
in 2.7, but I'm not sure that's enough.)
--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Micro-optimisations can speed up CPython"
On Mon, May 29, 2017 at 07:27:05PM -0700, Guido van Rossum wrote:
> > https://www.corsix.org/content/why-are-slots-so-slow
> >
>
> Is the author of that article using non-standard terminology? The article
> doesn't appear to be about __slots__ at all.
Quoting Armin Ronacher:
By far my biggest problem with the language is the stupid slot
system. I do not mean the __slots__ but the internal type slots
for special methods.
http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see/
Armin shows the history of these "slots" (or however they're called)
back to Python in 1990!
static number_methods int_as_number = {
intadd, /*tp_add*/
intsub, /*tp_subtract*/
intmul, /*tp_multiply*/
... etc
I don't know where the name "slot" comes for from the various tp_*
members (fields? attributes? slots?) but I'm pretty sure I've heard it
before. I don't normally pay attention to what happens in the C
implementation, but I'm fairly sure he's referring to these tp_*
thingies.
Oh yes, here you go:
https://docs.python.org/2/extending/newtypes.html#type-methods
refers to "tp_* slot" regularly. What does "tp" stand for? Type
something, I guess.
--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Language proposal: variable assignment in functional context
Welcome Robert. My response below. Follow-ups to Python-Ideas, thanks. You'll need to subscribe to see any further discussion. On Fri, Jun 16, 2017 at 11:32:19AM +, Robert Vanden Eynde wrote: > In a nutshell, I would like to be able to write: > y = (b+2 for b = a + 1) I think this is somewhat similar to a suggestion of Nick Coghlan's. One possible syntax as a statement might be: y = b + 2 given: b = a + 1 https://www.python.org/dev/peps/pep-3150/ In mathematics, I might write: y = b + 2 where b = a + 1 although of course I wouldn't do so for anything so simple. Here's a better example, the quadratic formula: -b ± √Δ x = ─ 2a where Δ = b² - 4ac although even there I'd usually write Δ in place. > Python already have the "functional if", lambdas, list comprehension, > but not simple assignment functional style. I think you mean "if *expression*" rather than "functional if". The term "functional" in programming usually refers to a particular paradigm: https://en.wikipedia.org/wiki/Functional_programming -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Impact of Namedtuple on startup time
On Mon, Jul 17, 2017 at 02:43:19PM +0200, Antoine Pitrou wrote:
>
> Hello,
>
> Cost of creating a namedtuple has been identified as a contributor to
> Python startup time. Not only Python core and the stdlib, but any
> third-party library creating namedtuple classes (there are many of
> them). An issue was created for this:
> https://bugs.python.org/issue28638
Some time ago, I needed to backport a version of namedtuple to Python
2.4, so I started with Raymond's recipe on Activestate and modified it
to only exec the code needed for __new__. The rest of the class is an
ordinary inner class:
# a short sketch
def namedtuple(...):
class Inner(tuple):
...
exec(source, ns)
Inner.__new__ = ns['__new__']
return Inner
Here's my fork of Raymond's recipe:
https://code.activestate.com/recipes/578918-yet-another-namedtuple/
Out of curiosity, I took that recipe, updated it to work in Python 3,
and compared it to the std lib version. Here are some representative
timings:
[steve@ando ~]$ python3.5 -m timeit -s "from collections import
namedtuple" "K = namedtuple('K', 'a b c')"
1000 loops, best of 3: 1.02 msec per loop
[steve@ando ~]$ python3.5 -m timeit -s "from nt3 import namedtuple" "K =
namedtuple('K', 'a b c')"
1000 loops, best of 3: 255 usec per loop
I think that proves that this approach is viable and can lead to a big
speed up.
I don't think that merely dropping the _source attribute will save much
time. It might save a bit of memory, but in my experiements dropping it
only saves about 10µs more. I think the real bottleneck is the cost of
exec'ing the entire class.
--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Impact of Namedtuple on startup time
On Mon, Jul 17, 2017 at 09:31:20PM +, Brett Cannon wrote: > As for removing exec() as a goal, I'll back up Christian's point and the > one Steve made at the language summit that removing the use of exec() from > the critical path in Python is a laudable goal from a security perspective. I'm sorry, I don't understand this point. What do you mean by "critical path"? Is the intention to remove exec from builtins? From the entire language? If not, how does its use in namedtuple introduce a security problem? -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Impact of Namedtuple on startup time
On Tue, Jul 18, 2017 at 01:17:24AM +0200, Giampaolo Rodola' wrote:
> The extra memory overhead is a price I would be happy to pay considering
> that collections.namedtuple is considerably slower than a plain tuple.
> Other than the additional overhead on startup / import time, instantiation
> is 4.5x slower than a plain tuple:
>
> $ python3.7 -m timeit -s "from collections import namedtuple; nt =
> namedtuple('xxx', ('x', 'y'))" "nt(1, 2)"
> 100 loops, best of 5: 313 nsec per loop
>
> $ python3.7 -m timeit "tuple((1, 2))"
> 500 loops, best of 5: 68.4 nsec per loop
I don't think that is a fair comparision. As far as I can tell, that
gets compiled to a name lookup for "tuple" which then returns its
argument unchanged, the tuple itself being constant-folded at compile
time.
py> dis.dis("tuple((1, 2))")
1 0 LOAD_NAME0 (tuple)
3 LOAD_CONST 2 ((1, 2))
6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
9 RETURN_VALUE
--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup time
On Wed, Jul 19, 2017 at 04:11:24PM -0700, Chris Barker wrote: > As long as we are talking anecdotes: > > If it could save a person’s life, could you find a way to save ten seconds > off the boot time? If there were five million people using the Mac, and it > took ten seconds extra to turn it on every day, that added up to three > hundred million or so hours per year people would save, which was the > equivalent of at least one hundred lifetimes saved per year. > > Steve Jobs. And about a fifth of the time they spent standing in lines waiting to buy the latest unnecessary iGadget... But seriously, that calculation is completely bogus. Not only is Steve Job's arithmetic *completely* wrong, but the whole premise is nonsense. Do the maths yourself: ten seconds per day is 3650 seconds in a year, which is slightly over an hour (3600 seconds). Multiply by five million users, that's about five million hours, not 300 million. So Jobs exaggerates the time saved by a factor of sixty. (Or maybe Jobs was warning that Macs crash sixty times a day...) But the premise is wrong too. Those hypothetical people don't turn their Macs on in sequence, each person turning their computer on only after the previous person's Mac had finished booting. They effectively boot them up in parallel but offset, spread out over a 24 hour period, so about 3472 people booting up at the same time each minute of the day. Time savings for parallel processes don't add in the way Jobs adds them, if we treat this as 1440 parallel processes (one per minute of the day) we save 1440 hours a year. But really, the only meaningful calculation is the each person saves 10 seconds per day. We can't even meaningfully say they save one hour a year: it doesn't come nicely packaged up for you all at once, so you can actually do something useful with it, nor can you save those ten seconds from one day to the next. You only get one shot at using them. What can you do with ten seconds per day? By the time you decide what to do with the extra time, it's already gone. There are good reasons for speeding up boot time, but this sort of calculation is not one of them. I think it is in particularly bad taste to exaggerate the significance of it by putting it in terms of saving lives. You want to save real lives? How about fixing the conditions in the sweatshops that make Apple phones? And installing suicide nets around the building doesn't count. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] for...else
Hello Kiuhnm, and welcome.
On Mon, Jul 24, 2017 at 05:35:03PM +0200, Kiuhnm via Python-Dev wrote:
> Hello,
>
> I think that the expression "for...else" or "while...else" is completely
> counter-intuitive.
You may be right -- this has been discussed many, many times before. In
my personal opinion, the best (and only accurate!) phrase would have
been:
for item in sequence:
# block
then:
# block
If you look at the byte-code generated by a for...else statement, you
see that the "else" block is unconditionally executed after the for loop
completes, unless something causes a jump outside of the entire
statement: return, break, or raise. So it is more like:
- run the loop;
- *then* run the following block
rather than:
- run the loop;
- otherwise ("else") run the following block.
Others disagree and would prefer other keywords. But regardless,
backwards compatibility means that we must keep "for...else", so I'm
afraid that discussing alternatives is *almost certainly* a waste of
time.
> Wouldn't it be possible to make it clearer? Maybe
> something like
At this point, no, it is not practical to change the syntax used. Maybe
when Python 3.0 was first introduced, but that ship has long sailed. It
is very, very unlikely that the syntax for this will ever change, but if
it does, it probably won't be until something in the distant future like
Python 5.
But not Python 4: Guido has already ruled that Python 4 will not include
major backwards-incompatible changes. Going from 3 to 4 will not be as
disruptive as going from 2 to 3.
So depending on how you look at it: discussing alternative syntax to
for...else is either ten years too late or ten years too early.
--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 551: Security transparency in the Python runtime
Very nicely written. A few comments below. On Mon, Aug 28, 2017 at 04:55:19PM -0700, Steve Dower wrote: [...] > This PEP describes additions to the Python API and specific behaviors > for the > CPython implementation that make actions taken by the Python runtime > visible to > security and auditing tools. The goals in order of increasing importance [...] Check your line lengths, I think they may be too long? (Or maybe my mail client is set too short?) [...] > To summarize, defenders have a need to audit specific uses of Python in > order to > detect abnormal or malicious usage. Currently, the Python runtime does not > provide any ability to do this, which (anecdotally) has led to organizations > switching to other languages. It would help if the PEP addressed the state of the art in other languages. [...] > For example, ``sys.addaudithook()`` and ``sys.audit()`` should exist but > may do > nothing. This allows code to make calls to ``sys.audit()`` without having to > test for existence, but it should not assume that its call will have any > effect. > (Including existence tests in security-critical code allows another > vector to > bypass auditing, so it is preferable that the function always exist.) That suggests a timing attack to infer the existence of auditing. A naive attempt: from time import time f = lambda: None t = time() f() time_to_do_nothing = time() - t audit = sys.audit t = time() audit() time_to_do_audit = time() - t if time_to_do_audit <= time_to_do_nothing: do_something_bad() This is probably too naive to work in real code, but the point is that the attacker may be able to exploit timing differences in sys.audit and related functions to infer whether or not auditing is enabled. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553: Built-in debug()
On Tue, Sep 05, 2017 at 06:14:12PM -0700, Barry Warsaw wrote: > I’ve written a PEP proposing the addition of a new built-in function > called debug(). Adding this to your code would invoke a debugger > through the hook function sys.debughook(). [...] > P.S. This came to me in a nightmare on Sunday night, and the more I > explored the idea the more it frightened me. I know exactly what I > was dreaming about and the only way to make it all go away was to > write this thing up. Sorry, are we to interpret this as you asking that the PEP be rejected? I can't tell whether you are being poetic and actually think the PEP is a good idea, or whether you have written it to have it rejected and prevent anyone else ever implementing this? -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?
On Sun, Oct 10, 2021 at 09:15:30AM +, Patrick Reader wrote: > - a LOAD_FAST cannot possibly have any side-effects outside the > interpreter stack [1] > [1]: global variables are probably supposed to have the same > guarantee, but in practice this is not the case I don't think that lookups are guaranteed to be side-effect free, since raising an exception is a side-effect. def func(): x can raise NameError if there is no global or builtin "x". So that cannot be optimized away. Similarly for locals: def func(x): if random.random() > 0.5: del x x Will that raise or not? No way of telling without actually running the code. Now I guess a sufficiently smart optimizer might be able to optimize away the lookup of a local variable if it cannot possibly be undefined, as in your initial example, but that brings us to how much benefit is gained for how much developer effort and code complexity. If the benefit is small, or negligible, then the developer effort had better be even smaller. In this case, the cost of the lookup is small (it is a *FAST* lookup). And furthermore, the likelihood that anyone will actually write that code is even smaller. (There are times that we might test for the existence of a global, by looking up the name. There might even be times that we would do the same for a local. We might even, under some rare circumstances, do so without guarding it in a try...except block. But why on earth would we test for the existence of a local that cannot possibly fail to exist, which is the only case that is safe to optimize away?) So the benefit is correspondingly tiny, and optimizing it away had better be effectively free in both developer effort and ongoing code complexity to make it worthwhile. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HWSHZLW64Q26EIUPX4S3TME5QXYBRLPY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: What is __int__ still useful for?
On Thu, Oct 14, 2021 at 11:52:11AM +1300, Greg Ewing wrote: > Scratch that, it seems __trunc__ also returns an int, at least > for floats. Not sure what the logic behind that is. I'm not sure about the logic either, but it is documented as returning an Integral: "Truncates the Real x to the nearest Integral toward 0." so the option is there for third-party types to return some integral type apart from int. For the stdlib, the only Integral type we have is int. So I think we have the following intended behaviour. * Round a numeric (Real) value to an Integral value: - round to nearest (ties to even): __round__ - round down (towards negative infinity): __floor__ - round up (towards positive infinity): __ceil__ - round towards zero: __trunc__ * Convert a numeric Integral value to an actual int: (intended for indexing of sequences): __index__ * Convert any arbitrary value to an actual int: __int__ Does that seem right? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TFKQPKGUU3G2HI5AVIFFKSULU3YCLLRY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11
Hello Doug, On Thu, Oct 14, 2021 at 03:45:07PM -, Doug Swarin wrote: > I believe strong and valid arguments can be made about the use of None > being a fundamental flaw in some types of coding Can you elaborate on that? Obviously it is not always appropriate to use None, but I've never seen it called a *fundamental* flaw. I know that the null pointer has been called a billion-dollar mistake, but Python's None is not a null pointer. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IGA7GYE2M6KQVOGIP3X2EESOXYUOMXMC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 except* formatting
Rob Cliffe is having problems posting to the Python-Dev list, so he posted an alternative suggestion to the Python-Ideas list: https://mail.python.org/archives/list/python-id...@python.org/message/6KQUQBKFGJSGDNXFZBSM5OXD2ISLIQTT/ Rob's idea is to use "except for ..." with exception groups, instead of a new keyword or symbol. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/62ZX4T226BCRQE7ZNJPSJNNBAP735K2J/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11
On Tue, Oct 19, 2021 at 05:09:42PM -0700, Michael Selik wrote: > None and its ilk often conflate too many qualities. For example, is it > missing because it doesn't exist, it never existed, or because we never > received a value, despite knowing it must exist? Does it matter if different functions have different semantic interpretations for None? > The languages SAS and R > support at least 27 varieties of NA, allowing un-tagged, and tagged with > the letters A-Z to help someone create distinctions between different kinds > of nothingness. IEEE-754 allows about 16 million possible NaNs, which I > believe was intended to allow floating point instructions to pass error > messages along. Yes, and after something like 30-40 years of IEEE-754 supporting NAN payloads, the number of systems that actually use them can probably be counted on the fingers of one hand :-( Ironically, one of those systems is R, which -- so I have been lead to believe -- uses distict NANs to represent those 27 tagged NA values. Back in the 1980s, one of the earliest systems which supported IEEE-754 maths was the Apple Numeric Toolkit. Apple's maths routines generated NANs with documented payloads for certain errors, e.g: * NAN(1) invalid sqrt * NAN(2) invalid addition such as INF + -INF * NAN(34) invalid argument to inverse trig functions In a complex computation, it was sometimes useful to see why a NAN was generated. Alas, when Apple moved their maths routines into hardware, the MC68881 coprocessor always generated NANs with payload 255, and that useful debugging information was lost. 30+ years later, and we cannot easily, reliably or portably use NAN payloads. Most people don't care. If we offerred them a dozen or a thousand distinct sentinels for all the various kinds of missing data, how many people would use them and how many would just stick to plain old None? > If the motivation for this operator is chained lookups, how about adding a > feature to the operator module, first? It seems natural to add a > keyword-only argument to `attrgetter`, and it's a lighter touch than > implementing a new operator. If use becomes widespread, that gives more > weight to PEP 505. I agree that this is a nice way forward, and a useful function in its own right. The only thing is that I would argue for a different colour of the bikeshed: def getattr_chain(obj, *attrs, default): # like obj.a.b.c.d # if any attribute is missing, # raises if default is not given # otherwise returns default getattr is far more commonly used than attrgetter. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AAHVRF7WXIZRUGBHOVBEB7NEZAYHHL26/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11
On Wed, Oct 20, 2021 at 06:17:59PM +0200, Piotr Waszkiewicz wrote: > Do you think about something along those lines? > ``` > phone = book.publisher.owner.phone except AttributeError: None > ``` This is not equivalent to PEP 505's None-aware operators. The semantics are very different, and it is much less safe. If you misspell an attribute: book.publisher.onwer = Owner(...) then the `except AttributeError` code will silently hide that error and return None. PEP 505 does not do that. If you use the wrong type: book.publisher = "O'Reilly Books" then the `except` version will silently hide the error and return None. PEP 505 does not. Versions of this that rely on catching AttributeError are simply wrong and are an anti-pattern. They catch too much and silently turn errors into silent wrong behaviour. PEP 505 does not fall into that trap. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/B5LUO5K562IM6667YKDW6N57YEUMJKXG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11
On Thu, Oct 21, 2021 at 01:46:27PM +1100, Steven D'Aprano wrote: > On Tue, Oct 19, 2021 at 05:09:42PM -0700, Michael Selik wrote: > > If the motivation for this operator is chained lookups, how about adding a > > feature to the operator module, first? It seems natural to add a > > keyword-only argument to `attrgetter`, and it's a lighter touch than > > implementing a new operator. If use becomes widespread, that gives more > > weight to PEP 505. > > I agree that this is a nice way forward, and a useful function in its > own right. On further thought, I no longer agree. Or at least, I think we need to think a lot harder about the API before adding any sort of chained attribute getter into the stdlib. If we get it wrong, we'll be stuck with it until Python 5000. The problem is that any sort of code equivalent to: try: return obj.chain.of.attribute.lookups except AttributeError: return None risks silently hiding genuine coding errors. This seems to be an anti-pattern, or at least a foot-gun. And it is certainly not equivalent to, or a replacement for, PEP 505. Same applies to variants similar to attrgetter. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IRO7Z2IEV3I7X5EZZRNWCLDMJRCWALGS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11
On Thu, Oct 21, 2021 at 10:49:35AM +0200, Baptiste Carvello wrote:
> > Versions of this that rely on catching AttributeError are simply wrong
> > and are an anti-pattern. They catch too much and silently turn
> > errors into silent wrong behaviour.
> >
> > PEP 505 does not fall into that trap.
>
> This is not true as a general rule: the PEP 505 examples with
> `dict.get()` do catch too much.
The problem there is not the None-aware operators, but the use of
dict.get. That's a good reason to re-think None-aware subscripting.
dict?['key']
will still raise if you mistype the key, while dict.get does not.
Even if we limit ourselves to dict.get:
return obj?.get('spam')?.get('eggs')
doesn't protect against mispellings of the keys, *due to dict.get*, but
it does protect against mispelling "get" (unlikely). More importantly it
also protects against type errors:
obj['spam'] = {'eggs', value} # oops, a set, not a dict
or equivalent. (I don't mean to limit this to just typos.)
Now if we write:
obj?.get('spam')?.get('eggs')
the first attribute lookup will return a set and the second will fail
because sets don't have a get method.
Where as if we use exceptions:
try:
obj['spam]['eggs']
except (TypeError, KeyError):
return None
the error is silently suppressed and we get None when we should get a
TypeError.
--
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/WYEIZ3IC6P6KN4L2MROV4SBCZ5XOQUV6/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Type annotations, PEP 649 and PEP 563
On Thu, Oct 21, 2021 at 04:48:28PM -0400, Larry Hastings wrote: > In Python, if you evaluate an undefined name, Python raises a > NameError. This is so consistent I'm willing to call it a "rule". > Various folks have proposed an exception to this "rule": evaluating an > undefined name in an PEP 649 delayed annotation wouldn't raise > NameError, instead evaluating to some yet-to-be-determined value > (ForwardRef, AnnotationName, etc). I don't think annotations are > special enough to "break the rules" in this way. Can we have a code snippet illustrating that? I think this is what you mean. Please correct me if I have anything wrong. If I have this: from typing import Any def function(arg:Spam) -> Any: ... then we have four sets of (actual or proposed) behaviour: 1. The original, current and standard behaviour is that Spam raises a NameError at function definition time, just as it would in any other context where the name Spam is undefined, e.g. `obj = Spam()`. 2. Under PEP 563 (string annotations), there is no NameError, as the annotations stay as strings until you attempt to explicitly resolve them using eval. Only then would it raise NameError. 3. Under PEP 649 (descriptor annotations), there is no NameError at function definition time, as the code that resolves the name Spam (and Any for that matter) is buried in a descriptor. It is only on inspecting `function.__annotations__` at runtime that the code in the descriptor is run and the name Spam will generate a NameError. 4. Guido would(?) like PEP 649 to be revised so that inspecting the annotations at runtime would not generate a NameError. Since Spam is unresolvable, some sort of proxy or ForwardRef (or maybe just a string?) would have to be returned instead of raising. Am I close? My initial thought was to agree with Larry about special cases, but perhaps we could borrow part of PEP 563 and return a string if the name Spam is unresolvable. Runtime type checkers already have to deal with forward refs that are strings, as this is legal, and always will be: def function(arg:'Spam') -> Any: ... so we're not putting any extra burden on them. And we had already agreed to implicitly use strings for annotations. So if I have understood the options correctly, I like the idea of a hybrid descriptor + stringy annotations solution. - defer evaluation of the annotations using descriptors (PEP 649); - on runtime evaluation, if a name does not resolve, stringify it (as PEP 563 would have done implicitly); - anyone who really wants to force a NameError can eval the string. More practically, folks will more likely delay evaluating the string until Spam has been created/imported and will resolve. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/F4BZLEQ36MIIBDRUIMNGWCSSE6AMYM5K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Type annotations, PEP 649 and PEP 563
On Thu, Oct 21, 2021 at 09:36:20PM -0700, Christopher Barker wrote: > On Thu, Oct 21, 2021 at 5:24 PM Steven D'Aprano wrote: > > > Runtime type checkers already have to deal with forward refs that are > > strings, as this is legal, and always will be: > > > > def function(arg:'Spam') -> Any: ... > > > > so we're not putting any extra burden on them. And we had already > > agreed to implicitly use strings for annotations. > > > > I'll take your word for it. However, other runtime uses for annotations may > not already need to support strings as types. > > Pydantic is the classic example. Pydantic supports stringified annotations. https://pydantic-docs.helpmanual.io/usage/postponed_annotations/ Any other runtime annotation tool has to support strings, otherwise the "from __future__ import annotations" directive will have already broken it. If the tool does type-checking, then it should support stringified annotations. They have been a standard part of type-hinting since 2014 and Python 3.5: https://www.python.org/dev/peps/pep-0484/#forward-references Any type-checking tool which does not already support stringified references right now is broken. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WVVBETE7UZ4WI6HOF7WCNHYOK6HCXRTA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Type annotations, PEP 649 and PEP 563
On Sat, Oct 23, 2021 at 09:49:10AM -0400, Larry Hastings wrote: > It's an debatable point since "from future" behavior is always off by > default. I'd certainly agree that libraries /should/ support stringized > annotations by now, considering they were nearly on by default in 3.10. > But I wouldn't say stringized annotations were a "standard" part of > Python, yet. As yet they are optional. Optional things aren't > standard, and standard things aren't optional. You misunderstand me. I'm not referring to PEP 563, which is still optional and requires the user to opt-in with a future import. I'm referring to the *manual* use of strings for forward references, which has been part of type-hinting since PEP 484 way back in 2014. https://www.python.org/dev/peps/pep-0484/#forward-references I expect that people were using strings for forward references before PEP 484, but it was 484 that made it official. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NGW3E43JXZ4N25GPXZIMSIAKHSMHUTSA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Oh wow, this is really impressive
I was using Python 3.10 and got this NameError when I mistyped a name: NameError: name 'KeyboardInterrupted' is not defined. Did you mean: 'KeyboardInterrupt'? It even works for attribute errors too. That's fantastic! This is a really amazing useability improvement, thank you everyone who was involved! I literally squeed :-) I just may spend a few days deliberately mistyping names in the REPL just to see this :-) -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2CUCVKZ44IYCW5QRV4CMVC7YDYBCJFYG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The list.sort(reverse=True) method not consistent with description
On Sun, Oct 31, 2021 at 01:32:29PM +1300, Greg Ewing wrote: > On 31/10/21 5:47 am, Raymond Bisdorff wrote: > >Should the tuples comparison is in this case, I thought, not be solely > >based on the first tuple component? > > Whether you think it should or shouldn't, the fact is that it's not. > This is documented in Section 5.6 of the Library Reference: > > "tuples and lists are compared lexicographically by comparing > corresponding elements. This means that to compare equal, every element > must compare equal and the two sequences must be of the same type and > have the same length." I don't think that applies if you provide a key function. In context, Raymond is referring to using itemgetter(0) as the key function, so only the first item in the tuple will be considered. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NOTXDVEMIAE5YKREK274QEPJGX6OA7JO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python
Thanks for writing this Petr! A few comments below. On Mon, Nov 01, 2021 at 01:17:02PM +0100, Petr Viktorin wrote: > >ASCII-only Considerations > >- > > > >ASCII is a subset of Unicode > > > >While issues with the ASCII character set are generally well understood, > >the're presented here to help better understanding of the non-ASCII cases. You should mention that some very common typefaces (fonts) are more confusable than others. For instance, Arial (a common font on Windows systems) makes the two letter combination 'rn' virtually indistinguishable from the single letter 'm'. > >Before the age of computers, most mechanical typewriters lacked the keys > >for the digits ``0`` and ``1`` I'm not sure that "most" is justifed here. One of the most popular typewriters in history, the Underwood #5 (from 1900 to 1920), lacked the 1 key but had a 0 distinct from O. https://i1.wp.com/curiousasacathy.com/wp-content/uploads/2016/04/underwood-no-5-standard-typewriter-circa-1901.jpg The Oliver 5 (1894 – 1928) had both a 0 and a 1, as did the 1895 Ford Typewriter. As did possibly the best selling typewriter in history, the IBM Selectric (introduced in 1961). http://www.technocrazed.com/the-interesting-history-of-evolution-of-typewriters-photo-gallery Perhaps you should say "many older mechanical typewriters"? > >Bidirectional Text > >-- The section on bidirectional text is interesting, because reading it in my email client mutt, all the examples are left to right. You might like to note that not all applications support bidirectional text. > >Unicode includes alorithms to *normalize* variants like these to a > >single form, and Python identifiers are normalized. Typo: "algorithms". This is a good and useful document, thank you again. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CHGK6LLBMVRQ6GGEMRWYJNRLUL7KUMVS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python
On Mon, Nov 01, 2021 at 11:41:06AM -0700, Toshio Kuratomi wrote:
> Unicode specifies the mapping of glyphs to code points. Then a second
> mapping from code points to sequences of bytes is what is actually
> recorded by the computer. The second mapping is what programmers
> using Python will commonly think of as the encoding while the majority
> of what you're writing about has more to do with the first mapping.
I don't think that is correct.
According to the Unicode consortium -- and I hope that they would know
*wink* -- Unicode is the universal character encoding. In other words:
"Unicode provides a unique number for every character"
https://www.unicode.org/standard/WhatIsUnicode.html
Not glyphs.
("Character" in natural language is a bit of a fuzzy concept, so I think
that Unicode here is referring to what their glossary calls an abstract
character.)
The usual meaning of glyph is for the graphical images used
by fonts (typefaces) for display. Sense 2 in the Unicode glossary here:
https://www.unicode.org/glossary/#glyph
I'm not really sure what they mean by sense 1, unless they mean a
representative glyph, which is intended to stand in as an example of the
entire range of glyphs.
Unicode does not specify what the glyphs for code points are, although
it does provide representative samples. See, for example, their comment
on emoji:
"The Unicode Consortium provides character code charts that show a
representative glyph"
http://www.unicode.org/faq/emoji_dingbats.html
Their code point charts likewise show representative glyphs for other
letters and symbols, not authoritative. And of course, many abstract
characters do not have glyphs at all, e.g. invisible joiners, control
characters, variation selectors, noncharacters, etc.
The mapping from bytes to code points and abstract characters is also
part of Unicode. The UTF encodings are part of Unicode:
https://www.unicode.org/faq/utf_bom.html#gen2
The "U" in UTF literally stands for Unicode :-)
--
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/I7ZRNIHSQ7UL4NSKOXFRYBYHQEXGNBPA/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python
On Wed, Nov 03, 2021 at 03:03:54AM +1100, Chris Angelico wrote: > On Wed, Nov 3, 2021 at 1:06 AM Petr Viktorin wrote: > > Let me know if it's clear in the newest version, with this note: > > > > > Here, ``encoding: unicode_escape`` in the initial comment is an encoding > > > declaration. The ``unicode_escape`` encoding instructs Python to treat > > > ``\u0027`` as a single quote (which can start/end a string), ``\u002c`` as > > > a comma (punctuator), etc. > > > > Huh. Is that level of generality actually still needed? Can Python > deprecate all but a small handful of encodings? To be clear, are you proposing to deprecate the encodings *completely* or just as the source code encoding? Personally, I think that using obscure encodings as the source encoding is one of those "linters and code reviews should check it" issues. Besides, now that I've learned about this unicode_escape encoding, I think that's going to be *awesome* for winning obfuscated Python competitions! *wink* -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/27IDDKAADVBAZSRZ2I5EO5SLXZIY6ANW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)
On Tue, Nov 02, 2021 at 05:55:55PM +0200, Serhiy Storchaka wrote: > All control characters except CR, LF, TAB and FF are banned outside > comments and string literals. I think it is worth to ban them in > comments and string literals too. In string literals you can use > backslash-escape sequences, and comments should be human readable, there > are no reason to include control characters in them. There is a > precedence of emitting warnings for some superficial escapes in strings. Agreed. I don't think there is any good reason for including control characters (apart from whitespace) in comments. In strings, I would consider allowing VT (vertical tab) as well, that is whitespace. >>> '\v'.isspace() True But I don't have a strong opinion on that. [Petr] > > For homoglyphs/confusables, should there be a SyntaxWarning when an > > identifier looks like ASCII but isn't? Let's not enshrine as a language "feature" that non Western European languages are dangerous second-class citizens. > It would virtually ban Cyrillic. There is a lot of Cyrillic letters > which look like Latin letters, and there are complete words written in > Cyrillic which by accident look like other words written in Latin. Agreed. > It is a work for linters, which can have many options for configuring > acceptable scripts, use spelling dictionaries and dictionaries of > homoglyphs, etc. Linters and editors. I have no objection to people using editors that highlight non-ASCII characters in blinking red letters, so long as I can turn that option off :-) -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RWE5FIWHUM5PSOJ6BI2PAO5TDE3KLC5D/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python
On Wed, Nov 03, 2021 at 11:21:53AM +1100, Chris Angelico wrote:
> TBH, I'm not entirely sure how valid it is to talk about *security*
> considerations when we're dealing with Python source code and variable
> confusions, but that's a term that is well understood.
It's not like Unicode is the only way to write obfuscated code,
malicious or otherwise.
> But to the extent that it is a security concern, it's not one that
> linters can really cope with. I'm not sure how a linter would stop
> someone from publishing code on PyPI that causes confusion by its
> character encoding, for instance.
Do we require that PyPI prevents people from publishing code that causes
confusion by its poorly written code and obfuscated and confusing
identifiers?
The linter is to *flag the issue* during, say, code review or before
running the code, like other code quality issues.
If you're just running random code you downloaded from the internet
using pip, then Unicode confusables are the least of your worries.
I'm not really sure why people get so uptight about Unicode confusables,
while being blasé about the opportunities to smuggle malicious code into
pure ASCII code.
https://en.wikipedia.org/wiki/Underhanded_C_Contest
Is it unfamiliarity? Worse? "Real programmers write identifiers in
English." And the ironic thing is, while it is very difficult indeed for
automated checkers to detect underhanded code in ASCII, it is trivially
easier for editors, linters and other tools to spot the sort of Unicode
confusables we're talking about here. But we spend all our energy
worrying about the minor issue, and almost none on the broader problem
of malicious code in general.
I'm pretty sure I could upload a library to PyPI that included
os.system('rm -rf .')
and nobody would blink an eye, but if I write:
A = 1
А = 2
Α = 3
print(A, А, Α)
everyone goes insane. Let's keep the threat in perspective. Writing an
informational PEP for the education of people is a great idea. Rushing
into making wholesale changes to the interpreter, not so much.
--
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/YGPSWZL4Z7LKTUHC25JVMHA5LUSLLQEL/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python
On Wed, Nov 03, 2021 at 11:11:00AM +0100, Marc-Andre Lemburg wrote: > Coming back to the thread topic, many of the Unicode security > considerations don't apply to non-Unicode encodings, since those > usually don't support e.g. changing the bidi direction within a > stream of text or other interesting features you have in Unicode > such as combining code points, invisible (space) code points, font > rendering hint code points, etc. > > So in a sense, those non-Unicode encodings are safer than > using UTF-8 :-) Thank you MAL for that timely reminder that most encodings are not Unicode. I have to admit that I often forget that there is a whole universe of non-Unicode, non-ASCII encodings. > Please also note that most character lookalikes are not encoding > issues, but instead font issues, which then result in the characters > looking similar. +1 -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NJFO5C7367F4NLLQTJRNNNUCRRLA6BES/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: containment and the empty container
On Mon, Nov 08, 2021 at 01:43:03PM -0800, Ethan Furman wrote: > When is an empty container contained by a non-empty container? [...] > SomeFlag.nothing in SomeFlag.something <-- ??? I don't think that consistency with other containers is particularly relevant here. More useful is consistency with other flag objects. What's SomeFlag? I presume it is some sort of Enum, or bitset. Presumably SomeFlag.nothing is equivalent to a bitset of all zeroes. (If it means something else, then I have no clue what to suggest.) You might have: something = 0b11010 0 in something # ??? 1 in something # False 2 in something # True 4 in something # False 8 in something # True 16 in something # True 32 in something # False So much is obvious. But what about `3 in something`? If that is interpreted as an *all* operation, you get: 3 in something --> all(i in something for i in (1, 2)) # False but if it is an *any* operation: 3 in something --> any(i in something for i in (1, 2)) # True I don't mean to imply that you must actually use all/any in your implementation. I mean only that it is conceptually equivalent to ANDing each of the flags (all) versus ORing the flags (any). *If* that is how you interpret your containment tests, that implies a natural interpretation for `nothing in something`. The vacuous truth of all is True, and of any is False: 0 in something --> all(i in something for i in ()) # True 0 in something --> any(i in something for i in ()) # False Vacuous truth is not the only possible interpretation. If you have some other obvious and natural interpretation of `0 in something` then you probably wouldn't be asking here, but if you did, you could follow that interpretation. With a good reason to violate the vacuous truth rules, it would only be a *little* surprising. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZRSZG4UM67GGZLHMUG5GQ4BINLMFC2O4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proposal: Allow non-default after default arguments
By the way, this discussion is probably better suited to the Python-Ideas mailing list. But since we're here... On Tue, Nov 09, 2021 at 11:37:40AM +0100, Sebastian Rittau wrote: > >>To me, the "natural" solution looks like this: > >> > >>def foo(x=None, y): ... [...] Chris Angelico asked: > >What would this mean, though: > > > >foo(2) > > > >Is that legal? > > No. This would be equal to foo(x=2) (same as now), meaning the required > argument "y" is missing. That's an odd interpretation. What you described earlier is very similar to the calling convention of range, which conceptually looks like this: range([start=0,] end, [step=1]) With your example of "foo(x=None, y)" I would expect foo(2) to mean that x gets the default and y gets the passed in argument 2, similar to the way that range(2) works. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S3USWLRSJYLOYMS52UH2YWMFOLTAT7EV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Having Sorted Containers in stdlib?
On Tue, Nov 09, 2021 at 04:23:50PM -0800, Hasan Diwan wrote:
> As of 3.7. dicts are sorted[1], but I'm unsure if the order can be
> overridden.
Dicts are not sorted, they are kept in insertion order.
>>> d = {3: 'a', 4: 'b', 1: 'c', 2: 'd'}
>>> d
{3: 'a', 4: 'b', 1: 'c', 2: 'd'}
See the docs:
https://docs.python.org/3/library/stdtypes.html#dict
although you have to scroll almost all the way to then end, just before
the dict view objects, to see it documented.
Sorting dicts has been discussed on the Python-Ideas mailing list, it is
too hard and expensive to justify for the limited use-cases for it. If
you want to sort a dict, you are best to sort the dict's keys, then
create a new dict. Or possibly use a dedicated Sorted Mapping type like
a red-black tree or similar.
Hasan wrote:
> >>> print(cmp(Girls, Boys))
> Traceback (most recent call last):
> File "", line 1, in
> NameError: name 'cmp' is not defined
The 'cmp' built-in was removed in Python 3.0.
In any case, dicts don't support order comparisons.
--
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/AV5RVB37YR7TZ3ZURI4ZIHCS5Y2XV67Y/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Having Sorted Containers in stdlib?
Hi Bob and welcome, Before we could even consider adding the sortedcontainers library to the standard library, we would need to hear from the maintainer(s) of the library that they agree to the move and would be able to continue maintaining the library under our release schedule and backwards compatibility guarantees. Otherwise, you would need to find a core developer willing to re-implement the containers and maintain them. I don't want to discourage you, but even if the maintainer is willing, we night not decide to add it. Every new feature, class and function adds to the weight of learning Python, and the cost of maintenance. We must balance that against the benefit, and only add features where the benefits are greater than the costs. Our decision making is usually very conservative, because we have strong requirements for backwards-compatibility. Once we add something to the stdlib, we can't easily change our mind and remove it again. So we follow the Zen of Python: >>> import this The Zen of Python, by Tim Peters [...] Now is better than never. Although never is often better than *right* now. Have you looked at the Python PEPs? If you are serious about pushing this proposal, your first step should be to read PEP 1 and then browse through the collection of successful and unsuccessful PEPs: https://www.python.org/dev/peps/pep-0001/ https://www.python.org/dev/peps/ -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VCM5PUEHCGRIYTPPRAZZUSEEJ3KW2DTN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Having Sorted Containers in stdlib?
On Tue, Nov 09, 2021 at 10:01:35PM -0800, Christopher Barker wrote: > Maybe a stupid question: > > What are use cases for sorted dicts? > > I don’t think I’ve ever needed one. Good question :-) > Also, I can’t quite tell from the discussion If a “sorted dict” implements > something new, or is an internal data structure that gives better > performance for particular use cases. I.e. is a sorted dict a Mapping? All dicts are mappings :-) I would expect any kind of sorted dict to support the full mutable mapping interface. Some mappings are not necessarily implemented as hash tables, as dict is. Some variety of tree is a common choice. I expect that a sorted dict (however it is implemented) would be a mapping that preserves sorted order on insertions and deletions, rather than insertion order. (Or some arbitrary order.) I haven't used one myself, but I would expect an API where you create a Sorted Dict (of whatever library or implementation you prefer), set an optional key function and direction (ascending or descending), then as you insert keys, the mapping keeps the keys in sort order. Note that this is different from *sorting* a dict, which (if it were supported) has to be done explicitly. Between sorts, the dict would be capable of getting out of sorted order. The idea of a *sorted* dict is that the sort order is an invariant, rather than something that can come and go as you insert and delete items. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JCFQYOSD5NQ3GFGLRVW7QWISXQL5LHS5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Having Sorted Containers in stdlib?
On Wed, Nov 10, 2021 at 05:11:33PM +1100, Chris Angelico wrote: > Nothing's technically new. You could make an inefficient sorted dict like > this: > > class SortedDict(dict): > def __iter__(self): return iter(sorted(self.keys())) You would need more than that. You would want to ensure that the dict's repr shows it is sorted order. You would need popitem to pop items in the appropriate order. There may be other APIs that sorted dicts provide than merely sorting the keys on iteration. You don't actually need to implement this to see how repeatedly sorting the keys would give terrible performance for anything above small sets of data. Here's Java's standard SortedMap: https://docs.oracle.com/javase/7/docs/api/java/util/SortedMap.html That should give you some idea of the interface a sorted dict would likely provide. > IMO this is a great tool to have on PyPI. Someone can start the naive > way, run into major performance problems, and then go "huh, maybe > someone else has already solved this problem". Doesn't need to be in > the stdlib for that. You may have missed that this thread has already discussed two mature, good quality sorted dict implementations that have been on PyPI for years: https://pypi.org/project/sortedcontainers/ https://pypi.org/project/treap-python/ Here are a few more: https://pypi.org/project/ruamel.ordereddict/ https://github.com/surenkov/PySortedDict https://pypi.org/project/sorteddict/ -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HPEXFUZR6Z6AHW4MHABWXT7UKCM6ELYZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Having Sorted Containers in stdlib?
On Thu, Nov 11, 2021 at 11:01:32AM -0800, Richard Levasseur wrote: > Should the stdlib have e.g. SortedList? Probably not, because the use cases > of such data types are too niche to a one-size-fits-all implementation, and > there are too many implementations with too many of their own settings. > Should it have e.g. BTree, RedBlackTree, SortedArrayList, etc? Probably so, > because those are somewhat fundamental data structures and implementing, > testing, and verifying them is very much non-trivial. While niche, having > them at your immediate disposal is very powerful. By that reasoning, we shouldn't have dicts, for exactly the same reason: for anyone wanting an associative array, there are so many implementation variants to choose from: - hash table with linear probing - hash table with chaining - AVL tree - red-black tree - judy array - splay tree - treap - scapegoat tree and many more, and most of them can be tuned. If Python didn't already have dict, your argument against SortedList would equally apply to it: they are "fundamental data structures and implementing, testing, and verifying them is very much non-trivial". So if your argument is correct, that would imply that standardizing on one single dict implementation, one that isn't even tunable by the caller, was a mistake. We should have provided a dozen different hash tables, trees and treaps. But... we didn't, and I don't think that Python is a worse language because we only have one associative array implementation in the stdlib. Whenever I need an associative array, I don't lose sleep over whether I could get 2% better performance for hits, at a cost of 17% worse performance for misses, by swapping over to some other implementation. I just reach for dict, knowing that it will almost always be good enough. > Last year, for fun, after wishing there was a SortedSet in the standard > lib, I ended up implementing a Red-Black Tree and BTree based sorted > dictionary/set[1]. After then trying to use them for my use case[2], I > found that, in order to fully and truly exploit their benefits, the basic > Sequence/Collection/Set/Dict APIs didn't really suffice. I needed APIs that > would let me, e.g. binary search to a particular spot and then iterate, or > give me a range between two points, etc. I believe that sortedcontainers.SortedSet provides that functionality via the irange() method. http://www.grantjenks.com/docs/sortedcontainers/sortedlist.html#sortedcontainers.SortedList.irange -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DWGJAXOVAY64XU7FBQI6D3D5NNFKZEHW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Having Sorted Containers in stdlib?
On Fri, Nov 12, 2021 at 10:07:13AM -0500, Paul Ganssle wrote: > I knew about sortedcontainers and I also don't remember ever seeing a > situation where I needed one or recommended its use. We have a very odd situation where apparently sortedcontainers is one of the most well-known, popular, most heavily downloaded libraries on PyPI. According to here: https://hugovk.github.io/top-pypi-packages/ sortedcontainers is the 290th most popular package on PyPI, ahead of such luminaries as pylint, black, selenium, mypy, django and nose. And yet, nobody(?) admits to either using it or knowing what it could be used for. How very curious :-/ -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7JVLJE4G6ZWM6PKXTMSZL7MVAIUWGZRH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)
Out of all the approximately thousand bazillion ways to write obfuscated Python code, which may or may not be malicious, why are Unicode confusables worth this level of angst and concern? I looked up "Unicode homoglyph" on CVE, and found a grand total of seven hits: https://www.cvedetails.com/google-search-results.php?q=unicode+homoglyph all of which appear to be related to impersonation of account names. I daresay if I expanded my search terms, I would probably find some more, but it is clear that Unicode homoglyphs are not exactly a major threat. In my opinion, the other Steve's (Stestagg) example of obfuscated code with homoglyphs for e (as well as a few similar cases, such as homoglyphs for A) mostly makes for an amusing curiosity, perhaps worth a plugin for Pylint and other static checkers, but not much more. I'm not entirely sure what Paul's more lurid examples are supposed to indicate. If your threat relies on a malicious coder smuggling in identifiers like "𝚑𝓮𝖑𝒍𝑜" or "ªº" and having the reader not notice, then I'm not going to lose much sleep over it. Confusable account names and URL spoofing are proven, genuine threats. Beyond that, IMO the actual threat window from confusables is pretty small. Yes, you can write obfuscated code, and smuggle in calls to unexpected functions: result = lеn(sequence) # Cyrillic letter small Ie but you still have to smuggle in a function to make it work: def lеn(obj): # something malicious And if you can do that, the Unicode letter is redundant. I'm not sure why any attacker would bother. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XNRW6JSFGO4DQOGVNY2FEZAUBN6P2HRR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)
On Mon, Nov 15, 2021 at 12:33:54PM +0400, Abdur-Rahmaan Janhangeer wrote: > Yet another issue is adding vulnerabilities in plain sight. > Human code reviewers will see this: > > if user.admin == "something": > > Static analysers will see > > if user.admin == "something": Okay, you have a string literal with hidden characters. Assuming that your editor actually renders them as invisible characters, rather than "something???" or "something□□□" or "something���" or equivalent. Now what happens? where do you go from there to a vunerability or backdoor? I think it might be a bit obvious that there is something funny going on if I see: if (user.admin == "root" and check_password_securely() or user.admin == "root" # Second string has hidden characters, do not remove it. ): elevate_privileges() even without the comment :-) In another thread, Serhiy already suggested we ban invisible control characters (other than whitespace) in comments and strings. https://mail.python.org/archives/list/python-dev@python.org/message/DN24FK3A2DSO4HBGEDGJXERSAUYK6VK6/ I think that is a good idea. But beyond the C0 and C1 control characters, we should be conservative about banning "hidden characters" without a *concrete* threat. For example, variation selectors are "hidden", but they change the visual look of emoji and other characters. Even if you think that being able to set the skin tone of your emoji or choose different national flags using variation selectors is pure frippery, they are also necessary for Mongolian and some CJK ideographs. http://unicode.org/reports/tr28/tr28-3.html#13_7_variation_selectors I'm not sure about bidirectional controls; I have to leave that to people with more experience in bidirectional text than I do. I think that many editors in common use don't support bidirectional text, or at least the ones I use don't seem to support it fully or correctly. But for what little it is worth, my feeling is that people who use RTL or bidirectional strings and have editors that support them will be annoyed if we ban them from strings for the comfort of people who may never in their life come across a string containing such bidirectional text. But, if there is a concrete threat beyond "it looks weird", that it another issue. > but will not flag it as it's up to the user to verify the logic of > things There is no reason why linters and code checkers shouldn't check for invisible characters, Unicode confusables or mixed script identifiers and flag them. The interpreter shouldn't concern itself with such purely stylistic issues unless there is a concrete threat that can only be handled by the interpreter itself. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KSIBL3KMONIETBKXSBPPMA27MACWIH33/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)
On Sun, Nov 14, 2021 at 10:12:39PM -0800, Christopher Barker wrote: > I am, however, surprised and disappointed by the NKFC normalization. > > For example, in writing math we often use different scripts to mean > different things (e.g. TeX's Blackboard Bold). So if I were to use > some of the Unicode Mathematical Alphanumeric Symbols, I wouldn't want > them to get normalized. Hmmm... would you really want these to all be different identifiers? 𝕭 𝓑 𝑩 𝐁 B You're assuming the reader of the code has the right typeface to view them (rather than as mere boxes), and that their eyesight is good enough to distinguish the variations even if their editor applies bold or italic as part of syntax highlighting. That's very bold of you :-) In any case, the question of NFKC versus NFC was certainly considered, but unfortunately PEP 3131 doesn't document why NFKC was chosen. https://www.python.org/dev/peps/pep-3131/ Before we change the normalisation rules, it would probably be a good idea to trawl through the archives of the mailing list and work out why NFKC was chosen in the first place, or contact Martin von Löwis and see if he remembers. > Then there's the question of when this normalization happens (and when it > doesn't). If one is doing any kind of metaprogramming, even just using > getattr() and setattr(), things could get very confusing: For ordinary identifiers, they are normalised at some point during compilation or interpretation. It probably doesn't matter exactly when. Strings should *not* be normalised when using subscripting on a dict, not even on globals(): https://bugs.python.org/issue42680 I'm not sure about setattr and getattr. I think that they should be normalised. But apparently they aren't: >>> from types import SimpleNamespace >>> obj = SimpleNamespace(B=1) >>> setattr(obj, '𝕭', 2) >>> obj namespace(B=1, 𝕭=2) >>> obj.B 1 >>> obj.𝕭 1 See also here: https://bugs.python.org/issue35105 -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7XZJPFED3YJSJ73YSPWCQPN6NLTNEMBI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)
On Mon, Nov 15, 2021 at 12:28:01PM -0500, Terry Reedy wrote: > On 11/15/2021 5:45 AM, Steven D'Aprano wrote: > > >In another thread, Serhiy already suggested we ban invisible control > >characters (other than whitespace) in comments and strings. > > He said in string *literals*. One would put them in stromgs by using > visible escape sequences. Thanks Terry for the clarification, of course I didn't mean to imply that we should ban control characters in strings completely. Only actual control characters embedded in string literals in the source, just as we already currently ban them outside of comments and strings. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XCPSQYKOX4YXDIAACDLL3I5OYWFGFLD7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)
On Mon, Nov 15, 2021 at 03:20:26PM +0400, Abdur-Rahmaan Janhangeer wrote: > Well, it's not so obvious. From Ross Anderson and Nicholas Boucher > src: https://trojansource.codes/trojan-source.pdf Thanks for the link. But it discusses a whole range of Unicode attacks, and the specific attack you mentioned (Invisible Character Attacks) is described in section D page 7 as "unlikely to work in practice". As they say, compilers and interpreters in general already display errors, or at least a warning, for invisible characters in code. In addition, there is the difficulty that its not just enough to use invisible characters to call a different function, you have to smuggle in the hostile function that you actually want to call. It does seem that the Trojan-Source attack listed in the paper is new, but others (such as the homoglyph attacks that get most people's attention) are neither new nor especially easy to actually exploit. Unicode has been warning about it for many years. We discussed it in PEP 3131. This is not new, and not easy to exploit. Perhaps that's why there are no, or very few, actual exploits of this in the wild. Homoglyph attacks against user-names and URLs, absolutely, but homoglyph attacks against source code are a different story. Yes, you can cunningly have two classes like Α and A and the Python interpreter will treat them as distinct, but you still have to smuggle in your hostile code in Α (greek Alpha) without anyone noticing, and you have to avoid anyone asking why you have two classes with the same name. And that's the hard part. We don't need Unicode for homoglyph attacks. func0 and funcO may look identical, or nearly identical, but you still have to smuggle in your hostile code into funcO without anyone noticing, and that's why there are so few real-world homoglyph attacks. Whereas the Trojan Source attacks using BIDI controls does seem to be genuinely exploitable. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FSHGS4AOAGTWKSWAADZWH5L2GGBWHHXE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)
On Mon, Nov 15, 2021 at 10:43:12PM +1100, Chris Angelico wrote: > The problems here are not Python's, they are code reviewers', and that > means they're really attacks against the code review tools. I think that's a bit strong. Boucher and Anderson's paper describes multiple kinds of vulnerabilities. At a fairly quick glance, the BIDI attacks does seem to be a novel attack, and probably exploitable. But unfortunately it seems to be the Unicode confusables or homoglyph attack that seems to be getting most of the attention, and that's not new, it is as old as ASCII, and not so easily exploitable. Being able to have А (Cyrillic) Α (Greek alpha) and A (Latin) in the same code base makes for a nice way to write obfuscated code, but it's *obviously* obfuscated and not so easy to smuggle in hostile code. Whereas the BIDI attacks do (apparently) make it easy to smuggle in code: using invisible BIDI control codes, you can introduce source code where the way the editor renders the code, and the way the coder reads it, is different from the way the interpreter or compiler runs it. That is, I think, new and exploitable: something that looks like a comment is actually code that the interpreter runs, and something that looks like code is actually a string or comment which is not executed, but editors may syntax-colour it as if it were code. Obviously we can mitigate against this by improving the editors (at the very least, all editors should have a Show Invisible Characters option). Linters and code checks should also flag problematic code containing BIDI codes, or attacks against docstrings. Beyond that, it is not clear to me what, if anything, we should do in response to this new class of Trojan Source attacks, beyond documenting it. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SXF2BG47UZTI7QM7GB3XCTGEV576UZOE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The current state of typing PEPs
On Sat, Nov 20, 2021 at 11:46:56PM -0800, Christopher Barker wrote: > One example is a use case of mine -- I have built a hierarchical object > system, built on dataclasses, in which the annotation absolutely has to be > an actual type(class) object. PEP 563 will very much break this use case. I don't think that's an insurmountable problem. I think that all you need is a small class decorator to evaluate the stringified annotations back to the real things. The PEP tells us the right way to evaluate annotations, so all(?) you need is a decorator to do that to each method in your class, and Bob's your uncle. Maybe PEP 563 could include a decorator in the typing module to destringify all the annotations in a class or function? As far as I can see from a brief scan of the PEP, and based on knowing next to nothing about your use-case, the only hypothetical problem might be this line in the PEP: "Consequently, using local state in annotations is no longer possible in general." but it's not entirely clear to me what Łukasz means by that, or whether it will affect your use-case. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3RYHEEARXUCK3S6WIZALAU75PEPG7GR2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The current state of typing PEPs
On Fri, Nov 26, 2021 at 12:15:10AM +0900, Stephen J. Turnbull wrote:
> I will grant that restricting the type of compiled annotations from
> typing.Any to "string serializing an object's AST" could reasonably be
> said to be "going out of your way to break other use cases". You have
> a valid beef here, although it's not obvious to me how it should be
> resolved.
I don't think that's what PEP 563 says. Annotations are not *restricted*
to only be strings, it is merely that when the function or class object
is built, the annotations are left as strings.
So we can post-process annotations and destringify them:
>>> from __future__ import annotations
>>> def func(arg:float) -> int:
... pass
...
>>> func.__annotations__
{'arg': 'float', 'return': 'int'}
>>> func.__annotations__['return'] = int
>>> func.__annotations__['arg'] = float
>>> func.__annotations__
{'arg': , 'return': }
There may be scoping issues to be sorted out, but I don't think they are
insurmountable.
--
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/24PKPJTI5SIES32C2KZVF6R6D3RLF6BI/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Optimizing literal comparisons and contains
On Mon, Nov 29, 2021 at 12:32:19PM +, Mark Shannon wrote: > Hi, > > I am surprised by the insistence on this thread for excluding comparisons > from constant folding. > Why should we special case comparisons? Am I missing something here? We[1] are worried that the benefit gained will not be worth the maintenance burden of constant folding comparisons. Unlike constant-folding arithmetic expressions, the benefit for comparisons is small: code that compares two literals e.g. `3 < 5` is probably very rare, outside of tests. So this will help almost nobody, but still require maintenance. (And the tests will need to be changed, if we add this, otherwise they will only be testing the keyhole optimizer, not the runtime comparison!) I have no idea of how much maintenance the keyhole optimizer requires. We have a volunteer willing to do the work (Jeremy). Do we have a core developer willing to review their work, and mentor them if if it is not up to standard? If not, then this conversation will go nowhere. [1] That's an editorial "we". Personally, I don't have an opinion one way or another :-) -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MUL7KTCPUGBTYZF4AXYI5UDE7UIC5ANR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The current state of typing PEPs
On Mon, Nov 29, 2021 at 07:56:16PM -0500, Terry Reedy wrote: > Maybe it should be reiterated with whatever decision comes forth that > > >>> def muladd(x, y, z): > ... return x * (y+z) > ... [...] > and other duck-typed code will always be legal, idiomatic, and even > expected as good practice for beginner, informal, exploratory, and > similar python code. Many of the typing-related PEP comes with such a disclaimer, listed as "Non-goals". For example: https://www.python.org/dev/peps/pep-0484/#non-goals Łukasz's stringified annotations PEP has a non-goals section: https://www.python.org/dev/peps/pep-0563/#non-goals Larry's deferred evaluation PEP does not: https://www.python.org/dev/peps/pep-0649/ but I don't think we should hold it against PEP-649. It's not trying to sneakily sneak mandatory static typechecking in by the back door like some sort of sneaking sneak :-) -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XRBNYBEK6ZBZUC2KIDBGEYKYXZ2CBF4C/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The current state of typing PEPs
On Tue, Nov 30, 2021 at 09:17:13AM +, Paul Moore wrote: > Also, related to the question Terry raised, IMO it would be useful to > have a clear statement on code that *does* use type annotations, but > violates them at runtime. To be specific, is the following considered > as an error? > > >>> def muladd(x: int, y: int, z: int) -> int: > ... return x * (y+z) > ... > >>> muladd(3.1459, 87.33, 2.7e2) > 1124.124447 Isn't that just duck-typing? You've got a function that is documented as taking arguments of one type (int), but actually will work with any numeric type (and some non-numeric types) that quacks like an int. muladd(2, 'ab', 'cd') # returns 'abcdabcd' If you replaced the type annotations with a docstring that listed the parameters x, y, z as int, would duck-typing be wrong? Maybe. I don't think we can make any concrete claims here. Sometimes duck-typing is perfectly fine. Sometimes its not. It depends on the function's implementation and its semantics. Sometimes calling a function with a duck-typed value seems to work fine but the result is meaningless junk. (Garbage In, Garbage Out.) I guess all we can say is that the Python language is agnostic and neutral on this matter. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/W24VGDUH6F43C6VT4OUGERNEZFW223MP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The current state of typing PEPs
On Tue, Nov 30, 2021 at 02:30:18PM +, Paul Moore wrote: > And to be clear, it's often very non-obvious how to annotate something > - in https://github.com/pfmoore/editables I basically gave up because > I couldn't work out how to write a maintainable annotation for an > argument that is "a Path, or something that can be passed to the Path > constructor to create a Path" (it's essentially impossible without > copy/pasting the argument annotation for the Path constructor). I thought that type inference was supposed to solve that sort of problem? If the typechecker can see that an argument is passed to the Path constructor, it should be able to infer that it must be the same types as accepted by Path. Aside: I'm a little disappointed in the way the typing ecosystem has developed. What I understood was that we'd get type inference like ML or Haskell use, so we wouldn't need to annotate *everything*, only the bits needed to resolve ambiguity. But what we seem to have got is typing like C, Pascal and Java, except gradual. Am I being unreasonable to be disappointed? I'm not a heavy mypy user, I just dabble with it occasionally, so maybe I've missed something. > Anyway, we're *way* off topic now, and I doubt there's much that the > SC or python-dev can do to change the community view on typing, > anyway. Heh. We could update PEP 8 to ban type annotations, then watch as the people who over-zealously apply PEP 8 to everything AND over-zealously insist on adding type annotations to everything have their heads explode. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Y6GOWIZV5JCOG5TP4ZZ4SVLXL4BGDJTI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Should isinstance call __getattribute__?
I'm looking for some guidance on a bug report involving isinstance and __getattribute__` please. The issue is that if your class overloads `__getattribute__`, calling isinstance on an instance will call the overloaded `__getattribute__` method when looking up `__class__`, potentially causing isinstance to fail, or return the wrong result. See b.p.o. #32683 https://bugs.python.org/issue32683 I see no reason why this isn't working as designed, __getattribute__ is intended to overload attribute access, and that could include the `__class__` attribute. Am I wrong? It has been suggested that isinstance should call `object.__getattribute__` and bypass the class' overloaded method, but I expect that would probably break objects which intentionally lie about their class. (Mocks? Stubs? Proxies?) Thanks, Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/W55NIIXBI2QNZUMTGTZNTXZ7TZSWUQ7F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should isinstance call __getattribute__?
On Thu, Dec 09, 2021 at 05:19:00PM +0100, Ronald Oussoren wrote: > https://mail.python.org/pipermail/python-dev/2015-October/141953.html > is an old thread about the difference between type(x)/Py_TYPE(x) and > x.__class__ that contains some insight about this. Thanks for the link Ronald, I remember that thread. It didn't really clarify things to me at the time, and re-reading it, it still doesn't. > Proxy types are one use case, although with some sharp edges. I'm not looking for use cases. I'm looking for a better understanding of how type() and isinstance() (and presumably issubclass) work. The best I can see is that type() sometimes believes __class__ but not always, that you can sometimes change __class__ but not always, but the rules that control when and why (or why not) are not clear or documented, as far as I can see. Is there a reference for how type(obj) and isinstance(obj, T) are intended to work, or is the implementation the only reference? Thanks in advance, -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GB2S2SMNDGS5UV5GG6O7HQUQSZP27OOI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should dataclasses add__set__ (and possibly __get __) descriptors ?
On Mon, Dec 13, 2021 at 06:43:27AM -, Vioshim wrote: > Anyways, at the moment that I write this message in python3.10.1, It > happens that when making a class with the dataclasses module, this > class can't actually be used in Multiple inheritance for Enum > purposes, this is mostly to avoid code repetition by having to write > the methods over again. [...] > class Entries(Foo, Enum): > ENTRY1 = Foo(1) > ENTRY2 = Foo(2) > ENTRY3 = Foo(3) I think this may be what you are looking for: >>> class Entries(Foo, Enum): ... ENTRY1 = 1 ... ENTRY2 = 2 ... >>> Entries.ENTRY1 Entries(a=1) >>> Entries.ENTRY1.a 1 >>> isinstance(Entries.ENTRY2, Foo) True -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UTNUZZUHTAXMEDJDZMTAWO6ZYXUBQD7G/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Python release announcement format
On Tue, Dec 14, 2021 at 11:56:09AM +0100, Yann Droneaud wrote: > Hi, > > I'm not familiar with the Python release process, but looking at the latest > release > > https://www.python.org/downloads/release/python-3101/ > > we can see MD5 is still used ... which doesn't sound right in 2021 ... > especially since we proved it's possible to build different .tar.gz that > have the same MD5 Please ignore my ignorant question, but what is the actual threat you are concerned about here? Anyone who gains access to the Python web site sufficient to smuggle in a different .tar.gz file will surely also be able to just change the checksum on the website to whatever they want. It doesn't matter what the checksum is or how strong it is, MD5 or SHA256 or any other checksum. An attacker who can change the file can change the checksum to match it. And both the download and the webpage listing the checksum are over https. If we don't trust https, the whole internet is broken and changing to a stronger checksum won't help. A hypothetical MITM attacker capable of breaking https and injecting new content into the download file can likewise change the checksum. Checksums are, as far as I understand it, only good for verifying data integrity against download errors, not malicious attack. And for that, MD5 is more than sufficient. Have I missed something? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IFBRCJ3LKLV3MINIFJF5WO6QHN54ERVC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)
On Wed, Dec 15, 2021 at 02:57:46PM -0800, Guido van Rossum wrote: > Another potential issue is that there may be some applications that take > refcounts at face value (perhaps obtained using sys.getrefcount()). These > would find that immortal objects have a very large refcount, which might > surprise them. But technically a very large refcount is totally valid, and > the kinds of objects that we plan to immortalize are all widely shared -- > who cares if the refcount for None is 5000 or 1610612736? As long as the > refcount of *mortal* objects is the same as it was before, this shouldn't > be a problem. I agree with your reasoning. But can we agree to document the presence and interpretation of the magic bit, so that if anyone actually does care (for whatever reason, good bad or indifferent) they can mask off the immortal bit to get the real ref num? Or maybe even have getrefcount() automatically mask the bit off. If we reserve the bit as the immortal bit, then is there any reason to keep that bit visible when returning refcounts? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DV4OYEFTJV6XLVYLIJBYR76SPTCHNTUT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)
On Thu, Dec 16, 2021 at 12:23:09PM +0100, Antoine Pitrou wrote: > > The "real number of references" would not be known for immortal objects. Oh that surprises me. How does that work? Does that imply that some code might not increment the ref count, while other code will? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EBSQGPCX67AFL5YK2UBKOR7PQS2LNLXZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)
On Thu, Dec 16, 2021 at 01:46:38PM +0100, Antoine Pitrou wrote: > If an object is immortal, then its refcount wouldn't change at all. Ah, that makes sense, thanks for the explanation. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/O45UDYGAKMB7HHBB4WYLV76M3Y2XKTAY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL
On Fri, Dec 17, 2021 at 11:35:24AM +1300, Greg Ewing wrote: > On 17/12/21 6:52 am, Eddie Elizondo via Python-Dev wrote: > >I've just updated the original Immortal Instances PR > > Is it just me, or does Immortal Instances sound like a > video game franchise? Or a Doctor Who episode. Doctor Who and the Immortal Instances of Doom. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6LCAF3XZQQB3OSOGH5ECMCJPSRJHTOIA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Function Prototypes
On Thu, Dec 23, 2021 at 03:00:03PM -, asleep.c...@gmail.com wrote:
> Mark Shannon initially proposed that functions be used as types and provided
> this example:
>
> @Callable
> def IntToIntFunc(a:int)->int:
> pass
>
> def flat_map(
> l: list[int],
> func: IntToIntFunc
> ) -> list[int]:
>
I have to admit, when Mark Shannon initially proposed that as an
improvement over both the current and the proposed syntax, I was so
taken aback that I initially thought he was being sarcastic and had
to read it a second time to realise he was serious :-(
The status quo is an in-place declaration using an anonymous type:
func: Callable[[int], list[int]]
# proposed equivalent
func: (int) -> list[int]
The anonymous type using Callable requires requires 26 characters,
including 7 punctuation characters.
The PEP 677 proposal cuts that down to 18 chars (6 punctuation chars)
while increasing readability: the arrow syntax is "executable pseudo
code". As far back as PEP 484 in 2014 this same syntax was used in
type-comments for Python2 straddling code:
https://www.python.org/dev/peps/pep-0484/#id50
Extending that to refer to the signature of a function is an obvious
step. Many other languages have converged on the same, or very similar,
syntax.
I've been using similar `param -> result` pseudo-syntax when sketching
out code using pencil and paper, or on a whiteboard, for years, and
nobody has failed to understand it.
In comparison, Mark's version:
@Callable
def IntToIntFunc(a:int)->int:
pass
# in the type declaration
func: IntToIntFunc
uses 54 characters, plus spaces and newlines (including 7 punctuation
characters); it takes up three extra lines, plus a blank line. As
syntax goes it is double the size of Callable.
It separates the type declaration from the point at which it is used,
potentially far away from where it is used.
It adds a new name to the global namespace, bloating the output of
introspection tools like dir(), help() etc.
And it *requires* a named (non-anonymous) type where an anonymous type
is all that is needed or wanted.
Being able to name types using an alias when it helps readability is
good. Being required to name them even at the cost of hurting
readability is not. Naming is hard, and bad names are worse than no
names.
Consider Mark's name for the function: "IntToIntFunc", which tells us
nothing that the signature (int)->int doesn't already tell us. It is the
naming equivalent of the comment:
x += 1 # Add 1 to x.
Your proposal is slightly more compact than Mark's: you drop the ending
colon and the body ("pass"), saving one line and five characters out of
the 54. But it suffers from the same major flaws:
- verbose and relatively heavy on vertical space;
- bloats the output of introspection tools;
- separating the definition of the type from where it is used;
- requiring a name for something which doesn't need a name.
If I had the choice between using the current syntax with Callable[] and
the proposed PEP 677 arrow syntax, I would almost always use the arrow
syntax. It matches the pseudo-syntax I already use when writing pseudo-
code on paper.
If I had the choice between Callable[] and this proposed function-as-a-
type syntax, I would stick to Callable. If I wanted to give the type a
name, for some reason, I would still use Callable, and just write an
alias. I cannot imagine any scenario where I would prefer this function-
as-a-type syntax over the other two alternatives.
> I further proposed that we make the body of a function non-mandatory
> and create a function prototype if it is omitted. I provided these
> examples:
>
> import typing
>
> @typing.Callable
> def IntToIntFunc(a: int) -> int
What do you get when the inevitable occurs, and you forget the
decorator? If I just write this:
def IntToIntFunc(a: int) -> int
it will create what sort of object?
[...]
> This new lambda syntax also allows you to create a function prototype
> by omitting the body. The original example can be rewritten as
> follows:
At least that brings back the ability to write it as an anonymous type,
but at the cost of adding a totally unnecessary keyword "lambda" and an
unused, redundant parameter name:
func: (int) -> int
func: lambda (a: int) -> int
--
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/SW4NYIYWIX7VBYTGIBCTOVMDWFQSTFB4/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Function Prototypes
On Thu, Dec 23, 2021 at 02:09:50PM -0800, Guido van Rossum wrote: > Without decorator too (that was Lukasz’ idea). Why bother with the > decorator (*if* we were to go there)? So that def func(params): pass creates a function object, and def func(params) makes a Callable type object? I'm going to *love* explaining the difference to beginners when they accidently do this and try to call func. Python is not Scratch and is not intended as a teaching language for kids, but some proposals are actively hostile to beginners, and I think this is one. Have we considered how this would effect something as simple as the meaning of keywords? - `class` creates classes, also known as types; - `def` creates functions, and also types, but not types you can use for anything except type-checking. Given that annotations are optional everywhere else, what happens if you leave out the annotations in the type definition? def SomethingFunction(x) Is that the same as Callable[[Any], Any] or Callable[[None], None]? Or are we going to create a rule that annotations are mandatory in some `def`s but optional in others? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5GXCY64PHTNBXXUHL4ULDVSECX2DWWIC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Function Prototypes
On Thu, Dec 23, 2021 at 07:04:17PM -, asleep.c...@gmail.com wrote: > One thing that you must consider is that function prototypes Just to be clear, by "function prototype", do you mean what PEP 677 calls a Callable Type? > have a few implications beyond typing but it seems like you're > only looking at it as a tool for type hinting. The interpreter will > create a function prototype object regardless of if you forget your > decorator, it needs to pass something to the decorator after all. What sort of implications beyond typing? Some questions: 1. What kind of object does `def Func(a:int)->int` create, if you leave out the decorator? Is that what you are calling a "function prototype object"? 2. What does it need the @Callable decorator for? 3. What can you do with it, apart from typing? 4. What happens if you leave out the annotations and just say `def Func(a)` alone? 5. Do you have any response to my other criticisms about this syntax? > After reading through your reply, I am seeing that the main concern > is the bloat added by the lambda keyword. Perhaps you should re-read my reply. The lambda keyword is the *least* part of my dislike of this proposal. > The fact that lambda is hard to type has been felt by almost everyone > who has ever used Python, Such exaggerations don't help your case. I have never felt the need to type a lambda expression, and doubt I'm so very unusual. Annotations have only existed for less than half of Python's existence. Even today, I doubt that as many as half of Python's user base are heavy users of typing. Or even casual users of typing. And many of them would not use lambda, or at least not in places where adding typing to it would add any value. In any case, allowing type-hints in lambda expressions is independent of the question of using the lambda keyword to declare a function prototype. I have no objection in principle to allowing annotations in lambda expressions if such a thing would actually be useful. But that doesn't mean I want to see lambda used as function prototype syntax: def map(func: lambda (obj: Any) -> Any, items: Sequence[Any]) -> Sequence[Any] especially not in preference to just using arrow syntax: def map(func: (Any)-> Any, items: Sequence[Any]) -> Sequence[Any] > this isn't just a problem that would be introduced by > function prototypes. PEP 677 has taken the lazy approach to solving > this issue What does that mean? What is lazy about it? > and has prioritized type hinting over functionality. PEP 667 > also suggests the usage of => for lambdas which would likely > never be accepted because of the confusion it would cause. Syntactic sugar for lambda is not part of PEP 667, it merely references the fact that people have suggested using => as shorthand for a lambda. For what its worth, I was skeptical about using two different arrows (one for declaring callable types, one for functions) when I first heard the idea (I think it was Guido who mentioned it?). But I've come to believe that whatever confusion there might be in using two arrows "do I use -> or => here? I never remember which is which" will be less, not more, than the confusion due to using the same arrow for both contexts. That is, I think, the experience from other languages. (Kotlin if I remember correctly? Maybe not.) But that's a discussion for when somebody writes a PEP for lambda shortcut syntax. > As someone who has used typing with Python, I do think that a new > callable syntax is needed, but I truly believe that PEP 677 is taking the > wrong approach. > > So what if we broke every Python program in existence by creating a > new lambda syntax, how would it look? Creating new syntax is backwards compatible: it doesn't break existing code that is syntactically correct. Only removing, or changing the meaning of, existing syntax will break "every Python program in existence". I doubt the Steering Council would accept such breakage. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MTWBA5T5ISPCL7S4PM6KE77AIEMPFDTT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Function Prototypes
On Thu, Dec 23, 2021 at 05:09:18PM -0800, Guido van Rossum wrote: > > def func(params) > > > > makes a Callable type object? > > > > No, no, no. That syntax has already been discredited. It has? Where? Have I missed something? This thread is about using that syntax as an alternative to Mark's proposal. If it is already ruled out, then somebody should mention it to asleep.cult (the original poster of this thread). I'm curious: what objections to asleep.cult's proposal don't equally apply to Mark's proposal? From what I can see, Mark's original proposal has all the same disadvantages, plus it is even more verbose. Dropping the need for the @Callable decorator reduces the level of verbosity somewhat (one less line, nine fewer characters) but all the other negatives remain. Honestly, I cannot see a single positive to using `def` statements. Sure, we can do it, and an experienced programmer could infer the meaning of it, but given the choice of writing an anonymous type in place where you want it, versus having to pre-declare a function prototype with a name that adds nothing to the readability of the code before using it, why would I prefer the `def` version? This is not a rhetorical question. The fact that the existing feature (Callable) and the PEP 677 arrow syntax are anonymous, and can be written in place rather than needing to be pre-declared with a name, are positives. *Requiring* a name to use this `def` syntax is a point against it. If I need a named type alias, I can already create one, and name it: IntToIntFunc = Callable[[int], int] and while I can see that there are complicated signatures where a named alias would be useful: FileOpener = Callable[ ... ] # complicated signature we can already do that, so the `def` syntax adds nothing. For simple cases we don't need a name. The name IntToIntFunc adds nothing that isn't just as clear, if not more so, in the signature itself. It is like the comment: x += 1 # add one to x > Mark's proposal was > ``` > @Callable > def func(params): pass > ``` Indeed, and I already wrote a criticism of that proposal. Removing the decorator saves one line and nine characters, but the other criticisms remain. > My question is, why does it need `@Callable`? Lukasz proposed just using > any (undecorated) function, with the convention being that the body is > `...` (to which I would add the convention that the function *name* be > capitalized, since it is a type). But without the Callable decorator, it isn't a type, its a function. You're just using it as a type (or to be precise, a function prototype). I'm okay with naming conventions reflecting usage (we sort of already do that, with int, float, etc) but we should be clear about what's really happening. `def` creates a function. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VTOOJLPU2HPIW6TQBBYTW64W4DFGCQEG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Function Prototypes
On Fri, Dec 24, 2021 at 01:28:44PM +1100, Steven D'Aprano wrote: > Honestly, I cannot see a single positive to using `def` statements. ... > This is not a rhetorical question. Hmm, I think I may have come up with one. If we did teach type checkers to use actual functions as prototypes, that would allow the same function object to do double duty as the type-hint (as a prototype) and as an implementation of that prototype. So if you wanted a function that has the same signature as builtin `print`, you could just say something like: def traverse_graph( g: Graph, maxdepth: int = -1, # Same signature as print, defaults to print visitor: print = print, ) -> None: instead of having to replicate print's signature. The duplication `: print = print` is a bit on the nose, but not too much. And maybe type-checkers could infer that if a parameter defaults to a function, its type-hint should clearly be the same as the default? This would require builtins to gain annotations, of course. Which they don't currently have :-( And for the avoidance of doubt, I am not suggesting this be limited to only builtin functions. Any function with annotations would work. So to my mind, that moves Mark's proposal into the category of an independent new feature separate to PEP 677, rather than a competitor or alternative proposal: * Teach type-checkers to use functions made with `def` as function prototypes (regardless of what the body is). * Likewise for classes (use the `__call__` method's signature as the prototype). while PEP 677 remains as a nice-looking shorter syntax for in-place anonymous Callable Types. I remain strongly -1 on `def ...` as an alternative to PEP 677, I don't think that the `def` syntax makes a good alternative to either Callable or the proposed arrow syntax. But being able to use an existing function, complete with implementation in the body, as a prototype, yeah, I'll buy that. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YDQOLLRRWKYJHCTFKQIJDXCJEOEOFLZJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Function Prototypes
On Fri, Dec 24, 2021 at 01:54:35PM +1100, Chris Angelico wrote: > My reading of this is that a function IS the type of a function with > that signature, just like how None means the type NoneType. Is that > correct? That's not the status quo, but I think the idea is that it will be. Except that I think that the preferred terminology is that it is a *function prototype* rather than a type, since it is a prototype, not an actual class you can instantiate. A template, if you will: "Any callable with the same signature (modulo parameter names) as this template / protocol is acceptable." > Or putting it another way: is this (silly) example legal? [snip example] I think maybe it should be, but right now, mypy reports it as illegal: [steve ~]$ mypy ~/func_prototype.py /home/steve/func_prototype.py:7: error: Function "func_prototype.repeat_string" is not valid as a type If we allow this, I see no reason why functions used as prototypes should be required to have an empty body. (Ellipsis or pass.) I think that actual, useful functions with an implementation should be allowed, as in your example. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JU2JCG5TLKA35AGD636OJH7NWQ75HFMX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Function Prototypes
On Fri, Dec 24, 2021 at 11:53:22AM +0200, Serhiy Storchaka wrote: > Some library provide function foo() which returns an instance of private > type _Foo and people start using it as a type hint. If people want to shoot themselves in the foot, is it our job to stop them? Typically the only promise we make is that the interpreter won't segfault, not to protect coders from making bad decisions in their code. If _Foo is private, you shouldn't be using it as a type hint, not even indirectly through a function prototype. *Everything* about _Foo is subject to change without notice, including its very existence. It seems a bit strange to accept the risk of unpredictable changes to a private implementation detail, while worrying about backwards- incompatible changes to a public function. > A new version > converts foo() into a class. It is usually a safe backward compatible > change, except that now all type hints became wrong. It is a safe backwards compatible change only if you don't care about the type of foo. As soon as you use foo as a function prototype, then you now care about its type, just as much as if you inspected it with type() or isinstance(). In other words, it's not *actually* a backwards compatible change in a language like Python where functions are first-class objects. We can just get away with that in a context where the type of the callable usually doesn't matter, just as we usually don't care if the repr() changes. In this case, if we introduce function prototypes, then people will learn that changing a factory function to a class is a breaking change for people who do type-checking, just as changing the repr of objects is a breaking change for people who use doctests. On the other hand, changing a function to a callable instance should not be a breaking change, if we follow my earlier suggestion that objects with a `__call__` method should be usable as prototypes too. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/O7IHYHVGJMXO6WJQOIJACARRU6QJC232/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Function Prototypes
On Fri, Dec 24, 2021 at 06:24:03PM -, Jim J. Jewett wrote: > Steven D'Aprano wrote: > > In comparison, Mark's version: > > @Callable > > def IntToIntFunc(a:int)->int: > > pass > > # in the type declaration > > func: IntToIntFunc > > uses 54 characters, plus spaces and newlines (including 7 punctuation > > characters); it takes up three extra lines, plus a blank line. As > > syntax goes it is double the size of Callable. > > I think it takes only the characters needed to write the name IntToIntFunc. That's only true if IntToIntFunc is a builtin, otherwise it needs to be defined somewhere. It doesn't just magically happen. If you are using the declaration many times, then I acknowledge that it may be worth the effort of pre-declaration and naming. (But see below.) Particularly if the signature is complicated, although I think that will be relatively rare. But in the worst case, you may only use it once. So the entire cognitive burden of pre-declaration (both writing it and reading it) applies to that one use. > The @callable def section is a one-time definition, and not logically > part of each function definition where it is used. The status quo is that we can use an anonymous type in the annotation without pre-defining it, using Callable. PEP 677 proposes a new, more compact syntax for the same. Any proposal for function prototypes using `def` is directly competing against Callable or arrow syntax for the common case that we want an anonymous, unnamed type written in place. Even in the case that we want to give the type a name that we plan to use repeatedly, this `def` syntax is still competing directly against what is already possible using the status quo: use Callable to create a named type alias. But with the `def` syntax, you can *only* use it as a named, pre-defined object. So half, maybe 90%, of your use-cases disappear. Any time that we have a short, simple Callable that doesn't require a name, why would we bother creating a do-nothing function just so we can use it as a prototype? I don't think many people would. I know I wouldn't. That would be the equivalent of filling your program with trivial plus_one(x) and times_two(y) functions instead of just using `x+1` and `2*y`. So the benefit of the `def` syntax comes from that relatively small subset of cases: - the callable signature is complicated; - we wish to refer it it multiple times; - giving it a name (like "FileOpener", say, not "IntToInt") aids clarity. That's not to be sneered at. But in those circumstances, we don't need the `def` syntax, because we can already use Callable and a type alias. So the `def` syntax adds nothing we don't already have, it is no easier to use, it is more verbose, not less. But if we can use an existing function as the prototype instead of having to declare the prototype, that shifts the balance. If we already have some function, then there is no extra cost in having to declare it and give it a name, it already has been declared and given a name. > I get that some people prefer an inline lambda to a named function, > and others hate naming an infrastructure function, but ... > > Why are you even bothering to type the callback function? If it is > complicated enough to be worth explicitly typing, then it is > complicated enough to chunk off with a name. I would say the opposite: most callback or key functions have very simple signatures. If my function takes a key function, let's say: def spam(mylist:[str], a: int, b: float, c: bool|None, key: Callable[[str], str], ) -> Eggs: mylist = sorted(mylist, key=key) ... the relevant signature is (str) -> str. Do we really need to give that a predefined named prototype? def StrToStr(s: str) -> str: pass I would argue that very few people would bother. If somebody did, they probably also defined type aliases for ListOfStr and BoolOrNone, and wish they were using Java or Pascal *wink* It seems to me that most callbacks and key functions have short signatures. Certainly all the ones I have written do: they typically take a single argument, of a known type, and return a known type. > Having to switch parsing modes to understand an internal ([int, float, > int] -> List[int]), and then to pop that back off the stack is much > harder. I notice that you just used something very close to PEP 677 arrow syntax totally unself-consciously, without any need to explain it. I think this is good evidence that far from being confusing, this is a completely natural syntax that we already interpret as a function prototype. > Hard enough that you really ought to help your reader out with a > name, What are you g
[Python-Dev] Re: Function Prototypes
I've done some more thinking more about Serhiy's worry about changing a factory function to a class, and how that would change the meaning of type-hints. Say: def spam(x: Eggs, y:Cheese) -> _Aardvark: # actual factory function implementation # later, we use it as a function protocol def myfunction(a: int, callback: spam) -> str: ... If spam changes implementation from a factory function to the actual class of _Aardvark, keeping the name: class spam: # body of _Ardvark goes here that would completely change the meaning of the myfunction type declaration. I still think that is not a scenario we need to care about. In my mind, that counts as a user-applied footgun. But if people disagree, and Serhiy's argument persuades them, then we can still use functions as their own prototype. @Callable def spam(x: Eggs, y: Cheese) -> _Aardvark: # actual factory function implementation The Callable decorator would just flag the function as *permitted* to be used as a prototype, with no other change. Or perhaps we could have @FunctionPrototype. Alternatively, we could write the consumer's annotation like this: def myfunction(a: int, callback: Prototype[spam]) -> str: ... where Prototype[x] uses: - the signature of x if x is a function or method; - the signature of `x.__call__` if x is a callable instance; - the signature of `x.__new__` or `x.__init__` is x is a class; as the prototype. However it is spelled, we might require functions to opt-in before they can be used as prototypes, in other words the decorator is the author's promise that they aren't going to change the function into a class, or change its signature, without the usual deprecation warnings etc. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MOZOXB73UIYGE745ZWORG4DWYM2PD72N/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] issues-test-2 spam?
Apologies if this is the wrong place to raise this (where is the right place?) but over the last four days, I've received ten subscription notices for python/issues-test-2 on Github. Is anyone else also getting multiple subscription notices? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q37XLFRF2H3OQFV55D7ASILCQ57XO6XE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Suggestion: a little language for type definitions
On Fri, Jan 07, 2022 at 04:27:37PM -0800, Christopher Barker wrote: > Note that if PEP 563 is ultimately accepted, then Annotations would be > strings, and type checkers could use any language they wanted. Annotations still have to be syntactically valid Python expressions. >>> from __future__ import annotations >>> def func(arg: array[1...10] of float) -> str: pass File "", line 1 def func(arg: array[1...10] of float) -> str: pass ^^^ SyntaxError: invalid syntax Of course if you explicitly wrap your annotation in quotation marks, you can use any syntax you like (think: regexes, SQL, etc). But without a *standard* annotation syntax: - static checkers will disagree on what annotations mean; - runtime introspection will be difficult; and - IDEs and syntax colourisers are going to just treat them as strings. We can write little DSLs with any syntax we like using explicitly quoted strings. We're not limited to do this in annotations. But while DSLs tend to be specific to your own library or application, annotations exist in a wide ecosystem of both static and runtime tools that expect to interpret annotations. Writing your own little DSL for annotations cuts you off from the rest of the Python ecosystem. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/65UGW2D5ATEVDQVNPYX4B7QZFQJOLXOO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Suggestion: a little language for type definitions
On Sat, Jan 08, 2022 at 12:32:35PM +0200, Serhiy Storchaka wrote: > 08.01.22 01:59, jack.jan...@cwi.nl пише: > >> If I can make a wild suggestion: why not create a little language for > >> type specifications? > > We need a way to define aliases. For example, write: > > Data = Mapping[str, Sequence[Tuple[int, T]]] > Factory = Callable[[int, Iterable[str]], Optional[list[Data[T Can't we already do that? https://docs.python.org/3/library/typing.html#type-aliases Type aliases are great. But there are times where we don't want to write an alias when we can just write the type in-place, just as there are times where we don't want to write a function when we can just use an in-place expression. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZRYSXMF2HRC5HXIZLQ4FTSL44PF6TXPS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Suggestion: a little language for type definitions
On Sat, Jan 08, 2022 at 12:59:38AM +0100, jack.jan...@cwi.nl wrote:
> I posted this suggestion earlier in the callable type syntax discussion, at
> which point it was completely ignored. Possibly because it’s a really stupid
> idea, but let me post it again on the off chance that it isn’t a stupid idea
> but was overlooked.
>
> > If I can make a wild suggestion: why not create a little language
> > for type specifications?
Any time we are tempted to prefix a question with "Why not ...", the
question is backwards. The right question is, "why should we ...".
Python is 30 years old and mature, with a HUGE ecosystem of users,
libraries, tooling, etc. It is far, far easier to get changes wrong than
to get them right, which is why we should be conservative when it comes
to language changes (including syntax). Changes to the language are
"default deny", not "default accept", and it is up to the proponents of
the change to prove their case, not for opponents to disprove it.
https://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html
https://www.curiousefficiency.org/posts/2011/04/musings-on-culture-of-python-dev.html
> > If you look at other programming languages you’ll see that the “type
> > definition sub-language” is often completely different from the
> > “execution sub-language”
[...]
Yes, and that makes other programming languages harder to use and harder
to learn. Even Pascal, which has a nice human-readable declaration
syntax, requires you to learn more before you can do anything. C is
particularly bad, since there are effectively three distinct languages
to learn: macros, type declarations, and C code.
Anyone who has learned enough C to get good at it is likely to under-
estimate how hard C is to learn, due to survivorship bias (those who
have survived their apprenticeship as a newbie C coder to become an
expert are not representative of all newbie C coders) and the curse of
knowledge (experts often find it hard to put themselves in the shoes of
non-experts).
So I think that we should avoid the temptation to have a distinct
language for type annotations. I think that Guido's initial reasoning,
that type annotations are Python expressions, rather than distinct
syntax for declarations, was correct, and I think that the Steering
Council is right to insist on keeping annotations aligned as Python
code.
Mind you, that doesn't necessarily mean that we cannot introduce new
syntax for use in annotations. (Just as we added three-argument slicing
and Ellipsis specifically for use in Numpy.) So long as they are usable
outside of annotations, they remain "part of the Python language" rather
than a distinct "type declaration syntax".
For example, the arrow syntax for Callable `(int) -> str` (if accepted)
could be a plain old Python expression, usable anywhere the plain old
Python expression `Callable[[int], str]` would be.
> > Python typing uses basically a subset of the execution expression
> > syntax as its declarative typing language.
I think a good way of putting it is that the typing module defines a
little DSL (Domain Specific Language) for type hints, but annotations
themselves are just Python expressions.
> > What if we created a little language that is clearly flagged, for
> > example as t”….” or t’….’? Then we could simply define the
> > typestring language to be readable, so you could indeed say t”(int,
> > str) -> bool”. And we could even allow escapes (similar to
> > f-strings) so that the previous expression could also be specified,
> > if you really wanted to, as t”{typing.Callable[[int, str], bool}”.
The following are not rhetorical questions. I don't know the answers,
which is why I am asking.
1. Are these t-strings only usable inside annotations, or are they
expressions that are usable everywhere?
2. If only usable inside annotations, why bother with the extra prefix
t" and suffix "? What benefit do they give versus just the rule
"annotations use this syntax"?
3. If usable outside of annotations, what runtime effect do they have?
The t-string must evaluate to an object. What object?
4. If the syntax allowed inside the t-string is specified as part of the
Python language definition, why do we need the prefix and suffix?
E.g. if we can write:
# Equivalent to T = Callable[[int], str]
T = t"(int) -> str"
and have the arrow syntax defined by the language, then surely the
prefix and suffix is redundant.
Likewise, if this is allowed:
def func(arr: t"array [1...10] of int") -> str: ...
why not just allow this?
def func(arr: array [1...10] of int) -> str: ...
5. What difference, if any, is there between `t"{expression}"` and
`expression`?
If there is no difference, then I don't think that the t-string syntax
adds anything to this proposal. Remove the t-string syntax, and just
write the type expression.
This is not the case with f-strings, where they actually do add
something to the code:
[Python-Dev] Re: Suggestion: a little language for type definitions
On Sun, Jan 09, 2022 at 01:30:33AM +0100, jack.jan...@cwi.nl wrote: > In the last year I’ve contributed mods to two packages that were > rejected because I hadn’t provided typing (and black formatting, but > that’s a different subject). I’ve reluctantly done so. And while I > *hated* it because of the unreadability I _do_ like the outcome Typing is hard because it is often more abstract and less concrete than the code you are adding type hints to. It is *metaprogramming*. And it involves unfamiliar jargon (covariant, contravariant, typeguards, etc). But the annotations themselves are plain ol' Python expressions. So if you hate type annotations because they are unreadable, then you hate Python because Python is unreadable. There may be some unfamiliarity if you aren't doing a lot of typing (what's a ParamSpec?), and due to the use of square brackets instead of round, but if you can read expressions: spam(eggs | cheese, aardvark) then you can read type expressions: Spam[Eggs | Cheese, Aardvark] Creating a whole new language to describe type hints will go against that. All the existing complexity of typing will still exist, but on top of that, there will *also* be the problem that the syntax used for type expressions will *not be Python* but a second language. Wrapping that second language in t"..." will not change that. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7L2GBLWKQYRKLUZAJBSH3EH77JXG5NLO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Suggestion: a little language for type definitions
On Sun, Jan 09, 2022 at 02:20:45AM +0100, jack.jan...@cwi.nl wrote: > Barry: > > The advantage to users of keeping the languages the same is that > > readers of your code don’t have to learn two disparate syntaxes to > > make sense of what they’re reading. One of Python’s enduring > > strengths has been its readability. > > Agreed. But if the little language is (a) clearly flagged and (b) has > a different domain I think this is much less of a problem. I disagree. > I don’t > think f-strings are seen as a problem, far from it, because they’re > clearly flagged. On the contrary, f-strings are not really a second language. f-strings involve *ordinary Python expressions* plus a set of formatting codes which are mostly similar or identical to those used by the format method and string interpolation with the `%` operator. > That’s why I suggested t-strings. And while from a > Python parser point-of-view the grammar of current type expressions > are the same as the grammar of other Python code I think that for > human readers this is not the case: there’s a lot of square brackets > but nothing is being indexed, to name one major readability issue... Nothing is being *indexed* here either: mydict[key] I don't think that Mathematica code is harder to read because it uses square brackets for function calls instead of round brackets. I challenge you to say that you cannot read these: Range[10] FindShortestPath[graph, start_vertex, target_vertext] Even when Mathematica uses syntax that is unfamiliar, I expect that you would be able to guess what this does: StringReplace["abbaabbaa", "ab" -> "X"] and if you can't, it's not because of the square brackets. As I mentioned in a previous email, I can see a number of reasons why typing is hard, but the syntax (ordinary Python expressions) is not why it is hard. We can, I think, improve elements of the typing DSL. `T|S` is, I think, an improvement over `Union[T, S]` for the same reason that `a|b` would be an improvement over `bitwise_and(a, b)` or `set_intersection(a, b)`. Likewise, I think that we should accept PEP 677 (arrow notation as sugar for Callable). But I wouldn't want to see type hints diverge into a completely different language from Python. > I introduced the t-strings specifically because I think it would be > beneficial to have the little language a clearly flagged and have as > little interaction with “normal” Python as possible. And that is exactly why I think that it is not a good idea. Having type hints use regular Python syntax is not a design flaw to be fixed. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N42DDG3MKIVBSWHMONSBJIVMVK5H2COA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Suggestion: a little language for type definitions
On Sat, Jan 08, 2022 at 06:30:53PM -0800, Ethan Furman wrote: > On 1/8/22 5:46 PM, Steven D'Aprano wrote: > > > [...] if you hate type annotations because they are unreadable, then you > > hate Python because Python is unreadable. > > Not so. Are you disputing that annotations use the same syntax as other Python expressions? If not, I don't see how you can deny that "type annotations are unreadable" implies "Python expressions are unreadable", which in turn implies "Python is unreadable". > A simple list comprehension is (usually) quite readable, while a > triply-nested list comprehension all on one line is not. Indeed. We can abuse any syntax. So do we conclude that comprehensions are "unreadable" because we can write obfuscated triply-nested list comprehensions? > Similarly, adding type information in between a variable name and its value > is not (for me, and apparently others too) readable. I think that "unreadable" or "not readable" is a complaint that gets overused, often for the most trivial cases, to the point that it loses all credibility. Especially when it comes from people who are fluent in C (which may not be you, Ethan). http://unixwiz.net/techtips/reading-cdecl.html "Easily learned", huh. I think that this is one of the clearest examples of the curse of knowledge as it applies to programming that one could hope to find. Anyway, let's compare: # C int n = 44; # Pascal var n: integer; n := 44; # Typescript var n: number = 44; # Java int n = 44; # Python n: int = 44 There are millions who find the C, Pascal, TypeScript and Java perfectly readable. I don't find it credible that people are incapable of reading the last one. Aside: such a type hint is redundant, as mypy is perfectly capable of inferring that n = 44 makes n an int. Style guides should recommend against such redundancy, and I would certainly flag that in a code review. A better example of a *useful* type hint would be: L: list[str] = [] > Most horribly of all, cluttering a function header with type information is > most unreadable. I hear you. Adding redundant or unnecessary type hints to parameters just for the sake of having type hints is just clutter, especially if they are never tested by actually running a type checker over the file. (Untested code is broken code. If not right now, it soon will be.) Fortunately, we have *gradual typing*, and nobody should be forced to use type hints in their projects if they don't want them. Just as we don't make linters mandatory, we don't make typing mandatory either. I think that, outside of very simple functions, once we make the decision to annotate a function, we should space them out: # Better def func(spam: list[str], eggs: float, cheese: str = 'cheddar', aardvark: str|bytes = "", eels: Optional[Tuple[int, str]] = None ) -> Hovercraft: which makes them much easier to read. Trying to cram them all into one line is abuse of the syntax every bit as bad as cramming a triply-nested list comp into one line: # Worse def func(spam: list[str], eggs: float, cheese: str = 'cheddar', aardvark: str|bytes = "", eels: Optional[Tuple[int, str]] = None) -> Hovercraft: I can read it, I just don't want to. It is too much like hard work compared to the previous layout. Even if you don't run a type-checker, those annotations can make useful documentation. (At least *sometimes*.) If the parameter name doesn't make it clear what types are allowed, then the annotation can make it clear. So if you don't use a static checker, you can think of type annotations as introspectable documentation. > I started using Python at 2.5. It was simple, clean, and elegant. And I started using Python at 1.5, when the syntax was even simpler and cleaner. And to this day I will never forget the first time I read Python code, after being told repeatedly how easy to read it, and I couldn't make head or tails of it. All those colons and square brackets, it might as well have been APL. (Not that I knew what APL was back then.) I knew what a for-loop was, from Pascal, Hypertalk and HP RPN calculators: # Pascal for i := 0 to 10 do begin block; end; # Hypertalk repeat with i = 0 to 10 block end repeat # HP-48 RPN language 0 10 FOR I block NEXT but I kept seeing loops like this in Python: for i in range(11): or worse: for somename in [stuff, thing, another_thing, widget]: and worse of all: for somename in values[1:-1]: Python for loops looked nothing like any for loop I had seen before, and they freaked me out, and at the
[Python-Dev] Re: Suggestion: a little language for type definitions
On Sat, Jan 08, 2022 at 08:36:57PM -0600, Skip Montanaro wrote: > > > > So if you hate type annotations because they are unreadable, then you > > hate Python because Python is unreadable. > > > > That seems rather harsh. I suspect if those of us who are uncomfortable > with the typing subsystem actually hated Python we would have found our way > to the exits long ago. Right. That's my point, or at least part of it. I've read heaps of complicated, confusing, even outright obfuscated Python expressions. Some of it even written by me, that a couple of months later I couldn't work out what I had done. So did I conclude that "Python expressions are unreadable"? No I did not. When I read an obfuscated, confusing, complex type hint, do I decide that all type hints are "unreadable"? No I don't do that either. Here is the type hint for `len`, taken from the stub file in typeshed: def len(__obj: Sized) -> int: ... Putting the mysterious double underscore naming convention aside, I do not find it credible that anyone capable of programming Python beyond a beginner level can find that "unreadable". Not by any definition of unreadable I can think of. Even if you can't guess what "Sized" means, it isn't that hard to track it down: from typing import Sized # among many others and from there to collections.abc. This is not brain surgery folks :-) https://www.youtube.com/watch?v=THNPmhBl-8I We should not dismiss all of typing as "unreadable". Type hints are just expressions. If you can read Python expressions, you can read type hints. Some are simple, some are complex, some are confusing. We should do our best to improve the complex and confusing ones, using all the tools at our disposal: * named type aliases; * style guides for laying out the annotations to make them physically easier to read (one parameter per line in complex function signatures works for me); * improving the `typing` module DSL; * perhaps even adding new syntax like the arrow syntax; but we shouldn't dismiss the whole thing as "unreadable" if what we actually mean is "its unfamiliar and I don't like it". > Typing was always supposed to be optional, And it still is. Alas, we can't do anything about third-party projects mandating type hints (maybe they have a good reason for mandating them!). But perhaps we can help discourage some of the excessive zeal for annotating everything in sight. > so I > didn't worry too much about it at the time. As Jack indicated though, while > it may be optional at the language level, it's often not truly optional at > the organizational level. We can't prevent organisations and third-parties mandating the use of linters, or IDEs, or particular naming conventions, or any other style convention they want. PEP 8 zealotry is especially prevalent out in the world. I've come across people with their own idiosyncratic style, like "Never use comprehensions, only for-loops", and others who insist "Never use for-loops, only comprehensions". What are we going to do, dismiss comprehensions as a bad idea because some people are irrationally pro- or anti-comprehensions? I don't think so. > As you indicated, there are two things going on, Python syntax and the > semantics which go along with it. Python's economical syntax is a terrific > reflection of its runtime semantics, hence the use of the phrase > "executable pseudocode" to describe Python (at least in the past). Right. Beyond the easy cases, typing is hard. It is often easier to write code that works for typical data you care about, than to convince the type-checker that the code works :-) I don't know if this applies to gradual typing, but I imagine it probably does. Type checking is a hard problem, and if your type system is powerful enough to use, it is undecidable: http://composition.al/blog/2017/02/27/why-does-a-turing-complete-type-system-make-type-checking-undecidable/ https://forums.swift.org/t/swift-type-checking-is-undecidable/39024 Even if your type system is not Turing complete, it is still going to be pretty powerful. We're not using Pascal any more :-) And that means that the types themselves are communicating some fairly complex semantics. Blaming the syntax for something which is inherently hard is not helpful. > Just because you are using Python syntax for your declarations doesn't mean > that (a) mapping the semantics of the desired declarations onto existing > syntax will be straightforward or (b) that the semantics of those > declarations will be reflected as effortlessly as it reflects runtime > semantics. Indeed. And we can say the same thing about using Python syntax as code. Mapping the semantics of your desired behaviour into syntax is not always straightforward, nor is reading the code and inferring the semantics. If it were, anyone could be a rockstar ninja coder, and programming would be a minimum wage job. We accept without blinking comprehensions, decorators, descri
[Python-Dev] Re: Suggestion: a little language for type definitions
On Mon, Jan 10, 2022 at 05:39:42AM +1100, Chris Angelico wrote: > From my understanding, "x->y" would create a Callable if given two > *types*, but its meaning if given two other objects is still > undefined. The PEP requires parentheses around the argument list, so that would be a SyntaxError. The PEP also states that the arrow syntax would be equivalent to calling Callable. Callable currently enforces that the return type actually is a type, but doesn't check the input types. (I don't know if that is a deliberate design or an oversight.) Assuming that it is an oversight, I would expect that only the following values would be legal for the x and y objects: - a type; - a string (which gets converted to a ForwardRef); - None; and anything else would result in a TypeError. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OIHDPKBSWSRV3QP2VOKP6PPA7AV65V5P/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Suggestion: a little language for type definitions
On Sun, Jan 09, 2022 at 08:42:14AM -0800, Christopher Barker wrote: > Perhaps it's worth remembering that this thread spun off one about adding > syntax to Python because the current syntax isn't capable of easily > expressing an important type hinting concept (i.e. Callable). I shall quote the PEP's justification (in part) for the benefit of those who haven't read it. [quote PEP 677] There are a few usability challenges with Callable we can see here: • It is verbose, particularly for more complex function signatures. • It relies on two levels of nested brackets, unlike any other generic type. This can be especially hard to read when some of the type parameters are themselves generic types. • The bracket structure is not visually similar to how function signatures are written. • It requires an explicit import, unlike many of the other most common types like list. Possibly as a result, programmers often fail to write complete Callable types. Such untyped or partially-typed callable types do not check the parameter types or return types of the given callable and thus negate the benefits of static typing. [/quote] These are legitimate issues with the Callable type hint. Does that mean that the existing syntax "isn't capable of easily" expressing callables? I don't think so. "Easily" is subjective, and at the risk of undermining the justification for the PEP, I think it is fairly easy: # type hint for a function of two parameters, the first takes # an int, float or complex, the second a bool, and returns # either a str or bytes. from typing import Callable T = Callable[[int|float|complex, bool], str|bytes] It was harder to write it out in English than to write the type hint. So its not *difficult*, there's one import and one class with a subscript needed. The type system is perfectly capable of expressing that. It is no more hard to read than any other expression of equivalent complexity: from aardvark import Surveying T = Surveying((spam+eggs+cheese, eels), foo*bar) But can we do better? The PEP authors think we can, and I am inclined to agree with them. While it is true that "not everything needs to be a builtin", in the case of typing, we are evolving to use builtins where possible: * we use int, and always have, not typing.Int; * what was once typing.List[int] is now list[int]; * what was once Union[float, int] is now float|int etc, but Callable is the last remaining common compound type without a syntactic shortcut. Its not that declaring callables are *too hard* (let alone impossible) but that we can do better than what we've got. Callable[[int|float|complex, bool], str|bytes] (int|float|complex, bool) -> str|bytes The second looks like executable pseudo-code for a function declaration. It avoids the import and the nested square brackets, and it is shorter without being so terse it becomes cryptic. The improved syntax looks just like a function signature with the parameter names excised: def func(a: int|float|complex, b: bool) -> str|bytes which I think is a win. > So arguing that Python is completely readable for type hints is a bit > off-mark, isn't it? What does "completely readable" mean? If you mean that it is impossible to compose a type-hint of such stupendous complexity and obfuscatory brilliance that even the infamous RFC-822 email address validator regex appears trivial in comparison? http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html Of course I don't mean that. Type hints are expressions, and expressions can have arbitrary complexity, including "too damn high". I argued that type hints are no harder to read than other expressions of the equivalent complexity. I didn't argue that they can't be abused or that the syntax cannot be improved. More to follow... -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZCOX33PTCQI3TUZPIO4GLLUPZJDXYIE5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Suggestion: a little language for type definitions
TL;DR: declaration only syntax is a non-starter. Even if we tried to add
specialised syntax that only appears in annotations, it would become a
regular Python expression almost immediately.
On Sun, Jan 09, 2022 at 08:42:14AM -0800, Christopher Barker wrote:
> Is it more clear for readers to have two different (but related) related
> syntaxes for two different (but related) purposes, or to have one Sytax
> that is used in different ways?
That is not a valid "either/or" choice. The choice is:
* Python expressions that have multiple meanings (the status quo);
(e.g. subscripting `obj[x]` already had 4+ meanings before its
use in typing)
* or Python expressions that have multiple meanings, PLUS new
declaration syntax that is only valid in annotations.
And I will argue below that the culture of Python is such that the
second choice is practically *impossible* without a radical shift in the
culture.
Let's say that we decide to add specialist declaration-only syntax for
type-hints. What next?
First, we need a way to distinguish between type-hints that are
declarations from type-hints that are expressions. Let's say that we use
the proposed t-string syntax:
spam: t"declaration goes here"
That tells the interpreter to treat everything inside the quotes as a
pure declaration, not an expression, with no runtime effect.
I'm going to put aside the question of what syntax is allowed for the
declarations inside the quotes, because that's actually not that
important. What happens after we add this declaration syntax?
The first thing that happens is that people interested in runtime
processing of annotations will want to inspect the declaration, without
having to scan the source code (which might not be available).
Python has a strong culture of runtime introspection, starting with
dir() way back in Python 1.x days, so if you think the core devs are
going to resist the call to make those annotations visible at runtime,
I think you are remarkably ~~wrong~~ optimistic :-)
That means that the declaration has to have a runtime effect.
Something has to go into the `__annotations__` dict:
__annotations__['spam'] = ... # what goes here?
(It doesn't matter precisely what the value that goes into the mapping
is, it could be a string, or a type object, whatever it is, it is a
value.)
So our declaration is no longer a declaration, like `global`, it is an
expression that returns a value. The parser merely restricts its use to
annotations like `spam: t"declaration"`.
Why can't we use that nifty new declaration syntax in type aliases? I
mean, it already evaluates to a value that can be inspected at runtime,
so we just have to relax the restriction on where it can appear so we
can use it in type aliases as well as directly in annotations:
T = t"declaration goes here"
spam: T|None
But if we can do that, then our declaration is just another expression.
It has a value, it can appear outside of annotations. People are going
to want to treat it as a first-class value (no pun intended), even if
only for testing purposes:
# Unit tests
for T in [t"declaration",
t"another declaration",
t"yet another",
t"my hovercraft is full of eels"]:
self.assertIsInstance(T, typing.Generic)
...
Even if it doesn't happen from day 1, rapidly whatever restrictions on
where the syntax is allowed will be relaxed, because that's the sort of
language Python is. Once we make that first (and inevitable!) step to
give these type declarations an introspectable value, then they will
become first-class expressions as sure as death and taxes.
So I maintain that *declaration only syntax* is doomed. Whatever syntax
we invent, there is going to be irresistable pressure to make it a
first-class expression and not just restrict it to annotations.
And if that is the case, then why do we need the t-string quotes?
That's not a rhetorical question.
Something like t-string quotes, or some other pair of delimiters, *may*
be useful if we have our hearts set on syntax which is radically
different from regular Python expressions:
spam: t"array 1 to 50 of int" # Hybrid of Pascal and Hyperscript
eggs: t"{(+⌿⍵)÷≢⍵}" # I don't even...
Just as we have `[ ... ]` delimiters for list comprehensions. Maybe it
turns out that there are cases where we need delimiters. But if so, I
hope they aren't quotation marks (with or without the t-prefix), since
that will be confusing as far as the existing use of strings as forward
references.
But why do we want type hints to be so radically different from regular
Python expressions? How does that make typing easier to read and the
language less complicated, if we have to learn *two* languages to be
fluent in Python instead of one?
Outside of that, the t-string delimiters are redundant. There is no need
to wrap the proposed arrow syntax in quotes, as the PEP makes clear they
can be handled j
[Python-Dev] Re: PEP 677 (Callable Type Syntax): Final Call for Comments
On Thu, Jan 13, 2022 at 06:41:14PM +1000, Nick Coghlan wrote: > If such a protocol were to be proposed, then int.__arrow__ could be defined > such that "1 -> 100" was equivalent to "range(1, 100)" Why? What's the benefit? It is slightly shorter, but no more clear than range. I would never guess that `1 -> 100` would mean a range object. I would think it was a transformation of 1 -> 100. In e.g. Ruby the equivalent of range has syntax 1...100, which matches how we write ranges of values in English. If we had an arrow operator beyond the typing proposal, I think it would be a terrible wasted opportunity to use it for something as unclear and trivial as a range object :-( -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L6XT3PVYSUKTA6PZDPKLGATXA32NT5AB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 677 (Callable Type Syntax): Final Call for Comments
On Thu, Jan 13, 2022 at 04:23:09AM -, Dennis Sweeney wrote: > Like others expressed, I don't like the idea of the typing and > non-typing parts of Python separating. Its a good thing that this PEP doesn't separate the typing and non-typing world. The arrow syntax will be a plain old Python expression that is usable anywhere, not just in annotations. It will only be *useful* in code that makes use of generic types, whether that is for annotations or runtime introspection, or exploration in the REPL, but that's okay. `is` is a built-in operator despite have exceedingly limited use-cases and sometimes being an attractive nuisance. > Has anyone considered adding a new special method like `__arrow__` or > something, In the absense of any useful functionality for this arrow syntax, I think that is a clear case of YAGNI. As it turns out, I do have a use-case for an arrow operator, but it wouldn't use a dunder either. And since my use-case doesn't have a PEP written, it would be unfair of me to derail the conversation with a half-baked proposal that is nowhere near ready to be a PEP. But if it gets rejected, all bets are off :-) If you do have some non-trivial uses for the arrow operator, it would be good to hear what they are. > that would be user-definable, but also defined for tuples > and types as returning a Callable? For example `int -> str` could mean > Callable[[int], str], and (int, str) -> bool could mean Callable[[int, > str], bool]. That wouldn't work. The PEP explains that they don't want people to be able to write: int -> bool without the parentheses. I agree with them. I think it is important that the syntax be visually similar to an anonymous function signature with the parameter names scrubbed out: def func(a:int) -> str https://www.python.org/dev/peps/pep-0677/#parenthesis-free-syntax Analogy: we require generator comprehensions to be surrounded by parentheses: it = (expr for i in sequence) In the desired syntax: (int) -> bool that is not a tuple. Its a parenthesised comma-separated expression. The same objection applies to using the `>>` operator. There is no way to force `(int) >> str` or prevent `int >> str`. > I would find that sort of semantics more agreeable since > Python already has operators that dispatch to dunder methods, And Python already has operators which don't: - `or` and `and` - `is` and `is not` - name binding (assignment) `=` - walrus assignment operator `:=` - ternary `if else` operator. So there is plenty of precedent for dunder-less operators. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/W7Y4Z7M7QBJXKRE6IL6LEIVUSFNVFRE4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request to revert unittest and configparser incompatible changes in Python 3.11
On Wed, Jan 26, 2022 at 02:40:32PM -0800, Neil Schemenauer wrote: > On 2022-01-18 23:14, Gregory P. Smith wrote: > > > >Our stdlib unittest already enables warnings by default per > >https://bugs.python.org/issue10535. > > > >Getting the right people to pay attention to them is always the hard part. > > I wonder if we can do a bit better in that regard. When I install 3rd > party packages, I create a usercustomize.py file that uses > filterwarnings() to turn off all the warnings I don't care about. I > don't know how but maybe we could make that easier to do. That way, you > don't get buried in warnings coming from code you don't maintain. Or maybe, as a developer (not an end-user of an app), you could be more proactive in reporting those warnings to the third party, and encouraging them to fix them. Maybe even submitting a patch? If we use a library, then we surely care about that library working correctly, which means that if the library generates warnings, we *should* care about them. They are advanced notice that the library is going to break in the future. Of course I understand that folks are busy maintaining their own project, and have neither the time nor the inclination to take over the maintenance of every one of their dependencies. But we shouldn't just dismiss warnings in those dependencies as "warnings I don't care about" and ignore them as Not My Problem. Like it or not, it is My Problem and we should care about them. Especially in the case of open source software, the lines of responsibility are blurred. Open source libraries are not proprietary black boxes with a strict division between the vender that supplies the library and the people who use the library. They are fully transparent, we can see the warnings and, at least potentially, see how to fix them. And we have the legal right to. This is a hard problem, but it is not solely a technical problem. It is partly a social problem, and you cannot fix social problems with technology. People are ignoring the warnings, and not just the immediate developers of the software, but their downstream users. The open source mantra about many eyes making bugs shallow doesn't work when everyone is intentionally closing their eyes to the warnings of pending bugs. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BCEKHDT76GGG2E65DCCIX4DC6OL5CIW4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Increase of Spammy PRs and PR reviews
On Sun, Jan 30, 2022 at 08:36:43AM -0800, Jelle Zijlstra wrote: > Agree, the count of 1.6k open PRs is not a good look for new contributors. How does that compare to other comparable open source projects? Number of open PRs per KLOC seems like a more useful metric than just the number of open PRs. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QV4TXPY6KOERZTYG6FWPYRM7NGRIWH6K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?
On Tue, Feb 08, 2022 at 12:44:46PM +, Steve Dower wrote: > Agreed. CPython should be specific, Python should be as vague as > possible. Otherwise, we would prevent _by specification_ using Python as > a scripting language for things where floats may not even be relevant. I don't think that being *vague* is desirable. Being *permissive* on the other hand may be. > It's not even about hardware capabilities (unless you believe Python is > only for running in a terminal, in which case we'll just have to > disagree). It could be about an app using short Python snippets as a > customisation language for some domain-specific task rather than > inventing a brand new language for it. I'm sorry, I don't understand what point you are trying to make here. Regarding "running in a terminal", I don't see how the environment (terminal, GUI, something else) is relevent. Regarding your second point, about domain-specific tasks, I don't understand. If your domain-specific task doesn't need to use floats, just don't use floats. There's surely no need to invent a whole new language for some task because your app only needs to do string processing, say. Have I misunderstood what you are trying to say? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CFLJJN64M4WW6DTRLSGZZTQLU7OHW5OH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?
On Mon, Feb 07, 2022 at 05:35:17PM -0800, Gregory P. Smith wrote: > CPython: yes. we use a double. > Python the language: no. (float is single precision on many micropython > platforms as it saves precious ram and performance, plus microcontroller > fpu hardware like an M4 is usually single precision 32bit) If we are to *officially* support non-double floats, it would be nice if sys.float_info were to tell us explicitly how wide the floats are rather than having to try to reverse engineer it from the other information there. A floating point expert can probably look at this: sys.float_info(max=1.7976931348623157e+308, max_exp=1024, max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, radix=2, rounds=1) and immediately recognise that those values imply a 64-bit float, but I expect most people will not. If Python the language is going to support single, double, quad precision floats, and maybe even minifloats with just 16 or even fewer bits, then can we please add a field to float_info telling us how many bits the floats have? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EUWN4TDI62FK7L6SOG6IAV3AOINDBH6O/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?
On Mon, Feb 07, 2022 at 06:23:52PM +, Mark Dickinson wrote: > - Should we require IEEE 754 floating-point for CPython-the-implementation? > - Should we require IEEE 754 floating-point for Python-the-language? If the answer to those questions are Yes, that rules out using Unums, posits, sigmoid numbers etc as the builtin float. (The terminology is a bit vague, sorry.) Do we want that? https://ieeexplore.ieee.org/document/808 https://en.wikipedia.org/wiki/Unum_%28number_format%29 https://github.com/interplanetary-robot/SigmoidNumbers Posits are hyped as "better than IEEE-754", I have no idea if it is all hype or if they actually are better or just different. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7LUAQ32ZAHKWBJHLHUYEB7I5BZNDXGB7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?
On Tue, Feb 08, 2022 at 05:48:46PM -0800, Gregory P. Smith wrote: > On Tue, Feb 8, 2022 at 2:41 PM Steven D'Aprano wrote: > > If the answer to those questions are Yes, that rules out using Unums, > > posits, sigmoid numbers etc as the builtin float. (The terminology is a > > bit vague, sorry.) Do we want that? > > It does not rule anything else out should they become viable. This is just > a statement that to build cpython we require ieee754 support. It does not > say anything about how our Python float type is implemented internally. Posits do not implement IEEE-754. They aren't merely a different internal representation of the IEEE-754 floating point standard, they implement a different numeric system altogether. Things that IEEE-754 require, such as signed zero and signed infinity, are not supported by posits. Posits include a single unsigned zero and a single unsigned infinity. Earlier, I made a mistake: I misremembered that posits support a single NAN, and so I removed Mark's question about requiring NANs from my quoting. I was wrong to do so: posits do not have any NANs. So if we require NANs, or IEEE-754, that rules out posits as the builtin float. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IKJSL3CEL7YZX3XPCJM4SA6BXVFN27Q3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Make __mro_entries__ mandatory for non-types
On Sat, Mar 05, 2022 at 11:27:44AM +0200, Serhiy Storchaka wrote: > Currently the class can inherit from arbitrary objects, not only types. Is that intentionally supported? I know that metaclasses do not have to be actual classes, they can be any callable with the correct signature, but I didn't know that classes can inherit from non-classes. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/K44D3H2L2B4KNKMYURZHL5OOSOGDOAY6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Make __mro_entries__ mandatory for non-types
On Sat, Mar 05, 2022 at 04:42:55PM -, Jason Madden wrote: > zope.interface relies on this behaviour. The example you give shows that Interface is a class. It merely has a metaclass which is not `type`. (I presume that is what's going on behind the scenes.) I'm asking about the example that Serhiy shows, where a class inherits from something which is not a class at all. In his example, the base is 1, although it gives a TypeError. I'm asking if that sort of thing is suppposed to work, and if so, how? -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/E2IWXPFY32R6JS22XXFORMNQTI4S6AOK/ Code of Conduct: http://python.org/psf/codeofconduct/
