Re: [Python-ideas] add a time decorator to timeit.py
> On 7 Oct 2018, at 22:44, Guido van Rossum wrote: > > So someone ought to submit a PR that adds (brief) documentation for this, > with reference to this thread. I was trying to write this documentation when I noticed that the docs already mention this! "The stmt and setup parameters can also take objects that are callable without arguments." Maybe we should reword this? Obviously many people here have looked at the docs and not spotted it. I'll post a PR with lambda examples under "Basic Examples", I think this should make people take notice of this feature. / Anders ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
Chris Angelico writes: > On Wed, Oct 10, 2018 at 5:09 AM Stephen J. Turnbull > wrote: > > > > Chris Angelico writes: > > > > > Both processes are using the virtual memory. Either or both could be > > > using physical memory. Assuming they haven't written to the pages > > > (which is the case with executables - the system mmaps the binary into > > > your memory space as read-only), and assuming that those pages are > > > backed by physical memory, which process is using that memory? > > > > One doesn't know. Clever side-channel attacks aside, I don't care, > > and I don't see how it could matter. > > It matters a lot when you're trying to figure out what your system > is doing. Sure, but knowing how your system works is far more important. Eg, create a 1TB file on a POSIX system, delete it while a process still has it opened, and it doesn't matter how you process the output of du or ls, you still have 1TB of used file space not accounted for. The same applies to swapfiles. But "df" knows and will tell you. In fact, "ps" will tell you how much shared memory a process is using. I just don't see a problem here, on the "I'm not getting the data I need" side. You do have access to the data you need. > > > > > Tell me, which process is responsible for libc being in memory? > > > > > Other than, like, all of them? > > > > > > > > Yes. Why would you want a different answer? > > > > > > Because that would mean that I have way more *physical* memory in use > > > than I actually have chips on the motherboard for. > > > > No, that's like saying that because you have multiple links to a file > > on disk you're using more physical disk than you have. > > Actually, that's exactly the same problem, with exactly the same > consequences. How do you figure out why your disk is full? How do you > enforce disk quotas? How can you get any sort of reasonable metrics > about anything when the sum of everything vastly exceeds the actual > usage? You add up the right things, of course, and avoid paradoxes. The disk quota enforcement problem is indeed hard. This sounds to me like a special problem studied in cost accounting, a problem which was solved (a computation that satisfies certain axioms was shown to exist and be unique) in a sense by Aumann and Shapley in the 1970s. The A-S prices have been used by telephone carriers to allocate costs of fixed assets with capacity constraints to individual calls, though I don't know if the method is still in use. I'm not sure if the disk quota problem fits the A-S theorem (which imposes certain monotonicity conditions), but the general paradigm does. However, the quota problem (and in general, the problem of allocation of overhead) is "hard" even if you have complete information about the system, because it's a values problem: what events are bad? what events are worse? what events are unacceptable (result in bankruptcy and abandonment of the system)? Getting very complete, accurate information about the physical consequences of individual events in the system (linking to a file on disk, allocating a large quantity of viritual memory) is not difficult, in the sense that you throw money and engineers at it, and you get "df". Getting very complete, accurate information about the values you're trying to satisfy is possible only for an omniscient god, even if, as in business, they can be measured in currency units. Steve ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
On 10Oct2018 00:42, Stephen J. Turnbull wrote: Chris Angelico writes: > On Tue, Oct 9, 2018 at 10:05 PM Greg Ewing wrote: > > Chris Angelico wrote: > > > In contrast, a mmap'd file is memory that you do indeed own. > > > > Although it's not really accurate to say that it's owned by > > a particular process. If two processes mmap the same file, > > the physical memory pages holding it appear in the address > > spaces of both processes. Subject to COW, I presume. Probably in units smaller than the whole file (per page?) Yes, pages (whatever using the memory paging service works in). But not COW. Changes to the memory affect the file contents and are visible to the other process. It is mapped, not pseudocopied. Cheers, Cameron Simpson ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
Stephen J. Turnbull wrote: Subject to COW, I presume. Probably in units smaller than the whole file (per page?) It can be COW or not, depending on the options passed to mmap. And yes, it's mapped in units of pages. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
On Wed, Oct 10, 2018 at 5:09 AM Stephen J. Turnbull wrote: > > Chris Angelico writes: > > > Both processes are using the virtual memory. Either or both could be > > using physical memory. Assuming they haven't written to the pages > > (which is the case with executables - the system mmaps the binary into > > your memory space as read-only), and assuming that those pages are > > backed by physical memory, which process is using that memory? > > One doesn't know. Clever side-channel attacks aside, I don't care, > and I don't see how it could matter. It matters a lot when you're trying to figure out what your system is doing. > > > > Tell me, which process is responsible for libc being in memory? > > > > Other than, like, all of them? > > > > > > Yes. Why would you want a different answer? > > > > Because that would mean that I have way more *physical* memory in use > > than I actually have chips on the motherboard for. > > No, that's like saying that because you have multiple links to a file > on disk you're using more physical disk than you have. Actually, that's exactly the same problem, with exactly the same consequences. How do you figure out why your disk is full? How do you enforce disk quotas? How can you get any sort of reasonable metrics about anything when the sum of everything vastly exceeds the actual usage? ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
Chris Angelico writes: > Both processes are using the virtual memory. Either or both could be > using physical memory. Assuming they haven't written to the pages > (which is the case with executables - the system mmaps the binary into > your memory space as read-only), and assuming that those pages are > backed by physical memory, which process is using that memory? One doesn't know. Clever side-channel attacks aside, I don't care, and I don't see how it could matter. > > > Tell me, which process is responsible for libc being in memory? > > > Other than, like, all of them? > > > > Yes. Why would you want a different answer? > > Because that would mean that I have way more *physical* memory in use > than I actually have chips on the motherboard for. No, that's like saying that because you have multiple links to a file on disk you're using more physical disk than you have. > Yes, virtual memory can be over-allocated, but physical memory > can't. How do you measure where your physical memory is being used, > if adding up the usage of all processes exceeds the available > resources? lsof can do it, I guess. The kernel knows which pages of which processes' virtual memory are backed by physical memory. But just as running over the array of inodes tells you how much physical disk is allocated, without looking at directories to find out what it's allocated to, the kernel probably has a bitmap or similar so it knows which pages of physical memory are in use. All problems are easy once you have enough indirection. :^) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] support toml for pyproject support
On Mon, 8 Oct 2018 09:26:12 -0400 David Mertz wrote: > I agree here. I briefly urged against using the less used TOML format, but > I have no real skin in the game around packaging. I like YAML, but that's > also not in the standard library, even if more widely used. Agreed with David. Also, please note that one argument against YAML is its complexity, which logically also entails implementation complexity (and therefore makes it less likely that a YAML implementation would enter the stdlib). Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] support toml for pyproject support
If I had the energy to argue it I would also argue against using TOML > in those PEPs. I partook in that discussion, and I still have no idea why TOML was chosen, over, say, a defined subset of YAML, or a slightly extended JSON. But the folks that were highly invested and putting the work in made a choice, so here we are. But if it’s in the PEPs, it seems time to define a version used ( 1.0 would be good, but often that’s actually pretty arbitrary) and get an implementation into the stdlib. If the TOML folks don’t think it’s stable enough for that, I’ve got to wonder if it was a good choice! We’re have enough trouble with really slow adoption of the new ways of doing packaging, not even providing the tools seems to really defeat the purpose. -CHB ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
On Wed, Oct 10, 2018 at 2:42 AM Stephen J. Turnbull wrote: > > Chris Angelico writes: > > On Tue, Oct 9, 2018 at 10:05 PM Greg Ewing > wrote: > > > > > > Chris Angelico wrote: > > > > In contrast, a mmap'd file is memory that you do indeed own. > > > > > > Although it's not really accurate to say that it's owned by > > > a particular process. If two processes mmap the same file, > > > the physical memory pages holding it appear in the address > > > spaces of both processes. > > Subject to COW, I presume. Probably in units smaller than the whole > file (per page?) Both processes are using the virtual memory. Either or both could be using physical memory. Assuming they haven't written to the pages (which is the case with executables - the system mmaps the binary into your memory space as read-only), and assuming that those pages are backed by physical memory, which process is using that memory? > > Tell me, which process is responsible for libc being in memory? > > Other than, like, all of them? > > Yes. Why would you want a different answer? Because that would mean that I have way more *physical* memory in use than I actually have chips on the motherboard for. Yes, virtual memory can be over-allocated, but physical memory can't. How do you measure where your physical memory is being used, if adding up the usage of all processes exceeds the available resources? It's like trying to figure out which countries the people of this planet live in, and saying that there are X billion in China, Y million in Australia, etc, etc, etc, and then discovering that you're counting people multiple times if they ever move... which means that there are ten or twenty billion people on the planet. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
Chris Angelico writes: > On Tue, Oct 9, 2018 at 10:05 PM Greg Ewing > wrote: > > > > Chris Angelico wrote: > > > In contrast, a mmap'd file is memory that you do indeed own. > > > > Although it's not really accurate to say that it's owned by > > a particular process. If two processes mmap the same file, > > the physical memory pages holding it appear in the address > > spaces of both processes. Subject to COW, I presume. Probably in units smaller than the whole file (per page?) > Tell me, which process is responsible for libc being in memory? > Other than, like, all of them? Yes. Why would you want a different answer? ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Introduce typing.SupportsFsPath
On Tue, Oct 9, 2018 at 3:16 AM Ivan Levkivskyi wrote: > class PathLike(Protocol[AnyStr]): > I had been working on this same problem intermittently for several months, so thanks, but... error: Invariant type variable 'AnyStr' used in protocol where covariant one is expected is called out on the class by mypy 0.630 (Python 3.6.6). Do I just need to wait for 0.640? Or should I define a new TypeVar for AnyStr_co and use that? ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
On Tue, Oct 9, 2018 at 10:05 PM Greg Ewing wrote: > > Chris Angelico wrote: > > In contrast, a mmap'd file is memory that you do indeed own. > > Although it's not really accurate to say that it's owned by > a particular process. If two processes mmap the same file, > the physical memory pages holding it appear in the address > spaces of both processes. > Yeah, which is a constant problem when you ask "why am I out of memory". Same thing happens if you have any other (non-mmap'd) pages and then you fork, or if two processes are using the same executable (though I think that's actually mmap again), etc, etc, etc. Tell me, which process is responsible for libc being in memory? Other than, like, all of them? ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Support parsing stream with `re`
Chris Angelico wrote: In contrast, a mmap'd file is memory that you do indeed own. Although it's not really accurate to say that it's owned by a particular process. If two processes mmap the same file, the physical memory pages holding it appear in the address spaces of both processes. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Introduce typing.SupportsFsPath
On Tue, 9 Oct 2018 at 03:13, Guido van Rossum wrote: > In typeshed there is os.PathLike which is close. You should be able to use > Union[str, os.PathLike[str]] for what you want (or define an alias). > > We generally don't want to add more things to typing that aren't closely > related to the type system. (Adding the io and re classes was already less > than ideal, and we don't want to do more of those.) > The problem however is that `PathLike` is not a protocol in typeshed. This should be updated when protocols will be official. Until that, you can just easily define your own protocol: from typing import AnyStr from typing_extensions import Protocol class PathLike(Protocol[AnyStr]): def __fspath__(self) -> AnyStr: ... -- Ivan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
On Mon, Oct 08, 2018 at 03:34:51PM -0400, Terry Reedy wrote: > >> I said above that functions may be specified by > >> process rather than result. > > > > Fine. What of it? Can you describe what the function does? > > > > "It sorts the list in place." > > > > "It deletes the given record from the database." > > > These are all post-conditions. > > No they are not. Yes they are; they are textual descriptions of the post-condition, not executable code, but since software contracts are both specification and code, they certainly count as post-conditions. By the way, you seem to have deleted one of my examples, leaving only the two relatively simple ones. I don't understand why you deleted the example, but here it is again: "It deducts the given amount from Account A and transfers it to Account B, guaranteeing that either both transactions occur or neither of them, but never one and not the other." > They are descriptions of the process. Additional > mental work is required to turn them into formal descriptions of the > result that can be coded. Yes. That's called "programming". You might have done it from time to time :-) > Marko appears to claim that such coded formal > descriptions are easier to read and understand than the short English > description. I disagree. What does "sort in place"? *Precisely*? How do you know if the software *actually does* sort in place? They are not rhetorical questions. You can document that the function sorts in place, but if it actually reverses a copy of the list, how would you know? How do we know the software meets the specified requirements if you haven't nailed down the specification, and if the specification exists only in documentation and is never checked in the code? Design By Contract is a methodology that helps ensure that the specification is (1) nailed down and (2) actually checked. > It is therefore not obvious to me that the > extra work is worthwhile. Tell me how you, personally, would find out whether or not the function actually sorts the list in place. What process would YOU do? As I see it, you either have to take it on trust, or you have to write tests. Would you argue that the "extra work" to write tests are not worthwhile? > >>def append_first(seq): > >> "Append seq[0] to seq." > >[...] > > The snipped body (revised to omit irrelevant 'return') > seq.append(seq[0]) > > >>But with duck-typing, no post condition is possible. > > > >That's incorrect. > > > >def append_first(seq): > > require: > > len(seq) > 0 > > seq does not neccessarily have a __len__ method Yes it must, because I wrote the specification and I demand that it must support len. You lost your opportunity to write the spec yourself when you wrongly said no post-condition is possible. I just showed that post-conditions ARE possible, by writing some. We could debate what those conditions are: can we just test for the Sequence interface, using isinstance() and the Sequence ABC? Do I need to bother checking that seq supports __getitem__? But I'm not really interested in getting bogged down in the minutia of what specific post-conditions need to go with this toy function. Its a made-up example, so who cares? In practice, you specify in as much detail as you need. You can always come back and add more detail later. > > hasattr(seq, "append") > > The actual precondition is that seq[0] be in the domain of seq.append. That is not part of the contract of the append_first function. It doesn't care about any domain limits of seq. If an object could be appended into seq[0] in the first place, surely it can append again later. We don't have to try to detect every imaginable bizarre weird piece of code intentionally written to perplex us. > The only absolutely sure way to test this is to run the code body. Sure. So what? The code body does actually get run you know. If it raises an exception because seq can't append seq[0] to itself, that's a bug that needs to be fixed. > Or > one could run seq[0] and check it against the preconditions, if formally > specified, of seq.append. That is not the responsibility of the append_first function. > > ensure: > > len(seq) == len(OLD.seq) + 1 > > seq[0] == seq[-1] > > Not even all sequences implement negative indexing. I don't care. Since I wrote the spec, I demand that only those which meet my requirements are used. But okay, fine, I'll re-write it: seq[0] == seq[len(seq)-1] and then you can tell me off for writing un-Pythonic code, and then I'll change it back. > This is true for lists, as I said, but not for every object the meets > the preconditions. As others have said, duck typing means that we don't > know what unexpected things methods of user-defined classes might do. That's not how we use duck-typing. When I call len(obj), I don't expect that it will format my hard drive, even though obj.__len__ might do that, and
