Re: [Python-Dev] Store timestamps as decimal.Decimal objects
> I think this is definitely worth elaborating in a PEP (to recap the
> long discussion in #11457 if nothing else).
The discussion in issues #13882 and #11457 already lists many
alternatives with their costs and benefits, but I can produce a PEP if
you need a summary.
> In particular, I'd want to
> see a very strong case being made for supporting multiple formats over
> standardising on a *single* new higher precision format (for example,
> using decimal.Decimal in conjunction with integration of Stefan's
> cdecimal work) that can then be converted to other formats (like
> datetime) via the appropriate APIs.
To convert a Decimal to a datetime object, we have already the
datetime.datetime.fromtimestamp() function (it converts Decimal to
float, but the function can be improved without touching its API). But
I like the possibility of getting the file modification time directly
as a datetime object to have something like:
>>> s=os.stat("setup.py", timestamp="datetime")
>>> print(s.st_atime - s.st_ctime)
52 days, 1:44:06.191293
We have already more than one timestamp format: os.stat() uses int or
float depending on os.stat_float_times() value. In 5 years, we may
prefer to use directly float128 instead of Decimal. I prefer to have
an extensible API to prepare future needs, even if we just add Decimal
today.
Hum, by the way, we need a "int" format for os.stat(), so
os.stat_float_times() can be deprecated. So there will be a minimum of
3 types:
- int
- float
- decimal.Decimal
Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
On Tue, Jan 31, 2012 at 7:42 PM, Victor Stinner wrote: >> I think this is definitely worth elaborating in a PEP (to recap the >> long discussion in #11457 if nothing else). > > The discussion in issues #13882 and #11457 already lists many > alternatives with their costs and benefits, but I can produce a PEP if > you need a summary. PEPs are about more than just providing a summary - they're about presenting the alternatives in a clear form instead of having them scattered across a long meandering tracker discussion. Laying out the alternatives and clearly articulating their pros and cons (as Larry attempted to do on the tracker) *helps to make better decisions*. I counted several options presented as possibilities and I probably missed some: - expose the raw POSIX (seconds, nanoseconds) 2-tuples (lots of good reasons not to go that way) - use decimal.Decimal (with or without cdecimal) - use float128 (nixed due to cross-platform supportability problems) - use datetime (bad idea for the reasons Martin mentioned) - use timedelta (not mentioned on the tracker, but a *much* better fit for a timestamp than datetime, since timestamps are relative to the epoch while datetime objects try to be absolute) A PEP would also allow the following items to be specifically addressed: - a survey of what other languages are doing to cope with nanosecond time resolutions (as suggested by Raymond but not actually done as far I could see on the tracker) - how to avoid a negative performance impact on os.stat() (new API? flag argument? new lazily populated attributes accessed by name only?) Guido's admonition against analysis paralysis doesn't mean we should go to the other extreme and skip clearly documenting our analysis of complex problems altogether (particularly for something like this which may end up having ramifications for a lot of other time related code). Having a low-level module like os needing to know about higher-level types like decimal.Decimal and datetime.datetime (or even timedelta) should be setting off all kinds of warning bells. Of all the possibilties that offer decent arithmetic support, timedelta is probably the one currently most suited to being pushed down to the os level, although decimal.Decimal is also a contender if backed up by Stefan's C implementation. You're right that supporting this does mean being able to at least select between 'int', 'float' and output, but that's the kind of case that can be made most clearly in a PEP. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] cdecimal (Was: Store timestamps as decimal.Decimal objects)
On 31 January 2012 11:11, Nick Coghlan wrote: > although decimal.Decimal is also a contender if backed up by > Stefan's C implementation. As you mention this, and given the ongoing thread about __preview__ and "nearly ready for stdlib" modules, what is the current position on cdecimal? I seem to recall it being announced some time ago, but I don't recall any particular discussions/conclusions about including it in the stdlib. Is it being considered for stdlib inclusion? What obstacles remain before inclusion (clearly not many, if it's being seriously considered as an option to support functions in something as fundamental as os)? Do Guido's comments on the __preview__ thread make any difference here? (Note - I don't have any particular *need* for cdecimal, I'm just curious...) Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
Hi,
2012/1/31 Matt Joiner :
> Sounds good, but I also prefer Alexander's method. The type information is
> already encoded in the class object.
Ok, I posted a patch version 6 to use types instead of strings. I also
prefer types because it solves the "hidden import" issue.
> This way you don't need to maintain a
> mapping of strings to classes, and other functions/third party can join in
> the fun without needing access to the latest canonical mapping. Lastly there
> will be no confusion or contention for duplicate keys.
My patch checks isinstance(format, type), format.__module__ and
format.__name__ to do the "mapping". It is not a direct mapping
because I don't always call the same method, the implementation is
completly differenet for each type.
I don't think that we need user defined timestamp formats. My last
patch provides 5 formats:
- int
- float
- decimal.Decimal
- datetime.datetime
- datetime.timedelta
(I removed the timespec format, I consider that we don't need it.)
Examples:
>>> time.time()
1328006975.681211
>>> time.time(format=int)
1328006979
>>> time.time(format=decimal.Decimal)
Decimal('1328006983.761119')
>>> time.time(format=datetime.datetime)
datetime.datetime(2012, 1, 31, 11, 49, 49, 409831)
>>> print(time.time(format=datetime.timedelta))
15370 days, 10:49:52.842116
If someone wants another format, he/she should pick up an existing
format to build his/her own format.
datetime.datetime and datetime.timedelta can be used on any function,
but datetime.datetime format gives surprising results on clocks using
an arbitrary start like time.clock() or time.wallclock(). We may raise
an error in these cases.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
On Tue, 31 Jan 2012 21:11:37 +1000 Nick Coghlan wrote: > > Having a low-level module like os needing to know about higher-level > types like decimal.Decimal and datetime.datetime (or even timedelta) > should be setting off all kinds of warning bells. Decimal is ideally low-level (it's a number), it's just that it has a complicated high-level implementation :) But we can't use Decimal by default, for the obvious reason (performance impact that threatens to contaminate other parts of the code through operator application). > Of all the > possibilties that offer decent arithmetic support, timedelta is > probably the one currently most suited to being pushed down to the os > level, although decimal.Decimal is also a contender if backed up by > Stefan's C implementation. I'm -1 on using timedelta. This is a purity proposition that will make no sense to the average user. By the way, datetimes are relative too, by the same reasoning. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
> - use datetime (bad idea for the reasons Martin mentioned) It is only a bad idea if it is the only available choice. > - use timedelta (not mentioned on the tracker, but a *much* better fit > for a timestamp than datetime, since timestamps are relative to the > epoch while datetime objects try to be absolute) Last version of my patch supports also timedelta. > - a survey of what other languages are doing to cope with nanosecond > time resolutions (as suggested by Raymond but not actually done as far > I could see on the tracker) I didn't check that right now. I don't know if it is really revelant because some languages don't have a builtin Decimal class or no "builtin" datetime module. > - how to avoid a negative performance impact on os.stat() (new API? > flag argument? new lazily populated attributes accessed by name only?) Because timestamp is an optional argument to os.stat() and the behaviour is unchanged by default, the performance impact of my patch on os.stat() is null (if you don't set timestamp argument). > Having a low-level module like os needing to know about higher-level > types like decimal.Decimal and datetime.datetime (or even timedelta) > should be setting off all kinds of warning bells. What is the problem of using decimal in the os module? Especially if it is an option. In my patch version 6, the timestamp argument is now a type (e.g. decimal.Decimal) instead of a string, so the os module doesn't import directly the module (well, to be exact, it does import the module, but the module should already be in the cache, sys.modules). > You're right that supporting this does mean being able to at least > select between 'int', 'float' and output, but that's > the kind of case that can be made most clearly in a PEP. Why do you want to limit the available formats? Why not giving the choice to the user between Decimal, datetime and timedelta? Each type has a different use case and different features, sometimes exclusive. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEPs and cons (was: Re: Store timestamps as decimal.Decimal objects)
Nick Coghlan, 31.01.2012 12:11: > On Tue, Jan 31, 2012 at 7:42 PM, Victor Stinner wrote: >>> I think this is definitely worth elaborating in a PEP (to recap the >>> long discussion in #11457 if nothing else). >> >> The discussion in issues #13882 and #11457 already lists many >> alternatives with their costs and benefits, but I can produce a PEP if >> you need a summary. > > PEPs are about more than just providing a summary - they're about > presenting the alternatives in a clear form instead of having them > scattered across a long meandering tracker discussion. There was a keynote by Jan Lehnardt (of CouchDB fame) on last year's PyCon-DE on the end of language wars and why we should just give each other a hug and get along and all that. To seed some better understanding, he had come up with mottoes for the Ruby and Python language communities, which find themselves in continuous quarrel. I remember the motto for Python being "you do it right - and you document it". A clear hit IMHO. Decisions about language changes and environmental changes (such as the stdlib) aren't easily taken in the Python world, but when they are taken, they tend to show a good amount of well reflected common sense, and we make it transparent how they come to be by writing a PEP about them, so that we (and others) can go back and read them up later on when they are being questioned again or when similar problems appear in other languages. That's a good thing, and we should keep that up. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 optimizations, continued, continued again...
> I assume "yes" here means "yes, I'm aware" and not "yes, I'm using Python > 2", right? And you're building on top of the existing support for threaded > code in order to improve it? > Your assumption is correct, I'm sorry for the sloppiness (I was heading out for lunch.) None of the code is 2.x compatible, all of my work has always targeted Python 3.x. My work does not improve threaded code (as in interpreter dispatch technique), but enables efficient and purely interpretative inline caching via quickening. (So, after execution of BINARY_ADD, I rewrite the specific occurence of the bytecode instruction to a, say, FLOAT_ADD instruction and ensure that my assumption is correct in the FLOAT_ADD instruction.) Thanks, --stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 optimizations, continued, continued again...
> If I read the patch correctly, most of it is auto-generated (and there > is probably a few spurious changes that blow it up, such as the > python-gdb.py file). Hm, honestly I don't know where the python-gdb.py file comes from, I thought it came with the switch from 3.1 to the tip version I was using. Anyways, I did not tuch it or at least have no recollection of doing so. Regarding the spurious changes: This might very well be, regression testing works, and it would actually be fairly easy to figure out crashes (e.g., by tracing all executed bytecode instructions and seeing if all of them are actually executed, I could easily do that if wanted/necessary.) > But the tool that actually generates the code > doesn't seem to be included? (Which means that in this form, the > patch couldn't possibly be accepted.) > Well, the tool is not included because it does a lot more (e.g., generate the code for elimination of reference count operations.) Unfortunately, my interpreter architecture that achieves the highest speedups is more complicated, and I got the feeling that this is not going well with python-dev. So, I had the idea of basically using just one (but a major one) optimization technique and going with that. I don't see why you would need my code generator, though. Not that I cared, but I would need to strip down and remove many parts of it and also make it more accessible to other people. However, if python-dev decides that it wants to include the optimizations and requires the code generator, I'll happily chip in the extra work an give you the corresponding code generator, too. Thanks, --stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cdecimal (Was: Store timestamps as decimal.Decimal objects)
On Tue, Jan 31, 2012 at 06:47, Paul Moore wrote: > On 31 January 2012 11:11, Nick Coghlan wrote: > > although decimal.Decimal is also a contender if backed up by > > Stefan's C implementation. > > As you mention this, and given the ongoing thread about __preview__ > and "nearly ready for stdlib" modules, what is the current position on > cdecimal? I seem to recall it being announced some time ago, but I > don't recall any particular discussions/conclusions about including it > in the stdlib. > > Is it being considered for stdlib inclusion? What obstacles remain > before inclusion (clearly not many, if it's being seriously considered > as an option to support functions in something as fundamental as os)? > Do Guido's comments on the __preview__ thread make any difference > here? > > (Note - I don't have any particular *need* for cdecimal, I'm just > curious...) > Because cdecimal is just an accelerated version of decimal there is no specific stdlib restriction from it going in. At this point I think it just needs to be finished and then committed. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading.Semaphore()'s counter can become negative for non-ints
On 2012-01-31 00:23, Benjamin Peterson wrote: 2012/1/30 Nick Coghlan: On Tue, Jan 31, 2012 at 8:11 AM, Matt Joiner wrote: It's also potentially lossy if you incremented and decremented until integer precision is lost. My vote is for an int type check. No casting. operator.index() is built for that purpose (it's what we use these days to restrict slicing to integers). +1 for the type restriction from me. We don't need a type check. Just pass integers (obviously the only right type) to it. When a float is used, think of debugging such a thing, e.g. a float from integer division. I don't care if float (or generally non-integers) are not allowed in threading.Semaphore, but please make it fail with a bang. Regards, TB ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
On Mon, Jan 30, 2012 at 6:31 PM, Victor Stinner wrote: > Alexander Belopolsky proposed to use > time.time(format=datetime.datetime) instead. Just to make sure my view is fully expressed: I am against adding flag arguments to time.time(). My preferred solution to exposing high resolution clocks is to do it in a separate module. You can even call the new function time() and access it as hirestime.time(). Longer names that reflect various time representation are also an option: hirestime.decimal_time(), hirestime.datetime_time() etc. The suggestion to use the actual type as a flag was motivated by the desire to require module import before fancy time.time() can be called. When you care about nanoseconds in your time stamps you won't tolerate an I/O delay between calling time() and getting the result. A separate module can solve this issue much better: simply import decimal or datetime or both at the top of the module. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
On Tue, Jan 31, 2012 at 7:13 AM, Antoine Pitrou wrote: > On Tue, 31 Jan 2012 21:11:37 +1000 > Nick Coghlan wrote: >> >> Having a low-level module like os needing to know about higher-level >> types like decimal.Decimal and datetime.datetime (or even timedelta) >> should be setting off all kinds of warning bells. > > Decimal is ideally low-level (it's a number), it's just that it has a > complicated high-level implementation :) FWIW, my vote is also for Decimal and against datetime or timedelta. (I dream of Decimal replacing float in Python 4000, so take my vote with an appropriate amount of salt. :-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading.Semaphore()'s counter can become negative for non-ints
On Jan 29, 2012, at 6:11 PM, John O'Connor wrote: > On Sat, Jan 28, 2012 at 3:07 PM, Benjamin Peterson > wrote: >> But why would you want to pass a float? It seems like API abuse to me. >> > > Agreed. Anything else seems meaningless. I concur. This is very much a non-problem. There is no need to add more code and slow running time with superfluous type checks. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
Am 31.01.2012 13:08, schrieb Victor Stinner: >> This way you don't need to maintain a >> mapping of strings to classes, and other functions/third party can join in >> the fun without needing access to the latest canonical mapping. Lastly there >> will be no confusion or contention for duplicate keys. > > My patch checks isinstance(format, type), format.__module__ and > format.__name__ to do the "mapping". It is not a direct mapping > because I don't always call the same method, the implementation is > completly differenet for each type. > > I don't think that we need user defined timestamp formats. My last > patch provides 5 formats: > > - int > - float > - decimal.Decimal > - datetime.datetime > - datetime.timedelta > > (I removed the timespec format, I consider that we don't need it.) Rather, I guess you removed it because it didn't fit the "types as flags" pattern. As I said in another message, another hint that this is the wrong API design: Will the APIs ever support passing in types other than these five? Probably not, so I strongly believe they should not be passed in as types. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 optimizations, continued, continued again...
Am 31.01.2012 16:46, schrieb stefan brunthaler: >> If I read the patch correctly, most of it is auto-generated (and there >> is probably a few spurious changes that blow it up, such as the >> python-gdb.py file). > > Hm, honestly I don't know where the python-gdb.py file comes from, I > thought it came with the switch from 3.1 to the tip version I was > using. Anyways, I did not tuch it or at least have no recollection of > doing so. Regarding the spurious changes: This might very well be, > regression testing works, and it would actually be fairly easy to > figure out crashes (e.g., by tracing all executed bytecode > instructions and seeing if all of them are actually executed, I could > easily do that if wanted/necessary.) There is also the issue of the two test modules removed from the test suite. >> But the tool that actually generates the code >> doesn't seem to be included? (Which means that in this form, the >> patch couldn't possibly be accepted.) >> > Well, the tool is not included because it does a lot more (e.g., > generate the code for elimination of reference count operations.) > Unfortunately, my interpreter architecture that achieves the highest > speedups is more complicated, and I got the feeling that this is not > going well with python-dev. So, I had the idea of basically using just > one (but a major one) optimization technique and going with that. I > don't see why you would need my code generator, though. Not that I > cared, but I would need to strip down and remove many parts of it and > also make it more accessible to other people. However, if python-dev > decides that it wants to include the optimizations and requires the > code generator, I'll happily chip in the extra work an give you the > corresponding code generator, too. Well, nobody wants to review generated code. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 optimizations, continued, continued again...
> There is also the issue of the two test modules removed from the > test suite. > Oh, I'm sorry, seems like the patch did contain too much of my development stuff. (I did remove them before, because they were always failing due to the instruction opcodes being changed because of quickening; they pass the tests, though.) > Well, nobody wants to review generated code. > I agree. The code generator basically uses templates that contain the information and a dump of the C-structure of several types to traverse and see which one of them implements which functions. There is really no magic there, the most "complex" thing is to get the inline-cache miss checks for function calls right. But I tried to make the generated code look pretty, so that working with it is not too much of a hassle. The code generator itself is a little bit more complicated, so I am not sure it would help a lot... best, --stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
> (I removed the timespec format, I consider that we don't need it.) > > Rather, I guess you removed it because it didn't fit the "types as flags" > pattern. I removed it because I don't like tuple: you cannot do arithmetic on tuple, like t2-t1. Print a tuple doesn't give you a nice output. It is used in C because you have no other choice, but in Python, we can do better. > As I said in another message, another hint that this is the wrong API design: > Will the APIs ever support passing in types other than these five? Probably > not, so I strongly believe they should not be passed in as types. I don't know if we should only support 3 types today, or more, but I suppose that we will add more later (e.g. if datetime is replaced by another new and better datetime module). You mean that we should use a string instead of type, so time.time(format="decimal")? Or do something else? Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading.Semaphore()'s counter can become negative for non-ints
I concur. This is very much a non-problem.
There is no need to add more code and slow
running time with superfluous type checks.
Raymond
What do you think about the following check from threading.py:
@@ -317,8 +317,6 @@
self._value = value
def acquire(self, blocking=True, timeout=None):
-if not blocking and timeout is not None:
-raise ValueError("can't specify timeout for non-blocking
acquire")
rc = False
(There are similar checks in Modules/_threadmodule.c)
Removing the check means that we ignore the timeout argument when
blocking=False. Currently in the multiprocessing docs there is an
outdated note concerning acquire() methods that also says: "If block is
False then timeout is ignored". This makes the acquire() methods of the
threading and multiprocessing modules have different behaviors.
Related: http://bugs.python.org/issue850728#msg103227
TB
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading.Semaphore()'s counter can become negative for non-ints
On 1/31/2012 3:10 PM, Raymond Hettinger wrote: On Jan 29, 2012, at 6:11 PM, John O'Connor wrote: On Sat, Jan 28, 2012 at 3:07 PM, Benjamin Peterson mailto:[email protected]>> wrote: But why would you want to pass a float? It seems like API abuse to me. Agreed. Anything else seems meaningless. I concur. This is very much a non-problem. There is no need to add more code and slow running time with superfluous type checks. If it does not now, the doc could be changed to say that the arg must be an int, and behavior is undefined otherwise. Then the contract is clear. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
Nick mentioned using a single type and converting upon return, I'm starting to like that more. A limited set of time formats is mostly arbitrary, and there will always be a performance hit deciding which type to return. The goal here is to allow high precision timings with minimal cost. A separate module, and an agreement on what the best performing high precision type is I think is the best way forward. On Feb 1, 2012 8:47 AM, "Victor Stinner" wrote: > > (I removed the timespec format, I consider that we don't need it.) > > > > Rather, I guess you removed it because it didn't fit the "types as flags" > > pattern. > > I removed it because I don't like tuple: you cannot do arithmetic on > tuple, like t2-t1. Print a tuple doesn't give you a nice output. It is > used in C because you have no other choice, but in Python, we can do > better. > > > As I said in another message, another hint that this is the wrong API > design: > > Will the APIs ever support passing in types other than these five? > Probably > > not, so I strongly believe they should not be passed in as types. > > I don't know if we should only support 3 types today, or more, but I > suppose that we will add more later (e.g. if datetime is replaced by > another new and better datetime module). > > You mean that we should use a string instead of type, so > time.time(format="decimal")? Or do something else? > > Victor > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
Alexander Belopolsky wrote: On Tue, Jan 31, 2012 at 7:13 AM, Antoine Pitrou wrote: On Tue, 31 Jan 2012 21:11:37 +1000 Nick Coghlan wrote: Having a low-level module like os needing to know about higher-level types like decimal.Decimal and datetime.datetime (or even timedelta) should be setting off all kinds of warning bells. Decimal is ideally low-level (it's a number), it's just that it has a complicated high-level implementation :) FWIW, my vote is also for Decimal and against datetime or timedelta. (I dream of Decimal replacing float in Python 4000, so take my vote with an appropriate amount of salt. :-) Why not add a new function rather than modifying time.time()? (after all its just a timestamp, does it really need nanosecond precision?) For those who do want super-accuracy then add a new function time.picotime() (it could be nanotime but why not future proof it :) ) which returns an int represent the number of picoseconds since the epoch. ints never loose precision and never overflow. Cheers, Mark. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
On Wed, Feb 1, 2012 at 8:58 AM, Mark Shannon wrote:
> Why not add a new function rather than modifying time.time()?
> (after all its just a timestamp, does it really need nanosecond precision?)
>
> For those who do want super-accuracy then add a new function
> time.picotime() (it could be nanotime but why not future proof it :) )
> which returns an int represent the number of picoseconds since the
> epoch. ints never loose precision and never overflow.
Because the problem is broader than that - it affects os.stat(), too,
along with a number of the other time module APIs that produce
timestamp values.
That's where Alexander's suggestion of a separate "hirestime" module
comes in - it would be based on the concept of *always* using a high
precision type in the API (probably decimal.Decimal()). Conceptually,
it's a very clean approach, and obviously has zero performance impact
on existing APIs, but the idea of adding
yet-another-time-related-module to the standard library is rather
questionable. Such an approach is also likely to lead to a lot of
duplicated code.
Victor's current approach, unfortunately, is a bit of a
"worst-of-both-worlds" approach. It couples the time and os modules to
various other currently unrelated modules (such as datetime and
decimal), but still doesn't provide a particularly extensible API
(whether indicated by flags or strings, each new supported output type
must be special cased in time and os).
Perhaps more fruitful would be to revisit the original idea from the
tracker of defining a conversion function protocol for timestamps
using some basic fixed point arithmetic. The objection to using a
conversion function that accepts a POSIX-style seconds+nanoseconds
timespec is that it isn't future-proof - what if at some point in the
future, nanonsecond resolution is considered inadequate?
The secret to future-proofing such an API while only using integers
lies in making the decimal exponent part of the conversion function
signature:
def from_components(integer, fraction=0, exponent=-9):
return Decimal(integer) + Decimal(fraction) * Decimal((0,
(1,), exponent))
>>> from_components(100)
Decimal('100.0')
>>> from_components(100, 100)
Decimal('100.00100')
>>> from_components(100, 100)
Decimal('100.00100')
>>> from_components(100, 100, -12)
Decimal('100.0100')
Such a protocol can easily be extended to any other type - the time
module could provide conversion functions for integers and float
objects (meaning results may have lower precision than the underlying
system calls), while the existing "fromtimestamp" APIs in datetime can
be updated to accept the new optional arguments (and perhaps an
appropriate class method added to timedelta, too). A class method
could also be added to the decimal module to construct instances from
integer components (as shown above), since that method of construction
isn't actually specific to timestamps.
With this approach, API usage might end up looking something like:
>>> time.time()
1328006975.681211
>>> time.time(convert=time.as_float)
1328006975.681211
>>> time.time(convert=time.as_int)
1328006979
>>> time.time(convert=time.as_tuple)
(1328006975, 681211, -9)
>>> time.time(convert=decimal.Decimal.from_components)
Decimal('1328006983.761119000')
>>> time.time(convert=datetime.datetime.fromtimestamp)
datetime.datetime(2012, 1, 31, 11, 49, 49, 409831)
>>> time.time(convert=datetime.datetime.utcfromtimestamp)
datetime.datetime(2012, 1, 31, 11, 49, 49, 409831)
>>> time.time(convert=datetime.date.fromtimestamp)
datetime.date(2012, 1, 31)
>>> print(time.time(convert=datetime.timedelta.fromtimestamp))
15370 days, 10:49:52.842116
This strategy would have negligible performance impact in already
supported cases (just an extra check to determine that no callback was
provided), and offer a very simple, yet fully general and
future-proof, integer based callback protocol when you want your
timestamps in a different format.
Regards,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
On Wed, 1 Feb 2012 10:35:08 +1000
Nick Coghlan wrote:
>
> With this approach, API usage might end up looking something like:
>
>>>> time.time()
>1328006975.681211
>>>> time.time(convert=time.as_float)
>1328006975.681211
>>>> time.time(convert=time.as_int)
>1328006979
>>>> time.time(convert=time.as_tuple)
>(1328006975, 681211, -9)
>>>> time.time(convert=decimal.Decimal.from_components)
>Decimal('1328006983.761119000')
It strikes me as inelegant to have to do so much typing for something
as simple as getting the current time. We should approach the
simplicity of ``time.time(format='decimal')`` or
``time.decimal_time()``.
(and I think the callback thing is overkill)
Regards
Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
On Tue, Jan 31, 2012 at 7:35 PM, Nick Coghlan wrote: > Such a protocol can easily be extended to any other type - the time > module could provide conversion functions for integers and float > objects (meaning results may have lower precision than the underlying > system calls), while the existing "fromtimestamp" APIs in datetime can > be updated to accept the new optional arguments (and perhaps an > appropriate class method added to timedelta, too). A class method > could also be added to the decimal module to construct instances from > integer components (as shown above), since that method of construction > isn't actually specific to timestamps. > Why not just make it something like __fromfixed__() and make it a standard protocol, implemented on floats, ints, decimals, etc. Then the API is just "time.time(type)", where type is any object providing a __fromfixed__ method. ;-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
Analysis paralysis commence. +1 for separate module using decimal. On Feb 1, 2012 1:44 PM, "PJ Eby" wrote: > On Tue, Jan 31, 2012 at 7:35 PM, Nick Coghlan wrote: > >> Such a protocol can easily be extended to any other type - the time >> module could provide conversion functions for integers and float >> objects (meaning results may have lower precision than the underlying >> system calls), while the existing "fromtimestamp" APIs in datetime can >> be updated to accept the new optional arguments (and perhaps an >> appropriate class method added to timedelta, too). A class method >> could also be added to the decimal module to construct instances from >> integer components (as shown above), since that method of construction >> isn't actually specific to timestamps. >> > > Why not just make it something like __fromfixed__() and make it a standard > protocol, implemented on floats, ints, decimals, etc. Then the API is just > "time.time(type)", where type is any object providing a __fromfixed__ > method. ;-) > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store timestamps as decimal.Decimal objects
On Wed, Feb 1, 2012 at 12:35 PM, Antoine Pitrou wrote: > It strikes me as inelegant to have to do so much typing for something > as simple as getting the current time. We should approach the > simplicity of ``time.time(format='decimal')`` or > ``time.decimal_time()``. Getting the current time is simple (you can already do it), getting access to high precision time without performance regressions or backwards incompatiblities or excessive code duplication is hard. There's a very simple rule in large scale software development: coupling is bad and you should do everything you can to minimise it. Victor's approach throws that out the window by requiring that time and os know about every possible output format for time values. That's why protocols are so valuable: instead of having MxN points of interconnection, you just define a standard protocol as the basis for interaction, and the consumer of the protocol doesn't need to care about the details of the provider, they just care about the protocol itself. So, the question becomes how to solve the problem of exposing high resolution timestamps to Python code in a way that: - is applicable not just to time.time(), but also to os.stat(), time.clock(), time.wall_clock() and any other timestamp sources I've forgotten. - is backwards compatible for all those use cases - doesn't cause a significant performance regression for any of those use cases - doesn't cause excessive coupling between the time and os modules and other parts of Python - doesn't excessively duplicate code - doesn't add too much machinery for a relatively minor problem The one key aspect that I think Victor's suggestion gets right is that we want a way to request high precision time from the *existing* APIs, and that this needs to be selected on a per call basis rather than globally for the whole application. The big advantage of going with a callback based approach is that it gives you flexibility and low coupling without any additional supporting infrastructure, and you have the full suite of Python tools available to deal with any resulting verbosity issues. For example, it would become *trivial* to write Alexander's suggested "hirestime" module that always returned decimal.Decimal objects: _hires = decimal.Decimal.from_components def time(): return time.time(convert=_hires) def clock(): return time.clock(convert=_hires) def stat(path): return os.stat(path, timestamps=_hires) # etc... PJE is quite right that using a new named protocol rather than a callback with a particular signature could also work, but I don't see a lot of advantages in doing so. On the other hand, if you go with the "named output format", "hires=True" or new API approaches, you end up having to decide what additional coupling you're going to introduce to time and os. Now, in this case, I actually think there *is* a reasonable option available if we decide to go down that path: - incorporate Stefan Krah's cdecimal work into the standard library - add a "hires=False" flag to affected APIs - return a Decimal instance with full available precision if "hires=True" is passed in. - make time and os explicitly depend on the ability to create decimal.Decimal instances A hirestime module is even easier to implement in that case: def time(): return time.time(hires=True) def clock(): return time.clock(hires=True) def stat(path): return os.stat(path, hires=True) # etc... All of the other APIs (datetime, timedelta, etc) can then just be updated to also accept a Decimal object as input, rather than handling the (integer, fraction, exponent) callback signature I suggested. Either extreme (full flexibility via a callback API or protocol, or else settling specifically on decimal.Decimal and explicitly making time and os dependent on that type) makes sense to me. A wishy-washy middle ground that introduces a dependency from time and os onto multiple other modules *without* making the API user extensible doesn't seem reasonable at all. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 409 - final?
I haven't seen any further discussion here or in the bug tracker. Below is the latest version of this PEP, now with a section on Language Details. Who makes the final call on this? Any idea how long that will take? (Not that I'm antsy, or anything... ;) PEP: 409 Title: Suppressing exception context Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 26-Jan-2012 Post-History: 30-Aug-2002, 01-Feb-2012 Abstract One of the open issues from PEP 3134 is suppressing context: currently there is no way to do it. This PEP proposes one. Rationale = There are two basic ways to generate exceptions: 1) Python does it (buggy code, missing resources, ending loops, etc.) 2) manually (with a raise statement) When writing libraries, or even just custom classes, it can become necessary to raise exceptions; moreover it can be useful, even necessary, to change from one exception to another. To take an example from my dbf module:: try: value = int(value) except Exception: raise DbfError(...) Whatever the original exception was (``ValueError``, ``TypeError``, or something else) is irrelevant. The exception from this point on is a ``DbfError``, and the original exception is of no value. However, if this exception is printed, we would currently see both. Alternatives Several possibilities have been put forth: * ``raise as NewException()`` Reuses the ``as`` keyword; can be confusing since we are not really reraising the originating exception * ``raise NewException() from None`` Follows existing syntax of explicitly declaring the originating exception * ``exc = NewException(); exc.__context__ = None; raise exc`` Very verbose way of the previous method * ``raise NewException.no_context(...)`` Make context suppression a class method. All of the above options will require changes to the core. Proposal I proprose going with the second option:: raise NewException from None It has the advantage of using the existing pattern of explicitly setting the cause:: raise KeyError() from NameError() but because the 'cause' is ``None`` the previous context, while retained, is not displayed by the default exception printing routines. Language Details Currently, ``__context__`` and ``__cause__`` start out as None, and then get set as exceptions occur. To support ``from None``, ``__context__`` will stay as it is, but ``__cause__`` will start out as ``False``, and will change to ``None`` when the ``raise ... from None`` method is used. The default exception printing routine will then: * If ``__cause__`` is ``False`` the ``__context__`` (if any) will be printed. * If ``__cause__`` is ``None`` the ``__context__`` will not be printed. * if ``__cause__`` is anything else, ``__cause__`` will be printed. This has the benefit of leaving the ``__context__`` intact for future logging, querying, etc., while suppressing its display if it is not caught. This is important for those times when trying to debug poorly written libraries with `bad error messages`_. Patches === There is a patch for CPython implementing this attached to `Issue 6210`_. References == Discussion and refinements in this `thread on python-dev`_. .. _bad error messages: http://bugs.python.org/msg152294 .. _Issue 6210: http://bugs.python.org/issue6210 .. _thread on python-dev: http://mail.python.org/pipermail/python-dev/2012-January/115838.html Copyright = This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 409 - now properly formatted (sorry for the noise)
PEP: 409 Title: Suppressing exception context Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 26-Jan-2012 Post-History: 30-Aug-2002, 01-Feb-2012 Abstract One of the open issues from PEP 3134 is suppressing context: currently there is no way to do it. This PEP proposes one. Rationale = There are two basic ways to generate exceptions: 1) Python does it (buggy code, missing resources, ending loops, etc.) 2) manually (with a raise statement) When writing libraries, or even just custom classes, it can become necessary to raise exceptions; moreover it can be useful, even necessary, to change from one exception to another. To take an example from my dbf module:: try: value = int(value) except Exception: raise DbfError(...) Whatever the original exception was (``ValueError``, ``TypeError``, or something else) is irrelevant. The exception from this point on is a ``DbfError``, and the original exception is of no value. However, if this exception is printed, we would currently see both. Alternatives Several possibilities have been put forth: * ``raise as NewException()`` Reuses the ``as`` keyword; can be confusing since we are not really reraising the originating exception * ``raise NewException() from None`` Follows existing syntax of explicitly declaring the originating exception * ``exc = NewException(); exc.__context__ = None; raise exc`` Very verbose way of the previous method * ``raise NewException.no_context(...)`` Make context suppression a class method. All of the above options will require changes to the core. Proposal I proprose going with the second option:: raise NewException from None It has the advantage of using the existing pattern of explicitly setting the cause:: raise KeyError() from NameError() but because the 'cause' is ``None`` the previous context, while retained, is not displayed by the default exception printing routines. Language Details Currently, ``__context__`` and ``__cause__`` start out as None, and then get set as exceptions occur. To support ``from None``, ``__context__`` will stay as it is, but ``__cause__`` will start out as ``False``, and will change to ``None`` when the ``raise ... from None`` method is used. The default exception printing routine will then: * If ``__cause__`` is ``False`` the ``__context__`` (if any) will be printed. * If ``__cause__`` is ``None`` the ``__context__`` will not be printed. * if ``__cause__`` is anything else, ``__cause__`` will be printed. This has the benefit of leaving the ``__context__`` intact for future logging, querying, etc., while suppressing its display if it is not caught. This is important for those times when trying to debug poorly written libraries with `bad error messages`_. Patches === There is a patch for CPython implementing this attached to `Issue 6210`_. References == Discussion and refinements in this `thread on python-dev`_. .. _bad error messages: http://bugs.python.org/msg152294 .. _Issue 6210: http://bugs.python.org/issue6210 .. _thread on python-dev: http://mail.python.org/pipermail/python-dev/2012-January/115838.html Copyright = This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 409 - final?
On Wed, Feb 1, 2012 at 1:57 PM, Ethan Furman wrote: > I haven't seen any further discussion here or in the bug tracker. Below is > the latest version of this PEP, now with a section on Language Details. > > Who makes the final call on this? Any idea how long that will take? (Not > that I'm antsy, or anything... ;) Guido still has the final say on PEP approvals as BDFL - it's just that sometimes he'll tap someone else and say "Your call!" (thus making them a BDFOP - Benevolent Dictator for One PEP). FWIW, I'm personally +1 on the latest version of this. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] docs fixes and PEP 409
I'm looking at the docs to make the relevant changes due to PEP 409, and I'm noticing some problems. E.g. The PyException_Get|Set_Context|Cause all talk about using NULL to clear the related attribute, when actually in should be Py_None. Only PyException_GetCause is directly related to PEP 409 -- should I only fix that one, and open up a new issue on the tracker for the other three, or should I fix all four now? ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] docs fixes and PEP 409
On Wed, Feb 1, 2012 at 3:07 PM, Ethan Furman wrote: > I'm looking at the docs to make the relevant changes due to PEP 409, and I'm > noticing some problems. > > E.g. The PyException_Get|Set_Context|Cause all talk about using NULL to > clear the related attribute, when actually in should be Py_None. > > Only PyException_GetCause is directly related to PEP 409 -- should I only > fix that one, and open up a new issue on the tracker for the other three, or > should I fix all four now? Passing in NULL is the right way to clear them using those APIs - the descriptors in exceptions.c then control how "not set" is exposed at the Python layer. So only Get/SetCause should need updating for PEP 409 to say to pass in NULL to clear the cause and fall back on displaying the context and Py_None to suppress the context in the default display. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] docs fixes and PEP 409
Ethan Furman wrote: Only PyException_GetCause is directly related to PEP 409 -- should I only fix that one, and open up a new issue on the tracker for the other three, or should I fix all four now? The specific question is now irrelevant (still learning the differences between the C code and the Python code ;) -- but the general question remains... ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 409 - now properly formatted (sorry for the noise)
What an appropriate title since I sent it to the wrong place. :( ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 optimizations, continued, continued again...
stefan brunthaler, 31.01.2012 22:17: >> Well, nobody wants to review generated code. >> > I agree. The code generator basically uses templates that contain the > information and a dump of the C-structure of several types to traverse > and see which one of them implements which functions. There is really > no magic there, the most "complex" thing is to get the inline-cache > miss checks for function calls right. But I tried to make the > generated code look pretty, so that working with it is not too much of > a hassle. The code generator itself is a little bit more complicated, > so I am not sure it would help a lot... How many times did you regenerate this code until you got it right? And how do you know that you really got it so right that it was the last time ever that you needed your generator for it? What if the C structure of any of those "several types" ever changes? Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
