Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-28 Thread Andreas Hilboll
On 19.04.2014 09:03, Andreas Hilboll wrote: On 14.04.2014 20:59, Chris Barker wrote: On Fri, Apr 11, 2014 at 4:58 PM, Stephan Hoyer sho...@gmail.com mailto:sho...@gmail.com wrote: On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris charlesr.har...@gmail.com

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-28 Thread Chris Barker
On Fri, Apr 25, 2014 at 4:57 AM, Andreas Hilboll li...@hilboll.de wrote: Array-wide access to the individual datetime components should work, i.e., datetime64array.year should yield an array of dtype int with the years. That would allow boolean indexing to filter data, like

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-24 Thread Chris Barker - NOAA Federal
On Apr 23, 2014, at 8:23 PM, Sankarshan Mudkavi smudk...@uwaterloo.ca wrote: I've been quite busy for the past few weeks but I should be much freer after next week and can pick up on this (fixing the code and actually implement things). wonderful! Thanks. Chris Cheers, Sankarshan On Apr 23,

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-24 Thread Charles R Harris
On Thu, Apr 24, 2014 at 10:26 AM, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: On Apr 23, 2014, at 8:23 PM, Sankarshan Mudkavi smudk...@uwaterloo.ca wrote: I've been quite busy for the past few weeks but I should be much freer after next week and can pick up on this (fixing the

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-24 Thread Chris Barker
On Thu, Apr 24, 2014 at 10:07 AM, Charles R Harris charlesr.har...@gmail.com wrote: Might want to take a look at the datetime proposalhttps://github.com/ContinuumIO/blaze/blob/master/docs/design/blaze-datetime.mdfor blaze. oh man! not again!. Oh well, that is a decidedly different

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-23 Thread Chris Barker
On Wed, Mar 19, 2014 at 7:07 PM, Sankarshan Mudkavi smudk...@uwaterloo.cawrote: I've written a rather rudimentary NEP, (lacking in technical details which I will hopefully add after some further discussion and receiving clarification/help on this thread). Please let me know how to proceed

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-23 Thread Sankarshan Mudkavi
Thank you very much, I will incorporate it! I've been quite busy for the past few weeks but I should be much freer after next week and can pick up on this (fixing the code and actually implement things). Cheers, Sankarshan On Apr 23, 2014, at 5:58 PM, Chris Barker chris.bar...@noaa.gov wrote:

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-19 Thread Andreas Hilboll
On 14.04.2014 20:59, Chris Barker wrote: On Fri, Apr 11, 2014 at 4:58 PM, Stephan Hoyer sho...@gmail.com mailto:sho...@gmail.com wrote: On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris charlesr.har...@gmail.com mailto:charlesr.har...@gmail.com wrote: Are we in a position

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-18 Thread Sankarshan Mudkavi
I think we'll be ready to start implementation once I get the conversion to datetime.datetime on the proposal with some decent examples. It would also be great to have opinions on what test cases should be used, so please speak up if you feel you have anything to say about that. Cheers,

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-18 Thread Stephan Hoyer
On Mon, Apr 14, 2014 at 11:59 AM, Chris Barker chris.bar...@noaa.govwrote: - datetime64 objects with high precision (e.g., ns) can't compare to datetime objects. That's a problem, but how do you think it should be handled? My thought is that it should round to microseconds, and then compare

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-14 Thread Chris Barker
On Fri, Apr 11, 2014 at 4:58 PM, Stephan Hoyer sho...@gmail.com wrote: On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris charlesr.har...@gmail.com wrote: Are we in a position to start looking at implementation? If so, it would be useful to have a collection of test cases, i.e., typical uses

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-11 Thread Sankarshan Mudkavi
So is the consensus that we don't accept any tags at all (not even temporarily)? Would that break too much existing code? Cheers, Sankarshan On Apr 1, 2014, at 2:50 PM, Alexander Belopolsky ndar...@mac.com wrote: On Tue, Apr 1, 2014 at 1:12 PM, Nathaniel Smith n...@pobox.com wrote: In [6]:

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-11 Thread Nathaniel Smith
On Fri, Apr 11, 2014 at 11:25 PM, Sankarshan Mudkavi smudk...@uwaterloo.ca wrote: So is the consensus that we don't accept any tags at all (not even temporarily)? Would that break too much existing code? Well, we don't know. If anyone has any ideas on how to figure it out then they should speak

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-11 Thread Charles R Harris
On Fri, Apr 11, 2014 at 4:25 PM, Sankarshan Mudkavi smudk...@uwaterloo.cawrote: So is the consensus that we don't accept any tags at all (not even temporarily)? Would that break too much existing code? Cheers, Sankarshan On Apr 1, 2014, at 2:50 PM, Alexander Belopolsky ndar...@mac.com

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-11 Thread Stephan Hoyer
On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris charlesr.har...@gmail.com wrote: Are we in a position to start looking at implementation? If so, it would be useful to have a collection of test cases, i.e., typical uses with specified results. That should also cover conversion from/(to?)

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-11 Thread Alexander Belopolsky
On Fri, Apr 11, 2014 at 7:58 PM, Stephan Hoyer sho...@gmail.com wrote: print datetime(2010, 1, 1) np.datetime64('2011-01-01') # raises exception This is somewhat consistent with from datetime import * datetime(2010, 1, 1) date(2010, 1, 1) Traceback (most recent call last): File stdin,

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-01 Thread Chris Barker
On Mon, Mar 31, 2014 at 7:19 PM, Nathaniel Smith n...@pobox.com wrote: The difference is that datetime.datetime doesn't provide any iso string parsing. Sure it does. datetime.strptime, with the %z modifier in particular. that's not ISO parsing, that's parsing according to a user-defined

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-01 Thread Alexander Belopolsky
On Tue, Apr 1, 2014 at 12:10 PM, Chris Barker chris.bar...@noaa.gov wrote: For a naive object, the %z and %Z format codes are replaced by empty strings. though I'm not entirely sure what that means -- probably only for writing. That's right: from datetime import *

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-01 Thread Alexander Belopolsky
On Tue, Apr 1, 2014 at 12:10 PM, Chris Barker chris.bar...@noaa.gov wrote: It seems this committee of two has come to a consensus on naive -- and you're probably right, raise an exception if there is a time zone specifier. Count me as +1 on naive, but consider converting garbage (including

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-01 Thread Nathaniel Smith
On Tue, Apr 1, 2014 at 5:22 PM, Alexander Belopolsky ndar...@mac.com wrote: On Tue, Apr 1, 2014 at 12:10 PM, Chris Barker chris.bar...@noaa.gov wrote: It seems this committee of two has come to a consensus on naive -- and you're probably right, raise an exception if there is a time zone

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-01 Thread Sankarshan Mudkavi
I agree with that interpretation of naive as well. I'll change the proposal to reflect that. So any modifier should raise an error then? (At the risk of breaking people's code.) The only question is, should we consider accepting the modifier and disregard it with a warning, letting the user

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-01 Thread Alexander Belopolsky
On Tue, Apr 1, 2014 at 1:12 PM, Nathaniel Smith n...@pobox.com wrote: In [6]: a[0] = garbage ValueError: could not convert string to float: garbage (Cf, Errors should never pass silently.) Any reason why datetime64 should be different? datetime64 is different because it has NaT support

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-31 Thread Chris Barker
On Sat, Mar 29, 2014 at 3:08 PM, Nathaniel Smith n...@pobox.com wrote: On 29 Mar 2014 20:57, Chris Barker chris.bar...@noaa.gov wrote: I think this is somewhat open for discussion -- yes, it's odd, but in the spirit of practicality beats purity, it seems OK. We could allow any TZ specifier

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-31 Thread Nathaniel Smith
On 31 Mar 2014 19:47, Chris Barker chris.bar...@noaa.gov wrote: On Sat, Mar 29, 2014 at 3:08 PM, Nathaniel Smith n...@pobox.com wrote: On 29 Mar 2014 20:57, Chris Barker chris.bar...@noaa.gov wrote: I think this is somewhat open for discussion -- yes, it's odd, but in the spirit of

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-29 Thread Nathaniel Smith
On Fri, Mar 28, 2014 at 9:30 PM, Sankarshan Mudkavi smudk...@uwaterloo.ca wrote: Hi Nathaniel, 1- You give as an example of naive datetime handling: np.datetime64('2005-02-25T03:00Z') np.datetime64('2005-02-25T03:00') This IIUC is incorrect. The Z modifier is a timezone offset, and for

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-29 Thread Chris Barker
On Sat, Mar 29, 2014 at 1:04 PM, Nathaniel Smith n...@pobox.com wrote: 1- You give as an example of naive datetime handling: np.datetime64('2005-02-25T03:00Z') np.datetime64('2005-02-25T03:00') This IIUC is incorrect. The Z modifier is a timezone offset, and for normal naive

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-29 Thread Nathaniel Smith
On 29 Mar 2014 20:57, Chris Barker chris.bar...@noaa.gov wrote: I think this is somewhat open for discussion -- yes, it's odd, but in the spirit of practicality beats purity, it seems OK. We could allow any TZ specifier for that matter -- that's kind of how naive or local timezone (non) handling

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-28 Thread Nathaniel Smith
On 28 Mar 2014 05:00, Sankarshan Mudkavi smudk...@uwaterloo.ca wrote: Hi all, Apologies for the delay in following up, here is an expanded version of the proposal, which hopefully clears up most of the details. I have not included specific implementation details for the code, such as which

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-28 Thread Sankarshan Mudkavi
Hi Nathaniel, 1- You give as an example of naive datetime handling: np.datetime64('2005-02-25T03:00Z') np.datetime64('2005-02-25T03:00') This IIUC is incorrect. The Z modifier is a timezone offset, and for normal naive datetimes would cause an error. If what I understand from

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-28 Thread Jeff Reback
FYI Here are docs for panda of timezone handling wesm worked thru the various issues w.r.t. conversion, localization, and ambiguous zone crossing. http://pandas.pydata.org/pandas-docs/stable/timeseries.html#time-zone-handling implementation is largely in here: (underlying impl is a

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-27 Thread Sankarshan Mudkavi
Hi all,Apologies for the delay in following up, here is an expanded version of the proposal, which hopefully clears up most of the details. I have not included specific implementation details for the code, such as which functions to modify etc. since I think those are not traditionally included in

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-23 Thread Chris Barker
On Fri, Mar 21, 2014 at 3:43 PM, Nathaniel Smith n...@pobox.com wrote: On Thu, Mar 20, 2014 at 11:27 PM, Chris Barker chris.bar...@noaa.gov wrote: * I think there are more or less three options: 1) a) don't have any timezone handling at all -- all datetime64s are UTC. Always

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-21 Thread Chris Barker
On Thu, Mar 20, 2014 at 4:55 PM, Sankarshan Mudkavi smudk...@uwaterloo.cawrote: Yes 2) is indeed what I was suggesting. My apologies for being unclear, I was unsure of how much detail and technical information I should include in the proposal. well, you need to put enough in that it's clear

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-21 Thread Chris Barker
On Thu, Mar 20, 2014 at 5:53 PM, Alexander Belopolsky ndar...@mac.comwrote: I recall that it was at some point suggested that epoch be part of dtype. I was not able to find the reasons for a rejection, I don't think it was rejected, it just wasn't adopted by anyone to write a NEP and write

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-21 Thread Chris Barker
On Thu, Mar 20, 2014 at 6:32 PM, Alexander Belopolsky ndar...@mac.comwrote: The difference comes down to I/O. It is more than I/O. It is also about interoperability with Python's datetime module. Sorry -- I was using I/O to mean converting to/from datetime64 and other types So that

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-21 Thread Alexander Belopolsky
On Fri, Mar 21, 2014 at 5:31 PM, Chris Barker chris.bar...@noaa.gov wrote: But this brings up a good point -- having time zone handling fully compatible ith datetime.datetime would have its advantages. I don't know if everyone is aware of this, but Python stdlib has support for fixed-offset

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-21 Thread Nathaniel Smith
On Thu, Mar 20, 2014 at 11:27 PM, Chris Barker chris.bar...@noaa.gov wrote: * I think there are more or less three options: 1) a) don't have any timezone handling at all -- all datetime64s are UTC. Always b) don't have any timezone handling at all -- all datetime64s are naive

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-20 Thread Nathaniel Smith
On 20 Mar 2014 02:07, Sankarshan Mudkavi smudk...@uwaterloo.ca wrote: I've written a rather rudimentary NEP, (lacking in technical details which I will hopefully add after some further discussion and receiving clarification/help on this thread). Please let me know how to proceed and what you

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-20 Thread Sankarshan Mudkavi
Hi Nathaniel, It differs by allowing time zone info to be preserved if supplied. A naive datetime64 would be unable to handle this, and would either have to ignore the tzinfo or would have to throw up an exception. The current suggestion is very similar to a naive datetime64 and only differs

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-20 Thread Chris Barker
On Thu, Mar 20, 2014 at 4:16 AM, Nathaniel Smith n...@pobox.com wrote: Your NEP suggests making all datetime64s be in UTC, and treating string representations from unknown timezones as UTC. How does this differ from, and why is it superior to, making all datetime64s be naive? This came up in

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-20 Thread Sankarshan Mudkavi
Hi Chris, I think there are more or less three options: 1) a) don't have any timezone handling at all -- all datetime64s are UTC. Always b) don't have any timezone handling at all -- all datetime64s are naive (the only difference between these two is I/O of

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-20 Thread Alexander Belopolsky
On Thu, Mar 20, 2014 at 7:16 AM, Nathaniel Smith n...@pobox.com wrote: Your NEP suggests making all datetime64s be in UTC, and treating string representations from unknown timezones as UTC. I recall that it was at some point suggested that epoch be part of dtype. I was not able to find the

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-20 Thread Alexander Belopolsky
On Thu, Mar 20, 2014 at 9:39 AM, Sankarshan Mudkavi smudk...@uwaterloo.cawrote: A naive datetime64 would be unable to handle this, and would either have to ignore the tzinfo or would have to throw up an exception. This is not true. Python's own datetime has no problem handling this: t1 =

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-20 Thread Alexander Belopolsky
On Thu, Mar 20, 2014 at 7:27 PM, Chris Barker chris.bar...@noaa.gov wrote: On Thu, Mar 20, 2014 at 4:16 AM, Nathaniel Smith n...@pobox.com wrote: Your NEP suggests making all datetime64s be in UTC, and treating string representations from unknown timezones as UTC. How does this differ from,

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-19 Thread Dave Hirschfeld
Sankarshan Mudkavi smudkavi at uwaterloo.ca writes: Hey all, It's been a while since the last datetime and timezones discussion thread was visited (linked below): http://thread.gmane.org/gmane.comp.python.numeric.general/53805 It looks like the best approach to follow is the UTC only

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-19 Thread Jeff Reback
Dave, your example is not a problem with numpy per se, rather that the default generation is in local timezone (same as what python datetime does). If you localize to UTC you get the results that you expect. In [49]: dates = pd.date_range('01-Apr-2014', '04-Apr-2014', freq='H')[:-1] In [50]:

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-19 Thread Dave Hirschfeld
Jeff Reback jeffreback at gmail.com writes: Dave, your example is not a problem with numpy per se, rather that the default generation is in local timezone (same as what python datetime does). If you localize to UTC you get the results that you expect.  The problem is that the default

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-19 Thread Sankarshan Mudkavi
On Mar 19, 2014, at 10:01 AM, Dave Hirschfeld novi...@gmail.com wrote: Jeff Reback jeffreback at gmail.com writes: Dave, your example is not a problem with numpy per se, rather that the default generation is in local timezone (same as what python datetime does). If you localize to

[Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-18 Thread Sankarshan Mudkavi
Hey all, It's been a while since the last datetime and timezones discussion thread was visited (linked below): http://thread.gmane.org/gmane.comp.python.numeric.general/53805 It looks like the best approach to follow is the UTC only approach in the linked thread with an optional flag to

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-18 Thread Chris Barker
On Tue, Mar 18, 2014 at 2:49 PM, Sankarshan Mudkavi smudk...@uwaterloo.cawrote: It's been a while since the last datetime and timezones discussion thread was visited (linked below): http://thread.gmane.org/gmane.comp.python.numeric.general/53805 It looks like the best approach to follow is