[Numpy-discussion] Re: advice for using datetime64 in C binding
Hello,
I found my ankser in own code ^^
static PyArray_Descr * create_datetime64_ms_dtype()
{
// Extrapolated from numpy sources
PyArray_Descr * dtype = PyArray_DescrFromType(NPY_DATETIME);
reinterpret_cast(dtype->c_metadata)->meta.base = NPY_FR_ms;
return dtype;
}
On Thu, 2022-01-20 at 09:20 +0100, Benoit Gschwind wrote:
> Hello,
>
> How can I create np.dtype("datetime64[ms]") in C to get the
> coresponding PyArray_Descr ?
>
> Thank you by advance
>
> On Wed, 2022-01-19 at 10:08 +0100, Benoit Gschwind wrote:
> > Hello Sebastian,
> >
> > Thanks for the precision
> >
> > Best regards
> >
> > On Tue, 2022-01-18 at 11:52 -0600, Sebastian Berg wrote:
> > > On Tue, 2022-01-18 at 18:29 +0100, Benoit Gschwind wrote:
> > > > Hello Sebastian,
> > > >
> > > > Thanks for detail.
> > > >
> > > > The last call has the NPY_ARRAY_C_CONTIGUOUS and
> > > > NPY_ARRAY_ALIGNED,
> > > > thus I guess it can be no-op most of the time but for some
> > > > unknown
> > > > case
> > > > it's may be safer ?
> > >
> > > Ah, yes, you are right, and it should indeed by very quick,
> > > anyway.
> > > My
> > > guess is, that you can do _only_ the last call with the appropriate
> > > `datetime64[ms]` descriptor passed in.
> > > (Since whatever C-code that follows is allowed to work with
> > > datetime64
> > > as if it were int64.)
> > >
> > > Cheers,
> > >
> > > Sebastian
> > >
> > >
> > > >
> > > > Best regards
> > > >
> > > > On Tue, 2022-01-18 at 09:22 -0600, Sebastian Berg wrote:
> > > > > On Tue, 2022-01-18 at 14:56 +0100, Benoit Gschwind wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I using the following code:
> > > > > >
> > > > > > if (PyArray_TYPE(arr1) == NPY_DATETIME) {
> > > > > > // Ensure datetime64[ms]
> > > > > > auto tmp =
> > > > > > reinterpret_cast(PyObject_CallMethod(reinterp
> > > > > > re
> > > > > > t_
> > > > > > ca
> > > > > > st
> > > > > > (arr1), "astype", "(s)", "datetime64[ms]"));
> > > > > > std::swap(arr1, tmp);
> > > > > > Py_XDECREF(tmp);
> > > > > > // Ensure integer
> > > > > > tmp =
> > > > > > reinterpret_cast(PyObject_CallMethod(reinterp
> > > > > > re
> > > > > > t_
> > > > > > ca
> > > > > > st
> > > > > > (arr1), "astype", "(s)", "i8"));
> > > > > > std::swap(arr1, tmp);
> > > > > > Py_XDECREF(tmp);
> > > > > > tmp =
> > > > > > reinterpret_cast(PyArray_FromArray(arr1,
> > > > > > PyArray_DescrFromType(NPY_INT64), NPY_ARRAY_IN_ARRAY));
> > > > > > std::swap(arr1, tmp);
> > > > > > Py_XDECREF(tmp);
> > > > > > }
> > > > > >
> > > > > > First, if something is wrong with my code let me known. Then
> > > > > > I
> > > > > > wonder
> > > > > > if I can have a safe shortcut to avoid converting datetime64
> > > > > > to
> > > > > > i8.
> > > > > > I
> > > > > > guess the internal data of datetime64[ms] is i8 thus copying
> > > > > > and
> > > > > > casting the array may be avoided.
> > > > >
> > > > > Yes, you can assume datetime64 is stored as an i8 with the unit
> > > > > and
> > > > > possible byteswapping. Both of which, your initial cast will
> > > > > ensure.
> > > > >
> > > > > The internal data is i8, except for the special NaT value.
> > > > >
> > > > > Code-wise, you can avoid calling `astype`, but if you do (also
> > > > > in
> > > > > python), I suggest to pass `copy=False`, so that it does not
> > > > > copy
> > > > > if
> > > > > it
> > > > > clearly is not necessary (I am hoping to improve on the
> > > > > "clearly"
> > > > > here
> > > > > at some point).
> > > > >
> > > > > The last call again seems to be a no-op? Just the last call
> > > > > with
> > > > > the
> > > > > correct `datetime64[ms]` descriptor could be enough.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Sebastian
> > > > >
> > > > >
> > > > > >
> > > > > > Thanks by advance
> > > > > > Best regards.
> > > > > >
> > > > > >
> > > > > > ___
> > > > > > NumPy-Discussion mailing list -- numpy-discussion@python.org
> > > > > > To unsubscribe send an email to
> > > > > > numpy-discussion-le...@python.org
> > > > > > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> > > > > > Member address: sebast...@sipsolutions.net
> > > > > >
> > > > >
> > > > > ___
> > > > > NumPy-Discussion mailing list -- numpy-discussion@python.org
> > > > > To unsubscribe send an email to
> > > > > numpy-discussion-le...@python.org
> > > > > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> > > > > Member address: gschw...@gnu-log.net
> > > >
> > > >
> > > > ___
> > > > NumPy-Discussion mailing list -- numpy-discussion@python.org
> > > > To unsubscribe send an email to numpy-discussion-le...@python.org
> > > > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> > > > Member address: sebast...@sipsolut
[Numpy-discussion] Re: advice for using datetime64 in C binding
On Fri, 2022-02-18 at 15:53 +0100, Benoit Gschwind wrote:
> Hello,
>
> I found my ankser in own code ^^
>
> static PyArray_Descr * create_datetime64_ms_dtype()
> {
> // Extrapolated from numpy sources
> PyArray_Descr * dtype = PyArray_DescrFromType(NPY_DATETIME);
> reinterpret_cast(dtype-
> >c_metadata)->meta.base = NPY_FR_ms;
> return dtype;
> }
>
Oh sorry, yeah, I guess that works, I am not sure we have anything
better in NumPy.
The code itself is wrong though, you must make use of
`PyArray_DescrNewFromType`: *never* mutate the result returned by
`PyArray_DescrFromType`.
After replacing, you should also check for `dtype == NULL` since errors
could happen (not that it is relevant in practice, it only fails if
Python fails to allocate that small descriptor object).
Cheers,
Sebastian
>
>
> On Thu, 2022-01-20 at 09:20 +0100, Benoit Gschwind wrote:
> > Hello,
> >
> > How can I create np.dtype("datetime64[ms]") in C to get the
> > coresponding PyArray_Descr ?
> >
> > Thank you by advance
> >
> > On Wed, 2022-01-19 at 10:08 +0100, Benoit Gschwind wrote:
> > > Hello Sebastian,
> > >
> > > Thanks for the precision
> > >
> > > Best regards
> > >
> > > On Tue, 2022-01-18 at 11:52 -0600, Sebastian Berg wrote:
> > > > On Tue, 2022-01-18 at 18:29 +0100, Benoit Gschwind wrote:
> > > > > Hello Sebastian,
> > > > >
> > > > > Thanks for detail.
> > > > >
> > > > > The last call has the NPY_ARRAY_C_CONTIGUOUS and
> > > > > NPY_ARRAY_ALIGNED,
> > > > > thus I guess it can be no-op most of the time but for some
> > > > > unknown
> > > > > case
> > > > > it's may be safer ?
> > > >
> > > > Ah, yes, you are right, and it should indeed by very quick,
> > > > anyway.
> > > > My
> > > > guess is, that you can do _only_ the last call with the
> > > > appropriate
> > > > `datetime64[ms]` descriptor passed in.
> > > > (Since whatever C-code that follows is allowed to work with
> > > > datetime64
> > > > as if it were int64.)
> > > >
> > > > Cheers,
> > > >
> > > > Sebastian
> > > >
> > > >
> > > > >
> > > > > Best regards
> > > > >
> > > > > On Tue, 2022-01-18 at 09:22 -0600, Sebastian Berg wrote:
> > > > > > On Tue, 2022-01-18 at 14:56 +0100, Benoit Gschwind wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > I using the following code:
> > > > > > >
> > > > > > > if (PyArray_TYPE(arr1) == NPY_DATETIME) {
> > > > > > > // Ensure datetime64[ms]
> > > > > > > auto tmp =
> > > > > > > reinterpret_cast(PyObject_CallMethod(rein
> > > > > > > terp
> > > > > > > re
> > > > > > > t_
> > > > > > > ca
> > > > > > > st
> > > > > > > (arr1), "astype", "(s)", "datetime64[ms]"));
> > > > > > > std::swap(arr1, tmp);
> > > > > > > Py_XDECREF(tmp);
> > > > > > > // Ensure integer
> > > > > > > tmp =
> > > > > > > reinterpret_cast(PyObject_CallMethod(rein
> > > > > > > terp
> > > > > > > re
> > > > > > > t_
> > > > > > > ca
> > > > > > > st
> > > > > > > (arr1), "astype", "(s)", "i8"));
> > > > > > > std::swap(arr1, tmp);
> > > > > > > Py_XDECREF(tmp);
> > > > > > > tmp =
> > > > > > > reinterpret_cast(PyArray_FromArray(arr1,
> > > > > > > PyArray_DescrFromType(NPY_INT64), NPY_ARRAY_IN_ARRAY));
> > > > > > > std::swap(arr1, tmp);
> > > > > > > Py_XDECREF(tmp);
> > > > > > > }
> > > > > > >
> > > > > > > First, if something is wrong with my code let me known.
> > > > > > > Then
> > > > > > > I
> > > > > > > wonder
> > > > > > > if I can have a safe shortcut to avoid converting
> > > > > > > datetime64
> > > > > > > to
> > > > > > > i8.
> > > > > > > I
> > > > > > > guess the internal data of datetime64[ms] is i8 thus
> > > > > > > copying
> > > > > > > and
> > > > > > > casting the array may be avoided.
> > > > > >
> > > > > > Yes, you can assume datetime64 is stored as an i8 with the
> > > > > > unit
> > > > > > and
> > > > > > possible byteswapping. Both of which, your initial cast
> > > > > > will
> > > > > > ensure.
> > > > > >
> > > > > > The internal data is i8, except for the special NaT value.
> > > > > >
> > > > > > Code-wise, you can avoid calling `astype`, but if you do
> > > > > > (also
> > > > > > in
> > > > > > python), I suggest to pass `copy=False`, so that it does
> > > > > > not
> > > > > > copy
> > > > > > if
> > > > > > it
> > > > > > clearly is not necessary (I am hoping to improve on the
> > > > > > "clearly"
> > > > > > here
> > > > > > at some point).
> > > > > >
> > > > > > The last call again seems to be a no-op? Just the last
> > > > > > call
> > > > > > with
> > > > > > the
> > > > > > correct `datetime64[ms]` descriptor could be enough.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Sebastian
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thanks by advance
> > > > > > > Best regards.
> > > > > > >
> > > > > > >
> > > > > > > ___
> > > > > > > NumPy-Discussion mailing list --
> > > > > > > numpy-di
[Numpy-discussion] Re: advice for using datetime64 in C binding
Thank you for fixing my code ! :)
On Fri, 2022-02-18 at 09:00 -0600, Sebastian Berg wrote:
> On Fri, 2022-02-18 at 15:53 +0100, Benoit Gschwind wrote:
> > Hello,
> >
> > I found my ankser in own code ^^
> >
> > static PyArray_Descr * create_datetime64_ms_dtype()
> > {
> > // Extrapolated from numpy sources
> > PyArray_Descr * dtype =
> > PyArray_DescrFromType(NPY_DATETIME);
> > reinterpret_cast(dtype-
> > > c_metadata)->meta.base = NPY_FR_ms;
> > return dtype;
> > }
> >
>
> Oh sorry, yeah, I guess that works, I am not sure we have anything
> better in NumPy.
>
> The code itself is wrong though, you must make use of
> `PyArray_DescrNewFromType`: *never* mutate the result returned by
> `PyArray_DescrFromType`.
>
> After replacing, you should also check for `dtype == NULL` since
> errors
> could happen (not that it is relevant in practice, it only fails if
> Python fails to allocate that small descriptor object).
>
> Cheers,
>
> Sebastian
>
>
> >
> >
> > On Thu, 2022-01-20 at 09:20 +0100, Benoit Gschwind wrote:
> > > Hello,
> > >
> > > How can I create np.dtype("datetime64[ms]") in C to get the
> > > coresponding PyArray_Descr ?
> > >
> > > Thank you by advance
> > >
> > > On Wed, 2022-01-19 at 10:08 +0100, Benoit Gschwind wrote:
> > > > Hello Sebastian,
> > > >
> > > > Thanks for the precision
> > > >
> > > > Best regards
> > > >
> > > > On Tue, 2022-01-18 at 11:52 -0600, Sebastian Berg wrote:
> > > > > On Tue, 2022-01-18 at 18:29 +0100, Benoit Gschwind wrote:
> > > > > > Hello Sebastian,
> > > > > >
> > > > > > Thanks for detail.
> > > > > >
> > > > > > The last call has the NPY_ARRAY_C_CONTIGUOUS and
> > > > > > NPY_ARRAY_ALIGNED,
> > > > > > thus I guess it can be no-op most of the time but for some
> > > > > > unknown
> > > > > > case
> > > > > > it's may be safer ?
> > > > >
> > > > > Ah, yes, you are right, and it should indeed by very quick,
> > > > > anyway.
> > > > > My
> > > > > guess is, that you can do _only_ the last call with the
> > > > > appropriate
> > > > > `datetime64[ms]` descriptor passed in.
> > > > > (Since whatever C-code that follows is allowed to work with
> > > > > datetime64
> > > > > as if it were int64.)
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Sebastian
> > > > >
> > > > >
> > > > > >
> > > > > > Best regards
> > > > > >
> > > > > > On Tue, 2022-01-18 at 09:22 -0600, Sebastian Berg wrote:
> > > > > > > On Tue, 2022-01-18 at 14:56 +0100, Benoit Gschwind wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I using the following code:
> > > > > > > >
> > > > > > > > if (PyArray_TYPE(arr1) == NPY_DATETIME) {
> > > > > > > > // Ensure datetime64[ms]
> > > > > > > > auto tmp =
> > > > > > > > reinterpret_cast(PyObject_CallMethod(re
> > > > > > > > in
> > > > > > > > terp
> > > > > > > > re
> > > > > > > > t_
> > > > > > > > ca
> > > > > > > > st
> > > > > > > > (arr1), "astype", "(s)", "datetime64[ms]"));
> > > > > > > > std::swap(arr1, tmp);
> > > > > > > > Py_XDECREF(tmp);
> > > > > > > > // Ensure integer
> > > > > > > > tmp =
> > > > > > > > reinterpret_cast(PyObject_CallMethod(re
> > > > > > > > in
> > > > > > > > terp
> > > > > > > > re
> > > > > > > > t_
> > > > > > > > ca
> > > > > > > > st
> > > > > > > > (arr1), "astype", "(s)", "i8"));
> > > > > > > > std::swap(arr1, tmp);
> > > > > > > > Py_XDECREF(tmp);
> > > > > > > > tmp =
> > > > > > > > reinterpret_cast(PyArray_FromArray(arr1
> > > > > > > > ,
> > > > > > > > PyArray_DescrFromType(NPY_INT64), NPY_ARRAY_IN_ARRAY));
> > > > > > > > std::swap(arr1, tmp);
> > > > > > > > Py_XDECREF(tmp);
> > > > > > > > }
> > > > > > > >
> > > > > > > > First, if something is wrong with my code let me known.
> > > > > > > > Then
> > > > > > > > I
> > > > > > > > wonder
> > > > > > > > if I can have a safe shortcut to avoid converting
> > > > > > > > datetime64
> > > > > > > > to
> > > > > > > > i8.
> > > > > > > > I
> > > > > > > > guess the internal data of datetime64[ms] is i8 thus
> > > > > > > > copying
> > > > > > > > and
> > > > > > > > casting the array may be avoided.
> > > > > > >
> > > > > > > Yes, you can assume datetime64 is stored as an i8 with
> > > > > > > the
> > > > > > > unit
> > > > > > > and
> > > > > > > possible byteswapping. Both of which, your initial cast
> > > > > > > will
> > > > > > > ensure.
> > > > > > >
> > > > > > > The internal data is i8, except for the special NaT
> > > > > > > value.
> > > > > > >
> > > > > > > Code-wise, you can avoid calling `astype`, but if you do
> > > > > > > (also
> > > > > > > in
> > > > > > > python), I suggest to pass `copy=False`, so that it does
> > > > > > > not
> > > > > > > copy
> > > > > > > if
> > > > > > > it
> > > > > > > clearly is not necessary (I am hoping to improve on the
> > > > > > > "clearly"
> > > > > > > here
> > > > > > > at some point).
> > > > > > >
> > > > > > > The last call again
