Re: [Numpy-discussion] difference between dtypes
On Wed, Jul 22, 2015 at 7:45 PM, josef.p...@gmail.com wrote: Is there an explanation somewhere of what different basic dtypes mean, across platforms and python versions? np.bool8 type 'numpy.bool_' np.bool_ type 'numpy.bool_' bool type 'bool' Are there any rules and recommendations or is it all folks lore? This may help a little: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html#arrays-dtypes-constructing Basically, we accept the builtin Python type objects as a dtype argument and do something sensible with them. float - np.float64 because Python floats are C doubles. int - np.int32 or np.int64 depending on whatever a C long is (i.e. depending on the 64bitness of your CPU and how your OS chooses to deal with that). We encode those precision choices as aliases to the corresponding specific numpy scalar types (underscored as necessary to avoid shadowing builtins of the same name): np.float_ is np.float64, for example. See here for why the aliases to Python builtin types, np.int, np.float, etc. still exist: https://github.com/numpy/numpy/pull/6103#issuecomment-123652497 If you just need to pass a dtype= argument and want the precision that matches the native integer and float for your platform, then I prefer to use the Python builtin types instead of the underscored aliases; they just look cleaner. If you need a true numpy scalar type (e.g. to construct a numpy scalar object), of course, you must use one of the numpy scalar types, and the underscored aliases are convenient for that. Never use the aliases to the Python builtin types. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] difference between dtypes
On Fri, Jul 24, 2015 at 3:46 AM, Robert Kern robert.k...@gmail.com wrote: On Wed, Jul 22, 2015 at 7:45 PM, josef.p...@gmail.com wrote: Is there an explanation somewhere of what different basic dtypes mean, across platforms and python versions? np.bool8 type 'numpy.bool_' np.bool_ type 'numpy.bool_' bool type 'bool' Are there any rules and recommendations or is it all folks lore? This may help a little: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html#arrays-dtypes-constructing Basically, we accept the builtin Python type objects as a dtype argument and do something sensible with them. float - np.float64 because Python floats are C doubles. int - np.int32 or np.int64 depending on whatever a C long is (i.e. depending on the 64bitness of your CPU and how your OS chooses to deal with that). We encode those precision choices as aliases to the corresponding specific numpy scalar types (underscored as necessary to avoid shadowing builtins of the same name): np.float_ is np.float64, for example. See here for why the aliases to Python builtin types, np.int, np.float, etc. still exist: https://github.com/numpy/numpy/pull/6103#issuecomment-123652497 If you just need to pass a dtype= argument and want the precision that matches the native integer and float for your platform, then I prefer to use the Python builtin types instead of the underscored aliases; they just look cleaner. If you need a true numpy scalar type (e.g. to construct a numpy scalar object), of course, you must use one of the numpy scalar types, and the underscored aliases are convenient for that. Never use the aliases to the Python builtin types. (I don't have time to follow up on this for at least two weeks) my thinking was that, if there is no actual difference between bool, np.bool and np.bool_, the np.bool could become an alias and a replacement for np.bool_, so we can get rid of a ugly trailing underscore. If np.float is always float64 it could be mapped to that directly. As the previous discussion on python int versus numpy int on python 3.x, int is at least confusing. Also I'm thinking that maybe adjusting the code to the (mis)interpretation, instead of adjusting killing np.float completely might be nicer, (but changing np.int would be riskier?) Josef -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] difference between dtypes
On Fri, Jul 24, 2015 at 10:05 AM, josef.p...@gmail.com wrote: On Fri, Jul 24, 2015 at 3:46 AM, Robert Kern robert.k...@gmail.com wrote: On Wed, Jul 22, 2015 at 7:45 PM, josef.p...@gmail.com wrote: Is there an explanation somewhere of what different basic dtypes mean, across platforms and python versions? np.bool8 type 'numpy.bool_' np.bool_ type 'numpy.bool_' bool type 'bool' Are there any rules and recommendations or is it all folks lore? This may help a little: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html#arrays-dtypes-constructing Basically, we accept the builtin Python type objects as a dtype argument and do something sensible with them. float - np.float64 because Python floats are C doubles. int - np.int32 or np.int64 depending on whatever a C long is (i.e. depending on the 64bitness of your CPU and how your OS chooses to deal with that). We encode those precision choices as aliases to the corresponding specific numpy scalar types (underscored as necessary to avoid shadowing builtins of the same name): np.float_ is np.float64, for example. See here for why the aliases to Python builtin types, np.int, np.float, etc. still exist: https://github.com/numpy/numpy/pull/6103#issuecomment-123652497 If you just need to pass a dtype= argument and want the precision that matches the native integer and float for your platform, then I prefer to use the Python builtin types instead of the underscored aliases; they just look cleaner. If you need a true numpy scalar type (e.g. to construct a numpy scalar object), of course, you must use one of the numpy scalar types, and the underscored aliases are convenient for that. Never use the aliases to the Python builtin types. (I don't have time to follow up on this for at least two weeks) my thinking was that, if there is no actual difference between bool, np.bool and np.bool_, the np.bool could become an alias and a replacement for np.bool_, so we can get rid of a ugly trailing underscore. If np.float is always float64 it could be mapped to that directly. Well, I'll tell you why that's a bad idea in when you get back in two weeks. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.?
On 07/23/2015 04:29 AM, Nathaniel Smith wrote: Hi all, So one of the things exposed in the numpy namespace are objects called np.int np.float np.bool etc. These are commonly used -- in fact, just yesterday on another project I saw a senior person reviewing a pull request instruct a more junior person that they should use np.float instead of float or np.float64. But AFAICT everyone who is actually using them is doing this based on a very easy-to-fall-for misconception, i.e., that these objects have something to do with numpy. I don't see the issue. They are just aliases so how is np.float worse than just float? Too me this does not seem worth the bother of deprecation. An argument could be made for deprecating creating dtypes from python builtin types as they are ambiguous (C float != python float) and platform dependent. E.g. dtype=int is just an endless source of bugs. But this is also so invasive that the deprecation would never be completed and just be a bother to everyone. So -1 from me. P.S.: using metamodule.py also gives us the option of making np.testing lazily imported, which last time this came up was benchmarked to improve numpy's import speed by ~35% [1] -- not too bad given that most production code will never touch np.testing. But this is just a teaser postscript; I'm not proposing that we actually do this at this time :-). [1] http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063147.html I doubt these numbers from 2012 are still correct. When this was last profiled last year the import there were two main offenders, add_docs and np.polynomial. Both have been fixed in 1.9. I don't recall np.testing even showing up. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.?
Julian Taylor jtaylor.deb...@googlemail.com wrote: I don't see the issue. They are just aliases so how is np.float worse than just float? I have burned my fingers on it. Since np.double is a C double I assumed np.float is a C float. It is not. np.int has the same problem by being a C long. Pure evil. Most users of NumPy probably expect the np.foobar dtype to map to the corresponding foobar C type. This is actually inconsistent and plain dangerous. It would be much better if dtype=float meant Python float, dtype=np.float meant C float, dtype=int meant Python int, and dtype=np.int meant C int. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: Scipy 0.16.0 release
Hi all, On behalf of the Scipy development team I'm pleased to announce the availability of Scipy 0.16.0. This release contains some exciting new features (see release notes below) and more than half a years' worth of maintenance work. 93 people contributed to this release. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.6.2 or greater. Sources, binaries and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.16.0 Enjoy, Ralf -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 == SciPy 0.16.0 Release Notes == SciPy 0.16.0 is the culmination of 7 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.16.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.6.2 or greater. Highlights of this release include: - - A Cython API for BLAS/LAPACK in `scipy.linalg` - - A new benchmark suite. It's now straightforward to add new benchmarks, and they're routinely included with performance enhancement PRs. - - Support for the second order sections (SOS) format in `scipy.signal`. New features Benchmark suite - --- The benchmark suite has switched to using `Airspeed Velocity http://spacetelescope.github.io/asv/`__ for benchmarking. You can run the suite locally via ``python runtests.py --bench``. For more details, see ``benchmarks/README.rst``. `scipy.linalg` improvements - --- A full set of Cython wrappers for BLAS and LAPACK has been added in the modules `scipy.linalg.cython_blas` and `scipy.linalg.cython_lapack`. In Cython, these wrappers can now be cimported from their corresponding modules and used without linking directly against BLAS or LAPACK. The functions `scipy.linalg.qr_delete`, `scipy.linalg.qr_insert` and `scipy.linalg.qr_update` for updating QR decompositions were added. The function `scipy.linalg.solve_circulant` solves a linear system with a circulant coefficient matrix. The function `scipy.linalg.invpascal` computes the inverse of a Pascal matrix. The function `scipy.linalg.solve_toeplitz`, a Levinson-Durbin Toeplitz solver, was added. Added wrapper for potentially useful LAPACK function ``*lasd4``. It computes the square root of the i-th updated eigenvalue of a positive symmetric rank-one modification to a positive diagonal matrix. See its LAPACK documentation and unit tests for it to get more info. Added two extra wrappers for LAPACK least-square solvers. Namely, they are ``*gelsd`` and ``*gelsy``. Wrappers for the LAPACK ``*lange`` functions, which calculate various matrix norms, were added. Wrappers for ``*gtsv`` and ``*ptsv``, which solve ``A*X = B`` for tri-diagonal matrix ``A``, were added. `scipy.signal` improvements - --- Support for second order sections (SOS) as a format for IIR filters was added. The new functions are: * `scipy.signal.sosfilt` * `scipy.signal.sosfilt_zi`, * `scipy.signal.sos2tf` * `scipy.signal.sos2zpk` * `scipy.signal.tf2sos` * `scipy.signal.zpk2sos`. Additionally, the filter design functions `iirdesign`, `iirfilter`, `butter`, `cheby1`, `cheby2`, `ellip`, and `bessel` can return the filter in the SOS format. The function `scipy.signal.place_poles`, which provides two methods to place poles for linear systems, was added. The option to use Gustafsson's method for choosing the initial conditions of the forward and backward passes was added to `scipy.signal.filtfilt`. New classes ``TransferFunction``, ``StateSpace`` and ``ZerosPolesGain`` were added. These classes are now returned when instantiating `scipy.signal.lti`. Conversion between those classes can be done explicitly now. An exponential (Poisson) window was added as `scipy.signal.exponential`, and a Tukey window was added as `scipy.signal.tukey`. The function for computing digital filter group delay was added as `scipy.signal.group_delay`. The functionality for spectral analysis and spectral density estimation has been significantly improved: `scipy.signal.welch` became ~8x faster and the functions `scipy.signal.spectrogram`, `scipy.signal.coherence` and `scipy.signal.csd` (cross-spectral density) were added. `scipy.signal.lsim` was rewritten - all known issues are fixed, so this function can now be used instead of ``lsim2``; ``lsim`` is orders of magnitude faster than ``lsim2`` in most cases. `scipy.sparse` improvements - --- The function `scipy.sparse.norm`, which computes sparse matrix norms, was added. The function `scipy.sparse.random`, which allows to draw random variates
Re: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.?
On Fri, Jul 24, 2015 at 10:03 AM, Sturla Molden sturla.mol...@gmail.com wrote: I don't see the issue. They are just aliases so how is np.float worse than just float? I have burned my fingers on it. I must have too -- but I don't recall, because I am VERY careful about not using np.float, no.int, etc... but I do have to constantly evangelize and correct code others put in my code base. This really is very, very, ugly. we get away with np.float, because every OS/compiler that gets any regular use has np.float == a c double, which is always 64 bit. but, as Sturla points our, no.int being a C long is a disaster! So +inf on deprecating this, though I have no opinion about the mechanism. Ans sadly, it will be a long time before we can actually remove them, so the evangelizing and code reviews will need to co continue for a long time... -Chris Since np.double is a C double I assumed np.float is a C float. It is not. np.int has the same problem by being a C long. Pure evil. Most users of NumPy probably expect the np.foobar dtype to map to the corresponding foobar C type. This is actually inconsistent and plain dangerous. It would be much better if dtype=float meant Python float, dtype=np.float meant C float, dtype=int meant Python int, and dtype=np.int meant C int. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] constructing record dtypes from the c-api
After drilling through the sources a second time, I found it was
numpy/core/src/multiarray/descriptor.c was the file to consult with
the primary routine being PyArray_DescrConverter and _convert_from_*
functions being the most interesting to read and glean the
capabilities of this with.
So in particular it looks like the _convert_from_dict based path is
the way to go to allow custom field offsets to safely and
transparently map C POD structs with the offset information generated
at compile time to hopefully keep dtype's in perfect sync with C
sources vs declaring on the python source side. I plan on building a
helper class to generate the dictionaries for this subroutine since
something akin to the list dtype specification is more user-friendly
(even towards me).
-Jason
On Thu, Jul 23, 2015 at 7:55 PM, Jason Newton nev...@gmail.com wrote:
Hi folks,
The moderator for the ML approved my subscription so I can now post
this back in the numpy list rather than scipy. Apologies for the
duplicate/cross posting.
I was trying to figure out how to make a dtype for a c-struct on the
c-side and storing that in some boost python libraries I'm making.
Imagine the following c-struct, greatly simplified of course from the
real ones I need to expose:
struct datum{
double position[3];
float velocity[3];
int32_t d0;
uint64_t time_of_receipt;
};
How would you make the dtype/PyArray_Descr for this?
I have as a point of reference compound types in HDF for similar
struct descriptions (h5py makes these nearly 1:1 and converts back and
forth to dtypes and hdf5 types, it uses Cython to accomplish this) -
but I don't want to bring in hdf for this task - I'm not sure how well
the offsets would go over in that translation to h5py too.
Proper/best solutions would make use of offsetof as we insert entries
to the dtype/PyArray_Descr. It's fine if you do this in straight C -
I just need to figure out how to accomplish this in a practical way.
The language I'm working in is C++11. The end goal is probably going
to be to create a helper infrastructure to allow this to be made
automatically-ish provided implementation of a [static] visitor
pattern in C++. The idea is to make numpy compatible c++ POD types
rather than use Boost.Python wrapped proxies for every object which
will cut down on some complicated and time consuming code (both of
computer and developer) when ndarrays are what's called for.
Related - would one work with record dtypes passed to C? How would
one lookup fields and offsets within a structure?
Thanks for any advisement!
-Jason
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
