Re: [Numpy-discussion] Numpy doesn't use RAM

An alternative solution may be https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html If you are sure your subsequent computation against the array data has enough locality to avoid thrashing, I think numpy

Re: [Numpy-discussion] Put type annotations in NumPy proper?

I'd like to offer a +1 from skimage's perspective (and napari's!) for having NumPy types directly in the repo. We have been wanting to start using type annotations, but the lack of types in NumPy proper, together with the uncertainty about whether numpy-stubs was "officially supported" and in-sy

[Numpy-discussion] NumPy Development Meeting - Triage Focus

Hi all, Our bi-weekly triage-focused NumPy development meeting is tomorrow (Wednesday, March 25) at 11 am Pacific Time. Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PR

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 162, Issue 27

mpy to create these large arrays? > > Thanks for your time and consideration > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > --

Re: [Numpy-discussion] Put type annotations in NumPy proper?

> That is, is this an all-or-nothing thing where as soon as we start, > numpy-stubs becomes unusable? Until NumPy is made PEP 561 compatible by adding a `py.typed` file, type checkers will ignore the types in the repo, so in theory you can avoid the all or nothing. In practice it's maybe trickier

Re: [Numpy-discussion] Numpy doesn't use RAM

Another thing to point out about having an array of that percentage of the available memory is that it severely restricts what you can do with it. Since you are above 50% of the available memory, you won't be able to create another array that would be the result of computing something with that arr

Re: [Numpy-discussion] Numpy doesn't use RAM

In addition to what Sebastian said about memory fragmentation and OS limits about memory allocations, I do think it will be hard to work with an array that close to the memory limit in NumPy regardless. Almost any operation will need to make a temporary array and exceed your memory limit. You mig

Re: [Numpy-discussion] Numpy doesn't use RAM

On Tue, 2020-03-24 at 13:59 -0400, Keyvis Damptey wrote: > Hi Numpy dev community, > > I'm keyvis, a statistical data scientist. > > I'm currently using numpy in python 3.8.2 64-bit for a clustering > problem, > on a machine with 1.9 TB RAM. When I try using np.zeros to create a > 600,000 > by 60

Re: [Numpy-discussion] Put type annotations in NumPy proper?

> Putting > aside ndarray, as more challenging, even annotations for numpy functions > and method parameters with built-in types would help, as a start. This is a good idea in principle, but one thing concerns me. If we add type annotations to numpy, does it become an error to have numpy-stubs i

[Numpy-discussion] Numpy doesn't use RAM

Hi Numpy dev community, I'm keyvis, a statistical data scientist. I'm currently using numpy in python 3.8.2 64-bit for a clustering problem, on a machine with 1.9 TB RAM. When I try using np.zeros to create a 600,000 by 600,000 matrix of dtype=np.float32 it says "Unable to allocate 1.31 TiB for a

Re: [Numpy-discussion] Put type annotations in NumPy proper?

Thanks for re-starting this discussion, Stephan! I think there is definitely significant interest in this topic: https://github.com/numpy/numpy/issues/7370 is the issue with the largest number of user likes in the issue tracker (FWIW). Having them in numpy, as opposed to a separate numpy-stubs

[Numpy-discussion] Put type annotations in NumPy proper?

When we started numpy-stubs [1] a few years ago, putting type annotations in NumPy itself seemed premature. We still supported Python 2, which meant that we would need to use awkward comments for type annotations. Over the past few years, using type annotations has become increasingly popular, eve

Re: [Numpy-discussion] Proposal: NEP 41 -- First step towards a new Datatype System

On Tue, Mar 24, 2020 at 12:12 PM Matti Picus wrote: > > On 24/3/20 11:48 am, Francesc Alted wrote: > > > > What I am trying to say is that NumPy should be rather agnostic about > > providing data types beyond the relatively simple set that already > > supports. I am suggesting that focusing on p

Re: [Numpy-discussion] Proposal: NEP 41 -- First step towards a new Datatype System

On 24/3/20 11:48 am, Francesc Alted wrote: What I am trying to say is that NumPy should be rather agnostic about providing data types beyond the relatively simple set that already supports.  I am suggesting that focusing on providing a way to allow the storage (not only in-memory, but also p

Re: [Numpy-discussion] Proposal: NEP 41 -- First step towards a new Datatype System

On Mon, Mar 23, 2020 at 9:49 PM Sebastian Berg wrote: > On Mon, 2020-03-23 at 18:23 +0100, Francesc Alted wrote: > > > > If we were designing a new programming language around array > > > computing > > > principles, I do think that would be the approach I would want to > > > take/consider. But I

Re: [Numpy-discussion] Good use of __dunder__ methods in numpy

Thanks for info! Kind Regards, Abdur-Rahmaan Janhangeer compileralchemy.com | github Mauritius On Mon, Mar 23, 2020 at 10:56 PM Chris Barker wrote: > On Thu, Mar 5, 2020 at 2:15 PM Gregory Lee wrote: > >> If i can get a l