RE: Why can't numpy array be restored to saved value?
> -Original Message- > From: Christian Gollwitzer > Sent: Thursday, November 26, 2020 3:26 AM > To: python-list@python.org > Subject: Re: Why can't numpy array be restored to saved value? > > Am 25.11.20 um 07:47 schrieb pjfarl...@earthlink.net: > > Why isn't the final value of the numpy array npary in the following code the > > same as the initial value before some but not all elements of the array were > > changed to a new value? > > > > I know I am missing something basic here. I thought I understood the > > concepts of immutable vs mutable values but obviously I missed something. > > > > Because this does not copy the array, rather it creates a view into the > original array. This is an optimization to avoid copying. If you want a > copy, do svary = npary.copy() Thank you for the explanation. I did indeed find that using the copy() function did what I needed to do. Peter -- https://mail.python.org/mailman/listinfo/python-list
RE: Why can't numpy array be restored to saved value?
> -Original Message- > From: Greg Ewing > Sent: Thursday, November 26, 2020 12:01 AM > To: python-list@python.org > Subject: Re: Why can't numpy array be restored to saved value? > > On 25/11/20 7:47 pm, pjfarl...@earthlink.net wrote: > > Why isn't the final value of the numpy array npary in the following code the > > same as the initial value before some but not all elements of the array were > > changed to a new value? > > Slicing a numpy array doesn't copy anything, it just > gives you another view of the underlying data. > > This is a trap you need to watch out for, since it's > different from the way sequences normally behave in > Python. > > -- > Greg Thank you for that explanation. I will certainly watch out for it in the future. Peter -- https://mail.python.org/mailman/listinfo/python-list
Re: Why can't numpy array be restored to saved value?
Am 25.11.20 um 07:47 schrieb pjfarl...@earthlink.net:
Why isn't the final value of the numpy array npary in the following code the
same as the initial value before some but not all elements of the array were
changed to a new value?
I know I am missing something basic here. I thought I understood the
concepts of immutable vs mutable values but obviously I missed something.
My environment is Win10-64, Python 3.8.5, numpy 1.19.2.
Code and output follows. TIA for any help you can provide to cure my
ignorance.
Peter
--- nptest.py ---
import numpy as np
import sys
if len(sys.argv) > 0:
try:
asz = int(sys.argv[1]) + 0
except:
asz = 4
npary = np.full([asz, asz, asz], 0, dtype=np.int32)
print("Array before change=\n{}".format(npary))
svary = npary[:, :, :]
Because this does not copy the array, rather it creates a view into the
original array. This is an optimization to avoid copying. If you want a
copy, do svary = npary.copy()
Christian
--
https://mail.python.org/mailman/listinfo/python-list
Re: Why can't numpy array be restored to saved value?
On 25/11/20 7:47 pm, pjfarl...@earthlink.net wrote: Why isn't the final value of the numpy array npary in the following code the same as the initial value before some but not all elements of the array were changed to a new value? Slicing a numpy array doesn't copy anything, it just gives you another view of the underlying data. This is a trap you need to watch out for, since it's different from the way sequences normally behave in Python. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
RE: Why can't numpy array be restored to saved value?
Never mind, I found the numpy.copy function does what I need. Revised code
below works.
Sorry for wasting bandwidth.
Peter
--- nptest.py ---
import numpy as np
import sys
if len(sys.argv) > 0:
try:
asz = int(sys.argv[1]) + 0
except:
asz = 4
npary = np.full([asz, asz, asz], 0, dtype=np.int32)
print("Array before change=\n{}".format(npary))
svary = np.copy(npary, order='C')
npary[1:-1, 1:-1, 1:-1] = 1
print("Array after change=\n{}".format(npary))
npary = svary
print("Array after restore=\n{}".format(npary))
--- nptest.py ---
> -Original Message-
> From: pjfarl...@earthlink.net
> Sent: Wednesday, November 25, 2020 1:48 AM
> To: 'python-list@python.org'
> Subject: Why can't numpy array be restored to saved value?
>
> Why isn't the final value of the numpy array npary in the following code
the
> same as the initial value before some but not all elements of the array
were
> changed to a new value?
>
> I know I am missing something basic here. I thought I understood the
concepts
> of immutable vs mutable values but obviously I missed something.
>
> My environment is Win10-64, Python 3.8.5, numpy 1.19.2.
>
> Code and output follows. TIA for any help you can provide to cure my
> ignorance.
>
> Peter
>
> --- nptest.py ---
> import numpy as np
> import sys
>
> if len(sys.argv) > 0:
> try:
> asz = int(sys.argv[1]) + 0
> except:
> asz = 4
>
> npary = np.full([asz, asz, asz], 0, dtype=np.int32)
> print("Array before change=\n{}".format(npary))
> svary = npary[:, :, :]
> npary[1:-1, 1:-1, 1:-1] = 1
> print("Array after change=\n{}".format(npary))
> npary = svary[:, :, :]
> print("Array after restore=\n{}".format(npary))
> --- nptest.py ---
>
> --- output ---
> Array before change=
> [[[0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]]]
> Array after change=
> [[[0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 1 1 0]
> [0 1 1 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 1 1 0]
> [0 1 1 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]]]
> Array after restore=
> [[[0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 1 1 0]
> [0 1 1 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 1 1 0]
> [0 1 1 0]
> [0 0 0 0]]
>
> [[0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]
> [0 0 0 0]]]
> --- output ---
--
https://mail.python.org/mailman/listinfo/python-list
Why can't numpy array be restored to saved value?
Why isn't the final value of the numpy array npary in the following code the
same as the initial value before some but not all elements of the array were
changed to a new value?
I know I am missing something basic here. I thought I understood the
concepts of immutable vs mutable values but obviously I missed something.
My environment is Win10-64, Python 3.8.5, numpy 1.19.2.
Code and output follows. TIA for any help you can provide to cure my
ignorance.
Peter
--- nptest.py ---
import numpy as np
import sys
if len(sys.argv) > 0:
try:
asz = int(sys.argv[1]) + 0
except:
asz = 4
npary = np.full([asz, asz, asz], 0, dtype=np.int32)
print("Array before change=\n{}".format(npary))
svary = npary[:, :, :]
npary[1:-1, 1:-1, 1:-1] = 1
print("Array after change=\n{}".format(npary))
npary = svary[:, :, :]
print("Array after restore=\n{}".format(npary))
--- nptest.py ---
--- output ---
Array before change=
[[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]
[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]
[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]
[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]]
Array after change=
[[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]
[[0 0 0 0]
[0 1 1 0]
[0 1 1 0]
[0 0 0 0]]
[[0 0 0 0]
[0 1 1 0]
[0 1 1 0]
[0 0 0 0]]
[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]]
Array after restore=
[[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]
[[0 0 0 0]
[0 1 1 0]
[0 1 1 0]
[0 0 0 0]]
[[0 0 0 0]
[0 1 1 0]
[0 1 1 0]
[0 0 0 0]]
[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]]
--- output ---
--
https://mail.python.org/mailman/listinfo/python-list
Re: numpy array question
jagmit sandhu wrote: > python newbie. I can't understand the following about numpy arrays: > > x = np.array([[0, 1],[2,3],[4,5],[6,7]]) > x > array([[0, 1], >[2, 3], >[4, 5], >[6, 7]]) > x.shape > (4, 2) > y = x[:,0] > y > array([0, 2, 4, 6]) > y.shape > (4,) > > Why is the shape for y reported as (4,) ? I expected it to be a (4,1) > array. thanks in advance Why do you expect every array to be a 2D matrix? Why not 3D or 4D? In the shape of y the 4 could be followed by an infinite amount of ones (4, 1, 1, 1, 1, 1,...) Instead numpy uses as many dimensions as you provide: >>> import numpy as np >>> np.array(42).shape () >>> np.array([42]).shape (1,) >>> np.array([[42]]).shape (1, 1) >>> np.array([[[42]]]).shape (1, 1, 1) I think this is a reasonable approach. Dont you? -- https://mail.python.org/mailman/listinfo/python-list
Re: numpy array question
Il giorno giovedì 2 aprile 2020 06:30:22 UTC+2, jagmit sandhu ha scritto: > python newbie. I can't understand the following about numpy arrays: > > x = np.array([[0, 1],[2,3],[4,5],[6,7]]) > x > array([[0, 1], >[2, 3], >[4, 5], >[6, 7]]) > x.shape > (4, 2) > y = x[:,0] > y > array([0, 2, 4, 6]) > y.shape > (4,) > > Why is the shape for y reported as (4,) ? I expected it to be a (4,1) array. > thanks in advance Because is not Matlab where everything is at least a 2d array. If you fix a dimension that dimension disappear. It is the same behaviour as that of Fortran. Personally I think that the Python behaviour is more natural and obvious. As always it is a choice of who has written the library what will happen with a slice. -- https://mail.python.org/mailman/listinfo/python-list
numpy array question
python newbie. I can't understand the following about numpy arrays: x = np.array([[0, 1],[2,3],[4,5],[6,7]]) x array([[0, 1], [2, 3], [4, 5], [6, 7]]) x.shape (4, 2) y = x[:,0] y array([0, 2, 4, 6]) y.shape (4,) Why is the shape for y reported as (4,) ? I expected it to be a (4,1) array. thanks in advance -- https://mail.python.org/mailman/listinfo/python-list
Re: numpy array - convert hex to int
Sharan Basappa writes:
> On Sunday, 8 September 2019 11:16:52 UTC-4, Luciano Ramalho wrote:
>> >>> int('C0FFEE', 16)
>> 12648430
>>
>> There you go!
>>
>> On Sun, Sep 8, 2019 at 12:02 PM Sharan Basappa
>> wrote:
>> >
>> > I have a numpy array that has data in the form of hex.
>> > I would like to convert that into decimal/integer.
>> > Need suggestions please.
>> > --
>
> I am sorry. I forgot to mention that I have the data in a numpy array.
> So, when I try to convert to int, I get the following error.
>
> sample code here
> #
> my_data_3 = int(my_data_2)
>
> my_data_4 = my_data_3.astype(np.float)
> #
>
> Error here
> ###
> #np.core.defchararray.replace(my_data_2,",'')
> 27
> ---> 28 my_data_3 = int(my_data_2)
> 29
> 30 my_data_4 = my_data_3.astype(np.float)
> TypeError: only length-1 arrays can be converted to Python scalars
> #
>>> my_data_2 = numpy.array(['0a', '2f', '38', 'ff'])
>>>
>>> a_toint = np.frompyfunc(lambda x: int(x, 16), 1, 1)
>>> my_data_3 = a_toint(my_data_2)
>>> my_data_3
array([10, 47, 56, 255], dtype=object)
--
Piet van Oostrum
WWW: http://piet.vanoostrum.org/
PGP key: [8DAE142BE17999C4]
--
https://mail.python.org/mailman/listinfo/python-list
Re: numpy array - convert hex to int
On Sunday, 8 September 2019 11:16:52 UTC-4, Luciano Ramalho wrote:
> >>> int('C0FFEE', 16)
> 12648430
>
> There you go!
>
> On Sun, Sep 8, 2019 at 12:02 PM Sharan Basappa
> wrote:
> >
> > I have a numpy array that has data in the form of hex.
> > I would like to convert that into decimal/integer.
> > Need suggestions please.
> > --
I am sorry. I forgot to mention that I have the data in a numpy array.
So, when I try to convert to int, I get the following error.
sample code here
#
my_data_3 = int(my_data_2)
my_data_4 = my_data_3.astype(np.float)
#
Error here
###
#np.core.defchararray.replace(my_data_2,",'')
27
---> 28 my_data_3 = int(my_data_2)
29
30 my_data_4 = my_data_3.astype(np.float)
TypeError: only length-1 arrays can be converted to Python scalars
#
--
https://mail.python.org/mailman/listinfo/python-list
Re: numpy array - convert hex to int
>>> int('C0FFEE', 16)
12648430
There you go!
On Sun, Sep 8, 2019 at 12:02 PM Sharan Basappa wrote:
>
> I have a numpy array that has data in the form of hex.
> I would like to convert that into decimal/integer.
> Need suggestions please.
> --
> https://mail.python.org/mailman/listinfo/python-list
--
Luciano Ramalho
| Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
| Technical Principal at ThoughtWorks
| Twitter: @ramalhoorg
--
https://mail.python.org/mailman/listinfo/python-list
numpy array - convert hex to int
I have a numpy array that has data in the form of hex. I would like to convert that into decimal/integer. Need suggestions please. -- https://mail.python.org/mailman/listinfo/python-list
Re: Numpy array
On 05/18/2018 09:50 PM, Sharan Basappa wrote:
This is regarding numpy array. I am a bit confused how parts of the array are
being accessed in the example below.
1 import scipy as sp
2 data = sp.genfromtxt("web_traffic.tsv", delimiter="\t")
3 print(data[:10])
4 x = data[:,0]
5 y = data[:,1]
Apparently, line 3 prints the first 10 entries in the array
line 4 & 5 is to extract all rows but only 1st and second columns alone for x
and y respectively.
I am confused as to how data[:10] gives the first 10 rows while data[:,0] gives
all rows
Numpy ordering is [rows,columns,...]. Things you don't provide an
explicit answer for are assumed to be "all", which in Python syntax is
":". The reason for THAT goes into how slices are constructed.
So your example 3 could also be written print(data[0:10,:]), which sets
rows to the range starting with 0 and ending before 10, and columns to
everything.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Numpy array
The "indexing" page of the documentation might help you with this:
https://docs.scipy.org/doc/numpy-1.14.0/reference/arrays.indexing.html
On 05/18/2018 09:50 PM, sharan.basa...@gmail.com wrote:
This is regarding numpy array. I am a bit confused how parts of the array are
being accessed in the example below.
1 import scipy as sp
2 data = sp.genfromtxt("web_traffic.tsv", delimiter="\t")
3 print(data[:10])
4 x = data[:,0]
5 y = data[:,1]
Apparently, line 3 prints the first 10 entries in the array
line 4 & 5 is to extract all rows but only 1st and second columns alone for x
and y respectively.
I am confused as to how data[:10] gives the first 10 rows while data[:,0] gives
all rows
--
Dr. Gary Herron
Professor of Computer Science
DigiPen Institute of Technology
(425) 895-4418
--
https://mail.python.org/mailman/listinfo/python-list
Numpy array
This is regarding numpy array. I am a bit confused how parts of the array are
being accessed in the example below.
1 import scipy as sp
2 data = sp.genfromtxt("web_traffic.tsv", delimiter="\t")
3 print(data[:10])
4 x = data[:,0]
5 y = data[:,1]
Apparently, line 3 prints the first 10 entries in the array
line 4 & 5 is to extract all rows but only 1st and second columns alone for x
and y respectively.
I am confused as to how data[:10] gives the first 10 rows while data[:,0] gives
all rows
--
https://mail.python.org/mailman/listinfo/python-list
Re: fastest way to read a text file in to a numpy array
Am 30.06.16 um 17:49 schrieb Heli: Dear all, After a few tests, I think I will need to correct a bit my question. I will give an example here. I have file 1 with 250 lines: X1,Y1,Z1 X2,Y2,Z2 Then I have file 2 with 3M lines: X1,Y1,Z1,value11,value12, value13, X2,Y2,Z2,value21,value22, value23,... I will need to interpolate values for the coordinates on file 1 from file 2. (using nearest) I am using the scipy.griddata for this. scipy.interpolate.griddata(points, values, xi, method='linear', fill_value=nan, rescale=False) This constructs a Delaunay triangulation and no wonder takes some time if you run it over 3M datapoints. You can probably save a factor of three, because: I need to repeat the griddata above to get interpolation for each of the column of values. I think this is wrong. It should, according to the docs, happily interpolate from a 2D array of values. BTW, you stated you want nearest interpolation, but you chose "linear". I think it won't make a big difference on runtime, though. (nearest uses a KDtree, Linear uses QHull) I was wondering if there are any ways to improve the time spent in interpolation. Are you sure you need the full generality of this algorithm? i.e., are your values given on a scattered cloud of points in the 3D space, or maybe the X,Y,Z in file2 are in fact on a rectangular grid? In the former case, there is probably nothing you can really do. In the latter, there should be a more efficient algorithm by looking up the nearest index from X,Y,Z by index arithmetics. Or maybe even reshaping it into a 3D-array. Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: fastest way to read a text file in to a numpy array
Dear all, After a few tests, I think I will need to correct a bit my question. I will give an example here. I have file 1 with 250 lines: X1,Y1,Z1 X2,Y2,Z2 Then I have file 2 with 3M lines: X1,Y1,Z1,value11,value12, value13, X2,Y2,Z2,value21,value22, value23,... I will need to interpolate values for the coordinates on file 1 from file 2. (using nearest) I am using the scipy.griddata for this. scipy.interpolate.griddata(points, values, xi, method='linear', fill_value=nan, rescale=False) When slicing the code, reading files in to numpy is not the culprit, but the griddata is. time to read file2= 2 min time to interpolate= 48 min I need to repeat the griddata above to get interpolation for each of the column of values. I was wondering if there are any ways to improve the time spent in interpolation. Thank you very much in advance for your help, -- https://mail.python.org/mailman/listinfo/python-list
Re: fastest way to read a text file in to a numpy array
On Tue, Jun 28, 2016 at 8:45 AM, Heli wrote: > Hi, > > I need to read a file in to a 2d numpy array containing many number of lines. > I was wondering what is the fastest way to do this? > > Is even reading the file in to numpy array the best method or there are > better approaches? > numpy.genfromtxt[1] is a pretty robust function for reading text files. If you're generating the file from a numpy array already, you should use arr.save()[2] and numpy.load()[3]. [1]: http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html [2]: http://docs.scipy.org/doc/numpy/reference/generated/numpy.save.html [3]: http://docs.scipy.org/doc/numpy/reference/generated/numpy.load.html Cody -- https://mail.python.org/mailman/listinfo/python-list
Re: fastest way to read a text file in to a numpy array
On Tue, Jun 28, 2016 at 10:08 AM Hedieh Ebrahimi wrote: > File 1 has : > x1,y1,z1 > x2,y2,z2 > > > and file2 has : > x1,y1,z1,value1 > x2,y2,z2,value2 > x3,y3,z3,value3 > ... > > I need to read the coordinates from file 1 and then interpolate a value > for these coordinates on file 2 to the closest coordinate possible. The > problem is file 2 is has around 5M lines. So I was wondering what would be > the fastest approach? > Is this a one-time task, or something you'll need to repeat frequently? How many points need to be interpolated? How do you define distance? Euclidean 3d distance? K-nearest? 5 million can probably fit into memory, so it's not so bad. NumPy is a good option for broadcasting the distance function across all 5 million labeled points for each unlabeled point. Given that file format, NumPy can probably read from file directly into an array. http://stackoverflow.com/questions/3518778/how-to-read-csv-into-record-array-in-numpy -- https://mail.python.org/mailman/listinfo/python-list
Re: fastest way to read a text file in to a numpy array
On Tue, Jun 28, 2016 at 9:51 AM Heli wrote: > Is even reading the file in to numpy array the best method or there are > better approaches? > What are you trying to accomplish? Summary statistics, data transformation, analysis...? -- https://mail.python.org/mailman/listinfo/python-list
fastest way to read a text file in to a numpy array
Hi, I need to read a file in to a 2d numpy array containing many number of lines. I was wondering what is the fastest way to do this? Is even reading the file in to numpy array the best method or there are better approaches? Thanks for your suggestions, -- https://mail.python.org/mailman/listinfo/python-list
Re: looping and searching in numpy array
On 10 March 2016 at 13:02, Peter Otten <__pete...@web.de> wrote:
> Heli wrote:
>
>> I need to loop over a numpy array and then do the following search. The
>> following is taking almost 60(s) for an array (npArray1 and npArray2 in
>> the example below) with around 300K values.
>>
>>
>> for id in np.nditer(npArray1):
>>newId=(np.where(npArray2==id))[0][0]
What are the dtypes of the arrays? And what are the typical sizes of
each of them. It can have a big effect on what makes a good solution
to the problem.
>> Is there anyway I can make the above faster? I need to run the script
>> above on much bigger arrays (50M). Please note that my two numpy arrays in
>> the lines above, npArray1 and npArray2 are not necessarily the same size,
>> but they are both 1d.
>
> You mean you are looking for the index of the first occurence in npArray2
> for every value of npArray1?
>
> I don't know how to do this in numpy (I'm not an expert), but even basic
> Python might be acceptable:
I'm not sure that numpy has any particular function that can be of use
here. Your approach below looks good though.
> lookup = {}
> for i, v in enumerate(npArray2):
> if v not in lookup:
> lookup[v] = i
Looking at this I wondered if there was a way to avoid the double hash
table lookup and realised it's the first time I've ever considered a
use for setdefault:
for i, v in enumerate(npArray2):
lookup.setdefault(i, v)
Another option would be to use this same algorithm in Cython. Then you
can access the ndarray data pointer directly and loop over it in C.
This is the kind of scenario where that sort of thing can be well
worth doing.
--
Oscar
--
https://mail.python.org/mailman/listinfo/python-list
Re: looping and searching in numpy array
problem is infact not related to numpy at all. the complexity of your
algorithm is O(len(npArray1) * len(npArray2))
which means the number of computations that you are doing is in the range
of 10**10,
if the absolute difference between the maximum element and minimum element
is less than 10**6, you can improve your code by pre-computing the first
occurrence of a number by using an array of size of that difference(afore
mentioned).
if your npArray2 doesn't have such a pattern, you have to precompute it by
using a dict (I don't know if numpy has such data structure)
an optimised pseudo code would look like
mmdiff = max(npArray2) - min(npArray2)
if mmdiff < 10**6:
precom = np.array([-1] * mmdiff)
offset = min(npArray2)
for i, x in enumerate(npArray2):
precom[x - offset] = i
for id in npArray1:
if 0 <= id - offset < mmdiff and precom[id - offset] != -1:
new_id = precom[id]
# your code
else:
precom = {}
for i, x in enumerate(npArray1):
if x not in precom:
precom[x] = i
for id in npArray1:
if id in precom:
new_id = precom[id]
# your code
you can just use the else case which will work for all cases but if your
npArray2 has such a pattern then the above code will perform better.
Regards
Srinivas Devaki
Junior (3rd yr) student at Indian School of Mines,(IIT Dhanbad)
Computer Science and Engineering Department
ph: +91 9491 383 249
telegram_id: @eightnoteight
On Mar 10, 2016 5:15 PM, "Heli" wrote:
Dear all,
I need to loop over a numpy array and then do the following search. The
following is taking almost 60(s) for an array (npArray1 and npArray2 in the
example below) with around 300K values.
for id in np.nditer(npArray1):
newId=(np.where(npArray2==id))[0][0]
Is there anyway I can make the above faster? I need to run the script above
on much bigger arrays (50M). Please note that my two numpy arrays in the
lines above, npArray1 and npArray2 are not necessarily the same size, but
they are both 1d.
Thanks a lot for your help,
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
RE: looping and searching in numpy array
> From: sjeik_ap...@hotmail.com > To: heml...@gmail.com; python-list@python.org > Subject: RE: looping and searching in numpy array > Date: Sun, 13 Mar 2016 13:51:23 + > > Hi, I suppose you have seen this already (in particular the first link): > http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.htmlI > don't thonk it's part of numpy yet. > Albert-Jan sorry, the correct url is: http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.html -- https://mail.python.org/mailman/listinfo/python-list
RE: looping and searching in numpy array
> Date: Thu, 10 Mar 2016 08:48:48 -0800
> Subject: Re: looping and searching in numpy array
> From: heml...@gmail.com
> To: python-list@python.org
>
> On Thursday, March 10, 2016 at 2:02:57 PM UTC+1, Peter Otten wrote:
> > Heli wrote:
> >
> > > Dear all,
> > >
> > > I need to loop over a numpy array and then do the following search. The
> > > following is taking almost 60(s) for an array (npArray1 and npArray2 in
> > > the example below) with around 300K values.
> > >
> > >
> > > for id in np.nditer(npArray1):
> > >
> > >newId=(np.where(npArray2==id))[0][0]
> > >
> > >
> > > Is there anyway I can make the above faster? I need to run the script
> > > above on much bigger arrays (50M). Please note that my two numpy arrays in
> > > the lines above, npArray1 and npArray2 are not necessarily the same size,
> > > but they are both 1d.
> >
> > You mean you are looking for the index of the first occurence in npArray2
> > for every value of npArray1?
> >
> > I don't know how to do this in numpy (I'm not an expert), but even basic
> > Python might be acceptable:
> >
> > lookup = {}
> > for i, v in enumerate(npArray2):
> > if v not in lookup:
> > lookup[v] = i
> >
> > for v in npArray1:
> > print(lookup.get(v, ""))
> >
> > That way you iterate once (in Python) instead of 2*len(npArray1) times (in
> > C) over npArray2.
>
> Dear Peter,
>
> Thanks for your reply. This really helped. It reduces the script time from
> 61(s) to 2(s).
>
> I am still very interested in knowing the correct numpy way to do this, but
> till then your fix works great.
Hi, I suppose you have seen this already (in particular the first link):
http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.htmlI
don't thonk it's part of numpy yet.
Albert-Jan
--
https://mail.python.org/mailman/listinfo/python-list
Re: looping and searching in numpy array
On Thursday, March 10, 2016 at 5:49:07 PM UTC+1, Heli wrote:
> On Thursday, March 10, 2016 at 2:02:57 PM UTC+1, Peter Otten wrote:
> > Heli wrote:
> >
> > > Dear all,
> > >
> > > I need to loop over a numpy array and then do the following search. The
> > > following is taking almost 60(s) for an array (npArray1 and npArray2 in
> > > the example below) with around 300K values.
> > >
> > >
> > > for id in np.nditer(npArray1):
> > >
> > >newId=(np.where(npArray2==id))[0][0]
> > >
> > >
> > > Is there anyway I can make the above faster? I need to run the script
> > > above on much bigger arrays (50M). Please note that my two numpy arrays in
> > > the lines above, npArray1 and npArray2 are not necessarily the same size,
> > > but they are both 1d.
> >
> > You mean you are looking for the index of the first occurence in npArray2
> > for every value of npArray1?
> >
> > I don't know how to do this in numpy (I'm not an expert), but even basic
> > Python might be acceptable:
> >
> > lookup = {}
> > for i, v in enumerate(npArray2):
> > if v not in lookup:
> > lookup[v] = i
> >
> > for v in npArray1:
> > print(lookup.get(v, ""))
> >
> > That way you iterate once (in Python) instead of 2*len(npArray1) times (in
> > C) over npArray2.
>
> Dear Peter,
>
> Thanks for your reply. This really helped. It reduces the script time from
> 61(s) to 2(s).
>
> I am still very interested in knowing the correct numpy way to do this, but
> till then your fix works great.
>
> Thanks a lot,
And yes, I am looking for the index of the first occurence in npArray2
for every value of npArray1.
--
https://mail.python.org/mailman/listinfo/python-list
Re: looping and searching in numpy array
On Thursday, March 10, 2016 at 2:02:57 PM UTC+1, Peter Otten wrote:
> Heli wrote:
>
> > Dear all,
> >
> > I need to loop over a numpy array and then do the following search. The
> > following is taking almost 60(s) for an array (npArray1 and npArray2 in
> > the example below) with around 300K values.
> >
> >
> > for id in np.nditer(npArray1):
> >
> >newId=(np.where(npArray2==id))[0][0]
> >
> >
> > Is there anyway I can make the above faster? I need to run the script
> > above on much bigger arrays (50M). Please note that my two numpy arrays in
> > the lines above, npArray1 and npArray2 are not necessarily the same size,
> > but they are both 1d.
>
> You mean you are looking for the index of the first occurence in npArray2
> for every value of npArray1?
>
> I don't know how to do this in numpy (I'm not an expert), but even basic
> Python might be acceptable:
>
> lookup = {}
> for i, v in enumerate(npArray2):
> if v not in lookup:
> lookup[v] = i
>
> for v in npArray1:
> print(lookup.get(v, ""))
>
> That way you iterate once (in Python) instead of 2*len(npArray1) times (in
> C) over npArray2.
Dear Peter,
Thanks for your reply. This really helped. It reduces the script time from
61(s) to 2(s).
I am still very interested in knowing the correct numpy way to do this, but
till then your fix works great.
Thanks a lot,
--
https://mail.python.org/mailman/listinfo/python-list
Re: looping and searching in numpy array
On 10/03/2016 11:43, Heli wrote: Dear all, I need to loop over a numpy array and then do the following search. The following is taking almost 60(s) for an array (npArray1 and npArray2 in the example below) with around 300K values. for id in np.nditer(npArray1): newId=(np.where(npArray2==id))[0][0] Is there anyway I can make the above faster? I need to run the script above on much bigger arrays (50M). Please note that my two numpy arrays in the lines above, npArray1 and npArray2 are not necessarily the same size, but they are both 1d. Thanks a lot for your help, I'm no numpy expert, but if you're using a loop my guess is that you're doing it wrong. I suggest your first port of call is the numpy docs if you haven't all ready been there, then the specific numpy mailing list or stackoverflow, as it seems very likely that this type of question has been asked before. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: looping and searching in numpy array
Heli wrote:
> Dear all,
>
> I need to loop over a numpy array and then do the following search. The
> following is taking almost 60(s) for an array (npArray1 and npArray2 in
> the example below) with around 300K values.
>
>
> for id in np.nditer(npArray1):
>
>newId=(np.where(npArray2==id))[0][0]
>
>
> Is there anyway I can make the above faster? I need to run the script
> above on much bigger arrays (50M). Please note that my two numpy arrays in
> the lines above, npArray1 and npArray2 are not necessarily the same size,
> but they are both 1d.
You mean you are looking for the index of the first occurence in npArray2
for every value of npArray1?
I don't know how to do this in numpy (I'm not an expert), but even basic
Python might be acceptable:
lookup = {}
for i, v in enumerate(npArray2):
if v not in lookup:
lookup[v] = i
for v in npArray1:
print(lookup.get(v, ""))
That way you iterate once (in Python) instead of 2*len(npArray1) times (in
C) over npArray2.
--
https://mail.python.org/mailman/listinfo/python-list
looping and searching in numpy array
Dear all, I need to loop over a numpy array and then do the following search. The following is taking almost 60(s) for an array (npArray1 and npArray2 in the example below) with around 300K values. for id in np.nditer(npArray1): newId=(np.where(npArray2==id))[0][0] Is there anyway I can make the above faster? I need to run the script above on much bigger arrays (50M). Please note that my two numpy arrays in the lines above, npArray1 and npArray2 are not necessarily the same size, but they are both 1d. Thanks a lot for your help, -- https://mail.python.org/mailman/listinfo/python-list
Re: 3D numpy array subset
On 9 Dec 2015 14:26, "Heli" wrote: > > Dear all, > > I am reading a dataset from a HDF5 file using h5py. my datasets are 3D. > > Then I will need to check if another 3d numpy array is a subset of this 3D array i am reading from the file. > > In general, is there any way to check if two 3d numpy arrays have intersections and if so, get the indices of the intersection area. > > By intersection, I exactly mean the intersection definition used in set theory. Does this help: http://docs.scipy.org/doc/numpy/reference/generated/numpy.intersect1d.html I'm not sure how the 3d part is relevant but that function finds the common elements of two arrays. Use .flat or something to make the 3d arrays be treated as 1d. Otherwise it's not clear what you mean by intersection. -- Oscar -- https://mail.python.org/mailman/listinfo/python-list
3D numpy array subset
Dear all, I am reading a dataset from a HDF5 file using h5py. my datasets are 3D. Then I will need to check if another 3d numpy array is a subset of this 3D array i am reading from the file. In general, is there any way to check if two 3d numpy arrays have intersections and if so, get the indices of the intersection area. By intersection, I exactly mean the intersection definition used in set theory. Thanks in Advance for your help, -- https://mail.python.org/mailman/listinfo/python-list
Re: Fast capture and 2D image stacking as 3D numpy array with Python and Raspberry Pi
On Mon, 6 Jul 2015 at 22:36 Agustin Cruz wrote:
> I'm working on a Python - Raspberry Pi project in which I need to take
> about 30 images per second (no movie) and stack each 2D image to a 3D array
> using numpy array, without saving each 2D capture as a file (because is
> slow).
>
> I found this Python code to take images as fast as possible, but i don't
> know how to stack all images fast to a 3D stack of images.
>
What does the code below actually do? Does it save the images as files? Do
you know how to modify the code so that you get each image as a matrix (2D
array)?
>
> import io
> import time
> import picamera
> #from PIL import Image
>
> def outputs():
> stream = io.BytesIO()
> for i in range(40):
> # This returns the stream for the camera to capture to
> yield stream
> # Once the capture is complete, the loop continues here
> # (read up on generator functions in Python to understand
> # the yield statement). Here you could do some processing
> # on the image...
> #stream.seek(0)
> #img = Image.open(stream)
>
I guess the commented lines above do what you want. From the Image object
there will be some way to retrieve a matrix representing the image. Is the
image black and white? If it's RGB then you would have size 640x480x3 3D
array.
> # Finally, reset the stream for the next capture
> stream.seek(0)
> stream.truncate()
>
> with picamera.PiCamera() as camera:
> camera.resolution = (640, 480)
> camera.framerate = 80
> time.sleep(2)
> start = time.time()
> camera.capture_sequence(outputs(), 'jpeg', use_video_port=True)
> finish = time.time()
> print('Captured 40 images at %.2ffps' % (40 / (finish - start)))
>
> Does anyone of you know how to stack the 2D images taken in this code to a
> 3D numpy array using Python and the Raspberry Pi camera module? Without
> saving each 2D capture as a file
>
I can't immediately see how to access the 2D images from the code above.
See if you can change it so that it gets the image matrix for each image
(and e.g. prints out the matrix). Then change it so that you build a list
of the 2D matrices then you can just do:
video = numpy.dstack(list_of_images)
And video should be a 3D array. (I'm assuming black and white images here).
--
Oscar
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast capture and 2D image stacking as 3D numpy array with Python and Raspberry Pi
On 07/07/2015 00:16, Agustin Cruz wrote:
On Monday, July 6, 2015 at 6:00:42 PM UTC-4, Mark Lawrence wrote:
On 06/07/2015 22:31, Agustin Cruz wrote:
I'm working on a Python - Raspberry Pi project in which I need to take about 30
images per second (no movie) and stack each 2D image to a 3D array using numpy
array, without saving each 2D capture as a file (because is slow).
I found this Python code to take images as fast as possible, but i don't know
how to stack all images fast to a 3D stack of images.
import io
import time
import picamera
#from PIL import Image
def outputs():
stream = io.BytesIO()
for i in range(40):
# This returns the stream for the camera to capture to
yield stream
# Once the capture is complete, the loop continues here
# (read up on generator functions in Python to understand
# the yield statement). Here you could do some processing
# on the image...
#stream.seek(0)
#img = Image.open(stream)
# Finally, reset the stream for the next capture
stream.seek(0)
stream.truncate()
with picamera.PiCamera() as camera:
camera.resolution = (640, 480)
camera.framerate = 80
time.sleep(2)
start = time.time()
camera.capture_sequence(outputs(), 'jpeg', use_video_port=True)
finish = time.time()
print('Captured 40 images at %.2ffps' % (40 / (finish - start)))
Does anyone of you know how to stack the 2D images taken in this code to a 3D
numpy array using Python and the Raspberry Pi camera module? Without saving
each 2D capture as a file
Best regards, Agustín
http://docs.scipy.org/doc/numpy/reference/generated/numpy.dstack.html is
the first hit on google for "numpy 3d array stack".
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
Hi Mark,
I know the dstack function can do the job, but i don't know how to implement it
in this case.
Sadly I don't know how either, but if I can find the above link in
seconds, I'm fairly certain that with a little searching you could find
something. Specific sites like nullege or koders might offer solutions.
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast capture and 2D image stacking as 3D numpy array with Python and Raspberry Pi
On Monday, July 6, 2015 at 6:00:42 PM UTC-4, Mark Lawrence wrote:
> On 06/07/2015 22:31, Agustin Cruz wrote:
> > I'm working on a Python - Raspberry Pi project in which I need to take
> > about 30 images per second (no movie) and stack each 2D image to a 3D array
> > using numpy array, without saving each 2D capture as a file (because is
> > slow).
> >
> > I found this Python code to take images as fast as possible, but i don't
> > know how to stack all images fast to a 3D stack of images.
> >
> > import io
> > import time
> > import picamera
> > #from PIL import Image
> >
> > def outputs():
> > stream = io.BytesIO()
> > for i in range(40):
> > # This returns the stream for the camera to capture to
> > yield stream
> > # Once the capture is complete, the loop continues here
> > # (read up on generator functions in Python to understand
> > # the yield statement). Here you could do some processing
> > # on the image...
> > #stream.seek(0)
> > #img = Image.open(stream)
> > # Finally, reset the stream for the next capture
> > stream.seek(0)
> > stream.truncate()
> >
> > with picamera.PiCamera() as camera:
> > camera.resolution = (640, 480)
> > camera.framerate = 80
> > time.sleep(2)
> > start = time.time()
> > camera.capture_sequence(outputs(), 'jpeg', use_video_port=True)
> > finish = time.time()
> > print('Captured 40 images at %.2ffps' % (40 / (finish - start)))
> >
> > Does anyone of you know how to stack the 2D images taken in this code to a
> > 3D numpy array using Python and the Raspberry Pi camera module? Without
> > saving each 2D capture as a file
> >
> > Best regards, Agustín
> >
>
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.dstack.html is
> the first hit on google for "numpy 3d array stack".
>
> --
> My fellow Pythonistas, ask not what our language can do for you, ask
> what you can do for our language.
>
> Mark Lawrence
Hi Mark,
I know the dstack function can do the job, but i don't know how to implement it
in this case.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast capture and 2D image stacking as 3D numpy array with Python and Raspberry Pi
On 06/07/2015 22:31, Agustin Cruz wrote:
I'm working on a Python - Raspberry Pi project in which I need to take about 30
images per second (no movie) and stack each 2D image to a 3D array using numpy
array, without saving each 2D capture as a file (because is slow).
I found this Python code to take images as fast as possible, but i don't know
how to stack all images fast to a 3D stack of images.
import io
import time
import picamera
#from PIL import Image
def outputs():
stream = io.BytesIO()
for i in range(40):
# This returns the stream for the camera to capture to
yield stream
# Once the capture is complete, the loop continues here
# (read up on generator functions in Python to understand
# the yield statement). Here you could do some processing
# on the image...
#stream.seek(0)
#img = Image.open(stream)
# Finally, reset the stream for the next capture
stream.seek(0)
stream.truncate()
with picamera.PiCamera() as camera:
camera.resolution = (640, 480)
camera.framerate = 80
time.sleep(2)
start = time.time()
camera.capture_sequence(outputs(), 'jpeg', use_video_port=True)
finish = time.time()
print('Captured 40 images at %.2ffps' % (40 / (finish - start)))
Does anyone of you know how to stack the 2D images taken in this code to a 3D
numpy array using Python and the Raspberry Pi camera module? Without saving
each 2D capture as a file
Best regards, Agustín
http://docs.scipy.org/doc/numpy/reference/generated/numpy.dstack.html is
the first hit on google for "numpy 3d array stack".
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
--
https://mail.python.org/mailman/listinfo/python-list
Fast capture and 2D image stacking as 3D numpy array with Python and Raspberry Pi
I'm working on a Python - Raspberry Pi project in which I need to take about 30
images per second (no movie) and stack each 2D image to a 3D array using numpy
array, without saving each 2D capture as a file (because is slow).
I found this Python code to take images as fast as possible, but i don't know
how to stack all images fast to a 3D stack of images.
import io
import time
import picamera
#from PIL import Image
def outputs():
stream = io.BytesIO()
for i in range(40):
# This returns the stream for the camera to capture to
yield stream
# Once the capture is complete, the loop continues here
# (read up on generator functions in Python to understand
# the yield statement). Here you could do some processing
# on the image...
#stream.seek(0)
#img = Image.open(stream)
# Finally, reset the stream for the next capture
stream.seek(0)
stream.truncate()
with picamera.PiCamera() as camera:
camera.resolution = (640, 480)
camera.framerate = 80
time.sleep(2)
start = time.time()
camera.capture_sequence(outputs(), 'jpeg', use_video_port=True)
finish = time.time()
print('Captured 40 images at %.2ffps' % (40 / (finish - start)))
Does anyone of you know how to stack the 2D images taken in this code to a 3D
numpy array using Python and the Raspberry Pi camera module? Without saving
each 2D capture as a file
Best regards, Agustín
--
https://mail.python.org/mailman/listinfo/python-list
Re: numpy array product driving me mad
>> I think that you want >> >> P * R[;,None] > > Sorry, I meant > > P * R[:, None] > > Manolo Muchísimas gracias, Manolo. Eres un genio y me has ayudado mucho. Te debo una. -- https://mail.python.org/mailman/listinfo/python-list
Re: numpy array product driving me mad
On 03/20/15 at 02:11pm, Manolo Martínez wrote: > On 03/20/15 at 01:46pm, Mr. Twister wrote: > > > I have two numpy arrays: [...] > > Is there a direct, single expression command to get this result? > > I think that you want > > P * R[;,None] Sorry, I meant P * R[:, None] Manolo -- https://mail.python.org/mailman/listinfo/python-list
Re: numpy array product driving me mad
On 03/20/15 at 01:46pm, Mr. Twister wrote: > I have two numpy arrays: > > >>> P > array([[[ 2, 3], > [33, 44], > [22, 11], > [ 1, 2]]]) > >>> R > array([0, 1, 2, 3]) > > the values of these may of course be different. The important fact is that: > > >>> P.shape > (1, 4, 2) > >>> R.shape > (4,) > > where the number 4 in the shape of both P and R may be another number as well > (same on both). > > > What I'd like to get is a new array Q with same shape as P so that the nth > pair > of Q is the nth pair of P multiplied by the nth element of R. I.e., in the > above > case it should produce: > > >>> Q > array([[[ 0, 0], > [33, 44], > [44, 22], > [ 3, 6]]]) > > > Is there a direct, single expression command to get this result? I think that you want P * R[;,None] Read about broadcasting (http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) for an explanation. I'm never sure I understand it myself :) Manolo -- https://mail.python.org/mailman/listinfo/python-list
numpy array product driving me mad
Hi everyone. Hope you can help me overcome this "noob" issue. I have two numpy arrays: >>> P array([[[ 2, 3], [33, 44], [22, 11], [ 1, 2]]]) >>> R array([0, 1, 2, 3]) the values of these may of course be different. The important fact is that: >>> P.shape (1, 4, 2) >>> R.shape (4,) where the number 4 in the shape of both P and R may be another number as well (same on both). What I'd like to get is a new array Q with same shape as P so that the nth pair of Q is the nth pair of P multiplied by the nth element of R. I.e., in the above case it should produce: >>> Q array([[[ 0, 0], [33, 44], [44, 22], [ 3, 6]]]) Is there a direct, single expression command to get this result? I have tried all I could imagine, including .T, extend_dims, h/vstack, repeat..., so far with no success. Please, any help will be welcomed. Thanks. -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract Indices of Numpy Array Based on Given Bit Information
On Sat, Oct 18, 2014 at 4:10 PM, Chris Angelico wrote: > On Sat, Oct 18, 2014 at 6:02 PM, Artur Bercik wrote: > > So, the Bit No. 2-5 for the following case is '1101', right? > > > > 1073741877: 1110101 > > > > If my required bit combination for Bit No. 2-5 is '1011', then the above > > number (1073741877) is not chosen, right?? > > > > Look forward to know your confirmation. > > (Side point: Please don't top-post. The convention on this mailing > list, and most other technical mailing lists, is what's sometimes > called "interleaved style"; you trim the quoted text to what's needed > for context, and put your text underneath what you're referring to. > See https://en.wikipedia.org/wiki/Posting_style#Interleaved_style for > more info.) > > >>> 1073741877&60 > 52 > >>> bin(52) > '0b110100' > > Everything I've written with the triple-angle-bracket marker can be > typed in at the Python prompt, and you'll see exactly what it does. In > this case, you can see that you're absolutely right: 1073741877 has > 1101 in those positions, so if you're looking for 1011, it won't > match: > > >>> 0b101100 > 44 > > However, 1073741869 would: > > >>> 1073741869&60 > 44 > > The way to test would be something like this: > > >>> (1073741877 & 0b00) == 0b101100 > False > >>> (1073741869 & 0b00) == 0b101100 > True > Thank you very much Chris Angelico, I have come to know it. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract Indices of Numpy Array Based on Given Bit Information
So, the Bit No. 2-5 for the following case is '1101', right? 1073741877: 1110101 If my required bit combination for Bit No. 2-5 is '1011', then the above number (1073741877) is not chosen, right?? Look forward to know your confirmation. On Sat, Oct 18, 2014 at 3:50 PM, Chris Angelico wrote: > On Sat, Oct 18, 2014 at 5:42 PM, Artur Bercik wrote: > > I got some sense, but could not imagine if required Bit No. 2–5, and Bit > > Combination . > > > > I hope example with the new case would make me more sense. > > > > Just write the number in binary, with the bits you're interested in > set to 1, and everything else 0: > > ... 876543210 > ... 00000 > > >>> 0b00000 > 60 > >>> 1073741877 & 0b00000 > 52 > > So this number does _not_ have all zeroes there. If it did, the result > would be zero. To look for some other pattern, just do the same thing > - suppose we want to find numbers where it's 1001, we just look for > the result to be 0b000100100, or 36. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract Indices of Numpy Array Based on Given Bit Information
Thanks Chris Angelico for your nice answer. I got some sense, but could not imagine if required Bit No. 2–5, and Bit Combination . I hope example with the new case would make me more sense. Artur On Sat, Oct 18, 2014 at 3:24 PM, Chris Angelico wrote: > On Sat, Oct 18, 2014 at 4:58 PM, Artur Bercik wrote: > > I want to get the index of my data where the following occurs: > > > > Bit No. 0–1 > > Bit Combination: 00 > > So, what you want to do is look at each number in binary, and find the > ones that have two zeroes in the last two places? > 1073741824: 100 > 1073741877: 1110101 > > The easiest way to do this is with simple bitwise operations: > > >>> 1073741824 & 3 > 0 > >>> 1073741877 & 3 > 1 > > 3 is 0011 in binary, so a bitwise AND with that will tell > you about just the last two bits. The result will be an integer - 0, > 1, 2, or 3, representing 00, 01, 10, or 11 for those last two bits. So > all you have to do is AND each number with 3, and when the result is > 0, that's what you want. > > Does that make sense? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract Indices of Numpy Array Based on Given Bit Information
On Sat, Oct 18, 2014 at 6:21 PM, Artur Bercik wrote: > Thank you very much Chris Angelico, I have come to know it. > You're most welcome. And thank you for taking heed of the request to not top-post. :) Hang around, you never know what weird and wonderful things you'll learn! -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract Indices of Numpy Array Based on Given Bit Information
On Sat, Oct 18, 2014 at 6:02 PM, Artur Bercik wrote: > So, the Bit No. 2-5 for the following case is '1101', right? > > 1073741877: 1110101 > > If my required bit combination for Bit No. 2-5 is '1011', then the above > number (1073741877) is not chosen, right?? > > Look forward to know your confirmation. (Side point: Please don't top-post. The convention on this mailing list, and most other technical mailing lists, is what's sometimes called "interleaved style"; you trim the quoted text to what's needed for context, and put your text underneath what you're referring to. See https://en.wikipedia.org/wiki/Posting_style#Interleaved_style for more info.) >>> 1073741877&60 52 >>> bin(52) '0b110100' Everything I've written with the triple-angle-bracket marker can be typed in at the Python prompt, and you'll see exactly what it does. In this case, you can see that you're absolutely right: 1073741877 has 1101 in those positions, so if you're looking for 1011, it won't match: >>> 0b101100 44 However, 1073741869 would: >>> 1073741869&60 44 The way to test would be something like this: >>> (1073741877 & 0b00) == 0b101100 False >>> (1073741869 & 0b00) == 0b101100 True ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract Indices of Numpy Array Based on Given Bit Information
On Sat, Oct 18, 2014 at 5:42 PM, Artur Bercik wrote: > I got some sense, but could not imagine if required Bit No. 2–5, and Bit > Combination . > > I hope example with the new case would make me more sense. > Just write the number in binary, with the bits you're interested in set to 1, and everything else 0: ... 876543210 ... 00000 >>> 0b00000 60 >>> 1073741877 & 0b00000 52 So this number does _not_ have all zeroes there. If it did, the result would be zero. To look for some other pattern, just do the same thing - suppose we want to find numbers where it's 1001, we just look for the result to be 0b000100100, or 36. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract Indices of Numpy Array Based on Given Bit Information
On Sat, Oct 18, 2014 at 4:58 PM, Artur Bercik wrote: > I want to get the index of my data where the following occurs: > > Bit No. 0–1 > Bit Combination: 00 So, what you want to do is look at each number in binary, and find the ones that have two zeroes in the last two places? 1073741824: 100 1073741877: 1110101 The easiest way to do this is with simple bitwise operations: >>> 1073741824 & 3 0 >>> 1073741877 & 3 1 3 is 0011 in binary, so a bitwise AND with that will tell you about just the last two bits. The result will be an integer - 0, 1, 2, or 3, representing 00, 01, 10, or 11 for those last two bits. So all you have to do is AND each number with 3, and when the result is 0, that's what you want. Does that make sense? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Extract Indices of Numpy Array Based on Given Bit Information
Dear Python and Numpy Users: My data are in the form of '32-bit unsigned integer' as follows: myData = np.array([1073741824, 1073741877, 1073742657, 1073742709, 1073742723, 1073755137, 1073755189,1073755969],dtype=np.int32) I want to get the index of my data where the following occurs: Bit No. 0–1 Bit Combination: 00 How can I do it? I heard this type of problem first time, please help me. Artur -- https://mail.python.org/mailman/listinfo/python-list
Re: Convert 3d NumPy array into 2d
On 08/27/2014 08:08 AM, phinn stuart wrote: Hi everyone, how can I convert (1L, 480L, 1440L) shaped numpy array into (480L, 1440L)? Thanks in the advance. phinn A simple assignment into the arrays shape does it: >>> a = numpy.zeros((1,480,1440)) >>> a.shape (1, 480, 1440) >>> a.shape = (480,1440) >>> a.shape (480, 1440) Gary Herron -- https://mail.python.org/mailman/listinfo/python-list
Convert 3d NumPy array into 2d
Hi everyone, how can I convert (1L, 480L, 1440L) shaped numpy array into (480L, 1440L)? Thanks in the advance. phinn -- https://mail.python.org/mailman/listinfo/python-list
Re: [SciPy-User] Convert 3d NumPy array into 2d
[source] <http://github.com/numpy/numpy/blob/v1.8.1/numpy/core/fromnumeric.py#L1072> <http://docs.scipy.org/doc/numpy/reference/generated/numpy.squeeze.html#numpy.squeeze> 2014-08-27 16:08 GMT+01:00 phinn stuart : > Hi everyone, how can I convert (1L, 480L, 1440L) shaped numpy array into > (480L, 1440L)? > If 'a' is your array then 'a.squeeze()' or 'a.squeeze(axis=0)' will do the job (see the docs at [1]). You can also say a.reshape(480, 1440) but obviously this is only useful if you know the exact dimensions of your array. Cheers, Max [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.squeeze.html -- https://mail.python.org/mailman/listinfo/python-list
Re: Numpy Array of Sets
Thank you very much! -- https://mail.python.org/mailman/listinfo/python-list
Re: Numpy Array of Sets
LJ wrote:
> Thank you for the reply.
>
> So, as long as I access and modify the elements of, for example,
>
> A=array([[set([])]*4]*3)
>
>
> as (for example):
>
> a[0][1] = a[0][1] | set([1,2])
>
> or:
>
> a[0][1]=set([1,2])
>
> then I should have no problems?
As long as you set (i. e. replace) elements you're fine, but modifying means
trouble. You can prevent accidental modification by using immutable values
-- in your case frozenset:
>>> b = numpy.array([[frozenset()]*4]*3)
>>> b[0,0].update("123")
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'frozenset' object has no attribute 'update'
Or you take the obvious approach and ensure that there are no shared values.
I don't know if there's a canonical form to do this in numpy, but
>>> a = numpy.array([[set()]*3]*4)
>>> a |= set()
works:
>>> assert len(set(map(id, a.flat))) == 3*4
--
https://mail.python.org/mailman/listinfo/python-list
Re: Numpy Array of Sets
Thank you for the reply. So, as long as I access and modify the elements of, for example, A=array([[set([])]*4]*3) as (for example): a[0][1] = a[0][1] | set([1,2]) or: a[0][1]=set([1,2]) then I should have no problems? -- https://mail.python.org/mailman/listinfo/python-list
Re: Numpy Array of Sets
LJ wrote: > Wolfgang, thank you very much for your reply. > > Following the example in the link, the problem appears: > >>>> A = [[0]*2]*3 You can see this as a shortcut for value = 0 inner = [value, value] A = [inner, inner, inner] When the value is mutable (like your original set) a modification of the value shows in all six entries. Likewise if you change the `inner` list the modification shows in all three rows. >>>> A > [[0, 0], [0, 0], [0, 0]] >>>> A[0][0] = 5 >>>> A > [[5, 0], [5, 0], [5, 0]] > > Now, if I use a numpy array: > >>>> d=array([[0]*2]*3) >>>> d > array([[0, 0], >[0, 0], >[0, 0]]) >>>> d[0][0]=5 >>>> d > array([[5, 0], >[0, 0], >[0, 0]]) > > > What is the difference here? Basically a numpy array doesn't reference the lists, it uses them to determine the required shape of the array. A simplified implementation might be class Array: def __init__(self, data): self.shape = (len(data), len(data[0])) self._data = [] for row in data: self._data.extend(row) def __getitem__(self, index): y, x = index return self._data[y * self.shape[1] + x] With that approach you may only see simultaneous changes of multiple entries when using mutable values. -- https://mail.python.org/mailman/listinfo/python-list
Re: Numpy Array of Sets
Wolfgang, thank you very much for your reply. Following the example in the link, the problem appears: >>> A = [[0]*2]*3 >>> A [[0, 0], [0, 0], [0, 0]] >>> A[0][0] = 5 >>> A [[5, 0], [5, 0], [5, 0]] Now, if I use a numpy array: >>> d=array([[0]*2]*3) >>> d array([[0, 0], [0, 0], [0, 0]]) >>> d[0][0]=5 >>> d array([[5, 0], [0, 0], [0, 0]]) What is the difference here? Thank you, -- https://mail.python.org/mailman/listinfo/python-list
Re: Numpy Array of Sets
On 25.05.2014 00:14, Robert Kern wrote: On 2014-05-24 23:05, Luis José Novoa wrote: Hi All, Hope you're doing great. One quick question. I am defining an array of sets using numpy as: a=array([set([])]*3) Has nothing to do with numpy, but the problem is exclusively with your innermost expression [set([])]*3. Now, if I want to add an element to the set in, lets say, a[0], and I use the .add(4) operation, which results in: with .add you are modifying the *existing* set. array([set([4]), set([4]), set([4])], dtype=object) which I do not want. If I use the union operator a[0] = a[0] | set([4]) here you are forming a *new* set and put it in a[0] replacing the old set at this position. then I obtain what I want: array([set([4]), set([]), set([])], dtype=object) Can anyone explain whay this happens? Same reason why you shouldn't make a list of lists like so: [[]]*3 https://docs.python.org/2/faq/programming.html#how-do-i-create-a-multidimensional-list The above link explains the underlying problem. Best, Wolfgang -- https://mail.python.org/mailman/listinfo/python-list
Re: Numpy Array of Sets
On 2014-05-24 23:05, Luis José Novoa wrote: Hi All, Hope you're doing great. One quick question. I am defining an array of sets using numpy as: a=array([set([])]*3) Now, if I want to add an element to the set in, lets say, a[0], and I use the .add(4) operation, which results in: array([set([4]), set([4]), set([4])], dtype=object) which I do not want. If I use the union operator a[0] = a[0] | set([4]) then I obtain what I want: array([set([4]), set([]), set([])], dtype=object) Can anyone explain whay this happens? Same reason why you shouldn't make a list of lists like so: [[]]*3 https://docs.python.org/2/faq/programming.html#how-do-i-create-a-multidimensional-list -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- https://mail.python.org/mailman/listinfo/python-list
Numpy Array of Sets
Hi All, Hope you're doing great. One quick question. I am defining an array of sets using numpy as: a=array([set([])]*3) Now, if I want to add an element to the set in, lets say, a[0], and I use the .add(4) operation, which results in: array([set([4]), set([4]), set([4])], dtype=object) which I do not want. If I use the union operator a[0] = a[0] | set([4]) then I obtain what I want: array([set([4]), set([]), set([])], dtype=object) Can anyone explain whay this happens? Thank you very much. -- https://mail.python.org/mailman/listinfo/python-list
Re: Convert numpy array to single number
Maybe something like this? >>> to_num = lambda array: np.sum(array * 2**np.arange(len(array)-1, -1, -1)) >>> to_num(np.array([1,0,1,0])) 10 2014-04-29 17:42 GMT+02:00 Tom P : > On 28.04.2014 15:04, mboyd02...@gmail.com wrote: > >> I have a numpy array consisting of 1s and zeros for representing binary >> numbers: >> >> e.g. >> >> >>> binary >> array([ 1., 0., 1., 0.]) >> >> I wish the array to be in the form 1010, so it can be manipulated. >> >> I do not want to use built in binary converters as I am trying to build >> my own. >> >> > Do you mean that each element in the array represents a power of two? > So array([ 1., 0., 1., 0.]) represents 2^3 + 2 = 6 decimal? > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Convert numpy array to single number
On 28.04.2014 15:04, mboyd02...@gmail.com wrote: I have a numpy array consisting of 1s and zeros for representing binary numbers: e.g. >>> binary array([ 1., 0., 1., 0.]) I wish the array to be in the form 1010, so it can be manipulated. I do not want to use built in binary converters as I am trying to build my own. Do you mean that each element in the array represents a power of two? So array([ 1., 0., 1., 0.]) represents 2^3 + 2 = 6 decimal? -- https://mail.python.org/mailman/listinfo/python-list
Re:Convert numpy array to single number
mboyd02...@gmail.com Wrote in message: > I have a numpy array consisting of 1s and zeros for representing binary > numbers: > > e.g. > > >>> binary > array([ 1., 0., 1., 0.]) > > I wish the array to be in the form 1010, so it can be manipulated. > > I do not want to use built in binary converters as I am trying to build my > own. > One thousand and ten is an int, not an array. So is 10, but it seems a more likely value you might be trying for. As for using the builtin binary converters, you'd better be a lot more specific, as you cannot even print an int in decimal form without utilizing them. The number is in binary already. Please copy the exact problem and constraints, as well as what you've written so far and what's wrong with it. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Convert numpy array to single number
On Mon, 28 Apr 2014 06:04:02 -0700, mboyd02255 wrote: > I have a numpy array consisting of 1s and zeros for representing binary > numbers: > > e.g. > > >>> binary > array([ 1., 0., 1., 0.]) > > I wish the array to be in the form 1010, so it can be manipulated. > > I do not want to use built in binary converters as I am trying to build > my own. Did you have a question, or are you just sharing? -- Steven D'Aprano http://import-that.dreamwidth.org/ -- https://mail.python.org/mailman/listinfo/python-list
Convert numpy array to single number
I have a numpy array consisting of 1s and zeros for representing binary numbers: e.g. >>> binary array([ 1., 0., 1., 0.]) I wish the array to be in the form 1010, so it can be manipulated. I do not want to use built in binary converters as I am trying to build my own. -- https://mail.python.org/mailman/listinfo/python-list
Re: numpy array operation
On 1/29/2013 1:49 PM, Alok Singhal wrote: On Tue, 29 Jan 2013 00:41:54 -0800, C. Ng wrote: Is there a numpy operation that does the following to the array? 1 2 ==> 4 3 3 4 2 1 Thanks in advance. How about: import numpy as np a = np.array([[1,2],[3,4]]) a array([[1, 2], [3, 4]]) a[::-1, ::-1] array([[4, 3], [2, 1]]) Nice. The regular Python equivalent is a = [[1,2],[3,4]] print([row[::-1] for row in a[::-1]]) >>> [[4, 3], [2, 1]] The second slice can be replaced with reversed(a), which returns an iterator, to get [row[::-1] for row in reversed(a)] The first slice would have to be list(reversed(a)) to get the same result. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: numpy array operation
On Tue, 29 Jan 2013 00:41:54 -0800, C. Ng wrote: > Is there a numpy operation that does the following to the array? > > 1 2 ==> 4 3 > 3 4 2 1 > > Thanks in advance. How about: >>> import numpy as np >>> a = np.array([[1,2],[3,4]]) >>> a array([[1, 2], [3, 4]]) >>> a[::-1, ::-1] array([[4, 3], [2, 1]]) -- http://mail.python.org/mailman/listinfo/python-list
Re: numpy array operation
On Tuesday, January 29, 2013 3:41:54 AM UTC-5, C. Ng wrote: > Is there a numpy operation that does the following to the array? > > > > 1 2 ==> 4 3 > > 3 4 2 1 > > > > Thanks in advance. >>> import numpy as np >>> a=np.array([[1,2],[3,4]]) >>> a array([[1, 2], [3, 4]]) >>> np.fliplr(np.flipud(a)) array([[4, 3], [2, 1]]) -- http://mail.python.org/mailman/listinfo/python-list
Re: numpy array operation
C. Ng wrote: > Is there a numpy operation that does the following to the array? > > 1 2 ==> 4 3 > 3 4 2 1 How about >>> a array([[1, 2], [3, 4]]) >>> a[::-1].transpose()[::-1].transpose() array([[4, 3], [2, 1]]) Or did you mean >>> a.reshape((4,))[::-1].reshape((2,2)) array([[4, 3], [2, 1]]) Or even >>> -a + 5 array([[4, 3], [2, 1]]) -- http://mail.python.org/mailman/listinfo/python-list
numpy array operation
Is there a numpy operation that does the following to the array? 1 2 ==> 4 3 3 4 2 1 Thanks in advance. -- http://mail.python.org/mailman/listinfo/python-list
Re: splitting numpy array unevenly
On 18/09/12 16:02:02, Wanderer wrote: > On Monday, September 17, 2012 7:43:06 PM UTC-4, Martin De Kauwe wrote: >> On Tuesday, September 18, 2012 8:31:09 AM UTC+10, Wanderer wrote: >>> I need to divide a 512x512 image array with the first horizontal >>> and vertical division 49 pixels in. Then every 59 pixels in after >>> that. hsplit and vsplit want to start at the edges and create a >>> bunch of same size arrays. Is there a command to chop off >>> different sized arrays? >> I don't know that I follow completely, but can't you just slice >> what you are after? >> x = np.random.rand(512*512).reshape(512,512) >> xx = x[0,:49] >> And put the rest of the slices in a loop...? > I was trying to avoid the loop. I figured it out. hsplit and vsplit > will work. I just need to give it a list of break points. I still > need a loop though. > breakPoints = range(49,512,59) > rowArrays = hsplit(InputArray, breakPoints) > OutArrays = [] > for r in rowArrays: > OutArrays.append(vsplit(r, breakPoints)) How about a list display: breakPoints = range(49,512,59) rowArrays = hsplit(InputArray, breakPoints) OutArrays = [vsplit(r, breakPoints) for r in rowArrays] In some sense, it's still a loop, but at least it doesn't look like one. Hope this helps, -- HansM -- http://mail.python.org/mailman/listinfo/python-list
Re: splitting numpy array unevenly
On Monday, September 17, 2012 7:43:06 PM UTC-4, Martin De Kauwe wrote: > On Tuesday, September 18, 2012 8:31:09 AM UTC+10, Wanderer wrote: > > > I need to divide a 512x512 image array with the first horizontal and > > vertical division 49 pixels in. Then every 59 pixels in after that. hsplit > > and vsplit want to start at the edges and create a bunch of same size > > arrays. Is there a command to chop off different sized arrays? > > > > > > > > > > > > Thanks > > > > I don't know that I follow completely, but can't you just slice what you are > after? > > > > x = np.random.rand(512*512).reshape(512,512) > > xx = x[0,:49] > > And put the rest of the slices in a loop...? I was trying to avoid the loop. I figured it out. hsplit and vsplit will work. I just need to give it a list of break points. I still need a loop though. breakPoints = range(49,512,59) rowArrays = hsplit(InputArray, breakPoints) OutArrays = [] for r in rowArrays: OutArrays.append(vsplit(r, breakPoints)) -- http://mail.python.org/mailman/listinfo/python-list
Re: splitting numpy array unevenly
On 17 September 2012 23:31, Wanderer wrote: > I need to divide a 512x512 image array with the first horizontal and > vertical division 49 pixels in. Then every 59 pixels in after that. hsplit > and vsplit want to start at the edges and create a bunch of same size > arrays. Is there a command to chop off different sized arrays? > hsplits = [49] sum_hsplits = 49 while sum_hsplits < 512: hsplits.append(sum_hsplits) sum_hsplits += 59 np.split(np.random.rand(512), hsplits) If I understand correctly, this is a one-dimensional version of what you want. Extending this to the second dimension should be quite easy. However, I am not sure if I do understand what you want. Could you elaborate on what you are trying to achieve? -- http://mail.python.org/mailman/listinfo/python-list
Re: splitting numpy array unevenly
On Tuesday, September 18, 2012 8:31:09 AM UTC+10, Wanderer wrote: > I need to divide a 512x512 image array with the first horizontal and vertical > division 49 pixels in. Then every 59 pixels in after that. hsplit and vsplit > want to start at the edges and create a bunch of same size arrays. Is there a > command to chop off different sized arrays? > > > > Thanks I don't know that I follow completely, but can't you just slice what you are after? x = np.random.rand(512*512).reshape(512,512) xx = x[0,:49] And put the rest of the slices in a loop...? -- http://mail.python.org/mailman/listinfo/python-list
splitting numpy array unevenly
I need to divide a 512x512 image array with the first horizontal and vertical division 49 pixels in. Then every 59 pixels in after that. hsplit and vsplit want to start at the edges and create a bunch of same size arrays. Is there a command to chop off different sized arrays? Thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: Convert ctypes 16 bit c_short array to a 32 bit numpy array
On Mar 24, 3:14 pm, Wanderer wrote: > I'm using ctypes to have a dll fill a buffer with 16 bit data. I then > want to convert this data to a numpy array. The code snippet below > converts the data from 16 bit to 32 bit, but two 16 bit numbers are > concatenated to make a 32 bit number and half the array is zero. > > Buffer = (c_short * byteSize)() > self.cam.Qframe.pBuffer = cast(pointer(Buffer), c_void_p) > perr = self.cam.GrabFrame() > image1 = np.frombuffer(Buffer, int) > xdim = self.cam.Qframe.width > ydim = self.cam.Qframe.height > image2 = image1.reshape(xdim, ydim) > > image2 looks like > > [[6291555 6357091 6160481 ..., 6488160 6226020 6553697] > [6488163 6422625 6684770 ..., 6422624 6553697 6553696] > [6488160 6357091 6226018 ..., 6815842 6422627 6553696] > ..., > [ 0 0 0 ..., 0 0 0] > [ 0 0 0 ..., 0 0 0] > [ 0 0 0 ..., 0 0 0]] > > How do convert 16 bit data to 32 bit data? > Thanks I figured it out. Buffer = (c_ubyte * byteSize)() self.cam.Qframe.pBuffer = cast(pointer(Buffer), c_void_p) perr = self.cam.GrabFrame() image1 = np.frombuffer(Buffer, np.uint16) xdim = self.cam.Qframe.width ydim = self.cam.Qframe.height image2 = image1.reshape(xdim, ydim) Though Eclipse thinks Buffer = (c_ubyte * byteSize)() is an error. -- http://mail.python.org/mailman/listinfo/python-list
Convert ctypes 16 bit c_short array to a 32 bit numpy array
I'm using ctypes to have a dll fill a buffer with 16 bit data. I then want to convert this data to a numpy array. The code snippet below converts the data from 16 bit to 32 bit, but two 16 bit numbers are concatenated to make a 32 bit number and half the array is zero. Buffer = (c_short * byteSize)() self.cam.Qframe.pBuffer = cast(pointer(Buffer), c_void_p) perr = self.cam.GrabFrame() image1 = np.frombuffer(Buffer, int) xdim = self.cam.Qframe.width ydim = self.cam.Qframe.height image2 = image1.reshape(xdim, ydim) image2 looks like [[6291555 6357091 6160481 ..., 6488160 6226020 6553697] [6488163 6422625 6684770 ..., 6422624 6553697 6553696] [6488160 6357091 6226018 ..., 6815842 6422627 6553696] ..., [ 0 0 0 ..., 0 0 0] [ 0 0 0 ..., 0 0 0] [ 0 0 0 ..., 0 0 0]] How do convert 16 bit data to 32 bit data? Thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading bz2 file into numpy array
Nobody wrote: > On Mon, 22 Nov 2010 11:37:22 +0100, Peter Otten wrote: > >>> is there a convenient way to read bz2 files into a numpy array? >> >> Try > >> f = bz2.BZ2File(filename) >> data = numpy.fromstring(f.read(), numpy.float32) > > That's going to hurt if the file is large. Yes, but memory usage will peak at about 2*sizeof(data), and most scripts need more data than just a single numpy.array. In short: the OP is unlikely to run into the problem. > You might be better off either extracting to a temporary file, or creating > a pipe with numpy.fromfile() reading the pipe and either a thread or > subprocess decompressing the data into the pipe. I like to keep it simple, so if available RAM turns out to be the limiting factor I think extracting the data into a temporary file is a good backup plan. Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading bz2 file into numpy array
On Mon, 22 Nov 2010 11:37:22 +0100, Peter Otten wrote: >> is there a convenient way to read bz2 files into a numpy array? > > Try > f = bz2.BZ2File(filename) > data = numpy.fromstring(f.read(), numpy.float32) That's going to hurt if the file is large. You might be better off either extracting to a temporary file, or creating a pipe with numpy.fromfile() reading the pipe and either a thread or subprocess decompressing the data into the pipe. E.g.: import os import threading class Pipe(threading.Thread): def __init__(self, f, blocksize = 65536): super(Pipe, self).__init__() self.f = f self.blocksize = blocksize rd, wr = os.pipe() self.rd = rd self.wr = wr self.daemon = True self.start() def run(self): while True: s = self.f.read(self.blocksize) if not s: break os.write(self.wr, s) os.close(self.wr) def make_real(f): return os.fdopen(Pipe(f).rd, 'rb') Given the number of situations where you need a "real" (OS-level) file handle or descriptor rather than a Python "file-like object", something like this should really be part of the standard library. -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading bz2 file into numpy array
Johannes Korn wrote: > I tried: > > from bz2 import * > from numpy import * > fd = BZ2File(filename, 'rb') > read_data = fromfile(fd, float32) > > but BZ2File doesn't seem to produce a transparent filehandle. > is there a convenient way to read bz2 files into a numpy array? Try import numpy import bz2 filename = ... f = bz2.BZ2File(filename) data = numpy.fromstring(f.read(), numpy.float32) print data -- http://mail.python.org/mailman/listinfo/python-list
Reading bz2 file into numpy array
Hi, is there a convenient way to read bz2 files into a numpy array? I tried: from bz2 import * from numpy import * fd = BZ2File(filename, 'rb') read_data = fromfile(fd, float32) but BZ2File doesn't seem to produce a transparent filehandle. Kind regards! Johannes -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing.Array bug / shared numpy array
On 2009-10-08 15:14 PM, Felix wrote:
I am trying to create a shared, read-only numpy.ndarray between
several processes. After some googling the basic idea is:
sarr = mp.Array('i',1000)
ndarr = scipy.frombuffer(sarr._obj,dtype='int32')
Since it will be read only (after being filled once in a single
process) I don't think I need any locking mechanism. However is this
really true given garbage collection, reference counts and other
implicit things going on?
Or is there a recommended better way to do this?
I recommend using memory-mapped arrays for such a purpose.
You will want to ask further numpy questions on the numpy mailing list:
http://www.scipy.org/Mailing_Lists
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Multiprocessing.Array bug / shared numpy array
Hi,
The documentation for the Multiprocessing.Array says:
multiprocessing.Array(typecode_or_type, size_or_initializer, *,
lock=True)¶
...
If lock is False then access to the returned object will not be
automatically protected by a lock, so it will not necessarily be
“process-safe”.
...
However:
In [48]: mp.Array('i',1,lock=False)
---
AssertionErrorTraceback (most recent call
last)
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/
multiprocessing/__init__.pyc in Array(typecode_or_type,
size_or_initializer, **kwds)
252 '''
253 from multiprocessing.sharedctypes import Array
--> 254 return Array(typecode_or_type, size_or_initializer,
**kwds)
255
256 #
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/
multiprocessing/sharedctypes.pyc in Array(typecode_or_type,
size_or_initializer, **kwds)
85 if lock is None:
86 lock = RLock()
---> 87 assert hasattr(lock, 'acquire')
88 return synchronized(obj, lock)
89
AssertionError:
---
I.e. it looks like lock=false is not actually supported. Or am I
reading this wrong? If not, I can submit a bug report.
I am trying to create a shared, read-only numpy.ndarray between
several processes. After some googling the basic idea is:
sarr = mp.Array('i',1000)
ndarr = scipy.frombuffer(sarr._obj,dtype='int32')
Since it will be read only (after being filled once in a single
process) I don't think I need any locking mechanism. However is this
really true given garbage collection, reference counts and other
implicit things going on?
Or is there a recommended better way to do this?
Thanks
--
http://mail.python.org/mailman/listinfo/python-list
Re: numpy array merge
On 2009-08-10 17:38, Nathan wrote: Is there an easy way to merge two numpy arrays with different rank sizes (terminology?). You will want to ask numpy questions on the numpy mailing list. http://www.scipy.org/Mailing_Lists I believe that "shape" is the term you are looking for. I want to make a single array by concatenating two arrays along a given direction and filling the excess cells with a dummy variable. numpy concatenate works well as long as the two arrays have the same dimension, but I want to do this on two array with a matching dimension size. An example: import numpy as np a = np.array([[1,2,3],[4,5,6]]) b = np.array([[1,2],[3,4],[5,6]]) np.concatenate((a,a),axis=1) #This works fine, but isn't what I need. Out: array([[1, 2, 3, 1, 2, 3], [4, 5, 6, 4, 5, 6]]) np.concatenate((a,b),axis=1) #This doesn't work, but this is what I need. I want to fill the third row of array "a" with a dummy variable (9 or NaN) and concatenate with "b" to make: [1,2,3,1,2] [4,5,6,4,5] [9,9,9,5,6] This just seems like it would be relatively common. So I thought I'd check if I'm missing something before I go write the logic. Thanks! I don't believe there is anything that does what you want out-of-box. You will probably want to determine the appropriate shape for the final array, use empty() to create it, use .fill(nan) (or whatever) to supply the default value, then assign the input arrays into the appropriate locations. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
numpy array merge
Is there an easy way to merge two numpy arrays with different rank sizes (terminology?). I want to make a single array by concatenating two arrays along a given direction and filling the excess cells with a dummy variable. numpy concatenate works well as long as the two arrays have the same dimension, but I want to do this on two array with a matching dimension size. An example: import numpy as np a = np.array([[1,2,3],[4,5,6]]) b = np.array([[1,2],[3,4],[5,6]]) np.concatenate((a,a),axis=1) #This works fine, but isn't what I need. Out: array([[1, 2, 3, 1, 2, 3], [4, 5, 6, 4, 5, 6]]) np.concatenate((a,b),axis=1) #This doesn't work, but this is what I need. I want to fill the third row of array "a" with a dummy variable (9 or NaN) and concatenate with "b" to make: [1,2,3,1,2] [4,5,6,4,5] [9,9,9,5,6] This just seems like it would be relatively common. So I thought I'd check if I'm missing something before I go write the logic. Thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: clean way prepend an element to a numpy array
On 2009-07-24 18:26, greg wrote:
Robert Kern wrote:
a[:0] = array('i', [0])
Not when 'a' is a numpy array rather than an array.array.
That's true, but I got the impression that the OP was
talking about array.array, not numpy.array.
Did you read the Subject: line? :-)
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Re: clean way prepend an element to a numpy array
Robert Kern wrote:
a[:0] = array('i', [0])
Not when 'a' is a numpy array rather than an array.array.
That's true, but I got the impression that the OP was
talking about array.array, not numpy.array.
It's very confusing having two widely-used types
both called 'array'. :-(
--
Greg
--
http://mail.python.org/mailman/listinfo/python-list
Re: clean way prepend an element to a numpy array
On 2009-07-22 01:51, greg wrote:
bdb112 wrote:
I saw this interest syntax on this site
x[:0]=0
I guess that is cute, but could be confusing(and doesn't work)
It does if you use an array of the appropriate
type on the right hand side:
a[:0] = array('i', [0])
Not when 'a' is a numpy array rather than an array.array.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Re: clean way prepend an element to a numpy array
bdb112 wrote:
I saw this interest syntax on this site
x[:0]=0
I guess that is cute, but could be confusing(and doesn't work)
It does if you use an array of the appropriate
type on the right hand side:
a[:0] = array('i', [0])
--
Greg
--
http://mail.python.org/mailman/listinfo/python-list
Re: clean way prepend an element to a numpy array
On 2009-07-20 22:56, bdb112 wrote: If I want to add an element at the beginning of an array, it seems like I must make a list, insert in place, then make an array again. the_array = numpy.concatenate([new_value, the_array]) You will want to ask numpy questions on the numpy mailing list. http://www.scipy.org/Mailing_Lists -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: clean way prepend an element to a numpy array
On Jul 21, 2:13 pm, Ben Finney wrote: > bdb112 writes: > > If I want to add an element at the beginning of an array, it seems > > like I must make a list, insert in place, then make an array again. > > The NumPy ‘ndarray’ type (which is what you get by default from the > ‘array’ factory function) is a far more complex type than (and is not > derived from) the Python list. > > For manipulating them, you'll want to study the NumPy documentation > http://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html>. > Yes, I had had a look through that - nothing there that allows writing a succint clear statement (except for my .resize example in the original post), although if efficiency was a concern, then maybe resize and shift right, but that would really obscure the code. -- http://mail.python.org/mailman/listinfo/python-list
Re: clean way prepend an element to a numpy array
bdb112 writes: > If I want to add an element at the beginning of an array, it seems > like I must make a list, insert in place, then make an array again. The NumPy ‘ndarray’ type (which is what you get by default from the ‘array’ factory function) is a far more complex type than (and is not derived from) the Python list. For manipulating them, you'll want to study the NumPy documentation http://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html>. -- \ “Are you pondering what I'm pondering?” “I think so, Brain, but | `\I don't think Kay Ballard's in the union.” —_Pinky and The | _o__) Brain_ | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
clean way prepend an element to a numpy array
If I want to add an element at the beginning of an array, it seems like I must make a list, insert in place, then make an array again. Of course, lists can be efficient too, and the insert() funtion works nicely in that case, but sometimes arrays are the best choice e.g. x=array([1,2,3]) # to put the element 0 at the head of the array listx=list(x) listx.insert(0,0) x=array(listx) # this extra effort distracts from the clarity of the code, and must slow things a little. # While I appreciate that array operations that cause memory re- allocation and copying are to be # avoided if at all possible, sometimes this is the best compromise between clarity and efficiency # A nicer piece of code would be x.insert(0,0) #or x.prepend(0) # It is a little easier to grow an array at the end (if it is not referenced) x.resize(4) I saw this interest syntax on this site x[:0]=0 I guess that is cute, but could be confusing(and doesn't work) -- http://mail.python.org/mailman/listinfo/python-list
Re: deleting certain entries in numpy array
On 2009-07-02 04:40, Sebastian Schabe wrote: Robert Kern schrieb: > You will want to ask numpy questions on the numpy mailing list. > > http://www.scipy.org/Mailing_Lists > I ever thought news-groups are the right way for questions like this. And the mailing list page just confuses me, but I'm trying to get used to it. You may want to try GMane, then: http://dir.gmane.org/gmane.comp.python.numeric.general It's how I read this list, for instance. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: deleting certain entries in numpy array
Robert Kern schrieb: First, convert the pos array to integers, and just the columns with indices in them: ipos = pos[:,:2].astype(int) Now check the values in the mask corresponding to these positions: mask_values = mask[ipos[:,0], ipos[:,1]] Now extract the rows from the original pos array where mask_values is nonzero: result = pos[mask_values != 0] Great!!! That's the way I wanted. After reading the numpy reference guide I supposed the delete function was not the right way, but that it's all about cerrect indexing. But I didn't really know how. So thanks a lot, also to Ben. > > You will want to ask numpy questions on the numpy mailing list. > > http://www.scipy.org/Mailing_Lists > I ever thought news-groups are the right way for questions like this. And the mailing list page just confuses me, but I'm trying to get used to it. Sebastian -- http://mail.python.org/mailman/listinfo/python-list
Re: deleting certain entries in numpy array
On 2009-07-01 09:51, Sebastian Schabe wrote: Hello everybody, I'm new to python and numpy and have a little/special problem: You will want to ask numpy questions on the numpy mailing list. http://www.scipy.org/Mailing_Lists I have an numpy array which is in fact a gray scale image mask, e.g.: mask = array([[ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 255, 255, 255, 0, 0, 255, 0], [ 0, 0, 255, 255, 255, 0, 0, 255, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=uint8) and I have another array of (y, x) positions in that mask (first two values of each row): pos = array([[ 3., 2., 0., 0.], [ 3., 4., 0., 0.], [ 5., 2., 0., 0.], [ 5., 4., 0., 0.], [ 6., 2., 0., 0.], [ 6., 7., 0., 0.], [ 0., 0., 0., 0.], [ 8., 8., 0., 0.]]) and now I only want to keep all lines from 2nd array pos with those indices that are nonzero in the mask, i.e. line 3-6 (pos[2]-pos[5]). F.e. line 4 in pos has the values (5, 4) and mask[5][4] is nonzero, so I want to keep it. While line 2 (pos[1]) has the values (4, 6) and mask[4][6] is zero, so shall be discarded. I want to avoid a for loop (if possible!!!) cause I think (but don't know) numpy array are handled in another way. I think numpy.delete is the right function for discarding the values, but I don't know how to build the indices. First, convert the pos array to integers, and just the columns with indices in them: ipos = pos[:,:2].astype(int) Now check the values in the mask corresponding to these positions: mask_values = mask[ipos[:,0], ipos[:,1]] Now extract the rows from the original pos array where mask_values is nonzero: result = pos[mask_values != 0] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: deleting certain entries in numpy array
Sebastian Schabe writes: > I want to avoid a for loop (if possible!!!) cause I think (but don't > know) numpy array are handled in another way. Yes, Numpy arrays can be indexed logically by a boolean array. > I think numpy.delete is the right function for discarding the values, > but I don't know how to build the indices. You don't need to discard the values. You can get a new array which is a filtered version of an existing array, by using an array of bool values as the index to the old array:: >>> import numpy >>> mask = numpy.array([ ... [ 0, 0, 0, 0, 0, 0, 0, 0, 0], ... [ 0, 0, 0, 0, 0, 0, 0, 0, 0], ... [ 0, 0, 0, 0, 0, 0, 0, 0, 0], ... [ 0, 0, 0, 0, 0, 0, 0, 0, 0], ... [ 0, 0, 0, 0, 0, 0, 0, 0, 0], ... [ 0, 0, 255, 255, 255, 0, 0, 255, 0], ... [ 0, 0, 255, 255, 255, 0, 0, 255, 0], ... [ 0, 0, 0, 0, 0, 0, 0, 0, 0], ... [ 0, 0, 0, 0, 0, 0, 0, 0, 0], ... [ 0, 0, 0, 0, 0, 0, 0, 0, 0], ... ], dtype=numpy.uint8) >>> mask > 0 array([[False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False], [False, False, True, True, True, False, False, True, False], [False, False, True, True, True, False, False, True, False], [False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False]], dtype=bool) >>> mask[mask > 0] array([255, 255, 255, 255, 255, 255, 255, 255], dtype=uint8) However, your case is somewhat more tricky: you need to construct the boolean array based on coordinates from a separate array. That doesn't require a for loop statement, but AFAICT it does require manually generating the array of bools. I've done it with a list comprehension:: >>> pos = numpy.array([ ... [ 3., 2., 0., 0.], ... [ 3., 4., 0., 0.], ... [ 5., 2., 0., 0.], ... [ 5., 4., 0., 0.], ... [ 6., 2., 0., 0.], ... [ 6., 7., 0., 0.], ... [ 0., 0., 0., 0.], ... [ 8., 8., 0., 0.], ... ]) >>> pos[numpy.array( ... [mask[int(x), int(y)] > 0 for (x, y) in pos[:, 0:2]])] array([[ 5., 2., 0., 0.], [ 5., 4., 0., 0.], [ 6., 2., 0., 0.], [ 6., 7., 0., 0.]]) Here it is again, showing my working steps:: >>> pos[:, 0:2] array([[ 3., 2.], [ 3., 4.], [ 5., 2.], [ 5., 4.], [ 6., 2.], [ 6., 7.], [ 0., 0.], [ 8., 8.]]) >>> [(int(x), int(y)) for (x, y) in pos[:, 0:2]] [(3, 2), (3, 4), (5, 2), (5, 4), (6, 2), (6, 7), (0, 0), (8, 8)] >>> [mask[int(x), int(y)] for (x, y) in pos[:, 0:2]] [0, 0, 255, 255, 255, 255, 0, 0] >>> [mask[int(x), int(y)] > 0 for (x, y) in pos[:, 0:2]] [False, False, True, True, True, True, False, False] >>> numpy.array( ... [mask[int(x), int(y)] > 0 for (x, y) in pos[:, 0:2]]) array([False, False, True, True, True, True, False, False], dtype=bool) >>> pos[numpy.array( ... [mask[int(x), int(y)] > 0 for (x, y) in pos[:, 0:2]])] array([[ 5., 2., 0., 0.], [ 5., 4., 0., 0.], [ 6., 2., 0., 0.], [ 6., 7., 0., 0.]]) -- \“Holy knit one purl two, Batman!” —Robin | `\ | _o__) | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
deleting certain entries in numpy array
Hello everybody, I'm new to python and numpy and have a little/special problem: I have an numpy array which is in fact a gray scale image mask, e.g.: mask = array([[ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 255, 255, 255, 0, 0, 255, 0], [ 0, 0, 255, 255, 255, 0, 0, 255, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=uint8) and I have another array of (y, x) positions in that mask (first two values of each row): pos = array([[ 3., 2., 0., 0.], [ 3., 4., 0., 0.], [ 5., 2., 0., 0.], [ 5., 4., 0., 0.], [ 6., 2., 0., 0.], [ 6., 7., 0., 0.], [ 0., 0., 0., 0.], [ 8., 8., 0., 0.]]) and now I only want to keep all lines from 2nd array pos with those indices that are nonzero in the mask, i.e. line 3-6 (pos[2]-pos[5]). F.e. line 4 in pos has the values (5, 4) and mask[5][4] is nonzero, so I want to keep it. While line 2 (pos[1]) has the values (4, 6) and mask[4][6] is zero, so shall be discarded. I want to avoid a for loop (if possible!!!) cause I think (but don't know) numpy array are handled in another way. I think numpy.delete is the right function for discarding the values, but I don't know how to build the indices. Maybe someone can help me with that or suggest which way to do this Sebastian -- http://mail.python.org/mailman/listinfo/python-list
Re: numpy array sorting weirdness
>> Is there any reason the 'axis' keyword argument doesn't default to the >> value that corresponds to python list behaviour? That would make lot >> of sense I think. Or retaining compatibility with python lists is not >> really a goal of numpy.array? > > Not at all. It's an entirely different data structure. > Thanks a lot! Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown -- http://mail.python.org/mailman/listinfo/python-list
