Re: [Numpy-discussion] Slice assignment and overlapping views (was: Strange behavior in setting masked array values in Numpy 1.1.0)
2008/5/31 Pauli Virtanen <[EMAIL PROTECTED]>: > The reason for the strange behavior of slice assignment is that when the > left and right sides in a slice assignment are overlapping views of the > same array, the result is currently effectively undefined. Same is true > for ndarrays: > import numpy a = numpy.array([1, 2, 3, 4, 5]) a[::-1] > array([5, 4, 3, 2, 1]) a[:] = a[::-1] a > array([5, 4, 3, 4, 5]) I think that the current rule is, slices are walked from low index to high index. This doesn't help with multidimensional arrays, where the order of the axes is (and should be) determined by efficiency considerations. Unfortunately there's really no good way to warn about overlapping copies. Remember that this is a frequent operation, so it has to be fast for small arrays. I think changing base so that it points to the real base and not the parent would help (and clear up a memory leak: try "while True: A = A[::-1]" some time) eliminate some cases where overlap cannot occur, but what about the following cases? A[:5] = A[-5:] A[::2] = A[1::2] A[1:] = A[-1:] The last is actually fairly common (I've needed it), and relies on numpy's ordering of copies. The middle one is very common, and the first one would be a royal pain to code around if the slices were not allowed to overlap. I think I at one point wrote an algorithm that would detect many cases where overlap could not occur (maybe in the smarter reshape code?) but I came to the conclusion that detecting all the cases was infeasible. It's a number-theoretic problem - "can you write the same number as the sum of multiples of these two lists of numbers with nonnegative integer coefficients less than these other two lists of numbers?" - and I suspect it's NP-complete. (Ah, yes, you can make a knapsack out of it - take an array with strides a_1, ... a_n and shape (2,...,2), and a single-element array starting at position N.) In any case it's infeasible to solve it every time somebody tries to do a slice assignment. In any case, many users need nearly-overlapping slices, and some need really-overlapping slices. Preventing problems is going to have to happen at a higher level. Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Slice assignment and overlapping views (was: Strange behavior in setting masked array values in Numpy 1.1.0)
su, 2008-06-01 kello 03:37 -0400, Anne Archibald kirjoitti:
> 2008/5/31 Pauli Virtanen <[EMAIL PROTECTED]>:
>
> > The reason for the strange behavior of slice assignment is that when
> the
> > left and right sides in a slice assignment are overlapping views of
> the
> > same array, the result is currently effectively undefined. Same is
> true
> > for ndarrays:
> >
> import numpy
> a = numpy.array([1, 2, 3, 4, 5])
> a[::-1]
> > array([5, 4, 3, 2, 1])
> a[:] = a[::-1]
> a
> > array([5, 4, 3, 4, 5])
>
> I think that the current rule is, slices are walked from low index to
> high index. This doesn't help with multidimensional arrays, where the
> order of the axes is (and should be) determined by efficiency
> considerations.
Yes, I figured that it is not possible to do what the user usually means
in all cases without using at least one scalar temporary.
[clip]
> Unfortunately there's really no good way to warn about overlapping
> copies. Remember that this is a frequent operation, so it has to be
> fast for small arrays.
I didn't actually mean detecting overlaps, but detecting when assigning
to a view:
while (a->base) a = a->base;
while (b->base) b = b->base;
if (a == b) {
raise warning, but continue
}
The leading loops can be removed if the bases are walked upwards on
array creation, and in this case an added single branch cannot be so bad
of an performance hit. If the branch is taken, then there's a
performance hit of course.
> I think changing base so that it points to the real base and not the
> parent would help (and clear up a memory leak: try "while True: A =
> A[::-1]" some time) eliminate some cases where overlap cannot occur,
> but what about the following cases?
>
> A[:5] = A[-5:]
> A[::2] = A[1::2]
> A[1:] = A[-1:]
>
> The last is actually fairly common (I've needed it), and relies on
> numpy's ordering of copies. The middle one is very common, and the
> first one would be a royal pain to code around if the slices were not
> allowed to overlap.
But maybe the warning is still too much hand-holding, and we can just
add a warning about undefined behavior in the documentation and leave it
at that. I can't argue much against your examples.
If the behavior in 1d for overlapping parts is well-defined and relied
upon, we should add a unit test that ensures it, if there isn't one yet
(didn't check yet).
[clip]
> In any case, many users need nearly-overlapping slices, and some need
> really-overlapping slices. Preventing problems is going to have to
> happen at a higher level.
"Higher level" means "Documentation" here, right?
Pauli
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What does "Ignoring attempt to set 'name' (from ... " mean ?
On Thu, May 29, 2008 at 6:58 AM, David Cournapeau <[EMAIL PROTECTED]> wrote: > Robert Kern wrote: >> >> They're fine. Ignore them. They are silenced from the main setup.py with >> >> config.set_options(quiet=True) >> > What are the cases where those message are meaningful ? I did not > understand from the distutils code what kind of issues were related to > this message, There probably aren't any significant ones in production. It was probably useful during development of the recursive setup.py feature. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: PyQwt-5.1.0 released
What is PyQwt ( http://pyqwt.sourceforge.net ) ? - it is a set of Python bindings for the Qwt C++ class library which extends the Qt framework with widgets for scientific and engineering applications. It provides a 2-dimensional plotting widget and various widgets to display and control bounded or unbounded floating point values. - it requires and extends PyQt, a set of Python bindings for Qt. - it supports the use of PyQt, Qt, Qwt, and optionally NumPy or SciPy in a GUI Python application or in an interactive Python session. - it runs on POSIX, Mac OS X and Windows platforms (practically any platform supported by Qt and Python). - it plots fast: fairly good hardware allows a rate of 100,000 points/second. (PyQwt with Qt-3 is faster than with Qt-4). - it is licensed under the GPL with an exception to allow dynamic linking with non-free releases of Qt and PyQt. The most important new features of PyQwt-5.1.0 are: - support for Qwt-5.1.0. - support for PyQt-4.4.2 and SIP-4.7.6. - support for Qt-4.4. - the CartesianDemo.py, MaskedDataDemo.py, PickerDemo.py examples. The most important bug fixes in PyQwt-5.1.0 are: - fixed QwtPicker::stateMachine() to allow for subclassing of QwtPicker in Python. PyQwt-5.1.0 supports: 1. Python-2.5, or -2.4. 2. PyQt-3.17. 3. PyQt-4.4, PyQt-4.3, or PyQt-4.2. 3 SIP-4.7, or SIP-4.6. 4. Qt-3.3, or Qt-3.2. 5. Qt-4.4, Qt-4.3, or Qt-4.2. 6. Qwt-5.1, or Qwt-5.0. 7. Recent versions of NumPy, numarray, and/or Numeric. Enjoy -- Gerard Vermeulen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Slice assignment and overlapping views (was: Strange behavior in setting masked array values in Numpy 1.1.0)
On Sun, Jun 1, 2008 at 1:37 AM, Anne Archibald <[EMAIL PROTECTED]> wrote: > 2008/5/31 Pauli Virtanen <[EMAIL PROTECTED]>: > > > The reason for the strange behavior of slice assignment is that when the > > left and right sides in a slice assignment are overlapping views of the > > same array, the result is currently effectively undefined. Same is true > > for ndarrays: > > > import numpy > a = numpy.array([1, 2, 3, 4, 5]) > a[::-1] > > array([5, 4, 3, 2, 1]) > a[:] = a[::-1] > a > > array([5, 4, 3, 4, 5]) > > I think that the current rule is, slices are walked from low index to > high index. This doesn't help with multidimensional arrays, where the > order of the axes is (and should be) determined by efficiency > considerations. > > Unfortunately there's really no good way to warn about overlapping > copies. Remember that this is a frequent operation, so it has to be > fast for small arrays. > > I think changing base so that it points to the real base and not the > parent would help (and clear up a memory leak: try "while True: A = > A[::-1]" some time) eliminate some cases where overlap cannot occur, > but what about the following cases? > > A[:5] = A[-5:] > A[::2] = A[1::2] > A[1:] = A[-1:] > > The last is actually fairly common (I've needed it), and relies on > numpy's ordering of copies. The middle one is very common, and the > first one would be a royal pain to code around if the slices were not > allowed to overlap. > > I think I at one point wrote an algorithm that would detect many cases > where overlap could not occur (maybe in the smarter reshape code?) but > I came to the conclusion that detecting all the cases was infeasible. > It's a number-theoretic problem - "can you write the same number as > the sum of multiples of these two lists of numbers with nonnegative > integer coefficients less than these other two lists of numbers?" - > and I suspect it's NP-complete. (Ah, yes, you can make a knapsack out > of it - take an array with strides a_1, ... a_n and shape (2,...,2), > and a single-element array starting at position N.) In any case it's > infeasible to solve it every time somebody tries to do a slice > assignment. > Since one could compute all the addresses and check for duplicates, I would guess it is O(n), maybe O(n*log(n)) worst case, but I can't convince myself that it would help much to know about overlaps. As you point out, a lot of times one needs overlapping ranges, it is when the user has designed an algorithm without taking the possibility into account that problems arise. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] array += masked_array gives a normal array, not masked array
Ok, so you guys shot down my last attempt at finding a bug :). Here's another attempt. array + masked_array outputs a masked array array += masked_array outputs an array. I'm actually not sure if this is a bug (works the same for both the old and new masked arrays), but I thought this was unexpected. *However*, if the right side were an old masked array, the masked values would not be added to the output array. With the new masked arrays, the data from the masked array is added regardless of whether the value is masked (example, below). -Tony Example: === In [1]: import numpy In [2]: normal = numpy.array([1, 1]) In [3]: masked = numpy.ma.array([1, 1], mask=[True, False]) In [4]: new_masked = normal + masked In [5]: new_masked Out[5]: masked_array(data = [-- 2], mask = [ True False], fill_value=99) In [6]: normal += masked In [7]: normal Out[7]: array([2, 2]) # If used old masked arrays in the above, the final output would be: array([1, 2]) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] array += masked_array gives a normal array, not masked array
On Sunday 01 June 2008 11:22:05 Tony Yu wrote: > array + masked_array > outputs a masked array That's expected: you create a new object from two objects (array and masked_array), the latter has a higher priority than the former, you end up with a masked array > array += masked_array > outputs an array. That's expected as well: you modify the values of array in place, but don't modify its type. Similarly, masked_array += array gives a masked array. > *However*, if the right side were an old masked array, the masked > values would not be added to the output array. With the new masked > arrays, the data from the masked array is added regardless of whether > the value is masked (example, below). With oldnumeric.ma, masked_array has to be converted to a ndarray. During the conversion, the masked data are filled with a value that depends on the context of the call: here, the function called is add, and the corresponding filling value is 0. Your line normal += masked_array is then equivalent to normal += [0,1] With numpy.ma, masked_array is already recognized as a ndarray, and no conversion takes place. In particular, the values of masked_array are not filled. You should use normal += masked_array.filled(0) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy import problem
Dear Alan, I am using python 2.4.3 that came pre-installed with my linux. I dnt have any other version. But i noticed another problem. In fact no other modules are getting imported now. I have pasted the error below. Suggest me what i should do now. $ python 'import site' failed; use -v for traceback Python 2.4.3 (#1, Aug 8 2006, 18:51:20) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-20)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in ? ImportError: No module named numpy >>> import string Traceback (most recent call last): File "", line 1, in ? ImportError: No module named string On Sat, May 31, 2008 at 8:45 PM, Alan McIntyre <[EMAIL PROTECTED]> wrote: > Bala, > > One thing I can think of is that you might have multiple versions of > Python installed. For example, I have Python 2.4 and 2.5 on my > machine, but numpy is only installed for 2.5. Since just running > "python" brings up 2.4, sometimes I find myself in the wrong > interpreter typing "import numpy", which fails in the way you > described. > > Cheers, > Alan > > On Sat, May 31, 2008 at 10:52 AM, Bala subramanian > <[EMAIL PROTECTED]> wrote: > > Dear friends, > > > > I installed numpy in a 32-bit machines running with RHEL3. The > installation > > was successful. I tested the installtion by importint numpy inside python > > interpreter. > > By once i shutdown the system and restart, and try the same, it says > > ImportError: No module named numpy. > > > > What could be the problem i dnt get. Kindly someone write me on the same. > > > > Thanks, > > Bala > > > > > > ___ > > Numpy-discussion mailing list > > Numpy-discussion@scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy import problem
On Mon, 2008-06-02 at 10:49 +0530, Bala subramanian wrote: > Dear Alan, > I am using python 2.4.3 that came pre-installed with my linux. I dnt > have any other version. But i noticed another problem. In fact no > other modules are getting imported now. I have pasted the error below. > Suggest me what i should do now. > That's strange, you definitely have something wrong with your python. Could you try this: python -c "import sys; print sys.path" And also tell us what your environment look like ? (just dump it using the env command if you are using bash). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
