[Numpy-discussion] failed to add routine to the core module
Dear Numpy developers,
I'm trying to add a routine to calculate the sum of a product of two
arrays (a dot product). But that would not increase the memory (from
what I saw np.dot is increasing the memory while it should not be
necessary). The idea is to avoid the use of the temporary array in the
calculation of the variance ( numpy/numpy/core/_methods.py line 112).
The routine that I want to implement look like this in python,
|arr = np.random.rand(10)|
|mean = arr.mean()|
|var = 0.0|
|for ai in arr: var += (ai-mean)**2|
I would like to implement it in the umath module. As a first step, I
tried to reproduce the divmod function of umath, but I did not manage to
do it, you can find my fork here
<https://github.com/mbarbry/numpy/tree/looking_around> (the branch with
the changes is call looking_around). During compilation I get the
following error,
|gcc: numpy/core/src/multiarray/number.c
In file included from numpy/core/src/multiarray/number.c:17:0:
numpy/core/src/multiarray/number.c: In function ‘array_sum_multiply’:
numpy/core/src/private/binop_override.h:176:39: error: ‘PyNumberMethods
{aka struct }’ has no member named ‘nb_sum_multiply’
(void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func))
^
numpy/core/src/private/binop_override.h:180:13: note: in expansion of
macro ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr,
test_func) && \
^
numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro
‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2,
nb_sum_multiply, array_sum_multiply);|
Sorry if my question seems basic, but I'm new in Numpy development.
Any help?
Thank you in advance,
Marc Barbry
PS: I opened an issues as well on the github repository
https://github.com/numpy/numpy/issues/9130
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] failed to add routine to the core module
Hello Marten,
Thank you for your help, effectively, the example that you propose is
much easier to imitate, I can now continue further.
Thanks,
Marc
On 05/18/2017 04:32 PM, Marten van Kerkwijk wrote:
Hi Marc,
ufuncs are quite tricky to compile. Part of your problem is that, I
think, you started a bit too high up: `divmod` is also a binary
operation, so that part you do not need at all. It may be an idea to
start instead with a PR that implemented a new ufunc, e.g.,
https://github.com/numpy/numpy/pull/8795, so that you can see what is
involved.
All the best,
Marten
On Thu, May 18, 2017 at 9:04 AM, marc wrote:
Dear Numpy developers,
I'm trying to add a routine to calculate the sum of a product of two arrays
(a dot product). But that would not increase the memory (from what I saw
np.dot is increasing the memory while it should not be necessary). The idea
is to avoid the use of the temporary array in the calculation of the
variance ( numpy/numpy/core/_methods.py line 112).
The routine that I want to implement look like this in python,
arr = np.random.rand(10)
mean = arr.mean()
var = 0.0
for ai in arr: var += (ai-mean)**2
I would like to implement it in the umath module. As a first step, I tried
to reproduce the divmod function of umath, but I did not manage to do it,
you can find my fork here (the branch with the changes is call
looking_around). During compilation I get the following error,
gcc: numpy/core/src/multiarray/number.c
In file included from numpy/core/src/multiarray/number.c:17:0:
numpy/core/src/multiarray/number.c: In function ‘array_sum_multiply’:
numpy/core/src/private/binop_override.h:176:39: error: ‘PyNumberMethods {aka
struct }’ has no member named ‘nb_sum_multiply’
(void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func))
^
numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro
‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \
^
numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro
‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply,
array_sum_multiply);
Sorry if my question seems basic, but I'm new in Numpy development.
Any help?
Thank you in advance,
Marc Barbry
PS: I opened an issues as well on the github repository
https://github.com/numpy/numpy/issues/9130
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Variance without temporary array
Dear Numpy developers, I'm trying to write a version of the variance function that does not use temporary array. I got something that start to be functional (the branch is located here https://github.com/mbarbry/numpy/tree/sum_square_shift) but when running the tests I get the following memory error, Running unit tests for numpy NumPy version 1.14.0.dev0+f9db20b NumPy relaxed strides checking option: True NumPy is installed in /home/marc/programs/python/my_numpy/local/lib/python2.7/site-packages/numpy-1.14.0.dev0+f9db20b-py2.7-linux-x86_64.egg/numpy Python version 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609] nose version 1.3.7 Program received signal SIGSEGV, Segmentation fault. 0x71ef1f6d in inittest_rational () at numpy/core/src/umath/test_rational.c.src:1262 1262REGISTER_UFUNC_BINARY_RATIONAL(add) This is weird because I didn't touch the umath module, I'm working in the multiarray one. Do you have a idea how I can debug this? Thanks for your help, Marc Barbry ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] compiler binary in numpy.distutils
Hi! I'm trying to write a python wrapper to ScaLapack using f2py. I have troubles to set up the binary path to the compiler using numpy.distutils. What is the correct way? You can find my actual setup.py at the code repository, https://gitlab.com/mbarbry/python-scalapack Thanks in advance, Marc ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Directives in numpy docstring convention
Hi there, Is there a reason why the numpy docstring convention doesn't use the sphinx directives .. deprecated:: [1] and .. seealso:: [2]? I see that the numpy convention [3] uses the .. note:: directive for the deprecation notes, and for the "See also" it uses a section in this form: See also - I guess those directives were added to the sphinx after the numpy docstring convention was created. And in this case, while is probably not worth to update numpy docstrings, I think they should be used for new projects that want to follow the numpy convention. Is that the case, or is there a reason why new projects shouldn't use them? Thanks! 1. http://www.sphinx-doc.org/en/stable/markup/para.html#directive-deprecated 2. http://www.sphinx-doc.org/en/stable/markup/para.html#directive-seealso 3. https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Stricter numpydoc validation
In pandas we've been, for more than a year now, enforcing a stricter numpydoc standard. And we've got a script to help with it, which validates things like capitalization and punctuation of paragraphs, the documented parameters (they must match the ones in the signature, have both a type and a description...), PEP-8 of the standards, and many more things, so all our docstrings are consistent. I saw that there is an issue with a discussion on having a more strict standard for numpydoc, I added a comment there on whether would make sense to move the pandas standard and validation code to numpydoc: https://github.com/numpy/numpydoc/issues/213#issuecomment-511760580 I think it's worth opening the discussion here too. Is there interest in the rest of the community on having a stricter standard, and move the pandas validation (with the required updates) to numpydoc? Of course we can discuss also the exact standard, but probably worth finding out first if a stricter numpydoc standard would make sense for everyone. You can find the documentation of our standard at: https://dev.pandas.io/development/contributing_docstring.html And the script that we use to validate, as well as the exact list of errors we detect in: https://github.com/pandas-dev/pandas/blob/master/scripts/validate_docstrings.py#L76 Cheers! ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] combinations of all rows and cols from a dataframe
Hello everyone,
Given a dataframe like this:
2 6
8 5
I want to yield the following list of lists:
[ [[2],[6,5]],
[[2],[6]],
[[2],[5]],
[[8],[6,5]],
[[8],[6]],
[[8],[5]],
[[6],[2,8]],
[[6],[8]],
[[6],[2]],
[[5],[2,8]],
[[5],[2]],
[[5],[8]],
[[6,5],[2,8]] ]
I have written the following (which doesn't yield the expected results)
import pandas as pd
> from itertools import combinations
> import numpy as np
> resList=[]
> resListTmp=[]
> resListTmp2=[]
> dataframe =
> pd.read_excel("C:\\Users\\user\\Desktop\\testData.xlsx",index_col=False,header=None)
for i in range(0, len(dataframe)+1):
> for j in range(0, len(dataframe.columns)):
> for k in range (0,len(dataframe)+1):
> for xVals in list(combinations(dataframe.iloc[k:i,j], i)):
> if list(xVals) not in resListTmp:
> resListTmp.append(list(xVals))
> resListTmp2.append(resListTmp)
> resList.append(resListTmp2)
> print(resList)
>
What is wrong with my code?
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] Does numpy depend upon a Fortran library?
Hello, I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib. While investigating the source of this problem, I downloaded the numpy source (https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce378259677de516bd899dd9eb8f9b32/numpy-1.16.0.zip), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation. I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS. I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing? thanks Marc ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
