Re: [Numpy-discussion] cPickle.loads and Numeric

[Raul Cota]

Robert is right, you can always implement your own function.

What version of numpy and Python are you using ?

There may be something you can add to your numpy installation related to 
the old Numeric support which I believe is now deprecated.

Raul




On 25/02/2014 4:28 AM, Robert Kern wrote:
> On Tue, Feb 25, 2014 at 8:19 AM, Chris  wrote:
>> I have some old code that uses cPickle.loads which used to work, but now
>> reports an error in loading the module Numeric. Since Numeric has been
>> replaced by numpy, this makes sense, but, how can I get cPickle.loads to
>> work? I tested the code again on an older machine and it works fine
>> there, but, I'd like to get it working again on a modern set-up as well.
> It's relatively straightforward to subclass Unpickler to redirect it
> when it goes to look for the array constructor that it expects from
> the Numeric module.
>
>
> from cStringIO import StringIO
> import pickle
>
> import numpy as np
>
>
> TEST_NUMERIC_PICKLE = ('\x80\x02cNumeric\narray_constructor\nq\x01(K\x05\x85U'
> 
> '\x01lU(\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00'
> '\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03'
> '\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00'
> '\x00\x00K\x01tRq\x02.')
>
>
> # Constant from Numeric.
> LittleEndian = 1
>
> def array_constructor(shape, typecode, thestr, Endian=LittleEndian):
>  """ The old Numeric array constructor for pickle, recast for numpy.
>  """
>  if typecode == "O":
>  x = np.array(thestr, "O")
>  else:
>  x = np.fromstring(thestr, typecode)
>  x.shape = shape
>  if LittleEndian != Endian:
>  return x.byteswapped()
>  else:
>  return x
>
>
> class NumericUnpickler(pickle.Unpickler):
>  """ Allow loading of pickles containing Numeric arrays and
>  converting them to numpy arrays.
>  """
>
>  def find_class(self, module, name):
>  """ Return the constructor callable for a given "class".
>
>  Overridden to handle Numeric.array_constructor specially.
>  """
>  if module == 'Numeric' and name == 'array_constructor':
>  return array_constructor
>  else:
>  return pickle.Unpickler.find_class(self, module, name)
>
>
> def load(fp):
>  return NumericUnpickler(fp).load()
>
>
> def loads(pickle_string):
>  fp = StringIO(pickle_string)
>  return NumericUnpickler(fp).load()
>
>
> if __name__ == '__main__':
>  import sys
>  print loads(TEST_NUMERIC_PICKLE)
>  # Look, Ma! No Numeric!
>  assert 'Numeric' not in sys.modules
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] switching from Numeric to numpy

[Raul Cota]

John,


Just noticed this message,

We are already cleaning up all of our code to not be numpy based but for 
porting from Numeric to numpy:


In our C code we settled for the following,

#define NUMPY

#if !defined(NUMPY)
 #include "arrayobject.h"
#else
 #include "numpy/oldnumeric.h"
#endif


And this,

#ifdef NUMPY
 pyArray = (PyArrayObject *) PyArray_SimpleNew( 1, dims, 
PyArray_DOUBLE);
#else
 pyArray = (PyArrayObject *) PyArray_FromDims( 1, dims, PyArray_DOUBLE);
#endif


And I think that was all the changes we needed and while we were 
transitioning we could easily compile for Numeric or numpy.


Most of our headaches were on the python code because of very subtle 
things but even there, the bulk of the changes worked out rather nicely 
using the oldnumeric stuff.


Raul




On 07/10/2013 5:27 PM, john fernando wrote:
> I have the following C code which is an extension to my python code.
>
> The python and C code use
> #include 
>
> what is the equivalent I can use in numpy that causes the minimum code change?
>
> I did look through the old messages but didn't really find the answer-any 
> help to a novice just starting out in numpy/python/C is much appreciated.
>
> 
> FREE ONLINE PHOTOSHARING - Share your photos online with your friends and 
> family!
> Visit http://www.inbox.com/photosharing to find out more!
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problems building from source

[Raul Cota]

  
  
On 05/08/2013 2:17 PM, Charles R Harris
  wrote:


  
  On Mon, Aug 5, 2013 at 2:00 PM, Raul Cota
<r...@virtualmaterials.com>
wrote:

  Hello,
  
  I had not updated my code for a few months. I updated today to
  the
  latest source and I cannot build anymore,
  (Windows, Python 2.6)
  
  
  When I do,
  python setup.py build
  
  
  I get,
  
  """
  Running from numpy source directory.
  Traceback (most recent call last):
     File "setup.py", line 192, in 
       setup_package()
     File "setup.py", line 169, in setup_package
       from numpy.distutils.core import setup
     File
  "C:\Users\Raul\Documents\GitHub\numpy\numpy\distutils\__init__.py",
  line
  37, in 
       from numpy.testing import Tester
     File
  "C:\Users\Raul\Documents\GitHub\numpy\numpy\testing\__init__.py",
  line 13
  , in 
       from .utils import *
     File
  "C:\Users\Raul\Documents\GitHub\numpy\numpy\testing\utils.py",
  line 13, i
  n 
       from numpy.core import float32, empty, arange
     File
  "C:\Users\Raul\Documents\GitHub\numpy\numpy\core\__init__.py",
  line 6, in
    
       from . import multiarray
  ImportError: cannot import name multiarray
  
  """
  
  I can see
  numpy\numpy\core\__init__.py
  
  changed the imports to be 'relative' based from when it used
  to work.
  
  Any hints or reference to a related discussion ? All I found
  was pr 3178
  "2to3 apply import fixer"


  I don't think anyone else has seen this problem, although I
  don't know who else uses window. Have you checked with a clean
  build and clean install?
  

  


Yes. 
I will try again starting from scratch. I will poke around a bit
more and report back if I find anything interesting.


Here is where I am at,

I commented out,
#if _INSTALLED:
#    from numpy.testing import Tester
#    test = Tester().test
#    bench = Tester().bench

and now it at least tries to build and now gives me compiler errors,

arrayobject.c
numpy\core\src\multiarray\arrayobject.c(1307) : error C2143: syntax
error : miss
ing ';' before 'type'
numpy\core\src\multiarray\arrayobject.c(1309) : error C2065: '_res'
: undeclared
 identifier
numpy\core\src\multiarray\arrayobject.c(1313) : error C2065: '_res'
: undeclared
 identifier
numpy\core\src\multiarray\arrayobject.c(1318) : error C2065: '_res'
: undeclared
 identifier
numpy\core\src\multiarray\arrayobject.c(1364) : error C2143: syntax
error : miss
ing ';' before 'type'
numpy\core\src\multiarray\arrayobject.c(1366) : error C2065: '_res'
: undeclared
 identifier
numpy\core\src\multiarray\arrayobject.c(1370) : error C2065: '_res'
: undeclared
 identifier
numpy\core\src\multiarray\arrayobject.c(1375) : error C2065: '_res'
: undeclared
 identifier




Raul





  
Chuck


  
  
  
  
  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



  

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Problems building from source

[Raul Cota]

Hello,

I had not updated my code for a few months. I updated today to the 
latest source and I cannot build anymore,
(Windows, Python 2.6)


When I do,
python setup.py build


I get,

"""
Running from numpy source directory.
Traceback (most recent call last):
   File "setup.py", line 192, in 
 setup_package()
   File "setup.py", line 169, in setup_package
 from numpy.distutils.core import setup
   File 
"C:\Users\Raul\Documents\GitHub\numpy\numpy\distutils\__init__.py", line
37, in 
 from numpy.testing import Tester
   File 
"C:\Users\Raul\Documents\GitHub\numpy\numpy\testing\__init__.py", line 13
, in 
 from .utils import *
   File "C:\Users\Raul\Documents\GitHub\numpy\numpy\testing\utils.py", 
line 13, i
n 
 from numpy.core import float32, empty, arange
   File "C:\Users\Raul\Documents\GitHub\numpy\numpy\core\__init__.py", 
line 6, in
  
 from . import multiarray
ImportError: cannot import name multiarray

"""

I can see
numpy\numpy\core\__init__.py

changed the imports to be 'relative' based from when it used to work.

Any hints or reference to a related discussion ? All I found was pr 3178 
"2to3 apply import fixer"


Raul



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GSoC : Performance parity between numpy arrays and Python scalars

[Raul Cota]

For the sake of completeness, I don't think I ever mentioned what I used 
to profile when I was working on speeding up the scalars. I used AQTime 
7. It is commercial and only for Windows (as far as I know). It works 
great and it gave me fairly accurate timings and all sorts of visual 
navigation features. I do have to mock around with the numpy code every 
time I want to compile it to get it to play nicely with Visual Studio to 
generate the proper bindings for the profiler.

Raul


On 02/05/2013 7:14 AM, Nathaniel Smith wrote:
> On Thu, May 2, 2013 at 6:26 AM, Arink Verma  wrote:
>> Yes, we need to ensure that..
>> Code generator can be made, which can create code for table of registered
>> dtype during build time itself.
> I'd probably just generate it at run-time on an as-needed basis.
> (I.e., use the full lookup logic the first time, then save the
> result.) New dtypes can be registered, which will mean the tables need
> to change size at runtime anyway. If someone does some strange thing
> like add float16's and float64's, we can do the lookup to determine
> that this should be handled by the float64/float64 loop, and then
> store that information so that the next time it's fast (but we
> probably don't want to be calculating all combinations at build-time,
> which would require running the full type resolution machinery, esp.
> since it wouldn't really bring any benefits that I can see).
>
> * Re: the profiling, I wrote a full oprofile->callgrind format script
> years ago: http://vorpus.org/~njs/op2calltree.py
> Haven't used it in years either but neither oprofile nor kcachegrind
> are terribly fast-moving projects so it's probably still working, or
> could be made so without much work.
> Or easier is to use the gperftools CPU profiler:
> https://gperftools.googlecode.com/svn/trunk/doc/cpuprofile.html
>
> Instead of linking to it at build time, you can just use ctypes:
>
> In [7]: profiler = ctypes.CDLL("libprofiler.so.0")
>
> In [8]: profiler.ProfilerStart("some-file-name-here")
> Out[8]: 1
>
> In [9]: # do stuff here
>
> In [10]: profiler.ProfilerStop()
> PROFILE: interrupts/evictions/bytes = 2/0/592
> Out[10]: 46
>
> Then all the pprof analysis tools are available as described on that webpage.
>
> * Please don't trust those random suggestions for possible
> improvements I threw out when writing the original description.
> Probably it's true that FP flag checking and ufunc type lookup are
> expensive, but one should fix what the profile says to fix, not what
> someone guessed might be good to fix based on a few minutes thought.
>
> * Instead of making a giant table of everything that needs to be done
> to make stuff fast first, before writing any code, I'd suggest picking
> one operation, figuring out what change would be the biggest
> improvement for it, making that change, checking that it worked, and
> then repeat until that operation is really fast. Then if there's still
> time pick another operation. Producing a giant todo list isn't very
> productive by itself if there's no time then to actually do all the
> things on the list :-).
>
> * Did you notice this line on the requirements page? "Having your
> first pull request merged before the GSoC application deadline (May 3)
> is required for your application to be accepted."
>
> -n
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GSoC : Performance parity between numpy arrays and Python scalars

[Raul Cota]

  
  

  It is great that you are looking into this !! We are currently
  running on a fork of numpy because we really need these
  performance improvements .
  
  
  I noticed that, as suggested, you took from the pull request I
  posted a while ago for the 
  PyObject_GetAttrString
  PyObject_GetBuffer
  
  issues.
  
  ( https://github.com/raulcota/numpy )
  
  
  A couple of comments on that,
  
  - Seems like you did not grab the latest revisions of that code
  that I posted that fixes the style of the comments and 'attempts'
  to fix an issue reported about Python 3 . I say 'attempts' because
  I thought it was fixed but I someone mentioned this was not
  correct.
  
  - There was also some feedback from Nathaniel about not liking the
  macros and siding for inline functions. I have not gotten around
  to it, but it would be nice if you jump on that boat.
  
  On the has lookup table, haven't looked at the implementation but
  the speed up is remarkable.
  
  
  Cheers !
  
  Raul
  
  
  On 30/04/2013 8:26 PM, Arink Verma wrote:


  
Hi all!
  I have written my application[1] for Performance
  parity between numpy arrays and Python scalars[2]. It
would be a great help if you view it. Does it
look achievable and deliverable according to the project.
  
  
  [1] http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/arinkverma/40001#
  
  [2] http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas
  
  


  -- 
  Arink
  Computer Science and Engineering 
  Indian Institute of Technology Ropar
  www.arinkverma.in

  
  
  
  
  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



  

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Gsoc : Performance parity between numpy arrays and Python scalars

[Raul Cota]

  
  
Few comments on the topic,
  
  The postings I did to the list were in numpy 1.6 but the pull
  requests were done on the latest code at that time I believe 1.7 .
  There are still a few comments pending that I have not had a
  chance to look into, but that is a separate topic.
  
  
  The main problems I addressed in those pull requests are related
  to the scalars. For example,
  timeit.timeit("x + x", setup="import numpy as np;x =
  np.array([1.0])[0]")
  
  The line above is much faster with the pull request I posted.
  
  
  The problem as posted in the Gsoc is a different one,
  timeit.timeit("x + x", setup="import numpy as np;x =
  np.asarray(1.0)")
  
  The line above is not much faster with the pull requests I posted.
  It would if the test was "x + 1.0"
  
  
  I profiled the line above and the main results are shown in the
  links below,
  
https://docs.google.com/file/d/0B3hgR3Pc2vPgUC1Kbng3SUx0OUE/edit?usp=sharing
https://docs.google.com/file/d/0B3hgR3Pc2vPgS3ZCUDVJUTZScEE/edit?usp=sharing
https://docs.google.com/file/d/0B3hgR3Pc2vPgZ3B5alFfYW5vLWc/edit?usp=sharing
https://docs.google.com/file/d/0B3hgR3Pc2vPgZnpaZEVFSkhhTkE/edit?usp=sharing
  
  They show the main time consumers and then a graphical
  representation of calls and their time spent in their children.
  The different images traverse the calls starting in
  ufunc_gneric_call. You can see a fair bit of time is spent in
  find_best_ufunc_inner_loop . The point here is that, this
  particular example is not bottle necked by error checking.
  
  Things to keep in mind,
  - these profile results are based on numpy 1.6 . Takes me quite a
  bit of doing/hacking to get the profiler to play nicely with
  getting me to the proper lines of code and I have not done it for
  the latest code.
  
  - I timed it, and in the latest code of numpy it is not any
  faster, so I assume all the findings still apply (note the word
  assume)
  
  - all my timings are on windows. I point this out, because
  Nathaniel had pointed me to a specific line of code that was a
  bottleneck to him which does not apply to Windows and found out
  that the Windows counter part of the code is not a bottleneck.
  
  
  
  Raul
  
  
  
  
  On 17/04/2013 9:03 AM, Arink Verma wrote:


  Hello everyone

  


I am Arink, computer science student and open source
  enthusiastic. This year I am interested to work on project
  "Performance parity between numpy arrays and Python
  scalars"[1]. 


I tried to adobt rald's work on numpy1.7[2] (which was
  done for numpy1.6 [3]). 
Till now by avoiding 
a) the uncessary Checking for floating point errors
  which is slow, 

  b) unnecessarily Creation / destruction of scalar array
  types


I am getting the speedup by ~ 1.05 times, which
  marginal offcourse. As in project's describtion it is
  mention that ufunc look up code is slow and inefficient. 


Few questions
1. Does it has to check every single data type possible
  until if finds the best match for the data that the
  operation is being performed on, or is there better way to
  find the best possible match?
2. If yes, so where are bottle-necks? Is the checks for
  proper data types are very expensive?






[1]http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas
[2]https://github.com/arinkverma/numpy/compare/master...gsoc_performance
[3]http://article.gmane.org/gmane.comp.python.numeric.general/52480



  

-- 
Arink
Computer Science and Engineering 
Indian Institute of Technology Ropar
www.arinkverma.in
  
  
  
  
  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



  

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Leaking memory problem

[Raul Cota]

Josef's suggestion is the first thing I'd try.

Are you doing any of this in C ? It is easy to end up duplicating memory 
that you need to Py_DECREF .
In the C debugger you should be able to monitor the ref count of your 
python objects.


btw, for manual tracking of reference counts you can do,
sys.getrefcount
it has come handy for me every once in a while but usually the garbage 
collector is all I've needed besides patience.


The way I usually run the gc is by doing

 gc.enable(  )
 gc.set_debug(gc.DEBUG_LEAK)

as pretty much my first lines and then after everything is said and done 
I do something along the lines of,

gc.collect(  )
for x in gc.garbage:
   s = str(x)
   print type(x)

you'd have to set up your program to quite before it runs out of memory 
of course but I understand you get to run for quite a few iterations 
before failure.

Raul




On 25/02/2013 9:03 AM, josef.p...@gmail.com wrote:
> On Mon, Feb 25, 2013 at 8:41 AM, Jaakko Luttinen
>  wrote:
>> Hi!
>>
>> I was wondering if anyone could help me in finding a memory leak problem
>> with NumPy. My project is quite massive and I haven't been able to
>> construct a simple example which would reproduce the problem..
>>
>> I have an iterative algorithm which should not increase the memory usage
>> as the iteration progresses. However, after the first iteration, 1GB of
>> memory is used and it steadily increases until at about 100-200
>> iterations 8GB is used and the program exits with MemoryError.
>>
>> I have a collection of objects which contain large arrays. In each
>> iteration, the objects are updated in turns by re-computing the arrays
>> they contain. The number of arrays and their sizes are constant (do not
>> change during the iteration). So the memory usage should not increase,
>> and I'm a bit confused, how can the program run out of memory if it can
>> easily compute at least a few iterations..
> There are some stories where pythons garbage collection is too slow to kick 
> in.
> try to call gc.collect in the loop to see if it helps.
>
> roughly what I remember: collection works by the number of objects, if
> you have a few very large arrays, then memory increases, but garbage
> collection doesn't start yet.
>
> Josef
>
>
>> I've tried to use Pympler, but I've understood that it doesn't show the
>> memory usage of NumPy arrays.. ?
>>
>> I also tried gc.set_debug(gc.DEBUG_UNCOLLECTABLE) and then printing
>> gc.garbage at each iteration, but that doesn't show anything.
>>
>> Does anyone have any ideas how to debug this kind of memory leak bug?
>> And how to find out whether the bug is in my code, NumPy or elsewhere?
>>
>> Thanks for any help!
>> Jaakko
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] NumPy int32 array to Excel through COM server is failing

[Raul Cota]

Hello,

We came across a problem trying to get an array across COM when wrapped 
as a COM server using the win32com extension.

What caught our attention is that arrays of type float64 work fine but 
do not work for any other array.

Does anyone know if there is something we could do at the NumPy level to 
make it work ?

Arrays are mapped onto Variant/SafeArray and the conversion to a Variant 
seems to be failing for anything that is not a float64.

It is not a huge deal for us to workaround the problem but it is kind of 
ugly and I just wanted to make sure there was not something simple that 
could be done (particularly if something like this would be considered a 
bug).

I include working sample code below that reproduces the problem where 
Excel instantiates a com server and requests an array of size and dtype. 
On the Python side, the code sets up the com server and exposes the 
function that just returns an array of ones.


Code in Excel to get an array:

Public Sub NumPyVariantTest()
 Dim npcom As Object
 Dim arr

 '... instantiate com object
 Set npcom = CreateObject("NPTest.COM")
 size = 7

 '... test float64. Works !
 arr = npcom.GetNPArray(size, "float64")

 '... test int32. Fails !
 arr = npcom.GetNPArray(size, "int32")

End Sub



Code in Python to set up com server and expose the GetNPArray function

import win32com.server.util
import win32com.client
from pythoncom import CLSCTX_LOCAL_SERVER, CLSCTX_INPROC

import sys, os
import numpy
from numpy import zeros, ones, array


class NPTestCOM(object):
 """COM accessible version of CommandInterface"""

 _reg_clsid_ = "{A0E551F5-2F22-4FB4-B28E-FF1B6809D21C}"
 _reg_desc_  = "NumPy COM Test"
 _reg_progid_ = "NPTest.COM"
 _reg_clsctx_ = CLSCTX_INPROC
 _public_methods_ = ['GetNPArray']

 _public_attrs_ = []
 _readonly_attrs_ = []


 def GetNPArray(self, size, dtype):
 """ Return an arbitrary NumPy array of type dtype to check
 conversion to Variant"""
 return ones(size, dtype=dtype)



if __name__ == '__main__':
 import win32com.server.register
 import _winreg

 dllkey = 'nptestdll'

 if len(sys.argv) > 1 and sys.argv[1] == 'unregister':
 win32com.server.register.UnregisterClasses(NPTestCOM)
 software_key = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, 
'SOFTWARE')
 vmg_key = _winreg.OpenKey(software_key, 'VMG')
 _winreg.DeleteKey(vmg_key, dllkey)
 _winreg.CloseKey(vmg_key)
 _winreg.CloseKey(software_key)
 else:
 win32com.server.register.UseCommandLine(NPTestCOM)
 software_key = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, 
'SOFTWARE')
 vmg_key = _winreg.CreateKey(software_key, 'VMG')
 _winreg.SetValue(vmg_key, dllkey, _winreg.REG_SZ, 
os.path.abspath(os.curdir))
 _winreg.CloseKey(vmg_key)
 _winreg.CloseKey(software_key)





Regards,
Raul

-- 
Raul Cota (P.Eng., Ph.D. Chemical Engineering)
Research & Development Manager
Phone: (403) 457 4598
Fax:   (403) 457 4637
Virtual Materials Group - Canada
www.virtualmaterials.com

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove support for numeric and numarray in 1.8

[Raul Cota]

  
  

  based on our situation
  
  On 10/01/2013 9:02 AM, Christopher Hanley wrote:


  I'm all for a big scary warning on import.  Fair
warning is good for everyone, not just our developers.


  

agree


  
As for testing, our software that uses the API is
  tested nightly.  So if our software stops working, and the
  compatibility layer is the cause, we would definitely be
  looking into what happened. :-)


  

same (but not necesairlly with the very latest numpy) . 



  
In any case, fair warning of dropped support in
  1.8 and removal in 1.9 is fine with us.


  


not sure. It all depends on the timing. I'd prefer to revisit the
topic once 1.8 is actually going to come out.


Raul





  
Thanks,
Chris


  
  


On Wed, Jan 9, 2013 at 6:38 PM,
  Nathaniel Smith 
  wrote:
  
On Wed, Jan 9, 2013 at 11:21 PM, Christopher
  Hanley 
  wrote:
  > After poking around our code base and talking to a
  few folks I predict that
  > we at STScI can remove our dependence on the
  numpy-numarray compatibility
  > layer by the end of this calendar year.  I'm unsure
  of what the timeline for
  > numpy 1.8 is so I don't know if this schedule
  supports removal of the
  > compatibility layer from 1.8 or not.
  

It'd be nice if 1.8 were out before that, but that doesn't
really
matter -- let us know when you get it sorted?

Also, would it help if we added a big scary warning at
import time to
annoy your more recalcitrant developers with? :-)

The basic issue is that none of us actually use this stuff,
it has no
tests, the rest of numpy is changing around it, and we have
no idea if
it works, so at some point it makes more sense for us to
just stop
shipping the compat layer and let anyone who still needs it
maintain
their own copy of the code.

-n
  

  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
  

  


  
  
  
  
  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



  

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove support for numeric and numarray in 1.8

[Raul Cota]

  
  
We can keep testing it in coming
  versions while we catch up.
  
  Raul Cota (P.Eng., Ph.D. Chemical Engineering)
Research & Development Manager
Virtual Materials Group - Canada
www.virtualmaterials.com

===
This e-mail may be privileged and/or confidential, and the sender does
not waive any related rights and obligations.
Any distribution, use or copying of this e-mail or the information it
contains by other than an intended recipient is unauthorized.
If you received this e-mail in error, please advise me (by return e-mail
or otherwise) immediately.
===
  On 07/01/2013 5:14 PM, Charles R Harris wrote:


  
  On Mon, Jan 7, 2013 at 4:08 PM, Raul Cota
<r...@virtualmaterials.com>
wrote:

  
Ran a fair bit of our test suite using numpy 1.7
  compiling against the corresponding 'numpy/oldnumeric.h'
  and everything worked well .
  
  All I saw was the warning below which is obviously
  expected:
  """
  Warning    23    warning Msg: Using deprecated NumPy API,
  disable it by #defining NPY_NO_DEPRECATED_API
  NPY_1_7_API_VERSION   
  c:\python27\lib\site-packages\numpy\core\include\numpy\npy_deprecated_api.h   

  8
  """
  

  


  Great! Thanks, not many are in a position to check that part
  of numpy. Looks like we will need to keep the deprecated api
  around for a while also.
  
  Chuck 


  
  
  
  
  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



  

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove support for numeric and numarray in 1.8

[Raul Cota]

  
  
Ran a fair bit of our test suite using
  numpy 1.7 compiling against the corresponding 'numpy/oldnumeric.h'
  and everything worked well .
  
  All I saw was the warning below which is obviously expected:
  """
  Warning    23    warning Msg: Using deprecated NumPy API, disable
  it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION   
  c:\python27\lib\site-packages\numpy\core\include\numpy\npy_deprecated_api.h   
  8
  """
  
  
  Raul Cota
  
  
  
  On 07/01/2013 9:22 AM, Charles R Harris wrote:


  
      On Sun, Jan 6, 2013 at 9:15 PM, Raul Cota
<r...@virtualmaterials.com>
wrote:

  
I realize we may be a minority but it would be very
  nice if support for numeric could be kept for a few more
  versions. We don't have any particular needs for numarray.
  
  We just under went through an extremely long and painful
  process to upgrade our software from Numeric to numpy and
  everything hinges on the "oldnumeric" stuff. This was the
  classical 80-20 scenario where we got most of the stuff to
  work in a just a few days and then we had to revisit
  several areas of our software to iron out all the bugs and
  subtle but meaningful differences between numpy and
  Numeric. The last round of work was related to speed. We
  still have not released the upgrade to our costumers
  therefore we expect still a few more bugs to surface.
  
  Bottom line, we are still about one or two years away from
  changing all our imports to numpy. Yes, I know, we move
  fairly slowly but that is our reality.
  

  


  Good to know. Have you tested oldnumeric in the upcoming 1.7
  release?
  
  Chuck 


  
  
  
  
  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



  

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove support for numeric and numarray in 1.8

[Raul Cota]

  
  
On 07/01/2013 9:22 AM, Charles R Harris
  wrote:


  
  On Sun, Jan 6, 2013 at 9:15 PM, Raul Cota
<r...@virtualmaterials.com>
wrote:

  
I realize we may be a minority but it would be very
  nice if support for numeric could be kept for a few more
  versions. We don't have any particular needs for numarray.
  
  We just under went through an extremely long and painful
  process to upgrade our software from Numeric to numpy and
  everything hinges on the "oldnumeric" stuff. This was the
  classical 80-20 scenario where we got most of the stuff to
  work in a just a few days and then we had to revisit
  several areas of our software to iron out all the bugs and
  subtle but meaningful differences between numpy and
  Numeric. The last round of work was related to speed. We
  still have not released the upgrade to our costumers
  therefore we expect still a few more bugs to surface.
  
  Bottom line, we are still about one or two years away from
  changing all our imports to numpy. Yes, I know, we move
  fairly slowly but that is our reality.
  

  


  Good to know. Have you tested oldnumeric in the upcoming 1.7
  release?

  


Not yet. Will get on it asap and report back.


Raul






  

  Chuck 


  
  
  
  
  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



  

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove support for numeric and numarray in 1.8

[Raul Cota]

  
  
I realize we may be a minority but it
  would be very nice if support for numeric could be kept for a few
  more versions. We don't have any particular needs for numarray.
  
  We just under went through an extremely long and painful process
  to upgrade our software from Numeric to numpy and everything
  hinges on the "oldnumeric" stuff. This was the classical 80-20
  scenario where we got most of the stuff to work in a just a few
  days and then we had to revisit several areas of our software to
  iron out all the bugs and subtle but meaningful differences
  between numpy and Numeric. The last round of work was related to
  speed. We still have not released the upgrade to our costumers
  therefore we expect still a few more bugs to surface.
  
  Bottom line, we are still about one or two years away from
  changing all our imports to numpy. Yes, I know, we move fairly
  slowly but that is our reality.
  
  
  Regards,
  Raul
  
  
  
  
  
  On 05/01/2013 7:38 PM, Charles R Harris wrote:

Thoughts?
  
  Chuck
  
  
  
  ___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



  

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy speed ups to simple tasks - final findings and suggestions

[Raul Cota]

On 04/01/2013 5:44 PM, Nathaniel Smith wrote:
> On Fri, Jan 4, 2013 at 11:36 PM, Raul Cota  wrote:
>> On 04/01/2013 2:33 PM, Nathaniel Smith wrote:
>>> On Fri, Jan 4, 2013 at 6:50 AM, Raul Cota  wrote:
>>>> On 02/01/2013 7:56 AM, Nathaniel Smith wrote:
>>>>> But, it's almost certainly possible to optimize numpy's float64 (and
>>>>> friends), so that they are themselves (almost) as fast as the native
>>>>> python objects. And that would help all the code that uses them, not
>>>>> just the ones where regular python floats could be substituted
>>>>> instead. Have you tried profiling, say, float64 * float64 to figure
>>>>> out where the bottlenecks are?
>>>> Seems to be split between
>>>> - (primarily) the memory allocation/deallocation of the float64 that is
>>>> created from the operation float64 * float64. This is the reason why 
>>>> float64
>>>> * Pyfloat got improved with one of my changes because PyFloat was being
>>>> internally converted into a float64 before doing the multiplication.
>>>>
>>>> - the rest of the time is the actual multiplication path way.
>>> Running a quick profile on Linux x86-64 of
>>> x = np.float64(5.5)
>>> for i in xrange(n):
>>>x * x
>>> I find that ~50% of the total CPU time is inside feclearexcept(), the
>>> function which resets the floating point error checking registers --
>>> and most of this is inside a single instruction, stmxcsr ("store sse
>>> control register").
>> I find strange you don't see bottleneck in allocation of a float64.
>>
>> is it easy for you to profile this ?
>>
>> x = np.float64(5.5)
>> y = 5.5
>> for i in xrange(n):
>>   x * y
>>
>> numpy internally translates y into a float64 temporarily and then
>> discards it and I seem to remember is a bit over two times slower than x * x
> Yeah, seems to be dramatically slower. Using ipython's handy interface
> to the timeit[1] library:
>
> In [1]: x = np.float64(5.5)
>
> In [2]: y = 5.5
>
> In [3]: timeit x * y
> 100 loops, best of 3: 725 ns per loop
>
> In [4]: timeit x * x
> 100 loops, best of 3: 283 ns per loop

I haven't been using timeit because the bulk of what we are doing 
includes comparing against Python 2.2 and Numeric and timeit did not 
exist then. Can't wait to finally officially upgrade our main product.


> But we already figured out how to (mostly) fix this part, right?

Correct


Cheers,

Raul



> I was
> curious about the Float64*Float64 case, because that's the one that
> was still slow after those first two patches. (And, yes, like you say,
> when I run x*y in the profiler then there's a huge amount of overhead
> in PyArray_GetPriority and object allocation/deallocation).
>
>> I will try to do your suggestions on
>>
>> PyUFunc_clearfperr/PyUFunc_getfperror
>>
>> and see what I get. Haven't gotten around to get going with being able
>> to do a pull request for the previous stuff. if changes are worth while
>> would it be ok if I also create one for this ?
> First, to be clear, it's always OK to do a pull request -- the worst
> that can happen is that we all look it over carefully and decide that
> it's the wrong approach and don't merge. In my email before I just
> wanted to give you some clear suggestions on a good way to get
> started, we wouldn't have like kicked you out or something if you did
> it differently :-)
>
> And, yes, assuming my analysis so far is correct we would definitely
> be interested in major speedups that have no other user-visible
> effects... ;-)
>
> -n
>
> [1] http://docs.python.org/2/library/timeit.html
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy speed ups to simple tasks - final findings and suggestions

[Raul Cota]

On 04/01/2013 2:33 PM, Nathaniel Smith wrote:
> On Fri, Jan 4, 2013 at 6:50 AM, Raul Cota  wrote:
>> On 02/01/2013 7:56 AM, Nathaniel Smith wrote:
>>> But, it's almost certainly possible to optimize numpy's float64 (and
>>> friends), so that they are themselves (almost) as fast as the native
>>> python objects. And that would help all the code that uses them, not
>>> just the ones where regular python floats could be substituted
>>> instead. Have you tried profiling, say, float64 * float64 to figure
>>> out where the bottlenecks are?
>> Seems to be split between
>> - (primarily) the memory allocation/deallocation of the float64 that is
>> created from the operation float64 * float64. This is the reason why float64
>> * Pyfloat got improved with one of my changes because PyFloat was being
>> internally converted into a float64 before doing the multiplication.
>>
>> - the rest of the time is the actual multiplication path way.
> Running a quick profile on Linux x86-64 of
>x = np.float64(5.5)
>for i in xrange(n):
>   x * x
> I find that ~50% of the total CPU time is inside feclearexcept(), the
> function which resets the floating point error checking registers --
> and most of this is inside a single instruction, stmxcsr ("store sse
> control register").

I find strange you don't see bottleneck in allocation of a float64.

is it easy for you to profile this ?

x = np.float64(5.5)
y = 5.5
for i in xrange(n):
 x * y

numpy internally translates y into a float64 temporarily and then 
discards it and I seem to remember is a bit over two times slower than x * x


I will try to do your suggestions on

PyUFunc_clearfperr/PyUFunc_getfperror

and see what I get. Haven't gotten around to get going with being able 
to do a pull request for the previous stuff. if changes are worth while 
would it be ok if I also create one for this ?



Thanks again,

Raul



> It's possible that this is different on windows
> (esp. since apparently our fpe exception handling apparently doesn't
> work on windows[1]), but the total time you measure for both
> PyFloat*PyFloat and Float64*Float64 match mine almost exactly, so most
> likely we have similar CPUs that are doing a similar amount of work in
> both cases.
>
> The way we implement floating point error checking is basically:
>  PyUFunc_clearfperr()
>  
>  if (PyUFunc_getfperror() & BAD_STUFF) {
>  
>  }
>
> Some points that you may find interesting though:
>
> - The way we define these functions, both PyUFunc_clearfperr() and
> PyUFunc_getfperror() clear the flags. However, for PyUFunc_getfperror,
> this is just pointless. We could simply remove this, and expect to see
> a ~25% speedup in Float64*Float64 without any downside.
>
> - Numpy's default behaviour is to always check for an warn on floating
> point errors. This seems like it's probably the correct default.
> However, if you aren't worried about this for your use code, you could
> disable these warnings with np.seterr(all="ignore"). (And you'll get
> similar error-checking to what PyFloat does.) At the moment, that
> won't speed anything up. But we could easily then fix it so that the
> PyUFunc_clearfperr/PyUFunc_getfperror code checks for whether errors
> are ignored, and disables itself. This together with the previous
> change should get you a ~50% speedup in Float64*Float64, without
> having to change any of numpy's semantics.
>
> - Bizarrely, Numpy still checks the floating point flags on integer
> operations, at least for integer scalars. So 50% of the time in
> Int64*Int64 is also spent in fiddling with floating point exception
> flags. That's also some low-hanging fruit right there... (to be fair,
> this isn't *quite* as trivial to fix as it could be, because the
> integer overflow checking code sets the floating point unit's
> "overflow" flag to signal a problem, and we'd need to pull this out to
> a thread-local variable or something before disabling the floating
> point checks entirely in integer code. But still, not a huge problem.)
>
> -n
>
> [1] https://github.com/numpy/numpy/issues/2350
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy speed ups to simple tasks - final findings and suggestions

[Raul Cota]

In my previous email I sent an image but I just thought that maybe the 
mailing list does not accept attachments or need approval.

I put a couple of images related to my profiling results (referenced to 
my previous email) here.


Sorted by time per function with a graph of calls at the bottom

http://raul-playground.appspot.com/static/images/numpy-profile-time.png


Sorted by Time with Children
http://raul-playground.appspot.com/static/images/numpy-profile-timewchildren.png


The test is a loop of
val = float64 * float64 * float64 * float64


Raul




On 02/01/2013 7:56 AM, Nathaniel Smith wrote:
> On Fri, Dec 21, 2012 at 7:20 PM, Raul Cota  wrote:
>> Hello,
>>
>>
>> On Dec/2/2012 I sent an email about some meaningful speed problems I was
>> facing when porting our core program from Numeric (Python 2.2) to Numpy
>> (Python 2.6). Some of our tests went from 30 seconds to 90 seconds for
>> example.
>
> Hi Raul,
>
> This is great work! Sorry you haven't gotten any feedback yet -- I
> guess it's a busy time of year for most people; and, the way you've
> described your changes makes it hard for us to use our usual workflow
> to discuss them.
>
>> These are the actual changes to the C code,
>> For bottleneck (a)
>>
>> In general,
>> - avoid calls to PyObject_GetAttrString when I know the type is
>> List, None, Tuple, Float, Int, String or Unicode
>>
>> - avoid calls to PyObject_GetBuffer when I know the type is
>> List, None or Tuple
>
> This definitely seems like a worthwhile change. There are possible
> quibbles about coding style -- the macros could have better names, and
> would probably be better as (inline) functions instead of macros --
> but that can be dealt with.
>
> Can you make a pull request on github with these changes? I guess you
> haven't used git before, but I think you'll find it makes things
> *much* easier (in particular, you'll never have to type out long
> awkward english descriptions of the changes you made ever again!) We
> have docs here:
>http://docs.scipy.org/doc/numpy/dev/gitwash/git_development.html
> and your goal is to get to the point where you can file a "pull request":
>
> http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html#asking-for-your-changes-to-be-merged-with-the-main-repo
> Feel free to ask on the list if you get stuck of course.
>
>> For bottleneck (b)
>>
>> b.1)
>> I noticed that PyFloat * Float64 resulted in an unnecessary "on the fly"
>> conversion of the PyFloat into a Float64 to extract its underlying C
>> double value. This happened in the function
>> _double_convert_to_ctype which comes from the pattern,
>> _@name@_convert_to_ctype
>
> This also sounds like an excellent change, and perhaps should be
> extended to ints and bools as well... again, can you file a pull
> request?
>
>> b.2) This is the change that may not be very popular among Numpy users.
>> I modified Float64 operations to return a Float instead of Float64. I
>> could not think or see any ill effects and I got a fairly decent speed
>> boost.
>
> Yes, unfortunately, there's no way we'll be able to make this change
> upstream -- there's too much chance of it breaking people's code. (And
> numpy float64's do act different than python floats in at least some
> cases, e.g., numpy gives more powerful control over floating point
> error handling, see np.seterr.)
>
> But, it's almost certainly possible to optimize numpy's float64 (and
> friends), so that they are themselves (almost) as fast as the native
> python objects. And that would help all the code that uses them, not
> just the ones where regular python floats could be substituted
> instead. Have you tried profiling, say, float64 * float64 to figure
> out where the bottlenecks are?
>
> -n
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy speed ups to simple tasks - final findings and suggestions

[Raul Cota]

On 02/01/2013 7:58 AM, Nathaniel Smith wrote:
> On Wed, Jan 2, 2013 at 2:56 PM, Nathaniel Smith  wrote:
>> On Fri, Dec 21, 2012 at 7:20 PM, Raul Cota  wrote:
>>> b.1)
>>> I noticed that PyFloat * Float64 resulted in an unnecessary "on the fly"
>>> conversion of the PyFloat into a Float64 to extract its underlying C
>>> double value. This happened in the function
>>> _double_convert_to_ctype which comes from the pattern,
>>> _@name@_convert_to_ctype
>>
>> This also sounds like an excellent change, and perhaps should be
>> extended to ints and bools as well... again, can you file a pull
>> request?
>
> Immediately after I hit 'send' I realized this might be unclear...
> what I mean is, please file two separate pull requests, one for the
> (a) changes and one for the (b.1) changes. They're logically separate
> so it'll be easier to review and merge them separately.
>


I understood it like that :)

I will give it a try.


Thanks for the feedback

Raul




> -n
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy speed ups to simple tasks - final findings and suggestions

[Raul Cota]

On 02/01/2013 7:56 AM, Nathaniel Smith wrote:

On Fri, Dec 21, 2012 at 7:20 PM, Raul Cota  wrote:

Hello,


On Dec/2/2012 I sent an email about some meaningful speed problems I was
facing when porting our core program from Numeric (Python 2.2) to Numpy
(Python 2.6). Some of our tests went from 30 seconds to 90 seconds for
example.


Hi Raul,

This is great work! Sorry you haven't gotten any feedback yet -- I
guess it's a busy time of year for most people; and, the way you've
described your changes makes it hard for us to use our usual workflow
to discuss them.



Sorry about that.




These are the actual changes to the C code,
For bottleneck (a)

In general,
- avoid calls to PyObject_GetAttrString when I know the type is
List, None, Tuple, Float, Int, String or Unicode

- avoid calls to PyObject_GetBuffer when I know the type is
List, None or Tuple


This definitely seems like a worthwhile change. There are possible
quibbles about coding style -- the macros could have better names, and
would probably be better as (inline) functions instead of macros --
but that can be dealt with.

Can you make a pull request on github with these changes? I guess you
haven't used git before, but I think you'll find it makes things
*much* easier (in particular, you'll never have to type out long
awkward english descriptions of the changes you made ever again!) We
have docs here:
   http://docs.scipy.org/doc/numpy/dev/gitwash/git_development.html
and your goal is to get to the point where you can file a "pull request":
   
http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html#asking-for-your-changes-to-be-merged-with-the-main-repo
Feel free to ask on the list if you get stuck of course.


For bottleneck (b)

b.1)
I noticed that PyFloat * Float64 resulted in an unnecessary "on the fly"
conversion of the PyFloat into a Float64 to extract its underlying C
double value. This happened in the function
_double_convert_to_ctype which comes from the pattern,
_@name@_convert_to_ctype


This also sounds like an excellent change, and perhaps should be
extended to ints and bools as well... again, can you file a pull
request?


b.2) This is the change that may not be very popular among Numpy users.
I modified Float64 operations to return a Float instead of Float64. I
could not think or see any ill effects and I got a fairly decent speed
boost.


Yes, unfortunately, there's no way we'll be able to make this change
upstream -- there's too much chance of it breaking people's code. (And
numpy float64's do act different than python floats in at least some
cases, e.g., numpy gives more powerful control over floating point
error handling, see np.seterr.)


I thought so. I may keep a fork of the changes for myself.




But, it's almost certainly possible to optimize numpy's float64 (and
friends), so that they are themselves (almost) as fast as the native
python objects. And that would help all the code that uses them, not
just the ones where regular python floats could be substituted
instead. Have you tried profiling, say, float64 * float64 to figure
out where the bottlenecks are?




Seems to be split between
- (primarily) the memory allocation/deallocation of the float64 that is 
created from the operation float64 * float64. This is the reason why 
float64 * Pyfloat got improved with one of my changes because PyFloat 
was being internally converted into a float64 before doing the 
multiplication.


- the rest of the time is the actual multiplication path way.


I attach an image of the profiler using the original numpy code with a 
loop on

val = float64 * float64 * float64 * float64


Let me know if something is not clear.


Raul






-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

<>___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 3D array problem in Python

[Raul Cota]

Quick comment since I was working on timing trivial operations,

If I run the triple loop with R with only zeros thus avoiding the 
integration, the loop takes in my computer about 1 second with the 
float() function and about 1.5 without it if R is dtype='float64' and 
3.3 seconds if dtype='float32'.

I didn't bother trying the other obvious speed up of avoiding the dot 
operator

quad = integrate.quad

and using quad inside the triple loop.

I do those things out of habit but they barely ever make a meaningful 
difference.


Conclusions:
1) The overhead of the triple loop is meaningless if the whole operation 
takes minutes to complete.

2) Using float() does make it faster but in this scenario the speed up 
is meaningless in the grand scheme of things.
* It is besides the point, but for what is worth, with the modified code 
for numpy I suggested a week ago, using the float() function is not 
needed to get it to run in 1 second.


Raul Cota




On 30/12/2012 5:35 PM, Chris Barker - NOAA Federal wrote:
> On Sun, Dec 30, 2012 at 3:41 AM, Happyman  wrote:
>> nums=32
>> rows=120
>> cols=150
>>
>> for k in range(0,nums):
>>for i in range(0,rows):
>>   for j in range(0,cols):
>>if float ( R[ k ] [ i ] [ j ] ) ==
>> 0.0:
> why the float() -- what data type is R?
>
>>else:
>>   val11[ i ] [ j ], val22[ i
>> ][ j ] = integrate.quad( lambda x :  F1(x)*F2(x) , 0 , pi)
> this is odd -- Do F1 and F2 depend on i,j, or k somehow? or are you
> somehow integerting over the k-dimension? In which case, I'm guessing
> that integration is you time killer anyway -- do some profiling to
> know for sure.
>
> -Chris
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Numpy speed ups to simple tasks - final findings and suggestions

[Raul Cota]

yp=(byte,ubyte,short,ushort,int,uint,long,ulong,longlong,ulonglong)*12,
  *  float*4, double*6,
  *   (half, float, double, longdouble, cfloat, cdouble, clongdouble)*6,
  *   (half, float, double, longdouble)*2#
  * 
#OName=(Byte,UByte,Short,UShort,Int,UInt,Long,ULong,LongLong,ULongLong)*12,
  *  Float*4, Double*6,
  *(Half, Float, Double, LongDouble, CFloat, CDouble, 
CLongDouble)*6,
  *(Half, Float, Double, LongDouble)*2#
  * 
#OutUseName=(Byte,UByte,Short,UShort,Int,UInt,Long,ULong,LongLong,ULongLong)*12,
  *  Float*4, out*6,
  *(Half, Float, out, LongDouble, CFloat, CDouble, CLongDouble)*6,
  *(Half, Float, out, LongDouble)*2#
  * #AsScalarArr=(1,1,1,1,1,1,1,1,1,1)*12,
  *  1*4, 0*6,
  *(1, 1, 0, 1, 1, 1, 1)*6,
  *(1, 1, 0, 1)*2#
  * 
#RetValCreate=(PyArrayScalar_New,PyArrayScalar_New,PyArrayScalar_New,PyArrayScalar_New,PyArrayScalar_New,PyArrayScalar_New,PyArrayScalar_New,PyArrayScalar_New,PyArrayScalar_New,PyArrayScalar_New)*12,
  *  PyArrayScalar_New*4, PyFloat_FromDouble*6,
  *(PyArrayScalar_New, PyArrayScalar_New, PyFloat_FromDouble, 
PyArrayScalar_New, PyArrayScalar_New, PyArrayScalar_New, 
PyArrayScalar_New)*6,
  *(PyArrayScalar_New, PyArrayScalar_New, PyFloat_FromDouble, 
PyArrayScalar_New)*2#
  */

#if !defined(CODEGEN_SKIP_@oper@_FLAG)

static PyObject *
@name@_@oper@(PyObject *a, PyObject *b)
{

... Same as before  and ends with...

#else
 ret = @RetValCreate@(@OutUseName@);
 if (ret == NULL) {
 return NULL;
 }
 if (@AsScalarArr@)
 PyArrayScalar_ASSIGN(ret, @OName@, out);
#endif
 return ret;
}
#endif

/**end repeat**/


I still need to do the section for when there are two return values and 
the power function. I am not sure what else could be there.


=====

That's about it. Sorry for the long email. I tried to summarize as much 
as possible.


Let me know if you have any questions or if you want the actual files I 
modified.


Cheers,

Raul Cota



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to drop python 2.4 support in numpy 1.8

[Raul Cota]

Point well taken. It is always a tradeoff / balancing act where you can 
have 'anything' but not 'everything'. Where would the fun be if we could 
have everything :) ? . In our situation, there were a couple of 
extensions that did not work (at least out of the box) in Python 2.7.

Raul






On 14/12/2012 1:09 AM, Sturla Molden wrote:
> So when upgrading everything you prefer to keep the bugs in 2.6 that were 
> squashed in 2.7? Who has taught IT managers that older and more buggy 
> versions of software are more "professional" and better for corporate 
> environments?
>
> Sturla
>
>
> Den 14. des. 2012 kl. 05:14 skrev Raul Cota :
>
>>
>> +1 from me
>>
>> For what is worth, we are just moving forward from Python 2.2 / Numeric
>> and are going to 2.6 and it has been rather painful because of the
>> several little details of extensions and other subtleties. I believe we
>> will settle there for a while. For companies like ours, it is a big
>> problem to upgrade versions. There is always this or that hiccup that
>> "works great" in a version but not so much in another and we also have
>> all sorts of extensions.
>>
>>
>>
>> Raul
>>
>>
>>
>>
>>
>> On 13/12/2012 9:34 AM, Charles R Harris wrote:
>>> Time to raise this topic again. Opinions welcome.
>>>
>>> Chuck
>>>
>>>
>>> ___
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to drop python 2.4 support in numpy 1.8

[Raul Cota]

+1 from me

For what is worth, we are just moving forward from Python 2.2 / Numeric 
and are going to 2.6 and it has been rather painful because of the 
several little details of extensions and other subtleties. I believe we 
will settle there for a while. For companies like ours, it is a big 
problem to upgrade versions. There is always this or that hiccup that 
"works great" in a version but not so much in another and we also have 
all sorts of extensions.



Raul





On 13/12/2012 9:34 AM, Charles R Harris wrote:
> Time to raise this topic again. Opinions welcome.
>
> Chuck
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] how do I specify maximum line length when using savetxt?

[Raul Cota]

assuming savetxt does not support it,

I modified a bit of code I had to do what I think you need ONLY works 
for a 1D array and wrapped it into a function that writes in properly 
formatted columns. I didn't really test it other than what is there. I 
"dressed" it like savetxt but the glaring difference is that it goes off 
significant digits as opposed to format.


"""
import numpy


def mysavetxt_forvector(fname, x, sigDigs, delimiter, newline, 
maxCharsPLine, fmode='w'):

 padSize = sigDigs + 6  #How many characters per number including 
empty space
 fmt = str(padSize) + '.' + str(sigDigs) + 'g' #e.g. 13.7g'
 asTxtLst = map(lambda val: format(val, fmt), a) #from array to list 
of formatted strings

 #how many cols max ?
 cols = maxCharsPLine/(padSize + len(delimiter))

 #write to file
 size = len(asTxtLst)
 col = 0
 f = open(fname, fmode)
 while col < size:
 f.write(delimiter.join(asTxtLst[col:col+cols]) )
 f.write(newline)
 col += cols
 f.close()


#Test it
a = numpy.ones(34, dtype='float64') * 7./3.
a[3] = 123564234.0002345
a[5] = 1
a[7] = -123564234.0002345
a[9] = -.23453456345
sigDigs = 7
maxCharsPLine = 80
delimiter = ','
newline = '\n'
fname = 'temp.out'
mysavetxt_forvector(fname, a, sigDigs, delimiter, newline, maxCharsPLine)

#append on this one
maxCharsPLine = 33
mysavetxt_forvector(fname, a, sigDigs, delimiter, newline, 
maxCharsPLine, fmode='a')


"""

Raul





On 05/12/2012 4:40 PM, Mark Bakker wrote:
> I guess I wasn't explicit enough.
> Say I have an array with 100 numbers and I want to write it to a file
> with 6 numbers on each line (and hence, only 4 on the last line).
> Can I use savetxt to do that?
> What other easy tool does numpy have to do that?
> Thanks,
> Mark
>
> On 5. des. 2012, at 22:35, Mark Bakker wrote:
>
>> Hello List,
>>
>> I want to write a large array to file, and each line can only be 80
>> characters long.
>> Can I use savetxt to do that? Where would I specify the maximum line length?
>
> If you specify the format, %10.3f for instance, you will know the max
> line length if you also know the array shape.
>
>
>> Or is there a better way to do this?
>
> Probably 1000 ways to accomplish the same thing out there, sure.
>
> Cheers
> Paul
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Speed bottlenecks on simple tasks - suggested improvement

[Raul Cota]

Chris,

thanks for the feedback,

fyi,
the minor changes I talked about have different performance enhancements 
depending on scenario,

e.g,

1) Array * Array
point = array( [2.0, 3.0])
scale = array( [2.4, 0.9] )

retVal = point * scale
#The line above runs 1.1 times faster with my new code (but it runs 3 
times faster in Numeric in Python 2.2)
#i.e. pretty meaningless but still far from old Numeric

2) Array * Tuple (item by item)
point = array( [2.0, 3.0])
scale =  (2.4, 0.9 )

retVal = point[0] < scale[0], point[1] < scale[1]
#The line above runs 1.8 times faster with my new code (but it runs 6.8 
times faster in Numeric in Python 2.2)
#i.e. pretty decent speed up but quite far from old Numeric


I am not saying that I would ever do something exactly like (2) in my 
code nor am I saying that the changes in NumPy Vs Numeric are not 
beneficial. My point is that performance in small size problems is 
fairly far from what it used to be in Numeric particularly when dealing 
with scalars and it is problematic at least to me.


I am currently looking around to see if there are practical ways to 
speed things up without slowing anything else down. Will keep you posted.


regards,

Raul


On 03/12/2012 12:49 PM, Chris Barker - NOAA Federal wrote:
> Raul,
>
> Thanks for doing this work -- both the profiling and actual
> suggestions for how to improve the code -- whoo hoo!
>
> In general, it seem that numpy performance for scalars and very small
> arrays (i.e (2,), (3,) maybe (3,3), the kind of thing that you'd use
> to hold a coordinate point or the like, not small as in "fits in
> cache") is pretty slow. In principle, a basic array scalar operation
> could be as fast as a numpy native numeric type, and it would be great
> is small array operations were, too.
>
> It may be that the route to those performance improvements is
> special-case code, which is ugly, but I think could really be worth it
> for the common types and operations.
>
> I'm really out of my depth for suggesting (or contributing) actual
> soluitons, but +1 for the idea!
>
> -Chris
>
> NOTE: Here's a example of what I'm talking about -- say you are
> scaling an (x,y) point by a (s_x, s_y) scale factor:
>
> def numpy_version(point, scale):
>  return point * scale
>
>
> def tuple_version(point, scale):
>  return (point[0] * scale[0], point[1] * scale[1])
>
>
> In [36]: point_arr, sca
> scale  scale_arr
>
> In [36]: point_arr, scale_arr
> Out[36]: (array([ 3.,  5.]), array([ 2.,  3.]))
>
> In [37]: timeit tuple_version(point, scale)
> 100 loops, best of 3: 397 ns per loop
>
> In [38]: timeit numpy_version(point_arr, scale_arr)
> 10 loops, best of 3: 2.32 us per loop
>
> It would be great if numpy could get closer to tuple performance for
> this sor tof thing...
>
>
> -Chris
>
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Speed bottlenecks on simple tasks - suggested improvement

[Raul Cota]

On 03/12/2012 4:14 AM, Nathaniel Smith wrote:
> On Mon, Dec 3, 2012 at 1:28 AM, Raul Cota  wrote:
>> I finally decided to track down the problem and I started by getting
>> Python 2.6 from source and profiling it in one of my cases. By far the
>> biggest bottleneck came out to be PyString_FromFormatV which is a
>> function to assemble a string for a Python error caused by a failure to
>> find an attribute when "multiarray" calls PyObject_GetAttrString. This
>> function seems to get called way too often from NumPy. The real
>> bottleneck of trying to find the attribute when it does not exist is not
>> that it fails to find it, but that it builds a string to set a Python
>> error. In other words, something as simple as "a[0] < 3.5" internally
>> result in a call to set a python error .
>>
>> I downloaded NumPy code (for Python 2.6) and tracked down all the calls
>> like this,
>>
>>ret = PyObject_GetAttrString(obj, "__array_priority__");
>>
>> and changed to
>>   if (PyList_CheckExact(obj) ||  (Py_None == obj) ||
>>   PyTuple_CheckExact(obj) ||
>>   PyFloat_CheckExact(obj) ||
>>   PyInt_CheckExact(obj) ||
>>   PyString_CheckExact(obj) ||
>>   PyUnicode_CheckExact(obj)){
>>   //Avoid expensive calls when I am sure the attribute
>>   //does not exist
>>   ret = NULL;
>>   }
>>   else{
>>   ret = PyObject_GetAttrString(obj, "__array_priority__");
>>
>> ( I think I found about 7 spots )
> If the problem is the exception construction, then maybe this would
> work about as well?
>
> if (PyObject_HasAttrString(obj, "__array_priority__") {
>  ret = PyObject_GetAttrString(obj, "__array_priority__");
> } else {
>  ret = NULL;
> }
>
> If so then it would be an easier and more reliable way to accomplish this.

I did think of that one but at least in Python 2.6 the implementation is 
just a wrapper to PyObject_GetAttrSting that clears the error

"""
PyObject_HasAttrString(PyObject *v, const char *name)
{
 PyObject *res = PyObject_GetAttrString(v, name);
 if (res != NULL) {
 Py_DECREF(res);
 return 1;
 }
 PyErr_Clear();
 return 0;
}
"""

so it is just as bad when it fails and a waste when it succeeds (it will 
end up finding it twice).
In my opinion, Python's source code should offer a version of 
PyObject_GetAttrString that does not raise an error but that is a 
completely different topic.


>> I also noticed (not as bad in my case) that calls to PyObject_GetBuffer
>> also resulted in Python errors being set thus unnecessarily slower code.
>>
>> With this change, something like this,
>>   for i in xrange(100):
>>   if a[1] < 35.0:
>>   pass
>>
>> went down from 0.8 seconds to 0.38 seconds.
> Huh, why is PyObject_GetBuffer even getting called in this case?

Sorry for being misleading in an already long and confusing email.
PyObject_GetBuffer
is not getting called doing an "if" call. This call showed up in my 
profiler as a time consuming task that raised python errors 
unnecessarily (not nearly as bad as often as PyObject_GetAttrString ) 
but since I was already there I decided to look into it as well.


The point I was trying to make was that I did both changes (avoiding 
PyObject_GetBuffer, PyObject_GetAttrString) when I came up with the times.


>> A bogus test like this,
>> for i in xrange(100):
>>   a = array([1., 2., 3.])
>>
>> went down from 8.5 seconds to 2.5 seconds.
> I can see why we'd call PyObject_GetBuffer in this case, but not why
> it would take 2/3rds of the total run-time...

Same scenario. This total time includes both changes (avoiding 
PyObject_GetBuffer, PyObject_GetAttrString).
If my memory helps, I believe PyObject_GetBuffer gets called once for 
every 9 times of a call to PyObject_GetAttrString in this scenario.


>> - The core of my problems I think boil down to things like this
>> s = a[0]
>> assigning a float64 into s as opposed to a native float ?
>> Is there any way to hack code to change it to extract a native float
>> instead ? (probably crazy talk, but I thought I'd ask :) ).
>> I'd prefer to not use s = a.item(0) because I would have to change too
>> much code and it is not even that much faster. For example,
>>   for i in xrange(100):
>>   if a.item(1) < 35.0:
>>   pass
>> is 0.23 seconds (as opposed to 0.38 seconds with my suggested changes)
> I'm confused here -- first you say that y

Re: [Numpy-discussion] Speed bottlenecks on simple tasks - suggested improvement

[Raul Cota]

On 02/12/2012 8:31 PM, Travis Oliphant wrote:
> Raul,
>
> This is *fantastic work*. While many optimizations were done 6 years ago 
> as people started to convert their code, that kind of report has trailed off 
> in the last few years.   I have not seen this kind of speed-comparison for 
> some time --- but I think it's definitely beneficial.

I'll clean up a bit as a Macro and comment.


> NumPy still has quite a bit that can be optimized.   I think your example is 
> really great.Perhaps it's worth making a C-API macro out of the short-cut 
> to the attribute string so it can be used by others.It would be 
> interesting to see where your other slow-downs are. I would be interested 
> to see if the slow-math of float64 is hurting you.It would be possible, 
> for example, to do a simple subclass of the ndarray that overloads 
> a[] to be the same as array.item().  The latter syntax 
> returns python objects (i.e. floats) instead of array scalars.
>
> Also, it would not be too difficult to add fast-math paths for int64, 
> float32, and float64 scalars (so they don't go through ufuncs but do 
> scalar-math like the float and int objects in Python.

Thanks. I'll dig a bit more into the code.


>
> A related thing we've been working on lately which might help you is Numba 
> which might help speed up functions that have code like:  "a[0] < 4" :  
> http://numba.pydata.org.
>
> Numba will translate the expression a[0] < 4 to a machine-code address-lookup 
> and math operation which is *much* faster when a is a NumPy array.
> Presently this requires you to wrap your function call in a decorator:
>
> from numba import autojit
>
> @autojit
> def function_to_speed_up(...):
>   pass
>
> In the near future (2-4 weeks), numba will grow the experimental ability to 
> basically replace all your function calls with @autojit versions in a Python 
> function.I would love to see something like this work:
>
> python -m numba filename.py
>
> To get an effective autojit on all the filename.py functions (and optionally 
> on all python modules it imports).The autojit works out of the box today 
> --- you can get Numba from PyPI (or inside of the completely free Anaconda 
> CE) to try it out.

This looks very interesting. Will check it out.


> Best,
>
> -Travis
>
>
>
>
> On Dec 2, 2012, at 7:28 PM, Raul Cota wrote:
>
>> Hello,
>>
>> First a quick summary of my problem and at the end I include the basic
>> changes I am suggesting to the source (they may benefit others)
>>
>> I am ages behind in times and I am still using Numeric in Python 2.2.3.
>> The main reason why it has taken so long to upgrade is because NumPy
>> kills performance on several of my tests.
>>
>> I am sorry if this topic has been discussed before. I tried parsing the
>> mailing list and also google and all I found were comments related to
>> the fact that such is life when you use NumPy for small arrays.
>>
>> In my case I have several thousands of lines of code where data
>> structures rely heavily on Numeric arrays but it is unpredictable if the
>> problem at hand will result in large or small arrays. Furthermore, once
>> the vectorized operations complete, the values could be assigned into
>> scalars and just do simple math or loops. I am fairly sure the core of
>> my problems is that the 'float64' objects start propagating all over the
>> program data structures (not in arrays) and they are considerably slower
>> for just about everything when compared to the native python float.
>>
>> Conclusion, it is not practical for me to do a massive re-structuring of
>> code to improve speed on simple things like "a[0] < 4" (assuming "a" is
>> an array) which is about 10 times slower than "b < 4" (assuming "b" is a
>> float)
>>
>>
>> I finally decided to track down the problem and I started by getting
>> Python 2.6 from source and profiling it in one of my cases. By far the
>> biggest bottleneck came out to be PyString_FromFormatV which is a
>> function to assemble a string for a Python error caused by a failure to
>> find an attribute when "multiarray" calls PyObject_GetAttrString. This
>> function seems to get called way too often from NumPy. The real
>> bottleneck of trying to find the attribute when it does not exist is not
>> that it fails to find it, but that it builds a string to set a Python
>> error. In other words, something as simple as "a[0] < 3.5" internally
>> result in a call to set a python error .
>>
&

Re: [Numpy-discussion] Speed bottlenecks on simple tasks - suggested improvement

[Raul Cota]

Thanks Christoph.

It seemed to work. Will do profile runs today/tomorrow and see what come 
out.


Raul



On 02/12/2012 7:33 PM, Christoph Gohlke wrote:
> On 12/2/2012 5:28 PM, Raul Cota wrote:
>> Hello,
>>
>> First a quick summary of my problem and at the end I include the basic
>> changes I am suggesting to the source (they may benefit others)
>>
>> I am ages behind in times and I am still using Numeric in Python 2.2.3.
>> The main reason why it has taken so long to upgrade is because NumPy
>> kills performance on several of my tests.
>>
>> I am sorry if this topic has been discussed before. I tried parsing the
>> mailing list and also google and all I found were comments related to
>> the fact that such is life when you use NumPy for small arrays.
>>
>> In my case I have several thousands of lines of code where data
>> structures rely heavily on Numeric arrays but it is unpredictable if the
>> problem at hand will result in large or small arrays. Furthermore, once
>> the vectorized operations complete, the values could be assigned into
>> scalars and just do simple math or loops. I am fairly sure the core of
>> my problems is that the 'float64' objects start propagating all over the
>> program data structures (not in arrays) and they are considerably slower
>> for just about everything when compared to the native python float.
>>
>> Conclusion, it is not practical for me to do a massive re-structuring of
>> code to improve speed on simple things like "a[0] < 4" (assuming "a" is
>> an array) which is about 10 times slower than "b < 4" (assuming "b" is a
>> float)
>>
>>
>> I finally decided to track down the problem and I started by getting
>> Python 2.6 from source and profiling it in one of my cases. By far the
>> biggest bottleneck came out to be PyString_FromFormatV which is a
>> function to assemble a string for a Python error caused by a failure to
>> find an attribute when "multiarray" calls PyObject_GetAttrString. This
>> function seems to get called way too often from NumPy. The real
>> bottleneck of trying to find the attribute when it does not exist is not
>> that it fails to find it, but that it builds a string to set a Python
>> error. In other words, something as simple as "a[0] < 3.5" internally
>> result in a call to set a python error .
>>
>> I downloaded NumPy code (for Python 2.6) and tracked down all the calls
>> like this,
>>
>> ret = PyObject_GetAttrString(obj, "__array_priority__");
>>
>> and changed to
>>if (PyList_CheckExact(obj) ||  (Py_None == obj) ||
>>PyTuple_CheckExact(obj) ||
>>PyFloat_CheckExact(obj) ||
>>PyInt_CheckExact(obj) ||
>>PyString_CheckExact(obj) ||
>>PyUnicode_CheckExact(obj)){
>>//Avoid expensive calls when I am sure the attribute
>>//does not exist
>>ret = NULL;
>>}
>>else{
>>ret = PyObject_GetAttrString(obj, "__array_priority__");
>>
>>
>>
>> ( I think I found about 7 spots )
>>
>>
>> I also noticed (not as bad in my case) that calls to PyObject_GetBuffer
>> also resulted in Python errors being set thus unnecessarily slower code.
>>
>>
>> With this change, something like this,
>>for i in xrange(100):
>>if a[1] < 35.0:
>>pass
>>
>> went down from 0.8 seconds to 0.38 seconds.
>>
>> A bogus test like this,
>> for i in xrange(100):
>>a = array([1., 2., 3.])
>>
>> went down from 8.5 seconds to 2.5 seconds.
>>
>>
>>
>> Altogether, these simple changes got me half way to the speed I used to
>> get in Numeric and I could not see any slow down in any of my cases that
>> benefit from heavy array manipulation. I am out of ideas on how to
>> improve further though.
>>
>> Few questions:
>> - Is there any interest for me to provide the exact details of the code
>> I changed ?
>>
>> - I managed to compile NumPy through setup.py but I am not sure how to
>> force it to generate pdb files from my Visual Studio Compiler. I need
>> the pdb files such that I can run my profiler on NumPy. Anybody has any
>> experience with this ? (Visual Studio)
>
> Change the compiler and linker flags in
> Python\Lib\distutils\msvc9compiler.py to:
>
> self.compile_options = ['/nologo', '/Ox', '/MD&#

[Numpy-discussion] Speed bottlenecks on simple tasks - suggested improvement

[Raul Cota]

Hello,

First a quick summary of my problem and at the end I include the basic 
changes I am suggesting to the source (they may benefit others)

I am ages behind in times and I am still using Numeric in Python 2.2.3. 
The main reason why it has taken so long to upgrade is because NumPy 
kills performance on several of my tests.

I am sorry if this topic has been discussed before. I tried parsing the 
mailing list and also google and all I found were comments related to 
the fact that such is life when you use NumPy for small arrays.

In my case I have several thousands of lines of code where data 
structures rely heavily on Numeric arrays but it is unpredictable if the 
problem at hand will result in large or small arrays. Furthermore, once 
the vectorized operations complete, the values could be assigned into 
scalars and just do simple math or loops. I am fairly sure the core of 
my problems is that the 'float64' objects start propagating all over the 
program data structures (not in arrays) and they are considerably slower 
for just about everything when compared to the native python float.

Conclusion, it is not practical for me to do a massive re-structuring of 
code to improve speed on simple things like "a[0] < 4" (assuming "a" is 
an array) which is about 10 times slower than "b < 4" (assuming "b" is a 
float)


I finally decided to track down the problem and I started by getting 
Python 2.6 from source and profiling it in one of my cases. By far the 
biggest bottleneck came out to be PyString_FromFormatV which is a 
function to assemble a string for a Python error caused by a failure to 
find an attribute when "multiarray" calls PyObject_GetAttrString. This 
function seems to get called way too often from NumPy. The real 
bottleneck of trying to find the attribute when it does not exist is not 
that it fails to find it, but that it builds a string to set a Python 
error. In other words, something as simple as "a[0] < 3.5" internally 
result in a call to set a python error .

I downloaded NumPy code (for Python 2.6) and tracked down all the calls 
like this,

  ret = PyObject_GetAttrString(obj, "__array_priority__");

and changed to
 if (PyList_CheckExact(obj) ||  (Py_None == obj) ||
 PyTuple_CheckExact(obj) ||
 PyFloat_CheckExact(obj) ||
 PyInt_CheckExact(obj) ||
 PyString_CheckExact(obj) ||
 PyUnicode_CheckExact(obj)){
 //Avoid expensive calls when I am sure the attribute
 //does not exist
 ret = NULL;
 }
 else{
 ret = PyObject_GetAttrString(obj, "__array_priority__");



( I think I found about 7 spots )


I also noticed (not as bad in my case) that calls to PyObject_GetBuffer 
also resulted in Python errors being set thus unnecessarily slower code.


With this change, something like this,
 for i in xrange(100):
 if a[1] < 35.0:
 pass

went down from 0.8 seconds to 0.38 seconds.

A bogus test like this,
for i in xrange(100):
 a = array([1., 2., 3.])

went down from 8.5 seconds to 2.5 seconds.



Altogether, these simple changes got me half way to the speed I used to 
get in Numeric and I could not see any slow down in any of my cases that 
benefit from heavy array manipulation. I am out of ideas on how to 
improve further though.

Few questions:
- Is there any interest for me to provide the exact details of the code 
I changed ?

- I managed to compile NumPy through setup.py but I am not sure how to 
force it to generate pdb files from my Visual Studio Compiler. I need 
the pdb files such that I can run my profiler on NumPy. Anybody has any 
experience with this ? (Visual Studio)

- The core of my problems I think boil down to things like this
s = a[0]
assigning a float64 into s as opposed to a native float ?
Is there any way to hack code to change it to extract a native float 
instead ? (probably crazy talk, but I thought I'd ask :) ).
I'd prefer to not use s = a.item(0) because I would have to change too 
much code and it is not even that much faster. For example,
 for i in xrange(100):
 if a.item(1) < 35.0:
 pass
is 0.23 seconds (as opposed to 0.38 seconds with my suggested changes)


I apologize again if this topic has already been discussed.


Regards,

Raul


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion