Re: [Numpy-discussion] load from text files Pull Request Review
On 9/8/11 1:43 PM, Christopher Jordan-Squire wrote:
> I just ran a quick test on my machine of this idea. With
>
> dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)])
> temp = np.empty((), dtype=dt)
> temp2 = np.zeros(1,dtype=dt)
>
> In [96]: def f():
> ...: l=[0]*3
> ...: l[0] = 2.54
> ...: l[1] = 4
> ...: l[2] = 2.3645
> ...: j = tuple(l)
> ...: temp2[0] = j
>
> vs
>
>
> In [97]: def g():
> ...: temp['x'] = 2.54
> ...: temp['y'] = 4
> ...: temp['z'] = 2.3645
> ...: temp2[0] = temp
> ...:
>
> The timing results were 2.73 us for f and 3.43 us for g. So good idea,
> but it doesn't appear to be faster. (Though the difference wasn't
> nearly as dramatic as I thought it would be, based on Pauli's
> comment.)
my guess is that the lines like: temp['x'] = 2.54 are slower (it
requires a dict lookup, and a conversion from a python type to a "raw" type)
and
temp2[0] = temp
is faster, as that doesn't require any conversion.
Which means that if you has a larger struct dtype, it would be even
slower, so clearly not the way to go for performance.
It would be nice to have a higher performing struct dtype scalar -- as
it is ordered, it might be nice to be able to index it with either the
name or an numeric index.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] load from text files Pull Request Review
On Wed, Sep 7, 2011 at 2:52 PM, Chris.Barker wrote:
> On 9/2/11 2:45 PM, Christopher Jordan-Squire wrote:
>> It doesn't have to parse the entire file to determine the dtypes. It
>> builds up a regular expression for what it expects to see, in terms of
>> dtypes. Then it just loops over the lines, only parsing if the regular
>> expression doesn't match. It seems that a regex match is fast, but a
>> regex fail is expensive.
>
> interesting -- I wouldn't have expected a regex to be faster that simple
> parsing, but that's why you profile!
>
>> Setting array elements is not as fast for the masked record arrays.
>> You must set entire rows at a time, so I have to build up each row as
>> a list, and convert to a tuple, and then stuff it in the array.
>
> hmmm -- that is a lot -- I was thinking of simple "set a value in an
> array". I"ve also done a bunch of this in C, where's it's really fast.
>
> However, rather than:
>
> build a row as a list
> build a row as a tuple
> stuff into array
>
> could you create an empty array scalar, and fill that, then put that in
> your array:
>
> In [4]: dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)])
>
> In [5]: dt
> Out[5]: dtype([('x', '
> In [6]: temp = np.empty((), dtype=dt)
>
> In [9]: temp['x'] = 3
>
> In [10]: temp['y'] = 4
>
> In [11]: temp['z'] = 5
>
> In [13]: a = np.zeros((4,), dtype = dt)
>
> In [14]: a[0] = temp
>
> In [15]: a
> Out[15]:
> array([(3.0, 4, 5.0), (0.0, 0, 0.0), (0.0, 0, 0.0), (0.0, 0, 0.0)],
> dtype=[('x', '
>
> (and you could pass the array scalar into accumulator as well)
>
> maybe it wouldn't be any faster, but with re-using temp, and one less
> list-tuple conversion, and fewer python type to numpy type conversions,
> maybe it would.
>
I just ran a quick test on my machine of this idea. With
dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)])
temp = np.empty((), dtype=dt)
temp2 = np.zeros(1,dtype=dt)
In [96]: def f():
...: l=[0]*3
...: l[0] = 2.54
...: l[1] = 4
...: l[2] = 2.3645
...: j = tuple(l)
...: temp2[0] = j
vs
In [97]: def g():
...: temp['x'] = 2.54
...: temp['y'] = 4
...: temp['z'] = 2.3645
...: temp2[0] = temp
...:
The timing results were 2.73 us for f and 3.43 us for g. So good idea,
but it doesn't appear to be faster. (Though the difference wasn't
nearly as dramatic as I thought it would be, based on Pauli's
comment.)
-Chris JS
>> it's even slower for the record arrays with missing data because I
>> must branch between adding missing data versus adding real data. Might
>> that be the reason for the slower performance than you'd expect?
>
> could be -- I haven't thought about the missing data part much.
>
>> I wonder if there are any really important cases where you'd actually
>> lose something by simply recasting an entry to another dtype, as Derek
>> suggested.
>
> In general, it won't be a simple re-cast -- it will be a copy to a
> subset -- which may be hard to write the code, but would save having to
> re-parse the data.
>
>
> Anyway, you know the issues, this is good stuff either way.
>
> -Chris
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> chris.bar...@noaa.gov
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] strange test failure in Cython code with numpy master
Probably has something to do with 'arr' having a statically defined type of cnp.ndarray. -Mark On Thu, Sep 8, 2011 at 12:59 PM, Ralf Gommers wrote: > Hi, > > There's a test failure in scipy/io/matlab/mio_utils that shows up with > numpy master but not 1.5.1, see > http://projects.scipy.org/scipy/ticket/1512 > I have a fix here: > https://github.com/rgommers/scipy/commit/4ade7829649b9e2c251f5f7f370781b16fc13612, > but I don't really understand why the code is failing in the first place. > This is the relevant part: > > -arr = np.squeeze(arr) > -if not arr.shape and arr.dtype.isbuiltin: # 0d coverted to scalar > -return arr.item() > -return arr > +arr2 = np.squeeze(arr) > +if (not arr2.shape) and arr2.dtype.isbuiltin: # 0d coverted to scalar > +return arr2.item() > +return arr2 > > All it does is rename arr to arr2. It fails with both Cython 0.13 and 0.15. > Any idea? > > Ralf > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] strange test failure in Cython code with numpy master
Hi, There's a test failure in scipy/io/matlab/mio_utils that shows up with numpy master but not 1.5.1, see http://projects.scipy.org/scipy/ticket/1512 I have a fix here: https://github.com/rgommers/scipy/commit/4ade7829649b9e2c251f5f7f370781b16fc13612, but I don't really understand why the code is failing in the first place. This is the relevant part: -arr = np.squeeze(arr) -if not arr.shape and arr.dtype.isbuiltin: # 0d coverted to scalar -return arr.item() -return arr +arr2 = np.squeeze(arr) +if (not arr2.shape) and arr2.dtype.isbuiltin: # 0d coverted to scalar +return arr2.item() +return arr2 All it does is rename arr to arr2. It fails with both Cython 0.13 and 0.15. Any idea? Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64
On 08/09/2011 16:16, Jim Vickroy wrote: > On 9/8/2011 6:09 AM, "V. Armando Solé" wrote: >> Have you tried to install Visual Studio 2008 Express edition (plus the >> windows SDK to be able to compile 64 bit code)? >> >> Armando > Armando, "Visual Studio 2008 Professional" is installed on the computer > as well as "Intel Visual Fortran Composer XE 2011". > > f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I > tried gfortran. > > The "Win64" reference, in the Exception, is puzzling to me since this is > a 32-bit computer. > Oh! I totally misunderstood the situation. I thought the problem was the missing compiler. All what I do with python and the intel fortran compiler is to compile numpy. Just in case it helps you, I set my environment from the console by running a bat file with the following content (I am on 64 bit but you could easily tailor it to your needs): "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" -version:v7.0 call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat" call "C:\Program Files (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell rem call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64" rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" /x64 /Release set PATH=C:\Python27;C:\Python27\Scripts;%PATH% set PATH="C:\Program Files (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% Perhaps that helps you to set a working environment. All what I can tell you is that with that environment, if I run "python f2py.py -c --help-fcompiler" it finds the intel compiler. Good luck, Armando ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64
On 9/8/2011 6:09 AM, "V. Armando Solé" wrote:
> Have you tried to install Visual Studio 2008 Express edition (plus the
> windows SDK to be able to compile 64 bit code)?
>
> Armando
Armando, "Visual Studio 2008 Professional" is installed on the computer
as well as "Intel Visual Fortran Composer XE 2011".
f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I
tried gfortran.
The "Win64" reference, in the Exception, is puzzling to me since this is
a 32-bit computer.
>
> On 08/09/2011 13:56, Jim Vickroy wrote:
>> Hello All, I'm attempting to create a python wrapper, for a Fortran
>> subroutine, using f2py.
>>
>> My system details are:
>>
>>>>> sys.version '2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500
>> 32 bit (Intel)]'
>>>>> sys.getwindowsversion() (5, 1, 2600, 2, 'Service Pack 3')
>>>>> scipy.__version__ '0.7.1'
>>>>> numpy.__version__ '1.4.0'
>> C:\>gfortran -dumpversion
>> 4.7.0
>>
>>
>> C:\Python26\Lib\site-packages\numpy\f2py>f2py.py -c --help-fcompiler
>> Traceback (most recent call last):
>> File "C:\Python26\Scripts\f2py.py", line 24, in
>>main()
>> File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557,
>> in main
>>run_compile()
>> File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543,
>> in run_compile
>>setup(ext_modules = [ext])
>> File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line
>> 186, in setup
>>return old_setup(**new_attr)
>> File "C:\Python26\lib\distutils\core.py", line 138, in setup
>>ok = dist.parse_command_line()
>> File "C:\Python26\lib\distutils\dist.py", line 460, in
>> parse_command_line
>>args = self._parse_command_opts(parser, args)
>> File "C:\Python26\lib\distutils\dist.py", line 574, in
>> _parse_command_opts
>>func()
>> File
>> "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py",
>> line 13, in show_fortran_compilers
>>show_fcompilers(dist)
>> File
>> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py",
>> line 855, in show_fcompilers
>>c.customize(dist)
>> File
>> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py",
>> line 525, in customize
>>self.set_libraries(self.get_libraries())
>> File
>> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\gnu.py", line
>> 306, in get_libraries
>>raise NotImplementedError("Only MS compiler supported with gfortran
>> on win64")
>> NotImplementedError: Only MS compiler supported with gfortran on win64
>>
>>
>> Could someone help me to resolve this?
>>
>> Thanks, -- jv
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64
Have you tried to install Visual Studio 2008 Express edition (plus the
windows SDK to be able to compile 64 bit code)?
Armando
On 08/09/2011 13:56, Jim Vickroy wrote:
> Hello All, I'm attempting to create a python wrapper, for a Fortran
> subroutine, using f2py.
>
> My system details are:
>
> >>> sys.version '2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500
> 32 bit (Intel)]'
> >>> sys.getwindowsversion() (5, 1, 2600, 2, 'Service Pack 3')
> >>> scipy.__version__ '0.7.1'
> >>> numpy.__version__ '1.4.0'
> C:\>gfortran -dumpversion
> 4.7.0
>
>
> C:\Python26\Lib\site-packages\numpy\f2py>f2py.py -c --help-fcompiler
> Traceback (most recent call last):
> File "C:\Python26\Scripts\f2py.py", line 24, in
> main()
> File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557,
> in main
> run_compile()
> File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543,
> in run_compile
> setup(ext_modules = [ext])
> File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line
> 186, in setup
> return old_setup(**new_attr)
> File "C:\Python26\lib\distutils\core.py", line 138, in setup
> ok = dist.parse_command_line()
> File "C:\Python26\lib\distutils\dist.py", line 460, in parse_command_line
> args = self._parse_command_opts(parser, args)
> File "C:\Python26\lib\distutils\dist.py", line 574, in
> _parse_command_opts
> func()
> File
> "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py",
> line 13, in show_fortran_compilers
> show_fcompilers(dist)
> File
> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py",
> line 855, in show_fcompilers
> c.customize(dist)
> File
> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py",
> line 525, in customize
> self.set_libraries(self.get_libraries())
> File
> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\gnu.py", line
> 306, in get_libraries
> raise NotImplementedError("Only MS compiler supported with gfortran
> on win64")
> NotImplementedError: Only MS compiler supported with gfortran on win64
>
>
> Could someone help me to resolve this?
>
> Thanks, -- jv
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64
Hello All, I'm attempting to create a python wrapper, for a Fortran
subroutine, using f2py.
My system details are:
>>> sys.version '2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500
32 bit (Intel)]'
>>> sys.getwindowsversion() (5, 1, 2600, 2, 'Service Pack 3')
>>> scipy.__version__ '0.7.1'
>>> numpy.__version__ '1.4.0'
C:\>gfortran -dumpversion
4.7.0
C:\Python26\Lib\site-packages\numpy\f2py>f2py.py -c --help-fcompiler
Traceback (most recent call last):
File "C:\Python26\Scripts\f2py.py", line 24, in
main()
File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557,
in main
run_compile()
File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543,
in run_compile
setup(ext_modules = [ext])
File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line
186, in setup
return old_setup(**new_attr)
File "C:\Python26\lib\distutils\core.py", line 138, in setup
ok = dist.parse_command_line()
File "C:\Python26\lib\distutils\dist.py", line 460, in parse_command_line
args = self._parse_command_opts(parser, args)
File "C:\Python26\lib\distutils\dist.py", line 574, in
_parse_command_opts
func()
File
"C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py",
line 13, in show_fortran_compilers
show_fcompilers(dist)
File
"C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py",
line 855, in show_fcompilers
c.customize(dist)
File
"C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py",
line 525, in customize
self.set_libraries(self.get_libraries())
File
"C:\Python26\lib\site-packages\numpy\distutils\fcompiler\gnu.py", line
306, in get_libraries
raise NotImplementedError("Only MS compiler supported with gfortran
on win64")
NotImplementedError: Only MS compiler supported with gfortran on win64
Could someone help me to resolve this?
Thanks, -- jv
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] what to use in buildbot config for numpy testing
There is much more simple: the command nosetests numpy
Of course, nosetests needs to be installed...
Cheers
JB
Le 7 sept. 2011 à 23:44, Chris Kees a écrit :
> Hi Derek,
>
> Thanks! I forgot that python would exit with 0 even if numpy.test()
> fails. That could have taken a while to realize.
>
> Chris
>
> On Wed, Sep 7, 2011 at 4:22 PM, Derek Homeier
> wrote:
>> On 07.09.2011, at 10:52PM, Chris Kees wrote:
>>
>>> Is there a recommended way to run the numpy test suite as a buildbot
>>> test? Just run ad python -c "import numpy; numpy.test" as ShellCommand
>>> object?
>>
>> It would be numpy.test() [or numpy.test('full')]; then it depends on what
>> you need
>> as the return value of your test. I am using for package verification
>>
>> python -c 'import numpy, sys; ret=numpy.test("full");
>> sys.exit(2*len(ret.errors+ret.failures))'
>>
>> so python will return with a value != 0 in case of an unsuccessful test
>> (which it
>> otherwise would not do). But this is just within a simple shell script.
>>
>> Cheers,
>>Derek
>> --
>>
>> Derek Homeier Centre de Recherche Astrophysique de Lyon
>> ENS Lyon 46, Allée d'Italie
>> 69364 Lyon Cedex 07, France +33 1133 47272-8894
>>
>>
>>
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] load from text files Pull Request Review
Wed, 07 Sep 2011 12:52:44 -0700, Chris.Barker wrote:
[clip]
> In [9]: temp['x'] = 3
>
> In [10]: temp['y'] = 4
>
> In [11]: temp['z'] = 5
[clip]
> maybe it wouldn't be any faster, but with re-using temp, and one less
> list-tuple conversion, and fewer python type to numpy type conversions,
> maybe it would.
Structured array assignments have plenty of overhead in Numpy, so it
could be slower, too:
x = np.array((1,2), dtype=[('a', int), ('b', float)])
x2 = [1,2,3]
%timeit x['a'] = 9
10 loops, best of 3: 2.83 us per loop
%timeit x2[0] = 9
100 loops, best of 3: 368 ns per loop
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
