[Numpy-discussion] genfromtxt converter question

I'm trying to read a file containing data formatted as in the following example using genfromtxt and I'm doing something wrong. It almost works. Can someone point out my error, or suggest a simpler solution to the ugly converter function? I thought I'd leave in the commented-out line for future ref

Re: [Numpy-discussion] Unicode characters in a numpy array

On Wed, Jun 15, 2011 at 1:30 PM, Gökhan Sever wrote: > Hello, > > The following snippet works fine for a regular string and prints out > the string without a problem. > > python > Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) > [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 > Type "help", "copy

[Numpy-discussion] Sorting library.

Hi All, Pull request 89 is a set of patches that move the numpy sorting patches into a library. I've tested it on fedora 15 with gcc and would greatly appreciate it if some of you with different os/arch/compilers could test it out to make sure it works correctly. Chuck ___

Re: [Numpy-discussion] code review/build & test for datetime business day API

On 17.06.2011, at 2:02AM, Mark Wiebe wrote: >> ok, that was a lengthy hunt, but it's in printing the string in >> make_iso_8601_date: >> >>tmplen = snprintf(substr, sublen, "%04" NPY_INT64_FMT, dts->year); >>fprintf(stderr, "printed %d[%d]: dts->year=%lld: %s\n", tmplen, sublen, >> dts-

Re: [Numpy-discussion] code review/build & test for datetime business day API

On Thu, Jun 16, 2011 at 5:52 PM, Derek Homeier < de...@astro.physik.uni-goettingen.de> wrote: > Hi Mark, > > On 16.06.2011, at 5:40PM, Mark Wiebe wrote: > > >> >>> np.datetime64('2011-06-16 02:03:04Z', 'D') > >> np.datetime64('-06-16','D') > >> > >> I've tried to track this down in datetime.c,

Re: [Numpy-discussion] code review/build & test for datetime business day API

Hi Mark, On 16.06.2011, at 5:40PM, Mark Wiebe wrote: >> >>> np.datetime64('2011-06-16 02:03:04Z', 'D') >> np.datetime64('-06-16','D') >> >> I've tried to track this down in datetime.c, but unsuccessfully so (i.e. I >> could not connect it >> to any of the dts->year assignments therein). >>

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

On 06/16/2011 02:05 PM, Brandt Belson wrote: Hi all, Thanks for the replies. As mentioned, I'm parallelizing so that I can take many inner products simultaneously (which I agree is embarrassingly parallel). The library I'm writing asks the user to supply a function that takes two objects and r

Re: [Numpy-discussion] code review/build & test for datetime business day API

On Thu, Jun 16, 2011 at 2:00 PM, Mark Wiebe wrote: > I've received some good feedback from Chuck and Ralf on the code and > documentation, respectively, and build testing with proposed fixes for > issues from the previous merge from Derek. I believe the current set of > changes are in good shape

Re: [Numpy-discussion] code review/build & test for datetime business day API

I've received some good feedback from Chuck and Ralf on the code and documentation, respectively, and build testing with proposed fixes for issues from the previous merge from Derek. I believe the current set of changes are in good shape to merge, so would like to proceed with that later today. Ch

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

The fact that you are still passing the myutil.arrayList argument to pool.map means the data is still being pickled. All arguments to pool.map are pickled to be passed to the subprocess. You need to change the function so that it accesses the data directly, then just pass indices to pool.map. Chee

[Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Hi all, Thanks for the replies. As mentioned, I'm parallelizing so that I can take many inner products simultaneously (which I agree is embarrassingly parallel). The library I'm writing asks the user to supply a function that takes two objects and returns their inner product. After all the discussi

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

On 06/16/2011 11:44 AM, Christopher Barker wrote: NOTE: I'm only taking part in this discussion because it's interesting and I hope to learn something. I do hope the OP chimes back in to clarify his needs, but in the meantime... Bruce Southey wrote: Remember that is what the OP wanted to do

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Sturla Molden wrote: > Den 16.06.2011 19:55, skrev Sturla Molden: >> The meta-data for the ndarray (strides, shape, dtype, etc.) and the name >> of the shared memory segment. As there is no fork, we must tell the >> other process where to find the shared memory and what to do with it. got it, than

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 19:55, skrev Sturla Molden: > > The meta-data for the ndarray (strides, shape, dtype, etc.) and the name > of the shared memory segment. As there is no fork, we must tell the > other process where to find the shared memory and what to do with it. > Which by the way is what the share

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 19:47, skrev Christopher Barker: > > What do you need to pickle if you're using shared memory? The meta-data for the ndarray (strides, shape, dtype, etc.) and the name of the shared memory segment. As there is no fork, we must tell the other process where to find the shared memory

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 19:23, skrev Robin: > > If you are on Linux or Mac then fork works nicely so you have read > only shared memory you just have to put it in a module before the fork > (so before pool = Pool() ) and then all the subprocesses can access it > without any pickling required. ie > myutil.da

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Sturla Molden wrote: > On Windows this sucks, because there is no fork system call. Darn -- I do need to support Windows. > Here we are > stuck with multiprocessing and pickle, even if we use shared memory. What do you need to pickle if you're using shared memory? -Chris -- Christopher Bar

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 18:44, skrev Christopher Barker: > > I'll bet this is a big issue, and one I'm curious about how to > address, I have another problem where I need to multi-process, and I'd > love to know a way to pass data to the other process and back > *without* going through pickle. maybe memm

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

On Thu, Jun 16, 2011 at 6:44 PM, Christopher Barker wrote: >> 2. There is also the question of when the process pool is spawned. Though >> I haven't checked, I suspect it happens prior to calling pool.map. But if it >> does not, this is a factor as well, particularly on Windows (less so on >> Linu

Re: [Numpy-discussion] Regression in choose()

Hi Ed, I'm pretty sure that this is "bug" is due to the compile-time constant NPY_MAXARGS defined in include/numpy/ndarraytypes.h I suspect that the versions you are trying it on where it succeeds as a different compile-time constant of that value. NumPy uses a multi-iterator object (PyA

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

NOTE: I'm only taking part in this discussion because it's interesting and I hope to learn something. I do hope the OP chimes back in to clarify his needs, but in the meantime... Bruce Southey wrote: Remember that is what the OP wanted to do, not me. Actually, I don't think that's what the

Re: [Numpy-discussion] 3d ODR

On Thu, Jun 16, 2011 at 06:28, Christian K. wrote: > Hi, > > I need to do fit a 3d surface to a point cloud. This sounds like a job for 3d > orthogonal distance regression. Does anybody know of an implementation? As eat points out, scipy.odr should do the job nicely. You should read at least part

Re: [Numpy-discussion] code review/build & test for datetime business day API

On Wed, Jun 15, 2011 at 5:16 PM, Derek Homeier < de...@astro.physik.uni-goettingen.de> wrote: > On 15.06.2011, at 1:34AM, Mark Wiebe wrote: > > > These functions are now fully implemented and documented. As always, code > reviews are welcome here: > > > > https://github.com/numpy/numpy/pull/87 > >

Re: [Numpy-discussion] datetimes with date vs time units, local time, and time zones

On Thu, Jun 16, 2011 at 9:18 AM, Benjamin Root wrote: > On Wednesday, June 15, 2011, Mark Wiebe wrote: > > Towards a reasonable behavior with regard to local times, I've made the > default repr for datetimes use the C standard library to print them in a > local ISO format. Combined with the ISO8

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

On 06/16/2011 09:30 AM, Sturla Molden wrote: > Den 16.06.2011 16:16, skrev Bruce Southey: >> The idea was to calculate: >> innerProduct =numpy.sum(array1*array2) where array1 and array2 are, in >> general, multidimensional. > If you want to do that in parallel, fast an with minimal overhead, it's >

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 16:16, skrev Bruce Southey: > The idea was to calculate: > innerProduct =numpy.sum(array1*array2) where array1 and array2 are, in > general, multidimensional. If you want to do that in parallel, fast an with minimal overhead, it's a typical OpenMP problem. If the dimensions are unk

Re: [Numpy-discussion] Using numpy.fromfile with structured array skipping some elements.

Den 16.06.2011 15:02, skrev Michael Klitgaard: > Now the data is read in and I can access it, but I have the 'junk' in > the array, which annoys me. > Is there a way to remove the junk data, or skip it with fromfile ? Not that I know of, unless you are ok with a loop (either in Python or C). If i

Re: [Numpy-discussion] datetimes with date vs time units, local time, and time zones

On Wednesday, June 15, 2011, Mark Wiebe wrote: > Towards a reasonable behavior with regard to local times, I've made the > default repr for datetimes use the C standard library to print them in a > local ISO format. Combined with the ISO8601-prescribed behavior of > interpreting datetime string

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

On 06/15/2011 07:25 PM, Sturla Molden wrote: > Den 15.06.2011 23:22, skrev Christopher Barker: >> I think the issue got confused -- the OP was not looking to speed up a >> matrix multiply, but rather to speed up a whole bunch of independent >> matrix multiplies. > I would do it like this: > > 1. Wr

Re: [Numpy-discussion] 3d ODR

Hi, On Thu, Jun 16, 2011 at 2:28 PM, Christian K. wrote: > Hi, > > I need to do fit a 3d surface to a point cloud. This sounds like a job for > 3d > orthogonal distance regression. Does anybody know of an implementation? > How bout http://docs.scipy.org/doc/scipy/reference/odr.htm

[Numpy-discussion] Using numpy.fromfile with structured array skipping some elements.

Hello, I hope this is the right mailings list for a numpy user questions, if not, I'm sorry. Im reading a binary file with numpy.fromfile() The binary file is constructed with some integers for checking the data for corruption. This is how the binary file is constructed: Timestamp       [ 12 by

[Numpy-discussion] 3d ODR

Hi, I need to do fit a 3d surface to a point cloud. This sounds like a job for 3d orthogonal distance regression. Does anybody know of an implementation? Regards, Christian K. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.sc

[Numpy-discussion] [ANN] OpenOpt suite 0.34

Hi all, I'm glad to inform you about new quarterly release 0.34 of the OOSuite package software (OpenOpt, FuncDesigner, SpaceFuncs, DerApproximator) . Main changes: * Python 3 compatibility * Lots of improvements and speedup for interval calculations * Now interalg can obtain

Re: [Numpy-discussion] Regression in choose()

If performance is not critical you could just write your own function for a quick fix, doing something like numpy.array([choices[j][i] for i, j in enumerate([2, 3, 1, 0])]) This 32-array limitation definitely looks weird to me though, doesn't seem to make sense. -=- Olivier 2011/6/16 Ed Schofiel