Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-04 Thread Gökhan Sever
On Thu, Sep 3, 2009 at 12:22 AM, Robert Kern robert.k...@gmail.com wrote: On Wed, Sep 2, 2009 at 23:59, Gökhan Severgokhanse...@gmail.com wrote: Robert, You must have thrown a couple RTFM's while replying my emails :) Not really. There's no manual for this. Greg Wilson's _Data

[Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
Hello, I want to be able to parse a binary file which hold information regarding to experiment configuration and data obviously. Both configuration and data sections are variable-length. A chuck this data is shown as below (after a binary read operation)

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Robert Kern
On Wed, Sep 2, 2009 at 09:38, Gökhan Severgokhanse...@gmail.com wrote: Hello, I want to be able to parse a binary file which hold information regarding to experiment configuration and data obviously. Both configuration and data sections are variable-length. A chuck this data is shown as below

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Sturla Molden
Gökhan Sever skrev: What would be wisest and fastest way to tackle this issue? Get the format, read the binary data directly, skip the ascii/regex part. I sometimes use recarrays with formatted binary data; just constructing a dtype and use numpy.fromfile to read. That works when the binary

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
On Wed, Sep 2, 2009 at 10:11 AM, Robert Kern robert.k...@gmail.com wrote: On Wed, Sep 2, 2009 at 09:38, Gökhan Severgokhanse...@gmail.com wrote: Hello, I want to be able to parse a binary file which hold information regarding to experiment configuration and data obviously. Both

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
On Wed, Sep 2, 2009 at 10:34 AM, Sturla Molden stu...@molden.no wrote: Gökhan Sever skrev: What would be wisest and fastest way to tackle this issue? Get the format, read the binary data directly, skip the ascii/regex part. I sometimes use recarrays with formatted binary data; just

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Robert Kern
On Wed, Sep 2, 2009 at 11:53, Gökhan Severgokhanse...@gmail.com wrote: How to use recarrays with variable-length data fields as well as metadata? You don't. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Citi, Luca
If I understand the problem... if you are 100% sure that ', ' only occurs between fields and never within, you can use the 'split' method of the string which could be faster than regexp in this simple case. ___ NumPy-Discussion mailing list

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
On Wed, Sep 2, 2009 at 12:01 PM, Citi, Luca lc...@essex.ac.uk wrote: If I understand the problem... if you are 100% sure that ', ' only occurs between fields and never within, you can use the 'split' method of the string which could be faster than regexp in this simple case.

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
On Wed, Sep 2, 2009 at 12:04 PM, Robert Kern robert.k...@gmail.com wrote: On Wed, Sep 2, 2009 at 11:53, Gökhan Severgokhanse...@gmail.com wrote: How to use recarrays with variable-length data fields as well as metadata? You don't. -- Robert Kern I have come to believe that the whole

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Robert Kern
On Wed, Sep 2, 2009 at 12:27, Gökhan Severgokhanse...@gmail.com wrote: On Wed, Sep 2, 2009 at 12:01 PM, Citi, Luca lc...@essex.ac.uk wrote: If I understand the problem... if you are 100% sure that ', ' only occurs between fields and never within, you can use the 'split' method of the string

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Robert Kern
On Wed, Sep 2, 2009 at 12:33, Gökhan Severgokhanse...@gmail.com wrote: How your find suggestion work? It just returns the location of the first occurrence. http://docs.python.org/library/stdtypes.html#str.find str.find(sub[, start[, end]]) Return the lowest index in the string where

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
On Wed, Sep 2, 2009 at 12:29 PM, Robert Kern robert.k...@gmail.com wrote: On Wed, Sep 2, 2009 at 12:27, Gökhan Severgokhanse...@gmail.com wrote: On Wed, Sep 2, 2009 at 12:01 PM, Citi, Luca lc...@essex.ac.uk wrote: If I understand the problem... if you are 100% sure that ', ' only

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
On Wed, Sep 2, 2009 at 12:29 PM, Robert Kern robert.k...@gmail.com wrote: On Wed, Sep 2, 2009 at 12:27, Gökhan Severgokhanse...@gmail.com wrote: On Wed, Sep 2, 2009 at 12:01 PM, Citi, Luca lc...@essex.ac.uk wrote: If I understand the problem... if you are 100% sure that ', ' only

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
On Wed, Sep 2, 2009 at 12:46 PM, Robert Kern robert.k...@gmail.com wrote: On Wed, Sep 2, 2009 at 12:33, Gökhan Severgokhanse...@gmail.com wrote: How your find suggestion work? It just returns the location of the first occurrence. http://docs.python.org/library/stdtypes.html#str.find

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Robert Kern
On Wed, Sep 2, 2009 at 13:28, Gökhan Severgokhanse...@gmail.com wrote: Put the reference manual in: http://drop.io/1plh5rt First few pages describe the data format they use. Ah. The fields are *not* delimited by a fixed value. Regexes are no help to you for pulling out the information you

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Gökhan Sever
On Wed, Sep 2, 2009 at 1:58 PM, Robert Kern robert.k...@gmail.com wrote: On Wed, Sep 2, 2009 at 13:28, Gökhan Severgokhanse...@gmail.com wrote: Put the reference manual in: http://drop.io/1plh5rt First few pages describe the data format they use. Ah. The fields are *not* delimited by

Re: [Numpy-discussion] Fastest way to parsing a specific binay file

2009-09-02 Thread Robert Kern
On Wed, Sep 2, 2009 at 23:59, Gökhan Severgokhanse...@gmail.com wrote: Robert, You must have thrown a couple RTFM's while replying my emails :) Not really. There's no manual for this. Greg Wilson's _Data Crunching_ may be a good general introduction to how to think about these problems.