Re: Elementary string-parsing
Dennis Lee Bieber wrote:
> On Tue, 05 Feb 2008 04:03:04 GMT, Odysseus
> <[EMAIL PROTECTED]> declaimed the following in
> comp.lang.python:
>
>> Sorry, translation problem: I am acquainted with Python's "for" -- if
>> far from fluent with it, so to speak -- but the PS operator that's most
>> similar (traversing a compound object, element by element, without any
>> explicit indexing or counting) is called "forall". PS's "for" loop is
>> similar to BASIC's (and ISTR Fortran's):
>>
>> start_value increment end_value {procedure} for
>>
>> I don't know the proper generic term -- "indexed loop"? -- but at any
>> rate it provides a counter, unlike Python's command of the same name.
>>
> The convention is Python is to use range() (or xrange() ) to
> generate a sequence of "index" values for the for statement to loop
> over:
>
> for i in range([start], end, [step]):
>
> with the caveat that "end" will not be one of the values, start defaults
> to 0, so if you supply range(4) the values become 0, 1, 2, 3 [ie, 4
> values starting at 0].
>
If you have a sequence of values s and you want to associate each with
its index value as you loop over the sequence the easiest way to do this
is the enumerate built-in function:
>>> for x in enumerate(['this', 'is', 'a', 'list']):
... print x
...
(0, 'this')
(1, 'is')
(2, 'a')
(3, 'list')
It's usually (though not always) much more convenient to bind the index
and the value to separate names, as in
>>> for i, v in enumerate(['this', 'is', 'a', 'list']):
... print i, v
...
0 this
1 is
2 a
3 list
[...]
> The whole idea behind the SGML parser is that YOU add methods to
> handle each tag type you need... Also, FYI, there IS an HTML parser (in
> module htmllib) that is already derived from sgmllib.
>
> class PageParser(SGMLParser):
> def __init__(self):
> #need to call the parent __init__, and then
> #initialize any needed attributes -- like someplace to collect
> #the parsed out cell data
> self.row = {}
> self.all_data = []
>
> def start_table(self, attrs):
> self.inTable = True
> .
>
> def end_table(self):
> self.inTable = False
> .
>
> def start_tr(self, attrs):
> if self.inRow:
> #unclosed row!
> self.end_tr()
> self.inRow = True
> self.cellCount = 0
> ...
>
> def end_tr(self):
> self.inRow = False
> # add/append collected row data to master stuff
> self.all_data.append(self.row)
> ...
>
> def start_td(self, attrs):
> if self.inCell:
> self.end_td()
> self.inCell = True
> ...
>
> def end_td(self):
> self.cellCount = self.cellCount + 1
> ...
>
> def handle_data(self, text):
> if self.inTable and self.inRow and self.inCell:
> if self.cellCount == 0:
> #first column stuff
> self.row["Epoch1"] = convert_if_needed(text)
> elif self.cellCount == 1:
> #second column stuff
> ...
>
>
> Hope you don't have nested tables -- it could get ugly as this style
> of parser requires the start_tag()/end_tag() methods to set instance
> attributes for the purpose of tracking state needed in later methods
> (notice the complexity of the handle_data() method just to ensure that
> the text is from a table cell, and not some random text).
>
There is, of course, nothing to stop you building a recursive data
structure, so that encountering a new opening tag such as adds
another level to some stack-like object, and the corresponding closing
tag pops it off again, but this *does* add to the complexity somewhat.
It seems natural that more complex input possibilities lead to more
complex parsers.
> And somewhere before you close the parser, get a handle on the
> collected data...
>
>
> parsed_data = parser.all_data
> parser.close()
> return parsed_data
>
>
>> Why wouldn't one use a dictionary for that?
>>
> The overhead may not be needed... Tuples can also be used as the
> keys /in/ a dictionary.
>
regards
Steve
--
Steve Holden+1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
--
http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
Marc 'BlackJack' Rintsch wrote: > On Tue, 05 Feb 2008 06:19:12 +, Odysseus wrote: > >> In article <[EMAIL PROTECTED]>, >> Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote: >> >>> Another issue is testing. If you rely on global names it's harder to test >>> individual functions. [...] >>> >>> In programs without such global names you see quite clearly in the >>> ``def`` line what the function expects as input. >> Good points, although thorough commenting can go a long way to help on >> both counts. In theory, at least ... > > Won't work in practice so well. Say we have function `f()` and > document that it expects global name `a` to be set to something before > calling it. `f()` is used by other functions so we have to document `a` in > all other functions too. If we change `f()` to rely on global name `b` > too we have to hunt down every function that calls `f()` and add the > documentation for `b` there too. It's much work and error prone. Easy to > get inconsistent or missing documentation this way. > Essentially what Marc is saying is that you want your functions to be as loosely coupled to their environment as practically possible. See http://en.wikipedia.org/wiki/Coupling_(computer_science) [...] regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
On Tue, 05 Feb 2008 06:19:12 +, Odysseus wrote:
> In article <[EMAIL PROTECTED]>,
> Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
>
>> Another issue is testing. If you rely on global names it's harder to test
>> individual functions. [...]
>>
>> In programs without such global names you see quite clearly in the
>> ``def`` line what the function expects as input.
>
> Good points, although thorough commenting can go a long way to help on
> both counts. In theory, at least ...
Won't work in practice so well. Say we have function `f()` and
document that it expects global name `a` to be set to something before
calling it. `f()` is used by other functions so we have to document `a` in
all other functions too. If we change `f()` to rely on global name `b`
too we have to hunt down every function that calls `f()` and add the
documentation for `b` there too. It's much work and error prone. Easy to
get inconsistent or missing documentation this way.
To write or check documentation for a function you have to scan the whole
function body for data in global names and calls to other functions and
repeat the search there. If you don't let functions communicate via global
names you just have to look at the argument list to see the input sources.
>> def main():
>> # Main program comes here.
>>
>> if __name__ == '__main__':
>> main()
>>
>> Then main is called when the script is called as program, but not called if
>> you just import the script as module. For example to test functions or to
>> reuse the code from other scripts.
>
> I'm using "if __name__ == 'main'" now, but only for test inputs (which
> will eventually be read from a config file or passed by the calling
> script -- or something). I hadn't thought of putting code that actually
> does something there. As for writing modules, that's way beyond where I
> want to go at this point: I don't know any C and am not sure I would
> want to ...
What does this have to do with C!? There's no specific C knowledge
involved here.
>> >> assert name.startswith('Name: ')
>
>> It checks if `name` really starts with 'Name: '. This way I turned the
>> comment into code that checks the assertion in the comment.
>
> Good idea to check, although this is actually only one of many
> assumptions I make about the data -- but what happens if the assertion
> fails? The program stops and the interpreter reports an AssertionError
> on line whatever?
Yes, you get an `AssertionError`:
In [314]: assert True
In [315]: assert False
---
Traceback (most recent call last)
/home/bj/ in ()
:
Ciao,
Marc 'BlackJack' Rintsch
--
http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
In article <[EMAIL PROTECTED]>,
Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
> The term "global" usually means "module global" in Python.
Because they're like the objects obtained from "import"?
> [T]he functions depend on some magic data coming from "nowhere" and
> it's much harder to follow the data flow in a program. If you work
> with globals you can't be sure what the following will print:
>
> def spam():
> global x
> x = 42
> beep()
> print x
>
> `beep()` might change `x` or any function called by `beep()` and so on.
I think I get the general point, but couldn't "beep()" get at "x" even
without the "global" statement, since they're both in "spam()"?
It seems natural to me to give the most important objects in a program
persistent names: I guess this something of a 'security blanket' I need
to wean myself from. I can appreciate the benefits of
context-independence when it comes to reusing code.
> Another issue is testing. If you rely on global names it's harder to test
> individual functions. [...]
>
> In programs without such global names you see quite clearly in the
> ``def`` line what the function expects as input.
Good points, although thorough commenting can go a long way to help on
both counts. In theory, at least ...
> It's easy to "enforce" if you have minimal code on the module level. The
> usual idiom is:
>
> def main():
> # Main program comes here.
>
> if __name__ == '__main__':
> main()
>
> Then main is called when the script is called as program, but not called if
> you just import the script as module. For example to test functions or to
> reuse the code from other scripts.
I'm using "if __name__ == 'main'" now, but only for test inputs (which
will eventually be read from a config file or passed by the calling
script -- or something). I hadn't thought of putting code that actually
does something there. As for writing modules, that's way beyond where I
want to go at this point: I don't know any C and am not sure I would
want to ...
[consolidating]
In article <[EMAIL PROTECTED]>,
Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
> Then you can either pass in `found` as argument instead of creating it
> here, or you collect the passes in the calling code with the `update()`
> method of `dict`. Something like this:
>
> found = dict()
> for pass in passes:
> # ...
> found.update(extract_data(names, na, cells))
Cool. I'll have to read more about dictionary methods.
> >> assert name.startswith('Name: ')
> It checks if `name` really starts with 'Name: '. This way I turned the
> comment into code that checks the assertion in the comment.
Good idea to check, although this is actually only one of many
assumptions I make about the data -- but what happens if the assertion
fails? The program stops and the interpreter reports an AssertionError
on line whatever?
> [I]f you can make the source simpler and easier to understand by
> using the `index()` method, use a list. :-)
Understood; thanks for all the tips.
--
Odysseus
--
http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
In article <[EMAIL PROTECTED]>,
Dennis Lee Bieber <[EMAIL PROTECTED]> wrote:
> On Mon, 04 Feb 2008 09:43:04 GMT, Odysseus
> <[EMAIL PROTECTED]> declaimed the following in
> comp.lang.python:
>
> >
> > Thanks, that will be very useful. I was casting about for a replacement
> > for PostScript's "for" loop, and the "while" loop (which PS lacks -- and
> > which I've never missed there) was all I could come up with.
> >
> Have you read the language reference manual yet? It is a rather
> short document given that the language syntactic elements are not that
> complex -- but would have exposed you to the "for" statement (along with
> "return" and passing arguments).
Sorry, translation problem: I am acquainted with Python's "for" -- if
far from fluent with it, so to speak -- but the PS operator that's most
similar (traversing a compound object, element by element, without any
explicit indexing or counting) is called "forall". PS's "for" loop is
similar to BASIC's (and ISTR Fortran's):
start_value increment end_value {procedure} for
I don't know the proper generic term -- "indexed loop"? -- but at any
rate it provides a counter, unlike Python's command of the same name.
> If your only other programming experience is base PostScript you
> wouldn't really be familiar with passing arguments or returning
> values -- as an RPN stack-based language, argument passing is just
> listing the arguments before a function call (putting a copy of them
> on the stack), and returns are whatever the function left on the
> stack at the end; hence they appear sort of global.
Working directly in the operand stack is efficient, but can make
interpretation by humans -- and debugging -- very difficult. So for the
sake of coder-friendliness it's generally advisable to use variables
(i.e. assign values to keys in a dictionary) in most cases instead of
passing values 'silently' via the stack. I'm beginning to realize that
for Python the situation is just about the opposite ...
Anyway, I have been reading the documentation on the website, but much
of the terminology is unfamiliar to me. When looking things up I seem to
get an inordinate number of 404 errors from links returned by the search
function, and often the language-reference or tutorial entries (if any)
are buried several pages down. In general I'm finding the docs rather
frustrating to navigate.
> After the language reference manual, the library reference manual
> chapter on built-ins and data types would be next for study -- the rest
> can usually be handled via search functions (working with time
> conversions, look for modules with date or time ).
As I mentioned elsethread, I did look at the "time" documentation; it
was there that I found a reference to the "calendar.timegm" function I
used in my first attempt.
> It looked a bit like you were using a SAX-style parser to collect
> "names" and "cells" -- and then passing the "bunch" to another function
> to trim out and convert data... It would take me a bit to restudy the
> SAX parsing scheme (I did it once, back in the days of v1.5 or so) but
> the way I'd /try/ to do it is to have the stream handler keep track of
> which cell ( tag) is currently being parsed, and convert the string
> data at that level. You'd initialize the record dictionary to {} (and
> cell position to 0) on the tag, and return the populated record on
> the tag.
This is what my setup looks like -- mostly cribbed from _Dive Into
Python_ -- where "PageParser" is a class based on "SGMLParser":
from sgmllib import SGMLParser
from urllib import urlopen
# ...
def parse_page(url):
usock = urlopen(url)
parser = PageParser()
parser.feed(usock.read())
parser.close()
usock.close()
return parser
# ...
captured = parse_page(base_url + suffix)
I only use "parse_page" the once at this stage, but my plan was to call
it repeatedly while varying "suffix" (depending on the data found by the
previous pass). On each pass the class will initialize itself, which is
why I was collecting the data into a 'standing' (global) dictionary. Are
you suggesting essentially that I'd do better to make the text-parsing
function into a method of "PageParser"? Can one add, to such a derived
class, methods that don't have protoypes in the parent?
> Might want to check into making a class/instance of the parser so
> you can make the record dictionary and column (cell) position instance
> attributes (avoiding globals).
AFAICT my "captured" is an instance of "PageParser", but I'm unclear on
how I would add attributes to it -- and as things stand it will get
rebuilt from scratch each time a page is read in.
> > [...] I'm somewhat intimidated by the whole concept of
> > exception-handling (among others). How do you know to expect a
> > "ValueError" if the string isn't a representation of a number?
>
> Read the library reference for the function in question? Though it
> appears the reference
Re: Elementary string-parsing
On Mon, 04 Feb 2008 09:43:04 +, Odysseus wrote:
> In article <[EMAIL PROTECTED]>,
> Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
>
>> def extract_data(names, na, cells):
>> found = dict()
>
> The problem with initializing the 'super-dictionary' within this
> function is that I want to be able to add to it in further passes, with
> a new set of "names" & "cells" each time.
Then you can either pass in `found` as argument instead of creating it
here, or you collect the passes in the calling code with the `update()`
method of `dict`. Something like this:
found = dict()
for pass in passes:
# ...
found.update(extract_data(names, na, cells))
> BTW what's the difference between the above and "found = {}"?
I find it more "explicit". ``dict`` and ``list`` are easier to
distinguish than ``{}`` and ``[]`` after a lng coding session or when
printed/displayed in a small font. It's just a matter of taste.
>> for i, name in enumerate(names):
>> data = dict()
>> cells_index = 10 * i + na
>> for cell_name, index, parse in (('epoch1', 0, parse_date),
>> ('epoch2', 1, parse_date),
>> ('time', 5, parse_number),
>> ('score1', 6, parse_number),
>> ('score2', 7, parse_number)):
>> data[cell_name] = parse(cells[cells_index + index])
>
> This looks a lot more efficient than my version, but what about the
> strings that don't need parsing? Would it be better to define a
> 'pass-through' function that just returns its input, so they can be
> handled by the same loop, or to handle them separately with another loop?
I'd handle them in the same loop. A "pass-through" function for strings
already exists:
In [255]: str('hello')
Out[255]: 'hello'
>> assert name.startswith('Name: ')
>
> I looked up "assert", but all I could find relates to debugging. Not
> that I think debugging is something I can do without ;) but I don't
> understand what this line does.
It checks if `name` really starts with 'Name: '. This way I turned the
comment into code that checks the assertion in the comment.
>> The `parse_number()` function could look like this:
>>
>> def parse_number(string):
>> try:
>> return float(string.replace(',', ''))
>> except ValueError:
>> return string
>>
>> Indeed the commas can be replaced a bit more elegant. :-)
>
> Nice, but I'm somewhat intimidated by the whole concept of
> exception-handling (among others). How do you know to expect a
> "ValueError" if the string isn't a representation of a number?
Experience. I just tried what happens if I feed `float()` with a string
that is no number:
In [256]: float('abc')
---
Traceback (most recent call last)
/home/bj/ in ()
: invalid literal for float(): abc
> Is there a list of common exceptions somewhere? (Searching for
> "ValueError" turned up hundreds of passing mentions, but I couldn't find
> a definition or explanation.)
The definition is quite vague. The type of an argument is correct, but
there's something wrong with the value.
See http://docs.python.org/lib/module-exceptions.html for an overview of
the built in exceptions.
>> As already said, that ``while`` loop should be a ``for`` loop. But if
>> you put `m_abbrevs` into a `list` you can replace the loop with a
>> single call to its `index()` method: ``dlist[1] =
>> m_abbrevs.index(dlist[1]) + 1``.
>
> I had gathered that lists shouldn't be used for storing constants. Is
> that more of a suggestion than a rule?
Some suggest this. Others say tuples are for data where the position of
an element has a "meaning" and lists are for elements that all have the
same "meaning" for some definition of meaning. As an example ('John',
'Doe', 'Dr.') vs. ['Peter', 'Paul', 'Mary']. In the first example we have
name, surname, title and in the second example all elements are just
names. Unless the second example models a relation like child, father,
mother, or something like that. Anyway, if you can make the source simpler
and easier to understand by using the `index()` method, use a list. :-)
Ciao,
Marc 'BlackJack' Rintsch
--
http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
On Mon, 04 Feb 2008 12:25:24 +, Odysseus wrote: > I'm not clear on what makes an object global, other than appearing as an > operand of a "global" statement, which I don't use anywhere. But "na" is > assigned its value in the program body, not within any function: does > that make it global? Yes. The term "global" usually means "module global" in Python. > Why is this not recommended? Because the functions depend on some magic data coming from "nowhere" and it's much harder to follow the data flow in a program. If you work with globals you can't be sure what the following will print: def spam(): global x x = 42 beep() print x `beep()` might change `x` or any function called by `beep()` and so on. Another issue is testing. If you rely on global names it's harder to test individual functions. If I want to test your `extract_data()` I first have to look through the whole function body and search all the global references and bind those names to values before I can call the function. This might not be enough, any function called by `extract_data()` might need some global assignments too. This way you'll get quite soon to a point where the single parts of a program can't be tested in isolation and are not reusable for other programs. In programs without such global names you see quite clearly in the ``def`` line what the function expects as input. > If I wrap the assignment in a function, making "na" a local variable, how > can "extract_data" then access it? Give it as an argument. As a rule of thumb values should enter a function as arguments and leave it as return values. It's easy to "enforce" if you have minimal code on the module level. The usual idiom is: def main(): # Main program comes here. if __name__ == '__main__': main() Then main is called when the script is called as program, but not called if you just import the script as module. For example to test functions or to reuse the code from other scripts. >> def extract_data(names, na, cells): >> >> and >> >> return > > What should it return? A Boolean indicating success or failure? All the > data I want should all have been stored in the "found" dictionary by the > time the function finishes traversing the list of names. Then create the `found` dictionary in that function and return it at the end. Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
In article <[EMAIL PROTECTED]>,
Dennis Lee Bieber <[EMAIL PROTECTED]> wrote:
> Rather complicated description... A sample of the real/actual input
> /file/ would be useful.
Sorry, I didn't want to go on too long about the background, but I guess
more context would have helped. The data actually come from a web page;
I use a class based on SGMLParser to do the initial collection. The
items in the "names" list were originally "title" attributes of anchor
tags and are obtained with a "start_a" method, while "cells" holds the
contents of the tags, obtained by a "handle_data" method according
to the state of a flag that's set to True by a "start_td" method and to
False by an "end_td". I don't care about anything else on the page, so I
didn't define most of the tag-specific methods available.
> cellRoot = 10 * i + na #where did na come from?
> #heck, where do
> names and cells
> #come from?
> Globals? Not recommended..
The variable "na" is the number of 'not applicable' items (headings and
whatnot) preceding the data I'm interested in.
I'm not clear on what makes an object global, other than appearing as an
operand of a "global" statement, which I don't use anywhere. But "na" is
assigned its value in the program body, not within any function: does
that make it global? Why is this not recommended? If I wrap the
assignment in a function, making "na" a local variable, how can
"extract_data" then access it?
The lists of data are attributes (?) of my SGMLParser class; in my
misguided attempt to pare irrelevant details from "extract_data" I
obfuscated this aspect. I have a "parse_page(url)" function that returns
an instance of the class, as "captured", and the lists in question are
actually called "captured.names" and "captured.cells". The
"parse_page(url)" function is called in the program body; does that make
its output global as well?
> use
>
> def extract_data(names, na, cells):
>
> and
>
> return
What should it return? A Boolean indicating success or failure? All the
data I want should all have been stored in the "found" dictionary by the
time the function finishes traversing the list of names.
> > for k in ('time', 'score1', 'score2'):
> > v = found[name][k]
> > if v != "---" and v != "n/a": # skip non-numeric data
> > v = ''.join(v.split(",")) # remove commas between 000s
> > found[name][k] = float(v)
>
> I'd suggest splitting this into a short function, and invoking it in
> the preceding... say it is called "parsed"
>
> "time" : parsed(cells[cellRoot + 5]),
Will do. I guess part of my problem is that being unsure of myself I'm
reluctant to attempt too much in a single complex statement, finding it
easier to take small and simple (but inefficient) steps. I'll have to
learn to consolidate things as I go.
> Did you check the library for time/date parsing/formatting
> operations?
>
> >>> import time
> >>> aTime = "03 Feb 2008 20:35:46 UTC"#DD Mth HH:MM:SS UTC
> >>> time.strptime(aTime, "%d %b %Y %H:%M:%S %Z")
> (2008, 2, 3, 20, 35, 46, 6, 34, 0)
I looked at the documentation for the "time" module, including
"strptime", but I didn't realize the "%b" directive would match the
month abbreviations I'm dealing with. It's described as "Locale's
abbreviated month name"; if someone were to run my program on a French
system e.g., wouldn't it try to find a match among "jan", "fév", ...,
"déc" (or whatever) and fail? Is there a way to declare a "locale" that
will override the user's settings? Are the locale-specific strings
documented anywhere? Can one assume them to be identical in all
English-speaking countries, at least? Now it's pretty unlikely in this
case that such an 'international situation' will arise, but I didn't
want to burn any bridges ...
I was also somewhat put off "strptime" on reading the caveat "Note: This
function relies entirely on the underlying platform's C library for the
date parsing, and some of these libraries are buggy. There's nothing to
be done about this short of a new, portable implementation of
strptime()." If it works, however, it'll be a lot tidier than what I was
doing. I'll make a point of testing it on its own, with a variety of
inputs.
> Note that the %Z is a problematic entry...
> ValueError: time data did not match format: data=03 Feb 2008
> 20:35:46 PST fmt=%d %b %Y %H:%M:%S %Z
All the times are UTC, so fortunately this is a non-issue for my
purposes of the moment. May I assume that leaving the zone out will
cause the time to be treated as UTC?
Thanks for your help, and for bearing with my elementary questions and
my fumbling about.
--
Odysseus
--
http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
On Feb 4, 8:43 pm, Odysseus <[EMAIL PROTECTED]> wrote:
> In article <[EMAIL PROTECTED]>,
> Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
> > found = dict()
> BTW what's the difference between the above and "found = {}"?
{} takes 4 fewer keystrokes, doesn't have the overhead of a function
call, and works with Pythons at least as far back as 1.5.2 -- apart
from that, it's got absolutely nothing going for it ;-)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
On Feb 4, 3:21 am, Odysseus <[EMAIL PROTECTED]> wrote: > The next one is much messier. A couple of the strings represent times, > which I think will be most useful in 'native' form, but the input is in > the format "DD Mth HH:MM:SS UTC". time.strptime will do this! You can find the documentation at http://docs.python.org/lib/module-time.html Untested: time.strptime(my_date, '%d %b %y %H:%M:%S %Z') -- Paul Hankin -- http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
In article <[EMAIL PROTECTED]>,
Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
> Here and in later code you use a ``while`` loop although it is known at
> loop start how many times the loop body will be executed. That's a job
> for a ``for`` loop. If possible not over an integer that is used later
> just as index into list, but the list itself. Here you need both, index
> and objects from `names`. There's the `enumerate()` function for creating
> an iterable of (index, name) from `names`.
Thanks, that will be very useful. I was casting about for a replacement
for PostScript's "for" loop, and the "while" loop (which PS lacks -- and
which I've never missed there) was all I could come up with.
> I'd put all the relevant information that describes a field of the
> dictionary that is put into `found` into tuples and loop over it. There
> is the cell name, the index of the cell and function that converts the
> string from that cell into an object that is stored in the dictionary.
> This leads to (untestet):
>
> def extract_data(names, na, cells):
> found = dict()
The problem with initializing the 'super-dictionary' within this
function is that I want to be able to add to it in further passes, with
a new set of "names" & "cells" each time.
BTW what's the difference between the above and "found = {}"?
> for i, name in enumerate(names):
> data = dict()
> cells_index = 10 * i + na
> for cell_name, index, parse in (('epoch1', 0, parse_date),
> ('epoch2', 1, parse_date),
> ('time', 5, parse_number),
> ('score1', 6, parse_number),
> ('score2', 7, parse_number)):
> data[cell_name] = parse(cells[cells_index + index])
This looks a lot more efficient than my version, but what about the
strings that don't need parsing? Would it be better to define a
'pass-through' function that just returns its input, so they can be
handled by the same loop, or to handle them separately with another loop?
> assert name.startswith('Name: ')
I looked up "assert", but all I could find relates to debugging. Not
that I think debugging is something I can do without ;) but I don't
understand what this line does.
> found[name[6:]] = data
> return found
>
> The `parse_number()` function could look like this:
>
> def parse_number(string):
> try:
> return float(string.replace(',', ''))
> except ValueError:
> return string
>
> Indeed the commas can be replaced a bit more elegant. :-)
Nice, but I'm somewhat intimidated by the whole concept of
exception-handling (among others). How do you know to expect a
"ValueError" if the string isn't a representation of a number? Is there
a list of common exceptions somewhere? (Searching for "ValueError"
turned up hundreds of passing mentions, but I couldn't find a definition
or explanation.)
>
> As already said, that ``while`` loop should be a ``for`` loop. But if you
> put `m_abbrevs` into a `list` you can replace the loop with a single call
> to its `index()` method: ``dlist[1] = m_abbrevs.index(dlist[1]) + 1``.
I had gathered that lists shouldn't be used for storing constants. Is
that more of a suggestion than a rule? I take it tuples don't have an
"index()" method.
Thanks for the detailed advice. I'll post back if I have any trouble
implementing your suggestions.
--
Odysseus
--
http://mail.python.org/mailman/listinfo/python-list
Re: Elementary string-parsing
On Mon, 04 Feb 2008 03:21:18 +, Odysseus wrote:
> def extract_data():
> i = 0
> while i < len(names):
> name = names[i][6:] # strip off "Name: "
> found[name] = {'epoch1': cells[10 * i + na],
>'epoch2': cells[10 * i + na + 1],
>'time': cells[10 * i + na + 5],
>'score1': cells[10 * i + na + 6],
>'score2': cells[10 * i + na + 7]}
Here and in later code you use a ``while`` loop although it is known at
loop start how many times the loop body will be executed. That's a job
for a ``for`` loop. If possible not over an integer that is used later
just as index into list, but the list itself. Here you need both, index
and objects from `names`. There's the `enumerate()` function for creating
an iterable of (index, name) from `names`.
I'd put all the relevant information that describes a field of the
dictionary that is put into `found` into tuples and loop over it. There
is the cell name, the index of the cell and function that converts the
string from that cell into an object that is stored in the dictionary.
This leads to (untestet):
def extract_data(names, na, cells):
found = dict()
for i, name in enumerate(names):
data = dict()
cells_index = 10 * i + na
for cell_name, index, parse in (('epoch1', 0, parse_date),
('epoch2', 1, parse_date),
('time', 5, parse_number),
('score1', 6, parse_number),
('score2', 7, parse_number)):
data[cell_name] = parse(cells[cells_index + index])
assert name.startswith('Name: ')
found[name[6:]] = data
return found
The `parse_number()` function could look like this:
def parse_number(string):
try:
return float(string.replace(',', ''))
except ValueError:
return string
Indeed the commas can be replaced a bit more elegant. :-)
`parse_date()` is left as an exercise for the reader.
> for k in ('epoch1', 'epoch2'):
> dlist = found[name][k].split(" ")
> m = 0
> while m < 12:
> if m_abbrevs[m] == dlist[1]:
> dlist[1] = m + 1
> break
> m += 1
> tlist = dlist[3].split(":")
> found[name][k] = timegm((int(dlist[2]), int(dlist[1]),
> int(dlist[0]), int(tlist[0]),
> int(tlist[1]), int(tlist[2]),
> -1, -1, 0))
> i += 1
>
> The function appears to be working OK as is, but I would welcome any &
> all suggestions for improving it or making it more idiomatic.
As already said, that ``while`` loop should be a ``for`` loop. But if you
put `m_abbrevs` into a `list` you can replace the loop with a single call
to its `index()` method: ``dlist[1] = m_abbrevs.index(dlist[1]) + 1``.
Ciao,
Marc 'BlackJack' Rintsch
--
http://mail.python.org/mailman/listinfo/python-list
Elementary string-parsing
I'm writing my first 'real' program, i.e. that has a purpose aside from
serving as a learning exercise. I'm posting to solicit comments about my
efforts at translating strings from an external source into useful data,
regarding efficiency and 'pythonicity' both. My only significant
programming experience is in PostScript, and I feel that I haven't yet
'found my feet' concerning the object-oriented aspects of Python, so I'd
be especially interested to know where I may be neglecting to take
advantage of them.
My input is in the form of correlated lists of strings, which I want to
merge (while ignoring some extraneous items). I populate a dictionary
called "found" with these data, still in string form. It contains
sub-dictionaries of various items keyed to strings extracted from the
list "names"; these sub-dictionaries in turn contain the associated
items I want from "cells". After loading in the strings (I have omitted
the statements that pick up strings that require no further processing,
some of them coming from a third list), I convert selected items in
place. Here's the function I wrote:
def extract_data():
i = 0
while i < len(names):
name = names[i][6:] # strip off "Name: "
found[name] = {'epoch1': cells[10 * i + na],
'epoch2': cells[10 * i + na + 1],
'time': cells[10 * i + na + 5],
'score1': cells[10 * i + na + 6],
'score2': cells[10 * i + na + 7]}
###
Following is my first parsing step, for those data that represent real
numbers. The two obstacles I'm contending with here are that the figures
have commas grouping the digits in threes, and that sometimes the data
are non-numeric -- I'll deal with those later. Is there a more elegant
way of removing the commas than the split-and-rejoin below?
###
for k in ('time', 'score1', 'score2'):
v = found[name][k]
if v != "---" and v != "n/a": # skip non-numeric data
v = ''.join(v.split(",")) # remove commas between 000s
found[name][k] = float(v)
###
The next one is much messier. A couple of the strings represent times,
which I think will be most useful in 'native' form, but the input is in
the format "DD Mth HH:MM:SS UTC". Near the beginning of my program
I have "from calendar import timegm". Before I can feed the data to this
function, though, I have to convert the month abbreviation to a number.
I couldn't come up with anything more elegant than look-up from a list:
the relevant part of my initialization is
'''
m_abbrevs = ("Jan", "Feb", "Mar", "Apr", "May", "Jun",
"Jul", "Aug", "Sep", "Oct", "Nov", "Dec")
'''
I'm also rather unhappy with the way I kluged the seventh and eighth
values in the tuple passed to timegm, the order of the date in the week
and in the year respectively. (I would hate to have to calculate them.)
The function doesn't seem to care what values I give it for these -- as
long as I don't omit them -- so I guess they're only there for the sake
of matching the output of the inverse function. Is there a version of
timegm that takes a tuple of only six (or seven) elements, or any better
way to handle this situation?
###
for k in ('epoch1', 'epoch2'):
dlist = found[name][k].split(" ")
m = 0
while m < 12:
if m_abbrevs[m] == dlist[1]:
dlist[1] = m + 1
break
m += 1
tlist = dlist[3].split(":")
found[name][k] = timegm((int(dlist[2]), int(dlist[1]),
int(dlist[0]), int(tlist[0]),
int(tlist[1]), int(tlist[2]),
-1, -1, 0))
i += 1
The function appears to be working OK as is, but I would welcome any &
all suggestions for improving it or making it more idiomatic.
--
Odysseus
--
http://mail.python.org/mailman/listinfo/python-list
