Re: [Tutor] parse text file
Hi Colin,
I'm taking the liberty of replying to your message back to the list, as
others hopefully may be able to make constructive comments. When
replying, please ensure that you reply to the tutor mailing list rather
than then individual.
On Thu, 3 Jun 2010 12:20:10 am Colin Talbert wrote:
> > Without seeing your text file, and the code you use to read the text
> > file, there's no way of telling what is going on, but I can guess
> > the most likely causes:
>
> Since the file is 9.2 gig it wouldn't make sense to send it to you.
And I am very glad you didn't try *smiles*
However, a file of that size changes things drastically. You can't
expect to necessarily be able to read the entire 9.2 gigabyte BZ2 file
into memory at once, let along the unpacked 131 GB text file, EVEN if
your computer has more than 9.2 GB of memory. So your tests need to
take this into account.
> > (2) There's a bug in your code so that you stop reading after
> > 900,000 bytes.
> The code is simple enough that I'm pretty sure there is not a
> bug in it.
>
> import bz2
> input_file =
> bz2.BZ2File(r'C:\temp\planet-latest.osm.bz2','rb') print
> len(input_file)
>
> returns 90
I'm pretty sure that this is not your code, because you can't call len()
on a bz2 file. If you try, you get an error:
>>> x = bz2.BZ2File('test.bz2', 'w') # create a temporary file
>>> x.write("some data")
>>> x.close()
>>> input_file = bz2.BZ2File('test.bz2', 'r') # open it
>>> print len(input_file)
Traceback (most recent call last):
File "", line 1, in
TypeError: object of type 'bz2.BZ2File' has no len()
So whatever your code actually is, I'm fairly sure it isn't what you say
here.
--
Steven D'Aprano
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] parse text file
Please always reply-all so a copy goes to the list. On 6/1/2010 6:49 PM, Colin Talbert wrote: Bob thanks for your response, The file is about 9.3 gig and no I don't want read the whole thing at once. I want to read it in line by line. Still it will read in to the same point (90 characters) and then act as if it came to the end of the file. Below is the code I using for this: import bz2 input_file = bz2.BZ2File(r"C:\temp\planet-latest.osm.bz2","rb") for uline in input_file: print linecount linecount+=1 Colin Talbert GIS Specialist US Geological Survey - Fort Collins Science Center 2150 Centre Ave. Bldg. C Fort Collins, CO 80526 (970) 226-9425 talbe...@usgs.gov From: bob gailer To: Colin Talbert Cc: tutor@python.org Date: 06/01/2010 04:43 PM Subject:Re: [Tutor] parse text file On 6/1/2010 5:40 PM, Colin Talbert wrote: I am also experiencing this same problem. (Also on a OSM bz2 file). It appears to be working but then partway through reading a file it simple ends. I did track down that file length is always 90 so it appears to be related to some sort of buffer constraint. Any other ideas? How big is the file? Is it necessary to read the entire thing at once? Try opening with mode rb import bz2 input_file = bz2.BZ2File(r"C:\temp\planet-latest.osm.bz2","r") try: all_data = input_file.read() print str(len(all_data)) finally: input_file.close() -- Bob Gailer 919-636-4239 Chapel Hill NC ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] OOP clarification needed
Steven D'Aprano wrote:
Case in point is this code snippet from a chapter on Tkinter.
def viewer(imgdir, kind=Toplevel, cols=None):
"""
make thumb links window for an image directory:
one thumb button per image; use kind=Tk to show
in main app window, or Frame container (pack);
imgfile differs per loop: must save with a default;
photoimage objs must be saved: erased if reclaimed;
"""
win = kind()
win.title('Viewer: ' + imgdir)
thumbs = makeThumbs(imgdir)
In the example you give, you have an argument named "kind". It is
expected to be some sort of function or class, and gets the default
value of TopLevel if not supplied. In the body of the function, this
function or class is called, to produce a value which is then
named "win". Judging by the default value and the name of the inner
variable, I would say it is expected to produce a window object, so any
function or class that returns a window object will be suitable.
I completely overlooked this expectation, which led to my confusion.
Thanks for the explanation.
Regards, Jim
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] OOP clarification needed
Alan Gauld wrote:
"Jim Byrnes" wrote
Whenever I teach myself a new language I have great difficulty
understanding the nuts and bolts of it's OO implementation.
Do you understand the OO concepts OK?
Is it only the language semantics you struggle with
or the underlying OO concepts?
I believe I understand the theory, but struggle with the actual
implementation.
some older procedural languages I always end up becoming confused by
the large number of built in methods.
C is one of the simplest procedural languages around
and yet it comes with a huge library of functions (several
hundred in some cases). The size of the library should be easier
to manage using OOP than with older function/procedure based
libraries, because the functions are not just logically grouped
in the documentation but in the code too.
I don't know C, I was thinking more along the lines of Basic or Rexx.I
could sit down and read through a list of keywords and built in
functions and it would be compact enough that I would have a good idea
of what was available. I can't seem to do that with the OO languages,
but of course I am older now also.
Case in point is this code snippet from a chapter on Tkinter.
def viewer(imgdir, kind=Toplevel, cols=None):
"""
make thumb links window for an image directory:
one thumb button per image; use kind=Tk to show
in main app window, or Frame container (pack);
imgfile differs per loop: must save with a default;
photoimage objs must be saved: erased if reclaimed;
"""
win = kind()
win.title('Viewer: ' + imgdir)
thumbs = makeThumbs(imgdir)
What is the relationship between kind=Toplevel in the first line and
win=kind() further down.
kind is a parameter ogf the function with a default value of Toplevel.
Toplevel being a class. Recall that in Python classes are objects
too and can be assigned to variables. This is similar to Smalltalk,
Lisp, Objective C and Delphi(Object Pascal) but different to C++
and Java (actually I'm not sure about Java?).
Isn't "kind" a variable and "kind()" a method?
No kind() is an invocation of a callable object.
In Python callables tend to be either functions
or classes or methods of objects.
In this case it is an instantiation of a class.
In C++ or Java it would look something like:
win = new kind();
Because classes can be treated as objects and passed to functions
this instantiates whatever kind of object was passed into viewer.
As the comment says this could be the top level window Tk or
a generic Frame container or the default Toplevel. So long as the
new object supports all the methods that will be invoked Python
doesn't care. This is polymorphism...
I had completely forgotten about the callable object. I saw the ()'s
and wrongly started to think of it as a method. Thanks for the explanation.
Regards, Jim
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] OOP clarification needed
Steve Willoughby wrote: On Tue, Jun 01, 2010 at 03:19:17PM -0500, Jim Byrnes wrote: def viewer(imgdir, kind=Toplevel, cols=None): win = kind() What is the relationship between kind=Toplevel in the first line and win=kind() further down. Isn't "kind" a variable and "kind()" a method? kind is a variable. Specifically, the second parameter passed to the viewer() function defined here. By saying kind() here, you are invoking it on the assumption that kind's value is a reference to some kind of callable object. So in theory you could pass a function as the "kind" parameter of viewer() and that function would get called, and its return value stored in "win". In this case, though, the intent is for "kind" to refer to an object class (the kind of widget this viewer is contained in or whatever). Recalling that object classes are themselves callable objects (specifically, calling them is how you construct new instances of a class), "win" will end up referring to a newly-constructed instance of whatever object class was passed as "kind". If no "kind" parameter was given, it will default to tk.Toplevel. Thanks for the explanation. I didn't understand how (or why) "kind" could change to "kind()". Sometimes I can manage to trip myself up over the silliest things. Regards, Jim ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
