date:20110221

Re: python and parsing an xml file

2011-02-21 Thread Stefan Behnel


Matt Funk, 21.02.2011 23:40:

On 2/21/2011 3:28 PM, Stefan Behnel wrote:

Matt Funk, 21.02.2011 23:07:

thank you for your advice.
I am running into an issue though (which is likely a newbie problem):

My xml file looks like (which i got from the internet):


 
Gambardella, Matthew
XML Developer's Guide
Computer
44.95
2000-10-01
An in-depth look at creating applications
with XML.
 


Then i try to access it as:

from lxml import etree
from lxml import objectify

inputfile="../input/books.xml"
parser = etree.XMLParser(ns_clean=True)
parser = etree.XMLParser(remove_comments=True)


Change that to

parser = objectify.makeparser(ns_clean=True, remove_comments=True)


root = objectify.parse(inputfile,parser)


Change that to

root = objectify.parse(inputfile,parser).getroot()


According to your instructions i do:

parser = objectify.makeparser(ns_clean=True, remove_comments=True)
root = objectify.parse(inputfile,parser).getroot()
print root.catalog.book.author.text


root == 

Try "print root.tag", and read the tutorial.

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Tutorial on Multithreading

2011-02-21 Thread Terry Reedy


On 2/21/2011 7:02 PM, KevinSimonson wrote:

On Feb 21, 4:04 pm, Alexander Kapps  wrote:


That tutorial seems to be wrong.

According to the official docs:

"If the subclass overrides the constructor, it must make sure to
invoke the base class constructor (Thread.__init__()) before doing
anything else to the thread."

http://docs.python.org/library/threading.html#thread-objects

So, change your __init__ to this:

class myThread (threading.Thread):
  def __init__(self, threadID, name, q):
  threading.Thread.__init__(self)
  self.threadID = threadID
  self.name = name
  self.q = q



Alexander, thanks!  Your suggestion fixed my problem.


Please report the problem and solution on the tracker so the tutorial 
can be fixed.


--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: Return Values & lambda

2011-02-21 Thread pradeepbpin

On Feb 22, 5:59 am, Steven D'Aprano  wrote:
> On Mon, 21 Feb 2011 16:43:49 -0800, Rafe Kettler wrote:
> > On Feb 21, 1:59 pm, pradeepbpin  wrote:
> >> I have a main program module that invokes an input dialog box via  a
> >> menu item. Now, the code for drawing and processing the input of dialog
> >> box is in another module, say 'dialogs.py'. I connect the menu item to
> >> this dialog box by a statement like,
>
> >> manu_item.connect('activate', lambda a: dialogs.open_dilaog())
>
> >> If this function open_dialog() returns a list of dialog inputs values,
> >> how can I access those values in the main module ?
>
> > Moreover, I don't see why you need a lambda in this case. Why not just
> > pass the function itself?
>
> My guess is that the callback function is passed a single argument, and
> open_dialog doesn't take any arguments, hence the wrapper which just
> ignores the argument and calls the function.
>
> --
> Steven


Thanks to all.

Pradeep
-- 
http://mail.python.org/mailman/listinfo/python-list

Ruby on rails training for Python developers

2011-02-21 Thread Wan Li Zhu

If you're a Python developer in the Boston area looking to learn Ruby,
Fairhaven Capital and thoughtbot are teaming up to offer Ruby on Rails
training courses in Boston at 50% off regular price ($600 for 2 full
days of training, intro and advanced levels).

Details at http://workshops.thoughtbot.com/fairhaven
-- 
http://mail.python.org/mailman/listinfo/python-list

Python 2 or 3 ? with Django , My SQL and YUI

2011-02-21 Thread Sumit

Python 2 or 3 ? with Django , My SQL and YUI

For a web project We have decided to work on Python 2 or 3 ? with
Django , My SQL and YUI, and this would be the first time to work with
Python, just now I explored a little and found Python -2 vs 3 Stuff ,

Is there experienced python dev can guide me in very simple terms
should I go for Python 2 or 3 ?

How about Python with Django and MySQL and YUI Stuff ?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Long Lists

2011-02-21 Thread Dan Stromberg

On Mon, Feb 21, 2011 at 7:24 PM, Dan Stromberg  wrote:

>
> On Mon, Feb 21, 2011 at 6:57 PM, Kelson Zawack <
> zawack...@gis.a-star.edu.sg> wrote:
>
>> I have a large (10gb) data file for which I want to parse each line into
>> an object and then append this object to a list for sorting and further
>> processing.  I have noticed however that as the length of the list increases
>> the rate at which objects are added to it decreases dramatically.  My first
>> thought was that  I was nearing the memory capacity of the machine and the
>> decrease in performance was due to the os swapping things in and out of
>> memory.  When I looked at the memory usage this was not the case.  My
>> process was the only job running and was consuming 40gb of the the total
>> 130gb and no swapping processes were running.  To make sure there was not
>> some problem with the rest of my code, or the servers file system, I ran my
>> program again as it was but without the line that was appending items to the
>> list and it completed without problem indicating that the decrease in
>> performance is the result of some part of the process of appending to the
>> list.  Since other people have observed this problem as well (
>> http://tek-tips.com/viewthread.cfm?qid=1096178&page=13,
>> http://stackoverflow.com/questions/2473783/is-there-a-way-to-circumvent-python-list-append-becoming-progressively-slower-i)
>> I did not bother to further analyze or benchmark it.  Since the answers in
>> the above forums do not seem very definitive  I thought  I would inquire
>> here about what the reason for this decrease in performance is, and if there
>> is a way, or another data structure, that would avoid this 
>> problem.
>
>
> Do you have 130G of physical RAM, or 130G of virtual memory?  That makes a
> big difference.  (Yeah, I know, 130G of physical RAM is probably pretty rare
> today)
>
> Disabling garbage collection is a good idea, but if you don't have well
> over 10G of physical RAM, you'd probably better also use a (partially)
> disk-based sort.  To do otherwise would pretty much beg for swapping and a
> large slowdown.
>
> Merge sort works very well for very large datasets.
> http://en.wikipedia.org/wiki/Merge_sort  Just make your sublists be disk
> files, not in-memory lists - until you get down to a small enough sublist
> that you can sort it in memory, without thrashing.  Timsort (list_.sort())
> is excellent for in memory sorting.
>
> Actually, GNU sort is very good at sorting huge datasets - you could
> probably just open a subprocess to it, as long as you can make your data fit
> the line-oriented model GNU sort expects, and you have enough temporary disk
> space.
>

Depending on what you're doing after the sort, you might also look at
bsddb.btopen
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Return Values & lambda

2011-02-21 Thread Rafe Kettler

On Feb 21, 7:59 pm, Steven D'Aprano  wrote:
> On Mon, 21 Feb 2011 16:43:49 -0800, Rafe Kettler wrote:
> > On Feb 21, 1:59 pm, pradeepbpin  wrote:
> >> I have a main program module that invokes an input dialog box via  a
> >> menu item. Now, the code for drawing and processing the input of dialog
> >> box is in another module, say 'dialogs.py'. I connect the menu item to
> >> this dialog box by a statement like,
>
> >> manu_item.connect('activate', lambda a: dialogs.open_dilaog())
>
> >> If this function open_dialog() returns a list of dialog inputs values,
> >> how can I access those values in the main module ?
>
> > Moreover, I don't see why you need a lambda in this case. Why not just
> > pass the function itself?
>
> My guess is that the callback function is passed a single argument, and
> open_dialog doesn't take any arguments, hence the wrapper which just
> ignores the argument and calls the function.
>
> --
> Steven

That would sound reasonable.

Rafe
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python on OSX

2011-02-21 Thread Jason Swails

On Mon, Feb 21, 2011 at 7:08 PM, Chris Rebert  wrote:

> On Mon, Feb 21, 2011 at 3:54 PM, Robert  wrote:
> > Can I install 2.7 and 3.2 side by side?
>
> Yes, of course. Just don't fiddle with the "System Python" (i.e. the
> copy preinstalled by Apple).
>

Good advice.  I second it.

> You may wish to install your additional Pythons via Fink or MacPorts.
>

I have python24, 25, 26, 27, and 32 installed with MacPorts, and
"python_select" (also from MacPorts) allows you to change the default python
version quickly and cleanly with a single command.  It's quite useful for
testing compatibility of scripts that use "#!/usr/bin/env python" as the
shebang line.  I haven't tried Fink, but the MacPorts approach works well
for me.

Good luck!
Jason

>
> Cheers,
> Chris
> --
> http://mail.python.org/mailman/listinfo/python-list
>

-- 
Jason M. Swails
Quantum Theory Project,
University of Florida
Ph.D. Candidate
352-392-4032
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Long Lists

2011-02-21 Thread Ben Finney

Kelson Zawack  writes:

> I have a large (10gb) data file for which I want to parse each line
> into an object and then append this object to a list for sorting and
> further processing.

What is the nature of the further processing?

Does that further processing access the items sequentially? If so, they
don't all need to be in memory at once, and you can produce them with a
generator http://docs.python.org/glossary.html#term-generator>.

Note that, if you just want lines of text from a file, the file object
itself is a generator for the lines of text within it.

If, on the other hand, you need arbitrary access all over that large
data set, you probably want a data type better suited. The standard
library has the ‘array’ module for this purpose; the third-party NumPy
library provides even more power.

-- 
 \   “Remember: every member of your ‘target audience’ also owns a |
  `\   broadcasting station. These ‘targets’ can shoot back.” —Michael |
_o__)   Rathbun to advertisers, news.admin.net-abuse.email |
Ben Finney
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Long Lists

2011-02-21 Thread Dan Stromberg

On Mon, Feb 21, 2011 at 6:57 PM, Kelson Zawack
wrote:

> I have a large (10gb) data file for which I want to parse each line into an
> object and then append this object to a list for sorting and further
> processing.  I have noticed however that as the length of the list increases
> the rate at which objects are added to it decreases dramatically.  My first
> thought was that  I was nearing the memory capacity of the machine and the
> decrease in performance was due to the os swapping things in and out of
> memory.  When I looked at the memory usage this was not the case.  My
> process was the only job running and was consuming 40gb of the the total
> 130gb and no swapping processes were running.  To make sure there was not
> some problem with the rest of my code, or the servers file system, I ran my
> program again as it was but without the line that was appending items to the
> list and it completed without problem indicating that the decrease in
> performance is the result of some part of the process of appending to the
> list.  Since other people have observed this problem as well (
> http://tek-tips.com/viewthread.cfm?qid=1096178&page=13,
> http://stackoverflow.com/questions/2473783/is-there-a-way-to-circumvent-python-list-append-becoming-progressively-slower-i)
> I did not bother to further analyze or benchmark it.  Since the answers in
> the above forums do not seem very definitive  I thought  I would inquire
> here about what the reason for this decrease in performance is, and if there
> is a way, or another data structure, that would avoid this 
> problem.


Do you have 130G of physical RAM, or 130G of virtual memory?  That makes a
big difference.  (Yeah, I know, 130G of physical RAM is probably pretty rare
today)

Disabling garbage collection is a good idea, but if you don't have well over
10G of physical RAM, you'd probably better also use a (partially) disk-based
sort.  To do otherwise would pretty much beg for swapping and a large
slowdown.

Merge sort works very well for very large datasets.
http://en.wikipedia.org/wiki/Merge_sort  Just make your sublists be disk
files, not in-memory lists - until you get down to a small enough sublist
that you can sort it in memory, without thrashing.  Timsort (list_.sort())
is excellent for in memory sorting.

Actually, GNU sort is very good at sorting huge datasets - you could
probably just open a subprocess to it, as long as you can make your data fit
the line-oriented model GNU sort expects, and you have enough temporary disk
space.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Long Lists

2011-02-21 Thread John Bokma

alex23  writes:

> On Feb 22, 12:57 pm, Kelson Zawack 
> wrote:
>> I did not bother to further analyze or benchmark it.  Since the answers
>> in the above forums do not seem very definitive  I thought  I would
>> inquire here about what the reason for this decrease in performance is,
>> and if there is a way, or another data structure, that would avoid this
>> problem.
>
> The first link is 6 years old and refers to Python 2.4. Unless you're
> using 2.4 you should probably ignore it.
>
> The first answer on the stackoverflow link was accepted by the poster
> as resolving his issue. Try disabling garbage collection.

I just read http://bugs.python.org/issue4074 which discusses a patch
that has been included 2 years ago. So using a recent Python 2.x also
doesn't have this issue?

-- 
John Bokma   j3b

Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma
Freelance Perl & Python Development: http://castleamber.com/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Long Lists

2011-02-21 Thread alex23

On Feb 22, 12:57 pm, Kelson Zawack 
wrote:
> I did not bother to further analyze or benchmark it.  Since the answers
> in the above forums do not seem very definitive  I thought  I would
> inquire here about what the reason for this decrease in performance is,
> and if there is a way, or another data structure, that would avoid this
> problem.

The first link is 6 years old and refers to Python 2.4. Unless you're
using 2.4 you should probably ignore it.

The first answer on the stackoverflow link was accepted by the poster
as resolving his issue. Try disabling garbage collection.
-- 
http://mail.python.org/mailman/listinfo/python-list

I have a large (10gb) data file for which I want to parse each line into
an object and then append this object to a list for sorting and further
processing. I have noticed however that as the length of the list
increases the rate at which objects are added to it decreases
dramatically. My first thought was that I was nearing the memory
capacity of the machine and the decrease in performance was due to the
os swapping things in and out of memory. When I looked at the memory
usage this was not the case. My process was the only job running and
was consuming 40gb of the the total 130gb and no swapping processes were
running. To make sure there was not some problem with the rest of my
code, or the servers file system, I ran my program again as it was but
without the line that was appending items to the list and it completed
without problem indicating that the decrease in performance is the
result of some part of the process of appending to the list. Since
other people have observed this problem as well
(http://tek-tips.com/viewthread.cfm?qid=1096178&page=13,
http://stackoverflow.com/questions/2473783/is-there-a-way-to-circumvent-python-list-append-becoming-progressively-slower-i)
I did not bother to further analyze or benchmark it. Since the answers
in the above forums do not seem very definitive I thought I would
inquire here about what the reason for this decrease in performance is,
and if there is a way, or another data structure, that would avoid this
problem.

77 matches

Mail list logo