The rational is to use the results from the database cursor directly
and incrementally so you don't need to load all the results into
memory before acting on them. For instance, let's say you have a
table that has 100,000 rows in it, and you need to return all of the
results in JSON in a single call (a contrived example, but bear with
me). Let's say your handler did this:
def GET(self):
web.header('Content', 'text/json')
rows = db.select('foo').list()
return json.dumps(rows)
What would happen is this: webpy's database wrapper would query your
database for all the rows and would load them into memory, very likely
causing your program to run out of memory. On linux, it would probably
be killed by the OOM functionality in the kernel after thrashing
around your swap space for a while (I don't really know how other
systems would behave). And if the table of 100,000 rows doesn't do
that, there is some point at which this will run out of memory.
If instead we use the database cursor to read rows in one at a time
and feed them back to the client as soon as we read them in, we only
load into memory one row at a time. Consider this handler:
def GET(self):
web.header('Content', 'text/json')
rows = db.select('foo')
yield '['
for row in rows:
yield json.dumps(row) + ', '
yield ']'
(Yes, it's not technically correct JSON due to the trailing comma in
out the output, but let's pretend I handle that...) Now for each row
in the table 'foo', we read in a row, dump it to json and flush it. We
never load in the entire database table, so it uses substantially less
memory in the python process.
And that's pretty the largest issue that the database returns an
iterator. There are some other less obvious reasons -- for instance,
by reading in the results all at once you put a brief but substantial
load on the database server which probably isn't necessary since it
can likely serve database queries faster than python can process them.
Is it worth it? Frankly I don't think it should be the default setting
in webpy because of the confusion that springs up as a result. It's
confusing to new users immediately and only really makes sense for
advanced users familiar with databases and dealing with heavy load.
Hope that helps!
Justin
On Nov 6, 12:47 pm, Dragan Espenschied <d...@a-blast.org> wrote:
> > db.select returns webpy's flavored iterator,
>
> By the way, what is the background of this feature?
> I do not understand what are the advantages of these iterators over lists.
> Could somebody knowledgeable explain?
>
> I think I am missing out on something. :)
>
> Thanks,
> Dragan
--
You received this message because you are subscribed to the Google Groups
"web.py" group.
To post to this group, send email to webpy@googlegroups.com.
To unsubscribe from this group, send email to
webpy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/webpy?hl=en.
