przemol...@poczta.fm, 11.08.2011 16:39:
On Thu, Aug 11, 2011 at 02:48:43PM +0100, Chris Angelico wrote:
On Thu, Aug 11, 2011 at 2:46 PM,<przemol...@poczta.fm> wrote:
This is the way I am going to use.
But what is the best data type to hold so many rows and then operate on them ?
List of strings. Take it straight from your Oracle interface and work
with it directly.
Can I use this list in the following way ?
subprocess_1 - run on list between 1 and 10000
subprocess_2 - run on list between 10001 and 20000
subprocess_3 - run on list between 20001 and 30000
etc
...
Sure. Just read the data as it comes in from the database and fill up a
chunk, then hand that on to a process. You can also distribute it in
smaller packets, just check what size gives the best throughput.
Still, I'd give each work parcel a number and then reorder the results
while collecting them, that allows you to vary the chunk size and the
process size independently, without having to wait for a process that
happens to take longer.
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
