On Fri, Aug 12, 2011 at 11:25:06AM +0200, Stefan Behnel wrote: > przemol...@poczta.fm, 11.08.2011 16:39: >> On Thu, Aug 11, 2011 at 02:48:43PM +0100, Chris Angelico wrote: >>> On Thu, Aug 11, 2011 at 2:46 PM,<przemol...@poczta.fm> wrote: >>>> This is the way I am going to use. >>>> But what is the best data type to hold so many rows and then operate on >>>> them ? >>>> >>> >>> List of strings. Take it straight from your Oracle interface and work >>> with it directly. >> >> Can I use this list in the following way ? >> subprocess_1 - run on list between 1 and 10000 >> subprocess_2 - run on list between 10001 and 20000 >> subprocess_3 - run on list between 20001 and 30000 >> etc >> ... > > Sure. Just read the data as it comes in from the database and fill up a > chunk, then hand that on to a process. You can also distribute it in > smaller packets, just check what size gives the best throughput.
Since the performance is critical I wanted to use multiprocessing module. But when I get all the source rows into one list of strings can I easly share it between X processes ? Regards Przemyslaw Bak (przemol) ---------------------------------------------------------------- Najwieksza baza najtanszych ofert mieszkaniowych http://linkint.pl/f2a0e -- http://mail.python.org/mailman/listinfo/python-list