Chris Angelico, 11.08.2011 12:59:
On Thu, Aug 11, 2011 at 7:40 AM,<przemol...@poczta.fm> wrote:
I am not a database developer so I don't want to change the whole process
of data flow between applications in my company. Another process is
reading this XML from particular Oracle table so I have to put the final XML
there.
I think you may be looking at a submission to
http://www.thedailywtf.com/ soon. You seem to be working in a rather
weird dataflow. :( Under the circumstances, you're probably going to
want to go with the original ''.join() option.
This server has 256 GB of RAM so memory is not a problem.
Also the select which fetches the data is sorted. That is why I have to
carefully divide into subtasks and then merge it in correct order.
There's no guarantee that all of that 256GB is available to you, of course.
What may be the easiest way is to do the select in a single process,
then partition it and use the Python multiprocessing module to split
the job into several parts. Then you need only concatenate the handful
of strings.
Or join them using an n-way merge.
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
