Hello all, I am building a workflow that basically does the following:
1) It receives a file as input 2) This file is read by a python tool that extracts a string for each line of the file. The file has 40.000 lines 3) Each string is the input for a Rest Service, (so we call to this Rest service 40.000 times) 4) The result of the calling to the previous Rest Service, is used as input for another python tool that extracts a string. 5) In the step 4 we get a list of strings, but I would like to have only one value, so I merge the list of string. After two hours running, this workflow gets an "out of memory" error. I guess this is because it has to keep in memory the result of each iteration, this result is very small, but if you multiply by 40000 it will be too much. So, I have divided my initial file in 4 files of 10000 lines, but Taverna becomes very slowly around 2500 iteration. So my questions are: Is there any limit on the number of iterations? Is there any way to split the number of iterations? I mean: if it is possible to give the 40000 lines file as input and say to Tavena "process it 2000 lines by 2000 lines" Or maybe, there is a way to keep in a file (in disk and not in memory), the results of each iteration ... ? Thanks -- Susana Sánchez Expósito Instituto de Astrofísica de Andalucía IAA (CSIC) Camino Bajo de Huétor, 50. Granada E-18008 Tel:(+34) 958 121 311 / (+34) 958 230 618 Fax:(+34) 958 814 530 e-mail: [email protected] ------------------------------------------------------------------------------ The demand for IT networking professionals continues to grow, and the demand for specialized networking skills is growing even more rapidly. Take a complimentary Learning@Cisco Self-Assessment and learn about Cisco certifications, training, and career opportunities. http://p.sf.net/sfu/cisco-dev2dev _______________________________________________ taverna-users mailing list [email protected] [email protected] Web site: http://www.taverna.org.uk Mailing lists: http://www.taverna.org.uk/about/contact-us/
