-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 15.04.2009 19:16 Uhr, matthew lange wrote: > I am developing a product that extracts large amounts of data from > public databases, crafts urls, and then extracts information from those > pages and makes archetypes from that information. > > I am dealing with after about 2000 records I get into memory/swap space > issues (yes there are hardware limitations). Can anyone recommend a > good strategy for me to batch this process so that I don't get into swap > space issues? >
Use transaction savepoints after N records: import transaction transaction.savepoint(True) Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAknmFyMACgkQCJIWIbr9KYxGrgCglaVJcIX+pUWiu3s+tOaPuZ9/ 7RgAnA4mBvcuilDDIm+Ofs9qgMNi3VfN =dmTm -----END PGP SIGNATURE-----
begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[email protected] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard
_______________________________________________ Product-Developers mailing list [email protected] http://lists.plone.org/mailman/listinfo/product-developers
