I'd just rewrite it to use tempfile and use add an option to do pagination (the counterindidation with pagination is that it may cause data loss or duplication although very rarely).
On Jan 30, 3:03 pm, Thadeus Burgess <thade...@thadeusb.com> wrote: > Rewrite or provide an alternative to ? > > -Thadeus > > On Sat, Jan 30, 2010 at 2:51 PM, mdipierro <mdipie...@cs.depaul.edu> wrote: > > I think we should consider rewriting the export function using a > > tmpfile instead of StringIO and doing pagination. > > > On Jan 30, 2:49 pm, Thadeus Burgess <thade...@thadeusb.com> wrote: > >> Perhaps you need to write your own csv export. Save a csv file to /tmp > >> instead and then when your finished creating your csv file then stream > >> it for download. > > >> This might reduce the memory requirements? Though not sure what you > >> can do about the fact web2py cache's your select into memory, maybe do > >> it in chunks as you generate the csv file? > > >> -Thadeus > > >> On Sat, Jan 30, 2010 at 2:37 PM, mdipierro <mdipie...@cs.depaul.edu> wrote: > >> > The problem is that all records are copied into memory before being > >> > serialized + the entire csv is copied into memory. We need to find a > >> > better way when exporting a large of records. > > >> > On Jan 30, 12:19 pm, Johann Spies <johann.sp...@gmail.com> wrote: > >> >> It happened more than once that a csv-export using appadmin resulted > >> >> in incomplete data both with Postgresql-backend and sqlite. > > >> >> This one was on sqlite: > > >> >> Selected records: 139413 > >> >> csv-exported (first effort): 287 records. > >> >> csv-exported (second effort) 216 records > > >> >> This after a lot of time and slowing down the computer to a near > >> >> standstill. > > >> >> I suppose it might be easier and more reliable directly form the > >> >> database. > > >> >> Regards > >> >> Johann > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "web2py-users" group. > >> > To post to this group, send email to web...@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > web2py+unsubscr...@googlegroups.com. > >> > For more options, visit this group > >> > athttp://groups.google.com/group/web2py?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "web2py-users" group. > > To post to this group, send email to web...@googlegroups.com. > > To unsubscribe from this group, send email to > > web2py+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/web2py?hl=en. > > -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
