Just for the record, my 160x speed improvement came with pretty naive code.
I loop to a record, build the text I need for that row, append the block to
the BLOB, clear the text and hit the next row. All-in-all, only a couple of
lines of extra code.

After what everyone has contributed and suggested, along with my own local
testing, my summary goes like this:

*Build a text var*
Yeah, don't do that. In my case, the final block size is ~3MB, not
huge...but the resizing is pretty aggressive.

*Build a BLOB var*
Works great. The downside is that if you ultimately need text, you (even if
briefly) need to double the amount of RAM you're consuming. I'm defaulting
to a max BLOB of 5MB and change, so this should never be a problem.

*Write to a text file*
This should never slow down, no matter how big the file gets. This has
always been true, the limiting factor being the speed of the drive. Drives
are fast now. Memory is also fine because you take no RAM to store the
document and can load it into RAM in one go very easily.

I'm going with a BLOB instead of documents. Why? Old prejudice, and I like
avoiding the file system. Permissions, file contention issues in 4D (been
there not long ago...and after *more than a year*, there's reportedly a
fix), data leakage. Plus, old prejudice. In my specific case, a BLOB should
be fine as I'm not pushing the envelope on memory usage. Under other
circumstances, I can easily see documents as being the best bet.
Particularly if you have other reasons to want the data in documents for
dispatching, archiving, etc.

Thanks for the help!
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**********************************************************************

Reply via email to