Graham Coster <graham_coster...@yahoo.co.uk> added the comment: This may be a silly question, however, does SpooledTemporaryFile need to exist at all?
>From some testing on macOS, SpooledTemporaryFile appeared to never have a >performance advantage over OS file caching, but with max_size greater than >4GB, it was a significant disadvantage. I found that the macOS built-in file cache was increasing in size as I wrote bigger TemporaryFile files, up to some limit the OS had decided. So, it seems the OS is automatically doing the same job as SpooledTemporaryFile. Once the OS decided to write to disk, there was no sudden hit to performance, it just slowed down. However, when SpooledTemporaryFile rolled-over large max_size files, there was a temporary big hit to performance, which then became a consistent slow down the same as TemporaryFile. A big issue came with very large SpooledTemporaryFile max_sizes hogging RAM and causing the OS to start swapping all processes. This caused a huge performance hit to my program and the system as a whole. Once my program did finish, it took the system considerable time to reclaim swap. I’m guessing SpooledTemporaryFile may have benefits on light weight embedded OSes that have no, or poor, file caching. However, tuning the max_size to work with embedded systems’ limited RAM could be tricky for developers and would be hardware dependent. So, perhaps leaving file caching to operating systems is actually a better, and safer, option than offering it in Python? If there are no benefits to SpooledTemporaryFile, should it be deprecated? If so, as it is phasesd out, could it be patched to be a TemporaryFile wrapper, with no rollover functionality? > On 25 Nov 2019, at 1:37 pm, Inada Naoki <rep...@bugs.python.org> wrote: > > > Inada Naoki <songofaca...@gmail.com> added the comment: > > SpooledTemporaryFile has very serious bug which causes data corruption > (#26730). Please don't use it with text mode until it is fixed. > > ---------- > > _______________________________________ > Python tracker <rep...@bugs.python.org> > <https://bugs.python.org/issue38625> > _______________________________________ ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue38625> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
