[issue41265] lzma/bz2 module: inefficient buffer growth algorithm

Ma Lin added the comment: A more thorough solution was used, see issue41486. So I close this issue. -- stage: -> resolved status: open -> closed ___ Python tracker ___

[issue41265] lzma/bz2 module: inefficient buffer growth algorithm

Ma Lin added the comment: I'm working on a patch. lzma decompressing speed increases: baseline: 0.275722 sec patched: 0.140405 sec (Uncompressed data size 52.57 MB) The new algorithm looks like this: #define INITIAL_BUFFER_SIZE (16*1024) static inline Py_ssize_t

[issue41265] lzma/bz2 module: inefficient buffer growth algorithm

Ma Lin added the comment: Maybe the zlib module can also use the same algorithm. zlib module's initial buffer size is 16KB [1], each time the size doubles [2]. [1] zlib module's initial buffer size: https://github.com/python/cpython/blob/v3.9.0b4/Modules/zlibmodule.c#L32 [2] zlib module

[issue41265] lzma/bz2 module: inefficient buffer growth algorithm

New submission from Ma Lin : lzma/bz2 modules are using the same buffer growth algorithm: [1][2] newsize = size + (size >> 3) + 6; lzma/bz2 modules' default output buffer is 8192 bytes [3][4], so the growth step is below. For many cases, maybe the buffer is resized too many times.