[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Mark Lawrence]

Change by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Berker Peksag]

Berker Peksag added the comment:

Can this be closed now?

--
nosy: +berker.peksag

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

No, I'm working on more advanced patch.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Mark Lawrence]

Mark Lawrence added the comment:

Oh, Berker, please stop shaking up bug tracker.

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Some buildbots crash:

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/11152/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20Debian%20PGO%203.x/builds/4/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1715/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20Solaris%2011%20%5BSB%5D%203.x/builds/3836/steps/test/logs/stdio

E.g.:

Fatal Python error: Objects/frameobject.c:429 object at 0x406dd878 has negative 
ref count -5

Current thread 0x40525ce0 (most recent call first):
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py, 
line 605 in run
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py, 
line 625 in __call__
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py, 
line 122 in run
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py, 
line 84 in __call__
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py, 
line 122 in run
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py, 
line 84 in __call__
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py, 
line 122 in run
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py, 
line 84 in __call__
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py, 
line 168 in run
  File 
/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py, 
line 1770 in _run_suite
  File 
/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py, 
line 1804 in run_unittest
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/regrtest.py, 
line 1283 in test_runner
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/regrtest.py, 
line 1284 in runtest_inner
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/regrtest.py, 
line 967 in runtest
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/regrtest.py, 
line 763 in main
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/regrtest.py, 
line 1568 in main_in_temp_cwd
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/__main__.py, 
line 3 in module
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py, line 85 
in _run_code
  File /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py, line 170 
in _run_module_as_main
Aborted
make: *** [buildbottest] Error 134

Trying to reproduce locally and investigate.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Roundup Robot]

Roundup Robot added the comment:

New changeset ea33b61cac74 by Serhiy Storchaka in branch 'default':
Issue #15381: Fixed a bug in BytesIO.write().
https://hg.python.org/cpython/rev/ea33b61cac74

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Roundup Robot]

Roundup Robot added the comment:

New changeset 3fdbdf4d2ac7 by Serhiy Storchaka in branch 'default':
Issue #15381: Try to fix refcount bug. Empty and 1-byte buffers are always 
shared.
https://hg.python.org/cpython/rev/3fdbdf4d2ac7

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2e29d54843a4 by Serhiy Storchaka in branch 'default':
Issue #15381: Optimized io.BytesIO to make less allocations and copyings.
https://hg.python.org/cpython/rev/2e29d54843a4

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Updated patch addresses Antoine's comment.

--
Added file: http://bugs.python.org/file37939/bytesio_resized_bytes-6.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ping.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Stefan Krah]

Changes by Stefan Krah stefan-use...@bytereef.org:


--
nosy:  -skrah

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is a patch which optimizes readline() and readlines() methods of BytesIO 
and the __next__() method of BytesIO iterator.

--
Added file: http://bugs.python.org/file36369/bytesio_faster_readline.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Antoine Pitrou]

Antoine Pitrou added the comment:

I suppose this takes advantage of the libc's optimized memchr(). Any benchmarks?
(patch looks fine, by the way)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Benchmark from issue22003:

$ ./python0 -m timeit -s 'import i' 'i.readlines()'

Before patch: 10 loops, best of 3: 36.1 msec per loop
After patch: 10 loops, best of 3: 27.5 msec per loop

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Roundup Robot]

Roundup Robot added the comment:

New changeset 601045ceff94 by Serhiy Storchaka in branch 'default':
Issue #15381: Optimized line reading in io.BytesIO.
http://hg.python.org/cpython/rev/601045ceff94

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Updated patch without just committed line reading optimization and committed in 
issue22193 the _PySys_GetSizeOf() function.

--
Added file: http://bugs.python.org/file36371/bytesio_resized_bytes-5.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Antoine Pitrou]

Antoine Pitrou added the comment:

Can you post any benchmark?
Also, if you wouldn't change the resizing factor it would make it easier to 
compare the two approaches on their own merits, IMO.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I posted benchmarks two years ago, in msg165795. Here are updated results:

$ ./python -m timeit -s import io; n=100; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1  
s=io.BytesIO(); w=s.write  for x in d: w(x)  s.getvalue()

Before patch: 10 loops, best of 3: 42.3 msec per loop
After patch: 10 loops, best of 3: 27.6 msec per loop

$ ./python -m timeit -s import io; n=1000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000  
s=io.BytesIO(); w=s.write  for x in d: w(x)  s.getvalue()

Before patch: 10 loops, best of 3: 28.7 msec per loop
After patch: 100 loops, best of 3: 14.8 msec per loop

They don't depend from the resizing factor on Linux. I increased it in hope it 
will help on Windows.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

What should be made to move this issue forward?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[David Wilson]

David Wilson added the comment:

Hey Serhiy,

The implementation for your readline optimization seems less contentious (and 
less risky) than the remainder of the patch -- it could perhaps be easily split 
off into a separate patch, which may be far more easily committed.

I love the concept of this patch, although from my last reading (weeks ago), 
it's slightly scary that it relies on Py_REFCNT() to know whether to mutate a 
string or not. In principle this should never break, in practice, however, it 
is uncertain that there are no strange edge cases that aren't immediately 
obvious.

The _PyBytes_Resize doc is quite clear: Only use this to build up a brand new 
bytes object; don’t use this if the bytes may already be known in other parts 
of the code

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Nick Coghlan]

Nick Coghlan added the comment:

It wouldn't be the first time we've used Py_REFCNT that way. We do it with 
tuples and normal strings as well.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
nosy: +benjamin.peterson, dw, hynek, kmike, python-dev, scoder, skrah, stutzbach

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Synchronized with tip. Unfortunately there are no common points with recent 
issue22003 changes so they should be reverted (except tests).

--
versions: +Python 3.5 -Python 3.4
Added file: http://bugs.python.org/file36180/bytesio_resized_bytes-4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Only one week left before feature freeze. Please benchmark this patch on 
Windows.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[STINNER Victor]

STINNER Victor added the comment:

 Only one week left before feature freeze.

This issue is an optimization, not a new feature.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Updated patch uses large overallocation factor (1/2 instead 1/8), it may 
increase the speed on Windows. Fixed implementation of __sizeof__() and some 
minor bugs.

--
stage: needs patch - patch review
Added file: http://bugs.python.org/file32536/bytesio_resized_bytes-3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

[Eli Bendersky]

Changes by Eli Bendersky eli...@gmail.com:


--
title: Optimize BytesIO to so less reallocations when written, similarly to 
StringIO - Optimize BytesIO to do  less reallocations when written, similarly 
to StringIO

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com