[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Steve Dower]

Change by Steve Dower :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[miss-islington]

miss-islington  added the comment:


New changeset a26201cd8ef17dc81431f768846291c9f4337550 by Miss Islington (bot) 
in branch '3.7':
bpo-5438: Update memory requirements and optimize test_bigmem.py. (GH-11123)
https://github.com/python/cpython/commit/a26201cd8ef17dc81431f768846291c9f4337550


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset b13a20f50789e153c18ed8efb4fbc5eecc50f2cd by Serhiy Storchaka in 
branch 'master':
bpo-5438: Update memory requirements and optimize test_bigmem.py. (GH-11123)
https://github.com/python/cpython/commit/b13a20f50789e153c18ed8efb4fbc5eecc50f2cd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[miss-islington]

Change by miss-islington :


--
pull_requests: +10489

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +10355
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

tuple(range(size)) should consume approximately 90 GiB of memory and run for 
hours. I think this happens because it creates enormous number of integer 
objects, and this can lead to requesting additional memory for memory pools.

If details of memory management for small objects is not the purpose of this 
test, I think we can replace it with tuple(iter([42]*size)). It executes the 
same code in tuple creation: allocates a tuple of known size (both range and 
list iterators have __length_hint__) and fills it with values produced by the 
iterator. This allows significantly reduce memory requirements (to 16 GiB) and 
execution time (to 20 seconds on my computer).

The proposed PR implements this rewriting. It also fixes memory requirements 
for other tests, optimizes repr tests for tuples and lists (repr(False) is 
faster and longer than repr(0)), and expresses memory requirements in terms of 
character and pointer sizes.

--
versions: +Python 3.7, Python 3.8 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Thomas Wouters]

Thomas Wouters added the comment:

Rewriting the tests shouldn't block this specific issue, no. Also, don't use 
multiprocessing for it. I would just use subprocess to start a separate process 
(which might after all be OOM-killed,) check the exitcode, and record its 
stderr in case of failure. Given that Steve ran the bigmem tests with a high 
limit and they all passed means that it's not pressing, though -- while it 
would be *good* to make these tests easier, and the skips more obvious, it's a 
good sign that nothing broke :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Joe Jevnik]

Joe Jevnik added the comment:

I can look into rewriting the framework to use multiprocessing; however, should 
we split this into a separate issue or block this one until that work is done.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

How long time the test run (if there is enough memory)? May be we should 
decrease memory requirements by using shared values?

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Joe Jevnik]

Joe Jevnik added the comment:

I am not sure yet; I don't have access to a machine that can run the test.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Steve Dower]

Steve Dower added the comment:

I'll soon have Windows and Ubuntu machines with 448GB RAM each - anything 
special I should test here?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Joe Jevnik]

Joe Jevnik added the comment:

The tests need to be run with higher memory caps, this can be set with the -M 
flag, like python -m test -M 64G test_bigmem. I am under the impression that 
these tests don't get run all the time so we might want to run that suite.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Steve Dower]

Steve Dower added the comment:

Just for fun, I attached the memory usage graph from the Windows run. You can 
see the long pauses in the middle for the two changed tests, and the graph only 
goes up to 57GB despite the limit being set at 440GB.

--
Added file: http://bugs.python.org/file38996/bigmem_usage_440G.png

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Joe Jevnik]

Joe Jevnik added the comment:

Has the test passed or is it still hung? Also, I guess this means that the 
tuple is over-allocating, maybe I need to revise the memory assumptions to 
account for the growth algorithm.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Joe Jevnik]

Joe Jevnik added the comment:

I think that the idea is not to get as close to the limit as possible but just 
to hard cap the memory usage of the test suite so that it doesn't get 
oom-killed. twouters, does this sound correct? Also, I think that this means 
that the new decorator is reporting a proper size hint.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Steve Dower]

Steve Dower added the comment:

It's passed. There was no memory usage increase from that point, but after 
about 20 minutes it completed successfully.

All the tests passed on Ubuntu, but again, they didn't really fill up RAM - 
50GB looks like the high point.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Steve Dower]

Steve Dower added the comment:

It's running fine, but not really exercising the limit. I ran with -M 440G and 
we haven't gone past 38GB usage yet. I'll post a timeline plot when it finishes 
(on Windows - not sure how to collect the data but I'm also running on Ubuntu), 
but it looks like we may need to tweak the tests to go closer to the limit.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Steve Dower]

Steve Dower added the comment:

I'm up to test_from_2G_generator. It says the peak will be 36GB. Memory grew in 
chunks up to about 40GB and has been stuck there for 10 minutes.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Thomas Wouters]

Thomas Wouters added the comment:

Yes. The test doesn't run with more than the requested units, but if you
run it without -M it does run with a very small number, in order to make
sure the tests don't bitrot.

The mechanism could do with an overhaul though; I wrote it in 2006 or so,
before we had test skipping and when memory limits were a lot lower (and we
had fewer, less powerful friends :) I also wonder if we should change it to
run each test in a separate process so we can more easily check the actual
memory usage.
On Apr 14, 2015 1:16 PM, Joe Jevnik rep...@bugs.python.org wrote:


 Joe Jevnik added the comment:

 I think that the idea is not to get as close to the limit as possible but
 just to hard cap the memory usage of the test suite so that it doesn't get
 oom-killed. twouters, does this sound correct? Also, I think that this
 means that the new decorator is reporting a proper size hint.

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue5438
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Joe Jevnik]

Joe Jevnik added the comment:

I am now computing the size based on the system's int and pointer size.

--
keywords: +patch
nosy: +ll
Added file: http://bugs.python.org/file38945/bigmem.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Joe Jevnik]

Joe Jevnik added the comment:

I have updated the test to make another test that does not have a __len__.

--
Added file: http://bugs.python.org/file38952/bigmem-update.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Joe Jevnik]

Joe Jevnik added the comment:

Steve, we talked about the possibility of having you run some of these tests on 
a machine with a large amount of ram; I am adding you to the nosy list.

--
nosy: +steve.dower

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Mark Lawrence]

Changes by Mark Lawrence breamore...@yahoo.co.uk:


--
nosy:  -BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Nadeem Vawda]

Changes by Nadeem Vawda nadeem.va...@gmail.com:


--
nosy: +nadeem.vawda

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Thomas Wouters]

Thomas Wouters tho...@python.org added the comment:

I'm not surprised this test takes up more memory... It claims to take up 9 
times the test size in memory, with a minimum useful test size of 1G. With a 
limit of 12G that would make the test size about 1.33G. However, it then 
proceeds to build a tuple of 159 million distinct integers and stuffs them into 
a tuple. Each integer is at least 24 bytes long, plus the 8 of the pointer in 
the tuple, making the total memory use over 32 times the test size. I'll fix 
the test.

--
assignee:  - twouters
nosy: +twouters
versions: +Python 3.3 -Python 2.7, Python 3.1, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Mark Lawrence]

Mark Lawrence breamore...@yahoo.co.uk added the comment:

Is this problem repeatable?  I haven't got a big enough box on which to try 
this out.

--
nosy: +BreamoreBoy
versions: +Python 2.7, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5438] test_bigmem.test_from_2G_generator uses more memory than expected

[Antoine Pitrou]

New submission from Antoine Pitrou pit...@free.fr:

It seems test_from_2G_generator doesn't respect its announced memory
consumption. I launched test_bigmem with -M 12G on a 16GB machine, and
the interpreter process got killed by the OOM killer (Out Of Memory)
while running this test case.

--
components: Tests
messages: 83295
nosy: pitrou
priority: normal
severity: normal
status: open
title: test_bigmem.test_from_2G_generator uses more memory than expected
type: resource usage
versions: Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5438
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com