[issue38923] Spurious OSError "Not enough memory resources" when allocating memory using multiprocessing.RawArray

[Brian Kardon]

Brian Kardon  added the comment:

Ah, thank you so much! That makes sense. I'll have to switch to 64-bit python. 

I've marked this as closed - hope that's the right thing to do here.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38923] Spurious OSError "Not enough memory resources" when allocating memory using multiprocessing.RawArray

[Christian Heimes]

Christian Heimes  added the comment:

Good catch, Zach! A 32bit version of Python would certainly explain your 
problem. A 32bit process can only address 4 GB of address space. Almost half of 
the address space is reserved for OS, stack, Python itself, and other things. 
That leaves a little more than 2 GB of free heap memory for user data. Due to 
memory fragmentation it is possible that the allocator can't find a continuous 
free block 1 GB.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38923] Spurious OSError "Not enough memory resources" when allocating memory using multiprocessing.RawArray

[Zachary Ware]

Zachary Ware  added the comment:

You appear to be using a 32-bit version of Python on Windows; do you get the 
same behavior using the 64-bit version?

--
components:  -ctypes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38923] Spurious OSError "Not enough memory resources" when allocating memory using multiprocessing.RawArray

[Brian Kardon]

New submission from Brian Kardon :

When I try to create a series of multiprocessing.RawArray objects, I get an 
"OSError: Not enough memory resources are available to process this command". 
However, by my calculations, the total amount of memory I'm trying to allocate 
is just about 1 GB, and my system reports that it has around 20 GB free memory. 
I can't find any information about any artificial memory limit, and my system 
seems to have plenty of memory for this, so it seems like a bug to me. This is 
my first issue report, so I apologize if I'm doing this wrong.

The attached script produces the following output on my Windows 10 64-bit 
machine with Python 3.7:

Creating buffer # 0
Creating buffer # 1
Creating buffer # 2
Creating buffer # 3
Creating buffer # 4
...etc...
Creating buffer # 276
Creating buffer # 277
Creating buffer # 278
Creating buffer # 279
Creating buffer # 280
Traceback (most recent call last):
  File ".\Cruft\memoryErrorTest.py", line 10, in 
buffers.append(mp.RawArray(imageDataType, imageDataSize))
  File "C:\Users\Brian 
Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py",
 line 129, in RawArray
return RawArray(typecode_or_type, size_or_initializer)
  File "C:\Users\Brian 
Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\sharedctypes.py",
 line 61, in RawArray
obj = _new_value(type_)
  File "C:\Users\Brian 
Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\sharedctypes.py",
 line 41, in _new_value
wrapper = heap.BufferWrapper(size)
  File "C:\Users\Brian 
Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py", 
line 263, in __init__
block = BufferWrapper._heap.malloc(size)
  File "C:\Users\Brian 
Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py", 
line 242, in malloc
(arena, start, stop) = self._malloc(size)
  File "C:\Users\Brian 
Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py", 
line 134, in _malloc
arena = Arena(length)
  File "C:\Users\Brian 
Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py", 
line 38, in __init__
buf = mmap.mmap(-1, size, tagname=name)
OSError: [WinError 8] Not enough memory resources are available to process this 
command

--
components: Library (Lib), Windows, ctypes
files: memoryErrorTest.py
messages: 357531
nosy: bkardon, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Spurious OSError "Not enough memory resources" when allocating memory 
using multiprocessing.RawArray
type: behavior
versions: Python 3.7
Added file: https://bugs.python.org/file48744/memoryErrorTest.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com