[issue11212] Python memory limit on AIX

[Mark Lawrence]

Change by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[David Edelsohn]

David Edelsohn added the comment:

Setting the environment variable LDR_CNTRL is discouraged.  It is much better 
to set the value in the executable header. Best to set it at link time, but one 
can use ldedit.

The issue with the segments in 32 bit mode is a trade off between heap memory 
and shmat attached shared memory. If one wanted to use Python with some forms 
of shared memory, one cannot allocate all of the segments to heap. Setting 
maxdata too large can cause unexpected failures if shared memory is used.

Increasing the default maxdata value seems like a good idea. The default GCC 
for AIX builds with a larger value. But it adds the command to LDFLAGS, which 
is applied everywhere, but not a significant problem and cc1* do not use shared 
memory.

One subtlety with LDFLAG is XLC wants the flag directly (-bmaxdata:0x4000) 
but GCC needs to use -Wl,-bmaxdata.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Mark Lawrence]

Mark Lawrence added the comment:

@David (Edelsohn) what is your take on this?

--
nosy: +BreamoreBoy, David.Edelsohn
versions: +Python 3.4, Python 3.5 -Python 2.6, Python 3.1, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

Sébastien Sablé sa...@users.sourceforge.net added the comment:

I updated the documentation in issue 10709 so that it mentions ldedit rather 
than the linker flags (more convenient to activate support for more memory).

Once Python 3.2 is released, we may add LDR_CNTRL or ldedit in the Makefile 
when running buildbottest.

I will ask IBM if there is any reason not to activate support for 2GB in an 
application by default (impact on performances or something).
If there is none, we may systematically call ldedit -b maxdata:0x8000 
${srcdir}/python when compiling python on AIX...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Here is a proofread diff.  I did make one substantive change: I added a comment 
about the test suite not running with the default maxdata and not raising 
memory errors with 512MB.  Is that correct?

--
nosy: +r.david.murray
Added file: http://bugs.python.org/file20776/aix-notes.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

David, I think you attached your patch to the wrong issue.

Sébastien, according to the IBM docs, the memory limit can be changed using the 
LDR_CNTRL variable, so running the test suite shouldn't need special linker 
flags, or?
(also, wouldn't it be simpler to use large pages?)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

Sébastien Sablé sa...@users.sourceforge.net added the comment:

The AIX documentation says:
http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_perftune113.htm
It is a good idea to unset the LDR_CNTRL environment variable, so that it 
does not unintentionally affect other processes.

I think people may be more interested in setting permanently the maximum memory 
that can be allocated by python rather than using an environment variable that 
may be lost and which impacts all the applications.

Another way to increase the memory available for an application is to use the 
ldedit command on the python executable:

ldedit -b maxdata:0x2000 ./python

That may be easier than passing it directly to the linker.

Concerning large pages, I found that in an IBM documentation:

Users are advised to be cautious in their use of the environment
variable to specify large page usage. Performance tests have shown there
can be a significant performance loss in environments where a number of
shell scripts or small, short running applications are invoked. One
example saw a shell script's execution time increase over 10 times when
the large page environment variable was specified. Customers are advised
to only set the large page environment variable around specific
applications which can benefit from large page usage.

Also large pages must be activated on the system by root. This does not seem to 
be available by default according to the doc.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

Sébastien Sablé sa...@users.sourceforge.net added the comment:

The doc would become:


To allocate more segment registers to Python, you must use the linker option 
-bmaxdata or ldedit to specify the number of bytes you need in the data
 segment.
 
For example, if you want to allow 512MB of memory for Python (this
is enough for the test suite to run without MemroyErrors), you should
run the following command at the end of compilation:

 ldedit -b maxdata:0x2000 ./python


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

oops.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


Removed file: http://bugs.python.org/file20776/aix-notes.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 I think people may be more interested in setting permanently the
 maximum memory that can be allocated by python rather than using an
 environment variable that may be lost and which impacts all the
 applications.

Agreed, but it's impossible to standardize the configure script on a
good enough value. Some people might need 512MB, others 2GB...

 Another way to increase the memory available for an application is to
 use the ldedit command on the python executable:
 
 ldedit -b maxdata:0x2000 ./python
 
 That may be easier than passing it directly to the linker.

Sounds better. That means that it can be decoupled from the build
process.
(although for a hypothetical buildbot, I think passing LDR_CNTRL would
be the most practical solution)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

Sébastien Sablé sa...@users.sourceforge.net added the comment:

This link flag should only be applied to the python executable.
There is already LINKFORSHARED which is great for that.
LDFLAGS will apply it to all the shared libraries which is not really what we 
want.

I can compile Python with support for 512MB without any modification to 
configure like that:

CC=xlc_r OPT=-O2 -qmaxmem=10 ./configure --without-computed-gotos 
--enable-shared
make LINKFORSHARED=-Wl,-bE:Modules/python.exp -lld -Wl,-bmaxdata:0x2000

It is just not very elegant and will make the builbdot script even more complex 
than it is currently.
Also it will make it more difficult for AIX users to find out about this 
limitation or build python with support for more memory (though I agree this 
represents very few people).

If you don't want it in configure though, I will add a note in AIX-NOTES about 
this limitation and how to build Python with support for more memory.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

Changes by Sébastien Sablé sa...@users.sourceforge.net:


--
versions: +Python 2.6, Python 2.7, Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 LDFLAGS will apply it to all the shared libraries which is not really
 what we want.

Is there a downside?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

Sébastien Sablé sa...@users.sourceforge.net added the comment:

I don't know distutils well enough, but I am afraid the LDFLAGS flag used to 
compile python is reused in some way when linking some python extensions or 
some application that embed python.
(I use an application that embed python on AIX, I should check what happens 
when I compile it).

Except for that, I can't see any problem now, but I would not be feel 
comfortable propagating this option in LDFLAGS.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

New submission from Sébastien Sablé sa...@users.sourceforge.net:

On AIX, executables are compiled by default so that they cannot allocate more 
than 256MB of memory.

This is not enough in some cases; for example this is not enough to get the 
Python test suite to run completely.

As explained in IBM documentation:

By default each program gets one segment register (see 2.24) for its
data segment. As each segment register covers 256 MB, any calls to
malloc more will fail. Also programs that declare large global or static
arrays may fail to load. To allocate more segment registers to your
program, use the linker option -bmaxdata to specify the number of bytes
you need in the data segment as follows:

cc -o myprog -bmaxdata:0x2000 myprog.c

The example above would allocate an additional segment register to allow
for 512MB of data.

export LDR_CNTRL=MAXDATA=0x2000
start_process
unset LDR_CNTRL

Marking An Executable For Large Page Use
The XCOFF header in an executable file contains a new flag to indicate that the 
program wants to use
large pages to back its data and heap segments. This flag can be set when the 
application is linked by
specifying the -blpdata option on the ld command. The flag can also be set or 
cleared using the ldedit
command. The ldedit -blpdata filename command sets the large page data/heap 
flag in the specified
file. The ldedit -bnolpdata filename clears the large page flag. The ldedit 
command may also be used
to set an executable's maxdata value

--
components: Build
messages: 128553
nosy: sable
priority: normal
severity: normal
status: open
title: Python memory limit on AIX
versions: Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

Sébastien Sablé sa...@users.sourceforge.net added the comment:

Here is a patch that would provide an --enable-aix-maxdata option to configure, 
in order to support 512MB of mermory.

I have not tested it yet.

--
keywords: +patch
Added file: http://bugs.python.org/file20761/patch_maxdata.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

I don't see why we would provide an option to replace a magic number with 
another magic number. I find it strange that AIX doesn't simply enable 
unbounded virtual address spaces.
(by unbounded I mean bounded by CPU limitations only)

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Sébastien Sablé]

Sébastien Sablé sa...@users.sourceforge.net added the comment:

Yes, welcome into the strange world of AIX...

You can allow up to 8 segments, which represent 2GB:
http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_perftune113.htm

We could change the option to become:
--with-aix-maxdata=value
where value could be between 0 and 8, with a default at 0 (nothing passed to 
the compiler).

Would that be OK with you?

The good news is that thanks to this patch I was able to play the full python 
test suite on AIX:

313 tests OK.
8 tests failed:
test_asyncore test_cmath test_distutils test_fileio test_locale
test_resource test_time test_wait4
16 tests skipped:
test_ctypes test_curses test_dbm_gnu test_epoll test_gdb
test_kqueue test_largefile test_ossaudiodev test_pep277
test_startfile test_tk test_ttk_guionly test_unicode_file
test_winreg test_winsound test_zipfile64

Not too bad.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11212] Python memory limit on AIX

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 We could change the option to become:
 --with-aix-maxdata=value
 where value could be between 0 and 8, with a default at 0 (nothing
 passed to the compiler).

Doesn't LDFLAGS work for that? I don't think we want a lot of
platform-specific options like that.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11212
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com