On 8/15/2015 3:21 AM, dieter wrote:
Ping Liu <yanzhiping...@gmail.com> writes:
...
For small cases, Python works well. But if we consider longer time period.
then it would fail due to the memory usage issues. We have tested several
case studies to check the memory use for different time period, including
1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
are feasible while the last case gives out the memory error.
When we are testing the forth case, the memory error comes out while
creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.
The solver is CPLEX (academic). It turns out that the solver is taking a
lot of memory as you can see in the memory test report. for the first three
cases, different memory usage is observed, and it grows up dramatically
with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
19.5 MB; 3) solver memory usage: 830.0742 MB.
Make sure that the solver is using numpy arrays.
The C implementation of Python (called "CPython") does not use
memory compaction and places most of its objects on the heap.
Those implementations tend to suffer from memory fragmentation in
long running processes with large memory use.
This can be helped somewhat by reusing arrays instead of deleting and
creating a new one.
You may consider "Jython" (a Python implementation in "Java") or
"IronPython" (a ".Net" based Python implementation) to get a
Python implementation with memory compaction.
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
