lol, quot damage!
Janderson Wrote:
> bearophile wrote:
> > While allocating lot of memory for a little memory-hungry program, I have
> > found results that I don't understand. So I have written the following test
> > programs. Maybe someone can give me some information on the matter.
> > I am using a default install of a 32 bit Win XP with 2 GB RAM (so for
> > example I can't allocate 3 GB of RAM). (I presume answers to my questions
> > are Windows-related).
> >
> > From C (MinGW 4.2.1) this is about the largest memory block I can allocate
> > (even it swaps and requires 7+ seconds to run), 1_920_000_000 bytes:
> >
> > #include "stdio.h"
> > #include "stdlib.h"
> > #define N 480000000
> > int main() {
> > unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
> > unsigned int i;
> > if (a != NULL)
> > for (i = 0; i < N; ++i)
> > a[i] = i;
> > else
> > printf("null!");
> > return 0;
> > }
> >
> >
> > But from D this is about the largest memory block I can allocate with
> > std.c.stdlib.malloc, 1_644_000_000 bytes, do you know why the difference?
> >
> > //import std.gc: malloc;
> > import std.c.stdlib: malloc;
> > void main() {
> > const uint N = 411_000_000;
> > uint* a = cast(uint*)malloc(N * uint.sizeof);
> > if (a !is null)
> > for (uint i; i < N; ++i)
> > a[i] = i;
> > else
> > printf("null!");
> > }
> >
> > (If I use std.gc.malloc the situation is different yet, and generally
> > worse).
> >
> > -----------------------
> >
> > So I have tried to use a sequence of smaller memory blocks, this is the C
> > code (every block is about 1 MB):
> >
> > #include "stdio.h"
> > #include "stdlib.h"
> >
> > #define N 250000
> >
> > int main(int argc, char** argv) {
> > unsigned int i, j;
> > unsigned int m = argc == 2 ? atoi(argv[1]) : 100;
> >
> > for (j = 0; j < m; ++j) {
> > unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
> >
> > if (a != NULL) {
> > for (i = 0; i < N; ++i)
> > a[i] = i;
> > } else {
> > printf("null! %d\n", j);
> > break;
> > }
> > }
> >
> > return 0;
> > }
> >
> >
> > And the D code:
> >
> > //import std.gc: malloc;
> > import std.c.stdlib: malloc;
> > import std.conv: toUint;
> >
> > void main(string[] args) {
> > const uint N = 250_000;
> > uint m = toUint(args[1]);
> >
> > for (uint j; j < m; ++j) {
> > uint* a = cast(uint*)malloc(N * uint.sizeof);
> >
> > if (a !is null) {
> > for (uint i; i < N; ++i)
> > a[i] = i;
> > } else {
> > printf("null! %d\n", j);
> > break;
> > }
> > }
> > }
> >
> > With such code I can allocate 1_708_000_000 bytes from D and up to
> > 2_038_000_000 bytes from C (but near the last 100-200 MB of RAM the C code
> > swaps a lot).
> > So can't I can't use all my RAM from my D code? And do you know why?
> >
> > Bye,
> > bearophile
>
>
> bearophile wrote:
> > While allocating lot of memory for a little memory-hungry program, I
> have found results that I don't understand. So I have written the
> following test programs. Maybe someone can give me some information on
> the matter.
> > I am using a default install of a 32 bit Win XP with 2 GB RAM (so for
> example I can't allocate 3 GB of RAM). (I presume answers to my
> questions are Windows-related).
> >
> > From C (MinGW 4.2.1) this is about the largest memory block I can
> allocate (even it swaps and requires 7+ seconds to run), 1_920_000_000
> bytes:
> >
> > #include "stdio.h"
> > #include "stdlib.h"
> > #define N 480000000
> > int main() {
> > unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
> > unsigned int i;
> > if (a != NULL)
> > for (i = 0; i < N; ++i)
> > a[i] = i;
> > else
> > printf("null!");
> > return 0;
> > }
> >
> >
> > But from D this is about the largest memory block I can allocate with
> std.c.stdlib.malloc, 1_644_000_000 bytes, do you know why the difference?
> >
> > //import std.gc: malloc;
> > import std.c.stdlib: malloc;
> > void main() {
> > const uint N = 411_000_000;
> > uint* a = cast(uint*)malloc(N * uint.sizeof);
> > if (a !is null)
> > for (uint i; i < N; ++i)
> > a[i] = i;
> > else
> > printf("null!");
> > }
> >
> > (If I use std.gc.malloc the situation is different yet, and generally
> worse).
> >
> > -----------------------
> >
> > So I have tried to use a sequence of smaller memory blocks, this is
> the C code (every block is about 1 MB):
> >
> > #include "stdio.h"
> > #include "stdlib.h"
> >
> > #define N 250000
> >
> > int main(int argc, char** argv) {
> > unsigned int i, j;
> > unsigned int m = argc == 2 ? atoi(argv[1]) : 100;
> >
> > for (j = 0; j < m; ++j) {
> > unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned
> int));
> >
> > if (a != NULL) {
> > for (i = 0; i < N; ++i)
> > a[i] = i;
> > } else {
> > printf("null! %d\n", j);
> > break;
> > }
> > }
> >
> > return 0;
> > }
> >
> >
> > And the D code:
> >
> > //import std.gc: malloc;
> > import std.c.stdlib: malloc;
> > import std.conv: toUint;
> >
> > void main(string[] args) {
> > const uint N = 250_000;
> > uint m = toUint(args[1]);
> >
> > for (uint j; j < m; ++j) {
> > uint* a = cast(uint*)malloc(N * uint.sizeof);
> >
> > if (a !is null) {
> > for (uint i; i < N; ++i)
> > a[i] = i;
> > } else {
> > printf("null! %d\n", j);
> > break;
> > }
> > }
> > }
> >
> > With such code I can allocate 1_708_000_000 bytes from D and up to
> 2_038_000_000 bytes from C (but near the last 100-200 MB of RAM the C
> code swaps a lot).
> > So can't I can't use all my RAM from my D code? And do you know why?
> >
> > Bye,
> > bearophile
>
> Different allocation schemes have different strengths and weaknesses.
> Some are fast, some fragment less, some have less overhead, some allow
> larger sized blocks. Often these things arn't mutual so there are
> always tradoffs. For example, to improve speed an allocator may
> allocate into particular buckets which might restrict the maximum size
> of one allocation.
>
> I wonder how Ned-Malloc or Hord perform with your tests?
>
> -Joel
