Howdy,
Here's a followup on my problem. I think it is clearly a problem with
valgrind in large memory conditions. Perhaps I can build valgrind in a way
that allows for larger blocks? (if that is true, please point me in the
right direction).
Since my program has an alternative allocation-tracking technique built in,
I wrote a converter that converts that allocation-tracker's output to a test
program that does the same series of allocations. This test program is
appended below my signature. It does nothing but allocate, reallocate, and
free.
I then built this on a linux-x86_64 machine with these gcc commands:
cc -c -g -O0 -pg lz_memory_test.c -o lz_memory_test_valgrind.o
cc lz_memory_test_valgrind.o -o lz_memory_test_valgrind
Then I ran valgrind on it, like this:
valgrind --leak-check=full lz_memory_test_valgrind
valgrind version is 3.5.0
Output from valgrind is below the program. I get the same sort of "set
address range perms" I get on my real program (please, if somebody could
give me more info about those, it would be helpful). Then it tells me I
have 2,736,652,288 bytes in use at exit. This would seem to relate to lines
84 and 85 of the program (which I have marked with trailing comments). Line
84 allocates exactly that many bytes, as part of a realloc of m14. Line 85
realloc's again, increasing the size. m14 is realloc'd several more times
after that, then freed at line 184. So, it seems that either (a) there's a
bug in realloc, and it did not dispose of the incoming m14 at line 85, or
(b) valgrind has lost track.
I also notice that the last "address range perms" complaint has a size that
corresponds exactly to the allocation of m14 at line 83, i.e. the last
allocation before the allocation that appears to be "lost"
Thanks for any help,
Bob H
My test program, lz_memory_test.c, which (I claim) allocates, reallocates,
and frees the same as my target program does.
#include <stdlib.h>
#include <stdio.h>
int main (int argc, char** argv);
int main (int argc, char** argv)
{
void* m1;
void* m2;
void* m3;
void* m4;
void* m5;
void* m6;
void* m7;
void* m8;
void* m9;
void* m10;
void* m11;
void* m12;
void* m13;
void* m14;
void* m15;
void* m16;
void* m17;
void* m18;
void* m19;
void* m20;
void* m21;
void* m22;
void* m23;
void* m24;
void* m25;
void* m26;
void* m27;
void* m28;
void* m29;
void* m30;
void* m31;
void* m32;
void* m33;
void* m34;
m1 = malloc((size_t)218);
m2 = malloc((size_t)51);
m3 = malloc((size_t)32);
m4 = malloc((size_t)20);
m5 = malloc((size_t)16);
m6 = malloc((size_t)263984);
m7 = malloc((size_t)184);
m8 = malloc((size_t)263984);
free(m4);
m9 = malloc((size_t)632);
m10 = malloc((size_t)1024);
m11 = malloc((size_t)51);
m12 = realloc(NULL,(size_t)16384);
m13 = realloc(NULL,(size_t)16384);
m14 = realloc(NULL,(size_t)16384);
m15 = realloc(NULL,(size_t)5);
m14 = realloc(m14,(size_t)249266176);
m16 = realloc(NULL,(size_t)156);
m17 = realloc(NULL,(size_t)156);
m18 = realloc(NULL,(size_t)1300108);
m19 = realloc(NULL,(size_t)1300108);
m20 = realloc(NULL,(size_t)5);
m14 = realloc(m14,(size_t)492453888);
m14 = realloc(m14,(size_t)690487296);
m14 = realloc(m14,(size_t)881639424);
m14 = realloc(m14,(size_t)1062551552);
m14 = realloc(m14,(size_t)1233666048);
m14 = realloc(m14,(size_t)1392803840);
m14 = realloc(m14,(size_t)1548075008);
m14 = realloc(m14,(size_t)1694433280);
m14 = realloc(m14,(size_t)1835646976);
m16 = realloc(m16,(size_t)160);
m17 = realloc(m17,(size_t)160);
m15 = realloc(m15,(size_t)6);
m14 = realloc(m14,(size_t)1971191808);
m20 = realloc(m20,(size_t)6);
m14 = realloc(m14,(size_t)2106195968);
m14 = realloc(m14,(size_t)2240053248);
m14 = realloc(m14,(size_t)2355216384);
m14 = realloc(m14,(size_t)2462564352);
m14 = realloc(m14,(size_t)2565095424);
m14 = realloc(m14,(size_t)2655453184); /* line 83 */
m14 = realloc(m14,(size_t)2736652288); /* line 84 */
m14 = realloc(m14,(size_t)2814722048); /* line 85 */
m14 = realloc(m14,(size_t)2877751296);
m14 = realloc(m14,(size_t)2937126912);
m14 = realloc(m14,(size_t)2996256768);
m14 = realloc(m14,(size_t)3047555072);
m14 = realloc(m14,(size_t)3095691264);
m15 = realloc(m15,(size_t)15);
m14 = realloc(m14,(size_t)3100606464);
m20 = realloc(m20,(size_t)15);
m14 = realloc(m14,(size_t)3105439744);
m14 = realloc(m14,(size_t)3110240256);
m14 = realloc(m14,(size_t)3114926080);
m14 = realloc(m14,(size_t)3119546368);
m14 = realloc(m14,(size_t)3124166656);
m14 = realloc(m14,(size_t)3128770560);
m15 = realloc(m15,(size_t)16);
m14 = realloc(m14,(size_t)3130458112);
m20 = realloc(m20,(size_t)16);
m14 = realloc(m14,(size_t)3131047936);
m15 = realloc(m15,(size_t)21);
m14 = realloc(m14,(size_t)3131588608);
m20 = realloc(m20,(size_t)21);
m14 = realloc(m14,(size_t)3131801600);
m14 = realloc(m14,(size_t)3131998208);
m14 = realloc(m14,(size_t)3132178432);
m14 = realloc(m14,(size_t)3132375040);
m14 = realloc(m14,(size_t)3132555264);
m14 = realloc(m14,(size_t)3132735488);
m14 = realloc(m14,(size_t)3132932096);
m14 = realloc(m14,(size_t)3133112320);
m14 = realloc(m14,(size_t)3133292544);
m14 = realloc(m14,(size_t)3133472768);
m15 = realloc(m15,(size_t)22);
m14 = realloc(m14,(size_t)3133636608);
m20 = realloc(m20,(size_t)22);
m14 = realloc(m14,(size_t)3133816832);
m14 = realloc(m14,(size_t)3133980672);
m14 = realloc(m14,(size_t)3134160896);
m14 = realloc(m14,(size_t)3134324736);
m14 = realloc(m14,(size_t)3134488576);
m14 = realloc(m14,(size_t)3134652416);
m14 = realloc(m14,(size_t)3134816256);
m14 = realloc(m14,(size_t)3134980096);
m14 = realloc(m14,(size_t)3135143936);
m14 = realloc(m14,(size_t)3135291392);
m14 = realloc(m14,(size_t)3135438848);
m14 = realloc(m14,(size_t)3135553536);
m14 = realloc(m14,(size_t)3135684608);
m14 = realloc(m14,(size_t)3135799296);
m14 = realloc(m14,(size_t)3135881216);
m14 = realloc(m14,(size_t)3135979520);
m14 = realloc(m14,(size_t)3136061440);
m14 = realloc(m14,(size_t)3136110592);
m14 = realloc(m14,(size_t)3136159744);
m14 = realloc(m14,(size_t)3136192512);
m14 = realloc(m14,(size_t)3136241664);
m14 = realloc(m14,(size_t)3136290816);
m14 = realloc(m14,(size_t)3136323584);
m14 = realloc(m14,(size_t)3136372736);
m14 = realloc(m14,(size_t)3136405504);
m14 = realloc(m14,(size_t)3136454656);
m14 = realloc(m14,(size_t)3136487424);
m14 = realloc(m14,(size_t)3136536576);
m14 = realloc(m14,(size_t)3136569344);
m14 = realloc(m14,(size_t)3136618496);
m14 = realloc(m14,(size_t)3136651264);
m14 = realloc(m14,(size_t)3136684032);
m14 = realloc(m14,(size_t)3136733184);
m14 = realloc(m14,(size_t)3136765952);
m14 = realloc(m14,(size_t)3136798720);
m14 = realloc(m14,(size_t)3136847872);
m14 = realloc(m14,(size_t)3136880640);
m14 = realloc(m14,(size_t)3136913408);
m14 = realloc(m14,(size_t)3136946176);
m14 = realloc(m14,(size_t)3136995328);
m14 = realloc(m14,(size_t)3137028096);
m14 = realloc(m14,(size_t)3137060864);
m14 = realloc(m14,(size_t)3137093632);
m14 = realloc(m14,(size_t)3137110016);
m14 = realloc(m14,(size_t)3137126400);
m14 = realloc(m14,(size_t)3137142784);
m14 = realloc(m14,(size_t)3137159168);
m14 = realloc(m14,(size_t)3137175552);
m21 = malloc((size_t)632);
m22 = malloc((size_t)1024);
m23 = malloc((size_t)32);
m24 = realloc(NULL,(size_t)16384);
m25 = realloc(NULL,(size_t)16384);
m26 = malloc((size_t)12615754384);
m27 = malloc((size_t)160032);
m28 = realloc(NULL,(size_t)16384);
m29 = realloc(NULL,(size_t)30);
m30 = realloc(NULL,(size_t)8);
m30 = realloc(m30,(size_t)9);
m31 = malloc((size_t)16777216);
m32 = malloc((size_t)16777216);
m33 = malloc((size_t)33554432);
m34 = malloc((size_t)16777216);
free(m2);
free(m14); /* line 184 */
free(m10);
free(m11);
free(m15);
free(m20);
free(m16);
free(m17);
free(m18);
free(m19);
free(m12);
free(m13);
free(m9);
free(m3);
free(m28);
free(m22);
free(m23);
free(m29);
free(m30);
free(m24);
free(m25);
free(m21);
free(m1);
free(m6);
free(m8);
free(m26);
free(m27);
free(m7);
free(m5);
free(m31);
free(m32);
free(m33);
free(m34);
return 0;
}
==12544== Memcheck, a memory error detector
==12544== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==12544== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==12544== Command: ./lz_memory_test_valgrind
==12544==
==12544== Warning: set address range perms: large range [0x13dde030,
0x31382050) (noaccess)
==12544== Warning: set address range perms: large range [0x3901d030,
0x6229d050) (noaccess)
==12544== Warning: set address range perms: large range [0x6229e030,
0x96b6a050) (noaccess)
==12544== Warning: set address range perms: large range [0x96b6b030,
0xd60bf050) (noaccess)
==12544== Warning: set address range perms: large range [0xd60c0030,
0x11f944050) (noaccess)
==12544== Warning: set address range perms: large range [0x11f945030,
0x17298d050) (noaccess)
==12544== Warning: set address range perms: large range [0x17298e030,
0x1cedea050) (noaccess)
==12544== Warning: set address range perms: large range [0x1cedeb030,
0x233ddb050) (noaccess)
==12544== Warning: set address range perms: large range [0x233ddc030,
0x2a1478050) (noaccess)
==12544== Warning: set address range perms: large range [0x2a1479030,
0x316c59050) (noaccess)
==12544== Warning: set address range perms: large range [0x316c5a030,
0x3944fa050) (noaccess)
==12544== Warning: set address range perms: large range [0x43dd9b030,
0x4c35e3050) (noaccess)
==12544== Warning: set address range perms: large range [0x4c766c030,
0x553c88050) (noaccess)
==12544== Warning: set address range perms: large range [0x557cad030,
0x5ea929050) (noaccess)
==12544== Warning: set address range perms: large range [0x5ee492030,
0x6872d6050) (noaccess)
==12544== Warning: set address range perms: large range [0x68a7e7030,
0x728c57050) (noaccess)
==12544==
==12544== HEAP SUMMARY:
==12544== in use at exit: 2,736,652,288 bytes in 1 blocks
==12544== total heap usage: 140 allocs, 64 frees, 33,075,445,557 bytes
allocated
==12544==
==12544== 2,736,652,288 bytes in 1 blocks are definitely lost in loss record
1 of 1
==12544== at 0x4A05F1D: realloc (vg_replace_malloc.c:476)
==12544== by 0x40091E: main (lz_memory_test.c:84)
==12544==
==12544== LEAK SUMMARY:
==12544== definitely lost: 2,736,652,288 bytes in 1 blocks
==12544== indirectly lost: 0 bytes in 0 blocks
==12544== possibly lost: 0 bytes in 0 blocks
==12544== still reachable: 0 bytes in 0 blocks
==12544== suppressed: 0 bytes in 0 blocks
==12544==
==12544== For counts of detected and suppressed errors, rerun with: -v
==12544== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)
On Mar 28, 2011, at 4:10 PM, I wrote:
> I'm trying to check for leaks in a program I've written, using "valgrind
> --leak-check=full ...". When my program stays within relatively small
memory
> usage, that is, less than 2G total, valgrind has no complaints. However,
one
> of my test cases involves my program allocating a total of about 16G with
> about 12.6G in a single block. In this case, valgrind gives me this
warning,
> repeatedly (maybe a dozen times):
==31171== Warning: set address range perms: large range [0x13dde030,
0x31382050) (noaccess)
Then complains about a dozen times about "Invalid write of size 1" in
memcpy
> called by fread, as well as one other complaint, "Syscall param read(buf)
> points to unaddressable byte(s)", also in something called by fread.
After
> this it tells me it has detected more than 10 million errors and tells me
to
> "Go fix [my] program!". Then after reporting a page or so of stats, I get
a
> segfault.
>
> The program doesn't segfault on this case when built without without
> valgrind, and to the extent that I can tell, is giving me the correct
result
> (when valgrind isn't used).
>
> The machine I'm running on is a 64-bit linux machine with 32G of RAM.
This
> occurred with valgrind version 3.5.0 and also with version 3.6.1.
>
> I have attempted to find info on how to "set address range perms", but the
> best I have been able to find so far is that this is a message intended
for
> valgrind developers, not users. Is this something I can change if I build
> the valgrind source code?
>
> So... my question is, should I expect valgrind to work correctly in such
an
> environment? Or, equivalently, what are valgrind's memory limitations?
>
> Also worth noting, possibly, is that the single 12.6G block is the result
of
> several reallocs. Since the target program does not know a priori how
large
> this needs to be, it grows the block on an as-needed basis. I believe
> valgrind is reporting a "set address range perms" message ever time
realloc
> runs out of space in the current block. The first report has
> large range [0x13dde030, 0x31382050)
> and the final one has
> large range [0x68c353030, 0x72a7c3050)
> But if that last one is indeed the memory allocated for my huge block, it
is
> only 2.7G.
>
> I'm trying to decide whether I truly have a problem in my program, or if
the
> problem that valgrind doesn't handle blocks of this size.
------------------------------------------------------------------------------
Create and publish websites with WebMatrix
Use the most popular FREE web apps or write code yourself;
WebMatrix provides all the features you need to develop and
publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
_______________________________________________
Valgrind-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
