Quoting "SourceForge.net" <[EMAIL PROTECTED]>:
>
https://sourceforge.net/tracker/?func=detail&atid=111005&aid=1906837&group_id=11005
>
> Summary: long list creation
>
> Initial Comment:
> Hi
>
> Creation of long strings can completely hang the system. The cause for the
> hanging is described as follows.
>
> In the following simple macro
>
> string = ""
> for(i=0; i<N ; i++)
> string = string "\n" i
>
> each iteration eats memory since new string is created
> and old one is also kept in memory.
>
> When all memory is occupied a system completely hangs.
>
> When fast-access memory (non-swap) is fully occupied,
> work is strongly slowed-down, CPU usage drops to 1-10%.
>
> Memory is not freed when macro is completed, as a result, the system goes
> in a "somnambular state". X-server restarting does not help in this case.
>
> This bug is independent of NEdit's version.
>
> Alexey Kuznetsov
> ----------------------------------------------------------------------
> Comment By: Tony Balinski (ajbj)
>
> It's actually quite easy to convert NEdit macro's mark/sweep technique so
> that it can run during macro execution. I have a working version of NEdit
> that does this. What needs to be done is to allow the mark part to run
> through the active macro call stacks, allowing marking of function-local
> strings and arrays. For this all continuation contexts need to be
> accessible. So it's all quite easy to do, although it's not clear how often
> it should be done (other than at the end of macro execution). My current
> implementation calls the GC after every 10000 allocations or so. A better
> scheme might relate amount of memory allocated and number of allocations
> instead, which might be better in cases such as this bug highlights.
>
> I have worked on a ref-counted memory management scheme, but that is
> rather more involved, requiring a lot of reorganising and care in
> programming.
>
> I'll make a patch when I have a moment so that others can try it.
>
> Tony
> ----------------------------------------------------------------------
> Comment By: Nobody/Anonymous (nobody)
> Hi,
>
> I think, even "simple" implementation of memory checking/freeing after a
> macro execution/breaking and after a number of allocations can be very
> helpful.
>
> Alexey
Since I find I am unable to add a patch to the sourceforge bug report
mentioned above (am I missing some privileges here?), I'm sending this
patch to the list for perusal.
Share and enjoy!
Tony
Allow macro data garbage collector to run during macro execution
An old limitation of the garbage collector (GC) is its inability to tidy
during macro execution.
This patch converts NEdit macro's mark/sweep technique so that it can run
during macro execution. What needs to be done is to allow the mark part to
run through the active macro call stacks, allowing marking of function-local
strings and arrays. For this all continuation contexts need to be
accessible. This is done by linking them together in a circular linked list.
It is also necessary to add a new data value type for array iterators, so
that the GC can mark their underlying arrays.
Although this is all quite easy to do, it is not clear how often it should
be done (other than at the end of macro execution). This implementation
invokes the GC following an earlier call after a maximum number of
allocations (MAX_N_ALLOC_LIMIT=10000), or after allocation of a certain
amount of space (SIZE_ALLOC_LIMIT=1MB), along with a minimum number of
allocations (MIN_N_ALLOC_LIMIT=200), has occurred. Since the mark/sweep
itself (particularly for situations where lots of global data are
accessible) itself requires much time, calling the GC shouldn't be performed
too frequently.
A more complete solution, which could, in particular, cut down on copying
array structural data, would be a reference counted approach. This patch
though seems a reasonable improvement.
diff -r nedit_official/source/interpret.c nedit_mod/source/interpret.c
72a73,76
> #define MAX_N_ALLOC_LIMIT 10000 /* max num allocs before garbage collect
> */
> #define MIN_N_ALLOC_LIMIT 200 /* min num allocs before garbage collect
> */
> #define SIZE_ALLOC_LIMIT 1048576 /* limit on new allocations before GC */
>
164a169,177
> static long numAllocatedSinceLastGC = 0;
> static size_t sizeAllocatedSinceLastGC = 0;
>
> /* structure to link allocated strings */
> typedef struct GCStringTag {
> struct GCStringTag *next;
> char buff[1]; /* buff[0] is the same as the inUse flag
> */
> } GCString;
>
166c179
< static char *AllocatedStrings = NULL;
---
> static GCString *AllocatedStrings = NULL;
169c182
< SparseArrayEntry data; /* LEAVE this as top entry */
---
> SparseArrayEntry data; /* LEAVE this as top entry */
174c187,188
< static SparseArrayEntryWrapper *AllocatedSparseArrayEntries = NULL;
---
> static SparseArrayEntryWrapper *AllocatedSparseArrayEntries = NULL;
> RestartData *Continuations = NULL;
460a475,511
> static void linkContext(RestartData *context)
> {
> if (Continuations) {
> /* add context behind Continuations */
> context->next = Continuations;
> context->prev = Continuations->prev;
> context->prev->next = Continuations->prev = context;
> }
> else {
> Continuations = context->next = context->prev = context;
> }
> }
> static void dropContext(RestartData *context)
> {
> /* unlink it from the list of continuations */
> Continuations = context->next;
>
> context->next->prev = context->prev;
> context->prev->next = context->next;
>
> if (Continuations == context)
> Continuations = 0;
> }
> static RestartData *newContext(WindowInfo *window, Program *prog)
> {
> RestartData *context;
>
> context = (RestartData *)XtMalloc(sizeof(RestartData));
> context->stack = (DataValue *)XtMalloc(sizeof(DataValue) * STACK_SIZE);
> context->stackP = context->stack;
> context->pc = prog->code;
> context->runWindow = window;
> context->focusWindow = window;
> linkContext(context);
> return context;
> }
>
479,485c530
< context = (RestartData *)XtMalloc(sizeof(RestartData));
< context->stack = (DataValue *)XtMalloc(sizeof(DataValue) * STACK_SIZE);
< *continuation = context;
< context->stackP = context->stack;
< context->pc = prog->code;
< context->runWindow = window;
< context->focusWindow = window;
---
> *continuation = context = newContext(window, prog);
560a606,613
> /**/
> if (numAllocatedSinceLastGC >= MAX_N_ALLOC_LIMIT ||
> (sizeAllocatedSinceLastGC >= SIZE_ALLOC_LIMIT &&
> numAllocatedSinceLastGC >= MIN_N_ALLOC_LIMIT)) {
> saveContext(continuation); /* make sure context is up to date */
> GarbageCollectStrings();
> }
> /**/
611a665
> dropContext(context);
838c892,895
< /* Allocate a new string buffer of length chars */
---
> /*
> ** Allocate a new string buffer of length chars
> ** length does not include terminating null
> */
841,845c898,903
< char *mem;
<
< mem = XtMalloc(length + sizeof(char *) + 1);
< *((char **)mem) = AllocatedStrings;
< AllocatedStrings = mem;
---
> GCString *mem;
> length += sizeof (GCString);
> mem = (GCString *)XtMalloc(length);
> if (mem) {
> mem->next = AllocatedStrings;
> AllocatedStrings = mem;
847c905
< ++numAllocatedStrings;
---
> ++numAllocatedStrings;
849c907,911
< return mem + sizeof(char *) + 1;
---
> ++numAllocatedSinceLastGC;
> sizeAllocatedSinceLastGC += length;
> return &mem->buff[1];
> }
> return NULL;
859c921
< char *mem;
---
> string->rep = AllocString(length);
861,863c923
< mem = XtMalloc(length + sizeof(char *) + 1);
< if (!mem) {
< string->rep = 0;
---
> if (!string->rep) {
867,873d926
<
< *((char **)mem) = AllocatedStrings;
< AllocatedStrings = mem;
< #ifdef TRACK_GARBAGE_LEAKS
< ++numAllocatedStrings;
< #endif
< string->rep = mem + sizeof(char *) + 1;
883,888c936,942
< if (!p)
< return p;
< if (!s)
< s = "";
< p[length] = '\0'; /* forced \0 */
< return strncpy(p, s, length);
---
> if (!p) {
> if (!s)
> s = "";
> p[length] = '\0'; /* forced \0 */
> strncpy(p, s, length);
> }
> return p;
938a993,994
> ++numAllocatedSinceLastGC;
> sizeAllocatedSinceLastGC += sizeof(SparseArrayEntryWrapper);
971,973c1027,1071
< ** Collect strings that are no longer referenced from the global symbol
< ** list. THIS CAN NOT BE RUN WHILE ANY MACROS ARE EXECUTING. It must
< ** only be run after all macro activity has ceased.
---
> ** Mark allocated data as active (so the garbage collection doesn't delete any
> ** of it)
> */
> static void garbageCollectMarkDataValue(DataValue *dv)
> {
> char *s;
> switch (dv->tag)
> {
> case STRING_TAG:
> /* test first because it may be read-only static string */
> s = dv->val.str.rep;
> if (!s[-1])
> s[-1] = 1;
> break;
> case ARRAY_TAG:
> MarkArrayContentsAsUsed(dv->val.arrayPtr);
> break;
> case ARITER_TAG:
> MarkArrayContentsAsUsed(dv->val.iter.array);
> break;
> default:
> break;
> }
> }
>
> /*
> ** Marks all strings and/or arrays accessible in all active context stacks
> */
> static void garbageCollectMarkActiveData()
> {
> RestartData sentinel;
> RestartData *ctx;
>
> linkContext(&sentinel);
> for (ctx = sentinel.next; ctx != &sentinel; ctx = ctx->next) {
> DataValue *dv;
> for (dv = ctx->stack; dv != ctx->stackP; ++dv) {
> garbageCollectMarkDataValue(dv);
> }
> }
> dropContext(&sentinel);
> }
>
> /*
> ** Collect strings that are no longer referenced from the global symbol list.
979c1077
< char *p, *next;
---
> GCString *p;
980a1079,1082
> #ifdef TRACK_GARBAGE_LEAKS
> long nSinceLastTime = numAllocatedSinceLastGC;
> long nTidied = 0;
> #endif
983,984c1085,1086
< for (p = AllocatedStrings; p != NULL; p = *((char **)p)) {
< *(p + sizeof(char *)) = 0;
---
> for (p = AllocatedStrings; p != NULL; p = p->next) {
> p->buff[0] = 0;
995,1003c1097
< if (s->value.tag == STRING_TAG) {
< /* test first because it may be read-only static string */
< if (!(*(s->value.val.str.rep - 1))) {
< *(s->value.val.str.rep - 1) = 1;
< }
< }
< else if (s->value.tag == ARRAY_TAG) {
< MarkArrayContentsAsUsed(s->value.val.arrayPtr);
< }
---
> garbageCollectMarkDataValue(&s->value);
1004a1099,1100
> /* Now mark all values in all existing macro run contexts */
> garbageCollectMarkActiveData();
1007c1103
< next = AllocatedStrings;
---
> p = AllocatedStrings;
1009,1015c1105,1110
< while (next != NULL) {
< p = next;
< next = *((char **)p);
< if (*(p + sizeof(char *)) != 0) {
< *((char **)p) = AllocatedStrings;
< AllocatedStrings = p;
< }
---
> while (p != NULL) {
> GCString *q = p->next;
> if (p->buff[0]) {
> p->next = AllocatedStrings;
> AllocatedStrings = p;
> }
1018a1114
> ++nTidied;
1020c1116
< XtFree(p);
---
> XtFree((char *)p);
1021a1118
> p = q;
1035a1133
> ++nTidied;
1042c1140,1148
< printf("str count = %d\nary count = %d\n", numAllocatedStrings,
numAllocatedSparseArrayElements);
---
> printf("\nNEDIT GARBAGE STATS:\nstrings = %d\narrays = %d\n"
> "since last time: allocs = %ld - frees = %ld => diff = %ld\n"
> " new bytes allocated: %lu\n"
> "Continuations is %sNULL\n",
> numAllocatedStrings, numAllocatedSparseArrayElements,
> nSinceLastTime, nTidied, nSinceLastTime - nTidied,
> (unsigned long)sizeAllocatedSinceLastGC,
> Continuations ? "not " : "");
> fflush(stdout);
1043a1150,1151
> numAllocatedSinceLastGC = 0;
> sizeAllocatedSinceLastGC = 0;
2519c2627
< STACKDUMP(nDim, 3);
---
> STACKDUMP(nDim+2, 3);
2639c2747
< iteratorValPtr->tag = INT_TAG;
---
> iteratorValPtr->tag = ARITER_TAG;
2644c2752,2753
< iteratorValPtr->val.arrayPtr = arrayIterateFirst(&arrayVal);
---
> iteratorValPtr->val.iter.array = arrayVal.val.arrayPtr;
> iteratorValPtr->val.iter.ptr = arrayIterateFirst(&arrayVal);
2707c2816
< thisEntry = iteratorValPtr->val.arrayPtr;
---
> thisEntry = iteratorValPtr->val.iter.ptr;
2713c2822
< iteratorValPtr->val.arrayPtr = arrayIterateNext(thisEntry);
---
> iteratorValPtr->val.iter.ptr = arrayIterateNext(thisEntry);
diff -r nedit_official/source/interpret.h nedit_mod/source/interpret.h
53c53
< enum typeTags {NO_TAG, INT_TAG, STRING_TAG, ARRAY_TAG};
---
> enum typeTags {NO_TAG, INT_TAG, STRING_TAG, ARRAY_TAG, ARITER_TAG};
77a78,82
> typedef struct ArrayIterTag {
> struct SparseArrayEntryTag *ptr;
> struct SparseArrayEntryTag *array;
> } ArrayIter;
>
88a94
> ArrayIter iter;
112c118
< typedef struct {
---
> typedef struct RestartDataTag {
118a125,126
> struct RestartDataTag *next;
> struct RestartDataTag *prev;
--
NEdit Develop mailing list - [email protected]
http://www.nedit.org/mailman/listinfo/develop
