Looks like pattern_db is a global. Are you sure you don’t have multiple Go
routines calling the load? The global changes may not be visible- so the free
is doing nothing.
You probably need synchronization or pinning the access routine to a thread.
> On Mar 16, 2020, at 9:03 AM, Nitish Saboo <nitish.sabo...@gmail.com> wrote:
>
>
> Hi,
>
> From reading the code, I assume that the `pattern_db_new()` does some sort of
> allocation of a new pattern DB. I don't see any code releasing the pattern
> DB. Is that just missing from your post or something that some automatic
> mechanism does?
>
> >>Apologies, that part got missed in the code snippet. I am making the free
> >>call in the C code.
>
> node.c
> ---------
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
> if(patterndb != NULL){
> pattern_db_free(patterndb);
> }
> patterndb = pattern_db_new();
> pattern_db_reload_ruleset(patterndb, configuration, file);
> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
> return 0;
> }
>
> You can try a similar C program that instantiates and frees the structure to
> check for similar behavior.
>
> >> To verify if the C code had some issue, I called the C wrapper code method
> >> 'load_pattern_db' from my main.c to completely eliminate Go code here.
> >> What I found is there is no increase in memory consumption after every
> >> call ('load_pattern_db' was called 5 times). Hence there is no issue from
> >> C code.
>
> Thanks,
> Nitish
>
>> On Mon, Mar 16, 2020 at 7:04 PM Gregor Best <b...@pferdewetten.de> wrote:
>> This might be a dumb question but...
>>
>> From reading the code, I assume that the `pattern_db_new()` does some sort
>> of allocation of a new pattern DB. I don't see any code releasing the
>> pattern DB. Is that just missing from your post or something that some
>> automatic mechanism does?
>>
>> If not, that might be your leak.
>>
>>> On 09.03.20 12:33, Nitish Saboo wrote:
>>> Hi
>>>
>>> Following are my Go code and C header file and C wrapper code
>>>
>>> parser.go
>>> ==========
>>> var f *os.File
>>>
>>> func LoadPatternDB(patterndb string) {
>>> path := C.CString(patterndb)
>>> defer C.free(unsafe.Pointer(path))
>>> C.load_pattern_db(path, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
>>> }
>>>
>>> //export ParsedData
>>> func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
>>> f.WriteString(C.GoString(k))
>>> f.WriteString("\n")
>>> }
>>>
>>> cfunc.go
>>> ========
>>> /*
>>> #include <stdio.h>
>>> // The gateway function
>>> void callOnMeGo_cgo(char *k, char *val, size_t val_len)
>>> {
>>> void ParsedData(const char *k, const char *val, size_t val_len);
>>> ParsedData(k, val, val_len);
>>> }
>>> */
>>> import "C"
>>>
>>> node.h
>>> =======
>>>
>>> #ifndef TEST_H_INCLUDED
>>> #define TEST_H_INCLUDED
>>>
>>> #include <stdlib.h>
>>>
>>> typedef void (*key_value_cb)(const char* k, const char* val, size_t
>>> val_len);
>>> int load_pattern_db(const char* file, key_value_cb cb);
>>>
>>> #endif
>>>
>>> node.c
>>> ---------
>>> int load_pattern_db(const gchar* file, key_value_cb cb)
>>> {
>>> patterndb = pattern_db_new();
>>> pattern_db_reload_ruleset(patterndb, configuration, file);
>>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>> return 0;
>>> }
>>>
>>>
>>> I am calling 'LoadPatternDB' method in my parser.go file that makes a cgo
>>> call 'C.load_pattern_db' where I am passing a callback function to the C
>>> code.
>>> The C code is a wrapper code that internally calls some syslog-ng library
>>> apis'
>>>
>>> What I observed is:
>>>
>>> 1)As soon as I call LoadPatternDB() method in parser.go there is some
>>> increase in memory consumption(some memory leak).Ideally that should not
>>> have happened.
>>>
>>> 2)To verify if the C code had some issue, I called the C wrapper code
>>> method 'load_pattern_db' from my main.c in the following manner to
>>> completely eliminate Go code here.What I found is there is no increase in
>>> memory consumption after every call ('load_pattern_db' was called 5
>>> times).Hence there is no memory leak from C code.So the issue lies in the
>>> Go code in 'LoadPatternDB' method in parser.go
>>>
>>> main.c
>>> =======
>>>
>>> void check(char *key, char *value, size_t value_len)
>>> {
>>> printf("I am in function check\n");
>>> }
>>>
>>> int main(void){
>>> char* filename = "/home/nitish/default.xml";
>>> key_value_cb s = check;
>>> int i;
>>> for (i=1; i<=5; i++)
>>> {
>>> load_pattern_db(filename, s);
>>> printf("Sleeping for 5 second.\n");
>>> sleep(5);
>>> }
>>> printf("Loading done 5 times.\n");
>>> return 0;
>>> }
>>>
>>> 3)Can someone please guide me and help me figure out the mem-leak in
>>> 'LoadPatternDB' method in parser.go at very first glance? Is the callback
>>> function pointer an issue here ?
>>>
>>> 4)What tool can I use to check this mem-leak ?
>>>
>>> Thanks,
>>> Nitish
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq5cAJ19CQ8OmMPSJmVB1P3t4hE0CKZ4HsEsH-mp6zm3Ng%40mail.gmail.com.
>> --
>> --
>> Gregor Best
>> b...@pferdewetten.de
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/fee07a13-e3d7-4a2e-6a6c-fdb11140ce21%40pferdewetten.de.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CALjMrq4-%3DcsuKRP4Xa-MVgrm5MJdWLNVe0v5_i3jswjesHQTqA%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/4A1AFEDD-2D70-4B33-A183-83887A79A9A0%40ix.netcom.com.
