[sr-dev] [kamailio/kamailio] DMQ memory leak with dmq_bcast_message function (#2600)

2021-01-11 Thread jklingenmeyer
### Description

We noticed a PKG memory leak when using DMQ module's function 
_dmq_bcast_message_ .

### Troubleshooting

 Reproduction

Issue can be reproduced with configuration below.

* 3 Kamailio nodes each one listening on dedicated port 5070 for DMQ on their 
private IP address.
* A dns name _[KAMAILIO_DNS]_ resolving these 3 nodes:

```
# dig +short A [KAMAILIO_DNS]
10.234.97.169
10.234.97.170
10.234.92.126
```

DMQ module configured as below (example for node 1)
```
loadmodule "dmq.so"
...
modparam("dmq", "server_address", "sip:10.234.97.169:5070")
modparam("dmq", "notification_address", "sip:[KAMAILIO_DNS]:5070")
modparam("dmq", "ping_interval", 10)
modparam("dmq", "multi_notify", 1)
```

Each time an INVITE comes in, we use function below to broadcast custom message 
to all of the nodes:

```
...
if(is_method("INVITE")) {
dmq_bcast_message("invitenotification", "invite received", "text/plain");
}
...
```

This custom message is handled as below:

```
request_route {
...
if(is_method("KDMQ")) {
if($Rp == 5070) route(DMQHANDLE);
else drop;
}
...
}
...

route[DMQHANDLE] {
if($tU == "invitenotification") {
# Doing some additional stuff here, but not the cause of the leak
send_reply("200", "OK");
} else {
dmq_handle_message();
}
}
```

Using **sipp** to send lots of messages, we can easily notice the free PKG 
memory decreasing over time (using `kamcmd pkg.stats`)

 Debugging Data

Processes impacted by the leak are the ones processing SIP messages. In case of 
TCP, it is one of the tcp receivers.
When doing a dump of such processes with command below:

```
kamcmd cfg.set_now_int core mem_dump_pkg [PID]
```

Results are full of records like these ones:

```
ALERT: qm_status:   alloc'd from core: core/parser/parse_from.c: 
parse_from_header(63)
ALERT: qm_status:   alloc'd from core: core/parser/parse_addr_spec.c: 
parse_to_param(285)
```

We are sure that the leak is generated by this DMQ broadcast function: we 
isolated it and noticed no leaks when removing it from a basic script 
configuration.

We tried different configurations:
* disabling multi_notify => memory leak still present
* using an IP address instead of the DNS name => memory leak still present

Valgrind did not help us to find out more precisely where the issue in the code 
could be located.

### Possible Solutions

Would a _free_ of _msg->from->parsed_ be missing somewhere in DMQ code?
Behavior looks very similar to https://github.com/kamailio/kamailio/pull/59

### Additional Information

  * **Kamailio Version** - output of `kamailio -v`

```
version: kamailio 5.3.7 (x86_64/linux) c16b3f-dirty
flags: , EXTRA_DEBUGUSE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, 
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, 
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, TIMER_DEBUG, USE_FUTEX, 
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, 
USE_DST_BLACKLIST, HAVE_RESOLV_RES
```

* **Operating System**:

```
Debian 9.13 under docker
Linux 4.9.0-9-amd64
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2600___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev


Re: [sr-dev] [kamailio/kamailio] DMQ memory leak with dmq_bcast_message function (#2600)

2021-03-23 Thread Daniel-Constantin Mierla
Could not spot any potential issue upon quick look at the code.

To use valgrind to track private memory issues, you have to compile Kamailio 
with system memory, like:

```
make MEMPKG=sys include_modules="..." cfg
make all
...
```

Try to see if you can get any leads with valgrind.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2600#issuecomment-804900118___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev


Re: [sr-dev] [kamailio/kamailio] DMQ memory leak with dmq_bcast_message function (#2600)

2021-03-24 Thread jklingenmeyer
Thanks for having a look into it. We did compile with system memory and use 
valgrind following the tips on dedicated page 
https://www.kamailio.org/wiki/tutorials/troubleshooting/memory

Unfortunately this gave no more clues (though it is possible I used it badly 
because it was the first time I used such tools).

We removed totally DMQ module from our Kamailios anyhow.
We replaced it with a solution based on mqueue/http_async_client modules.

This issue can be closed as I am no more able to help in investigating further.
Thank you.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2600#issuecomment-805614317___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev