shaop created an issue (kamailio/kamailio#4542)
## Environment
- **OS**: Linux (running in container/pod)
- **Hardware**: 8 vCPU cores, 16GB RAM
- **Kamailio Version**: 6.0.3
- **Workload**: ~1000 CPS (Calls Per Second)
- **Memory Issue**: Virtual memory usage reaches 8GB causing system instability
## Configuration Details
### Kamailio Process Configuration
```cfg
children = 32
async_workers = 16
```
### Dialog Module Configuration
```cfg
modparam("dialog", "enable_stats", 1)
modparam("dialog", "default_timeout", 25200) # 7 hours
modparam("dialog", "db_mode", 0) # memory-only mode
modparam("dialog", "hash_size", 32768)
modparam("dialog", "dlg_match_mode", 1)
```
### Transaction Module Configuration
```cfg
modparam("tm", "failure_reply_mode", 3)
modparam("tm", "fr_timer", 30000)
modparam("tm", "fr_inv_timer", 120000)
```
### Dispatcher Module Configuration
```cfg
modparam("dispatcher", "db_url",
"mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@${MYSQL_IP}/${DB_NAME}")
modparam("dispatcher", "table_name", "km_dispatcher")
modparam("dispatcher", "ds_probing_mode", 3)
modparam("dispatcher", "flags", 2)
modparam("dispatcher", "ds_probing_threshold", 3)
modparam("dispatcher", "ds_ping_interval", 30)
modparam("dispatcher", "ds_ping_reply_codes",
"class=2;code=403;code=488;class=3")
modparam("dispatcher", "xavp_dst", "_dsdst_")
modparam("dispatcher", "xavp_ctx", "_dsctx_")
```
### Rate Limiting
```lua
-- CPS limit set to 12000
KSR.pipelimit.pl_check_limit("CPS", "TAILDROP", 12000)
```
## Problem Description
When running at approximately **1000 CPS**, the system's virtual memory
consumption rapidly grows to **8GB**, exhausting available memory resources and
causing:
- System OOM (Out of Memory) conditions
- Process crashes or restarts
- Service degradation
This seems disproportionate for the given workload, as 1000 CPS should
theoretically be well within the capacity of an 8-core 16GB system.
## Expected Behavior
At 1000 CPS on an 8-core 16GB system with the current configuration:
- Memory usage should remain stable and reasonable (< 1GB)
- The system should handle the load without memory exhaustion
- With 32 children processes, each process should consume manageable memory
## Actual Behavior
- Virtual memory grows to 8GB and continues to climb
- Memory does not stabilize even after traffic becomes steady
- Potential memory leak or inefficient memory management
## Relevant Configuration Analysis
### Potential Contributing Factors
1. **High Process Count**
- 32 child processes + 16 async workers = 48 total processes
- Each process loads Lua interpreter and modules
- Base memory per process could be significant
2. **Lua Module Usage**
- Multiple Lua modules loaded: `lunajson`, custom modules
- Each worker process maintains separate Lua state
- Potential for Lua garbage collection issues
3. **No Shared Memory Database**
- Dialog module not using database backend
- All state kept in process memory
## Memory Usage Estimation
**Observed metrics:**
- At 1000 CPS, system shows **20,000 active calls** (average call duration ~20
seconds)
- With `hash_size = 32768` and no database mode
- 32 child processes each maintaining dialog state
- Estimated per-dialog memory: 1-5KB
- **Expected total**: 20,000 × 2KB = **~40MB** for dialog state (if shared via
shared memory)
- **Actual observation**: System consuming **8GB virtual memory**
**This indicates a ~200x memory overhead**, which suggests potential issues
with:
- Dialog state not being shared efficiently across processes
- Each of 32 child processes maintaining separate dialog copies: 20,000 × 2KB ×
32 = **~1.3GB**
- Additional overhead from Lua interpreter, transaction module, and other
components
However, this doesn't account for:
- Transaction memory
- Lua interpreter overhead
- Module memory
- Buffer allocations
## Questions
1. **Is this memory growth expected** given the configuration, or is there a
potential memory leak?
2. **Dialog module memory model**: With `db_mode = 0`, does each child process
maintain its own copy of dialog state, or is it shared via shared memory?
3. **Recommended configuration**: For 1000 CPS workload on 8-core 16GB system:
- What are the recommended values for:
- `children` count?
- `dialog.hash_size`?
- `dialog.default_timeout`?
- Should `db_mode` be enabled to reduce memory pressure?
4. **Lua memory management**: Are there known issues with Lua garbage
collection under high load? Should we configure Lua memory limits?
## Troubleshooting Steps Attempted
- [X] Reviewed configuration for obvious misconfigurations
## Requested Assistance
1. **Configuration review**: Are there obvious misconfigurations causing
excessive memory usage?
2. **Memory profiling guidance**: What tools/methods are recommended for
identifying memory leaks in Kamailio with Lua modules?
3. **Optimization recommendations**: How can we optimize this configuration for
1000 CPS on the given hardware?
4. **Best practices**: What are the recommended settings for:
- Process count (`children`, `async_workers`)
- Dialog module parameters for moderate CPS
- Lua memory management
## Additional Context
This is a **stateless SIP proxy** implementation using Kamailio with minimal
routing logic:
- **Simple dispatcher-based routing only**: `KSR.dispatcher.ds_select_dst("6",
"4")` for round-robin load distribution
- Dialog module enabled **only for call tracking/statistics**, not for B2BUA
functionality
- NAT traversal support via nathelper
- Rate limiting via pipelimit
- Minimal Lua logic - just basic routing decisions
Any guidance on memory optimization for this use case would be greatly
appreciated.
---
**Kamailio Configuration Snippets**
Full configuration available upon request. Key modules in use:
- tm, dialog, dispatcher, app_lua, nathelper, pipelimit, ndb_redis
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4542
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/[email protected]>_______________________________________________
Kamailio - Development Mailing List -- [email protected]
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not reply only to the
sender!
