SEMS Conference Memory Consumption Problem: The AmMultiPartMixer will consume memory unabated when more than one session in a conference room and all are listen-only. NOTE: I realize this is not an ordinary occurrence, however it is a possible scenario.
Steps to reproduce problem: 1. Clone https://github.com/sipwise/sems and build 1. git describe = upstream/1.6.0_20140819_0b6d12-58-g4a643f0 2. Configure sems.conf to load DSM, use test_conference.dsm Test #1 A. Connect to SEMS with two users, set both users to listen-only via DTMF 2 B. Use top and observe memory for process SEMS will consume unabated. Test #2 A. Restart SEMS with valgrind --tool=massif /usr/local/sbin/sems B. Connect to SEMS with two users, set both users to listen-only = true, via DTMF 2 C. Run for approximately 4 minutes D. Observe heap consumption with listen-only set to true (***below***) Test #3 A. Restart SEMS with valgrind --tool=massif /usr/local/sbin/sems B. Connect to SEMS with two users, leave listen-only = false, (default) C. Run for approximately 4 minutes D. Observe heap consumption with listen-only set to false (***below***) ***Test #2.D.*** - Observe the large heap allocations from AmMultiPartyMixer::findBufferStateForReading() ------------------------------------------------------------------------ -------- n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B) ------------------------------------------------------------------------ -------- 69 4,557,521,530 3,013,498,808 3,011,980,904 1,517,904 0 70 4,596,656,520 3,028,042,784 3,026,517,656 1,525,128 0 99.95% (3,026,517,656B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->50.28% (1,522,390,488B) 0x4B11A6: MixerBufferState::MixerBufferState(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&) (AmMultiPartyMixer.cpp:313) | ->50.28% (1,522,390,488B) 0x4B1572: AmMultiPartyMixer::findBufferStateForReading(unsigned int, unsigned long long) (AmMultiPartyMixer.cpp:278) | | ->50.28% (1,522,390,488B) 0x4B1720: AmMultiPartyMixer::GetChannelPacket(unsigned int, unsigned long long, unsigned char*, unsigned int&, unsigned int&) (AmMultiPartyMixer.cpp:160) | | ->50.28% (1,522,390,488B) 0x48AC5B: AmConferenceChannel::get(unsigned long long, unsigned char*, int, unsigned int) (AmConferenceChannel.cpp:42) | | ->50.28% (1,522,390,488B) 0x4B69BB: AmPlaylist::get(unsigned long long, unsigned char*, int, unsigned int) (AmPlaylist.cpp:69) | | ->50.28% (1,522,390,488B) 0x4E7AE7: AmSession::writeStreams(unsigned long long, unsigned char*) (AmSession.cpp:1382) | | ->50.28% (1,522,390,488B) 0x4A80F6: AmMediaProcessorThread::processAudio(unsigned long long) (AmMediaProcessor.cpp:296) | | ->50.28% (1,522,390,488B) 0x4A82E8: AmMediaProcessorThread::run() (AmMediaProcessor.cpp:260) | | ->50.28% (1,522,390,488B) 0x50C1E8: AmThread::_start(void*) (AmThread.cpp:66) | | ->50.28% (1,522,390,488B) 0x36FE6079CF: start_thread (in /lib64/libpthread-2.12.so) | | ->50.28% (1,522,390,488B) 0x36FE2E88FB: clone (in /lib64/libc-2.12.so) | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%) | ->49.52% (1,499,370,896B) 0x4B1138: MixerBufferState::MixerBufferState(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&) (AmMultiPartyMixer.cpp:310) | ->49.52% (1,499,370,896B) 0x4B1572: AmMultiPartyMixer::findBufferStateForReading(unsigned int, unsigned long long) (AmMultiPartyMixer.cpp:278) | | ->49.52% (1,499,370,896B) 0x4B1720: AmMultiPartyMixer::GetChannelPacket(unsigned int, unsigned long long, unsigned char*, unsigned int&, unsigned int&) (AmMultiPartyMixer.cpp:160) | | ->49.52% (1,499,370,896B) 0x48AC5B: AmConferenceChannel::get(unsigned long long, unsigned char*, int, unsigned int) (AmConferenceChannel.cpp:42) | | ->49.52% (1,499,370,896B) 0x4B69BB: AmPlaylist::get(unsigned long long, unsigned char*, int, unsigned int) (AmPlaylist.cpp:69) | | ->49.52% (1,499,370,896B) 0x4E7AE7: AmSession::writeStreams(unsigned long long, unsigned char*) (AmSession.cpp:1382) | | ->49.52% (1,499,370,896B) 0x4A80F6: AmMediaProcessorThread::processAudio(unsigned long long) (AmMediaProcessor.cpp:296) | | ->49.52% (1,499,370,896B) 0x4A82E8: AmMediaProcessorThread::run() (AmMediaProcessor.cpp:260) | | ->49.52% (1,499,370,896B) 0x50C1E8: AmThread::_start(void*) (AmThread.cpp:66) | | ->49.52% (1,499,370,896B) 0x36FE6079CF: start_thread (in /lib64/libpthread-2.12.so) | | ->49.52% (1,499,370,896B) 0x36FE2E88FB: clone (in /lib64/libc-2.12.so) | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%) | ->00.16% (4,756,272B) in 1+ places, all below ms_print's threshold (01.00%) ***Test #3.D.*** - Observe much lower heap allocations from AmMultiPartyMixer::findBufferStateForReading() ------------------------------------------------------------------------ -------- n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B) ------------------------------------------------------------------------ -------- 50 2,262,341,650 1,162,752 1,131,568 31,184 0 51 2,283,194,080 1,162,696 1,131,520 31,176 0 52 2,304,244,813 1,162,696 1,131,520 31,176 0 53 2,325,072,788 1,162,696 1,131,520 31,176 0 54 2,345,899,787 1,162,752 1,131,568 31,184 0 55 2,366,743,489 1,162,696 1,131,520 31,176 0 56 2,387,595,226 1,162,752 1,131,568 31,184 0 57 2,408,423,646 1,162,752 1,131,568 31,184 0 58 2,429,253,925 1,162,808 1,131,616 31,192 0 59 2,450,091,678 1,162,752 1,131,568 31,184 0 97.32% (1,131,568B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->26.52% (308,400B) 0x4635CC: AmSession::RTPStream() (AmSession.h:620) | ->26.52% (308,400B) 0x4F08F3: AmSession::getSdpAnswer(AmSdp const&, AmSdp&) (AmSession.cpp:937) | ->15.50% (180,224B) 0x4A2B68: AmEventDispatcher::instance() (AmEventDispatcher.cpp:53) | ->15.50% (180,224B) 0x5F00C40: DSMFactory::DSMFactory(std::string const&) (DSM.cpp:86) | ->09.33% (108,528B) 0x4C9EB7: AmRtpAudio::setPlayoutType(PlayoutType) (AmRtpAudio.cpp:465) | ->09.33% (108,528B) 0x614BE8D: ConfSetPlayoutTypeAction::execute(AmSession*, DSMSession*, DSMCondition::EventType, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >*) (ModConference.cpp:235) | ->09.33% (108,528B) 0x5F197AF: DSMStateEngine::runactions(__gnu_cxx::__normal_iterator<DSMElement**, std::vector<DSMElement*, std::allocator<DSMElement*> > >, __gnu_cxx::__normal_iterator<DSMElement**, std::vector<DSMElement*, std::allocator<DSMElement*> > >, AmSession*, DSMSession*, DSMCondition::EventType, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >*, bool&) (DSMStateEngine.cpp:249) ->08.46% (98,390B) in 227 places, all below massif's threshold (01.00%) | ->06.34% (73,728B) 0x565720: global constructors keyed to trans_table.cpp (hash_table.h:258) | ->06.34% (73,728B) 0x56A7F4: ??? (in /usr/local/sbin/sems) | ->06.34% (73,728B) 0x4495C1: ??? (in /usr/local/sbin/sems) | ->05.64% (65,544B) 0x4B11A6: MixerBufferState::MixerBufferState(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&) (AmMultiPartyMixer.cpp:313) | ->05.64% (65,544B) 0x4B194A: AmMultiPartyMixer::findOrCreateBufferState(unsigned int) (AmMultiPartyMixer.cpp:256) | ->05.64% (65,544B) 0x4B1AE9: AmMultiPartyMixer::PutChannelPacket(unsigned int, unsigned long long, unsigned char*, unsigned int) (AmMultiPartyMixer.cpp:123) | ->04.30% (49,968B) 0x4C65C5: AmPromptCollection::setPrompt(std::string const&, std::string const&, char const*) (AmPromptCollection.cpp:71) | ->04.30% (49,968B) 0x5EFE596: DSMFactory::loadPrompts(AmConfigReader&) (DSM.cpp:291) | ->04.30% (49,968B) 0x5F08FF8: DSMFactory::onLoad() (DSM.cpp:131) ->03.27% (38,000B) 0x5F0533A: DSMFactory::onInvite(AmSipRequest const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) (DSM.cpp:717) | ->03.27% (38,000B) 0x4E0B61: AmSessionContainer::createSession(AmSipRequest const&, std::string&, AmArg*) (AmSessionContainer.cpp:532) | ->03.27% (38,000B) 0x4E237F: AmSessionContainer::startSessionUAS(AmSipRequest&) (AmSessionContainer.cpp:286) | ->02.86% (33,248B) 0x5ECA6F9: DSMCall::playFile(std::string const&, bool, bool) (DSMCall.cpp:628) | ->02.86% (33,248B) 0x5EDBACF: SCPlayFileAction::execute(AmSession*, DSMSession*, DSMCondition::EventType, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >*) (DSMCoreModule.cpp:346) | ->02.86% (33,248B) 0x5F197AF: DSMStateEngine::runactions(__gnu_cxx::__normal_iterator<DSMElement**, std::vector<DSMElement*, std::allocator<DSMElement*> > >, __gnu_cxx::__normal_iterator<DSMElement**, std::vector<DSMElement*, std::allocator<DSMElement*> > >, AmSession*, DSMSession*, DSMCondition::EventType, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >*, bool&) (DSMStateEngine.cpp:249) | ->02.86% (33,248B) 0x5F1D577: DSMStateEngine::runEvent(AmSession*, DSMSession*, DSMCondition::EventType, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >*, bool) (DSMStateEngine.cpp:579) | ->02.86% (33,248B) 0x5EC38EA: DSMCall::startSession() (DSMCall.cpp:233) | ->02.83% (32,912B) 0x48BEB1: AmConferenceStatus::getChannel(std::string const&, int) (AmConferenceStatus.cpp:182) | ->02.83% (32,912B) 0x48C1AD: AmConferenceStatus::getChannel(std::string const&, std::string const&, int) (AmConferenceStatus.cpp:62) | ->02.83% (32,912B) 0x614DFD3: ConferenceJoinChannel(DSMConfChannel**, AmSession*, DSMSession*, std::string const&, std::string const&) (ModConference.cpp:133) | ->02.82% (32,776B) 0x4B1138: MixerBufferState::MixerBufferState(unsigned int, std::set<int, std::less<int>, std::allocator<int> >&) (AmMultiPartyMixer.cpp:310) | ->02.82% (32,776B) 0x4B194A: AmMultiPartyMixer::findOrCreateBufferState(unsigned int) (AmMultiPartyMixer.cpp:256) | ->02.82% (32,776B) 0x4B1AE9: AmMultiPartyMixer::PutChannelPacket(unsigned int, unsigned long long, unsigned char*, unsigned int) (AmMultiPartyMixer.cpp:123) | ->02.82% (32,776B) 0x4B0155: MixerBufferState::add_channel(unsigned int) (AmMultiPartyMixer.cpp:329) | ->02.82% (32,776B) 0x4B0F26: AmMultiPartyMixer::addChannel(unsigned int) (AmMultiPartyMixer.cpp:81) | ->02.82% (32,776B) 0x48BCA9: AmConferenceStatus::getChannel(std::string const&, int) (AmConferenceStatus.cpp:176) | ->01.45% (16,888B) 0x525F9F: main (singleton.h:18) | ->01.44% (16,800B) 0x48C0F8: AmConferenceStatus::getChannel(std::string const&, std::string const&, int) (AmConferenceStatus.cpp:58) | ->01.44% (16,800B) 0x614DFD3: ConferenceJoinChannel(DSMConfChannel**, AmSession*, DSMSession*, std::string const&, std::string const&) (ModConference.cpp:133) | ->01.44% (16,712B) 0x52758E: _SipCtrlInterface::run() (singleton.h:18) | ->01.44% (16,712B) 0x5254DD: main (sems.cpp:608) | ->01.15% (13,362B) 0x3708A9C3C7: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in /usr/lib64/libstdc++.so.6.0.13) | ->01.15% (13,362B) in 5 places, all below massif's threshold (01.00%) | ->01.14% (13,312B) 0x543080: _resolver::_resolver() (hash_table.h:258) ->01.14% (13,312B) 0x4CFD0D: AmRtpStream::setRAddr(std::string const&, unsigned short, unsigned short) (singleton.h:10) ->01.14% (13,312B) 0x4D1F11: AmRtpStream::init(AmSdp const&, AmSdp const&, bool) (AmRtpStream.cpp:708) Kind Regards, Joseph Stusick _______________________________________________ Semsdev mailing list [email protected] http://lists.iptel.org/mailman/listinfo/semsdev
