With further debugging I came to know that snmpd fails to unset the
delegated flag in the global agent_delegated_list (
agent_delegated_list->requests->delegated ) because of memory corruption
In the close_session flow snmpd is freeing all the requests in the
session_list but fails to unset the delegated flag because of corrupted
agent_delegated_list->requests->subtree structure
As this flag is not unset, agent_delegated_list is not getting freed.
After this when set operation is done Snmpd checks if
"agent_delegated_list" is set, as it is not unset set request is getting
queued. As set operation is not processed "netsnmp_processing_set" is
not getting unset. For all the further queries SNMPD first checks
whether "netsnmp_processing_set" is set , as this not is unset all the
requests are getting queued up without getting processed
Can anyone tell me if they have already seen this problem and any help
on this will be appreciated
Code snippet
--------------------
if(request->subtree->session != sess) ==> As memory got corrupted
this condition is getting failed
{
if( asp->pdu )
{
snmp_log(LOG_ERR,"Ravi: Req is not set to
Non_delegated rqid:%ld tid:%ld sid:%ld
\n",asp->pdu->reqid,asp->pdu->transid,sess->sessid);
if( request->subtree->session )
snmp_log(LOG_ERR,"Ravi: Request subtree
sessid:%ld \n",request->subtree->session->sessid);
}
continue;
}
/*
* matched! mark request as done
*/
netsnmp_request_set_error(request, SNMP_ERR_GENERR);
==>request->delegated flag will be unset in this flow which is not
getting executed
(gdb) p *agent_delegated_list->pdu
$16 = {version = 1, command = 163, reqid = 454562527, msgid = 0,
transid = 37457, sessid = 0, errstat = 0, errindex = 0, time = 0,
flags = 0,
securityModel = 2, securityLevel = 1, msgParseModel = 0,
transport_data = 0x100add68, transport_data_length = 16,
tDomain = 0xfe89718, tDomainLen = 7, variables = 0x1009ef28,
community = 0x1009e348 "public", community_len = 6, enterprise = 0x0,
enterprise_length = 0, trap_type = 0, specific_type = 0,
agent_addr = "\000\000\000", contextEngineID = 0x0, contextEngineIDLen
= 0,
contextName = 0x1009dea0 "", contextNameLen = 0, securityEngineID =
0x0,
securityEngineIDLen = 0, securityName = 0x0, securityNameLen = 0,
priority = 0, range_subid = 0, securityStateRef = 0x0}
(gdb)
(gdb) p *agent_delegated_list->requests->subtree
=======>this structure got corrupted
$48 = {name_a = 0x100c4040, namelen = 0 '\0', start_a = 0x6,
start_len = 0 '\0', end_a = 0x4, end_len = 0 '\0', variables = 0xa1,
variables_len = 10, variables_width = 3,
label_a = 0x2 <Address 0x2 out of bounds>, session = 0x1, flags = 0
'\0',
priority = 0 '\0', timeout = 4, next = 0x39, prev = 0x100c3560,
children = 0xfd2fdbc, range_subid = 6, range_ubound = 1, reginfo =
0x4,
cacheid = 1, global_cacheid = 161}
(gdb)
Syslog(redirected debug logs from snmpd):
----------
Jan 23 10:36:26 f101_slot7 snmpd[23515]: Ravi:Req add to sesion_list
rqid: 37458 TID:37457
Jan 23 10:36:26 f101_slot7 snmpd[23515]: Ravi:Reqtime:Wed Jan 23
10:36:26 2008
Jan 23 10:36:26 f101_slot7 snmpd[23515]: Ravi:exptime:Wed Jan 23
10:36:41 2008
Jan 23 10:36:26 f101_slot7 snmpd[23515]: Ravi: Add to Agent_delegated
list rqid:454562527 tid:37457 ***
Jan 23 10:36:26 f101_slot7 snmpd[23515]: Ravi:Select selecting earliest
time from rqid: 37458 sessid:6
Jan 23 10:36:26 f101_slot7 snmpd[23515]: Ravi:earliest exp_time:Wed Jan
23 10:36:41 2008 secs:1201064801 micros:664663
Jan 23 10:36:38 f101_slot7 snmpd[23515]: Ravi:Shut this session down
gracefully CALLBACK_OP_DISCONNECT sid:6
Jan 23 10:36:38 f101_slot7 snmpd[23515]: Ravi: Req is not set to
Non_delegated rqid:454562527 tid:37457 sid:6
Jan 23 10:36:38 f101_slot7 snmpd[23515]: Ravi: Request subtree
sessid:266901272 ===>junk value
Jan 23 10:36:38 f101_slot7 snmpd[23515]: Ravi:Closing socket : 12
sessid:6
Jan 23 10:36:38 f101_slot7 snmpd[23515]: Ravi:Closing session: 6
Jan 23 10:36:38 f101_slot7 snmpd[23515]: Ravi:freeing each element in
the session sessid:6
Jan 23 10:36:38 f101_slot7 snmpd[23515]: Ravi:freeing request:37458
exp_time:Wed Jan 23 10:36:41 2008 secs:1201064801 micros:664663
Thanks,
Ravi
> _____________________________________________
> From: Sekhar Reddy Ravi-G20226
> Sent: Thursday, December 24, 2007 1:11 PM
> To: '[EMAIL PROTECTED]'
> Subject: SNMPD not responding sometimes even though it is up and
> running
>
>
> Hi All,
>
> Sometimes SNMPD is not responding to snmp requests even though it is
> up and running, SNMPD is queuing all the requests without processing
> as "netsnmp_processing_set and agent_delegated_list" are not getting
> unset. However SNMPD is able to send traps originated by subagent.
>
> We are using Net-snmp-5.2.2 as master and subagent. We have
> 1) Applications which does set and walk operations on our enterprise
> mib
> 2) Monitoring application which does get operation on OID
> 1.3.6.1.2.1.1.3.0 (sysUpTime) every 10secs
> 3) Applications which sends number of traps due to changes in the
> system
>
> Our analysis:
> When the problem occurred we are doing walks on our enterprise MIB
> followed by set operation, when getnext operation is done
> "agent_delegated_list" is set and it did not get unset. After this
> when set operation is done in netsnmp_handle_request Snmpd checks if
> "agent_delegated_list" is set, as it is not unset set request is
> getting queued and as set operation is not processed
> "netsnmp_processing_set" is not getting unset. For all the further
> queries SNMPD first checks whether "netsnmp_processing_set" is set ,
> as this not is unset all the requests are getting queued up without
> getting processed
>
> So the root cause seems to be agent_delegated_list field not getting
> unset, even I did not see any pending timers in the alarm list
>
> This problem is coming very frequently. Can you please help us if this
> kind of problem is seen earlier and any help on this will be
> appreciated.
>
>
> gdb) p *netsnmp_processing_set->pdu
> $6 = {version = 1, command = 163, reqid = 34644814, msgid = 0, transid
> = 5095,
> sessid = 0, errstat = 0, errindex = 0, time = 0, flags = 0,
> securityModel = 2, securityLevel = 1, msgParseModel = 0,
> transport_data = 0x100b5ac0, transport_data_length = 16,
> tDomain = 0xfe893c4, tDomainLen = 7, variables = 0x100ca7e0,
> community = 0x10084050 "public", community_len = 6, enterprise =
> 0x0,
> enterprise_length = 0, trap_type = 0, specific_type = 0,
> agent_addr = "\000\000\000", contextEngineID = 0x0,
> contextEngineIDLen = 0,
> contextName = 0x100b92f8 "", contextNameLen = 0, securityEngineID =
> 0x0,
> securityEngineIDLen = 0, securityName = 0x0, securityNameLen = 0,
> priority = 0, range_subid = 0, securityStateRef = 0x0}
> (gdb)
>
> (gdb) p *agent_delegated_list->pdu
> $4 = {version = 1, command = 161, reqid = 1989587127, msgid = 0,
> transid = 2658, sessid = 0, errstat = 0, errindex = 0, time = 0,
> flags = 0,
> securityModel = 2, securityLevel = 1, msgParseModel = 0,
> transport_data = 0x100cc2f8, transport_data_length = 16,
> tDomain = 0xfe893c4, tDomainLen = 7, variables = 0x100ccf38,
> community = 0x100cc2e8 "public", community_len = 6, enterprise =
> 0x0,
> enterprise_length = 0, trap_type = 0, specific_type = 0,
> agent_addr = "\000\000\000", contextEngineID = 0x0,
> contextEngineIDLen = 0,
> contextName = 0x100cc3e0 "", contextNameLen = 0, securityEngineID =
> 0x0,
> securityEngineIDLen = 0, securityName = 0x0, securityNameLen = 0,
> priority = 0, range_subid = 0, securityStateRef = 0x0}
>
>
> Code snippet:
> netsnmp_handle_request(netsnmp_agent_session *asp, int status)
> {
> /*
> * if this isn't a delegated request trying to finish,
> * processing of a set request should not start until all
> * delegated requests have completed, and no other new requests
> * should be processed until the set request completes.
> */
> if ((0 == netsnmp_check_delegated_chain_for(asp)) &&
> (asp != netsnmp_processing_set)) {
> /*
> * if we are processing a set and this is not a delegated
> * request, queue the request
> */
> if (netsnmp_processing_set) {
> netsnmp_add_queued(asp);
> DEBUGMSGTL(("snmp_agent",
> "request queued while processing set, "
> "asp = %08p\n", asp));
> return 1;
> }
> /*
> * check for set request
> */
> if (asp->pdu->command == SNMP_MSG_SET) {
> netsnmp_processing_set = asp;
>
> /*
> * if there are delegated requests, we must wait for them
> * to finish.
> */
> if (agent_delegated_list) {
> DEBUGMSGTL(("snmp_agent", "SET request queued while "
> "delegated requests finish, asp = %08p\n",
> asp));
> netsnmp_add_queued(asp);
> return 1;
> }
>
>
>
> Thanks,
> Ravi
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Net-snmp-users mailing list
[email protected]
Please see the following page to unsubscribe or change other options:
https://lists.sourceforge.net/lists/listinfo/net-snmp-users
