The best solution would be if ARServer had a configuration option, a thread keep-alive if you will, that would do this. This would avoid the busy system errors that sessions will get if all threads are busy.
Axton Grams On 9/20/07, J.T. Shyman <[EMAIL PROTECTED]> wrote: > Axton, you're 100% correct. And we thought of that too. We just don't know a > way to ensure we hit _all_ of the open threads at least once an hour. BMC's > suggestion was to "hit the server hard enough to use all the queues like you > would under load testing" but I have the same doubts you do: will this cause > deterioration in the user experience or server performance? I'm guessing, as > you are, that it would. > > I'm 99% sure the firewall is Cisco of some sort and you may be right on the > state table only being created on SYN packets but that means that any SYN > packet passing through the firewall (the start of any TCP connection) that > passes a rule would be added to the state table. After that any traffic, > regardless of packet type, would be covered by the entry in the state table > as long as it was over the established connection, wouldn't it? Then the > problem arises when the state table, to save firewall resources, clears out > old, defunct connections. > > I'm glad someone else agrees that the best approach to this would be to > eliminate the network devices that may be causing the issue rather than > trying to engineer ARS to keep all the connections open. > > It does amaze me, though, that BMC can call ARS an "enterprise" product when > it behaves so badly with stateful firewalls. > > J.T. Shyman > Column Technologies > Cell: 404-242-5407 > > -----Original Message----- > From: Axton [mailto:[EMAIL PROTECTED] > Sent: Wednesday, September 19, 2007 10:25 PM > Subject: Re: ARS 7/Oracle and Firewalls/Network devices > > The escalation is (was) single threaded; in order to send traffic over > every db connection, you have to exercise every thread. Since the > escalation engine is single threaded, it will only occupy that one > thread. If you notice in the arerror.log that all filter errors > reported show 390693 as the rpc queue, it is executing everything on > that one thread. > > In either case (single/multi-threaded escalation engine), it is only > exercising the threads associated with the escalation engine, not the > fast, list, callback, external auth, or custom queues. > > Axton Grams > > On 9/19/07, patrick zandi <[EMAIL PROTECTED]> wrote: > > ** > > Why not an afterhour escalation... instead.. > > Say every 10 minutes.. to do table queries or a report or two.. > > from 1800 - 0712 or something... > > > > > > On 9/19/07, Axton <[EMAIL PROTECTED]> wrote: > > > > > > "Actually, now that I re-read your post I don't think putting a > > > specific rule will side-step state checking." > > > > > > Depends on your firewall and the rule. Typically, states are created > > > using only SYN packets, if state can be created on other packet types, > > > you are still using stateful packet inspection, you are just allowing > > > different packet types to add the session to the state table. > > > > > > "We talked to BMC a few weeks ago and they told us "theoretically" > > > that it would be possible to write a custom API that would run custom > > > workflow (neither of which they could give us) that would hit all of > > > the server's Oracle connections at the same time often enough to > > > prevent anything from seeing them as idle." > > > > > > I was thinking this as I was reading your email, though I am not sure > > > how you would hit the admin and every fast/list/custom queue's threads > > > without occupying all of them simultaneously. The api, to my > > > knowledge, does not give you the capability to control what thread you > > > are using, which means that your api will have to be multi-threaded > > > and will have to occupy the max number of configured threads per rpc > > > queue, which will cause your remedy server to appear to hang (i.e., > > > block other operations on those queues). > > > > > > Can you share what type of firewall you are using? > > > > > > If you really want to remove the firewall from the equation, remove it > > > from the network, or completely disable it. I can't see that vlan > > > tagging would cause any issues with this. vlan's are configured in > > > one of two way's, on the switch per port or the tagging is handled by > > > the end nodes. If it is on the switch, it will be transparent to the > > > client. > > > > > > Axton Grams > > > > > > On 9/19/07, J.T. Shyman < [EMAIL PROTECTED]> wrote: > > > > ** > > > > > > > > > > > > > > > > Axton, > > > > > > > > > > > > > > > > Appreciate your input! > > > > > > > > > > > > > > > > I should have mentioned that we've been up and down that highway > > and > > > > > > > > haven't seen a blasted thing. (apologies to Glen Frey) > > > > > > > > > > > > > > > > What you are saying is exactly what I thought and we've disabled > > the > > > > > > > > idle timeout on the firewall. I know this may not be the same thing as > > > > > > > > preventing the firewall from using a state table but the firewall > admin > > > > > > > > tells us he now sees idle connections with idle times > 60 minutes. > So, > > > > > > > > we're kind of thinking we've eliminated the firewall as a > > > > > > > > cause...although we may not have, we aren't pursuing that any longer. > > > > > > > > > > > > > > > > Actually, now that I re-read your post I don't think putting a > > > > > > > > specific rule will side-step state checking. The purpose of a state > > table > > > > > > > > on a firewall is to speed up handling of traffic by allowing already > > known > > > > > > > > good traffic to pass without undergoing validation against the > rulebase > > > > > > > > for every packet. Adding a rule that allows a single port connection, > > > > > > > > which is what we had before, doesn't stop the state table from > > > > > > > > functioning. In fact, it may actually be what causes the connection to > > be > > > > > > > > put in the state table in the first place, no? Also, turning the > > firewall > > > > > > > > into, effectively, a packet-based firewall might have a detrimental > > affect > > > > > > > > on network throughput not only between AR and Oracle but for any other > > > > > > > > connections on that firewall due to increased overhead...or am I > wrong? > > > > > > > > > > > > > > > > Additionally, we the firewall admin put in ANY<->ANY rule in > place > > a > > > > > > > > few nights ago and the problem is still occurring. I'd hoped that this > > > > > > > > would circumvent the state table but it apparently doesn't. > > > > > > > > > > > > > > > > I don't suppose there is a AR-based solution? We could keep > trying > > > > > > > > changes on the network until we effectively distroy any semblance of > the > > > > > > > > original network design but that wouldn't mollify anyone. In fact, it > > may > > > > > > > > have the opposite affect. > > > > > > > > > > > > > > > > We talked to BMC a few weeks ago and they told us "theoretically" > > > > > > > > that it would be possible to write a custom API that would run custom > > > > > > > > workflow (neither of which they could give us) that would hit all of > the > > > > > > > > server's Oracle connections at the same time often enough to prevent > > > > > > > > anything from seeing them as idle. Does this sound like a good > approach > > to > > > > > > > > anyone? Any and all thoughts and comments are welcome! > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > J.T. > > > > > > > > > > > > > > > > __20060125_______________________This posting was > > > > submitted with HTML in it___ > > > > > > > > > ____________________________________________________________________________ > ___ > > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where > > the Answers Are" > > > > > > > > > > > -- > > Patrick Zandi __20060125_______________________This posting > > was submitted with HTML in it___ > > ____________________________________________________________________________ > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the > Answers Are" > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the > Answers Are" > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
