Be aware that (mysterious) things in ITSM7 trigger recaches -- not just
definition imports or changes. I haven't quite figured out what -- but we
have one or two a day -- no admin involved. They show as Remedy
Application Service user. Some can be traced to group changes, which we
now perform after-hours. Others are a mystery.
I don't think there are db read differences when updating the cache - just
the server handling of the in-memory copy. If you watch the size of the
server when doing changes on vs. off you should see the differences.
What happens with the cache in the other servers in the server group?
I think this is dealt with in the server_cache, servgrp_board and
servgrp_cache and related servgrp_* tables. No server group here so I
can't say for sure. My thinking is the server watches for a signal to
trigger a recache. That signal is is an API call or a change in the
meta-data. Can anyone confirm how it really works?
Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores
N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262)
703-7763 | e-mail: tony.worthing...@kohls.com
From:
Misi Mladoniczky <m...@rrr.se>
To:
arslist@ARSLIST.ORG
Date:
11/20/2009 01:13 PM
Subject:
Re: Development Cache Mode revisited
Sent by:
"Action Request System discussion list(ARSList)" <arslist@ARSLIST.ORG>
Lyle and Tony,
Thank you for the reply.
In other words, there should be no impact whatsoever if you perform NO
admin-changes.
If you apply your changes when both users, email engine and escalations
are taking it easy, it would probably be faster to use Developer Cache
Mode even in production (Cache-Mode: 1).
Two more questions:
1. I turned on SQL-logs to see what happened when disabling an escalation
with both settings. Nothing much happened, and the server did not seem to
reread much from the database repository. I had expected a bunch of
SQL-selects from the repository. Maybe it just updated the memory cache
without rereading the DB?
2. What happens with the cache in the other servers in the server group?
Is the admin-change propagated to these in some magic way?
Best Regards - Misi, RRR AB, http://rrr.se
> I just worked through a longstanding issue that we have had in our
> environment related to Dev Cache Mode. Due to extremely slow
performance
> when making admin changes on a 7.0 server, I had gotten into the habit
of
> keeping our admin server set with Dev Cache Mode turned on, which
> significantly sped up migrations to production and such. However, in
this
> environment (7.1 p5 server), I was seeing exactly the opposite - admin
> changes were very frequently timing out and migrations were painfully
slow
> (for example, a patch that took a few minutes to apply in dev and stage
> took an hour and twenty minutes to apply in production). Thinking that
it
> should go faster since Dev Cache Mode was turned on, I looked everywhere
I
> could think of to try and isolate why it was so slow. I got BMC
involved
> in my search twice. This last time, the support rep forwarded this same
> KB article to me and recommended that I leave Dev Cache Mode turned OFF
in
> order to improve performance with admin actions. I tried it, and it
> worked. My migrations now go very quickly, and I haven't seen any
> timeouts since. The only slowness is in the initial period where the AR
> server creates a copy of the cache in memory, but that's generally
fairly
> quick, since it's just doing an in memory copy rather than trying to
pull
> it all from the database again (like I suspect it was doing on the 7.0
> server).
>
> In short, in our environment, we see significantly better performance
> making admin changes in production with Dev Cache Mode turned off. The
> support rep told me that since AR Server 7.1, their recommendation is to
> keep it turned off.
>
> That said, if I understand it correctly, your mileage may vary depending
> on what kind of activity you have going on on the server you're trying
to
> make changes on. As I understand it, when Dev Cache Mode is turned on,
> since it doesn't make a copy of the cache for existing users, it has to
> wait until user operations have ceased before it can make the change,
then
> it makes the change in between user operations. In our case, since this
> server also runs all our back-end processes (e-mail, escalations, etc.),
> and we apparently have some things going on that take a long time to
> complete, the admin thread ended up having to wait a long time (more
than
> 10 minutes in some cases) before it could get a lock on the cache and
make
> the changes. This was what was causing the timeouts and slow
performance
> that we saw. Now, since it simply makes a relatively quick copy of the
> cache for existing user operations (just system operations in our case,
> but still, they're ongoing), the overall time to make the changes is
> quick, because once the cache is copied, it can proceed with the changes
> without having to wait for a break between user operations.
>
> So, if you don't have much going on on the server you're trying to make
> admin changes on, it could be quicker to leave Dev Cache Mode turned on,
> because then you don't have to wait for the cache to get copied (I think
> it takes less than 30 seconds in our case to copy about 400MB of cache).
> However, if you have lots of stuff going on (e-mail, user access,
> escalations, etc.), you may see better performance with it turned off.
>
> I hope this helps a bit.
>
> Lyle
>
>
> From: Action Request System discussion list(ARSList)
> [mailto:arsl...@arslist.org] On Behalf Of Tony Worthington
> Sent: Friday, November 20, 2009 9:36 AM
> To: arslist@ARSLIST.ORG
> Subject: Re: Development Cache Mode revisited
>
> ** I missed your question... sorry.
>
> - No admin-changes were done
> - And No changes to Groups of the type View/Change were done
>
> This is from BMC KB KM-000000024487. It doesn't answer your ? directly,
> but is full of good information.
>
> What may cause poor performance in the Admin Tool?
>
> When poor performance is seen in the Admin Tool when saving objects, it
> may be a result of how AR Server handles information it has read into
> memory.
>
> At server startup, AR Server reads information from the database into
> memory, for example all field properties are read into memory, all users
> and groups and active links, etc.
>
> When this information that has been read into memory changes, there are
> two different ways for AR Server to handle the event.
>
> It can update the existing memory footprint so that users all see the
> change that was made.
> It can create a separate memory footprint so existing users still
utilize
> the existing memory footprint, while new users logging in would utilize
> the new memory footprint.
> Either method will cause a performance hit, but the big difference
between
> the two ways of handling the event is where the performance impact is
> seen.
>
> Note: How noticeable the performance hit is depends on several factors,
> but mostly the amount of data that needs to be read into memory. This is
> usually a direct result of the number of objects (forms and workflow)
that
> exist. The more objects, the greater the performance impact.
>
> Is there a way to improve performance or control which option AR Server
> uses?
>
> Yes. There is a server configuration, called Development Cache Mode,
that
> was introduced to help with performance when dealing with systems that
> have large amounts of forms and workflow objects in a development
> environment. The Cache-Mode setting is available in ARServer 6.0.1 patch
> 1339 and above, and in ARServer 5.1.2 patch 1338 and above as well as in
> 6.3 and above. Note: only the 6.3 and above Admin Tool shows the GUI
> checkbox for Development Cache Mode in the Server Information dialog on
> the Configuration tab. If you're using an Admin Tool that does not have
> the checkbox for Development Cache Mode, then the Cache-Mode parameter
> must be manually added to the ar.cfg file.
>
> It can be found in the Admin Tool under File->Server Information on the
> Configuration tab. It is a check box called "Development Cache Mode" and
> by default, it is not selected.
>
> When selected, it puts a parameter in the ar.cfg/ar.conf file called
> Cache-Mode: 1.
>
> There are two cache modes, production and development.
>
> In production mode (Cache-Mode: 0), which is the default behavior if the
> line is not set in ar.cfg/ar.conf file (i.e. the Development Cache Mode
is
> not checked in the Admin Tool), AR Server operates as it always has --
> whenever an admin operation will change the memory cache, the admin
thread
> makes a new copy of the memory cache and makes its changes there. While
> this slows performance for the admin making the change, it allows the
> change to be made without affecting other users of the system so it's
> appropriate for a production environment with many users.
>
> In development mode (Cache-Mode: 1 in the ar.cfg/ar.conf file or where
> the Development Cache Mode is selected in the Admin Tool), the admin
> thread will instead lock other threads out of the current copy of the
> cache while it makes its change in that copy. On each API call that
will
> cause a change to the cache, the admin thread locks the cache, waits for
> all other threads to stop using the cache, makes its modifications, and
> unlocks the cache. The thread locks the entire cache, which means other
> threads cannot process API calls or escalations while the admin thread
> processes its API call. In a development environment this should be an
> acceptable cost.
>
> This effectively gives priority to the administrator at the expense of
> other users. It is therefore not really suitable for a production
> environment (blocking all users for the entire time that that the change
> is taking place is not good in production), but is great where
> development is the primary aim of the system.
>
> This mode (development mode) has a couple of notable effects --
>
> The first is that the server will no longer copy the cache, though it
will
> wait for user operations to complete before it proceeds with making a
> change. This will almost always be faster than copying the cache and
the
> difference in time will be greater with the number of objects on the
> server. In other words, the larger the application and number of
objects
> (i.e. forms, active links, etc.) on the server, the greater the effect
of
> the cache mode will be.
>
> The second is that the server's memory footprint will be smaller so that
> it is less likely to cause paging; this will again improve the server
> performance.
>
> Changes to the cache mode only take effect after AR Server has been
> restarted.
>
> The expectation is that in a development environment, admin changes have
> priority over other users. In a production environment the opposite is
> true.
>
> So, the key thing to understand when using this parameter is that we
lock
> the object when a change is applied (when we issue a Set, Delete, or
> Create API call), not when it is opened. This means that any number of
> admin tools may have an object open; if one of them changes the object
and
> a second one makes another change, the first is overwritten. That
> behavior is no different than without the change, since there is only
one
> admin thread and all admin operations are serialized.
>
> This should improve performance immensely when saving a form because you
> are only dealing with the current form and not all objects on the server
> being cached.
>
> Is there any risk to having Development Cache Mode enabled in
production?
>
> Yes. An issue was discovered when development cache mode is enabled that
> can cause AR Server to appear hung. This is thoroughly described in
> KM-000000025743. This is NOT a common issue. Normally, there would be no
> risk other than the described behavior of AR Server when Admin changes
> occur.
>
> ------ Here's 25743
>
> There is an important side-effect that can cause AR Server to appear
hung
> and users receive timeout errors, like ARERR 93.
> This occurs under specific circumstances, but specifically when the
> Development Cache Mode is selected in the Admin Tool under File->Server
> Information on the Configuration tab (see KB 24487 for information on
> Development Cache Mode).
>
> Here is the specific course of events that leads to the problem:
>
> An escalation starts running and puts a Read Lock on the Server Cache
> tables (in memory).
>
> An Admin Operation (like a form or workflow save) starts but cannot put
an
> Exclusive lock on the cache until the Escalation is done with its lock.
> The escalation continues to run.
>
> Users and utilities try to perform ARServer work (APIs). But they need
a
> read lock on the cache and are forced to wait.
>
> The escalation keeps running.
>
> Users start getting timeouts (ARERR 93, etc.) and the Admin Tool times
> out.
>
> The escalation keeps running.
>
> Finally the escalation completes and the Admin Operation can start.
>
> Admin Operation completes and users work can start. Since the users
> already timed out, they've likely given up and moved into trouble
shooting
> mode, unaware of whats happening with AR Server.
>
> Normally, escalations run pretty quick, but in some cases, this isn't
> true. For instance, lets say an escalation fires on a form with 200,000
> records in it, updating each one. Each record touched causes Filters
to
> fire which do Push Fields actions to several other forms for example,
> resulting in each of the 200,000 records to take roughly 18 seconds to
> complete. This escalation would take a very long time to complete and
all
> the while, AR Server would appear as though its hung because of the
> Development Cache mode being enabled. When Development Cache mode is
> enabled, it will attempt to put an exclusive lock on the server cache
> tables in memory. When disabled, a separate memory footprint is created
so
> there is no locking conflict with the cache and therefore this problem
> would not occur.
>
> This behavior is not considered a bug, but rather a circumstantial issue
> caused by the design of the product. Currently there is no workaround to
> this issue, but being aware of it means that there are options. For
> instance, normally Development Cache Mode is only enabled on a
development
> server. If this is the case, then usually you can disable escalations or
> temporarily disable development cache mode in development (if the
> escalation must run for testing, etc.). You could also alter the
> escalation(s) so it doesn't touch so many records or doesn't cause
filters
> to fire, etc. In other words, speed up the escalation processing time so
> any locks are very quick, or disable the escalation (or all
escalations),
> or disable development cache mode.
>
>
> NOTICE: This email message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information.
Any
> unauthorized review, use, disclosure or distribution is prohibited. If
> you are not the intended recipient, please contact the sender by reply
> email and destroy all copies of the original message.
>
>
>
>
_______________________________________________________________________________
> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
> Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers
Are"
>
> --
> This message was scanned by ESVA and is believed to be clean.
>
>
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
**********************************************************************
CONFIDENTIALITY NOTICE:
This is a transmission from Kohl's Department Stores, Inc.
and may contain information which is confidential and proprietary.
If you are not the addressee, any disclosure, copying or distribution or use of
the contents of this message is expressly prohibited.
If you have received this transmission in error, please destroy it and notify
us immediately at 262-703-7000.
CAUTION:
Internet and e-mail communications are Kohl's property and Kohl's reserves the
right to retrieve and read any message created, sent and received. Kohl's
reserves the right to monitor messages by authorized Kohl's Associates at any
time
without any further consent.
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
