Great. Glad to hear that you're back in business. I've created a JIRA [1] for this issue just in case we resurrect the subclassing support to a first-class citizen. :-)
Thanks, Kevin [1] https://issues.apache.org/jira/browse/OPENJPA-1462 On Fri, Jan 15, 2010 at 10:15 AM, Kurt T Stam <[email protected]> wrote: > Thanks so much Kevin, that did the trick. Some of the issues you mentioned > where introduced by me not copying right from the > jUDDI project, but the main issue is that the openjpa-plugin simply does > not run for some reason. Anyway it is no longer leaking now! I learned a lot > from this exercise. And yes my class jumped from 2 to 10 k. BTW I'm pretty > sure this used to work.. > > Thanks again! Going to fix it the jUDDI build now :)! > > --Kurt > > Kevin Sutter wrote: > >> Hi Kurt, >> >> I have not run a profiler to verify the memory leak, but I think I found >> the >> problem... >> >> Although you thought you were running the PCEnhancer during build time, it >> was not being performed. When I looked at the Tomcat log with the >> original >> jar file, the following message was logged which indicates you were >> falling >> back to the "not ready for production" subclassing support: >> >> 1844 juddiDatabase INFO [http-8080-1] openjpa.Enhance - Creating >> subclass for "[class org.apache.juddi.AuthToken]". This means that your >> application will be less efficient and will consume more memory than it >> would if you ran the OpenJPA enhancer. Additionally, lazy loading will not >> be available for one-to-one and many-to-one persistent attributes in types >> using field access; they will be loaded eagerly instead. >> >> I made a few changes to your pom.xml and the "core" module to get the >> enhancement to work and I believe the memory leak has went away. Like I >> said, I did not run a profiler, but I added a "finalize" method to your >> AuthToken class and I now see that these entities are getting GC'd. >> >> One thing I would do is add the following property to your >> persistence.xml. >> >> <property name="openjpa.RuntimeUnenhancedClasses" value="warn"/> >> >> This property will produce the following warning if you accidentally fall >> into this subclassing support. We turn this off by default when running >> within WebSphere since we really don't want customers to accidentally use >> this subclassing. And, we have turned this off for 1.3.x and trunk due to >> the various problems associated with it. Unfortunately, the 1.2.x branch >> is >> kind of stuck with it... >> >> 1688 juddiDatabase WARN [http-8080-1] openjpa.Enhance - This >> configuration disallows runtime optimization, but the following listed >> types >> were not enhanced at build time or at class load time with a javaagent: >> "[class org.apache.juddi.AuthToken]". >> >> And, of course, your webservice no longer works either. You will get an >> HTTP Status 500 error with an error message and call stack that further >> explains the error. >> >> To get the maven plugin to work properly... I made several changes, so >> hopefully I remember everything... >> >> o The number one thing is that the openjpa:enhance goal doesn't seem to >> get >> automatically invoked when you compile. I have no idea why, but that >> doesn't seem to work. >> >> o You have to make your persistence.xml file available to the "core" >> module. For my testing, I just created a new directory in the "core" >> module >> called "\openjpa-leak\core\src\main\resources\META-INF\persistence.xml". >> This will automatically get copied over to "target" and then the >> openjpa:enhance goal will find it. >> >> o How you want to make your persistence.xml file available to both your >> "core" module and the "leak-war" module is up to you... :-) >> >> o I added a dependency in your core\pom.xml file for the 1.2.1 openjpa. >> The documentation [1] says it will default to 1.2.0, if none is specified. >> >> <dependencies> >> <dependency> >> <groupId>org.apache.openjpa</groupId> >> <artifactId>openjpa</artifactId> >> <version>1.2.1</version> >> </dependency> >> </dependencies> >> >> o Your plugin properties don't seem to be specified correctly, at least >> according to the documentation, but I didn't touch them and things still >> seemed to work (once I got around the other problems). >> >> o I also changed where to find the entities to enhance since this pom.xml >> still specified the "model" directory. There also seems to be confusion >> as >> to whether to specify <classes> or <includes>. I went with <includes> >> since >> that's what the example showed. >> >> o It's very easy to see if the enhancement process worked or not since >> the >> size of your entity classes increases by a few thousand bytes. And, with >> the extra property above, this will prevent you from accidentally using >> the >> unenhanced version. >> >> Hope this helps! Let me know if this gets you around the memory leak. >> >> Kevin >> >> [1] http://mojo.codehaus.org/openjpa-maven-plugin/usage.html >> >> >> >> On Thu, Jan 14, 2010 at 2:12 PM, Kurt T Stam <[email protected]> wrote: >> >> >> >>> 1. Cool that you got it running :) >>> >>> 2. I just ran with openjpa 1.2.2.SNAPSHOT and still see an accumulation >>> of >>> AuthToken objects. >>> >>> --K >>> >>> >>> Kevin Sutter wrote: >>> >>> >>> >>>> Hey, this wasn't so difficult... :-) Maybe I can still learn new >>>> things... I have Tomcat up and running and I can get your app running. >>>> >>>> On Thu, Jan 14, 2010 at 1:46 PM, Kevin Sutter <[email protected]> >>>> wrote: >>>> >>>> >>>> >>>> >>>> >>>>> Hi Kurt, >>>>> These instructions, of course, assume that I know something about >>>>> Tomcat. >>>>> This would not be a good assumption... :-) If you really want me to >>>>> see >>>>> this in action, I will need more information on how to start Tomcat and >>>>> how >>>>> to load applications. In the mean time, a trace log of OpenJPA would >>>>> be >>>>> an >>>>> excellent start. >>>>> >>>>> Thanks! >>>>> Kevin >>>>> >>>>> >>>>> On Thu, Jan 14, 2010 at 11:30 AM, Kurt T Stam <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Hi Kevin, >>>>>> >>>>>> I tried adding the javaagent stuff, and it still ends up leaking. I >>>>>> extracted the code into a tiny >>>>>> war with just one servlet which persist the authToken. The build >>>>>> creates >>>>>> a >>>>>> Tomcat distro backed by a >>>>>> derby database. You can download it from: >>>>>> >>>>>> http://people.apache.org/~kstam/leak.tgz<http://people.apache.org/%7Ekstam/leak.tgz> >>>>>> <http://people.apache.org/%7Ekstam/leak.tgz> >>>>>> <http://people.apache.org/%7Ekstam/leak.tgz> >>>>>> >>>>>> >>>>>> You can build it with: mvn install, you can find the final tomcat in >>>>>> tomcat/target/tomcat-1.0.SNAPSHOT.zip >>>>>> >>>>>> or if you don't want to build it: >>>>>> http://people.apache.org/~kstam/tomcat-1.0.SNAPSHOT.zip<http://people.apache.org/%7Ekstam/tomcat-1.0.SNAPSHOT.zip> >>>>>> <http://people.apache.org/%7Ekstam/tomcat-1.0.SNAPSHOT.zip> >>>>>> <http://people.apache.org/%7Ekstam/tomcat-1.0.SNAPSHOT.zip> >>>>>> >>>>>> >>>>>> After you bring up tomcat it persistent an authToken when you hit >>>>>> http://localhost:8080/leak/test >>>>>> >>>>>> In the profiler I see an org.apache.juddi.AuthToken left behind for >>>>>> each >>>>>> request. >>>>>> >>>>>> Hopefully this little example will pinpoint what's going on. >>>>>> >>>>>> Thx Kev for sticking with me :) >>>>>> >>>>>> --Kurt >>>>>> >>>>>> >>>>>> >>>>>> Kevin Sutter wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Hi Kurt, >>>>>>> Since you have the environment to reproduce this, could you run with >>>>>>> the >>>>>>> OpenJPA trace enabled? I'm not even sure what I'm looking for at >>>>>>> this >>>>>>> point. :-) But, maybe there's some tracing that would point us in >>>>>>> the >>>>>>> right direction. >>>>>>> >>>>>>> <property name="openjpa.Log" value="DefaultLevel=TRACE"/> >>>>>>> >>>>>>> We have been always been on the watch for memory leaks, so this one >>>>>>> is >>>>>>> puzzling. You did mention blocked threads as well. At this point, >>>>>>> you >>>>>>> are >>>>>>> probably hitting blocked threads due to the GC and Memory problems. >>>>>>> But, >>>>>>> we >>>>>>> did recently resolve some locking situations [1] in our >>>>>>> MetaDataRepository >>>>>>> that could affect scaling (under extreme load). This was resolved in >>>>>>> the >>>>>>> 1.2.x SNAPSHOT drivers (Mike is currently pushing for a 1.2.2 >>>>>>> release, >>>>>>> so >>>>>>> that should be coming soon). >>>>>>> >>>>>>> Does your WebService version of the problem use EntityManager >>>>>>> transactions, >>>>>>> or is the transaction service coming from Tomcat? And, does your >>>>>>> WebService >>>>>>> version also do the em.clear() and em.close()? Basically, are you >>>>>>> using >>>>>>> application-managed or container-managed persistence while running as >>>>>>> a >>>>>>> WebService? >>>>>>> >>>>>>> I'm very interested in helping to resolve this issue. Like I >>>>>>> mentioned, >>>>>>> we >>>>>>> do a lot of memory leakage and scalability testing of the OpenJPA >>>>>>> solution >>>>>>> and we're not aware of anything. Although there is always the >>>>>>> possibility >>>>>>> that something was resolved in the 1.2.x branch after the 1.2.1 >>>>>>> release >>>>>>> that >>>>>>> I'm not remembering... >>>>>>> >>>>>>> Thanks for your help, >>>>>>> Kevin >>>>>>> >>>>>>> [1] https://issues.apache.org/jira/browse/OPENJPA-250 >>>>>>> >>>>>>> On Wed, Jan 13, 2010 at 9:27 PM, Kurt T Stam <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Thanks Kevin, >>>>>>>> >>>>>>>> We're enhancing at build time: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> http://svn.apache.org/repos/asf/webservices/juddi/trunk/juddi-core/pom.xml >>>>>>>> >>>>>>>> Yeah we've been running load tests and things are nice and stable >>>>>>>> with >>>>>>>> Hibernate but with Openjpa we see increasing memory use, blocking >>>>>>>> threads >>>>>>>> and then an OOM. http://issues.apache.org/jira/browse/JUDDI-267. >>>>>>>> Our >>>>>>>> preference would be to ship with openjpa by default; but our build >>>>>>>> supports >>>>>>>> both hibernate and openjpa. >>>>>>>> >>>>>>>> And yes we use openjpa 1.2.1 (latest stable version). >>>>>>>> >>>>>>>> --Kurt >>>>>>>> >>>>>>>> >>>>>>>> Kevin Sutter wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Interesting detective work, Kurt. Thanks. >>>>>>>>> >>>>>>>>> Why the WebService version of the app would behave differently as >>>>>>>>> far >>>>>>>>> as >>>>>>>>> GC >>>>>>>>> is concerned is a mystery. And, you said that plugging in >>>>>>>>> Hibernate >>>>>>>>> into >>>>>>>>> this scenario, everything works okay? Very confusing. >>>>>>>>> >>>>>>>>> How are you performing the Entity enhancement processing? Are you >>>>>>>>> pre-enhancing via your build process? Or, are you using the >>>>>>>>> -javaagent >>>>>>>>> mechanism? Or, are you falling back to the subclassing support >>>>>>>>> within >>>>>>>>> OpenJPA? (See [1] for more information on these questions in case >>>>>>>>> they >>>>>>>>> don't make sense.) >>>>>>>>> >>>>>>>>> This would be one area that is different between Hibernate and >>>>>>>>> OpenJPA >>>>>>>>> -- >>>>>>>>> enhancement processing. >>>>>>>>> >>>>>>>>> In the Tomcat environment, you may be falling back to the >>>>>>>>> subclassing >>>>>>>>> support (which we do not recommend) and hitting a memory leak with >>>>>>>>> that. >>>>>>>>> >>>>>>>>> You said OpenJPA 1.2.x, right? >>>>>>>>> >>>>>>>>> Just a couple of thoughts on the subject... >>>>>>>>> Kevin >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> http://webspherepersistence.blogspot.com/2009/02/openjpa-enhancement.html >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jan 13, 2010 at 4:25 PM, Kurt T Stam <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> The same code executed straight from a java client (inVM) shows no >>>>>>>>>> memory >>>>>>>>>> leak. >>>>>>>>>> >>>>>>>>>> So is the fact that it is WebService significant then? What else >>>>>>>>>> can >>>>>>>>>> be >>>>>>>>>> different? I think one thread remains up, and somehow this causes >>>>>>>>>> openjpa >>>>>>>>>> not being able to clean up after itself. What can I do to debug >>>>>>>>>> this >>>>>>>>>> more? I >>>>>>>>>> can actually see in the profiler that the objects are allocated by >>>>>>>>>> the >>>>>>>>>> WebService, but why aren't they cleaned up? >>>>>>>>>> >>>>>>>>>> Thx, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --Kurt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Kurt T Stam wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks Kevin, thanks for your response. >>>>>>>>>>> >>>>>>>>>>> I just replaced the static call by: >>>>>>>>>>> >>>>>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>>>>> >>>>>>>>>>> apiAuthToken.setAuthInfo(modelAuthToken.getAuthToken()); >>>>>>>>>>> //MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>>>>> apiAuthToken); >>>>>>>>>>> >>>>>>>>>>> which did not make a difference. >>>>>>>>>>> >>>>>>>>>>> I'm wondering if the fact that my class is a webservice makes a >>>>>>>>>>> difference. I'll try extracting it into >>>>>>>>>>> a regular class with a main method and profile that. At least I >>>>>>>>>>> know >>>>>>>>>>> that >>>>>>>>>>> I didn't forget something >>>>>>>>>>> completely obvious.. >>>>>>>>>>> >>>>>>>>>>> --Kurt >>>>>>>>>>> >>>>>>>>>>> Kevin Sutter wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Kurt, >>>>>>>>>>>> I agree that this is very common usage of the JPA programming >>>>>>>>>>>> model. >>>>>>>>>>>> And, >>>>>>>>>>>> we are not aware of any memory leaks. About the only thing that >>>>>>>>>>>> jumps >>>>>>>>>>>> out >>>>>>>>>>>> at me is the following two lines: >>>>>>>>>>>> >>>>>>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>>>>>> MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>>>>>> apiAuthToken); >>>>>>>>>>>> >>>>>>>>>>>> What do these do? Can you comment these out and see if the >>>>>>>>>>>> memory >>>>>>>>>>>> leak >>>>>>>>>>>> still exists? Since you are passing the modelAuthToken into >>>>>>>>>>>> this >>>>>>>>>>>> method, >>>>>>>>>>>> I >>>>>>>>>>>> don't know what it's doing with the reference and could it be >>>>>>>>>>>> holding >>>>>>>>>>>> onto >>>>>>>>>>>> something to prevent the GC from cleaning up? >>>>>>>>>>>> >>>>>>>>>>>> The rest of your example seems very straight forward with >>>>>>>>>>>> creating >>>>>>>>>>>> and >>>>>>>>>>>> persisting objects. >>>>>>>>>>>> >>>>>>>>>>>> Kevin >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jan 13, 2010 at 2:09 PM, Rick Curtis < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> If you change the 1000 to something like 1000000... does your >>>>>>>>>>>>> application >>>>>>>>>>>>> go >>>>>>>>>>>>> OOM? Are you running in a JSE environment? What is >>>>>>>>>>>>> PersistenceManager? >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jan 13, 2010 at 2:05 PM, Kurt T Stam < >>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> BTW I'm running with the cache off >>>>>>>>>>>>>> >>>>>>>>>>>>>> <property name="openjpa.DataCache" value="false"/> >>>>>>>>>>>>>> >>>>>>>>>>>>>> (that turns it off right?) >>>>>>>>>>>>>> >>>>>>>>>>>>>> --Kurt >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kurt T Stam wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi guys, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [DESCRIPTION] The code below inserts a 1000 records in the >>>>>>>>>>>>>>> database. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> for (int i=1; i<1000; i++) { >>>>>>>>>>>>>>> EntityManager em = >>>>>>>>>>>>>>> PersistenceManager.getEntityManager(); >>>>>>>>>>>>>>> EntityTransaction tx = em.getTransaction(); >>>>>>>>>>>>>>> try { >>>>>>>>>>>>>>> tx.begin(); >>>>>>>>>>>>>>> // Generate auth token and store it! >>>>>>>>>>>>>>> String authInfo = AUTH_TOKEN_PREFIX + >>>>>>>>>>>>>>> UUID.randomUUID(); >>>>>>>>>>>>>>> org.apache.juddi.model.AuthToken modelAuthToken = >>>>>>>>>>>>>>> new >>>>>>>>>>>>>>> org.apache.juddi.model.AuthToken(); >>>>>>>>>>>>>>> if (authInfo != null) { >>>>>>>>>>>>>>> modelAuthToken.setAuthToken(authInfo); >>>>>>>>>>>>>>> modelAuthToken.setCreated(new Date()); >>>>>>>>>>>>>>> modelAuthToken.setLastUsed(new Date()); >>>>>>>>>>>>>>> modelAuthToken.setAuthorizedName(publisherId); >>>>>>>>>>>>>>> modelAuthToken.setNumberOfUses(0); >>>>>>>>>>>>>>> modelAuthToken.setTokenState(AUTHTOKEN_ACTIVE); >>>>>>>>>>>>>>> em.persist(modelAuthToken); >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>>>>>>>>> MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>>>>>>>>> apiAuthToken); >>>>>>>>>>>>>>> tx.commit(); >>>>>>>>>>>>>>> } finally { >>>>>>>>>>>>>>> if (tx.isActive()) { >>>>>>>>>>>>>>> tx.rollback(); >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> em.clear(); >>>>>>>>>>>>>>> em.close(); >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [ISSUE] >>>>>>>>>>>>>>> After it leaving this code I end up with a 1000 >>>>>>>>>>>>>>> org.apache.juddi.model.AuthToken objects in memory. I've been >>>>>>>>>>>>>>> using >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> profiler, and these objects cannot be garbage collected. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This seems to be pretty the most common use case of using an >>>>>>>>>>>>>>> OR-mapping >>>>>>>>>>>>>>> tool, so I find it hard to believe openjpa has a memory leak >>>>>>>>>>>>>>> here. >>>>>>>>>>>>>>> Does >>>>>>>>>>>>>>> anyone see what I'm doing wrong? Or can someone point me to >>>>>>>>>>>>>>> an >>>>>>>>>>>>>>> example >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> that >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> does not exhibit this behavior? BTW same code using hibernate >>>>>>>>>>>>>> does >>>>>>>>>>>>>> not >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> accumulate these objects. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We're using openjpa 1.2.1. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thx, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kurt >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Apache jUDDI. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Rick >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >> >> >> > >
