Regarding the deployment environment (which is not under my control, as I said before) There are 6 boxes: 2 webservers (one each for 2 different apps) 2 WO nodes with wonder's latest wotaskd 1 Mysql DB 1 JavaMonitor box
I noticed that the box that runs JavaMonitor doesn't have wotaskd running, is this normal? Matteo On Wed, Jan 30, 2013 at 5:56 PM, Matteo Centro <[email protected]> wrote: > From what I see some instances don't log the DB Connection error and > still have immortal sessions. > The main problems with immortal sessions is that eventually they end > up with a memory problem, I recall there is a way to make the instance > restart if memory errors happen but I can't remember how. > > I'm still in dire straits, no idea how to fix this... no idea what's > happening either. Any suggestions? > > Matteo > > On Tue, Jan 15, 2013 at 6:14 PM, Chuck Hill <[email protected]> wrote: >> >> On 2013-01-15, at 3:20 AM, Matteo Centro wrote: >> >>> Unfortunately I don't have full control of the deployment >>> environment... It looks like at some times the DB stops responding >>> with no apparent reason. >>> Could a simple DB Connection glitch cause a EOObjectStoreCoordinator lock? >> >> I recall a bug at some level related to this. I could be wrong. >> >> >>> Maybe I can tweak the connection dictionary to enable auto reconnect. >>> I'll try >> >> Chuck >> >> >>> On Mon, Jan 14, 2013 at 8:17 PM, Chuck Hill <[email protected]> >>> wrote: >>>> Hi Matteo, >>>> >>>> Something locked the EOObjectStoreCoordinator and did not unlock it. Does >>>> the log for this instance show any exceptions? >>>> >>>> This exception could perhaps cause this problem: >>>> >>>>>> ava.io.EOFException >>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394) >>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538) >>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929) >>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167) >>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278) >>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247) >>>>>> at >>>>>> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371) >>>>>> at >>>>>> com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265) >>>>>> at >>>>>> com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337) >>>>>> at >>>>>> com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296) >>>>>> at >>>>>> com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220) >>>>>> at >>>>>> com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897) >>>>>> at >>>>>> com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234) >>>>>> at >>>>>> com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055) >>>>>> at >>>>>> com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195) >>>>>> at >>>>>> com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488) >>>>>> at >>>>>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069) >>>>>> at >>>>>> er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305) >>>>>> at >>>>>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444) >>>> >>>> >>>> Chuck >>>> >>>> >>>> On 2013-01-14, at 10:05 AM, Matteo Centro wrote: >>>> >>>>> Hi all, suspecting there was some sort of deadlock I ran jstack >>>>> against one of the immortal sessions instances, I'm attaching the >>>>> output, anything helps... >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> Matteo >>>>> >>>>> On Mon, Jan 14, 2013 at 12:35 PM, Matteo Centro <[email protected]> >>>>> wrote: >>>>>> Sure, here it is: >>>>>> Hi Chuck, >>>>>> >>>>>> I'm posting just to you, I can't have google to index this... >>>>>> >>>>>> anyway, here is the stack trace: >>>>>> java.lang.IllegalMonitorStateException >>>>>> at >>>>>> java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127) >>>>>> at >>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239) >>>>>> at >>>>>> java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431) >>>>>> at >>>>>> com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:448) >>>>>> at >>>>>> com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684) >>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805) >>>>>> at >>>>>> com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281) >>>>>> at >>>>>> er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378) >>>>>> at >>>>>> com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614) >>>>>> at >>>>>> com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634) >>>>>> at >>>>>> com.tla.logic._RigaCarrello.clientiPerTicket(_RigaCarrello.java:275) >>>>>> at com.tla.logic.RigaCarrello.setPosti(RigaCarrello.java:94) >>>>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:164) >>>>>> at com.tla.calendar.Session.terminate(Session.java:81) >>>>>> at >>>>>> com.webobjects.appserver.WOSession._terminateByTimeout(WOSession.java:610) >>>>>> at >>>>>> com.webobjects.appserver.WOSessionStore$_SessionTimeoutManager.run(WOSessionStore.java:115) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> Sorry for the italian Class Names and Methods, as I said I inherited >>>>>> that... >>>>>> >>>>>> The exception is caught, that printout is the printStackTrace of the >>>>>> exception... >>>>>> But anyway, I'm not sure that's the real problem, I found this morning >>>>>> an instance with 295 sessions (still alive) and the only exception >>>>>> logged was a weird >>>>>> >>>>>> er.transaction.adaptor.Exceptions - Database Exception occured: No >>>>>> operations allowed after connection closed. >>>>>> >>>>>> Connection was closed due to the following exception: >>>>>> >>>>>> ** BEGIN NESTED EXCEPTION ** >>>>>> >>>>>> java.sql.SQLException >>>>>> MESSAGE: Communication link failure: java.io.EOFException, underlying >>>>>> cause: null >>>>>> >>>>>> ** BEGIN NESTED EXCEPTION ** >>>>>> >>>>>> java.io.EOFException >>>>>> >>>>>> STACKTRACE: >>>>>> >>>>>> java.io.EOFException >>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394) >>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538) >>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929) >>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167) >>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278) >>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247) >>>>>> at >>>>>> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371) >>>>>> at >>>>>> com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265) >>>>>> at >>>>>> com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337) >>>>>> at >>>>>> com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296) >>>>>> at >>>>>> com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220) >>>>>> at >>>>>> com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897) >>>>>> at >>>>>> com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234) >>>>>> at >>>>>> com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055) >>>>>> at >>>>>> com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195) >>>>>> at >>>>>> com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488) >>>>>> at >>>>>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069) >>>>>> at >>>>>> er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305) >>>>>> at >>>>>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444) >>>>>> at >>>>>> com.tla.logic.Performance.fetchFreshPerformancesWithQualifier(Performance.java:414) >>>>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:154) >>>>>> at >>>>>> com.tla.calendar.components.ALTComponent.invokeAction(ALTComponent.java:167) >>>>>> at >>>>>> com.webobjects.appserver.WOSession.invokeAction(WOSession.java:1357) >>>>>> at >>>>>> com.webobjects.appserver.WOApplication.invokeAction(WOApplication.java:1745) >>>>>> at >>>>>> er.extensions.appserver.ajax.ERXAjaxApplication.invokeAction(ERXAjaxApplication.java:117) >>>>>> at >>>>>> er.extensions.appserver.ERXApplication.invokeAction(ERXApplication.java:2018) >>>>>> at >>>>>> er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedPage(ERXComponentRequestHandler.java:157) >>>>>> at >>>>>> er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedSession(ERXComponentRequestHandler.java:235) >>>>>> at >>>>>> er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedApplication(ERXComponentRequestHandler.java:268) >>>>>> at >>>>>> er.extensions.appserver.ERXComponentRequestHandler._handleRequest(ERXComponentRequestHandler.java:302) >>>>>> at >>>>>> er.extensions.appserver.ERXComponentRequestHandler.handleRequest(ERXComponentRequestHandler.java:377) >>>>>> at >>>>>> com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687) >>>>>> at >>>>>> er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2139) >>>>>> at >>>>>> er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2104) >>>>>> at >>>>>> com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144) >>>>>> at >>>>>> com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> So I'm not that confident that I am on the right track... >>>>>> Looks like if anything goes wrong in the life of the instance sessions >>>>>> won't die... Which is kind of disturbing. >>>>>> >>>>>> >>>>>> Matteo >>>>>> >>>>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <[email protected]> >>>>>> wrote: >>>>>>> Can you post the entire stack trace? >>>>>>> >>>>>>> At the very least, I would catch that exception and now allow it out of >>>>>>> your finalize method. >>>>>> >>>>>> >>>>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <[email protected]> >>>>>> wrote: >>>>>>> Can you post the entire stack trace? >>>>>>> >>>>>>> At the very least, I would catch that exception and now allow it out of >>>>>>> your finalize method. >>>>>>> >>>>>>> >>>>>>> On 2013-01-12, at 2:01 PM, Matteo Centro wrote: >>>>>>> >>>>>>>> Thanks, I'll try that. >>>>>>>> Anyway (I can't say it's that but this happens often in instances with >>>>>>>> immortal sessions): >>>>>>>> I get this in my terminate() method >>>>>>>> java.lang.IllegalMonitorStateException >>>>>>>> at >>>>>>>> java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127) >>>>>>>> at >>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239) >>>>>>>> at >>>>>>>> java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431) >>>>>>>> at >>>>>>>> com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:460) >>>>>>>> at >>>>>>>> com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684) >>>>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805) >>>>>>>> at >>>>>>>> com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281) >>>>>>>> at >>>>>>>> er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378) >>>>>>>> at >>>>>>>> com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614) >>>>>>>> at >>>>>>>> com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634) >>>>>>>> >>>>>>>> I'm using ERXEC everywhere and I'm not locking explicitly, should I >>>>>>>> lock the ec in the terminate method? >>>>>>>> >>>>>>>> >>>>>>>> Matteo >>>>>>>> >>>>>>>> On Sat, Jan 12, 2013 at 8:39 PM, Simon <[email protected]> wrote: >>>>>>>>> stick this in your session constructor, it will log out whenever a >>>>>>>>> session is created. then you can start figuring out where you are >>>>>>>>> creating a session outside the RR loop which is the normal culprit... >>>>>>>>> >>>>>>>>> StringWriter stringWriter = new StringWriter(); >>>>>>>>> PrintWriter printWriter = new >>>>>>>>> PrintWriter(stringWriter); >>>>>>>>> (new Throwable()).printStackTrace(printWriter); >>>>>>>>> String trace = stringWriter.toString(); >>>>>>>>> log.debug("Session count = " + >>>>>>>>> application().activeSessionsCount()); >>>>>>>>> >>>>>>>>> ClickEventLogger2.getLogger().fatal(ClickEventCode.E00129, "Session >>>>>>>>> Created:\n\n " + trace, >>>>>>>>> this.getClass().getSimpleName()); >>>>>>>>> >>>>>>>>> hmmm, you'll have to replace the "ClickEventLogger2" line because >>>>>>>>> that's our logging mechanism. you could log.fatal it and set up an >>>>>>>>> email appender in log4j. >>>>>>>>> >>>>>>>>> simon >>>>>>>>> >>>>>>>>> >>>>>>>>> On 12 January 2013 17:43, Matteo Centro <[email protected]> >>>>>>>>> wrote: >>>>>>>>>> Sorry to resuscitate such an old discussion but I'm having the exact >>>>>>>>>> same issue... >>>>>>>>>> It's an old application that we "inherited", we wonderized it as much >>>>>>>>>> as it's possible but something weird happens in production, sessions >>>>>>>>>> on some instances simply won't die. >>>>>>>>>> Some instances go out of memory and they hang there. >>>>>>>>>> I'm in trouble and I needs some hints, both to fix the issue >>>>>>>>>> temporarily and to fix it for good (of course in that case I assume >>>>>>>>>> I'll have to rewrite the app, if only I could find the budget). >>>>>>>>>> What are the most common causes of sessions not dying? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Matteo >>>>>>>>>> >>>>>>>>>> On Thu, Aug 21, 2008 at 5:35 AM, Joe Little <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> I had something similar with sessions going bonkers on a public WO >>>>>>>>>>> page that our internal google search engine completely trashed. In >>>>>>>>>>> the >>>>>>>>>>> end, robots.txt and explicit rules to deny certain patterns were >>>>>>>>>>> added >>>>>>>>>>> to prevent this. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 20, 2008 at 8:17 PM, D Tim Cummings <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>>> We have a couple of sessionless apps that have started showing >>>>>>>>>>>> this problem >>>>>>>>>>>> with sessions that don't terminate. It turned out the sessions >>>>>>>>>>>> were being >>>>>>>>>>>> created by malformed urls coming from malicious robot web >>>>>>>>>>>> crawlers. The >>>>>>>>>>>> urls were of the form >>>>>>>>>>>> http://www.courses.qut.edu.au/cgi-bin/WebObjects/Courses.woa/wa/cgi-bin/WebObjects/Courses.woa >>>>>>>>>>>> Maybe see if you are getting incorrect links to your sessionless >>>>>>>>>>>> login page. >>>>>>>>>>>> We solved the problem by catching unknown direct actions in >>>>>>>>>>>> DirectAction.java >>>>>>>>>>>> @Override >>>>>>>>>>>> public WOActionResults performActionNamed(String actionName) { >>>>>>>>>>>> try { >>>>>>>>>>>> return super.performActionNamed(actionName); >>>>>>>>>>>> } catch (NSForwardException nsfe) { >>>>>>>>>>>> log.info("ns forward exception - prbalby no such method for " + >>>>>>>>>>>> actionName); >>>>>>>>>>>> } >>>>>>>>>>>> return defaultAction(); >>>>>>>>>>>> } >>>>>>>>>>>> and in Application.java directing exceptions back to the Main page >>>>>>>>>>>> (for URLs >>>>>>>>>>>> with more than one / after wa). >>>>>>>>>>>> @Override >>>>>>>>>>>> public WOComponent pageWithName(String namePage, WOContext >>>>>>>>>>>> context) { >>>>>>>>>>>> if ( "WOExceptionPage".equals(namePage) ) { >>>>>>>>>>>> namePage = "Main"; >>>>>>>>>>>> } >>>>>>>>>>>> if ( "WOSessionRestorationError".equals(namePage) ) { >>>>>>>>>>>> namePage = "Main"; >>>>>>>>>>>> } >>>>>>>>>>>> return super.pageWithName(namePage, context); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> and in Main.java >>>>>>>>>>>> public void setException ( Exception e ) { >>>>>>>>>>>> log.error("an exception occurred " + e); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> We are running apps with embedded Wonder 4 and WebObjects 5.3.3 on >>>>>>>>>>>> Mac OS X >>>>>>>>>>>> Server 10.5.4 with WebObjects 5.4.2 deployment. We didn't have >>>>>>>>>>>> the problem >>>>>>>>>>>> before we went to this setup, but maybe we weren't getting hit >>>>>>>>>>>> with the same >>>>>>>>>>>> url format then. >>>>>>>>>>>> Tim >>>>>>>>>>>> On 21/08/2008, at 4:02 AM, Chuck Hill wrote: >>>>>>>>>>>> >>>>>>>>>>>> On Aug 20, 2008, at 9:54 AM, Simon McLean wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi All - >>>>>>>>>>>> Wondering if someone can throw me some ideas on solving what looks >>>>>>>>>>>> like an >>>>>>>>>>>> immortal session problem. >>>>>>>>>>>> - Start up 1 instance in java monitor. >>>>>>>>>>>> - User lands on sessionless login page. No sessions. >>>>>>>>>>>> - User logs in. 1 session. >>>>>>>>>>>> - User logs out. 0 sessions. >>>>>>>>>>>> - User logs in. 1 session. >>>>>>>>>>>> - User does nothing. Session times out. 0 sessions. >>>>>>>>>>>> All looks good. However, we get to the end of the day and the >>>>>>>>>>>> instance has >>>>>>>>>>>> 376 active sessions. But I know for this particular app there are >>>>>>>>>>>> only 6 >>>>>>>>>>>> users, because they are all sitting next door :-) Also, If i now >>>>>>>>>>>> leave this >>>>>>>>>>>> instance overnight none of those active sessions will go away. >>>>>>>>>>>> So what could be going on ? Something appears to be creating >>>>>>>>>>>> immortal >>>>>>>>>>>> sessions, but forced session termination (by the user logging out) >>>>>>>>>>>> and >>>>>>>>>>>> session expiration seem to be behaving themselves. >>>>>>>>>>>> >>>>>>>>>>>> If the users don't notice any problems, my money would be on an >>>>>>>>>>>> exception >>>>>>>>>>>> thrown from your Session.terminate(). Also check the sleep() >>>>>>>>>>>> method to >>>>>>>>>>>> ensure it can never throw. >>>>>>>>>>>> I have seen a related problem where two requests come in for the >>>>>>>>>>>> same >>>>>>>>>>>> session (user double clicks, Ajax etc), and the first calls >>>>>>>>>>>> terminate() on >>>>>>>>>>>> the session. The second thread is stuck and the app can't >>>>>>>>>>>> gracefully shut >>>>>>>>>>>> down. IIRC, you would see zero sessions in this case. The "fix" >>>>>>>>>>>> for this >>>>>>>>>>>> is to not call terminate() and instead set the session timeout to >>>>>>>>>>>> a small >>>>>>>>>>>> number of seconds. >>>>>>>>>>>> Chuck >>>>>>>>>>>> -- >>>>>>>>>>>> Chuck Hill Senior Consultant / VP Development >>>>>>>>>>>> Practical WebObjects - for developers who want to increase their >>>>>>>>>>>> overall >>>>>>>>>>>> knowledge of WebObjects or who are trying to solve specific >>>>>>>>>>>> problems. >>>>>>>>>>>> http://www.global-village.net/products/practical_webobjects >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>>>>>>> http://lists.apple.com/mailman/options/webobjects-dev/timcu%40tpg.com.au >>>>>>>>>>>> This email sent to [email protected] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>>>>>>> http://lists.apple.com/mailman/options/webobjects-dev/jmlittle%40gmail.com >>>>>>>>>>>> >>>>>>>>>>>> This email sent to [email protected] >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>>>>>> http://lists.apple.com/mailman/options/webobjects-dev/wolists%40matteocentro.it >>>>>>>>>>> >>>>>>>>>>> This email sent to [email protected] >>>>>>>>>> _______________________________________________ >>>>>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>>>>> https://lists.apple.com/mailman/options/webobjects-dev/simon%40potwells.co.uk >>>>>>>>>> >>>>>>>>>> This email sent to [email protected] >>>>>>>> _______________________________________________ >>>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>>> https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net >>>>>>>> >>>>>>>> This email sent to [email protected] >>>>>>> >>>>>>> -- >>>>>>> Chuck Hill Senior Consultant / VP Development >>>>>>> >>>>>>> Practical WebObjects - for developers who want to increase their >>>>>>> overall knowledge of WebObjects or who are trying to solve specific >>>>>>> problems. >>>>>>> http://www.global-village.net/gvc/practical_webobjects >>>>>>> >>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest >>>>>>> Growing Companies in B.C! >>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking >>>>>>> of Canada’s Fastest-Growing Companies by PROFIT Magazine! >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> <jstackoutput.txt> >>>> >>>> -- >>>> Chuck Hill Senior Consultant / VP Development >>>> >>>> Practical WebObjects - for developers who want to increase their overall >>>> knowledge of WebObjects or who are trying to solve specific problems. >>>> http://www.global-village.net/gvc/practical_webobjects >>>> >>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest >>>> Growing Companies in B.C! >>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of >>>> Canada’s Fastest-Growing Companies by PROFIT Magazine! >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >> >> -- >> Chuck Hill Senior Consultant / VP Development >> >> Practical WebObjects - for developers who want to increase their overall >> knowledge of WebObjects or who are trying to solve specific problems. >> http://www.global-village.net/gvc/practical_webobjects >> >> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest >> Growing Companies in B.C! >> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of >> Canada’s Fastest-Growing Companies by PROFIT Magazine! >> >> >> >> >> >> >> >> _______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list ([email protected]) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to [email protected]
