Unfortunately I don't have full control of the deployment environment... It looks like at some times the DB stops responding with no apparent reason. Could a simple DB Connection glitch cause a EOObjectStoreCoordinator lock? Maybe I can tweak the connection dictionary to enable auto reconnect. I'll try
Matteo On Mon, Jan 14, 2013 at 8:17 PM, Chuck Hill <[email protected]> wrote: > Hi Matteo, > > Something locked the EOObjectStoreCoordinator and did not unlock it. Does > the log for this instance show any exceptions? > > This exception could perhaps cause this problem: > >>> ava.io.EOFException >>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394) >>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929) >>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167) >>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278) >>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247) >>> at >>> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371) >>> at >>> com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265) >>> at >>> com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337) >>> at >>> com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296) >>> at >>> com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220) >>> at >>> com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897) >>> at >>> com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234) >>> at >>> com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055) >>> at >>> com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195) >>> at >>> com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488) >>> at >>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069) >>> at >>> er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305) >>> at >>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444) > > > Chuck > > > On 2013-01-14, at 10:05 AM, Matteo Centro wrote: > >> Hi all, suspecting there was some sort of deadlock I ran jstack >> against one of the immortal sessions instances, I'm attaching the >> output, anything helps... >> >> Thanks, >> >> >> Matteo >> >> On Mon, Jan 14, 2013 at 12:35 PM, Matteo Centro <[email protected]> >> wrote: >>> Sure, here it is: >>> Hi Chuck, >>> >>> I'm posting just to you, I can't have google to index this... >>> >>> anyway, here is the stack trace: >>> java.lang.IllegalMonitorStateException >>> at >>> java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239) >>> at >>> java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431) >>> at >>> com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:448) >>> at >>> com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684) >>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805) >>> at >>> com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281) >>> at >>> er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378) >>> at >>> com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614) >>> at >>> com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634) >>> at >>> com.tla.logic._RigaCarrello.clientiPerTicket(_RigaCarrello.java:275) >>> at com.tla.logic.RigaCarrello.setPosti(RigaCarrello.java:94) >>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:164) >>> at com.tla.calendar.Session.terminate(Session.java:81) >>> at >>> com.webobjects.appserver.WOSession._terminateByTimeout(WOSession.java:610) >>> at >>> com.webobjects.appserver.WOSessionStore$_SessionTimeoutManager.run(WOSessionStore.java:115) >>> at java.lang.Thread.run(Thread.java:662) >>> >>> Sorry for the italian Class Names and Methods, as I said I inherited that... >>> >>> The exception is caught, that printout is the printStackTrace of the >>> exception... >>> But anyway, I'm not sure that's the real problem, I found this morning >>> an instance with 295 sessions (still alive) and the only exception >>> logged was a weird >>> >>> er.transaction.adaptor.Exceptions - Database Exception occured: No >>> operations allowed after connection closed. >>> >>> Connection was closed due to the following exception: >>> >>> ** BEGIN NESTED EXCEPTION ** >>> >>> java.sql.SQLException >>> MESSAGE: Communication link failure: java.io.EOFException, underlying >>> cause: null >>> >>> ** BEGIN NESTED EXCEPTION ** >>> >>> java.io.EOFException >>> >>> STACKTRACE: >>> >>> java.io.EOFException >>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394) >>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929) >>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167) >>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278) >>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247) >>> at >>> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371) >>> at >>> com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265) >>> at >>> com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337) >>> at >>> com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296) >>> at >>> com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220) >>> at >>> com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897) >>> at >>> com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234) >>> at >>> com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055) >>> at >>> com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195) >>> at >>> com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488) >>> at >>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069) >>> at >>> er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305) >>> at >>> com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444) >>> at >>> com.tla.logic.Performance.fetchFreshPerformancesWithQualifier(Performance.java:414) >>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:154) >>> at >>> com.tla.calendar.components.ALTComponent.invokeAction(ALTComponent.java:167) >>> at >>> com.webobjects.appserver.WOSession.invokeAction(WOSession.java:1357) >>> at >>> com.webobjects.appserver.WOApplication.invokeAction(WOApplication.java:1745) >>> at >>> er.extensions.appserver.ajax.ERXAjaxApplication.invokeAction(ERXAjaxApplication.java:117) >>> at >>> er.extensions.appserver.ERXApplication.invokeAction(ERXApplication.java:2018) >>> at >>> er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedPage(ERXComponentRequestHandler.java:157) >>> at >>> er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedSession(ERXComponentRequestHandler.java:235) >>> at >>> er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedApplication(ERXComponentRequestHandler.java:268) >>> at >>> er.extensions.appserver.ERXComponentRequestHandler._handleRequest(ERXComponentRequestHandler.java:302) >>> at >>> er.extensions.appserver.ERXComponentRequestHandler.handleRequest(ERXComponentRequestHandler.java:377) >>> at >>> com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687) >>> at >>> er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2139) >>> at >>> er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2104) >>> at >>> com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144) >>> at >>> com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226) >>> at java.lang.Thread.run(Thread.java:662) >>> >>> So I'm not that confident that I am on the right track... >>> Looks like if anything goes wrong in the life of the instance sessions >>> won't die... Which is kind of disturbing. >>> >>> >>> Matteo >>> >>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <[email protected]> >>> wrote: >>>> Can you post the entire stack trace? >>>> >>>> At the very least, I would catch that exception and now allow it out of >>>> your finalize method. >>> >>> >>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <[email protected]> >>> wrote: >>>> Can you post the entire stack trace? >>>> >>>> At the very least, I would catch that exception and now allow it out of >>>> your finalize method. >>>> >>>> >>>> On 2013-01-12, at 2:01 PM, Matteo Centro wrote: >>>> >>>>> Thanks, I'll try that. >>>>> Anyway (I can't say it's that but this happens often in instances with >>>>> immortal sessions): >>>>> I get this in my terminate() method >>>>> java.lang.IllegalMonitorStateException >>>>> at >>>>> java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239) >>>>> at >>>>> java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431) >>>>> at >>>>> com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:460) >>>>> at >>>>> com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684) >>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805) >>>>> at >>>>> com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281) >>>>> at >>>>> er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378) >>>>> at >>>>> com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614) >>>>> at >>>>> com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634) >>>>> >>>>> I'm using ERXEC everywhere and I'm not locking explicitly, should I >>>>> lock the ec in the terminate method? >>>>> >>>>> >>>>> Matteo >>>>> >>>>> On Sat, Jan 12, 2013 at 8:39 PM, Simon <[email protected]> wrote: >>>>>> stick this in your session constructor, it will log out whenever a >>>>>> session is created. then you can start figuring out where you are >>>>>> creating a session outside the RR loop which is the normal culprit... >>>>>> >>>>>> StringWriter stringWriter = new StringWriter(); >>>>>> PrintWriter printWriter = new >>>>>> PrintWriter(stringWriter); >>>>>> (new Throwable()).printStackTrace(printWriter); >>>>>> String trace = stringWriter.toString(); >>>>>> log.debug("Session count = " + >>>>>> application().activeSessionsCount()); >>>>>> >>>>>> ClickEventLogger2.getLogger().fatal(ClickEventCode.E00129, "Session >>>>>> Created:\n\n " + trace, >>>>>> this.getClass().getSimpleName()); >>>>>> >>>>>> hmmm, you'll have to replace the "ClickEventLogger2" line because >>>>>> that's our logging mechanism. you could log.fatal it and set up an >>>>>> email appender in log4j. >>>>>> >>>>>> simon >>>>>> >>>>>> >>>>>> On 12 January 2013 17:43, Matteo Centro <[email protected]> wrote: >>>>>>> Sorry to resuscitate such an old discussion but I'm having the exact >>>>>>> same issue... >>>>>>> It's an old application that we "inherited", we wonderized it as much >>>>>>> as it's possible but something weird happens in production, sessions >>>>>>> on some instances simply won't die. >>>>>>> Some instances go out of memory and they hang there. >>>>>>> I'm in trouble and I needs some hints, both to fix the issue >>>>>>> temporarily and to fix it for good (of course in that case I assume >>>>>>> I'll have to rewrite the app, if only I could find the budget). >>>>>>> What are the most common causes of sessions not dying? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> >>>>>>> Matteo >>>>>>> >>>>>>> On Thu, Aug 21, 2008 at 5:35 AM, Joe Little <[email protected]> wrote: >>>>>>>> I had something similar with sessions going bonkers on a public WO >>>>>>>> page that our internal google search engine completely trashed. In the >>>>>>>> end, robots.txt and explicit rules to deny certain patterns were added >>>>>>>> to prevent this. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 20, 2008 at 8:17 PM, D Tim Cummings <[email protected]> >>>>>>>> wrote: >>>>>>>>> We have a couple of sessionless apps that have started showing this >>>>>>>>> problem >>>>>>>>> with sessions that don't terminate. It turned out the sessions were >>>>>>>>> being >>>>>>>>> created by malformed urls coming from malicious robot web crawlers. >>>>>>>>> The >>>>>>>>> urls were of the form >>>>>>>>> http://www.courses.qut.edu.au/cgi-bin/WebObjects/Courses.woa/wa/cgi-bin/WebObjects/Courses.woa >>>>>>>>> Maybe see if you are getting incorrect links to your sessionless >>>>>>>>> login page. >>>>>>>>> We solved the problem by catching unknown direct actions in >>>>>>>>> DirectAction.java >>>>>>>>> @Override >>>>>>>>> public WOActionResults performActionNamed(String actionName) { >>>>>>>>> try { >>>>>>>>> return super.performActionNamed(actionName); >>>>>>>>> } catch (NSForwardException nsfe) { >>>>>>>>> log.info("ns forward exception - prbalby no such method for " + >>>>>>>>> actionName); >>>>>>>>> } >>>>>>>>> return defaultAction(); >>>>>>>>> } >>>>>>>>> and in Application.java directing exceptions back to the Main page >>>>>>>>> (for URLs >>>>>>>>> with more than one / after wa). >>>>>>>>> @Override >>>>>>>>> public WOComponent pageWithName(String namePage, WOContext context) { >>>>>>>>> if ( "WOExceptionPage".equals(namePage) ) { >>>>>>>>> namePage = "Main"; >>>>>>>>> } >>>>>>>>> if ( "WOSessionRestorationError".equals(namePage) ) { >>>>>>>>> namePage = "Main"; >>>>>>>>> } >>>>>>>>> return super.pageWithName(namePage, context); >>>>>>>>> } >>>>>>>>> >>>>>>>>> and in Main.java >>>>>>>>> public void setException ( Exception e ) { >>>>>>>>> log.error("an exception occurred " + e); >>>>>>>>> } >>>>>>>>> >>>>>>>>> We are running apps with embedded Wonder 4 and WebObjects 5.3.3 on >>>>>>>>> Mac OS X >>>>>>>>> Server 10.5.4 with WebObjects 5.4.2 deployment. We didn't have the >>>>>>>>> problem >>>>>>>>> before we went to this setup, but maybe we weren't getting hit with >>>>>>>>> the same >>>>>>>>> url format then. >>>>>>>>> Tim >>>>>>>>> On 21/08/2008, at 4:02 AM, Chuck Hill wrote: >>>>>>>>> >>>>>>>>> On Aug 20, 2008, at 9:54 AM, Simon McLean wrote: >>>>>>>>> >>>>>>>>> Hi All - >>>>>>>>> Wondering if someone can throw me some ideas on solving what looks >>>>>>>>> like an >>>>>>>>> immortal session problem. >>>>>>>>> - Start up 1 instance in java monitor. >>>>>>>>> - User lands on sessionless login page. No sessions. >>>>>>>>> - User logs in. 1 session. >>>>>>>>> - User logs out. 0 sessions. >>>>>>>>> - User logs in. 1 session. >>>>>>>>> - User does nothing. Session times out. 0 sessions. >>>>>>>>> All looks good. However, we get to the end of the day and the >>>>>>>>> instance has >>>>>>>>> 376 active sessions. But I know for this particular app there are >>>>>>>>> only 6 >>>>>>>>> users, because they are all sitting next door :-) Also, If i now >>>>>>>>> leave this >>>>>>>>> instance overnight none of those active sessions will go away. >>>>>>>>> So what could be going on ? Something appears to be creating immortal >>>>>>>>> sessions, but forced session termination (by the user logging out) and >>>>>>>>> session expiration seem to be behaving themselves. >>>>>>>>> >>>>>>>>> If the users don't notice any problems, my money would be on an >>>>>>>>> exception >>>>>>>>> thrown from your Session.terminate(). Also check the sleep() method >>>>>>>>> to >>>>>>>>> ensure it can never throw. >>>>>>>>> I have seen a related problem where two requests come in for the same >>>>>>>>> session (user double clicks, Ajax etc), and the first calls >>>>>>>>> terminate() on >>>>>>>>> the session. The second thread is stuck and the app can't gracefully >>>>>>>>> shut >>>>>>>>> down. IIRC, you would see zero sessions in this case. The "fix" for >>>>>>>>> this >>>>>>>>> is to not call terminate() and instead set the session timeout to a >>>>>>>>> small >>>>>>>>> number of seconds. >>>>>>>>> Chuck >>>>>>>>> -- >>>>>>>>> Chuck Hill Senior Consultant / VP Development >>>>>>>>> Practical WebObjects - for developers who want to increase their >>>>>>>>> overall >>>>>>>>> knowledge of WebObjects or who are trying to solve specific problems. >>>>>>>>> http://www.global-village.net/products/practical_webobjects >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>>>> http://lists.apple.com/mailman/options/webobjects-dev/timcu%40tpg.com.au >>>>>>>>> This email sent to [email protected] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>>>> http://lists.apple.com/mailman/options/webobjects-dev/jmlittle%40gmail.com >>>>>>>>> >>>>>>>>> This email sent to [email protected] >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>>> http://lists.apple.com/mailman/options/webobjects-dev/wolists%40matteocentro.it >>>>>>>> >>>>>>>> This email sent to [email protected] >>>>>>> _______________________________________________ >>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>> Webobjects-dev mailing list ([email protected]) >>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>> https://lists.apple.com/mailman/options/webobjects-dev/simon%40potwells.co.uk >>>>>>> >>>>>>> This email sent to [email protected] >>>>> _______________________________________________ >>>>> Do not post admin requests to the list. They will be ignored. >>>>> Webobjects-dev mailing list ([email protected]) >>>>> Help/Unsubscribe/Update your Subscription: >>>>> https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net >>>>> >>>>> This email sent to [email protected] >>>> >>>> -- >>>> Chuck Hill Senior Consultant / VP Development >>>> >>>> Practical WebObjects - for developers who want to increase their overall >>>> knowledge of WebObjects or who are trying to solve specific problems. >>>> http://www.global-village.net/gvc/practical_webobjects >>>> >>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest >>>> Growing Companies in B.C! >>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of >>>> Canada’s Fastest-Growing Companies by PROFIT Magazine! >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >> <jstackoutput.txt> > > -- > Chuck Hill Senior Consultant / VP Development > > Practical WebObjects - for developers who want to increase their overall > knowledge of WebObjects or who are trying to solve specific problems. > http://www.global-village.net/gvc/practical_webobjects > > Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing > Companies in B.C! > Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of > Canada’s Fastest-Growing Companies by PROFIT Magazine! > > > > > > > > _______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list ([email protected]) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to [email protected]
