One work around that I found is to create the read/write lock in ConcurrentBarrier in "fair" mode instead of the default mode. (fair mode changes the algorithm used to decide which thread gets the lock next)
That worked for me but subjectively things seemed a little slower (no actual measurement so take that for what it is worth) If it is fixed in 6u2 then I'd be tempted just to document the problem with 6u1 :) cheers Ben PS: I'll try to actually confirm that is it fixed in the next few days... may not get to it till the weekend tho > -----Original Message----- > From: Howard Lewis Ship [mailto:[EMAIL PROTECTED] > Sent: Wednesday, 11 July 2007 4:14 PM > To: Tapestry users > Subject: Re: T5: NPE in Base64InputStream and > locked/waitingCheckForUpdatesFilter > > Great find! People may need to deploy on JDK 1.5 to see if that's the > underlying cause. > > I wonder if we could create a work around by setting a wait time to > acquire the read lock? In a loop? > > On 7/10/07, Ben Sommerville <[EMAIL PROTECTED]> wrote: > > If you are running under jdk 6u1 and tapestry 5.0.5 (or > greater) then > > there is > > a jvm bug that can cause a deadlock. > > > > The bug report is at > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6571733. > > It is supposed to be fixed in jdk 6u2 (which was release > recently) but I > > haven't had a chance to test it yet. > > > > cheers > > Ben > > > > > -----Original Message----- > > > From: Martin Grotzke [mailto:[EMAIL PROTECTED] > > > Sent: Wednesday, 11 July 2007 9:06 AM > > > To: Tapestry users > > > Subject: Re: T5: NPE in Base64InputStream and > > > locked/waitingCheckForUpdatesFilter > > > > > > The NPE seems to be caused by a missing t:formdata > request parameter. > > > > > > I just "reproduced" this (weird) situation by removing > the hidden form > > > field "t:formdata" with firebug, so that I could produce the NPE. > > > > > > Although, following request went through well, and the number > > > of request > > > processing threads did not increase. > > > > > > So there seems to be no (direct) interrelationship between the NPE > > > and the locked threads. > > > > > > Cheers, > > > Martin > > > > > > > > > > > > On Tue, 2007-07-10 at 23:43 +0200, Martin Grotzke wrote: > > > > Hi, > > > > > > > > we had an issue with our deployed application that did > not respond > > > > anymore. This happened two or three times in the last 4 > days, but > > > > I was not able to reproduce it until now. > > > > > > > > The analysis of the logs showed, that there was a NPE in > > > > Base64InputStream, and afterwards the application did > not respond > > > > anymore. > > > > > > > > When I triggered a thread dump, all 200 tomcat threads were > > > in status > > > > WAITING, like this one: > > > > > > > > "http-9090-1" daemon prio=10 tid=0x00002aaaf7e1fc00 > > > nid=0x3f05 waiting on condition > > > [0x000000004459e000..0x000000004459fbc0] > > > > java.lang.Thread.State: WAITING (parking) > > > > at sun.misc.Unsafe.park(Native Method) > > > > - parking to wait for <0x00002aaab8228360> (a > > > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndC > > > heckInterrupt(AbstractQueuedSynchronizer.java:712) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquir > > > eShared(AbstractQueuedSynchronizer.java:842) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireS > > > hared(AbstractQueuedSynchronizer.java:1162) > > > > at > > > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.loc > > k(ReentrantReadWriteLock.java:594) > > > > at > > > org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRe > > > ad(ConcurrentBarrier.java:70) > > > > at > > > org.apache.tapestry.internal.services.CheckForUpdatesFilter.se > > > rvice(CheckForUpdatesFilter.java:110) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > > $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java) > > > > at > > > org.apache.tapestry.services.TapestryModule$11.service(Tapestr > > yModule.java:1044) > > > > at > > > $HttpServletRequestHandler_1139c29ae40.service($HttpServletReq > > uestHandler_1139c29ae40.java) > > > > at > > > > org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135) > > > > at > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > > er(ApplicationFilterChain.java:235) > > > > at > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > > cationFilterChain.java:206) > > > > at > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > > > rapperValve.java:230) > > > > at > > > org.apache.catalina.core.StandardContextValve.invoke(StandardC > > > ontextValve.java:175) > > > > at > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > > > Valve.java:128) > > > > at > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > > > Valve.java:104) > > > > at > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > > > gineValve.java:109) > > > > at > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdap > > > ter.java:261) > > > > at > > > org.apache.coyote.http11.Http11Processor.process(Http11Process > > > or.java:844) > > > > at > > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandle > > r.process(Http11Protocol.java:581) > > > > at > > > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint. > > java:447) > > > > at java.lang.Thread.run(Thread.java:619) > > > > > > > > I'm not sure if the NPE that happened before is the reason > > > for this, as I don't > > > > see the relationship between both issues (apart from the > > > correlation of events/time). > > > > > > > > Hopefully someone closer to the code can have a look at > > > this and tell what's the > > > > problem here? Then I'd like to submit a bug report / help > > > fixing(?)... > > > > > > > > More output of the logs you find at the end of this email. > > > > > > > > Thanx a lot in advance, > > > > cheers, > > > > Martin > > > > > > > > > > > > > > > > [INFO ] 2007-07-09 11:56:35,513 TP-Processor1 > > > org.comp.proj.search.presentation.pages.Search.onActivate: > > > > Got invoked args hose, cat:Herrenbekleidung, n, 100 > > > > > > > > [INFO ] 2007-07-09 11:56:35,518 TP-Processor1 > > > org.comp.proj.search.presentation.pages.Search.setupRender: > > > > Starting search... > > > > > > > > [DEBUG] 2007-07-09 11:56:35,520 TP-Processor1 > > > org.comp.proj.search.business.SearchServiceSolrImpl.search: > > > > Starting query: > > > > > > > q=hose;score+desc&fq=cat:Herrenbekleidung&start=0&rows=100&fac > > et=true&facet.field=cat&facet.field=brand&facet.field=type&facet> > > .field=color&facet > > > > > > > .field=price&facet.limit=5&facet.zeros=false&facet.missing=fal > > se&version=2.2 > > > > > > > > [DEBUG] 2007-07-09 11:56:35,610 TP-Processor1 > > > org.comp.proj.search.business.SearchServiceSolrImpl.logResponse: > > > > Got response: numFound: 207, queryTime: 43 > > > > > > > > [INFO ] 2007-07-09 11:56:35,803 TP-Processor1 > > > org.comp.proj.search.presentation.services.AppModule.TimingFil > > > ter.service: > > > > Request time: 292 ms > > > > > > > > [INFO ] 2007-07-09 11:57:05,982 TP-Processor5 > > > org.comp.proj.search.presentation.pages.Search.onActivate: > > > > Got invoked args hose, n, 100 > > > > > > > > [INFO ] 2007-07-09 11:57:05,988 TP-Processor5 > > > org.comp.proj.search.presentation.pages.Search.setupRender: > > > > Starting search... > > > > > > > > [DEBUG] 2007-07-09 11:57:05,989 TP-Processor5 > > > org.comp.proj.search.business.SearchServiceSolrImpl.search: > > > > Starting query: > > > > > > > q=hose;score+desc&start=0&rows=100&facet=true&facet.field=cat& > > facet.field=brand&facet.field=type&facet.field=color&facet.field> > > =price&facet.limit > > > > =5&facet.zeros=false&facet.missing=false&version=2.2 > > > > > > > > [DEBUG] 2007-07-09 11:57:06,078 TP-Processor5 > > > org.comp.proj.search.business.SearchServiceSolrImpl.logResponse: > > > > Got response: numFound: 2001, queryTime: 44 > > > > > > > > [INFO ] 2007-07-09 11:57:06,276 TP-Processor5 > > > org.comp.proj.search.presentation.services.AppModule.TimingFil > > > ter.service: > > > > Request time: 296 ms > > > > > > > > [INFO ] 2007-07-09 12:07:22,379 TP-Processor5 > > > org.comp.proj.search.presentation.pages.Search.onActivate: > > > > Got invoked args > > > > > > > > [INFO ] 2007-07-09 12:07:22,382 TP-Processor5 > > > org.comp.proj.search.presentation.services.AppModule.TimingFil > > > ter.service: > > > > Request time: 605041 ms > > > > ^^^^^^^^^^^^^^^^^^^^^^^ --> 10 minutes?! > > > > > > > > [ERROR] 2007-07-09 12:07:22,382 TP-Processor5 > > > org.apache.tapestry.internal.services.InternalModule.RequestEx > > > ceptionHandler.handleRequestExceptio > > > > n: > > > > Processing of request failed with uncaught exception: > > > org.apache.tapestry.ioc.internal.util.TapestryException > > > > > > > > org.apache.tapestry.ioc.internal.util.TapestryException > > > > at > > > > org.apache.tapestry.corelib.components.Form.onAction(Form.java:375) > > > > at > > > org.apache.tapestry.corelib.components.Form.handleComponentEve > > > nt(Form.java) > > > > at > > > org.apache.tapestry.internal.structure.ComponentPageElementImp > > > l.handleEvent(ComponentPageElementImpl.java:893) > > > > at > > > org.apache.tapestry.internal.structure.ComponentPageElementImp > > > l.triggerEvent(ComponentPageElementImpl.java:998) > > > > at > > > org.apache.tapestry.internal.services.ComponentActionRequestHa > > > ndlerImpl.handle(ComponentActionRequestHandlerImpl.java:81) > > > > at > > > org.apache.tapestry.internal.services.InternalModule$11.handle > > (InternalModule.java:541) > > > > at > > > $ComponentActionRequestHandler_1139c29ae5a.handle($ComponentAc > > tionRequestHandler_1139c29ae5a.java) > > > > at > > > $ComponentActionRequestHandler_1139c29ae55.handle($ComponentAc > > tionRequestHandler_1139c29ae55.java) > > > > at > > > org.apache.tapestry.internal.services.ComponentActionDispatche > > > r.dispatch(ComponentActionDispatcher.java:116) > > > > at > > > $Dispatcher_1139c29ae57.dispatch($Dispatcher_1139c29ae57.java) > > > > at > > > $Dispatcher_1139c29ae49.dispatch($Dispatcher_1139c29ae49.java) > > > > at > > > org.apache.tapestry.services.TapestryModule$12.service(Tapestr > > yModule.java:1066) > > > > at > > > org.comp.proj.search.presentation.services.AppModule$2.service > > (AppModule.java:117) > > > > at > > > > $RequestFilter_1139c29ae48.service($RequestFilter_1139c29ae48.java) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > org.comp.proj.search.presentation.services.AppModule$1.service > > (AppModule.java:76) > > > > at > > > > $RequestFilter_1139c29ae47.service($RequestFilter_1139c29ae47.java) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > org.apache.tapestry.internal.services.LocalizationFilter.servi > > > ce(LocalizationFilter.java:43) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > org.apache.tapestry.services.TapestryModule$2.service(Tapestry > > Module.java:657) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > org.apache.tapestry.internal.services.StaticFilesFilter.servic > > > e(StaticFilesFilter.java:63) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > org.apache.tapestry.internal.services.CheckForUpdatesFilter$2. > > invoke(CheckForUpdatesFilter.java:97) > > > > at > > > org.apache.tapestry.internal.services.CheckForUpdatesFilter$2. > > invoke(CheckForUpdatesFilter.java:88) > > > > at > > > org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRe > > > ad(ConcurrentBarrier.java:77) > > > > at > > > org.apache.tapestry.internal.services.CheckForUpdatesFilter.se > > > rvice(CheckForUpdatesFilter.java:110) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > > $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java) > > > > at > > > org.apache.tapestry.services.TapestryModule$11.service(Tapestr > > yModule.java:1044) > > > > at > > > $HttpServletRequestHandler_1139c29ae40.service($HttpServletReq > > uestHandler_1139c29ae40.java) > > > > at > > > > org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135) > > > > at > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > > er(ApplicationFilterChain.java:235) > > > > at > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > > cationFilterChain.java:206) > > > > at > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > > > rapperValve.java:230) > > > > at > > > org.apache.catalina.core.StandardContextValve.invoke(StandardC > > > ontextValve.java:175) > > > > at > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > > > Valve.java:128) > > > > at > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > > > Valve.java:104) > > > > at > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > > > gineValve.java:109) > > > > at > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdap > > > ter.java:261) > > > > at > > > > org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) > > > > at > > > > org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283) > > > > at > > > org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767) > > > > at > > > org.apache.jk.common.ChannelSocket.processConnection(ChannelSo > > > cket.java:697) > > > > at > > > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(Chan > > nelSocket.java:889) > > > > at > > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( > > ThreadPool.java:686) > > > > at java.lang.Thread.run(Thread.java:619) > > > > Caused by: java.lang.NullPointerException > > > > at > > > org.apache.tapestry.internal.util.Base64InputStream.decode(Bas > > > e64InputStream.java:34) > > > > at > > > org.apache.tapestry.internal.util.Base64InputStream.<init>(Bas > > > e64InputStream.java:29) > > > > at > > > org.apache.tapestry.internal.util.Base64ObjectInputStream.<ini > > > t>(Base64ObjectInputStream.java:30) > > > > at > > > > org.apache.tapestry.corelib.components.Form.onAction(Form.java:355) > > > > ... 47 more > > > > Jul 9, 2007 2:27:57 PM > > > org.apache.tomcat.util.threads.ThreadPool logFull > > > > SEVERE: All threads (200) are currently busy, waiting. > > > Increase maxThreads (200) or check the servlet status > > > > 2007-07-09 17:13:35 > > > > Full thread dump Java HotSpot(TM) 64-Bit Server VM > > > (1.6.0_01-b06 mixed mode): > > > > > > > > "http-9090-1" daemon prio=10 tid=0x00002aaaf7e1fc00 > > > nid=0x3f05 waiting on condition > > > [0x000000004459e000..0x000000004459fbc0] > > > > java.lang.Thread.State: WAITING (parking) > > > > at sun.misc.Unsafe.park(Native Method) > > > > - parking to wait for <0x00002aaab8228360> (a > > > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndC > > > heckInterrupt(AbstractQueuedSynchronizer.java:712) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquir > > > eShared(AbstractQueuedSynchronizer.java:842) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireS > > > hared(AbstractQueuedSynchronizer.java:1162) > > > > at > > > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.loc > > k(ReentrantReadWriteLock.java:594) > > > > at > > > org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRe > > > ad(ConcurrentBarrier.java:70) > > > > at > > > org.apache.tapestry.internal.services.CheckForUpdatesFilter.se > > > rvice(CheckForUpdatesFilter.java:110) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > > $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java) > > > > at > > > org.apache.tapestry.services.TapestryModule$11.service(Tapestr > > yModule.java:1044) > > > > at > > > $HttpServletRequestHandler_1139c29ae40.service($HttpServletReq > > uestHandler_1139c29ae40.java) > > > > at > > > > org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135) > > > > at > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > > er(ApplicationFilterChain.java:235) > > > > at > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > > cationFilterChain.java:206) > > > > at > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > > > rapperValve.java:230) > > > > at > > > org.apache.catalina.core.StandardContextValve.invoke(StandardC > > > ontextValve.java:175) > > > > at > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > > > Valve.java:128) > > > > at > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > > > Valve.java:104) > > > > at > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > > > gineValve.java:109) > > > > at > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdap > > > ter.java:261) > > > > at > > > org.apache.coyote.http11.Http11Processor.process(Http11Process > > > or.java:844) > > > > at > > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandle > > r.process(Http11Protocol.java:581) > > > > at > > > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint. > > java:447) > > > > at java.lang.Thread.run(Thread.java:619) > > > > > > > > "TP-Processor200" daemon prio=10 tid=0x00002aaaf8896400 > > > nid=0x3ce9 waiting on condition > > > [0x000000004455d000..0x000000004455ec40] > > > > java.lang.Thread.State: WAITING (parking) > > > > at sun.misc.Unsafe.park(Native Method) > > > > - parking to wait for <0x00002aaab8228360> (a > > > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndC > > > heckInterrupt(AbstractQueuedSynchronizer.java:712) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquir > > > eShared(AbstractQueuedSynchronizer.java:842) > > > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireS > > > hared(AbstractQueuedSynchronizer.java:1162) > > > > at > > > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.loc > > k(ReentrantReadWriteLock.java:594) > > > > at > > > org.apache.tapestry.ioc.internal.util.ConcurrentBarrier.withRe > > > ad(ConcurrentBarrier.java:70) > > > > at > > > org.apache.tapestry.internal.services.CheckForUpdatesFilter.se > > > rvice(CheckForUpdatesFilter.java:110) > > > > at > > > > $RequestHandler_1139c29ae4a.service($RequestHandler_1139c29ae4a.java) > > > > at > > > > $RequestHandler_1139c29ae41.service($RequestHandler_1139c29ae41.java) > > > > at > > > org.apache.tapestry.services.TapestryModule$11.service(Tapestr > > yModule.java:1044) > > > > at > > > $HttpServletRequestHandler_1139c29ae40.service($HttpServletReq > > uestHandler_1139c29ae40.java) > > > > at > > > > org.apache.tapestry.TapestryFilter.doFilter(TapestryFilter.java:135) > > > > at > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > > er(ApplicationFilterChain.java:235) > > > > at > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > > cationFilterChain.java:206) > > > > at > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > > > rapperValve.java:230) > > > > at > > > org.apache.catalina.core.StandardContextValve.invoke(StandardC > > > ontextValve.java:175) > > > > at > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > > > Valve.java:128) > > > > at > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > > > Valve.java:104) > > > > at > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > > > gineValve.java:109) > > > > at > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdap > > > ter.java:261) > > > > at > > > > org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) > > > > at > > > > org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283) > > > > at > > > org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767) > > > > at > > > org.apache.jk.common.ChannelSocket.processConnection(ChannelSo > > > cket.java:697) > > > > at > > > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(Chan > > nelSocket.java:889) > > > > at > > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( > > ThreadPool.java:686) > > > > at java.lang.Thread.run(Thread.java:619) > > > > > > > > ... and 199 more of them > > > > > > > > > > > -- > > > Martin Grotzke > > > http://www.javakaffee.de/blog/ > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > -- > Howard M. Lewis Ship > TWD Consulting, Inc. > Independent J2EE / Open-Source Java Consultant > Creator and PMC Chair, Apache Tapestry > Creator, Apache HiveMind > > Professional Tapestry training, mentoring, support > and project work. http://howardlewisship.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
