jstack requires a JDK to be installed, not just a JRE (as I discovered earlier today).
On Thu, Sep 15, 2011 at 2:59 PM, Raju Buchi <raju.bu...@contentpilot.net> wrote: > The mails are now stuck up in spool on my server. Can any one tell me how to > debug. I tried to use jstack, but it says jstack: command not found. > > - Raju > > On Thu, Sep 15, 2011 at 1:16 PM, Zach Cox <zcox...@gmail.com> wrote: > >> OK so the cause of this shutdown problem was setting jmx.address= >> public IP address of the server in conf/jmx.properties. Just had the >> exact same problem on a fresh install of James. I changed jmx.address >> back to 127.0.0.1 and James now starts up properly. >> >> There is also no mention of anything related to jmx in the logs when >> James is unable to start up because of using the public IP address for >> jmx.address. >> >> >> >> >> On Thu, Sep 15, 2011 at 1:02 PM, Zach Cox <zcox...@gmail.com> wrote: >> > Sorry, had to replace that EC2 instance with a new one, need to keep >> > emails flowing... >> > >> > So that server is gone, but we still have all of the logs. >> > >> > If this happens again I will try moving activemq dir. >> > >> > Thanks, >> > Zach >> > >> > >> > On Thu, Sep 15, 2011 at 1:00 PM, Norman Maurer >> > <norman.mau...@googlemail.com> wrote: >> >> Hmm.. >> >> >> >> never saw this problem. But could you try to move the activemq dir out >> >> of the way. >> >> >> >> # mv /path/to/james/var/store/activemq >> /path/to/james/var/store/activemq.crc >> >> >> >> After that try to start again.. >> >> >> >> Bye, >> >> Norman >> >> >> >> >> >> 2011/9/15 Zach Cox <zcox...@gmail.com>: >> >>> And now every time I start James, after it completes startup, it just >> >>> shuts down: >> >>> >> >>> >> >>> 2011-09-15 17:56:26 INFO smtpserver:? - Init SMTP Service done >> >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >> >>> ConfigurationUtils.locate(): base is null, name is null >> >>> 2011-09-15 17:56:26 INFO lmtpserver:? - LMTP Service disabled by >> configuration >> >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >> >>> ConfigurationUtils.locate(): base is null, name is null >> >>> 2011-09-15 17:56:26 INFO pop3server:? - POP3 Service disabled by >> configuration >> >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >> >>> ConfigurationUtils.locate(): base is null, name is null >> >>> 2011-09-15 17:56:27 INFO imapserver:? - IMAP Service disabled by >> configuration >> >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >> >>> ConfigurationUtils.locate(): base is null, name is null >> >>> 2011-09-15 17:56:27 INFO usersrepository23:? - Init >> >>> org.apache.james.repository.file.FilePersistentObjectRepository Store >> >>> 2011-09-15 17:56:27 INFO usersrepository23:? - >> >>> org.apache.james.repository.file.FilePersistentObjectRepository opened >> >>> in ../var/users >> >>> 2011-09-15 17:56:27 DEBUG usersrepository23:? - >> >>> org.apache.james.user.file.UsersFileRepository created in >> >>> file://var/users/ >> >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service >> >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service done >> >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service >> >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service done >> >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service >> >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service done >> >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service >> >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service done >> >>> 2011-09-15 17:56:27 INFO mailspooler:? - >> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >> >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 >> >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 >> >>> 2011-09-15 17:56:27 INFO mailspooler:? - >> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler thread shutdown >> >>> completed. >> >>> 2011-09-15 17:56:30 INFO SocketNode:94 - Caught java.io.EOFException >> >>> closing conneciton. >> >>> >> >>> >> >>> Any advice would be greatly appreciated. >> >>> >> >>> >> >>> >> >>> On Thu, Sep 15, 2011 at 12:54 PM, Zach Cox <zcox...@gmail.com> wrote: >> >>>> OK now several minutes after I restarted James, everything was working >> >>>> fine, then all of a sudden I see this while tailing james-server.log: >> >>>> >> >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Timeout >> >>>> occurred. Now forcing the routes to be shutdown now. >> >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Interrupted >> >>>> while waiting during graceful shutdown, will force shutdown now. >> >>>> 2011-09-15 17:51:24 WARN DefaultInflightRepository:? - Shutting down >> >>>> while there are still 8 in flight exchanges. >> >>>> 2011-09-15 17:51:26 INFO SocketNode:94 - Caught java.io.EOFException >> >>>> closing conneciton. >> >>>> >> >>>> >> >>>> And James just shuts down. This is incredibly bad to have a mail >> >>>> server just shut down out of the blue... >> >>>> >> >>>> >> >>>> >> >>>> On Thu, Sep 15, 2011 at 12:46 PM, Zach Cox <zcox...@gmail.com> wrote: >> >>>>> Also when James is starting up I see this in logs: >> >>>>> >> >>>>> 2011-09-15 17:45:05 WARN AMQPersistenceAdapter:? - The >> ReferenceStore >> >>>>> is not valid - recovering ... >> >>>>> >> >>>>> >> >>>>> I'm guessing that's not a good thing? >> >>>>> >> >>>>> >> >>>>> On Thu, Sep 15, 2011 at 11:54 AM, Zach Cox <zcox...@gmail.com> >> wrote: >> >>>>>> I don't think we're seeing the problem again yet, but I just had to >> >>>>>> restart James and noticed this on shutdown: >> >>>>>> >> >>>>>> 2011-09-15 16:48:31 INFO JamesServerApplicationContext:? - Closing >> >>>>>> >> org.apache.james.container.spring.context.JamesServerApplicationContext@86fe26 >> : >> >>>>>> startup date [Thu Sep 15 12:20:21 UTC 2011]; root of context >> hierarchy >> >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service >> >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service done >> >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service >> >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service done >> >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service >> >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service done >> >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service >> >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service done >> >>>>>> 2011-09-15 16:48:31 INFO mailspooler:? - >> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >> >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 >> >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 >> >>>>>> 2011-09-15 16:48:51 INFO SocketNode:94 - Caught >> java.io.EOFException >> >>>>>> closing conneciton. >> >>>>>> >> >>>>>> >> >>>>>> Is that java.io.EOFException normal and expected? Or is it a sign of >> >>>>>> some problems? >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Zach >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Thu, Sep 15, 2011 at 8:35 AM, Norman Maurer >> >>>>>> <norman.mau...@googlemail.com> wrote: >> >>>>>>> Using: >> >>>>>>> >> >>>>>>> # jstack <pid> >> >>>>>>> >> >>>>>>> >> >>>>>>> Please take 3 within on minute ... >> >>>>>>> >> >>>>>>> Bye, >> >>>>>>> Norman >> >>>>>>> >> >>>>>>> 2011/9/15 Zach Cox <zcox...@gmail.com>: >> >>>>>>>> I had to restart James several times this morning so nothing is >> >>>>>>>> blocked at the moment. >> >>>>>>>> >> >>>>>>>> There is nothing indicated in the log files. Things look normal, >> and >> >>>>>>>> then they look like this: >> >>>>>>>> >> >>>>>>>> 2011-09-15 01:25:26 INFO smtpserver:? - ID=8443803 Connection >> >>>>>>>> established from x >> >>>>>>>> 2011-09-15 01:25:31 INFO smtpserver:? - ID=8443803 Successfully >> >>>>>>>> spooled mail >> Mail1316049931087-05b8199e-e01f-4d14-84d5-3d7ac2443bb3 >> >>>>>>>> from x on x for [x] >> >>>>>>>> 2011-09-15 01:25:36 INFO smtpserver:? - ID=8443803 Connection >> closed for x >> >>>>>>>> 2011-09-15 01:25:47 INFO smtpserver:? - ID=11521429 Connection >> >>>>>>>> established from x >> >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Successfully >> >>>>>>>> spooled mail >> Mail1316049948254-abd67e8f-bafe-41ab-8eef-c551420fd56e >> >>>>>>>> from x on x for [x] >> >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Connection >> closed for x >> >>>>>>>> 2011-09-15 01:26:41 INFO smtpserver:? - ID=5675333 Connection >> >>>>>>>> established from x >> >>>>>>>> 2011-09-15 01:26:52 INFO smtpserver:? - ID=5675333 Successfully >> >>>>>>>> spooled mail >> Mail1316050012621-522b1f2a-2851-46fb-940c-711591f637cc >> >>>>>>>> from x on x for [x] >> >>>>>>>> 2011-09-15 01:26:57 INFO smtpserver:? - ID=5675333 Connection >> closed for x >> >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection >> >>>>>>>> established from localhost (127.0.0.1) >> >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection >> closed >> >>>>>>>> for localhost (127.0.0.1) >> >>>>>>>> 2011-09-15 01:28:12 INFO smtpserver:? - ID=11673762 Connection >> >>>>>>>> established from x >> >>>>>>>> 2011-09-15 01:28:22 INFO smtpserver:? - ID=11673762 Successfully >> >>>>>>>> spooled mail >> Mail1316050102393-6fcc21aa-8f32-4935-b0eb-df892d54a5e8 >> >>>>>>>> from x on x for [x] >> >>>>>>>> 2011-09-15 01:28:27 INFO smtpserver:? - ID=11673762 Connection >> closed for x >> >>>>>>>> >> >>>>>>>> So every mail that comes in just gets spooled and not processed. >> >>>>>>>> >> >>>>>>>> When this starts happening again, what's the best way to take a >> thread >> >>>>>>>> dump from James? >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> Zach >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Thu, Sep 15, 2011 at 8:13 AM, Norman Maurer >> >>>>>>>> <norman.mau...@googlemail.com> wrote: >> >>>>>>>>> Hi there, >> >>>>>>>>> >> >>>>>>>>> please take a thread-dump to see why its blocked... >> >>>>>>>>> >> >>>>>>>>> Anything in the logs ? >> >>>>>>>>> >> >>>>>>>>> Bye, >> >>>>>>>>> Norman >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> 2011/9/15 Zach Cox <zcox...@gmail.com>: >> >>>>>>>>>> We upgraded to James 3.0-beta3 on Tuesday night. Things worked >> fine >> >>>>>>>>>> until around 9pm last night, where exactly the same problem >> started >> >>>>>>>>>> occurring: mail would get spooled, but not processed. After >> >>>>>>>>>> restarting James, some of the mail was then processed >> successfully, >> >>>>>>>>>> but most was lost. >> >>>>>>>>>> >> >>>>>>>>>> We consider this an emergency situation for our company and need >> to >> >>>>>>>>>> find a resolution ASAP. What kind of information can we provide >> to >> >>>>>>>>>> help find a resolution? If financial resources need to be >> involved to >> >>>>>>>>>> find a resolution or move things along, that is an option. >> >>>>>>>>>> >> >>>>>>>>>> Any help is most greatly appreciated. >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> Zach >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Tue, Sep 13, 2011 at 12:41 PM, Norman Maurer >> >>>>>>>>>> <norman.mau...@googlemail.com> wrote: >> >>>>>>>>>>> No problems... >> >>>>>>>>>>> >> >>>>>>>>>>> If you still can replicate then please open a issue with a >> stack trace >> >>>>>>>>>>> attached.. >> >>>>>>>>>>> >> >>>>>>>>>>> Bye, >> >>>>>>>>>>> Norman >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> 2011/9/13 Raju Buchi <raju.bu...@contentpilot.net>: >> >>>>>>>>>>>> James 3.0-M2. I will try to stand up a new server with beta3 >> and see if I >> >>>>>>>>>>>> can replicate the issue. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks >> >>>>>>>>>>>> >> >>>>>>>>>>>> - Raju >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Tue, Sep 13, 2011 at 12:37 PM, Norman Maurer < >> >>>>>>>>>>>> norman.mau...@googlemail.com> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> Do you use beta3 ? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Bye, >> >>>>>>>>>>>>> Norman >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> 2011/9/13 Raju Buchi <raju.bu...@contentpilot.net>: >> >>>>>>>>>>>>> > Hello >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > We are also facing exactly the same issues. Tried to debug >> my custom >> >>>>>>>>>>>>> mailets >> >>>>>>>>>>>>> > and seems to be everything is working fine. Logs did not >> show any >> >>>>>>>>>>>>> exception >> >>>>>>>>>>>>> > messages except it will always spool a mail and will never >> attempt to >> >>>>>>>>>>>>> send >> >>>>>>>>>>>>> > it. i.e. all mails are stuck in the spool. >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > - Raju >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > On Tue, Sep 13, 2011 at 12:20 PM, Zach Cox < >> zcox...@gmail.com> wrote: >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> >> We are having a severe problem with James 3.0-M2 where it >> will operate >> >>>>>>>>>>>>> >> correctly for hours or days, and then all of a sudden it >> will not >> >>>>>>>>>>>>> >> process a lot of the mail it spools. Sometimes it will >> not process >> >>>>>>>>>>>>> >> *any* mail that it spools, and other times it will spool a >> bunch of >> >>>>>>>>>>>>> >> mail, and then process others seemingly at random. I have >> to restart >> >>>>>>>>>>>>> >> James to get that spooled mail to be processed. But even >> then, >> >>>>>>>>>>>>> >> sometimes the spooled mail is completely lost after a >> restart. I have >> >>>>>>>>>>>>> >> seen several other threads lately with similar problems, >> but they >> >>>>>>>>>>>>> >> haven't helped our situation. >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> We do not store mail in James for personal accounts and we >> have IMAP & >> >>>>>>>>>>>>> >> POP3 disabled. Every inbound email has one of two things >> done to it: >> >>>>>>>>>>>>> >> - it is processed locally by custom mailets & matchers >> >>>>>>>>>>>>> >> - or it is relayed to our corporate Google Apps account >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> A previous thread recommended checking all custom mailets >> and >> >>>>>>>>>>>>> >> matchers. As far as I can tell, they are all working >> properly. There >> >>>>>>>>>>>>> >> are no exceptions or errors in log/james-server.log, and >> in the >> >>>>>>>>>>>>> >> scenario where James spools some mail & processes others, >> the mail >> >>>>>>>>>>>>> >> that is processed is handled properly by our custom >> mailets & >> >>>>>>>>>>>>> >> matchers. >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> I've tried connecting to James JMX remotely via JConsole >> but have not >> >>>>>>>>>>>>> >> been able to connect. Which port does James run JMX on? >> Any help >> >>>>>>>>>>>>> >> with JMX would be appreciated. >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> Does anyone have any other ideas about what we should look >> for? I >> >>>>>>>>>>>>> >> noticed 3.0-beta3 is out now, were there fixes in beta3 >> related to >> >>>>>>>>>>>>> >> mail being spooled but not processed? >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> Thanks, >> >>>>>>>>>>>>> >> Zach >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> >> --------------------------------------------------------------------- >> >>>>>>>>>>>>> >> To unsubscribe, e-mail: >> server-user-unsubscr...@james.apache.org >> >>>>>>>>>>>>> >> For additional commands, e-mail: >> server-user-h...@james.apache.org >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> --------------------------------------------------------------------- >> >>>>>>>>>>>>> To unsubscribe, e-mail: >> server-user-unsubscr...@james.apache.org >> >>>>>>>>>>>>> For additional commands, e-mail: >> server-user-h...@james.apache.org >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> --------------------------------------------------------------------- >> >>>>>>>>>>> To unsubscribe, e-mail: >> server-user-unsubscr...@james.apache.org >> >>>>>>>>>>> For additional commands, e-mail: >> server-user-h...@james.apache.org >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> --------------------------------------------------------------------- >> >>>>>>>>>> To unsubscribe, e-mail: >> server-user-unsubscr...@james.apache.org >> >>>>>>>>>> For additional commands, e-mail: >> server-user-h...@james.apache.org >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> --------------------------------------------------------------------- >> >>>>>>>>> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org >> >>>>>>>>> For additional commands, e-mail: >> server-user-h...@james.apache.org >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> --------------------------------------------------------------------- >> >>>>>>>> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org >> >>>>>>>> For additional commands, e-mail: >> server-user-h...@james.apache.org >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> --------------------------------------------------------------------- >> >>>>>>> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org >> >>>>>>> For additional commands, e-mail: server-user-h...@james.apache.org >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org >> >>> For additional commands, e-mail: server-user-h...@james.apache.org >> >>> >> >>> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org >> >> For additional commands, e-mail: server-user-h...@james.apache.org >> >> >> >> >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org >> For additional commands, e-mail: server-user-h...@james.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org
