2012/10/9 Karl Wright <daddy...@gmail.com>: > "- all worker threads are gone," ??? > > Really??
yes... really.. this is why I am also writing that this is strange... this is list of currently active threads: system Reference Handler Waiting Finalizer Waiting Signal Dispatcher Running Attach Listener Running main main Waiting Timer-0 Evaluating... HashSessionScavenger-0 Waiting HashSessionScavenger-1 Evaluating... HashSessionScavenger-2 Waiting qtp1792381350-157 Evaluating... qtp1792381350-158 Waiting qtp1792381350-159 Evaluating... qtp1792381350-161 Waiting Job start thread Waiting Startup thread Waiting Delete startup thread Waiting Finisher thread Waiting Job notification thread Waiting Job delete thread Waiting Stuffer thread Waiting Expire stuffer thread Waiting Set priority thread Waiting Expiration thread '0' Waiting Expiration thread '1' Waiting Expiration thread '2' Waiting Expiration thread '3' Waiting Expiration thread '4' Waiting Expiration thread '5' Evaluating... Expiration thread '6' Waiting Expiration thread '7' Waiting Expiration thread '8' Waiting Expiration thread '9' Evaluating... Document cleanup stuffer thread Waiting Document cleanup thread '0' Waiting Document cleanup thread '1' Waiting Document cleanup thread '2' Waiting Document cleanup thread '3' Evaluating... Document cleanup thread '4' Waiting Document cleanup thread '5' Waiting Document cleanup thread '6' Evaluating... Document cleanup thread '7' Waiting Document cleanup thread '8' Evaluating... Document cleanup thread '9' Waiting Document delete stuffer thread Waiting Document delete thread '0' Waiting Document delete thread '1' Waiting Document delete thread '2' Evaluating... Document delete thread '3' Waiting Document delete thread '4' Evaluating... Document delete thread '5' Waiting Document delete thread '6' Waiting Document delete thread '7' Waiting Document delete thread '8' Waiting Document delete thread '9' Evaluating... Job reset thread Waiting Seeding thread Waiting Idle cleanup thread Evaluating... Connection pool reaper Sleeping qtp1792381350-160 Running qtp1792381350-162 Running qtp1792381350-163 Running qtp1792381350-156 Evaluating... derby.daemons derby.rawStoreDaemon Waiting but still UI shows: Name Status Start Time End Time Documents Active Processed Restart Pause Abort msol-x Running Tue Oct 09 15:15:29 CEST 2012 5689 10 5689 (10 documents left active to process and status = "running") I checked the code of connector and there are only "ManifoldCFException" throwed... > > I can think of no scenario where the worker threads disappear except > if shutdown of the agents process is attempted and fails. > WorkerThread catches all Throwable's and logs them and repeats in a > loop. The only kind of exception that can cause the thread to exit is > an InterruptedException. > > This isn't making a heck of a lot of sense... > > Karl > > > On Tue, Oct 9, 2012 at 9:58 AM, Maciej Liżewski > <maciej.lizew...@gmail.com> wrote: >> Just looking at threads... but nothing special. >> - all worker threads are gone, >> - stuffer thread runs in a loop but finds nothing to do... >> - other threads just waits on 'sleep' commands. >> >> is there any particular thread I should look at? >> >> I could guess that there was some exception (maybe in my connector, >> but I could not repeat it) that was not handled and some worker thread >> just disappeared... >> >> How to enable debug logs so I could see verbose output from core functions? >> >> >> 2012/10/9 Karl Wright <daddy...@gmail.com>: >>> FWIW, getting thread dumps from the process running the agents process >>> when it is "hung" may (or may not) help determine the underlying >>> clause. >>> >>> Karl >>> >>> On Tue, Oct 9, 2012 at 9:21 AM, Karl Wright <daddy...@gmail.com> wrote: >>>> What is your deployment model? Is this a multiprocess deployment? >>>> What database are you using? >>>> >>>> There are various load tests for each database, which do far more than >>>> 7000 documents. I am concerned that you are seeing this because of >>>> some kind of cross-process synchronization issues, which might occur >>>> (for instance) if you are using a multiprocess environment with a >>>> single-process properties.xml file. >>>> >>>> Karl >>>> >>>> On Tue, Oct 9, 2012 at 9:12 AM, Maciej Liżewski >>>> <maciej.lizew...@gmail.com> wrote: >>>>> Ok... it is not a getMaxDocumentRequest issue, because I was able to >>>>> get it even with getMaxDocumentRequest=1. Seems it occurs when >>>>> indenxing large sets of documents (in my case ~7000). It also happened >>>>> once for CIFS connecotr (with samba share)... >>>>> >>>>> result is like this: >>>>> >>>>> Name Status Start Time End Time Documents Active >>>>> Processed >>>>> Restart Pause Abort Mantis Running Tue Oct 09 13:56:59 CEST >>>>> 2012 5689 1600 4400 >>>>> >>>>> there is "active" documents count 1600 for about an hour now but there >>>>> is no server load and nothing changes... seems that it is hanging >>>>> somwhere inside manifold core. >>>>> >>>>> also - when hitting abort - nothing happens (job process remains in >>>>> "aborting" state)... >>>>> >>>>> Problem is that it happens irregularly (sometime 10 documents, >>>>> sometime 1600 and sometime all documents are indexed). Tried to check >>>>> that locally but on first pass everything went ok... really strange... >>>>> >>>>> >>>>> 2012/10/3 Karl Wright <daddy...@gmail.com>: >>>>>> Hi Maciej, >>>>>> >>>>>> It sounds like your loop condition must be somehow incorrect. You may >>>>>> not receive the full number of documents specified by >>>>>> getMaxDocumentRequest(), but rather a number less than that. >>>>>> >>>>>> We have a number of connectors that use document batches > 1, e.g. the >>>>>> LiveLink connector, so this is likely not the problem. >>>>>> >>>>>> I'd recommend adding System.out.println() diagnostics to see exactly >>>>>> what is happening inside both getDocumentVersions() and >>>>>> processDocuments(). >>>>>> >>>>>> Karl >>>>>> >>>>>> >>>>>> On Wed, Oct 3, 2012 at 4:30 PM, Maciej Liżewski >>>>>> <maciej.lizew...@gmail.com> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> I have noticed strange problem with Connector (new one I am developing >>>>>>> right now) and getMaxDocumentRequest parameter. >>>>>>> When it returns 1 (default) everything seems ok, but when I set it to >>>>>>> anything higher (5, 10, 20) indexing job does not end but hangs when >>>>>>> there is only getMaxDocumentRequest documents left (when it should >>>>>>> process 5 documents in a row - 5 documents stays "active") >>>>>>> All document related functions seem written ok (they all iterate >>>>>>> throug passed arrays), there are no exceptions thrown (at least I do >>>>>>> not see any in console). >>>>>>> >>>>>>> What can be wrong and what should I look at to? any ideas? >>>>>>> >>>>>>> By the way - the new connector is for Mantis Bug tracker to index >>>>>>> issues. >>>>>>> >>>>>>> TIA >>>>>>> Redguy
