Hi, Raising the concern above again, to process only 60k of document (when clock issue is fixed too), job process is not progressing , its being stuck for like days. So had to restart the docker container every time for it to process. This time now we are getting this :- Timeout Exception. What we can be the reason for it and how it can be fixed .? ... 24 more [Worker thread '23'] WARN jcifs.util.transport.Transport - sendrecv failed jcifs.util.transport.RequestTimeoutException: Transport40 timedout waiting for response to command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=86 at jcifs.util.transport.Transport.waitForResponses(Transport.java:365) at jcifs.util.transport.Transport.sendrecv(Transport.java:232) at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021) at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) at jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614) at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568) at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) at jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551) at jcifs.smb.SmbFile.length(SmbFile.java:1541) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileLength(SharedDriveConnector.java:2340) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector$ProcessDocumentsFilter.accept(SharedDriveConnector.java:4935) at jcifs.smb.SmbEnumerationUtil$ResourceFilterWrapper.accept(SmbEnumerationUtil.java:331) at jcifs.smb.FileEntryAdapterIterator.advance(FileEntryAdapterIterator.java:82) at jcifs.smb.FileEntryAdapterIterator.<init>(FileEntryAdapterIterator.java:52) at jcifs.smb.DirFileEntryAdapterIterator.<init>(DirFileEntryAdapterIterator.java:37) at jcifs.smb.SmbEnumerationUtil.doEnum(SmbEnumerationUtil.java:223) at jcifs.smb.SmbEnumerationUtil.listFiles(SmbEnumerationUtil.java:279) at jcifs.smb.SmbFile.listFiles(SmbFile.java:1273) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileListFiles(SharedDriveConnector.java:2380) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:818) at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [Worker thread '23'] WARN jcifs.smb.SmbTransportImpl - Disconnecting transport while still in use Transport40[backup002.directory.intra/ 136.231.158.172:445,state=5,signingEnforced=false,usage=5]: [SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]] [Worker thread '23'] WARN jcifs.smb.SmbSessionImpl - Logging off session while still in use SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINPROJECTS,service=?????,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]] [Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed jcifs.util.transport.RequestTimeoutException: Transport41 timedout waiting for response to command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=80 at jcifs.util.transport.Transport.waitForResponses(Transport.java:365) at jcifs.util.transport.Transport.sendrecv(Transport.java:232) at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021) at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) at jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614) at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568) at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) at jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551) at jcifs.smb.SmbFile.exists(SmbFile.java:845) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2220) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:610) at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [Worker thread '10'] WARN jcifs.smb.SmbTransportImpl - Disconnecting transport while still in use Transport41[homestore.directory.intra/ 136.231.158.104:445,state=5,signingEnforced=false,usage=5]: [SmbSession[credentials=svc_EScrawl,targetHost=homestore.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]] [Worker thread '10'] WARN jcifs.smb.SmbSessionImpl - Logging off session while still in use SmbSession[credentials=svc_EScrawl,targetHost=homestore.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINHOMES,service=?????,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]] [Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed jcifs.util.transport.RequestTimeoutException: Transport44 timedout waiting for response to command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=72 at jcifs.util.transport.Transport.waitForResponses(Transport.java:365) at jcifs.util.transport.Transport.sendrecv(Transport.java:232) at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1
On Tue, Nov 9, 2021 at 6:19 PM Karl Wright <daddy...@gmail.com> wrote: > One hour is quite a lot and will wreak havoc on the document queue. > Karl > > > On Tue, Nov 9, 2021 at 7:08 AM ritika jain <ritikajain5...@gmail.com> > wrote: > >> I have checked, there is only one hour time difference between docker >> container and docker host >> >> On Tue, Nov 9, 2021 at 4:41 PM Karl Wright <daddy...@gmail.com> wrote: >> >>> If your docker image's clock is out of sync badly with the real world, >>> then System.currentTimeMillis() may give bogus values, and ManifoldCF uses >>> that to manage throttling etc. I don't know if that is the correct >>> explanation but it's the only thing I can think of. >>> >>> Karl >>> >>> >>> On Tue, Nov 9, 2021 at 4:56 AM ritika jain <ritikajain5...@gmail.com> >>> wrote: >>> >>>> >>>> Hi All, >>>> >>>> I am using window shares connector , manifoldcf 2.14 and ES as output. >>>> I have configured a job to process 60k of documents, Also these documents >>>> are new and do not have corresponding values in DB and ES index. >>>> >>>> So ideally it should process/Index the documents as soon as the job >>>> starts. >>>> But Manifoldcf does not process anything for many hours of job start >>>> up.I have tried restarting the docker container as well. But it didn't help >>>> much. Also logs only correspond to Long running queries. >>>> >>>> Why does the manifold behave like that? >>>> >>>> Thanks >>>> Ritika >>>> >>>