SMB exceptions with jcifs in the trace tell us that JCIFS couldn't talk to your windows share server. That's all we can tell though.
Karl On Mon, Nov 15, 2021 at 7:24 AM ritika jain <ritikajain5...@gmail.com> wrote: > Hi, > > Raising the concern above again, to process only 60k of document (when > clock issue is fixed too), job process is not progressing , its being stuck > for like days. So had to restart the docker container every time for it to > process. > This time now we are getting this :- Timeout Exception. What we can be the > reason for it and how it can be fixed .? > ... 24 more > [Worker thread '23'] WARN jcifs.util.transport.Transport - sendrecv failed > jcifs.util.transport.RequestTimeoutException: Transport40 timedout waiting > for response to > command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=86 > at > jcifs.util.transport.Transport.waitForResponses(Transport.java:365) > at jcifs.util.transport.Transport.sendrecv(Transport.java:232) > at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021) > at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) > at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) > at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) > at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) > at > jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614) > at > jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568) > at > jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) > at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) > at > jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) > at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551) > at jcifs.smb.SmbFile.length(SmbFile.java:1541) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileLength(SharedDriveConnector.java:2340) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector$ProcessDocumentsFilter.accept(SharedDriveConnector.java:4935) > at > jcifs.smb.SmbEnumerationUtil$ResourceFilterWrapper.accept(SmbEnumerationUtil.java:331) > at > jcifs.smb.FileEntryAdapterIterator.advance(FileEntryAdapterIterator.java:82) > at > jcifs.smb.FileEntryAdapterIterator.<init>(FileEntryAdapterIterator.java:52) > at > jcifs.smb.DirFileEntryAdapterIterator.<init>(DirFileEntryAdapterIterator.java:37) > at jcifs.smb.SmbEnumerationUtil.doEnum(SmbEnumerationUtil.java:223) > at > jcifs.smb.SmbEnumerationUtil.listFiles(SmbEnumerationUtil.java:279) > at jcifs.smb.SmbFile.listFiles(SmbFile.java:1273) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileListFiles(SharedDriveConnector.java:2380) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:818) > at > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) > [Worker thread '23'] WARN jcifs.smb.SmbTransportImpl - Disconnecting > transport while still in use Transport40[backup002.directory.intra/ > 136.231.158.172:445,state=5,signingEnforced=false,usage=5]: > [SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]] > [Worker thread '23'] WARN jcifs.smb.SmbSessionImpl - Logging off session > while still in use > SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINPROJECTS,service=?????,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]] > [Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed > jcifs.util.transport.RequestTimeoutException: Transport41 timedout waiting > for response to > command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=80 > at > jcifs.util.transport.Transport.waitForResponses(Transport.java:365) > at jcifs.util.transport.Transport.sendrecv(Transport.java:232) > at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021) > at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) > at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) > at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) > at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) > at > jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614) > at > jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568) > at > jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) > at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) > at > jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) > at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551) > at jcifs.smb.SmbFile.exists(SmbFile.java:845) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2220) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:610) > at > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) > [Worker thread '10'] WARN jcifs.smb.SmbTransportImpl - Disconnecting > transport while still in use Transport41[homestore.directory.intra/ > 136.231.158.104:445,state=5,signingEnforced=false,usage=5]: > [SmbSession[credentials=svc_EScrawl,targetHost=homestore.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]] > [Worker thread '10'] WARN jcifs.smb.SmbSessionImpl - Logging off session > while still in use > SmbSession[credentials=svc_EScrawl,targetHost=homestore.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINHOMES,service=?????,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]] > [Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed > jcifs.util.transport.RequestTimeoutException: Transport44 timedout waiting > for response to > command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=72 > at > jcifs.util.transport.Transport.waitForResponses(Transport.java:365) > at jcifs.util.transport.Transport.sendrecv(Transport.java:232) > at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1 > > > On Tue, Nov 9, 2021 at 6:19 PM Karl Wright <daddy...@gmail.com> wrote: > >> One hour is quite a lot and will wreak havoc on the document queue. >> Karl >> >> >> On Tue, Nov 9, 2021 at 7:08 AM ritika jain <ritikajain5...@gmail.com> >> wrote: >> >>> I have checked, there is only one hour time difference between docker >>> container and docker host >>> >>> On Tue, Nov 9, 2021 at 4:41 PM Karl Wright <daddy...@gmail.com> wrote: >>> >>>> If your docker image's clock is out of sync badly with the real world, >>>> then System.currentTimeMillis() may give bogus values, and ManifoldCF uses >>>> that to manage throttling etc. I don't know if that is the correct >>>> explanation but it's the only thing I can think of. >>>> >>>> Karl >>>> >>>> >>>> On Tue, Nov 9, 2021 at 4:56 AM ritika jain <ritikajain5...@gmail.com> >>>> wrote: >>>> >>>>> >>>>> Hi All, >>>>> >>>>> I am using window shares connector , manifoldcf 2.14 and ES as output. >>>>> I have configured a job to process 60k of documents, Also these documents >>>>> are new and do not have corresponding values in DB and ES index. >>>>> >>>>> So ideally it should process/Index the documents as soon as the job >>>>> starts. >>>>> But Manifoldcf does not process anything for many hours of job start >>>>> up.I have tried restarting the docker container as well. But it didn't >>>>> help >>>>> much. Also logs only correspond to Long running queries. >>>>> >>>>> Why does the manifold behave like that? >>>>> >>>>> Thanks >>>>> Ritika >>>>> >>>>
