Null pointer exception when accessed the job info thru mcf api service
Hi Karl, We are facing some issues with manifoldCF. In ManifoldUI, everything shows up except, list of jobs. There are no logs displayed in log file. Also when we try to get job information of a particular job through mcf api service, we are getting null pointer exception. We tried lock clean, restarted server. But its of no use. Could you please help us in identifying the root cause. Please find the response below. h1HTTP Status 500 - /h1HR size=1 noshade=noshadepbtype/b Exception report/ppbmessage/b u/u/ppbdescription/b uThe server encountered an internal error that prevented it from fulfilling this request./u/ppbexception/b prejava.lang.NullPointerException org.apache.manifoldcf.crawler.jobs.PipelineManager.getRows(PipelineManager.java:232) org.apache.manifoldcf.crawler.jobs.Jobs.getJobsChunk(Jobs.java:3521) org.apache.manifoldcf.crawler.jobs.Jobs.getJobsMultiple(Jobs.java:3451) org.apache.manifoldcf.crawler.jobs.Jobs$JobObjectExecutor.create(Jobs.java:3644) org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146) org.apache.manifoldcf.crawler.jobs.Jobs.loadMultipleInternal(Jobs.java:857) org.apache.manifoldcf.crawler.jobs.Jobs.loadMultiple(Jobs.java:838) org.apache.manifoldcf.crawler.jobs.Jobs.load(Jobs.java:816) org.apache.manifoldcf.crawler.jobs.JobManager.load(JobManager.java:794) org.apache.manifoldcf.crawler.system.ManifoldCF.apiReadJob(ManifoldCF.java:1296) org.apache.manifoldcf.crawler.system.ManifoldCF.executeReadCommand(ManifoldCF.java:2898) org.apache.manifoldcf.apiservlet.APIServlet.executeRead(APIServlet.java:232) org.apache.manifoldcf.apiservlet.APIServlet.doGet(APIServlet.java:78) javax.servlet.http.HttpServlet.service(HttpServlet.java:621) javax.servlet.http.HttpServlet.service(HttpServlet.java:728) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)/pre/ppbnote/b Thanks Regards, Smitha S CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS***
MCF API Services
Hi Karl, I am using MCF REST API services for creating connectors and jobs. I am stuck in two places. Could you please help me here. 1. Repository Connector Creation: How do we specify password for creating a repository connector with window share. Is MCF expecting the password to be encrypted in any special format. When I try to pass the password as a normal string, the connector is created with a connection status as Couldn't connect to server: Logon failure: unknown user name or bad password.. How do I pass password here. 2. Job Status history: I have created and scheduled a job using MCF Rest API. The requirement is to get the job running history with info of job start time, job end time, number of documents processed etc. Using jobstatus/jobid, I can get the last job run details. But how do I fetch the previous run details. Could you please provide some insights for the above two points. I really appreciate your help in resolving all MCF queries. Thanks in advance Regards, Smitha
ManifoldCF upgradation to 2.0.2- DB Issues
Hi, Currently we are using MCF 1.6 version. I wanted to upgrade to MCF2.0.2 version. Since so many jobs has been configured in MCF1.6 version, I was planning to use the same MCF DB for 2.0.2 version. But I am getting error related to some db column when I specify the same DB in new properties.xml. Is there any problem in using the old MCF DB. Or should I start fresh and re-create all the connectors and jobs. How do I migrate the jobs from old DB to new DB. Please help me in getting the process correct. Thanks in advance. Regards, Smitha
ManifoldCF Authentication
Hi Karl, I have a requirement to crawl documents shared using window share and index them to solr. The sharing permissions of window share decides which user can access documents. The window share is under Active Directory. I have an web application where the search results are displayed. Currently we are handling authentication and authorization with a DB approach. Now we are planning to use the Authentication module of ManifoldCF to decide the access rights of documents. Because of some client side restrictions, I cannot use Active Directory for login into the web application. In that case, do u see any challenges in using ManifoldCF for providing access rights for documents? Thanks Regards, Smitha S CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS***
RE: Help need for connecting to window share
Hi Karl, I am worried that I am not getting your point correctly. May be further explanation from my side might throw some clarity on the problem which I am facing. My requirement is to crawl the data resides in a window server. I have made this working by using the file system repository connection. Please find the steps which I followed below. 1. Mount the shared folder of windows fileserver to a linux folder using cifs mount command. 2. Mount worked successfully and I could access the data in the linux mounted folder. 3. I have configured filesystem repository connection and while configuring job using this repository connection, I have given the linux mounted folder in the repository path of Job UI. 4. Everything worked well and I could crawl the data by starting the job. Now I want to perform the crawling using Window File Share repository connection. As per my understanding ManifoldCF uses the cifs connection to window share. So if I can manually mount the windows folder in linux, then why I should get an error while configuring window share using the same server in ManifoldCF. Could you please throw some light on this. If you could explain the steps involved in configuring a job using window file server respository it would be great help. Currently I am refereeing End user documentation for this purpose. Regards, Smitha S From: Karl Wright [mailto:daddy...@gmail.com] Sent: Monday, March 3, 2014 1:24 PM To: user@manifoldcf.apache.org Subject: Re: Help need for connecting to window share Hi Smitha, The problem is that, without some research, you may have no idea what server you are actually talking to, because when you drill down in the Paths tab, you may be traversing DFS nodes that refer to other servers. So even though you may be starting at the root talking to a Windows 2008 Server machine, the path you select may lead you to some other machine. If you start out seeing no problem in the UI, but only as you descend to a particular place you run into issues, that would tend to confirm this story. Karl On Mon, Mar 3, 2014 at 2:47 AM, Smitha S smitha_...@infosys.commailto:smitha_...@infosys.com wrote: Thanks Karl for the response. I have my doubts here. I am using windows server 2008 as my repository server. The status of my repository connection with this window share says connection working. I hope I get error in establishing connection if the issue is due to credentials. I am getting the error when I go to PATH tab of job creation UI. Is there anything extra I need to do when using Window Share repository connection. Regards, Smitha From: Karl Wright [mailto:daddy...@gmail.commailto:daddy...@gmail.com] Sent: Monday, March 3, 2014 1:05 PM To: user@manifoldcf.apache.orgmailto:user@manifoldcf.apache.org Cc: user-h...@manifoldcf.apache.orgmailto:user-h...@manifoldcf.apache.org Subject: Re: Help need for connecting to window share Hi Smitha, This is apparently a credentials issue. The error is from the server, and apparently has to do with smart-card login. See: http://support.microsoft.com/kb/891849 and: http://social.technet.microsoft.com/Forums/windowsserver/en-US/e8cf1a1b--4eaa-b55a-cbfad3ef32d6/error-0xc0bb-in-smart-card-logon?forum=winserversecurity If this is an XP system, it means you need SP3 installed. If this is not an old XP system that you are crawling, I can find no indication of the circumstances leading to the error. Karl On Mon, Mar 3, 2014 at 1:14 AM, Smitha S smitha_...@infosys.commailto:smitha_...@infosys.com wrote: Hi, I am working on a project where I need to use the window file server for crawling documents. I am trying to use Window Share repository connection. I could configure the window share repository connection with server details and I managed to get Connection Working status. But while configuring the job using this repository, I am getting the following error in UI No starting points defined SmbException thrown: 0xC0BB In the tomcat server console I am getting following exception org.apache.manifoldcf.core.interfaces.ManifoldCFException: SmbException thrown: 0xC0BB at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processSMBException(SharedDriveConnector.java:1289) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getChildFolderNames(SharedDriveConnector.java:4703) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.outputSpecificationBody(SharedDriveConnector.java:3170) at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:2252) I am able to perform successful mount with the window fileserver using cifs mount commands in linux. Could you please help me with resolving this window share issue. Please let me know if I am missing anything. Regards, Smith CAUTION - Disclaimer * This e-mail contains PRIVILEGED
Help need for connecting to window share
Hi, I am working on a project where I need to use the window file server for crawling documents. I am trying to use Window Share repository connection. I could configure the window share repository connection with server details and I managed to get Connection Working status. But while configuring the job using this repository, I am getting the following error in UI No starting points defined SmbException thrown: 0xC0BB In the tomcat server console I am getting following exception org.apache.manifoldcf.core.interfaces.ManifoldCFException: SmbException thrown: 0xC0BB at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processSMBException(SharedDriveConnector.java:1289) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getChildFolderNames(SharedDriveConnector.java:4703) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.outputSpecificationBody(SharedDriveConnector.java:3170) at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:2252) I am able to perform successful mount with the window fileserver using cifs mount commands in linux. Could you please help me with resolving this window share issue. Please let me know if I am missing anything. Regards, Smith CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS***
ManifoldCF with multiple Repository Connections.
Hi, I am working on project where I need to crawl multiple window file servers. What is the best way to handle this without having much performance impact. Option1 : Having multiple repository connection for each window share and corresponding jobs for crawling. Option 2: Mount the fileservers in one Linux folder and define one file system repository to handle all the fileservers. We can have single job or multiple job here. Please ponder your ideas. Please let me know if there is any other options. Regards, Smitha S CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS***
RE: Help need for connecting to window share
Thanks Karl for the response. I have my doubts here. I am using windows server 2008 as my repository server. The status of my repository connection with this window share says connection working. I hope I get error in establishing connection if the issue is due to credentials. I am getting the error when I go to PATH tab of job creation UI. Is there anything extra I need to do when using Window Share repository connection. Regards, Smitha From: Karl Wright [mailto:daddy...@gmail.com] Sent: Monday, March 3, 2014 1:05 PM To: user@manifoldcf.apache.org Cc: user-h...@manifoldcf.apache.org Subject: Re: Help need for connecting to window share Hi Smitha, This is apparently a credentials issue. The error is from the server, and apparently has to do with smart-card login. See: http://support.microsoft.com/kb/891849 and: http://social.technet.microsoft.com/Forums/windowsserver/en-US/e8cf1a1b--4eaa-b55a-cbfad3ef32d6/error-0xc0bb-in-smart-card-logon?forum=winserversecurity If this is an XP system, it means you need SP3 installed. If this is not an old XP system that you are crawling, I can find no indication of the circumstances leading to the error. Karl On Mon, Mar 3, 2014 at 1:14 AM, Smitha S smitha_...@infosys.commailto:smitha_...@infosys.com wrote: Hi, I am working on a project where I need to use the window file server for crawling documents. I am trying to use Window Share repository connection. I could configure the window share repository connection with server details and I managed to get Connection Working status. But while configuring the job using this repository, I am getting the following error in UI No starting points defined SmbException thrown: 0xC0BB In the tomcat server console I am getting following exception org.apache.manifoldcf.core.interfaces.ManifoldCFException: SmbException thrown: 0xC0BB at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processSMBException(SharedDriveConnector.java:1289) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getChildFolderNames(SharedDriveConnector.java:4703) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.outputSpecificationBody(SharedDriveConnector.java:3170) at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:2252) I am able to perform successful mount with the window fileserver using cifs mount commands in linux. Could you please help me with resolving this window share issue. Please let me know if I am missing anything. Regards, Smith CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS***
RE: ManifoldCF with multiple Repository Connections.
Hi Karl, So I think you are suggesting to create multiple window share repository connection for each window server and corresponding multiple jobs for crawling. Please correct me if I am wrong. Regards, Smitha S From: Karl Wright [mailto:daddy...@gmail.com] Sent: Monday, March 3, 2014 12:56 PM To: user@manifoldcf.apache.org Cc: user-h...@manifoldcf.apache.org Subject: Re: ManifoldCF with multiple Repository Connections. Hi Smitha, There is no ManifoldCF-related reason why you should need to create a special linux share just to crawl multiple windows shares. In fact, I suspect that would slow things down rather than speed them up. Karl On Mon, Mar 3, 2014 at 1:23 AM, Smitha S smitha_...@infosys.commailto:smitha_...@infosys.com wrote: Hi, I am working on project where I need to crawl multiple window file servers. What is the best way to handle this without having much performance impact. Option1 : Having multiple repository connection for each window share and corresponding jobs for crawling. Option 2: Mount the fileservers in one Linux folder and define one file system repository to handle all the fileservers. We can have single job or multiple job here. Please ponder your ideas. Please let me know if there is any other options. Regards, Smitha S CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS***
