You can just change the setup provided you point to the same database. Thanks, Karl
On Mon, Dec 3, 2018 at 9:57 AM krishna agrawal <krish.a...@gmail.com> wrote: > thanks Karl, > > I have deployed in my local as simple example and in Dev and QA with the > recomendation of Dev Ops team we deployed as multiprocess file example we > had brief discussion about considering multiprocess-zk-example and at that > time we were unsure of multiprocess-zk-example. > > But i will check and let you know if we can change the set up now? > > One question do we need to do fresh install or can we upgrade to > multiprocess-zk-example? > > Thanks for anticipation. > > Thanks, > Krishna A > > On Sat, Dec 1, 2018 at 3:05 PM Karl Wright <daddy...@gmail.com> wrote: > > > Another thing: it's quite important to guarantee a working setup here, > > otherwise you're just wasting everyone's time. So, please base your > > installation on the multiprocess-zk-example. Start off by running the > > example as is, on a small test crawl. Once you know how it works, then > > move next to changing only what you have to -- namely, the database > > properties in the global properties file, to point to your MySQL > instance. > > Try that also on a small test case (crawl some files for instance), > before > > trying it on your large case. Every step of the way should work, and if > it > > doesn't, figure out why not before you move onto the next step. > > > > Thanks, > > Karl > > > > > > On Sat, Dec 1, 2018 at 2:59 PM Karl Wright <daddy...@gmail.com> wrote: > > > > > Zookeeper does not require a locking directory. It is a process that > > > synchronizes other processes, and they connect to it by port. > > > > > > Karl > > > > > > > > > On Sat, Dec 1, 2018 at 2:55 PM krishna agrawal <krish.a...@gmail.com> > > > wrote: > > > > > >> Thanks for the information. > > >> if we use Zookeeper how can we make sure all our ManifoldCF processes > > use > > >> same locking directory does it can be done at the configuration level > > >> while > > >> installing. > > >> > > >> thanks, > > >> Krishna A > > >> > > >> On Sat, Dec 1, 2018 at 1:39 PM Karl Wright <daddy...@gmail.com> > wrote: > > >> > > >> > That error is the result of the database not managing transactions > > >> > properly. It can occur if the locking system is not set up > properly, > > >> or if > > >> > you are using multiple agents processes and each process does not > have > > >> its > > >> > own ID. We have also seen it reported before just because MySQL > seems > > >> to > > >> > have bugs and sometimes writes are delayed or don't go through. > > >> > > > >> > My recommendation would be to: > > >> > (1) use zookeeper, not file locking > > >> > (2) Make sure all your ManifoldCF processes use the SAME locking > > >> directory > > >> > or Zookeeper instance > > >> > (3) If you are using multiple agents process, be certain that each > > such > > >> > process gets its own ID (as is done in the examples). > > >> > > > >> > Karl > > >> > > > >> > > > >> > On Sat, Dec 1, 2018 at 11:43 AM krishna agrawal < > krish.a...@gmail.com > > > > > >> > wrote: > > >> > > > >> > > Thanks Karl, > > >> > > > > >> > > I will take a look at it > > >> > > > > >> > > But there is the error keep on tossing at manifold log > > >> > > > > >> > > ERROR 2018-12-01T11:13:26,297 (Job reset thread) - Exception > tossed: > > >> > > Unexpected job status encountered: 33 > > >> > > org.apache.manifoldcf.core.interfaces.ManifoldCFException: > > Unexpected > > >> job > > >> > > status encountered: 33 > > >> > > at > > >> > > > > >> > > org.apache.manifoldcf.crawler.jobs.Jobs.returnJobToActive(Jobs.java:2145) > > >> > > ~[mcf-pull-agent.jar:?] > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.manifoldcf.crawler.jobs.JobManager.resetJobs(JobManager.java:8449) > > >> > > ~[mcf-pull-agent.jar:?] > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:77) > > >> > > [mcf-pull-agent.jar:?] > > >> > > > > >> > > Thanks, > > >> > > Krishna A > > >> > > > > >> > > > > >> > > On Fri, Nov 30, 2018 at 7:00 PM Karl Wright <daddy...@gmail.com> > > >> wrote: > > >> > > > > >> > > > Hi Krishna, > > >> > > > > > >> > > > First of all I suggest that you *not* use > > multiprocess-file-example, > > >> > and > > >> > > > instead use multiprocess-zk-example. > > >> > > > > > >> > > > Your symptoms suggest many possibilities. But if you move to > > >> Zookeeper > > >> > > we > > >> > > > will be able to eliminate dangling file locks as a complication. > > So > > >> > > please > > >> > > > do that first. > > >> > > > > > >> > > > Karl > > >> > > > > > >> > > > > > >> > > > On Fri, Nov 30, 2018 at 6:29 PM krishna agrawal < > > >> krish.a...@gmail.com> > > >> > > > wrote: > > >> > > > > > >> > > > > Yeah in our local set up we did Simple example but in server > we > > >> did > > >> > > > > multiprocess-file-example are you suggesting us to upgrade > from > > >> 2.10 > > >> > to > > >> > > > > 2.11 ? > > >> > > > > > > >> > > > > and we are using MY Sql database , > > >> > > > > > > >> > > > > So most of time i saw nothing is running and still it say job > is > > >> > > running > > >> > > > > and you have to wait for it to complete. > > >> > > > > > > >> > > > > and restarting also not helping. > > >> > > > > > > >> > > > > Any other solution woould be greatly appreciated. > > >> > > > > > > >> > > > > Thanks, > > >> > > > > Krishna A > > >> > > > > > > >> > > > > On Fri, Nov 30, 2018 at 10:50 AM Karl Wright < > > daddy...@gmail.com> > > >> > > wrote: > > >> > > > > > > >> > > > > > It also may be useful to start with the simple example, > which > > is > > >> > not > > >> > > > > > multiprocess, and get familiar with using ManifoldCF that > way, > > >> > before > > >> > > > you > > >> > > > > > try to go to a more complicated setup. > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Karl > > >> > > > > > > > >> > > > > > > > >> > > > > > On Fri, Nov 30, 2018 at 9:46 AM Karl Wright < > > daddy...@gmail.com > > >> > > > >> > > > wrote: > > >> > > > > > > > >> > > > > > > "simplified multi-process"? There is no such example. > > >> > > > > > > > > >> > > > > > > These are the examples available. Which one are you > using? > > >> > > > > > > > > >> > > > > > > 11/15/2018 03:40 AM <DIR> example > > >> > > > > > > 11/15/2018 03:40 AM <DIR> example-proprietary > > >> > > > > > > 11/15/2018 03:40 AM <DIR> > > >> multiprocess-file-example > > >> > > > > > > 11/15/2018 03:40 AM <DIR> > > >> > > > > > > multiprocess-file-example-proprietary > > >> > > > > > > 11/15/2018 03:40 AM <DIR> > > multiprocess-zk-example > > >> > > > > > > 11/15/2018 03:40 AM <DIR> > > >> > > > > > multiprocess-zk-example-proprietary > > >> > > > > > > > > >> > > > > > > Cleaning locks makes no sense unless you are using the > > >> > > > > multiprocess-file > > >> > > > > > > setup. This is deprecated, by the way, in favor of the > > >> Zookeeper > > >> > > > > setup. > > >> > > > > > > > > >> > > > > > > As for the buttons, please read: > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://manifoldcf.apache.org/release/release-2.11/en_US/end-user-documentation.html#outputs > > >> > > > > > > > > >> > > > > > > The buttons in question are "Reindex all..." and "Remove > > >> all..." > > >> > > > > > > > > >> > > > > > > Karl > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > On Fri, Nov 30, 2018 at 9:36 AM krishna agrawal < > > >> > > > krish.a...@gmail.com> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > >> We have deployed the Manifold using > > >> > > > > > >> > > >> > > > > > >> - Simplified multi-process model > > >> > > > > > >> > > >> > > > > > >> We did try clean up of lock Sh but that also did not > work. > > >> > > > > > >> > > >> > > > > > >> I dont have forget all document button in output > connector. > > >> > > > > > >> > > >> > > > > > >> [image: image.png] > > >> > > > > > >> > > >> > > > > > >> On Thu, Nov 29, 2018 at 6:52 PM Karl Wright < > > >> daddy...@gmail.com > > >> > > > > >> > > > > wrote: > > >> > > > > > >> > > >> > > > > > >>> Hi Krishna, > > >> > > > > > >>> > > >> > > > > > >>> Please give us some background as to how you've deployed > > >> > > > ManifoldCF. > > >> > > > > > Are > > >> > > > > > >>> you using one of the examples? If so, which one? > > >> > > > > > >>> > > >> > > > > > >>> The detailed answer to your question is: the job must > > delete > > >> > all > > >> > > > > > >>> documents > > >> > > > > > >>> it indexed before it can be deleted. That is the > typical > > >> way > > >> > > jobs > > >> > > > > > work. > > >> > > > > > >>> Thus, if you shut down the target of your output > > connection, > > >> > you > > >> > > > may > > >> > > > > be > > >> > > > > > >>> blocked in deleting your job. > > >> > > > > > >>> > > >> > > > > > >>> At that point, you can either (a) restart the target of > > your > > >> > > output > > >> > > > > > >>> connection, or (b) go to the "view" page for the output > > >> > > connection > > >> > > > > and > > >> > > > > > >>> click both of the "forget all documents" buttons on it. > > >> (b) is > > >> > > not > > >> > > > > > >>> recommended unless you really want to start over fresh > on > > >> your > > >> > > > output > > >> > > > > > >>> index. > > >> > > > > > >>> > > >> > > > > > >>> Thanks, > > >> > > > > > >>> Karl > > >> > > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> On Thu, Nov 29, 2018 at 3:21 PM krishna agrawal < > > >> > > > > krish.a...@gmail.com> > > >> > > > > > >>> wrote: > > >> > > > > > >>> > > >> > > > > > >>> > Hi We are facing issue of action button is not > available > > >> > > > > > >>> > > > >> > > > > > >>> > [image: image.png] > > >> > > > > > >>> > > > >> > > > > > >>> > I have stop the agent process but still i am not able > > to > > >> > > remove > > >> > > > > the > > >> > > > > > >>> job > > >> > > > > > >>> > it say it > > >> > > > > > >>> > > > >> > > > > > >>> > there should be some way to forcefully restart and > stop > > >> the > > >> > > > running > > >> > > > > > >>> > process ? > > >> > > > > > >>> > > > >> > > > > > >>> > Job 1542835910915 is busy; you must wait and/or shut > it > > >> down > > >> > > > before > > >> > > > > > >>> > deleting it > > >> > > > > > >>> > but there is no job running, and i am seeing this > > message > > >> > from > > >> > > > > past 3 > > >> > > > > > >>> days. > > >> > > > > > >>> > > > >> > > > > > >>> > is there any ways to clear this? > > >> > > > > > >>> > > > >> > > > > > >>> > > > >> > > > > > >>> > Any help in this matter will be appreciated. > > >> > > > > > >>> > > > >> > > > > > >>> > Thanks, > > >> > > > > > >>> > Krishna A > > >> > > > > > >>> > > > >> > > > > > >>> > > >> > > > > > >> > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
