You can enable repository connector debug logging by adding this to your properties.xml:
<property name="org.apache.manifoldcf.connectors" value="DEBUG"/> Having said that, the cleanup phase for all connectors is executed by the framework. We know the framework works because we have numerous integration tests that exercise it. But it's up to the ES connector to delete documents and log the fact that it is deleting documents. So I suspect that it is the ES connector's delete functionality that is not working properly. If you told me that *no* documents show up in the Simple History as being deleted during the cleanup phase, then there would obviously be a simple ES connector bug involved. But if there are multiple documents that *do* get deleted, it's more complex than that. Do you ever see *any* documents deleted during the cleanup phase in the Simple History with the ES connector? Another easy check is to set up exactly the same job but with the output going to the Null Output Connector. This connector definitely logs everything it sees. Compare and contrast vs the ES output connector. If you see a difference, it's likely a bug in the ES connector that we'll have to figure out. Thanks, Karl Karl On Mon, Oct 29, 2018 at 12:39 PM Gustavo Beneitez < gustavo.benei...@gmail.com> wrote: > Hi, > > we made a new test, job created several documents that never where removed > from Elastic Search after job deletion, and the Simple History never showed > them as deleted. > > I also looked for an error on logs without luck. > > I think it could be 2) case, can I increase log detail for web repository? > This, and the Elastic, are both default connectors, no code changes here. > > Thanks. > > El lun., 29 oct. 2018 a las 16:12, Karl Wright (<daddy...@gmail.com>) > escribió: > > > It is only possible if: > > > > (1) You run a job in a "minimal" configuration, or > > (2) There is a bug in either the repository connector that doesn't > properly > > signal the status of a deleted document to the pipeline, or > > (3) There is a bug in the output connector so that deletion of a document > > silently fails but is nevertheless reported as having succeeded. > > > > The way to figure this out is to look at the Simple History for one of > the > > documents you expect to have been deleted to see how it was handled. > > > > Thanks, > > Karl > > > > > > On Mon, Oct 29, 2018 at 11:06 AM Gustavo Beneitez < > > gustavo.benei...@gmail.com> wrote: > > > > > Hi Karl, > > > > > > after several tests I did manage to create, run and delete a job with > > > Elastic output connector, and all its documents where also deleted from > > > database while they were not deleted from repository. > > > > > > Under which cases is this possible? Maybe if they share repo? > > > > > > Thanks in advance! > > > > > > > > > El mié., 17 oct. 2018 a las 14:40, Gustavo Beneitez (< > > > gustavo.benei...@gmail.com>) escribió: > > > > > > > Ok thanks! > > > > > > > > El mié., 17 oct. 2018 a las 14:27, Karl Wright (<daddy...@gmail.com > >) > > > > escribió: > > > > > > > >> Ok, the schema is described in ManifoldCF In Action. > > > >> > > > >> https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs > > > >> > > > >> Karl > > > >> > > > >> > > > >> On Wed, Oct 17, 2018 at 7:41 AM Gustavo Beneitez < > > > >> gustavo.benei...@gmail.com> > > > >> wrote: > > > >> > > > >> > Hi Karl, > > > >> > > > > >> > as far as I was able to gather information from history records, I > > > could > > > >> > see MCF is behaving as expected. The "problem" shows when > > > ElasticSearch > > > >> is > > > >> > down or performing bad, MCF says it was requested to be deleted, > but > > > >> while > > > >> > it has been erased from database, it is alive on ElasticSearch > side, > > > so > > > >> I > > > >> > need to find whether or not there are those kind of > inconsistencies > > > >> exist. > > > >> > > > > >> > Please allow us to check those documents and make new tests in > order > > > to > > > >> see > > > >> > what really happens,we don't modify any database record by hand. > > > >> > > > > >> > Thanks! > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > El mar., 16 oct. 2018 a las 19:27, Karl Wright (< > daddy...@gmail.com > > >) > > > >> > escribió: > > > >> > > > > >> > > Hi, you can look at ManifoldCF In Action. There's a link to it > on > > > the > > > >> > > manifoldcf page. > > > >> > > > > > >> > > However, you should be aware that we consider it a severe bug if > > > >> > ManifoldCF > > > >> > > doesn't clean up after itself. The only time that is not > expected > > > is > > > >> > when > > > >> > > people write buggy connectors or mess with database tables > > > >> themselves. I > > > >> > > would urge you to examine the Simple History report and try to > > come > > > up > > > >> > with > > > >> > > a reproducible test case rather than trying to reverse engineer > > MCF. > > > >> > > Should you go directly to the database, we will be unable to > give > > > you > > > >> any > > > >> > > support. > > > >> > > > > > >> > > Thanks, > > > >> > > Karl > > > >> > > > > > >> > > > > > >> > > On Tue, Oct 16, 2018 at 11:51 AM Gustavo Beneitez < > > > >> > > gustavo.benei...@gmail.com> wrote: > > > >> > > > > > >> > > > Hi all, > > > >> > > > > > > >> > > > how do you do? I was wandering if there is any technical > > document > > > >> about > > > >> > > > what is the meaning of each table in database, the > relationship > > > >> between > > > >> > > > documents, repositories, jobs and any other output connector > > (some > > > >> kind > > > >> > > of > > > >> > > > a database model). > > > >> > > > > > > >> > > > We are facing some "garbage issues", jobs are created, > > duplicated, > > > >> > > related > > > >> > > > to transformations, linked to outputs (Elastic Search), played > > and > > > >> > > finally > > > >> > > > deleted, but in the end documents that should be also deleted > > > >> against > > > >> > the > > > >> > > > output connector, sometimes they still are there, don't know > if > > > >> they > > > >> > are > > > >> > > > visible because they point to an existing job, an unexpected > job > > > >> end or > > > >> > > any > > > >> > > > other failure. > > > >> > > > > > > >> > > > We need to understand the database model in order to check > when > > > >> > documents > > > >> > > > stored in Elastic can be safely removed since they no longer > are > > > >> > referred > > > >> > > > by any process. A process that should be executed periodically > > > every > > > >> > > week, > > > >> > > > for example. > > > >> > > > > > > >> > > > Thanks in advance! > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
