[jira] [Assigned] (CONNECTORS-1552) Apache ManifoldCF Elastic Connector for Basic Authorisation

[Karl Wright (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karl Wright reassigned CONNECTORS-1552:
---

Assignee: Steph van Schalkwyk  (was: Karl Wright)

> Apache ManifoldCF Elastic Connector for Basic Authorisation
> ---
>
> Key: CONNECTORS-1552
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1552
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Elastic Search connector
>Affects Versions: ManifoldCF 2.10
>Reporter: Krishna Agrawal
>Assignee: Steph van Schalkwyk
>Priority: Major
> Fix For: ManifoldCF 2.12
>
>
> We are using the Apache Manifold CF to connect the elastic search as our 
> Elastic server is protected url there is no way we are able to connect from 
> the Admin console.
> If we remove the authentication connector works well but we want to access by 
> passing username and password.
> Please guide us so that we can complete our set up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jira] [Commented] (CONNECTORS-1552) Apache ManifoldCF Elastic Connector for Basic Authorisation

[Steph van Schalkwyk]

I included all the fixes to ES and I have to debug before unleashing on an
unsuspecting audience.



*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk




On Mon, Oct 29, 2018 at 4:14 PM Karl Wright  wrote:

> If you have this ready, I can assign to you -- or take it yourself.
>
> Karl
>
>
> On Mon, Oct 29, 2018 at 3:33 PM Steph van Schalkwyk 
> wrote:
>
> > I'm working on that one as well. Bit of a fix with a client right now.
> Will
> > issue patch.
> >
> >
> >
> > *Steph van Schalkwyk*
> > Principal, Remcam Search Engines
> > +1.314.452. <+1+314+452+2896>2896st...@remcam.net
> http://remcam.net
> >  Skype: svanschalkwyk
> > 
> > 
> >
> >
> > On Mon, Oct 29, 2018 at 1:45 PM Karl Wright (JIRA) 
> > wrote:
> >
> > >
> > > [
> > >
> >
> https://issues.apache.org/jira/browse/CONNECTORS-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667589#comment-16667589
> > > ]
> > >
> > > Karl Wright commented on CONNECTORS-1552:
> > > -
> > >
> > > The ES connector does not currently support any ES authentication
> > > requirements whatsoever.  This is therefore an enhancement to the
> current
> > > connector, not a bug.  Enhancement requests are looked at based on time
> > and
> > > availability of the volunteers working on the ManifoldCF project.
> > >
> > > I would suggest that if you have time-critical need for a new feature,
> > you
> > > consider adding it yourself.  The earliest I could look at this would
> be
> > > next weekend and that is not guaranteed.
> > >
> > >
> > > > Apache ManifoldCF Elastic Connector for Basic Authorisation
> > > > ---
> > > >
> > > > Key: CONNECTORS-1552
> > > > URL:
> > > https://issues.apache.org/jira/browse/CONNECTORS-1552
> > > > Project: ManifoldCF
> > > >  Issue Type: Improvement
> > > >  Components: Elastic Search connector
> > > >Affects Versions: ManifoldCF 2.10
> > > >Reporter: Krishna Agrawal
> > > >Assignee: Karl Wright
> > > >Priority: Major
> > > > Fix For: ManifoldCF 2.12
> > > >
> > > >
> > > > We are using the Apache Manifold CF to connect the elastic search as
> > our
> > > Elastic server is protected url there is no way we are able to connect
> > from
> > > the Admin console.
> > > > If we remove the authentication connector works well but we want to
> > > access by passing username and password.
> > > > Please guide us so that we can complete our set up.
> > >
> > >
> > >
> > > --
> > > This message was sent by Atlassian JIRA
> > > (v7.6.3#76005)
> > >
> >
>

Re: [jira] [Commented] (CONNECTORS-1552) Apache ManifoldCF Elastic Connector for Basic Authorisation

[Karl Wright]

If you have this ready, I can assign to you -- or take it yourself.

Karl


On Mon, Oct 29, 2018 at 3:33 PM Steph van Schalkwyk 
wrote:

> I'm working on that one as well. Bit of a fix with a client right now. Will
> issue patch.
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>  Skype: svanschalkwyk
> 
> 
>
>
> On Mon, Oct 29, 2018 at 1:45 PM Karl Wright (JIRA) 
> wrote:
>
> >
> > [
> >
> https://issues.apache.org/jira/browse/CONNECTORS-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667589#comment-16667589
> > ]
> >
> > Karl Wright commented on CONNECTORS-1552:
> > -
> >
> > The ES connector does not currently support any ES authentication
> > requirements whatsoever.  This is therefore an enhancement to the current
> > connector, not a bug.  Enhancement requests are looked at based on time
> and
> > availability of the volunteers working on the ManifoldCF project.
> >
> > I would suggest that if you have time-critical need for a new feature,
> you
> > consider adding it yourself.  The earliest I could look at this would be
> > next weekend and that is not guaranteed.
> >
> >
> > > Apache ManifoldCF Elastic Connector for Basic Authorisation
> > > ---
> > >
> > > Key: CONNECTORS-1552
> > > URL:
> > https://issues.apache.org/jira/browse/CONNECTORS-1552
> > > Project: ManifoldCF
> > >  Issue Type: Improvement
> > >  Components: Elastic Search connector
> > >Affects Versions: ManifoldCF 2.10
> > >Reporter: Krishna Agrawal
> > >Assignee: Karl Wright
> > >Priority: Major
> > > Fix For: ManifoldCF 2.12
> > >
> > >
> > > We are using the Apache Manifold CF to connect the elastic search as
> our
> > Elastic server is protected url there is no way we are able to connect
> from
> > the Admin console.
> > > If we remove the authentication connector works well but we want to
> > access by passing username and password.
> > > Please guide us so that we can complete our set up.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v7.6.3#76005)
> >
>

Re: [jira] [Commented] (CONNECTORS-1552) Apache ManifoldCF Elastic Connector for Basic Authorisation

[Steph van Schalkwyk]

I'm working on that one as well. Bit of a fix with a client right now. Will
issue patch.



*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk




On Mon, Oct 29, 2018 at 1:45 PM Karl Wright (JIRA)  wrote:

>
> [
> https://issues.apache.org/jira/browse/CONNECTORS-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667589#comment-16667589
> ]
>
> Karl Wright commented on CONNECTORS-1552:
> -
>
> The ES connector does not currently support any ES authentication
> requirements whatsoever.  This is therefore an enhancement to the current
> connector, not a bug.  Enhancement requests are looked at based on time and
> availability of the volunteers working on the ManifoldCF project.
>
> I would suggest that if you have time-critical need for a new feature, you
> consider adding it yourself.  The earliest I could look at this would be
> next weekend and that is not guaranteed.
>
>
> > Apache ManifoldCF Elastic Connector for Basic Authorisation
> > ---
> >
> > Key: CONNECTORS-1552
> > URL:
> https://issues.apache.org/jira/browse/CONNECTORS-1552
> > Project: ManifoldCF
> >  Issue Type: Improvement
> >  Components: Elastic Search connector
> >Affects Versions: ManifoldCF 2.10
> >Reporter: Krishna Agrawal
> >Assignee: Karl Wright
> >Priority: Major
> > Fix For: ManifoldCF 2.12
> >
> >
> > We are using the Apache Manifold CF to connect the elastic search as our
> Elastic server is protected url there is no way we are able to connect from
> the Admin console.
> > If we remove the authentication connector works well but we want to
> access by passing username and password.
> > Please guide us so that we can complete our set up.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>

[jira] [Commented] (CONNECTORS-1552) Apache ManifoldCF Elastic Connector for Basic Authorisation

[Karl Wright (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667589#comment-16667589
 ] 

Karl Wright commented on CONNECTORS-1552:
-

The ES connector does not currently support any ES authentication requirements 
whatsoever.  This is therefore an enhancement to the current connector, not a 
bug.  Enhancement requests are looked at based on time and availability of the 
volunteers working on the ManifoldCF project.

I would suggest that if you have time-critical need for a new feature, you 
consider adding it yourself.  The earliest I could look at this would be next 
weekend and that is not guaranteed.


> Apache ManifoldCF Elastic Connector for Basic Authorisation
> ---
>
> Key: CONNECTORS-1552
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1552
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Elastic Search connector
>Affects Versions: ManifoldCF 2.10
>Reporter: Krishna Agrawal
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
>
> We are using the Apache Manifold CF to connect the elastic search as our 
> Elastic server is protected url there is no way we are able to connect from 
> the Admin console.
> If we remove the authentication connector works well but we want to access by 
> passing username and password.
> Please guide us so that we can complete our set up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (CONNECTORS-1552) Apache ManifoldCF Elastic Connector for Basic Authorisation

[Karl Wright (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karl Wright reassigned CONNECTORS-1552:
---

 Assignee: Karl Wright
 Priority: Major  (was: Blocker)
Fix Version/s: ManifoldCF 2.12
  Component/s: Elastic Search connector
   Issue Type: Improvement  (was: Bug)

> Apache ManifoldCF Elastic Connector for Basic Authorisation
> ---
>
> Key: CONNECTORS-1552
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1552
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Elastic Search connector
>Affects Versions: ManifoldCF 2.10
>Reporter: Krishna Agrawal
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
>
> We are using the Apache Manifold CF to connect the elastic search as our 
> Elastic server is protected url there is no way we are able to connect from 
> the Admin console.
> If we remove the authentication connector works well but we want to access by 
> passing username and password.
> Please guide us so that we can complete our set up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CONNECTORS-1552) Apache ManifoldCF Elastic Connector for Basic Authorisation

[Krishna Agrawal (JIRA)]

Krishna Agrawal created CONNECTORS-1552:
---

 Summary: Apache ManifoldCF Elastic Connector for Basic 
Authorisation
 Key: CONNECTORS-1552
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1552
 Project: ManifoldCF
  Issue Type: Bug
Affects Versions: ManifoldCF 2.10
Reporter: Krishna Agrawal


We are using the Apache Manifold CF to connect the elastic search as our 
Elastic server is protected url there is no way we are able to connect from the 
Admin console.

If we remove the authentication connector works well but we want to access by 
passing username and password.

Please guide us so that we can complete our set up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: ManifoldCF database model

[Karl Wright]

You can enable repository connector debug logging by adding this to your
properties.xml:



Having said that, the cleanup phase for all connectors is executed by the
framework.  We know the framework works because we have numerous
integration tests that exercise it.  But it's up to the ES connector to
delete documents and log the fact that it is deleting documents.  So I
suspect that it is the ES connector's delete functionality that is not
working properly.

If you told me that *no* documents show up in the Simple History as being
deleted during the cleanup phase, then there would obviously be a simple ES
connector bug involved.  But if there are multiple documents that *do* get
deleted, it's more complex than that.  Do you ever see *any* documents
deleted during the cleanup phase in the Simple History with the ES
connector?

Another easy check is to set up exactly the same job but with the output
going to the Null Output Connector.  This connector definitely logs
everything it sees.  Compare and contrast vs the ES output connector.  If
you see a difference, it's likely a bug in the ES connector that we'll have
to figure out.

Thanks,
Karl

Karl

On Mon, Oct 29, 2018 at 12:39 PM Gustavo Beneitez <
gustavo.benei...@gmail.com> wrote:

> Hi,
>
> we made a new test, job created several documents that never where removed
> from Elastic Search after job deletion, and the Simple History never showed
> them as deleted.
>
> I also looked for an error on logs without luck.
>
> I think it could be 2) case, can I increase log detail for web repository?
> This, and the Elastic, are both default connectors, no code changes here.
>
> Thanks.
>
> El lun., 29 oct. 2018 a las 16:12, Karl Wright ()
> escribió:
>
> > It is only possible if:
> >
> > (1) You run a job in a "minimal" configuration, or
> > (2) There is a bug in either the repository connector that doesn't
> properly
> > signal the status of a deleted document to the pipeline, or
> > (3) There is a bug in the output connector so that deletion of a document
> > silently fails but is nevertheless reported as having succeeded.
> >
> > The way to figure this out is to look at the Simple History for one of
> the
> > documents you expect to have been deleted to see how it was handled.
> >
> > Thanks,
> > Karl
> >
> >
> > On Mon, Oct 29, 2018 at 11:06 AM Gustavo Beneitez <
> > gustavo.benei...@gmail.com> wrote:
> >
> > > Hi Karl,
> > >
> > > after several tests I did manage to create, run and delete a job with
> > > Elastic output connector, and all its documents where also deleted from
> > > database while they were not deleted from repository.
> > >
> > > Under which cases is this possible? Maybe if they share repo?
> > >
> > > Thanks in advance!
> > >
> > >
> > > El mié., 17 oct. 2018 a las 14:40, Gustavo Beneitez (<
> > > gustavo.benei...@gmail.com>) escribió:
> > >
> > > > Ok thanks!
> > > >
> > > > El mié., 17 oct. 2018 a las 14:27, Karl Wright ( >)
> > > > escribió:
> > > >
> > > >> Ok, the schema is described in ManifoldCF In Action.
> > > >>
> > > >> https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs
> > > >>
> > > >> Karl
> > > >>
> > > >>
> > > >> On Wed, Oct 17, 2018 at 7:41 AM Gustavo Beneitez <
> > > >> gustavo.benei...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Karl,
> > > >> >
> > > >> > as far as I was able to gather information from history records, I
> > > could
> > > >> > see MCF is behaving as expected. The "problem" shows when
> > > ElasticSearch
> > > >> is
> > > >> > down or performing bad, MCF says it was requested to be deleted,
> but
> > > >> while
> > > >> > it has been erased from database, it is alive on ElasticSearch
> side,
> > > so
> > > >> I
> > > >> > need to find whether or not there are those kind of
> inconsistencies
> > > >> exist.
> > > >> >
> > > >> > Please allow us to check those documents and make new tests in
> order
> > > to
> > > >> see
> > > >> > what really happens,we don't modify any database record by hand.
> > > >> >
> > > >> > Thanks!
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > El mar., 16 oct. 2018 a las 19:27, Karl Wright (<
> daddy...@gmail.com
> > >)
> > > >> > escribió:
> > > >> >
> > > >> > > Hi, you can look at ManifoldCF In Action.  There's a link to it
> on
> > > the
> > > >> > > manifoldcf page.
> > > >> > >
> > > >> > > However, you should be aware that we consider it a severe bug if
> > > >> > ManifoldCF
> > > >> > > doesn't clean up after itself.  The only time that is not
> expected
> > > is
> > > >> > when
> > > >> > > people write buggy connectors or mess with database tables
> > > >> themselves.  I
> > > >> > > would urge you to examine the Simple History report and try to
> > come
> > > up
> > > >> > with
> > > >> > > a reproducible test case rather than trying to reverse engineer
> > MCF.
> > > >> > > Should you go directly to the database, we will be unable to
> give
> > > you
> > > >> any
> > > >> > > support.
> > > >> > >
> > > >> > 

Re: ManifoldCF database model

[Gustavo Beneitez]

Hi,

we made a new test, job created several documents that never where removed
from Elastic Search after job deletion, and the Simple History never showed
them as deleted.

I also looked for an error on logs without luck.

I think it could be 2) case, can I increase log detail for web repository?
This, and the Elastic, are both default connectors, no code changes here.

Thanks.

El lun., 29 oct. 2018 a las 16:12, Karl Wright ()
escribió:

> It is only possible if:
>
> (1) You run a job in a "minimal" configuration, or
> (2) There is a bug in either the repository connector that doesn't properly
> signal the status of a deleted document to the pipeline, or
> (3) There is a bug in the output connector so that deletion of a document
> silently fails but is nevertheless reported as having succeeded.
>
> The way to figure this out is to look at the Simple History for one of the
> documents you expect to have been deleted to see how it was handled.
>
> Thanks,
> Karl
>
>
> On Mon, Oct 29, 2018 at 11:06 AM Gustavo Beneitez <
> gustavo.benei...@gmail.com> wrote:
>
> > Hi Karl,
> >
> > after several tests I did manage to create, run and delete a job with
> > Elastic output connector, and all its documents where also deleted from
> > database while they were not deleted from repository.
> >
> > Under which cases is this possible? Maybe if they share repo?
> >
> > Thanks in advance!
> >
> >
> > El mié., 17 oct. 2018 a las 14:40, Gustavo Beneitez (<
> > gustavo.benei...@gmail.com>) escribió:
> >
> > > Ok thanks!
> > >
> > > El mié., 17 oct. 2018 a las 14:27, Karl Wright ()
> > > escribió:
> > >
> > >> Ok, the schema is described in ManifoldCF In Action.
> > >>
> > >> https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs
> > >>
> > >> Karl
> > >>
> > >>
> > >> On Wed, Oct 17, 2018 at 7:41 AM Gustavo Beneitez <
> > >> gustavo.benei...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Karl,
> > >> >
> > >> > as far as I was able to gather information from history records, I
> > could
> > >> > see MCF is behaving as expected. The "problem" shows when
> > ElasticSearch
> > >> is
> > >> > down or performing bad, MCF says it was requested to be deleted, but
> > >> while
> > >> > it has been erased from database, it is alive on ElasticSearch side,
> > so
> > >> I
> > >> > need to find whether or not there are those kind of inconsistencies
> > >> exist.
> > >> >
> > >> > Please allow us to check those documents and make new tests in order
> > to
> > >> see
> > >> > what really happens,we don't modify any database record by hand.
> > >> >
> > >> > Thanks!
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > El mar., 16 oct. 2018 a las 19:27, Karl Wright ( >)
> > >> > escribió:
> > >> >
> > >> > > Hi, you can look at ManifoldCF In Action.  There's a link to it on
> > the
> > >> > > manifoldcf page.
> > >> > >
> > >> > > However, you should be aware that we consider it a severe bug if
> > >> > ManifoldCF
> > >> > > doesn't clean up after itself.  The only time that is not expected
> > is
> > >> > when
> > >> > > people write buggy connectors or mess with database tables
> > >> themselves.  I
> > >> > > would urge you to examine the Simple History report and try to
> come
> > up
> > >> > with
> > >> > > a reproducible test case rather than trying to reverse engineer
> MCF.
> > >> > > Should you go directly to the database, we will be unable to give
> > you
> > >> any
> > >> > > support.
> > >> > >
> > >> > > Thanks,
> > >> > > Karl
> > >> > >
> > >> > >
> > >> > > On Tue, Oct 16, 2018 at 11:51 AM Gustavo Beneitez <
> > >> > > gustavo.benei...@gmail.com> wrote:
> > >> > >
> > >> > > > Hi all,
> > >> > > >
> > >> > > > how do you do? I was wandering if there is any technical
> document
> > >> about
> > >> > > > what is the meaning of each table in database, the relationship
> > >> between
> > >> > > > documents, repositories, jobs and any other output connector
> (some
> > >> kind
> > >> > > of
> > >> > > > a database model).
> > >> > > >
> > >> > > > We are facing some "garbage issues", jobs are created,
> duplicated,
> > >> > > related
> > >> > > > to transformations, linked to outputs (Elastic Search), played
> and
> > >> > > finally
> > >> > > > deleted, but in the end documents that should be also deleted
> > >> against
> > >> > the
> > >> > > > output connector,  sometimes they still are there, don't know if
> > >> they
> > >> > are
> > >> > > > visible because they point to an existing job, an unexpected job
> > >> end or
> > >> > > any
> > >> > > > other failure.
> > >> > > >
> > >> > > > We need to understand the database model in order to check when
> > >> > documents
> > >> > > > stored in Elastic can be safely removed since they no longer are
> > >> > referred
> > >> > > > by any process. A process that should be executed periodically
> > every
> > >> > > week,
> > >> > > > for example.
> > >> > > >
> > >> > > > Thanks in advance!
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: ManifoldCF database model

[Karl Wright]

It is only possible if:

(1) You run a job in a "minimal" configuration, or
(2) There is a bug in either the repository connector that doesn't properly
signal the status of a deleted document to the pipeline, or
(3) There is a bug in the output connector so that deletion of a document
silently fails but is nevertheless reported as having succeeded.

The way to figure this out is to look at the Simple History for one of the
documents you expect to have been deleted to see how it was handled.

Thanks,
Karl


On Mon, Oct 29, 2018 at 11:06 AM Gustavo Beneitez <
gustavo.benei...@gmail.com> wrote:

> Hi Karl,
>
> after several tests I did manage to create, run and delete a job with
> Elastic output connector, and all its documents where also deleted from
> database while they were not deleted from repository.
>
> Under which cases is this possible? Maybe if they share repo?
>
> Thanks in advance!
>
>
> El mié., 17 oct. 2018 a las 14:40, Gustavo Beneitez (<
> gustavo.benei...@gmail.com>) escribió:
>
> > Ok thanks!
> >
> > El mié., 17 oct. 2018 a las 14:27, Karl Wright ()
> > escribió:
> >
> >> Ok, the schema is described in ManifoldCF In Action.
> >>
> >> https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs
> >>
> >> Karl
> >>
> >>
> >> On Wed, Oct 17, 2018 at 7:41 AM Gustavo Beneitez <
> >> gustavo.benei...@gmail.com>
> >> wrote:
> >>
> >> > Hi Karl,
> >> >
> >> > as far as I was able to gather information from history records, I
> could
> >> > see MCF is behaving as expected. The "problem" shows when
> ElasticSearch
> >> is
> >> > down or performing bad, MCF says it was requested to be deleted, but
> >> while
> >> > it has been erased from database, it is alive on ElasticSearch side,
> so
> >> I
> >> > need to find whether or not there are those kind of inconsistencies
> >> exist.
> >> >
> >> > Please allow us to check those documents and make new tests in order
> to
> >> see
> >> > what really happens,we don't modify any database record by hand.
> >> >
> >> > Thanks!
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > El mar., 16 oct. 2018 a las 19:27, Karl Wright ()
> >> > escribió:
> >> >
> >> > > Hi, you can look at ManifoldCF In Action.  There's a link to it on
> the
> >> > > manifoldcf page.
> >> > >
> >> > > However, you should be aware that we consider it a severe bug if
> >> > ManifoldCF
> >> > > doesn't clean up after itself.  The only time that is not expected
> is
> >> > when
> >> > > people write buggy connectors or mess with database tables
> >> themselves.  I
> >> > > would urge you to examine the Simple History report and try to come
> up
> >> > with
> >> > > a reproducible test case rather than trying to reverse engineer MCF.
> >> > > Should you go directly to the database, we will be unable to give
> you
> >> any
> >> > > support.
> >> > >
> >> > > Thanks,
> >> > > Karl
> >> > >
> >> > >
> >> > > On Tue, Oct 16, 2018 at 11:51 AM Gustavo Beneitez <
> >> > > gustavo.benei...@gmail.com> wrote:
> >> > >
> >> > > > Hi all,
> >> > > >
> >> > > > how do you do? I was wandering if there is any technical document
> >> about
> >> > > > what is the meaning of each table in database, the relationship
> >> between
> >> > > > documents, repositories, jobs and any other output connector (some
> >> kind
> >> > > of
> >> > > > a database model).
> >> > > >
> >> > > > We are facing some "garbage issues", jobs are created, duplicated,
> >> > > related
> >> > > > to transformations, linked to outputs (Elastic Search), played and
> >> > > finally
> >> > > > deleted, but in the end documents that should be also deleted
> >> against
> >> > the
> >> > > > output connector,  sometimes they still are there, don't know if
> >> they
> >> > are
> >> > > > visible because they point to an existing job, an unexpected job
> >> end or
> >> > > any
> >> > > > other failure.
> >> > > >
> >> > > > We need to understand the database model in order to check when
> >> > documents
> >> > > > stored in Elastic can be safely removed since they no longer are
> >> > referred
> >> > > > by any process. A process that should be executed periodically
> every
> >> > > week,
> >> > > > for example.
> >> > > >
> >> > > > Thanks in advance!
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: ManifoldCF database model

[Gustavo Beneitez]

Hi Karl,

after several tests I did manage to create, run and delete a job with
Elastic output connector, and all its documents where also deleted from
database while they were not deleted from repository.

Under which cases is this possible? Maybe if they share repo?

Thanks in advance!


El mié., 17 oct. 2018 a las 14:40, Gustavo Beneitez (<
gustavo.benei...@gmail.com>) escribió:

> Ok thanks!
>
> El mié., 17 oct. 2018 a las 14:27, Karl Wright ()
> escribió:
>
>> Ok, the schema is described in ManifoldCF In Action.
>>
>> https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs
>>
>> Karl
>>
>>
>> On Wed, Oct 17, 2018 at 7:41 AM Gustavo Beneitez <
>> gustavo.benei...@gmail.com>
>> wrote:
>>
>> > Hi Karl,
>> >
>> > as far as I was able to gather information from history records, I could
>> > see MCF is behaving as expected. The "problem" shows when ElasticSearch
>> is
>> > down or performing bad, MCF says it was requested to be deleted, but
>> while
>> > it has been erased from database, it is alive on ElasticSearch side, so
>> I
>> > need to find whether or not there are those kind of inconsistencies
>> exist.
>> >
>> > Please allow us to check those documents and make new tests in order to
>> see
>> > what really happens,we don't modify any database record by hand.
>> >
>> > Thanks!
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > El mar., 16 oct. 2018 a las 19:27, Karl Wright ()
>> > escribió:
>> >
>> > > Hi, you can look at ManifoldCF In Action.  There's a link to it on the
>> > > manifoldcf page.
>> > >
>> > > However, you should be aware that we consider it a severe bug if
>> > ManifoldCF
>> > > doesn't clean up after itself.  The only time that is not expected is
>> > when
>> > > people write buggy connectors or mess with database tables
>> themselves.  I
>> > > would urge you to examine the Simple History report and try to come up
>> > with
>> > > a reproducible test case rather than trying to reverse engineer MCF.
>> > > Should you go directly to the database, we will be unable to give you
>> any
>> > > support.
>> > >
>> > > Thanks,
>> > > Karl
>> > >
>> > >
>> > > On Tue, Oct 16, 2018 at 11:51 AM Gustavo Beneitez <
>> > > gustavo.benei...@gmail.com> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > how do you do? I was wandering if there is any technical document
>> about
>> > > > what is the meaning of each table in database, the relationship
>> between
>> > > > documents, repositories, jobs and any other output connector (some
>> kind
>> > > of
>> > > > a database model).
>> > > >
>> > > > We are facing some "garbage issues", jobs are created, duplicated,
>> > > related
>> > > > to transformations, linked to outputs (Elastic Search), played and
>> > > finally
>> > > > deleted, but in the end documents that should be also deleted
>> against
>> > the
>> > > > output connector,  sometimes they still are there, don't know if
>> they
>> > are
>> > > > visible because they point to an existing job, an unexpected job
>> end or
>> > > any
>> > > > other failure.
>> > > >
>> > > > We need to understand the database model in order to check when
>> > documents
>> > > > stored in Elastic can be safely removed since they no longer are
>> > referred
>> > > > by any process. A process that should be executed periodically every
>> > > week,
>> > > > for example.
>> > > >
>> > > > Thanks in advance!
>> > > >
>> > >
>> >
>>
>