RE: Zookeeper locks issue

[Cedric Ulmer]

Hi,


would you also want to have a look at the disconnect() method ?


Cedric



De : Cedric Ulmer 
Envoyé : jeudi 6 janvier 2022 16:35:55
À : dev
Objet : RE: Zookeeper locks issue

Hi Karl,


to bounce back on this issue, since you asked about how we initiate the 
connections, here is how we implement the connect() method. We are talking here 
about a sharepoint datasource :


  public void connect(final ConfigParams configParams) {
super.connect(configParams);

spVersion = 
configParams.getParameter(SharePointConfiguration.Server.VERSION);
connectionType = 
configParams.getParameter(SharePointConfiguration.Server.CONNECTION_TYPE);
sts = configParams.getParameter(SharePointConfiguration.Server.STS);
protocol = 
configParams.getParameter(SharePointConfiguration.Server.PROTOCOL);
host = configParams.getParameter(SharePointConfiguration.Server.HOST);
port = configParams.getParameter(SharePointConfiguration.Server.PORT);
path = configParams.getParameter(SharePointConfiguration.Server.PATH);
username = 
configParams.getParameter(SharePointConfiguration.Server.USERNAME);
password = 
configParams.getObfuscatedParameter(SharePointConfiguration.Server.PASSWORD);
socketTimeoutString = 
configParams.getParameter(SharePointConfiguration.Server.SOCKET_TIMEOUT);
connectionTimeoutString = 
configParams.getParameter(SharePointConfiguration.Server.CONNECTION_TIMEOUT);

proxyHost = 
configParams.getParameter(SharePointConfiguration.Server.PROXY_HOST);
proxyPortStr = 
configParams.getParameter(SharePointConfiguration.Server.PROXY_PORT);
proxyUser = 
configParams.getParameter(SharePointConfiguration.Server.PROXY_USER);
proxyPassword = 
configParams.getObfuscatedParameter(SharePointConfiguration.Server.PROXY_PASSWORD);
proxyDomain = 
configParams.getParameter(SharePointConfiguration.Server.PROXY_DOMAIN);

sharePointUrl = protocol + "://" + host;
if (port != null && !port.isEmpty() && !port.equals("80") && 
!port.equals("443")) {
  sharePointUrl += ":" + port;
}
sharePointUrl += path;
if (sharePointUrl.endsWith("/")) {
  sharePointUrl = sharePointUrl.substring(0, sharePointUrl.length() - 1);
}

final String fullDelta = 
configParams.getParameter(SharePointConfiguration.Delta.FULL_DELTA);
if (fullDelta != null && !fullDelta.isEmpty()) {
  final boolean isFullDelta = Boolean.parseBoolean(fullDelta);
  if (isFullDelta) {
connectorModel = MODEL_ADD_CHANGE;
  } else {
connectorModel = MODEL_ADD_CHANGE_DELETE;
  }
} else {
  connectorModel = MODEL_ADD_CHANGE_DELETE;
}

  }

Regards,


Cédric


France Labs – Search Experts
Datafari – Discover our Version 5 released in 2021
www.datafari.com<http://www.datafari.com/>



De : Karl Wright 
Envoyé : vendredi 10 décembre 2021 15:32
À : dev
Objet : Re: Zookeeper locks issue

You haven't told me anything about the connectors involved in this job.
For every connector, there are connection pools of limited sizes,
partitioned by configuration parameters.  Throttling is done also on these
pools, and those require locks.

So if a connector has code for establishing a connection that can hang, it
can cause carnage.  Connection establishment in the connector contract
involves several methods, which I am sure you know.  If any one of those
can block, that is bad.  Instead they should just throw an exception.

Karl


On Fri, Dec 10, 2021 at 8:15 AM  wrote:

> Hi,
>
> Not sure to understand what you mean by establishing a connection, what
> kind of connection ? If you can explain to me when the agent needs to set
> and release locks, I'll be able to better investigate.
>
> Julien
>
> -Message d'origine-
> De : Karl Wright 
> Envoyé : jeudi 9 décembre 2021 15:37
> À : dev 
> Objet : Re: Zookeeper locks issue
>
> The fact that you only see this on one job is pretty clearly evidence that
> we are seeing a hang of some kind due something a specific connector or
> connection is doing.
>
> I'm going to have to guess wildly here to focus us on a productive path.
> What I want to rule out is a case where the connector hangs while
> establishing a connection.  If this can happen then I could well believe
> there would be a train wreck.  Is this something you can confirm or
> disprove?
>
> Karl
>
>
> On Thu, Dec 9, 2021 at 9:07 AM Julien Massiera <
> julien.massi...@francelabs.com> wrote:
>
> > Actually, I have several jobs, but only one job is running at a time,
> > and currently the error always happens on the same one. The problem is
> > that I can't access the environment in debug mode, I also can't
> > activate debug log because I am limited in log size, so the only th

RE: Zookeeper locks issue

[Cedric Ulmer]

Hi Karl,


to bounce back on this issue, since you asked about how we initiate the 
connections, here is how we implement the connect() method. We are talking here 
about a sharepoint datasource :


  public void connect(final ConfigParams configParams) {
super.connect(configParams);

spVersion = 
configParams.getParameter(SharePointConfiguration.Server.VERSION);
connectionType = 
configParams.getParameter(SharePointConfiguration.Server.CONNECTION_TYPE);
sts = configParams.getParameter(SharePointConfiguration.Server.STS);
protocol = 
configParams.getParameter(SharePointConfiguration.Server.PROTOCOL);
host = configParams.getParameter(SharePointConfiguration.Server.HOST);
port = configParams.getParameter(SharePointConfiguration.Server.PORT);
path = configParams.getParameter(SharePointConfiguration.Server.PATH);
username = 
configParams.getParameter(SharePointConfiguration.Server.USERNAME);
password = 
configParams.getObfuscatedParameter(SharePointConfiguration.Server.PASSWORD);
socketTimeoutString = 
configParams.getParameter(SharePointConfiguration.Server.SOCKET_TIMEOUT);
connectionTimeoutString = 
configParams.getParameter(SharePointConfiguration.Server.CONNECTION_TIMEOUT);

proxyHost = 
configParams.getParameter(SharePointConfiguration.Server.PROXY_HOST);
proxyPortStr = 
configParams.getParameter(SharePointConfiguration.Server.PROXY_PORT);
proxyUser = 
configParams.getParameter(SharePointConfiguration.Server.PROXY_USER);
proxyPassword = 
configParams.getObfuscatedParameter(SharePointConfiguration.Server.PROXY_PASSWORD);
proxyDomain = 
configParams.getParameter(SharePointConfiguration.Server.PROXY_DOMAIN);

sharePointUrl = protocol + "://" + host;
if (port != null && !port.isEmpty() && !port.equals("80") && 
!port.equals("443")) {
  sharePointUrl += ":" + port;
}
sharePointUrl += path;
if (sharePointUrl.endsWith("/")) {
  sharePointUrl = sharePointUrl.substring(0, sharePointUrl.length() - 1);
}

final String fullDelta = 
configParams.getParameter(SharePointConfiguration.Delta.FULL_DELTA);
if (fullDelta != null && !fullDelta.isEmpty()) {
  final boolean isFullDelta = Boolean.parseBoolean(fullDelta);
  if (isFullDelta) {
connectorModel = MODEL_ADD_CHANGE;
  } else {
connectorModel = MODEL_ADD_CHANGE_DELETE;
  }
} else {
  connectorModel = MODEL_ADD_CHANGE_DELETE;
}

  }

Regards,


Cédric


France Labs – Search Experts
Datafari – Discover our Version 5 released in 2021
www.datafari.com



De : Karl Wright 
Envoyé : vendredi 10 décembre 2021 15:32
À : dev
Objet : Re: Zookeeper locks issue

You haven't told me anything about the connectors involved in this job.
For every connector, there are connection pools of limited sizes,
partitioned by configuration parameters.  Throttling is done also on these
pools, and those require locks.

So if a connector has code for establishing a connection that can hang, it
can cause carnage.  Connection establishment in the connector contract
involves several methods, which I am sure you know.  If any one of those
can block, that is bad.  Instead they should just throw an exception.

Karl


On Fri, Dec 10, 2021 at 8:15 AM  wrote:

> Hi,
>
> Not sure to understand what you mean by establishing a connection, what
> kind of connection ? If you can explain to me when the agent needs to set
> and release locks, I'll be able to better investigate.
>
> Julien
>
> -Message d'origine-
> De : Karl Wright 
> Envoyé : jeudi 9 décembre 2021 15:37
> À : dev 
> Objet : Re: Zookeeper locks issue
>
> The fact that you only see this on one job is pretty clearly evidence that
> we are seeing a hang of some kind due something a specific connector or
> connection is doing.
>
> I'm going to have to guess wildly here to focus us on a productive path.
> What I want to rule out is a case where the connector hangs while
> establishing a connection.  If this can happen then I could well believe
> there would be a train wreck.  Is this something you can confirm or
> disprove?
>
> Karl
>
>
> On Thu, Dec 9, 2021 at 9:07 AM Julien Massiera <
> julien.massi...@francelabs.com> wrote:
>
> > Actually, I have several jobs, but only one job is running at a time,
> > and currently the error always happens on the same one. The problem is
> > that I can't access the environment in debug mode, I also can't
> > activate debug log because I am limited in log size, so the only thing
> > I can do, is to add specific logs in specific places in the code to
> > try to understand what is happening. Where would you suggest me to add
> > log entries to optimise our chances to spot the issue ?
> >
> > Julien
> >
> > Le 09/12/2021 à 13:27, Karl Wright a écrit :
> > > The large number of connections can happen but usually that means
> > something
> > > is stuck somewhere and there is a "train wreck" of other locks
> 

RE: GSOC - Mavenisation of MCF ?

[Cedric Ulmer]

Hi Karl and Piergiorgio,

Ok got it, was a bad idea!
Wrt connectors, not sure either, doing a good connector (incl. documentation) 
is rather time consuming, and 2 months flies very fast. The best a student 
could do would probably be to look at a source API and devise how to best use 
it to create a proper connector.

Regards,

Cedric
CEO
France Labs – Makers of Datafari Enteprise Search
Vainqueur du trophée du Jury à IMAgineDAY 2021


-Message d'origine-
De : Piergiorgio Lucidi  
Envoyé : lundi 8 février 2021 13:56
À : dev 
Objet : Re: GSOC - Mavenisation of MCF ?

I agree with Karl,
for GSoC purposes it's better to propose something related to an independent 
and well defined connector.
PJ

Il giorno lun 8 feb 2021 alle ore 12:57 Karl Wright  ha
scritto:

> There are already poms throughout.  However, release distribution 
> structure cannot be built with Maven at this time because the 
> directory structure of the final release artifact is complex, and 
> because we want individual external connector developers to be able to "add 
> into" this.
>
> In other words, it is far too big a project for a GSOC exercise.
>
> Karl
>
>
> On Mon, Feb 8, 2021 at 6:49 AM Cedric Ulmer 
> 
> wrote:
>
> > Hi Karl,
> >
> > With regards to the Google Summer of Code, we have a suggestion: 
> > would it be a good idea to propose as a subject the mavenisation of 
> > MCF ? If yes,
> is
> > it you who needs to apply since we as France Labs are not 
> > responsible for the Apache MCF project ? We can off course act as mentors 
> > of the student.
> > Also, if Manifoldians have other ideas for the GSOC, I think it's 
> > time to talk about it because the deadline for organizations is Feb. 19th.
> >
> > Cedric
> > CEO
> > France Labs - Makers of Datafari Enteprise Search< 
> > https://www.datafari.com/en> Winners of the trophy of the Jury at 
> > IMAgineDAY< https://www.ima-dt.org/ima/event/detail.html/idConf/938> 
> > 2021
> >
> >
> >
>


--
Piergiorgio

GSOC - Mavenisation of MCF ?

[Cedric Ulmer]

Hi Karl,

With regards to the Google Summer of Code, we have a suggestion: would it be a 
good idea to propose as a subject the mavenisation of MCF ? If yes, is it you 
who needs to apply since we as France Labs are not responsible for the Apache 
MCF project ? We can off course act as mentors of the student.
Also, if Manifoldians have other ideas for the GSOC, I think it's time to talk 
about it because the deadline for organizations is Feb. 19th.

Cedric
CEO
France Labs - Makers of Datafari Enteprise Search
Winners of the trophy of the Jury at 
IMAgineDAY 2021

RE: [VOTE] Release Apache ManifoldCF 2.18, RC0

[Cedric Ulmer]

Hi Karl,

We are not committers, but for info we have updated MCF to 2.18RC0 in our 
Datafari 5.0Beta to see what happens; we have tested the following and it runs 
fine:
1.  Indexing with the web connector
2.  Indexing with the file connector (without security)
3.  Using a transformation connector for the file connector job
4.  Using the API to backup and restore jobs
5.  Using the API to restart MCF
6.  Updating password (without accents since it still does not work)
7.  Running with Java 11 openjdk
8.  Output connector is Solr 8.5.2

Regards,

Cedric and Olivier 

France Labs – Makers of Datafari Enterprise Search
www.datafari.com

-Message d'origine-
De : Karl Wright  
Envoyé : dimanche 20 décembre 2020 21:20
À : dev 
Objet : Re: [VOTE] Release Apache ManifoldCF 2.18, RC0

Tests passed, and I was able to use the UI too.  +1 from me.

Karl


On Wed, Dec 16, 2020 at 1:44 PM Karl Wright  wrote:

> Hi,
>
> Please vote on whether to release Apache ManifoldCF 2.18, RC0.  The 
> release artifact can be found here:
> https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.
> 18
> .  There is also a release tag at
> https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.18-RC0 .
>
> This release has a brand new connector (the solr ingester connector), 
> as well as build support for Java 11.  We still do not guarantee that 
> all connectors operate properly under JDK 11, but hopefully we can get 
> feedback from this release sufficient to close that hole.
>
> Thanks,
> Karl
>
>

The SMBv2 topic - again :P

[Cedric Ulmer]

Hi Karl,

I noticed you had closed this ticket as won't fix 
(https://issues.apache.org/jira/browse/CONNECTORS-1498), until JCIFS implements 
SMBv2. From what I could gather on the JCIFS mailing list, this obviously will 
never happen. Still, there is now the fork of jcifs, namely jcifs-ng, that 
claims to be SMBv2 compatible. Any chance that this makes you take a look at an 
update of this connector ?

Regards,

Cedric

[jira] [Commented] (CONNECTORS-1508) Add support for French Language

[Cedric Ulmer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512133#comment-16512133
 ] 

Cedric Ulmer commented on CONNECTORS-1508:
--

[~kwri...@metacarta.com] yes you're right, seems like I completely overlooked 
the framework/ui-core ... Sorry for that. I won't have time for now to work on 
it, so I'll let for instance [~Moltroon] have a look if he has a need for it. 
Well at least, the remainder is translated already !

> Add support for French Language
> ---
>
> Key: CONNECTORS-1508
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1508
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: ManifoldCF 2.10
>Reporter: Cedric Ulmer
>Assignee: Karl Wright
>Priority: Minor
> Fix For: ManifoldCF 2.11
>
> Attachments: cedricmanifold_fr.zip
>
>
> Some users may need a French version of the ressource bundle. I attached a 
> preliminary translation that France Labs made some time ago (probably around 
> summer 2016), but that we halted due to lack of time (and priority). It is 
> probably almost complete, but some quality checking needs to be done. Note 
> also that I forgot to check the version when I did the translations, so 
> anyone interested would need to check any modifications that may have 
> occurred between this version and the current MCF version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CONNECTORS-1508) Add support for French Language

[Cedric Ulmer (JIRA)]

Cedric Ulmer created CONNECTORS-1508:


 Summary: Add support for French Language
 Key: CONNECTORS-1508
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1508
 Project: ManifoldCF
  Issue Type: Improvement
  Components: Documentation
Affects Versions: ManifoldCF 2.10
Reporter: Cedric Ulmer
 Fix For: ManifoldCF next
 Attachments: cedricmanifold_fr.zip

Some users may need a French version of the ressource bundle. I attached a 
preliminary translation that France Labs made some time ago (probably around 
summer 2016), but that we halted due to lack of time (and priority). It is 
probably almost complete, but some quality checking needs to be done. Note also 
that I forgot to check the version when I did the translations, so anyone 
interested would need to check any modifications that may have occurred between 
this version and the current MCF version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)