Need a Velocity fix, then a patch release
Hi all, The release just pushed out the door has two fatal flaws. The first one is that the new LDAP mapper has a name collision with the Regexp mapper. This makes the examples not work; the system fails while registering connectors. Already fixed on trunk. The second problem is that we had conflicting versions of Velocity in the release. The older version of Velocity was used by the UI, and the newer by Tika, but because of the conflict we get exceptions in the UI randomly depending on load order of the jars. This is much harder to correct. I've made some progress with it but there are two outstanding problems that need to be solved. See: https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=17533362#comment-17533362 . If anyone has the time to look inside the Velocity configuration process to figure out how the ExtProperties object gets set up, and how I can send in an object to it at engine configuration time, I'd love to hear about it. I'm out of time for this weekend. Karl
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533362#comment-17533362
]
Karl Wright commented on CONNECTORS-1712:
-
The problem is that we allow velocity templates to be associated with a class
instance, and in order for that to work right somehow the class instance has to
get into our custom velocity resource loader. We were passing it as a
constructor argument but we no longer are allowed to instantiate it ourselves.
The velocity resource loader does receive an ExtProperties object which can
include a Class property, but I don't know how to set this in the velocity
properties so that it makes it through somehow to the resource loader.
There's also a fundamental change to how logging is done by Velocity that will
require work, but it is not insurmountable. The first problem is currently
insurmountable however and we may need to request Velocity changes to allow
this to be done again.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.Abs
[jira] [Assigned] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1712:
---
Assignee: Karl Wright
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
> [tomcat-util.jar:9.0.56]
> at
> org
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533358#comment-17533358
]
Karl Wright commented on CONNECTORS-1712:
-
So what changed was that Tika now requires Velocity 2.x and our UI requires
Velocity 1.7 still.
The upgrade is nowhere near as simple as just what is stated; I'm working on it
but seems like we need to fundamentally change much more about how we set up
Velocity under 2.x.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadP
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533351#comment-17533351
]
Karl Wright commented on CONNECTORS-1712:
-
Looking into why the old dependency is still downloaded. It should not be
unless it was not properly cleaned.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPo
[RESULT] [VOTE] Release Apache ManifoldCF 2.22, RC3
Three +1's, >72 hrs. Vote passes. Karl On Tue, May 3, 2022 at 11:06 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Ran all tests without error. +1 > > Julien > > -Message d'origine- > De : Karl Wright > Envoyé : mardi 3 mai 2022 12:09 > À : dev > Objet : Re: [VOTE] Release Apache ManifoldCF 2.22, RC3 > > Ran all the tests. Looks good! +1 from me. > > Karl > > > On Fri, Apr 29, 2022 at 10:34 AM Piergiorgio Lucidi < > piergior...@apache.org> > wrote: > > > Hi folks, > > > > ran all the tests correctly. > > +1 from me > > > > Cheers, > > PG > > > > Il giorno ven 29 apr 2022 alle ore 15:41 Karl Wright > > ha scritto: > > > > > Hi, > > > > > > Please vote on whether to release Apache ManifoldCF 2.22, RC3. The > > release > > > artifact is at > > > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf- > > > 2.22 > > . > > > There is also a release tag at > > > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC3 . > > > > > > This release contains a Tika update which required extensive > > > dependency updates, which needs some degree of testing, as well as a > > > new LDAP > > Mapper. > > > > > > RC1 also contains a CMIS fix from Piergiorgio, as well as an LDAP > > > Mapper pom fix. > > > RC2 has a fix for the MongoDB test download issue. > > > RC3 has several fixes for other connectors. > > > > > > Thanks in advance, > > > Karl > > > > > > > > > -- > > Piergiorgio > > > >
Re: [VOTE] Release Apache ManifoldCF 2.22, RC3
Ran all the tests. Looks good! +1 from me. Karl On Fri, Apr 29, 2022 at 10:34 AM Piergiorgio Lucidi wrote: > Hi folks, > > ran all the tests correctly. > +1 from me > > Cheers, > PG > > Il giorno ven 29 apr 2022 alle ore 15:41 Karl Wright > ha scritto: > > > Hi, > > > > Please vote on whether to release Apache ManifoldCF 2.22, RC3. The > release > > artifact is at > > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22 > . > > There is also a release tag at > > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC3 . > > > > This release contains a Tika update which required extensive dependency > > updates, which needs some degree of testing, as well as a new LDAP > Mapper. > > > > RC1 also contains a CMIS fix from Piergiorgio, as well as an LDAP Mapper > > pom fix. > > RC2 has a fix for the MongoDB test download issue. > > RC3 has several fixes for other connectors. > > > > Thanks in advance, > > Karl > > > > > -- > Piergiorgio >
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
ok, I know what the issue is. When I do the mcf release build, which does clean all the mcf downloaded artifacts, apparently it misses the test-materials directory under connectors/mongodb : Directory of C:\wip\mcf\release-2.22-branch\connectors\mongodb\test-materials 04/27/2022 09:07 AM . 04/27/2022 09:07 AM .. 04/22/2022 02:28 AM 120,551 de.flapdoodle.embed.mongo-2.2.0.jar 04/29/2022 09:19 AM 189,132 de.flapdoodle.embed.mongo-3.4.5.jar 04/29/2022 09:19 AM 111,113 de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar 04/22/2022 02:28 AM 160,837 de.flapdoodle.embed.process-2.0.5.jar 04/29/2022 09:19 AM 215,729 de.flapdoodle.embed.process-3.1.10.jar 04/29/2022 09:19 AM 107,870 de.flapdoodle.os-1.1.8.jar 04/18/2022 06:17 AM 1,197 README.txt 7 File(s)906,429 bytes 2 Dir(s) 104,271,101,952 bytes free Note the duplication here of various incompatible mongodb versions. We should fix the clean target in the mongodb/build.xml. But that's not a stop-ship. Karl On Fri, Apr 29, 2022 at 9:13 AM Karl Wright wrote: > Hmm, now I get the same thing as well. > Ok, let's call this fixed. I'll spin a new RC. > Karl > > > On Fri, Apr 29, 2022 at 8:57 AM Piergiorgio Lucidi > wrote: > >> Nope, no errors, I have a BUILD SUCCESSFUL >> >> Il giorno ven 29 apr 2022 alle ore 14:54 Karl Wright >> ha scritto: >> >> > once again, I have the same. >> > >> > Can you try this: >> > >> > cd connectors/mongodb >> > ant run-tests >> > >> > Do you get a compilation exception? >> > >> > Karl >> > >> > >> > On Fri, Apr 29, 2022 at 8:49 AM Piergiorgio Lucidi < >> piergior...@apache.org >> > > >> > wrote: >> > >> > > -rw-r--r-- 1 pjlucidi staff 613352 29 Apr 11:46 >> > > mongo-java-driver-2.14.3.jar >> > > >> > > Il giorno ven 29 apr 2022 alle ore 14:48 Karl Wright < >> daddy...@gmail.com >> > > >> > > ha scritto: >> > > >> > > > I have same: >> > > > >> > > > 04/28/2022 06:24 AM . >> > > > 04/28/2022 06:24 AM .. >> > > > 04/28/2022 06:24 AM 189,132 >> > > de.flapdoodle.embed.mongo-3.4.5.jar >> > > > 04/28/2022 06:24 AM 111,113 >> > > > de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar >> > > > 04/28/2022 06:24 AM 215,729 >> > > > de.flapdoodle.embed.process-3.1.10.jar >> > > > 04/28/2022 06:24 AM 107,870 de.flapdoodle.os-1.1.8.jar >> > > > 04/19/2020 05:47 AM 1,197 README.txt >> > > >5 File(s)625,041 bytes >> > > >2 Dir(s) 106,148,036,608 bytes free >> > > > >> > > > What version of mongo* do you have in your lib directory? >> > > > >> > > > Karl >> > > > >> > > > >> > > > On Fri, Apr 29, 2022 at 8:42 AM Piergiorgio Lucidi < >> > > piergior...@apache.org >> > > > > >> > > > wrote: >> > > > >> > > > > -rw-r--r-- 1 pjlucidi staff 107870 29 Apr 11:47 >> > > > > de.flapdoodle.os-1.1.8.jar >> > > > > -rw-r--r-- 1 pjlucidi staff 215729 29 Apr 11:47 >> > > > > de.flapdoodle.embed.process-3.1.10.jar >> > > > > -rw-r--r-- 1 pjlucidi staff 13 29 Apr 11:47 >> > > > > de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar >> > > > > -rw-r--r-- 1 pjlucidi staff 189132 29 Apr 11:47 >> > > > > de.flapdoodle.embed.mongo-3.4.5.jar >> > > > > >> > > > >> > > >> > > >> > > -- >> > > Piergiorgio >> > > >> > >> >> >> -- >> Piergiorgio >> >
[VOTE] Release Apache ManifoldCF 2.22, RC3
Hi, Please vote on whether to release Apache ManifoldCF 2.22, RC3. The release artifact is at https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22 . There is also a release tag at https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC3 . This release contains a Tika update which required extensive dependency updates, which needs some degree of testing, as well as a new LDAP Mapper. RC1 also contains a CMIS fix from Piergiorgio, as well as an LDAP Mapper pom fix. RC2 has a fix for the MongoDB test download issue. RC3 has several fixes for other connectors. Thanks in advance, Karl
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
Hmm, now I get the same thing as well. Ok, let's call this fixed. I'll spin a new RC. Karl On Fri, Apr 29, 2022 at 8:57 AM Piergiorgio Lucidi wrote: > Nope, no errors, I have a BUILD SUCCESSFUL > > Il giorno ven 29 apr 2022 alle ore 14:54 Karl Wright > ha scritto: > > > once again, I have the same. > > > > Can you try this: > > > > cd connectors/mongodb > > ant run-tests > > > > Do you get a compilation exception? > > > > Karl > > > > > > On Fri, Apr 29, 2022 at 8:49 AM Piergiorgio Lucidi < > piergior...@apache.org > > > > > wrote: > > > > > -rw-r--r-- 1 pjlucidi staff 613352 29 Apr 11:46 > > > mongo-java-driver-2.14.3.jar > > > > > > Il giorno ven 29 apr 2022 alle ore 14:48 Karl Wright < > daddy...@gmail.com > > > > > > ha scritto: > > > > > > > I have same: > > > > > > > > 04/28/2022 06:24 AM . > > > > 04/28/2022 06:24 AM .. > > > > 04/28/2022 06:24 AM 189,132 > > > de.flapdoodle.embed.mongo-3.4.5.jar > > > > 04/28/2022 06:24 AM 111,113 > > > > de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar > > > > 04/28/2022 06:24 AM 215,729 > > > > de.flapdoodle.embed.process-3.1.10.jar > > > > 04/28/2022 06:24 AM 107,870 de.flapdoodle.os-1.1.8.jar > > > > 04/19/2020 05:47 AM 1,197 README.txt > > > >5 File(s)625,041 bytes > > > >2 Dir(s) 106,148,036,608 bytes free > > > > > > > > What version of mongo* do you have in your lib directory? > > > > > > > > Karl > > > > > > > > > > > > On Fri, Apr 29, 2022 at 8:42 AM Piergiorgio Lucidi < > > > piergior...@apache.org > > > > > > > > > wrote: > > > > > > > > > -rw-r--r-- 1 pjlucidi staff 107870 29 Apr 11:47 > > > > > de.flapdoodle.os-1.1.8.jar > > > > > -rw-r--r-- 1 pjlucidi staff 215729 29 Apr 11:47 > > > > > de.flapdoodle.embed.process-3.1.10.jar > > > > > -rw-r--r-- 1 pjlucidi staff 13 29 Apr 11:47 > > > > > de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar > > > > > -rw-r--r-- 1 pjlucidi staff 189132 29 Apr 11:47 > > > > > de.flapdoodle.embed.mongo-3.4.5.jar > > > > > > > > > > > > > > > > > > -- > > > Piergiorgio > > > > > > > > -- > Piergiorgio >
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
once again, I have the same. Can you try this: cd connectors/mongodb ant run-tests Do you get a compilation exception? Karl On Fri, Apr 29, 2022 at 8:49 AM Piergiorgio Lucidi wrote: > -rw-r--r-- 1 pjlucidi staff 613352 29 Apr 11:46 > mongo-java-driver-2.14.3.jar > > Il giorno ven 29 apr 2022 alle ore 14:48 Karl Wright > ha scritto: > > > I have same: > > > > 04/28/2022 06:24 AM . > > 04/28/2022 06:24 AM .. > > 04/28/2022 06:24 AM 189,132 > de.flapdoodle.embed.mongo-3.4.5.jar > > 04/28/2022 06:24 AM 111,113 > > de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar > > 04/28/2022 06:24 AM 215,729 > > de.flapdoodle.embed.process-3.1.10.jar > > 04/28/2022 06:24 AM 107,870 de.flapdoodle.os-1.1.8.jar > > 04/19/2020 05:47 AM 1,197 README.txt > >5 File(s)625,041 bytes > >2 Dir(s) 106,148,036,608 bytes free > > > > What version of mongo* do you have in your lib directory? > > > > Karl > > > > > > On Fri, Apr 29, 2022 at 8:42 AM Piergiorgio Lucidi < > piergior...@apache.org > > > > > wrote: > > > > > -rw-r--r-- 1 pjlucidi staff 107870 29 Apr 11:47 > > > de.flapdoodle.os-1.1.8.jar > > > -rw-r--r-- 1 pjlucidi staff 215729 29 Apr 11:47 > > > de.flapdoodle.embed.process-3.1.10.jar > > > -rw-r--r-- 1 pjlucidi staff 13 29 Apr 11:47 > > > de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar > > > -rw-r--r-- 1 pjlucidi staff 189132 29 Apr 11:47 > > > de.flapdoodle.embed.mongo-3.4.5.jar > > > > > > > > -- > Piergiorgio >
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
I have same: 04/28/2022 06:24 AM . 04/28/2022 06:24 AM .. 04/28/2022 06:24 AM 189,132 de.flapdoodle.embed.mongo-3.4.5.jar 04/28/2022 06:24 AM 111,113 de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar 04/28/2022 06:24 AM 215,729 de.flapdoodle.embed.process-3.1.10.jar 04/28/2022 06:24 AM 107,870 de.flapdoodle.os-1.1.8.jar 04/19/2020 05:47 AM 1,197 README.txt 5 File(s)625,041 bytes 2 Dir(s) 106,148,036,608 bytes free What version of mongo* do you have in your lib directory? Karl On Fri, Apr 29, 2022 at 8:42 AM Piergiorgio Lucidi wrote: > -rw-r--r-- 1 pjlucidi staff 107870 29 Apr 11:47 > de.flapdoodle.os-1.1.8.jar > -rw-r--r-- 1 pjlucidi staff 215729 29 Apr 11:47 > de.flapdoodle.embed.process-3.1.10.jar > -rw-r--r-- 1 pjlucidi staff 13 29 Apr 11:47 > de.flapdoodle.embed.mongo.packageresolver-1.0.3.jar > -rw-r--r-- 1 pjlucidi staff 189132 29 Apr 11:47 > de.flapdoodle.embed.mongo-3.4.5.jar >
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
Hi Piergiorgio, can you list the jars in the connectors/mongodb/test-materials directory on the system that you ran on? Karl On Fri, Apr 29, 2022 at 6:51 AM Piergiorgio Lucidi wrote: > Hi folks, > > I tried again from scratch, downloading the RC source bundle and using the > following command, I had no issues at all: > ant make-core-deps make-deps build test > > Then I tried to re-execute all the lifecycle with the following command and > again with no issues at all: > ant clean clean-deps clean-core-deps make-core-deps make-deps build test > > Unfortunately it seems that I can't reproduce this issue :-P > Any ideas? > > Cheers, > PG > > Il giorno gio 28 apr 2022 alle ore 13:12 Karl Wright > ha scritto: > > > The issue with MongoDB is a missing jar. That jar contains the classes > > used to execute MongoDB locally. These are no longer found in the driver > > jar. > > > > I won't have time for the next week to figure out where the needed > classes > > live now. > > > > Karl > > > > On Thu, Apr 28, 2022 at 6:37 AM Karl Wright wrote: > > > > > The mongo java driver version may need to change, or maybe it's already > > > changed and the code hasn't been updated to follow. Not sure. Still > > > researching. > > > > > > > > > Karl > > > > > > > > > On Thu, Apr 28, 2022 at 4:57 AM Julien Massiera < > > > julien.massi...@francelabs.com> wrote: > > > > > >> Hi everyone, > > >> > > >> I still have a problem with the mongodb tests, here are the ant > commands > > >> I execute: > > >> > > >> - ant clean > > >> - ant clean-core-deps > > >> - ant clean-deps > > >> - ant make-core-deps > > >> - ant make-deps > > >> - ant test > > >> > > >> Still I have the following error: > > >> > > >> compile-tests: > > >> [javac] Compiling 9 source files to > > >> > > > /home/francelabs/workspace/mcf-svn/connectors/mongodb/build/connector-tests/classes > > >> [javac] warning: [options] bootstrap class path not set in > > >> conjunction with -source 8 > > >> [javac] > > >> > > > /home/francelabs/workspace/mcf-svn/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/BaseITHSQLDB.java:71: > > >> error: incompatible types: MongodConfig cannot be converted to > > IMongodConfig > > >> [javac] mongodExecutable = > > starter.prepare(mongodConfig); > > >> [javac]^ > > >> [javac] Note: Some messages have been simplified; recompile with > > >> -Xdiags:verbose to get full output > > >> [javac] 1 error > > >> [javac] 1 warning > > >> > > >> BUILD FAILED > > >> > > >> Please let me know if I missed something > > >> > > >> Julien > > >> > > >> -Message d'origine- > > >> De : Karl Wright > > >> Envoyé : mercredi 27 avril 2022 17:43 > > >> À : dev > > >> Objet : Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2 > > >> > > >> Perhaps the user didn't actually make the proper deps? There is a > check > > >> in the ant build for whether the dependency is there, but I bet it > goes > > >> against the original downloaded file, not the patched one. > > >> > > >> Karl > > >> > > >> > > >> On Wed, Apr 27, 2022 at 11:30 AM Piergiorgio Lucidi < > > >> piergior...@apache.org> > > >> wrote: > > >> > > >> > Hi folks, > > >> > > > >> > I have just finished the ant test execution and for me no issues at > > all: > > >> > > > >> > BUILD SUCCESSFUL > > >> > > Total time: 41 minutes 58 seconds > > >> > > > >> > > > >> > I would say +1 from me but now I would like to understand how it is > > >> > possible to have that problem on CMIS Output Connector, I tried to > > >> > rerun this specific test many times but I don't see this issue. > > >> > > > >> > I'm investigating... > > >> > > > >> > Cheers, > > >> > PJ
Re: Job Service Interruption- and stops
" repeated service interruption" means that it happens again and again. For this particular document, the problem is that the error we are seeing is: "The process cannot access the file because it is being used by another process." ManifoldCF assumes that if it retries enough it should be able to read the document eventually. In this case, if it cannot read the document after 6+ hours, it assumes something is wrong and stops the job. We can make it continue at this point but the issue is that you shouldn't be seeing such an error for such a long period of time. Perhaps you might want to research why this is taking place. Karl On Fri, Apr 29, 2022 at 4:54 AM ritika jain wrote: > Hi All, > > With the window shares connector, on the server I am getting this > exception and due to repeated service interruption *job stops.* > > Error: Repeated service interruptions - failure processing document: The > process cannot access the file because it is being used by another process. > > How we can prevent this. > I read in the code that it retries the document. But due to service > interruptions, the jobs stopped. > > > Thanks > Ritika >
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
The issue with MongoDB is a missing jar. That jar contains the classes used to execute MongoDB locally. These are no longer found in the driver jar. I won't have time for the next week to figure out where the needed classes live now. Karl On Thu, Apr 28, 2022 at 6:37 AM Karl Wright wrote: > The mongo java driver version may need to change, or maybe it's already > changed and the code hasn't been updated to follow. Not sure. Still > researching. > > > Karl > > > On Thu, Apr 28, 2022 at 4:57 AM Julien Massiera < > julien.massi...@francelabs.com> wrote: > >> Hi everyone, >> >> I still have a problem with the mongodb tests, here are the ant commands >> I execute: >> >> - ant clean >> - ant clean-core-deps >> - ant clean-deps >> - ant make-core-deps >> - ant make-deps >> - ant test >> >> Still I have the following error: >> >> compile-tests: >> [javac] Compiling 9 source files to >> /home/francelabs/workspace/mcf-svn/connectors/mongodb/build/connector-tests/classes >> [javac] warning: [options] bootstrap class path not set in >> conjunction with -source 8 >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/BaseITHSQLDB.java:71: >> error: incompatible types: MongodConfig cannot be converted to IMongodConfig >> [javac] mongodExecutable = starter.prepare(mongodConfig); >> [javac]^ >> [javac] Note: Some messages have been simplified; recompile with >> -Xdiags:verbose to get full output >> [javac] 1 error >> [javac] 1 warning >> >> BUILD FAILED >> >> Please let me know if I missed something >> >> Julien >> >> -Message d'origine- >> De : Karl Wright >> Envoyé : mercredi 27 avril 2022 17:43 >> À : dev >> Objet : Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2 >> >> Perhaps the user didn't actually make the proper deps? There is a check >> in the ant build for whether the dependency is there, but I bet it goes >> against the original downloaded file, not the patched one. >> >> Karl >> >> >> On Wed, Apr 27, 2022 at 11:30 AM Piergiorgio Lucidi < >> piergior...@apache.org> >> wrote: >> >> > Hi folks, >> > >> > I have just finished the ant test execution and for me no issues at all: >> > >> > BUILD SUCCESSFUL >> > > Total time: 41 minutes 58 seconds >> > >> > >> > I would say +1 from me but now I would like to understand how it is >> > possible to have that problem on CMIS Output Connector, I tried to >> > rerun this specific test many times but I don't see this issue. >> > >> > I'm investigating... >> > >> > Cheers, >> > PJ >> > >> > Il giorno mer 27 apr 2022 alle ore 16:29 Markus Schuch < >> > markus_sch...@web.de> ha scritto: >> > >> > > Hi, >> > > >> > > before we spin the next RC WE should also check the ant precommit >> task. >> > > >> > > Travis Ci reports a failing task in the release branch and in trunk. >> > > >> > > Another issue i found: Changes.txt seems not reflecting all the >> > > resolved issues. >> > > >> > > Cheers, >> > > Markus >> > > >> > > >> > > Am 27. April 2022 16:22:06 MESZ schrieb Karl Wright >> > >> > >: >> > > >On Wed, Apr 27, 2022 at 10:21 AM Karl Wright >> > wrote: >> > > > >> > > >> This is Piergiorgio's patch; I think he missed something. >> > Piergiorgio? >> > > >> >> > > >> Karl >> > > >> >> > > >> >> > > >> On Wed, Apr 27, 2022 at 10:19 AM Julien Massiera < >> > > >> julien.massi...@francelabs.com> wrote: >> > > >> >> > > >>> Build is ok but I still have a problem with the cmisoutput test : >> > > >>> >> > > >>> run-IT-HSQLDB: >> > > >>> [junit] Testsuite: >> > > >>> >> > org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT >> > > >>> [junit] Configuration file successfully read >> > > >>> [junit] [main] INFO org.eclipse.jetty.util.log - Logging >> > > in
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
The mongo java driver version may need to change, or maybe it's already changed and the code hasn't been updated to follow. Not sure. Still researching. Karl On Thu, Apr 28, 2022 at 4:57 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi everyone, > > I still have a problem with the mongodb tests, here are the ant commands I > execute: > > - ant clean > - ant clean-core-deps > - ant clean-deps > - ant make-core-deps > - ant make-deps > - ant test > > Still I have the following error: > > compile-tests: > [javac] Compiling 9 source files to > /home/francelabs/workspace/mcf-svn/connectors/mongodb/build/connector-tests/classes > [javac] warning: [options] bootstrap class path not set in conjunction > with -source 8 > [javac] > /home/francelabs/workspace/mcf-svn/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/BaseITHSQLDB.java:71: > error: incompatible types: MongodConfig cannot be converted to IMongodConfig > [javac] mongodExecutable = starter.prepare(mongodConfig); > [javac]^ > [javac] Note: Some messages have been simplified; recompile with > -Xdiags:verbose to get full output > [javac] 1 error > [javac] 1 warning > > BUILD FAILED > > Please let me know if I missed something > > Julien > > -Message d'origine- > De : Karl Wright > Envoyé : mercredi 27 avril 2022 17:43 > À : dev > Objet : Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2 > > Perhaps the user didn't actually make the proper deps? There is a check > in the ant build for whether the dependency is there, but I bet it goes > against the original downloaded file, not the patched one. > > Karl > > > On Wed, Apr 27, 2022 at 11:30 AM Piergiorgio Lucidi < > piergior...@apache.org> > wrote: > > > Hi folks, > > > > I have just finished the ant test execution and for me no issues at all: > > > > BUILD SUCCESSFUL > > > Total time: 41 minutes 58 seconds > > > > > > I would say +1 from me but now I would like to understand how it is > > possible to have that problem on CMIS Output Connector, I tried to > > rerun this specific test many times but I don't see this issue. > > > > I'm investigating... > > > > Cheers, > > PJ > > > > Il giorno mer 27 apr 2022 alle ore 16:29 Markus Schuch < > > markus_sch...@web.de> ha scritto: > > > > > Hi, > > > > > > before we spin the next RC WE should also check the ant precommit task. > > > > > > Travis Ci reports a failing task in the release branch and in trunk. > > > > > > Another issue i found: Changes.txt seems not reflecting all the > > > resolved issues. > > > > > > Cheers, > > > Markus > > > > > > > > > Am 27. April 2022 16:22:06 MESZ schrieb Karl Wright > > > > >: > > > >On Wed, Apr 27, 2022 at 10:21 AM Karl Wright > > wrote: > > > > > > > >> This is Piergiorgio's patch; I think he missed something. > > Piergiorgio? > > > >> > > > >> Karl > > > >> > > > >> > > > >> On Wed, Apr 27, 2022 at 10:19 AM Julien Massiera < > > > >> julien.massi...@francelabs.com> wrote: > > > >> > > > >>> Build is ok but I still have a problem with the cmisoutput test : > > > >>> > > > >>> run-IT-HSQLDB: > > > >>> [junit] Testsuite: > > > >>> > > org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT > > > >>> [junit] Configuration file successfully read > > > >>> [junit] [main] INFO org.eclipse.jetty.util.log - Logging > > > initialized > > > >>> @1550ms to org.eclipse.jetty.util.log.Slf4jLog > > > >>> [junit] [main] INFO org.eclipse.jetty.server.Server - > > > >>> jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git: > > > >>> a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm > > > >>> 11.0.15+10-Ubuntu-0ubuntu0.20.04.1 > > > >>> [junit] [main] INFO org.eclipse.jetty.server.session - > > > >>> DefaultSessionIdManager workerName=node0 > > > >>> [junit] [main] INFO org.eclipse.jetty.server.session - No > > > >>> SessionScavenger set, using defaults > > > >>> [junit] [main] INFO org.eclipse.jetty.server.sess
Are we still waiting for a CMIS fix?
The reported CMIS problem was that the ant build could not find the "*_patched" version of the file. Piergiorgio, this seemed to me like it was operator error - the user neglected to run the updated ant command "ant make deps" and thus the patched version of the CMIS jar was not available. If this seems correct, please confirm and I'll spin a new RC. Testers, please also confirm whether you saw this even after you properly did ant clean ; ant clean-core-deps ; ant make-core-deps ; ant make-deps on your workarea. Or, if you see this at all on a fresh checkout. Thanks, Karl
Re: [VOTE] Release Apache ManifoldCF 2.22, RC2
The 2.22 release artifacts got inadvertently committed to that release branch but they have been deleted. We can't remove them from SVN history however, because we don't have the rights. If you committed to trunk I will try to pull up your changes to the release branch. Karl On Wed, Apr 27, 2022 at 3:11 PM Markus Schuch wrote: > Hi, > > I still miss the 2.22 release in JIRA to be able to tag all the issues > properly. Can someone create this? I don't have the rights. > > Another open issue is that the 2.22 release branch causes a problem with > the GitHub sync: https://issues.apache.org/jira/browse/INFRA-23212 > Somehow we have to get the big blobs out of the SVN history. Any ideas? > > Cheers, > Markus > > Am 27.04.2022 um 15:50 schrieb Karl Wright: > > Hi, > > > > Please vote on whether to release Apache ManifoldCF 2.22, RC2. The > release > > artifact is at > > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22 > . > > There is also a release tag at > > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC2 . > > > > This release contains a Tika update which required extensive dependency > > updates, which needs some degree of testing, as well as a new LDAP > Mapper. > > > > RC1 also contains a CMIS fix from Piergiorgio, as well as an LDAP Mapper > > pom fix. > > RC2 has a fix for the MongoDB test download issue. > > > > Thanks in advance, > > Karl > > >
[jira] [Commented] (CONNECTORS-1622) Upgrade to Tika 1.28.x
[ https://issues.apache.org/jira/browse/CONNECTORS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528999#comment-17528999 ] Karl Wright commented on CONNECTORS-1622: - yes. Resolving. > Upgrade to Tika 1.28.x > -- > > Key: CONNECTORS-1622 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1622 > Project: ManifoldCF > Issue Type: Improvement > Components: Tika extractor >Affects Versions: ManifoldCF 2.13 >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF next > > > Tika has released 1.28.1. Changes can be found from here: > [http://tika.apache.org/1.28.1/|http://tika.apache.org/1.23/] > > please see :[TIKA-3625|https://issues.apache.org/jira/browse/TIKA-3625] and > CONNECTORS-1683 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (CONNECTORS-1622) Upgrade to Tika 1.28.x
[ https://issues.apache.org/jira/browse/CONNECTORS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1622. - Fix Version/s: ManifoldCF 2.21 (was: ManifoldCF next) Resolution: Fixed > Upgrade to Tika 1.28.x > -- > > Key: CONNECTORS-1622 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1622 > Project: ManifoldCF > Issue Type: Improvement > Components: Tika extractor >Affects Versions: ManifoldCF 2.13 >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.21 > > > Tika has released 1.28.1. Changes can be found from here: > [http://tika.apache.org/1.28.1/|http://tika.apache.org/1.23/] > > please see :[TIKA-3625|https://issues.apache.org/jira/browse/TIKA-3625] and > CONNECTORS-1683 -- This message was sent by Atlassian Jira (v8.20.7#820007)
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
Perhaps the user didn't actually make the proper deps? There is a check in
the ant build for whether the dependency is there, but I bet it goes
against the original downloaded file, not the patched one.
Karl
On Wed, Apr 27, 2022 at 11:30 AM Piergiorgio Lucidi
wrote:
> Hi folks,
>
> I have just finished the ant test execution and for me no issues at all:
>
> BUILD SUCCESSFUL
> > Total time: 41 minutes 58 seconds
>
>
> I would say +1 from me but now I would like to understand how it is
> possible to have that problem on CMIS Output Connector, I tried to rerun
> this specific test many times but I don't see this issue.
>
> I'm investigating...
>
> Cheers,
> PJ
>
> Il giorno mer 27 apr 2022 alle ore 16:29 Markus Schuch <
> markus_sch...@web.de>
> ha scritto:
>
> > Hi,
> >
> > before we spin the next RC WE should also check the ant precommit task.
> >
> > Travis Ci reports a failing task in the release branch and in trunk.
> >
> > Another issue i found: Changes.txt seems not reflecting all the resolved
> > issues.
> >
> > Cheers,
> > Markus
> >
> >
> > Am 27. April 2022 16:22:06 MESZ schrieb Karl Wright >:
> > >On Wed, Apr 27, 2022 at 10:21 AM Karl Wright
> wrote:
> > >
> > >> This is Piergiorgio's patch; I think he missed something.
> Piergiorgio?
> > >>
> > >> Karl
> > >>
> > >>
> > >> On Wed, Apr 27, 2022 at 10:19 AM Julien Massiera <
> > >> julien.massi...@francelabs.com> wrote:
> > >>
> > >>> Build is ok but I still have a problem with the cmisoutput test :
> > >>>
> > >>> run-IT-HSQLDB:
> > >>> [junit] Testsuite:
> > >>>
> org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT
> > >>> [junit] Configuration file successfully read
> > >>> [junit] [main] INFO org.eclipse.jetty.util.log - Logging
> > initialized
> > >>> @1550ms to org.eclipse.jetty.util.log.Slf4jLog
> > >>> [junit] [main] INFO org.eclipse.jetty.server.Server -
> > >>> jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git:
> > >>> a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm
> > >>> 11.0.15+10-Ubuntu-0ubuntu0.20.04.1
> > >>> [junit] [main] INFO org.eclipse.jetty.server.session -
> > >>> DefaultSessionIdManager workerName=node0
> > >>> [junit] [main] INFO org.eclipse.jetty.server.session - No
> > >>> SessionScavenger set, using defaults
> > >>> [junit] [main] INFO org.eclipse.jetty.server.session - node0
> > >>> Scavenging every 66ms
> > >>> [junit] [main] INFO
> > org.eclipse.jetty.server.handler.ContextHandler -
> > >>> Started o.e.j.w.WebAppContext@27a0a5a2{ManifoldCF Crawler
> > >>>
> >
> Interface,/mcf-crawler-ui,file:///tmp/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-11141316626452716674.dir/webapp/,AVAILABLE}{/home/francelabs/workspace/mcf-svn/dist/web/war/mcf-crawler-ui.war}
> > >>> [junit] [main] INFO
> > org.eclipse.jetty.server.handler.ContextHandler -
> > >>> Started o.e.j.w.WebAppContext@7692cd34{ManifoldCF Authorities API
> > >>>
> >
> Webapp,/mcf-authority-service,file:///tmp/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-17202752865589121447.dir/webapp/,AVAILABLE}{/home/francelabs/workspace/mcf-svn/dist/web/war/mcf-authority-service.war}
> > >>> [junit] [main] INFO
> > org.eclipse.jetty.server.handler.ContextHandler -
> > >>> Started o.e.j.w.WebAppContext@33aa93c{ManifoldCF General API
> > >>>
> >
> Webapp,/mcf-api-service,file:///tmp/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-7754762847799735649.dir/webapp/,AVAILABLE}{/home/francelabs/workspace/mcf-svn/dist/web/war/mcf-api-service.war}
> > >>> [junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
> > >>> Started ServerConnector@120f38e6{HTTP/1.1,[http/1.1]}{0.0.0.0:8346}
> > >>> [junit] [main] INFO org.eclipse.jetty.server.Server - Started
> > @2576ms
> > >>> [junit] [main] INFO org.eclipse.jetty.server.Server -
> > >>> jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git:
> > >>> a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm
> > >>> 11.0.15+10-Ubuntu-0ubuntu0.20.04.1
> > >>> [junit] [main] WARN org.eclipse.jetty.webapp.WebInfConfiguration
> -
[CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC2
On Wed, Apr 27, 2022 at 10:21 AM Karl Wright wrote:
> This is Piergiorgio's patch; I think he missed something. Piergiorgio?
>
> Karl
>
>
> On Wed, Apr 27, 2022 at 10:19 AM Julien Massiera <
> julien.massi...@francelabs.com> wrote:
>
>> Build is ok but I still have a problem with the cmisoutput test :
>>
>> run-IT-HSQLDB:
>> [junit] Testsuite:
>> org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT
>> [junit] Configuration file successfully read
>> [junit] [main] INFO org.eclipse.jetty.util.log - Logging initialized
>> @1550ms to org.eclipse.jetty.util.log.Slf4jLog
>> [junit] [main] INFO org.eclipse.jetty.server.Server -
>> jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git:
>> a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm
>> 11.0.15+10-Ubuntu-0ubuntu0.20.04.1
>> [junit] [main] INFO org.eclipse.jetty.server.session -
>> DefaultSessionIdManager workerName=node0
>> [junit] [main] INFO org.eclipse.jetty.server.session - No
>> SessionScavenger set, using defaults
>> [junit] [main] INFO org.eclipse.jetty.server.session - node0
>> Scavenging every 66ms
>> [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
>> Started o.e.j.w.WebAppContext@27a0a5a2{ManifoldCF Crawler
>> Interface,/mcf-crawler-ui,file:///tmp/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-11141316626452716674.dir/webapp/,AVAILABLE}{/home/francelabs/workspace/mcf-svn/dist/web/war/mcf-crawler-ui.war}
>> [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
>> Started o.e.j.w.WebAppContext@7692cd34{ManifoldCF Authorities API
>> Webapp,/mcf-authority-service,file:///tmp/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-17202752865589121447.dir/webapp/,AVAILABLE}{/home/francelabs/workspace/mcf-svn/dist/web/war/mcf-authority-service.war}
>> [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
>> Started o.e.j.w.WebAppContext@33aa93c{ManifoldCF General API
>> Webapp,/mcf-api-service,file:///tmp/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-7754762847799735649.dir/webapp/,AVAILABLE}{/home/francelabs/workspace/mcf-svn/dist/web/war/mcf-api-service.war}
>> [junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
>> Started ServerConnector@120f38e6{HTTP/1.1,[http/1.1]}{0.0.0.0:8346}
>> [junit] [main] INFO org.eclipse.jetty.server.Server - Started @2576ms
>> [junit] [main] INFO org.eclipse.jetty.server.Server -
>> jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git:
>> a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm
>> 11.0.15+10-Ubuntu-0ubuntu0.20.04.1
>> [junit] [main] WARN org.eclipse.jetty.webapp.WebInfConfiguration -
>> Web application not found
>> ../test-materials/chemistry-opencmis-server-inmemory-1.1.0-patched.war
>> [junit] [main] WARN org.eclipse.jetty.webapp.WebAppContext - Failed
>> startup of context o.e.j.w.WebAppContext@6c37bd27
>> {/chemistry-opencmis-server-inmemory,null,UNAVAILABLE}{../test-materials/chemistry-opencmis-server-inmemory-1.1.0-patched.war}
>> [junit] java.io.FileNotFoundException:
>> ../test-materials/chemistry-opencmis-server-inmemory-1.1.0-patched.war
>> [junit] at
>> org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:663)
>> [junit] at
>> org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:141)
>> [junit] at
>> org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:488)
>> [junit] at
>> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:523)
>> [junit] at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>> [junit] at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
>> [junit] at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> [junit] at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
>> [junit] at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>> [junit] at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
>> [junit] at org.eclipse.jetty.server.Server.start(Server.java:407)
>> [junit] at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
>> [junit] at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart
Re: [VOTE] Release Apache ManifoldCF 2.22, RC2
bapp,/mcf-authority-service,null,UNAVAILABLE}{/home/francelabs/workspace/mcf-svn/dist/web/war/mcf-authority-service.war}
> [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
> Stopped o.e.j.w.WebAppContext@27a0a5a2{ManifoldCF Crawler
> Interface,/mcf-crawler-ui,null,UNAVAILABLE}{/home/francelabs/workspace/mcf-svn/dist/web/war/mcf-crawler-ui.war}
> [junit] - ---
> [junit] Testcase:
> sanityCheck(org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT):
> Caused an ERROR
> [junit] Service Unavailable
> [junit]
> org.apache.chemistry.opencmis.commons.exceptions.CmisServiceUnavailableException:
> Service Unavailable
> [junit] at
> org.apache.chemistry.opencmis.client.bindings.spi.atompub.AbstractAtomPubService.convertStatusCode(AbstractAtomPubService.java:518)
> [junit] at
> org.apache.chemistry.opencmis.client.bindings.spi.atompub.AbstractAtomPubService.read(AbstractAtomPubService.java:701)
> [junit] at
> org.apache.chemistry.opencmis.client.bindings.spi.atompub.AbstractAtomPubService.getRepositoriesInternal(AbstractAtomPubService.java:873)
> [junit] at
> org.apache.chemistry.opencmis.client.bindings.spi.atompub.RepositoryServiceImpl.getRepositoryInfos(RepositoryServiceImpl.java:66)
> [junit] at
> org.apache.chemistry.opencmis.client.bindings.impl.RepositoryServiceImpl.getRepositoryInfos(RepositoryServiceImpl.java:92)
> [junit] at
> org.apache.chemistry.opencmis.client.runtime.SessionFactoryImpl.getRepositories(SessionFactoryImpl.java:120)
> [junit] at
> org.apache.chemistry.opencmis.client.runtime.SessionFactoryImpl.getRepositories(SessionFactoryImpl.java:107)
> [junit] at
> org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT.getCmisSourceClientSession(APISanityHSQLDBIT.java:83)
> [junit] at
> org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT.createTestArea(APISanityHSQLDBIT.java:143)
> [junit] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> [junit] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit] at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit]
> [junit]
> [junit] [Thread-186] INFO org.eclipse.jetty.server.AbstractConnector -
> Stopped ServerConnector@4d0b0fd4{HTTP/1.1,[http/1.1]}{0.0.0.0:9091}
> [junit] [Thread-186] INFO
> org.eclipse.jetty.server.handler.ContextHandler - Stopped
> o.e.j.w.WebAppContext@6c37bd27
> {/chemistry-opencmis-server-inmemory,null,UNAVAILABLE}{../test-materials/chemistry-opencmis-server-inmemory-1.1.0-patched.war}
> [junit] [Thread-186] INFO org.eclipse.jetty.server.AbstractConnector -
> Stopped ServerConnector@3adbe50f{HTTP/1.1,[http/1.1]}{0.0.0.0:9092}
> [junit] [Thread-186] INFO
> org.eclipse.jetty.server.handler.ContextHandler - Stopped
> o.e.j.w.WebAppContext@49aa766b
> {/chemistry-opencmis-server-inmemory,null,UNAVAILABLE}{../test-materials/chemistry-opencmis-server-inmemory-1.1.0-patched.war}
>
> BUILD FAILED
>
> -Message d'origine-
> De : Karl Wright
> Envoyé : mercredi 27 avril 2022 15:50
> À : dev
> Objet : [VOTE] Release Apache ManifoldCF 2.22, RC2
>
> Hi,
>
> Please vote on whether to release Apache ManifoldCF 2.22, RC2. The
> release artifact is at
> https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22 .
> There is also a release tag at
> https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC2 .
>
> This release contains a Tika update which required extensive dependency
> updates, which needs some degree of testing, as well as a new LDAP Mapper.
>
> RC1 also contains a CMIS fix from Piergiorgio, as well as an LDAP Mapper
> pom fix.
> RC2 has a fix for the MongoDB test download issue.
>
> Thanks in advance,
> Karl
>
>
>
[VOTE] Release Apache ManifoldCF 2.22, RC2
Hi, Please vote on whether to release Apache ManifoldCF 2.22, RC2. The release artifact is at https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22 . There is also a release tag at https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC2 . This release contains a Tika update which required extensive dependency updates, which needs some degree of testing, as well as a new LDAP Mapper. RC1 also contains a CMIS fix from Piergiorgio, as well as an LDAP Mapper pom fix. RC2 has a fix for the MongoDB test download issue. Thanks in advance, Karl
Re: svn commit: r1900317 - in /manifoldcf/trunk: ./ connectors/mongodb/ connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/
Pulled this up (adding CHANGES.txt) to release-2.22-branch.
Unless proven otherwise, I think the CXF issue is not an actual problem; it
sounds like human error (failure to do ant clean-core-deps). So I will
spin a new RC.
Karl
On Wed, Apr 27, 2022 at 8:46 AM wrote:
> Author: piergiorgio
> Date: Wed Apr 27 12:46:26 2022
> New Revision: 1900317
>
> URL: http://svn.apache.org/viewvc?rev=1900317=rev
> Log:
> fix for the build process of MongoDB Output Connector: updated
> dependencies and changed some deprecated methods (CONNECTORS-1708)
>
> Modified:
> manifoldcf/trunk/build.xml
> manifoldcf/trunk/connectors/mongodb/build.xml
>
> manifoldcf/trunk/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/APISanityHSQLDBIT.java
>
> manifoldcf/trunk/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/BaseITHSQLDB.java
> manifoldcf/trunk/connectors/mongodb/pom.xml
>
> Modified: manifoldcf/trunk/build.xml
> URL:
> http://svn.apache.org/viewvc/manifoldcf/trunk/build.xml?rev=1900317=1900316=1900317=diff
>
> ==
> --- manifoldcf/trunk/build.xml (original)
> +++ manifoldcf/trunk/build.xml Wed Apr 27 12:46:26 2022
> @@ -37,7 +37,7 @@
>
>
>
> -
> +
>
>
>
>
> Modified: manifoldcf/trunk/connectors/mongodb/build.xml
> URL:
> http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/mongodb/build.xml?rev=1900317=1900316=1900317=diff
>
> ==
> --- manifoldcf/trunk/connectors/mongodb/build.xml (original)
> +++ manifoldcf/trunk/connectors/mongodb/build.xml Wed Apr 27 12:46:26 2022
> @@ -60,23 +60,35 @@
>
>
>
> -
> +
> value="de.flapdoodle.embed.mongo"/>
> value="de/flapdoodle/embed"/>
> value="${embedded-mongodb.artifactId}-${embedded-mongodb.version.label}.jar"/>
>
> - value="2.0.5"/>
> + value="3.1.10"/>
> value="de.flapdoodle.embed.process"/>
> value="de/flapdoodle/embed"/>
> value="${embedded-mongodb.process.artifactId}-${embedded-mongodb.process.version.label}.jar"/>
>
> +value="1.0.3"/>
> + value="de.flapdoodle.embed.mongo.packageresolver"/>
> + value="de/flapdoodle/embed"/>
> + value="${embedded-mongodb.packageresolver.artifactId}-${embedded-mongodb.packageresolver.version.label}.jar"/>
> +
> +
> + value="de.flapdoodle.os"/>
> +
>
> + value="${embedded-mongodb.os.artifactId}-${embedded-mongodb.os.version.label}.jar"/>
> +
>
> property="tests-present"/>
> property="tests-present"/>
> +file="test-materials/${embedded-mongodb.packageresolver.jar}"
> property="tests-present"/>
> +property="tests-present"/>
>
>
>
> -
> +
>
>
>
> @@ -89,6 +101,9 @@
>https://oss.sonatype.org/content/repositories/public; />
> src="${sonatype-maven-repo-releases}/${embedded-mongodb-package}/${embedded-mongodb.artifactId}/${embedded-mongodb.version.label}/${embedded-mongodb.snapshot-jar}"
> dest="test-materials/${embedded-mongodb.snapshot-jar}"/>
> src="${sonatype-maven-repo-releases}/${embedded-mongodb.process-package}/${embedded-mongodb.process.artifactId}/${embedded-mongodb.process.version.label}/${embedded-mongodb.process.jar}"
> dest="test-materials/${embedded-mongodb.process.jar}"/>
> + src="${sonatype-maven-repo-releases}/${embedded-mongodb-packageresolver-package}/${embedded-mongodb.packageresolver.artifactId}/${embedded-mongodb.packageresolver.version.label}/${embedded-mongodb.packageresolver.jar}"
> dest="test-materials/${embedded-mongodb.packageresolver.jar}"/>
> + src="${sonatype-maven-repo-releases}/${embedded-mongodb-os-package}/${embedded-mongodb.os.artifactId}/${embedded-mongodb.os.version.label}/${embedded-mongodb.os.jar}"
> dest="test-materials/${embedded-mongodb.os.jar}"/>
> +
>
>
>
>
> Modified:
> manifoldcf/trunk/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/APISanityHSQLDBIT.java
> URL:
> http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/APISanityHSQLDBIT.java?rev=1900317=1900316=1900317=diff
>
> ==
> ---
> manifoldcf/trunk/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/APISanityHSQLDBIT.java
> (original)
> +++
> manifoldcf/trunk/connectors/mongodb/connector/src/test/java/org/apache/manifoldcf/agents/output/mongodboutput/tests/APISanityHSQLDBIT.java
> Wed Apr 27 12:46:26 2022
> @@ -305,12 +305,12 @@ public class APISanityHSQLDBIT extends B
> count =
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528703#comment-17528703
]
Karl Wright commented on CONNECTORS-1707:
-
Perhaps the problem seen here is simply due to the fact that the version of CXF
has been updated and the people trying this so far have not done
clean-core-deps? I can believe that.
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding saxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding jaxb
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws21
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend cxf
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'class.resource.loader.class' has been deprecated in favor of
> 'resource.loader.class.class'
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'resource.loader' has been deprecated in favor of 'resource.loaders'
> [java]
> [java] WSDLToJava Error:
> org.apache.cxf.bus.extension.ExtensionException: Could not load extension
> class org.apache.cxf.common.util.ASMHelperImpl.
> [java]
> [java] Java Result: 1classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528701#comment-17528701
]
Karl Wright commented on CONNECTORS-1707:
-
[~julienFL], you missed doing ant clean-core-deps then.
This is what you get when you do ant clean-core-deps , then ant make-core-deps:
{code}
C:\wip\mcf\trunk\lib>dir cxf*.jar
Volume in drive C is Windows
Volume Serial Number is F4D8-E4E0
Directory of C:\wip\mcf\trunk\lib
04/27/2022 04:34 AM 1,433,236 cxf-core-3.5.0.jar
04/27/2022 04:33 AM 182,637 cxf-rt-bindings-soap-3.5.0.jar
04/27/2022 04:33 AM38,461 cxf-rt-bindings-xml-3.5.0.jar
04/27/2022 04:33 AM 115,309 cxf-rt-databinding-jaxb-3.5.0.jar
04/27/2022 04:34 AM 704,040 cxf-rt-frontend-jaxrs-3.5.0.jar
04/27/2022 04:33 AM 371,928 cxf-rt-frontend-jaxws-3.5.0.jar
04/27/2022 04:33 AM 107,654 cxf-rt-frontend-simple-3.5.0.jar
04/27/2022 04:34 AM 190,074 cxf-rt-rs-client-3.5.0.jar
04/27/2022 04:34 AM41,604 cxf-rt-security-3.5.0.jar
04/27/2022 04:34 AM 392,007 cxf-rt-transports-http-3.5.0.jar
04/27/2022 04:33 AM62,930 cxf-rt-transports-http-hc-3.5.0.jar
04/27/2022 04:33 AM75,762 cxf-rt-ws-addr-3.5.0.jar
04/27/2022 04:33 AM 215,385 cxf-rt-ws-policy-3.5.0.jar
04/27/2022 04:33 AM 178,390 cxf-rt-wsdl-3.5.0.jar
04/27/2022 04:33 AM 149,625 cxf-tools-common-3.5.0.jar
04/27/2022 04:33 AM74,208 cxf-tools-validator-3.5.0.jar
04/27/2022 04:33 AM65,948 cxf-tools-wsdlto-core-3.5.0.jar
04/27/2022 04:33 AM45,178
cxf-tools-wsdlto-databinding-jaxb-3.5.0.jar
04/27/2022 04:33 AM 172,561 cxf-tools-wsdlto-frontend-jaxws-3.5.0.jar
19 File(s) 4,616,937 bytes
0 Dir(s) 111,555,883,008 bytes free
{code}
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databind
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528652#comment-17528652
]
Karl Wright commented on CONNECTORS-1707:
-
If you want to try experimenting with the ant method that invokes WSDL-to-Java,
here is the ant production you need to modify (from connector-build.xml):
{code}
http://localhost/${wsdlname}"/>
{code}
Note that it contains references needed for Java 11, as well as ALL the cxf
jars in the classpath.
The connector-build.xml jar lives in framework/buildfiles. It is copied to the
dist area when the framework part of MCF is built, before the connectors get
built, so be sure you modify the copy that is being used by the connector build.
As I said before, I cannot much help here since there is no failure for me.
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO org.apache.cx
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528648#comment-17528648
]
Karl Wright commented on CONNECTORS-1707:
-
The class in question is in:
Artifact cxf-core
Group org.apache.cxf
Version 3.5.2
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding saxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding jaxb
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws21
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend cxf
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'class.resource.loader.class' has been deprecated in favor of
> 'resource.loader.class.class'
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'resource.loader' has been deprecated in favor of 'resource.loaders'
> [java]
> [java] WSDLToJava Error:
> org.apache.cxf.bus.extension.ExtensionException: Could not load extension
> class org.apache.cxf.common.util.ASMHelperImpl.
> [java]
> [java] Java Result: 1classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
>
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528645#comment-17528645
]
Karl Wright commented on CONNECTORS-1707:
-
On a windows system running either Java 8 or Java 11, this does not occur.
It sounds like CXF does some kind of autolocate for this class, and it cannot
find it on a Mac. Not sure that we can fix this in ManifoldCF.
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding saxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding jaxb
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws21
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend cxf
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'class.resource.loader.class' has been deprecated in favor of
> 'resource.loader.class.class'
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'resource.loader' has been deprecated in favor of 'resource.loaders'
> [java]
> [java] WSDLToJava Error:
> org.apache.cxf.bus.extension.ExtensionException: Could not load extension
> class org.apache.cxf.common.util.ASMHelperImpl.
> [java]
> [java] Java Result: 1classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf
[CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC1
Withdrawn until there's a fix for MongoDB download issue. On Tue, Apr 26, 2022 at 6:39 AM Piergiorgio Lucidi wrote: > It seems to be a problem related to the de.flapdoodle library, probably > because the URL is hardcoded, I don't find a way to configure it. > We should consider updating that dependency and check if this problem will > be fixed. > > > Il giorno mar 26 apr 2022 alle ore 12:22 Karl Wright > ha scritto: > > > Sounds like a very interesting project! > > > > But we need to resolve the question of the MongoDB testing download > > failing. Should I cancel this RC given that? Is there any hope of a > fix? > > Downloading one for the wrong OS doesn't sound workable to me. > > > > Karl > > > > > > On Tue, Apr 26, 2022 at 5:19 AM Piergiorgio Lucidi < > piergior...@apache.org > > > > > wrote: > > > > > Hi folks, > > > > > > I'm working on a ManifoldCF SDK, a Maven project with Docker > integration > > > that we could also use for testing purposes. > > > Below you can find the current status of this development, totally > > > separated from the ManifoldCF source code because it is a totally > > > independent project. > > > In my mind I would like to have something like the following: > > > > > >1. Creating a new Docker image with OpenJDK 11, Ant and Maven > > installed > > >- DONE > > >2. Downloading ManifoldCF source code (MCF version is an argument) - > > > DONE > > >3. Running the Docker container for installing all the Maven > > >dependencies: ant make-core-deps make-deps build; mvn clean install > > >-DskipTests,skipITs - DONE > > >4. Storing the Maven Repo and the installation in two separated > Docker > > >Volumes - DONE > > >5. Running the Docker container for executing tests: ant test - DONE > > (I > > >have issues with Elasticsearch at the moment for this) > > >6. Copy the overall Maven Repo locally on the host in order to have > > all > > >the Maven dependencies available the MCF SDK classpath - DONE > > >7. Compiling a mcf-extensions-jar module in order to compile new > > >connectors or extensions that developers want to implement (TODO) > > >8. Overlaying the ManifoldCF Docker image with extensions compiled > > with > > >the SDK (TODO) > > >9. Running the final container with extensions (TODO) > > > > > > I'll update you on this, I'll add this status in a new issue related to > > the > > > Docker integration. > > > > > > Cheers, > > > PJ > > > > > > > > > Il giorno lun 25 apr 2022 alle ore 23:28 Karl Wright < > daddy...@gmail.com > > > > > > ha scritto: > > > > > > > Used the lib from the RC. > > > > > > > > Worked fine for me: > > > > > > > > BUILD SUCCESSFUL > > > > Total time: 2 minutes 53 seconds > > > > > > > > C:\wip\mcf\release-2.22-branch>echo %JAVA_HOME% > > > > c:\Program Files\Java\jdk-11.0.4 > > > > > > > > > > > > On Mon, Apr 25, 2022 at 5:19 PM Karl Wright > > wrote: > > > > > > > > > hmm, I'm having network problems today and that's blocking me. > This > > > > might > > > > > have to wait until tomorrow. > > > > > > > > > > Karl > > > > > > > > > > > > > > > On Mon, Apr 25, 2022 at 4:33 PM Karl Wright > > > wrote: > > > > > > > > > >> CXF is used to produce the missing java files from the checked-in > > > WSDLs. > > > > >> In order for the binary to have built it was possible to do this, > > but > > > > that > > > > >> was on JDK 8. Possibly JDK 11 has a missing CXF dependency; I'll > > have > > > > to > > > > >> look into that. > > > > >> > > > > >> Karl > > > > >> > > > > >> > > > > >> On Mon, Apr 25, 2022 at 4:04 PM Julien Massiera < > > > > >> julien.massi...@francelabs.com> wrote: > > > > >> > > > > >>> Hi Karl and Piergiorgio, > > > > >>> > > > > >>> On my side, I am unable to compile the csws connector with ant or > > > mave
Re: [VOTE] Release Apache ManifoldCF 2.22, RC1
Sounds like a very interesting project! But we need to resolve the question of the MongoDB testing download failing. Should I cancel this RC given that? Is there any hope of a fix? Downloading one for the wrong OS doesn't sound workable to me. Karl On Tue, Apr 26, 2022 at 5:19 AM Piergiorgio Lucidi wrote: > Hi folks, > > I'm working on a ManifoldCF SDK, a Maven project with Docker integration > that we could also use for testing purposes. > Below you can find the current status of this development, totally > separated from the ManifoldCF source code because it is a totally > independent project. > In my mind I would like to have something like the following: > >1. Creating a new Docker image with OpenJDK 11, Ant and Maven installed >- DONE >2. Downloading ManifoldCF source code (MCF version is an argument) - > DONE >3. Running the Docker container for installing all the Maven >dependencies: ant make-core-deps make-deps build; mvn clean install >-DskipTests,skipITs - DONE >4. Storing the Maven Repo and the installation in two separated Docker >Volumes - DONE >5. Running the Docker container for executing tests: ant test - DONE (I >have issues with Elasticsearch at the moment for this) >6. Copy the overall Maven Repo locally on the host in order to have all >the Maven dependencies available the MCF SDK classpath - DONE >7. Compiling a mcf-extensions-jar module in order to compile new >connectors or extensions that developers want to implement (TODO) >8. Overlaying the ManifoldCF Docker image with extensions compiled with >the SDK (TODO) >9. Running the final container with extensions (TODO) > > I'll update you on this, I'll add this status in a new issue related to the > Docker integration. > > Cheers, > PJ > > > Il giorno lun 25 apr 2022 alle ore 23:28 Karl Wright > ha scritto: > > > Used the lib from the RC. > > > > Worked fine for me: > > > > BUILD SUCCESSFUL > > Total time: 2 minutes 53 seconds > > > > C:\wip\mcf\release-2.22-branch>echo %JAVA_HOME% > > c:\Program Files\Java\jdk-11.0.4 > > > > > > On Mon, Apr 25, 2022 at 5:19 PM Karl Wright wrote: > > > > > hmm, I'm having network problems today and that's blocking me. This > > might > > > have to wait until tomorrow. > > > > > > Karl > > > > > > > > > On Mon, Apr 25, 2022 at 4:33 PM Karl Wright > wrote: > > > > > >> CXF is used to produce the missing java files from the checked-in > WSDLs. > > >> In order for the binary to have built it was possible to do this, but > > that > > >> was on JDK 8. Possibly JDK 11 has a missing CXF dependency; I'll have > > to > > >> look into that. > > >> > > >> Karl > > >> > > >> > > >> On Mon, Apr 25, 2022 at 4:04 PM Julien Massiera < > > >> julien.massi...@francelabs.com> wrote: > > >> > > >>> Hi Karl and Piergiorgio, > > >>> > > >>> On my side, I am unable to compile the csws connector with ant or > maven > > >>> : > > >>> > > >>> compile-connector: > > >>> [javac] Compiling 8 source files to > > >>> > > > /home/francelabs/workspace/mcf-svn/connectors/csws/build/connector/classes > > >>> [javac] warning: [options] bootstrap class path not set in > > >>> conjunction with -source 8 > > >>> [javac] > > >>> > > > /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:21: > > >>> error: package com.opentext.livelink.service.memberservice does not > > exist > > >>> [javac] import > > >>> com.opentext.livelink.service.memberservice.MemberRight; > > >>> [javac] ^ > > >>> [javac] > > >>> > > > /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:29: > > >>> error: package com.opentext.livelink.service.memberservice does not > > exist > > >>> [javac] import com.opentext.livelink.service.memberservice.User; > > >>> [javac] ^ > > >>> [javac] > > >>> > > > /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main
Re: [VOTE] Release Apache ManifoldCF 2.22, RC1
Used the lib from the RC. Worked fine for me: BUILD SUCCESSFUL Total time: 2 minutes 53 seconds C:\wip\mcf\release-2.22-branch>echo %JAVA_HOME% c:\Program Files\Java\jdk-11.0.4 On Mon, Apr 25, 2022 at 5:19 PM Karl Wright wrote: > hmm, I'm having network problems today and that's blocking me. This might > have to wait until tomorrow. > > Karl > > > On Mon, Apr 25, 2022 at 4:33 PM Karl Wright wrote: > >> CXF is used to produce the missing java files from the checked-in WSDLs. >> In order for the binary to have built it was possible to do this, but that >> was on JDK 8. Possibly JDK 11 has a missing CXF dependency; I'll have to >> look into that. >> >> Karl >> >> >> On Mon, Apr 25, 2022 at 4:04 PM Julien Massiera < >> julien.massi...@francelabs.com> wrote: >> >>> Hi Karl and Piergiorgio, >>> >>> On my side, I am unable to compile the csws connector with ant or maven >>> : >>> >>> compile-connector: >>> [javac] Compiling 8 source files to >>> /home/francelabs/workspace/mcf-svn/connectors/csws/build/connector/classes >>> [javac] warning: [options] bootstrap class path not set in >>> conjunction with -source 8 >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:21: >>> error: package com.opentext.livelink.service.memberservice does not exist >>> [javac] import >>> com.opentext.livelink.service.memberservice.MemberRight; >>> [javac] ^ >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:29: >>> error: package com.opentext.livelink.service.memberservice does not exist >>> [javac] import com.opentext.livelink.service.memberservice.User; >>> [javac] ^ >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:30: >>> error: package com.opentext.livelink.service.memberservice does not exist >>> [javac] import com.opentext.livelink.service.memberservice.Member; >>> [javac] ^ >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:31: >>> error: package com.opentext.livelink.service.memberservice does not exist >>> [javac] import >>> com.opentext.livelink.service.memberservice.MemberPrivileges; >>> [javac] ^ >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:31: >>> error: package com.opentext.livelink.service.memberservice does not exist >>> [javac] import >>> com.opentext.livelink.service.memberservice.MemberRight; >>> [javac] ^ >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:41: >>> error: package com.opentext.ecm.api does not exist >>> [javac] import com.opentext.ecm.api.OTAuthentication; >>> [javac]^ >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:42: >>> error: package com.opentext.livelink.service.core does not exist >>> [javac] import com.opentext.livelink.service.core.PageHandle; >>> [javac] ^ >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:43: >>> error: package com.opentext.livelink.service.core does not exist >>> [javac] import com.opentext.livelink.service.core.Authentication; >>> [javac] ^ >>> [javac] >>> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:44: >>> error: package com.opentext.livelink.service.core does not exist >>> [javac] import >>> com.opentext.livelink.service.core.Authentication_Ser
Re: [VOTE] Release Apache ManifoldCF 2.22, RC1
hmm, I'm having network problems today and that's blocking me. This might have to wait until tomorrow. Karl On Mon, Apr 25, 2022 at 4:33 PM Karl Wright wrote: > CXF is used to produce the missing java files from the checked-in WSDLs. > In order for the binary to have built it was possible to do this, but that > was on JDK 8. Possibly JDK 11 has a missing CXF dependency; I'll have to > look into that. > > Karl > > > On Mon, Apr 25, 2022 at 4:04 PM Julien Massiera < > julien.massi...@francelabs.com> wrote: > >> Hi Karl and Piergiorgio, >> >> On my side, I am unable to compile the csws connector with ant or maven : >> >> compile-connector: >> [javac] Compiling 8 source files to >> /home/francelabs/workspace/mcf-svn/connectors/csws/build/connector/classes >> [javac] warning: [options] bootstrap class path not set in >> conjunction with -source 8 >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:21: >> error: package com.opentext.livelink.service.memberservice does not exist >> [javac] import >> com.opentext.livelink.service.memberservice.MemberRight; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:29: >> error: package com.opentext.livelink.service.memberservice does not exist >> [javac] import com.opentext.livelink.service.memberservice.User; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:30: >> error: package com.opentext.livelink.service.memberservice does not exist >> [javac] import com.opentext.livelink.service.memberservice.Member; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/csws/CswsAuthority.java:31: >> error: package com.opentext.livelink.service.memberservice does not exist >> [javac] import >> com.opentext.livelink.service.memberservice.MemberPrivileges; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:31: >> error: package com.opentext.livelink.service.memberservice does not exist >> [javac] import >> com.opentext.livelink.service.memberservice.MemberRight; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:41: >> error: package com.opentext.ecm.api does not exist >> [javac] import com.opentext.ecm.api.OTAuthentication; >> [javac]^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:42: >> error: package com.opentext.livelink.service.core does not exist >> [javac] import com.opentext.livelink.service.core.PageHandle; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:43: >> error: package com.opentext.livelink.service.core does not exist >> [javac] import com.opentext.livelink.service.core.Authentication; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:44: >> error: package com.opentext.livelink.service.core does not exist >> [javac] import >> com.opentext.livelink.service.core.Authentication_Service; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/csws/CswsSession.java:45: >> error: package com.opentext.livelink.service.core does not exist >> [javac] import com.opentext.livelink.service.core.ContentService; >> [javac] ^ >> [javac] >> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/s
Re: [VOTE] Release Apache ManifoldCF 2.22, RC1
gt; [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:176:
> error: cannot find symbol
> [javac] private Map workspaceNodes = new HashMap<>();
> [javac] ^
> [javac] symbol: class Node
> [javac] location: class CswsConnector
> [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:1285:
> error: cannot find symbol
> [javac] private static long getID(final SGraph sg, final int
> nodeIndex) {
> [javac] ^
> [javac] symbol: class SGraph
> [javac] location: class CswsConnector
> [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:1290:
> error: cannot find symbol
> [javac] private static String getString(final SGraph sg, final int
> nodeIndex) {
> [javac] ^
> [javac] symbol: class SGraph
> [javac] location: class CswsConnector
> [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3380:
> error: cannot find symbol
> [javac] protected NodeRights getObjectRights(long objID)
> [javac] ^
> [javac] symbol: class NodeRights
> [javac] location: class CswsConnector
> [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:4402:
> error: cannot find symbol
> [javac] protected static String[] lookupTokens(final NodeRights
> rights, final ObjectInformation objInfo)
> [javac]^
> [javac] symbol: class NodeRights
> [javac] location: class CswsConnector
> [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:4432:
> error: cannot find symbol
> [javac] protected static boolean evaluateRight(final NodeRight nr) {
> [javac]^
> [javac] symbol: class NodeRight
> [javac] location: class CswsConnector
> [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3662:
> error: cannot find symbol
> [javac] protected Node objectValue = null;
> [javac] ^
> [javac] symbol: class Node
> [javac] location: class CswsConnector.ObjectInformation
> [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3855:
> error: cannot find symbol
> [javac] public List getAttributeGroups()
> [javac] ^
> [javac] symbol: class AttributeGroup
> [javac] location: class CswsConnector.ObjectInformation
> [javac]
> /home/francelabs/workspace/mcf-svn/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3966:
> error: cannot find symbol
> [javac] public NodePermissions getPermissions()
> [javac]^
> [javac] symbol: class NodePermissions
> [javac] location: class CswsConnector.ObjectInformation
> [javac] 100 errors
> [javac] 1 warning
>
> Julien
>
> -Message d'origine-
> De : Karl Wright
> Envoyé : dimanche 24 avril 2022 11:22
> À : dev@manifoldcf.apache.org
> Objet : Re: [VOTE] Release Apache ManifoldCF 2.22, RC1
>
> It looks like they've locked it down.
> We really don't have a choice other than to disable the test I fear. Have
> you been able to find an alternate URL?
>
> Karl
>
>
> On Fri, Apr 22, 2022 at 9:50 AM Piergiorgio Lucidi >
> wrote:
>
> > Hi Karl,
> >
> > I ran ant test command and now I have a problem with
> > MongoDBOutputConnector:
> >
> > [junit] java.io.IOException: Could not open inputStream for
> > https://fastdl.mongodb.org/osx/mongodb-osx-i386-2.6.8.tgz
> >
> > Maybe MongoDB has changed this URL?
> > I'm investigating...
> >
> > Below all the exception returned by Ant
> >
> > run-IT-HSQLDB:
> > [junit] Testsuite:
> > org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT
> > [junit] Configuration file successfully read
>
Re: [VOTE] Release Apache ManifoldCF 2.22, RC1
venging
> [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
> Stopped o.e.j.w.WebAppContext@34cf5a97{ManifoldCF General API
>
> Webapp,/mcf-api-service,null,UNAVAILABLE}{/Users/pjlucidi/Downloads/apache-manifoldcf-2.22/dist/web/war/mcf-api-service.war}
> [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
> Stopped o.e.j.w.WebAppContext@3703bf3c{ManifoldCF Authorities API
>
> Webapp,/mcf-authority-service,null,UNAVAILABLE}{/Users/pjlucidi/Downloads/apache-manifoldcf-2.22/dist/web/war/mcf-authority-service.war}
> [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
> Stopped o.e.j.w.WebAppContext@4482469c{ManifoldCF Crawler
>
> Interface,/mcf-crawler-ui,null,UNAVAILABLE}{/Users/pjlucidi/Downloads/apache-manifoldcf-2.22/dist/web/war/mcf-crawler-ui.war}
> [junit] - ---
> [junit] Testcase:
>
> sanityCheck(org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT):
> Caused an ERROR
> [junit] java.io.IOException: Could not open inputStream for
> https://fastdl.mongodb.org/osx/mongodb-osx-i386-2.6.8.tgz
> [junit] de.flapdoodle.embed.process.exceptions.DistributionException:
> java.io.IOException: Could not open inputStream for
> https://fastdl.mongodb.org/osx/mongodb-osx-i386-2.6.8.tgz
> [junit] at
> de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:69)
> [junit] at
> de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:49)
> [junit] at
>
> org.apache.manifoldcf.agents.output.mongodboutput.tests.BaseITHSQLDB.setUpMongoDB(BaseITHSQLDB.java:69)
> [junit] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> [junit] at
>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit] at
>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit] Caused by: java.io.IOException: Could not open inputStream for
> https://fastdl.mongodb.org/osx/mongodb-osx-i386-2.6.8.tgz
> [junit] at
>
> de.flapdoodle.embed.process.store.Downloader.downloadInputStream(Downloader.java:131)
> [junit] at
> de.flapdoodle.embed.process.store.Downloader.download(Downloader.java:69)
> [junit] at
>
> de.flapdoodle.embed.process.store.ArtifactStore.checkDistribution(ArtifactStore.java:66)
> [junit] at
>
> de.flapdoodle.embed.process.store.ExtractedArtifactStore.checkDistribution(ExtractedArtifactStore.java:60)
> [junit] at
> de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:56)
> [junit] Caused by: java.io.IOException: Server returned HTTP response
> code: 403 for URL:
> https://fastdl.mongodb.org/osx/mongodb-osx-i386-2.6.8.tgz
> [junit] at
>
> java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)
> [junit] at
>
> java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
> [junit] at
>
> java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
> [junit] at
>
> de.flapdoodle.embed.process.store.Downloader.downloadInputStream(Downloader.java:127)
> [junit]
> [junit]
> [junit] Testcase:
>
> sanityCheck(org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT):
> Caused an ERROR
> [junit] null
> [junit] java.lang.NullPointerException
> [junit] at
>
> org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT.removeTestArea(APISanityHSQLDBIT.java:109)
> [junit] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> [junit] at
>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit] at
>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit]
> [junit]
>
> BUILD FAILED
> /Users/pjlucidi/Downloads/apache-manifoldcf-2.22/build.xml:516: The
> following error occurred while executing this line:
> /Users/pjlucidi/Downloads/apache-manifoldcf-2.22/build.xml:470: The
> following error occurred while executing this line:
>
> /Users/pjlucidi/Downloads/apache-manifoldcf-2.22/dist/connector-build.xml:1102:
> Test
> org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT
> failed
>
>
> Il giorno ven 22 apr 2022 alle ore 08:48 Karl Wright
> ha scritto:
>
> > Hi,
> >
> > Please vote on whether to release Apache ManifoldCF 2.22, RC0. The
> release
> > artifact is at
> > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22
> .
> > There is also a release tag at
> > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC1 .
> >
> > This release contains a Tika update which required extensive dependency
> > updates, which needs some degree of testing, as well as a new LDAP
> Mapper.
> >
> > RC1 also contains a CMIS fix from Piergiorgio, as well as an LDAP Mapper
> > pom fix.
> >
> > Thanks in advance,
> > Karl
> >
>
>
> --
> Piergiorgio
>
[VOTE] Release Apache ManifoldCF 2.22, RC1
Hi, Please vote on whether to release Apache ManifoldCF 2.22, RC0. The release artifact is at https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22 . There is also a release tag at https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC1 . This release contains a Tika update which required extensive dependency updates, which needs some degree of testing, as well as a new LDAP Mapper. RC1 also contains a CMIS fix from Piergiorgio, as well as an LDAP Mapper pom fix. Thanks in advance, Karl
Re: [jira] [Resolved] (CONNECTORS-1705) CMIS Integration Test suite is broken
Thanks, Piergiorgio, I will pull this up into the release branch now. On Thu, Apr 21, 2022 at 10:40 AM Piergiorgio Lucidi (Jira) wrote: > > [ > https://issues.apache.org/jira/browse/CONNECTORS-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > ] > > Piergiorgio Lucidi resolved CONNECTORS-1705. > > Resolution: Fixed > > rr1900112. > > > CMIS Integration Test suite is broken > > - > > > > Key: CONNECTORS-1705 > > URL: > https://issues.apache.org/jira/browse/CONNECTORS-1705 > > Project: ManifoldCF > > Issue Type: Bug > > Components: CMIS connector > >Reporter: Piergiorgio Lucidi > >Assignee: Piergiorgio Lucidi > >Priority: Major > > Fix For: ManifoldCF next > > > > Original Estimate: 24h > > Remaining Estimate: 24h > > > > The current integration test suite of the CMIS Connector is broken due > to some classloading issue related to the OpenCMIS Server. > > So we have to patch the WAR file including the following changes. > > Here the list of dependencies that we have to remove from the OpenCMIS > Server WAR: > > * cxf-core-3.0.12.jar > > * cxf-rt-bindings-soap-3.0.12.jar > > * cxf-rt-bindings-xml-3.0.12.jar > > * cxf-rt-databinding-jaxb-3.0.12.jar > > * cxf-rt-frontend-jaxws-3.0.12.jar > > * cxf-rt-frontend-simple-3.0.12.jar > > * cxf-rt-transports-http-3.0.12.jar > > * cxf-rt-ws-addr-3.0.12.jar > > * cxf-rt-ws-policy-3.0.12.jar > > * cxf-rt-wsdl-3.0.12.jar > > * log4j-slf4j-impl-2.6.1.jar > > * log4j-core-2.6.1.jar > > * log4j-api-2.6.1.jar > > > > And below the list of JARs we need to add in the OpenCMIS Server WAR: > > * jax-ws-api-2.3.1.jar > > * log4j-slf4j-impl-2.17.2.jar > > * log4j-core-2.17.2.jar > > * log4j-api-2.17.2.jar > > * activation-1.1.1.jar > > * cxf-core-3.0.12.jar > > * cxf-rt-bindings-soap-3.5.2.jar > > * cxf-rt-bindings-xml-3.5.2.jar > > * cxf-rt-databinding-jaxb-3.5.2.jar > > * cxf-rt-frontend-jaxws-3.5.2.jar > > * cxf-rt-frontend-simple-3.5.2.jar > > * cxf-rt-transports-http-3.5.2.jar > > * cxf-rt-ws-addr-3.5.2.jar > > * cxf-rt-ws-policy-3.5.2.jar > > * cxf-rt-wsdl-3.5.2.jar > > > > > > > > -- > This message was sent by Atlassian Jira > (v8.20.7#820007) >
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC0
Sounds like a good idea. One suggestion: I would not structure the ant logic to overwrite the war. I would download one copy and then transfer/create a different copy. Ant works best that way. So no deletion of the original download. On Thu, Apr 21, 2022 at 6:59 AM Piergiorgio Lucidi wrote: > I'm going to patch the the WAR file in the download-dependencies task in > Ant :-P > I'll use the download-via-maven for downloading all the libraries to > update, then unzip, delete and the jar task, so it should be enough, what > do you think? > PJ > > Il giorno gio 21 apr 2022 alle ore 12:49 Karl Wright > ha scritto: > > > Right now the war is downloaded via ant, I think. We should "patch" it > > when we download it, by unpacking, updating, and repacking. We leave it > in > > a specific place and I don't believe the pom.xml refers to it directly. > > > > If you can confirm that picture, I can suggest ant commands to do the > > unpack / copy / repack. Or I can look at this much later today myself. > > > > Karl > > > > > > On Thu, Apr 21, 2022 at 6:46 AM Piergiorgio Lucidi < > piergior...@apache.org > > > > > wrote: > > > > > Ok I solved the classloading issue :) > > > > > > I found all the dependencies that we have to update in the OpenCMIS > > Server > > > WAR, practically we have to: > > > 1. Update log4j from 2.6.1 to 2.17.2 > > > 2. Update CXF from 3.0.12 to 3.5.2 > > > 3. Add activation-1.1.1.jar > > > 4. Add jaxws-api-2.3.1.jar > > > > > > Here the list of dependencies that we have to remove from the WAR: > > > > > >- cxf-core-3.0.12.jar > > >- cxf-rt-bindings-soap-3.0.12.jar > > >- cxf-rt-bindings-xml-3.0.12.jar > > >- cxf-rt-databinding-jaxb-3.0.12.jar > > >- cxf-rt-frontend-jaxws-3.0.12.jar > > >- cxf-rt-frontend-simple-3.0.12.jar > > >- cxf-rt-transports-http-3.0.12.jar > > >- cxf-rt-ws-addr-3.0.12.jar > > >- cxf-rt-ws-policy-3.0.12.jar > > >- cxf-rt-wsdl-3.0.12.jar > > >- log4j-slf4j-impl-2.6.1.jar > > >- log4j-core-2.6.1.jar > > >- log4j-api-2.6.1.jar > > > > > > And below the list of JARs to add in the WAR: > > > > > >- jax-ws-api-2.3.1.jar > > >- log4j-slf4j-impl-2.17.2.jar > > >- log4j-core-2.17.2.jar > > >- log4j-api-2.17.2.jar > > >- activation-1.1.1.jar > > >- cxf-core-3.0.12.jar > > >- cxf-rt-bindings-soap-3.5.2.jar > > >- cxf-rt-bindings-xml-3.5.2.jar > > >- cxf-rt-databinding-jaxb-3.5.2.jar > > >- cxf-rt-frontend-jaxws-3.5.2.jar > > >- cxf-rt-frontend-simple-3.5.2.jar > > >- cxf-rt-transports-http-3.5.2.jar > > >- cxf-rt-ws-addr-3.5.2.jar > > >- cxf-rt-ws-policy-3.5.2.jar > > >- cxf-rt-wsdl-3.5.2.jar > > > > > > We only need to overlay the > chemistry-opencmis-server-inmemory-1.1.0.war > > > updating log4j and cxf dependencies in order to solve the deployment > > issue. > > > I ran tests correctly using the new WAR file. > > > > > > What could be the best way to overlay the WAR for making it available > for > > > both Ant and Maven? > > > Maybe using an Ant script and then reuse the same in Maven? > > > > > > Cheers, > > > PJ > > > > > > Il giorno gio 21 apr 2022 alle ore 12:27 Karl Wright < > daddy...@gmail.com > > > > > > ha scritto: > > > > > > > Ok, maybe there's a conflict then. We cannot back out versions of > CXF > > > > because we use that for the Livelink connector now. > > > > > > > > > > > > On Thu, Apr 21, 2022 at 6:20 AM Piergiorgio Lucidi < > > > piergior...@apache.org > > > > > > > > > wrote: > > > > > > > > > Sure, I'm investigating the OpenCMIS Server issue, I resolved the > > Log4j > > > > > issue updating the jar in the WAR but now I see a problem with CXF > > old > > > > > dependencies. > > > > > The strange thing is that using the same patched WAR in a standard > > > Docker > > > > > container based on OpenJDK 11 it works perfectly but trying to use > it > > > > using > > > > > Ant or Maven return issues. > > > > > > > > > > We could disable integration tests for CMIS connectors eventually > for > > > &g
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC0
Right now the war is downloaded via ant, I think. We should "patch" it when we download it, by unpacking, updating, and repacking. We leave it in a specific place and I don't believe the pom.xml refers to it directly. If you can confirm that picture, I can suggest ant commands to do the unpack / copy / repack. Or I can look at this much later today myself. Karl On Thu, Apr 21, 2022 at 6:46 AM Piergiorgio Lucidi wrote: > Ok I solved the classloading issue :) > > I found all the dependencies that we have to update in the OpenCMIS Server > WAR, practically we have to: > 1. Update log4j from 2.6.1 to 2.17.2 > 2. Update CXF from 3.0.12 to 3.5.2 > 3. Add activation-1.1.1.jar > 4. Add jaxws-api-2.3.1.jar > > Here the list of dependencies that we have to remove from the WAR: > >- cxf-core-3.0.12.jar >- cxf-rt-bindings-soap-3.0.12.jar >- cxf-rt-bindings-xml-3.0.12.jar >- cxf-rt-databinding-jaxb-3.0.12.jar >- cxf-rt-frontend-jaxws-3.0.12.jar >- cxf-rt-frontend-simple-3.0.12.jar >- cxf-rt-transports-http-3.0.12.jar >- cxf-rt-ws-addr-3.0.12.jar >- cxf-rt-ws-policy-3.0.12.jar >- cxf-rt-wsdl-3.0.12.jar >- log4j-slf4j-impl-2.6.1.jar >- log4j-core-2.6.1.jar >- log4j-api-2.6.1.jar > > And below the list of JARs to add in the WAR: > >- jax-ws-api-2.3.1.jar >- log4j-slf4j-impl-2.17.2.jar >- log4j-core-2.17.2.jar >- log4j-api-2.17.2.jar >- activation-1.1.1.jar >- cxf-core-3.0.12.jar >- cxf-rt-bindings-soap-3.5.2.jar >- cxf-rt-bindings-xml-3.5.2.jar >- cxf-rt-databinding-jaxb-3.5.2.jar >- cxf-rt-frontend-jaxws-3.5.2.jar >- cxf-rt-frontend-simple-3.5.2.jar >- cxf-rt-transports-http-3.5.2.jar >- cxf-rt-ws-addr-3.5.2.jar >- cxf-rt-ws-policy-3.5.2.jar >- cxf-rt-wsdl-3.5.2.jar > > We only need to overlay the chemistry-opencmis-server-inmemory-1.1.0.war > updating log4j and cxf dependencies in order to solve the deployment issue. > I ran tests correctly using the new WAR file. > > What could be the best way to overlay the WAR for making it available for > both Ant and Maven? > Maybe using an Ant script and then reuse the same in Maven? > > Cheers, > PJ > > Il giorno gio 21 apr 2022 alle ore 12:27 Karl Wright > ha scritto: > > > Ok, maybe there's a conflict then. We cannot back out versions of CXF > > because we use that for the Livelink connector now. > > > > > > On Thu, Apr 21, 2022 at 6:20 AM Piergiorgio Lucidi < > piergior...@apache.org > > > > > wrote: > > > > > Sure, I'm investigating the OpenCMIS Server issue, I resolved the Log4j > > > issue updating the jar in the WAR but now I see a problem with CXF old > > > dependencies. > > > The strange thing is that using the same patched WAR in a standard > Docker > > > container based on OpenJDK 11 it works perfectly but trying to use it > > using > > > Ant or Maven return issues. > > > > > > We could disable integration tests for CMIS connectors eventually for > > this > > > release. > > > > > > Probably it will take some time to understand all the dependencies to > > > update in the OpenCMIS Server, anyway when I have a working WAR file > then > > > we should create an ant script to update everything starting from the > > > original distribution WAR. > > > > > > I'll keep you updated. > > > > > > Cheers, > > > PJ > > > > > > Il giorno gio 21 apr 2022 alle ore 12:11 Karl Wright < > daddy...@gmail.com > > > > > > ha scritto: > > > > > > > I can fix the bad pom but not the CMIS issue, which will require more > > > > research. > > > > Karl > > > > > > > > On Thu, Apr 21, 2022 at 5:54 AM Piergiorgio Lucidi < > > > piergior...@apache.org > > > > > > > > > wrote: > > > > > > > > > Hi Karl, > > > > > > > > > > -1 from me because I found these problems: > > > > > > > > > > 1. Trying to build with Maven I see the following error: > > > > > > > > > > The build could not read 1 project -> [Help 1] > > > > > [ERROR] > > > > > [ERROR] The project > > > > > org.apache.manifoldcf:mcf-ldapmapper-connector:2.22-SNAPSHOT > > > > > > > > > > > > > > > > > > > > (/Users/pjlucidi/Downloads/apache-manifoldcf-2.22/connectors/ldapmapper/pom.xml) > > > > > has 1 error > >
Re: [CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC0
Ok, maybe there's a conflict then. We cannot back out versions of CXF because we use that for the Livelink connector now. On Thu, Apr 21, 2022 at 6:20 AM Piergiorgio Lucidi wrote: > Sure, I'm investigating the OpenCMIS Server issue, I resolved the Log4j > issue updating the jar in the WAR but now I see a problem with CXF old > dependencies. > The strange thing is that using the same patched WAR in a standard Docker > container based on OpenJDK 11 it works perfectly but trying to use it using > Ant or Maven return issues. > > We could disable integration tests for CMIS connectors eventually for this > release. > > Probably it will take some time to understand all the dependencies to > update in the OpenCMIS Server, anyway when I have a working WAR file then > we should create an ant script to update everything starting from the > original distribution WAR. > > I'll keep you updated. > > Cheers, > PJ > > Il giorno gio 21 apr 2022 alle ore 12:11 Karl Wright > ha scritto: > > > I can fix the bad pom but not the CMIS issue, which will require more > > research. > > Karl > > > > On Thu, Apr 21, 2022 at 5:54 AM Piergiorgio Lucidi < > piergior...@apache.org > > > > > wrote: > > > > > Hi Karl, > > > > > > -1 from me because I found these problems: > > > > > > 1. Trying to build with Maven I see the following error: > > > > > > The build could not read 1 project -> [Help 1] > > > [ERROR] > > > [ERROR] The project > > > org.apache.manifoldcf:mcf-ldapmapper-connector:2.22-SNAPSHOT > > > > > > > > > (/Users/pjlucidi/Downloads/apache-manifoldcf-2.22/connectors/ldapmapper/pom.xml) > > > has 1 error > > > [ERROR] Non-resolvable parent POM for > > > org.apache.manifoldcf:mcf-ldapmapper-connector:2.22-SNAPSHOT: Could not > > > find artifact org.apache.manifoldcf:mcf-connectors:pom:2.22-SNAPSHOT > and > > > 'parent.relativePath' points at wrong local POM @ line 2, column 11 -> > > > [Help 2] > > > > > > So it seems that the version is wrong in the new LDAP Mapper Connector, > > so > > > updating the version element from 2.22-SNAPSHOT to 2.22 in that pom > then > > > the build starts correctly. > > > > > > 2. Trying to execute tests with Ant or Maven, I see a problem related > to > > > the OpenCMIS server during the execution of integration tests related > to > > > the CMIS Connectors. It's a classCastException related to Log4j, below > > the > > > exception that I see: > > > > > > [main] INFO org.eclipse.jetty.server.Server - Started @1830ms > > > [main] INFO org.eclipse.jetty.server.Server - jetty-9.4.25.v20191220; > > > built: 2019-12-20T17:00:00.294Z; git: > > > a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm 11.0.13+8-LTS > > > [main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor - NO > JSP > > > Support for /chemistry-opencmis-server-inmemory, did not find > > > org.eclipse.jetty.jsp.JettyJspServlet > > > SLF4J: Class path contains multiple SLF4J bindings. > > > SLF4J: Found binding in > > > > > > > > > [jar:file:/private/var/folders/g1/vlf2szxj7tbbgm92n39l8kj8gn/T/jetty-0_0_0_0-9090-chemistry-opencmis-server-inmemory_war-_chemistry-opencmis-server-inmemory-any-6022774582585212771.dir/webapp/WEB-INF/lib/log4j-slf4j-impl-2.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > > SLF4J: Found binding in > > > > > > > > > [jar:file:/Users/pjlucidi/.m2/repository/org/slf4j/slf4j-simple/1.7.7/slf4j-simple-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > > explanation. > > > SLF4J: Actual binding is of type > > > [org.apache.logging.slf4j.Log4jLoggerFactory] > > > 2022-04-21 11:49:07,026 main ERROR Unable to create Lookup for event > > > java.lang.ClassCastException: class > > > org.apache.logging.log4j.core.lookup.EventLookup > > > at java.base/java.lang.Class.asSubclass(Class.java:3640) > > > at > > > > > > > > > org.apache.logging.log4j.core.lookup.Interpolator.(Interpolator.java:65) > > > at > > > > > > > > > org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:467) > > > at > > > > > > > > > org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:219) > > > at > > > > &g
[CANCEL] [VOTE] Release Apache ManifoldCF 2.22, RC0
.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
>
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:350)
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
> at
>
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822)
> at
>
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
> at
>
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
>
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
>
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
>
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
> at
>
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
>
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.server.Server.start(Server.java:407)
> at
>
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
>
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
> at org.eclipse.jetty.server.Server.doStart(Server.java:371)
> at
>
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
>
> org.apache.manifoldcf.crawler.connectors.cmis.tests.CMISServer.start(CMISServer.java:49)
> at
>
> org.apache.manifoldcf.crawler.connectors.cmis.tests.BaseITHSQLDB.setUpCMIS(BaseITHSQLDB.java:66)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
>
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
>
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
>
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
>
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
>
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
>
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
>
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
>
> [main] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager
> workerName=node0
> [main] INFO org.eclipse.jetty.server.session - No SessionScavenger set,
> using defaults
> [main] INFO org.eclipse.jetty.server.session - node0 Scavenging every
> 60ms
> [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started
> o.e.j.w.WebAppContext@7318daf8{OpenCMIS
>
> Server,/chemistry-opencmis-server-inmemory,file:///private/var/folders/g1/vlf2szxj7tbbgm92n39l8kj8gn/T/jetty-0_0_0_0-9090-chemistry-opencmis-server-inmemory_war-_chemistry-opencmis-server-inmemory-any-6022774582585212771.dir/webapp/,AVAILABLE}{../dependency/chemistry-opencmis-server-inmemory.war}
> [main] INFO org.eclipse.jetty.server.AbstractConnector - Started
> ServerConnector@66153688{HTTP/1.1,[http/1.1]}{0.0.0.0:9090}
> [main] INFO org.eclipse.jetty.server.Server - Started @4208ms
>
>
>
> Il giorno gio 21 apr 2022 alle ore 09:23 Karl Wright
> ha scritto:
>
> > Hi,
> >
> > Please vote on whether to release Apache ManifoldCF 2.22, RC0. The
> release
> > artifact is at
> > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22
> .
> > There is also a release tag at
> > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC0 .
> >
> > This release contains a Tika update which required extensive dependency
> > updates, which needs some degree of testing, as well as a new LDAP
> Mapper.
> >
> > Thanks in advance,
> > Karl
> >
>
>
> --
> Piergiorgio
>
[VOTE] Release Apache ManifoldCF 2.22, RC0
Hi, Please vote on whether to release Apache ManifoldCF 2.22, RC0. The release artifact is at https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.22 . There is also a release tag at https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.22-RC0 . This release contains a Tika update which required extensive dependency updates, which needs some degree of testing, as well as a new LDAP Mapper. Thanks in advance, Karl
Re: April release
1.28.1 Ah, you mean from build.xml. Got it. Karl On Wed, Apr 20, 2022 at 10:27 AM Karl Wright wrote: > Hmm. From root pom.xml, tika.version is set as follows: > > > On Wed, Apr 20, 2022 at 10:25 AM Karl Wright wrote: > >> Hi, I take it you mean the pom? That would be a trivial fix. >> Karl >> >> >> On Wed, Apr 20, 2022 at 10:00 AM Julien Massiera < >> julien.massi...@francelabs.com> wrote: >> >>> Hi Karl, >>> >>> First of all we wish you a quick recovery ! Concerning the release, we >>> ran our tests on the 2.22 branch today, using the Web, Winshare, >>> SharePoint repository connectors and the transformation connectors Tika >>> and TikaServiceRmeta. >>> >>> There was only one problem with the Tika transformation connector, you >>> updated the dependencies for the 1.28.1 version but you forgot to update >>> tika core and tika parsers to this version, they are still in 1.21. Once >>> those libs are up to date everything works fine. >>> >>> How do we proceed for that issue ? >>> >>> Regards, >>> Julien >>> >>> On 14/04/2022 23:49, Karl Wright wrote: >>> > Hi all, >>> > >>> > April is now coming to an end and we are scheduled to put out a >>> ManifoldCF >>> > release. Quite a lot has been done for this release, mainly in >>> updating >>> > Tika and all of its myriad dependencies, so when the release is >>> available I >>> > think we need to run our tests on as many platforms as possible, and if >>> > possible do a hand smoke test as well (build connections and a job and >>> run >>> > it, for as many repositories as we have access to). >>> > >>> > This happens at a bad time for me; I am dealing with personal medical >>> > issues for which there is no easy solution, so my free computer time is >>> > almost nothing right now. If the community wants to do this release, I >>> > have enough time to press all the buttons but I won't have time for >>> > in-depth testing I fear. So I am looking for a show of hands as to who >>> > wants to do the due diligence on the release so we can be assured it's >>> not >>> > dead on arrival. Thanks in advance, >>> > >>> > Karl >>> > >>> -- >>> Julien MASSIERA >>> Directeur développement produit >>> France Labs – Les experts du Search >>> Datafari – Vainqueur du trophée Big Data 2018 au Digital Innovation >>> Makers Summit >>> www.francelabs.com >>> >>>
Re: April release
Hmm. From root pom.xml, tika.version is set as follows: On Wed, Apr 20, 2022 at 10:25 AM Karl Wright wrote: > Hi, I take it you mean the pom? That would be a trivial fix. > Karl > > > On Wed, Apr 20, 2022 at 10:00 AM Julien Massiera < > julien.massi...@francelabs.com> wrote: > >> Hi Karl, >> >> First of all we wish you a quick recovery ! Concerning the release, we >> ran our tests on the 2.22 branch today, using the Web, Winshare, >> SharePoint repository connectors and the transformation connectors Tika >> and TikaServiceRmeta. >> >> There was only one problem with the Tika transformation connector, you >> updated the dependencies for the 1.28.1 version but you forgot to update >> tika core and tika parsers to this version, they are still in 1.21. Once >> those libs are up to date everything works fine. >> >> How do we proceed for that issue ? >> >> Regards, >> Julien >> >> On 14/04/2022 23:49, Karl Wright wrote: >> > Hi all, >> > >> > April is now coming to an end and we are scheduled to put out a >> ManifoldCF >> > release. Quite a lot has been done for this release, mainly in updating >> > Tika and all of its myriad dependencies, so when the release is >> available I >> > think we need to run our tests on as many platforms as possible, and if >> > possible do a hand smoke test as well (build connections and a job and >> run >> > it, for as many repositories as we have access to). >> > >> > This happens at a bad time for me; I am dealing with personal medical >> > issues for which there is no easy solution, so my free computer time is >> > almost nothing right now. If the community wants to do this release, I >> > have enough time to press all the buttons but I won't have time for >> > in-depth testing I fear. So I am looking for a show of hands as to who >> > wants to do the due diligence on the release so we can be assured it's >> not >> > dead on arrival. Thanks in advance, >> > >> > Karl >> > >> -- >> Julien MASSIERA >> Directeur développement produit >> France Labs – Les experts du Search >> Datafari – Vainqueur du trophée Big Data 2018 au Digital Innovation >> Makers Summit >> www.francelabs.com >> >>
Re: April release
Hi, I take it you mean the pom? That would be a trivial fix. Karl On Wed, Apr 20, 2022 at 10:00 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi Karl, > > First of all we wish you a quick recovery ! Concerning the release, we > ran our tests on the 2.22 branch today, using the Web, Winshare, > SharePoint repository connectors and the transformation connectors Tika > and TikaServiceRmeta. > > There was only one problem with the Tika transformation connector, you > updated the dependencies for the 1.28.1 version but you forgot to update > tika core and tika parsers to this version, they are still in 1.21. Once > those libs are up to date everything works fine. > > How do we proceed for that issue ? > > Regards, > Julien > > On 14/04/2022 23:49, Karl Wright wrote: > > Hi all, > > > > April is now coming to an end and we are scheduled to put out a > ManifoldCF > > release. Quite a lot has been done for this release, mainly in updating > > Tika and all of its myriad dependencies, so when the release is > available I > > think we need to run our tests on as many platforms as possible, and if > > possible do a hand smoke test as well (build connections and a job and > run > > it, for as many repositories as we have access to). > > > > This happens at a bad time for me; I am dealing with personal medical > > issues for which there is no easy solution, so my free computer time is > > almost nothing right now. If the community wants to do this release, I > > have enough time to press all the buttons but I won't have time for > > in-depth testing I fear. So I am looking for a show of hands as to who > > wants to do the due diligence on the release so we can be assured it's > not > > dead on arrival. Thanks in advance, > > > > Karl > > > -- > Julien MASSIERA > Directeur développement produit > France Labs – Les experts du Search > Datafari – Vainqueur du trophée Big Data 2018 au Digital Innovation Makers > Summit > www.francelabs.com > >
Re: New LDAP Mapper Connector
I haven't yet put up a vote for the release candidate. I just merged this feature branch and pulled it into the release-2.22-branch branch. I will have to respin an RC later today. Karl On Tue, Apr 19, 2022 at 9:24 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi, > > I really would like this mapper connector to be included in the next > release of ManifoldCF. It does not conflict with another connector so we > only need to check that the build passes. > > In the mean time, we will run our tests on the new MCF release today or > tomorrow. > > Regards, > Julien > > -Message d'origine- > De : Karl Wright > Envoyé : lundi 14 février 2022 17:09 > À : dev > Objet : Re: New LDAP Mapper Connector > > We have received zero comment on it. So I think we'll just have to land > it and await comment. > Karl > > On Mon, Feb 14, 2022 at 5:42 AM Julien Massiera < > julien.massi...@francelabs.com> wrote: > > > Hi, > > > > Any news about this new connector integration ? > > > > Regards, > > Julien > > > > -Message d'origine- > > De : Karl Wright > > Envoyé : mardi 11 janvier 2022 18:51 > > À : dev > > Objet : Re: New LDAP Mapper Connector > > > > Thank you for doing this Julien. > > > > Let's see what others say and if we hear nothing in a few days, onto > > trunk it goes. > > > > Karl > > > > > > On Tue, Jan 11, 2022 at 12:37 PM Julien Massiera < > > julien.massi...@francelabs.com> wrote: > > > > > Hi everyone, > > > > > > > > > > > > I have created the following issue > > > https://issues.apache.org/jira/browse/CONNECTORS-1692 to develop a > > > new LDAP Mapper connector. I have created the corresponding branch > > > and commited the connector to it. I did an ant and a maven build to > > > make sure everything is fine. If anyone could check if it is also > > > fine on his side, it would be greatly appreciated, and we could then > > > merge it to the trunk ! > > > > > > > > > > > > Regards, > > > > > > Julien > > > > > > > > > > > > > > > > > >
April release
Hi all, April is now coming to an end and we are scheduled to put out a ManifoldCF release. Quite a lot has been done for this release, mainly in updating Tika and all of its myriad dependencies, so when the release is available I think we need to run our tests on as many platforms as possible, and if possible do a hand smoke test as well (build connections and a job and run it, for as many repositories as we have access to). This happens at a bad time for me; I am dealing with personal medical issues for which there is no easy solution, so my free computer time is almost nothing right now. If the community wants to do this release, I have enough time to press all the buttons but I won't have time for in-depth testing I fear. So I am looking for a show of hands as to who wants to do the due diligence on the release so we can be assured it's not dead on arrival. Thanks in advance, Karl
Re: WorkerThread runtime exceptions
cifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:559) ~[?:?] > at jcifs.smb.SmbFile.exists(SmbFile.java:859) ~[?:?] > at > com.francelabs.datafari.connectors.share.SharedDriveConnector.fileExists(SharedDriveConnector.java:2129) > ~[?:?] > at > com.francelabs.datafari.connectors.share.SharedDriveConnector.processDocuments(SharedDriveConnector.java:677) > ~[?:?] > at > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) > [mcf-pull-agent.jar:?] > > > So after that FATAL exception, we end up in the 'wait' state of the > documentQueue > > Julien > > -Message d'origine- > De : Karl Wright > Envoyé : mercredi 23 mars 2022 16:32 > À : dev > Objet : Re: WorkerThread runtime exceptions > > Specifically, the stuffer thread is responsible for finding documents to > process and getting them to the worker threads via the internal queue that > the worker threads wait on. The stuffer thread uses a query to do this. > Either the query is not finding any documents, or the stuffer thread is > down. Probably it is the former, and the reason it is not finding any > documents is because the job is in the wrong state due to that runtime > exception. > > Can you describe what code is throwing that runtime exception? It would > be very helpful if you could provide a stack trace for it from the log. > > Karl > > > On Wed, Mar 23, 2022 at 11:27 AM Karl Wright wrote: > > > ' 1/ On the first and only one document of the seeding phase > > encountered, a runtime exception is triggered' > > > > The worker threads do not handle seeding. If a runtime exception > > takes place during seeding, no documents will be queued, and that is > > the problem. The state of the job must be incorrectly updated even > > though the seeding failed. OR the job's state is properly updated but > > the corresponding thread that is supposed to know when the job is > > completed (by looking at the job queue) doesn't properly trigger. > > > > The architecture of ManifoldCF has many threads that are individually > > responsible for transitioning the job state based on the jobqueue. If > > somehow the jobstate winds up not in the right state then those > > threads will not do the right thing. > > > > Karl > > > > > > On Wed, Mar 23, 2022 at 11:08 AM Julien Massiera < > > julien.massi...@francelabs.com> wrote: > > > >> Hi Karl, > >> > >> I had some time to investigate the problem I exposed in my first > >> mail, and here is the behavior I observed: > >> > >> 1/ On the first and only one document of the seeding phase > >> encountered, a runtime exception is triggered 2/ The runtime > >> exception is catched by the WorkerThread, logged, and the > >> WorkerThread stays alive (line 856 of the WorkerThread class) 3/ The > >> WorkerThread calls the getDocument method of its documentQueue (line > >> 121 of the WorkerThread class) 4/ The documentQueue ends in an > >> infinite 'wait' state because the queue size is 0 and the resetFlag > >> is false (lines 109 and 110 of the DocumentQueue class) 5/ Because of > >> the infinite 'wait' state of the documentQueue, the job stays freezed > >> on the 'running' state and it is impossible to stop it until the > >> Agent is restarted > >> > >> I don't know much about the WorkerThread and the DocumentQueue logic, > >> so from there, I really need your help to understand this behavior > >> and to figure out what can be done to prevent the job from hanging in > >> that case, which, I assume, can happen in other circumstances with > >> other repository connectors > >> > >> Regards, > >> Julien > >> > >> -Message d'origine- > >> De : Julien Massiera Envoyé : jeudi > >> 24 février 2022 15:08 À : dev@manifoldcf.apache.org Objet : RE: > >> WorkerThread runtime exceptions > >> > >> Yes I understand > >> > >> -Message d'origine- > >> De : Karl Wright Envoyé : jeudi 24 février 2022 > >> 14:59 À : dev Objet : Re: WorkerThread > >> runtime exceptions > >> > >> I'm currently completely consumed with upgrading dependencies for > >> Tika and CXF. This is a massive job and won't be done for probably > >> another week or two. Once that is done I can try to look into your > concern. > >> > >> Karl > >> > >> > >> On Thu, Feb 24, 2022 at 8:47 AM Julien Massiera < > >> julien.massi...@francelabs.com> wrote: > >> >
Re: WorkerThread runtime exceptions
Specifically, the stuffer thread is responsible for finding documents to process and getting them to the worker threads via the internal queue that the worker threads wait on. The stuffer thread uses a query to do this. Either the query is not finding any documents, or the stuffer thread is down. Probably it is the former, and the reason it is not finding any documents is because the job is in the wrong state due to that runtime exception. Can you describe what code is throwing that runtime exception? It would be very helpful if you could provide a stack trace for it from the log. Karl On Wed, Mar 23, 2022 at 11:27 AM Karl Wright wrote: > ' 1/ On the first and only one document of the seeding phase encountered, > a runtime exception is triggered' > > The worker threads do not handle seeding. If a runtime exception takes > place during seeding, no documents will be queued, and that is the > problem. The state of the job must be incorrectly updated even though the > seeding failed. OR the job's state is properly updated but the > corresponding thread that is supposed to know when the job is completed (by > looking at the job queue) doesn't properly trigger. > > The architecture of ManifoldCF has many threads that are individually > responsible for transitioning the job state based on the jobqueue. If > somehow the jobstate winds up not in the right state then those threads > will not do the right thing. > > Karl > > > On Wed, Mar 23, 2022 at 11:08 AM Julien Massiera < > julien.massi...@francelabs.com> wrote: > >> Hi Karl, >> >> I had some time to investigate the problem I exposed in my first mail, >> and here is the behavior I observed: >> >> 1/ On the first and only one document of the seeding phase encountered, a >> runtime exception is triggered >> 2/ The runtime exception is catched by the WorkerThread, logged, and the >> WorkerThread stays alive (line 856 of the WorkerThread class) >> 3/ The WorkerThread calls the getDocument method of its documentQueue >> (line 121 of the WorkerThread class) >> 4/ The documentQueue ends in an infinite 'wait' state because the queue >> size is 0 and the resetFlag is false (lines 109 and 110 of the >> DocumentQueue class) >> 5/ Because of the infinite 'wait' state of the documentQueue, the job >> stays freezed on the 'running' state and it is impossible to stop it until >> the Agent is restarted >> >> I don't know much about the WorkerThread and the DocumentQueue logic, so >> from there, I really need your help to understand this behavior and to >> figure out what can be done to prevent the job from hanging in that case, >> which, I assume, can happen in other circumstances with other repository >> connectors >> >> Regards, >> Julien >> >> -Message d'origine- >> De : Julien Massiera >> Envoyé : jeudi 24 février 2022 15:08 >> À : dev@manifoldcf.apache.org >> Objet : RE: WorkerThread runtime exceptions >> >> Yes I understand >> >> -Message d'origine- >> De : Karl Wright >> Envoyé : jeudi 24 février 2022 14:59 >> À : dev >> Objet : Re: WorkerThread runtime exceptions >> >> I'm currently completely consumed with upgrading dependencies for Tika >> and CXF. This is a massive job and won't be done for probably another week >> or two. Once that is done I can try to look into your concern. >> >> Karl >> >> >> On Thu, Feb 24, 2022 at 8:47 AM Julien Massiera < >> julien.massi...@francelabs.com> wrote: >> >> > Hi, >> > >> > >> > >> > I have faced a situation where the MCF agent was still up but was not >> > doing anything after a runtime exception. >> > >> > >> > >> > My use case was the following : >> > I have updated the libs used by a repository connector but forgot one. >> > During doc processing, a runtime exception < >> > java.lang.NoSuchMethodError > has been throwed because the sub >> > dependency lib was not up to date and thus the method called was >> > missing. The exception was catched by the WorkerThread and displayed < >> > Error tossed: .. > but then nothing and the job stayed in running >> > status and I was not able to abort it until I killed and I restarted >> > the agent. >> > >> > >> > >> > The catching clause is located in the WorkerThread class at lines >> 853-857. >> > I >> > know this is a particular case but I am not sure that the fact the >> > agent hangs after this exception is a normal behavior and furthermore >> > I can imagine that it can happen with other unkown runtime exceptions. >> > Is there something we can do to avoid the agent to be hanging in those >> cases ? >> > >> > >> > >> > Regards, >> > >> > Julien >> > >> > >> > >> > >> >> >>
Re: WorkerThread runtime exceptions
' 1/ On the first and only one document of the seeding phase encountered, a runtime exception is triggered' The worker threads do not handle seeding. If a runtime exception takes place during seeding, no documents will be queued, and that is the problem. The state of the job must be incorrectly updated even though the seeding failed. OR the job's state is properly updated but the corresponding thread that is supposed to know when the job is completed (by looking at the job queue) doesn't properly trigger. The architecture of ManifoldCF has many threads that are individually responsible for transitioning the job state based on the jobqueue. If somehow the jobstate winds up not in the right state then those threads will not do the right thing. Karl On Wed, Mar 23, 2022 at 11:08 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi Karl, > > I had some time to investigate the problem I exposed in my first mail, and > here is the behavior I observed: > > 1/ On the first and only one document of the seeding phase encountered, a > runtime exception is triggered > 2/ The runtime exception is catched by the WorkerThread, logged, and the > WorkerThread stays alive (line 856 of the WorkerThread class) > 3/ The WorkerThread calls the getDocument method of its documentQueue > (line 121 of the WorkerThread class) > 4/ The documentQueue ends in an infinite 'wait' state because the queue > size is 0 and the resetFlag is false (lines 109 and 110 of the > DocumentQueue class) > 5/ Because of the infinite 'wait' state of the documentQueue, the job > stays freezed on the 'running' state and it is impossible to stop it until > the Agent is restarted > > I don't know much about the WorkerThread and the DocumentQueue logic, so > from there, I really need your help to understand this behavior and to > figure out what can be done to prevent the job from hanging in that case, > which, I assume, can happen in other circumstances with other repository > connectors > > Regards, > Julien > > -Message d'origine- > De : Julien Massiera > Envoyé : jeudi 24 février 2022 15:08 > À : dev@manifoldcf.apache.org > Objet : RE: WorkerThread runtime exceptions > > Yes I understand > > -Message d'origine- > De : Karl Wright > Envoyé : jeudi 24 février 2022 14:59 > À : dev > Objet : Re: WorkerThread runtime exceptions > > I'm currently completely consumed with upgrading dependencies for Tika and > CXF. This is a massive job and won't be done for probably another week or > two. Once that is done I can try to look into your concern. > > Karl > > > On Thu, Feb 24, 2022 at 8:47 AM Julien Massiera < > julien.massi...@francelabs.com> wrote: > > > Hi, > > > > > > > > I have faced a situation where the MCF agent was still up but was not > > doing anything after a runtime exception. > > > > > > > > My use case was the following : > > I have updated the libs used by a repository connector but forgot one. > > During doc processing, a runtime exception < > > java.lang.NoSuchMethodError > has been throwed because the sub > > dependency lib was not up to date and thus the method called was > > missing. The exception was catched by the WorkerThread and displayed < > > Error tossed: .. > but then nothing and the job stayed in running > > status and I was not able to abort it until I killed and I restarted > > the agent. > > > > > > > > The catching clause is located in the WorkerThread class at lines > 853-857. > > I > > know this is a particular case but I am not sure that the fact the > > agent hangs after this exception is a normal behavior and furthermore > > I can imagine that it can happen with other unkown runtime exceptions. > > Is there something we can do to avoid the agent to be hanging in those > cases ? > > > > > > > > Regards, > > > > Julien > > > > > > > > > > >
Re: Change way of logging OOM and InterruptedException in WorkerThread
Yes, you can do that. Thanks! Karl On Tue, Mar 15, 2022 at 2:19 PM Julien Massiera < julien.massi...@francelabs.com> wrote: > Then I'll create a ticket to just add a date to the system.err.println > output of the OOM, is it ok ? > > -Message d'origine- > De : Karl Wright > Envoyé : mardi 15 mars 2022 19:10 > À : dev > Objet : Re: Change way of logging OOM and InterruptedException in > WorkerThread > > (1) InterruptedExceptions are thrown during normal shutdown and should > generally not be logged. This is by design. > (2) OOM conditions can prevent logging from working. So if you want to > log those, fine, but please retain the System.err output. > > Karl > > > On Tue, Mar 15, 2022 at 1:51 PM Julien Massiera < > julien.massi...@francelabs.com> wrote: > > > Hi, > > > > > > > > In the WorkerThread class, OOM are logged using System.err (l849) and > > therefore cannot be formated, but more importantly, the timestamp is > > lost, which can be very problematic to investigate issues. Also, > > InterruptedExceptions are not logged at all (l844) ! > > > > Can we use Logging.threads.fatal for OOM and Logging.threads.error > > for InterruptedExceptions instead ? > > > > > > > > If it is ok, I can create a ticket and make the changes myself. > > > > > > > > Regards, > > > > Julien > > > > > > > > > > > > > >
Re: Change way of logging OOM and InterruptedException in WorkerThread
(1) InterruptedExceptions are thrown during normal shutdown and should generally not be logged. This is by design. (2) OOM conditions can prevent logging from working. So if you want to log those, fine, but please retain the System.err output. Karl On Tue, Mar 15, 2022 at 1:51 PM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi, > > > > In the WorkerThread class, OOM are logged using System.err (l849) and > therefore cannot be formated, but more importantly, the timestamp is lost, > which can be very problematic to investigate issues. Also, > InterruptedExceptions are not logged at all (l844) ! > > Can we use Logging.threads.fatal for OOM and Logging.threads.error for > InterruptedExceptions instead ? > > > > If it is ok, I can create a ticket and make the changes myself. > > > > Regards, > > Julien > > > > > >
Re: Log4j Update Doubt
We cannot do back patches of older versions of ManifoldCF. There is a new release which shipped in January that addresses log4j issues. I suggest updating to that. Karl On Tue, Mar 15, 2022 at 8:59 AM ritika jain wrote: > Hi, > > How manifoldcf uses log4j files in bin directory/distribution. If this is > the location "D:\\Manifoldcf\apache-manifoldcf-2.14\lib" that is the lib > folder only.(for physical file presence) > > Also if the log4j dependency issue has been resolved and the version 2.15 > or higher is updated, then will it be reflected in Manifoldcf 2.14 version > also. If not , can you help to let me know at what places the log4j2.4.1 > jar files should be replaced with 2.15. > > When the log4j2.15 jar has been manually replaced in 'lib' folder and the > older version is deleted (2.4.1), got this as error > [image: image.png] > > What other location is needed to have the latest jar file. > > Thanks > Ritika >
Re: WorkerThread runtime exceptions
I'm currently completely consumed with upgrading dependencies for Tika and CXF. This is a massive job and won't be done for probably another week or two. Once that is done I can try to look into your concern. Karl On Thu, Feb 24, 2022 at 8:47 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi, > > > > I have faced a situation where the MCF agent was still up but was not doing > anything after a runtime exception. > > > > My use case was the following : > I have updated the libs used by a repository connector but forgot one. > During doc processing, a runtime exception < java.lang.NoSuchMethodError > > has been throwed because the sub dependency lib was not up to date and thus > the method called was missing. The exception was catched by the > WorkerThread > and displayed < Error tossed: .. > but then nothing and the job stayed in > running status and I was not able to abort it until I killed and I > restarted > the agent. > > > > The catching clause is located in the WorkerThread class at lines 853-857. > I > know this is a particular case but I am not sure that the fact the agent > hangs after this exception is a normal behavior and furthermore I can > imagine that it can happen with other unkown runtime exceptions. Is there > something we can do to avoid the agent to be hanging in those cases ? > > > > Regards, > > Julien > > > >
[jira] [Commented] (CONNECTORS-1622) Upgrade to Tika 1.28.x
[
https://issues.apache.org/jira/browse/CONNECTORS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496796#comment-17496796
]
Karl Wright commented on CONNECTORS-1622:
-
This issue is now being worked on. There are several new dependencies we have
to include at the appropriate places:
{code}
[INFO] | +- com.googlecode.plist:dd-plist:jar:1.23:compile
[INFO] | +- org.apache.pdfbox:preflight:jar:2.0.25:compile
[INFO] | +- org.apache.pdfbox:xmpbox:jar:2.0.25:compile
[INFO] | | +- org.bouncycastle:bcutil-jdk15on:jar:1.70:compile
[INFO] | | \- com.zaxxer:SparseBitSet:jar:1.2:compile
[INFO] | +- org.tallison:isoparser:jar:1.9.41.7:compile
[INFO] | +- org.tallison:metadata-extractor:jar:2.15.0.1:compile
[INFO] | | \- org.tallison.xmp:xmpcore-shaded:jar:6.1.11:compile
[INFO] | | \- org.apache.cxf:cxf-rt-security:jar:3.5.0:compile
[INFO] | | \- com.zaxxer:HikariCP-java7:jar:2.4.13:compile
[INFO] | | \- com.mchange:mchange-commons-java:jar:0.2.19:compile
[INFO] | | | \- org.apache.sis.core:sis-feature:jar:1.1:compile
{code}
We also have to include mention of these in the licensing text files. This
will likely take a few more days to complete and test.
> Upgrade to Tika 1.28.x
> --
>
> Key: CONNECTORS-1622
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1622
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Tika extractor
>Affects Versions: ManifoldCF 2.13
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Tika has released 1.28.1. Changes can be found from here:
> [http://tika.apache.org/1.28.1/|http://tika.apache.org/1.23/]
>
> please see :[TIKA-3625|https://issues.apache.org/jira/browse/TIKA-3625] and
> CONNECTORS-1683
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1699) Upgrade to Tika 2.x
[
https://issues.apache.org/jira/browse/CONNECTORS-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496421#comment-17496421
]
Karl Wright commented on CONNECTORS-1699:
-
Attempting to find the dependencies for 2.3.0 comes up with this error:
{code}
[ERROR] Failed to execute goal on project mcf-tika-connector: Could not resolve
dependencies for project
org.apache.manifoldcf:mcf-tika-connector:jar:2.21-SNAPSHOT: Could not find
artifact org.apache.tika:tika-parsers:jar:2.3.0 in central
(https://repo.maven.apache.org/maven2)
{code}
It looks like 2.x repackaged everything. Somebody therefore needs to invest
the basic time to figure out where everything went. A port to 2.x will likely
require substantial work and time and I certainly do not have it right now.
Volunteers welcome.
> Upgrade to Tika 2.x
> ---
>
> Key: CONNECTORS-1699
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1699
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Tika extractor
>Affects Versions: ManifoldCF 2.21
>Reporter: Cihad Guzel
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Tika has a new version as 2.x . We can support the new version instead of 1.x
> . There is a migration document here:
> [https://cwiki.apache.org/confluence/display/TIKA/Migrating+to+Tika+2.0.0]
> Tika has released 2.3.0. Changes can be found from here:
> https://tika.apache.org/2.3.0/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: New LDAP Mapper Connector
We have received zero comment on it. So I think we'll just have to land it and await comment. Karl On Mon, Feb 14, 2022 at 5:42 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi, > > Any news about this new connector integration ? > > Regards, > Julien > > -Message d'origine- > De : Karl Wright > Envoyé : mardi 11 janvier 2022 18:51 > À : dev > Objet : Re: New LDAP Mapper Connector > > Thank you for doing this Julien. > > Let's see what others say and if we hear nothing in a few days, onto trunk > it goes. > > Karl > > > On Tue, Jan 11, 2022 at 12:37 PM Julien Massiera < > julien.massi...@francelabs.com> wrote: > > > Hi everyone, > > > > > > > > I have created the following issue > > https://issues.apache.org/jira/browse/CONNECTORS-1692 to develop a new > > LDAP Mapper connector. I have created the corresponding branch and > > commited the connector to it. I did an ant and a maven build to make > > sure everything is fine. If anyone could check if it is also fine on > > his side, it would be greatly appreciated, and we could then merge it > > to the trunk ! > > > > > > > > Regards, > > > > Julien > > > > > > > > > >
Re: Manifoldcf freezes and sit idle
As I've mentioned before, the best way to diagnose problems like this is to get a thread dump of the agents process. There are many potential reasons it could occur, ranging from stuck locks to resource starvation. What locking model are you using? Karl On Mon, Jan 31, 2022 at 6:02 AM ritika jain wrote: > Hi, > > I am using Manifoldcf 2.14, web connector and Elastic as output. > I have observed after a certain time period of continuous run job freezes > and does not do/process anything. Simple history shows nothing after a > certain process, and it's not for one job it has been observed for 3 > different jobs , also checked for a certain document (that seems NOT to be > the case). > > Only after restarting the docker container helps. After restarting the job > continues > > What can be the possible reason for this? How it can be prevented for PROD. > > Thanks > Ritika >
[jira] [Commented] (CONNECTORS-1695) Sitemap xml not detected in version 2.17 webconnector
[
https://issues.apache.org/jira/browse/CONNECTORS-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482114#comment-17482114
]
Karl Wright commented on CONNECTORS-1695:
-
The interestingMimeTypes are mime types that may be HTML or XHTML, or documents
where links cannot be extracted but where the content can be indexed. What
actually happens to a given document depends on what is actually in it rather
than what the mime type says. The mimetypes are a crude filter only.
The code that we'd want to modify would be the extractLinks code:
{code}
// Now, extract links.
// We'll call the "link extractor" series, so we can plug more
stuff in over time.
boolean indexDocument =
extractLinks(documentIdentifier,activities,filter);
{code}
The code for this method is:
{code}
/** Code to extract links from an already-fetched document. */
protected boolean extractLinks(String
documentIdentifier, IProcessActivity activities, DocumentURLFilter filter)
throws ManifoldCFException, ServiceInterruption
{
ProcessActivityRedirectionHandler redirectHandler = new
ProcessActivityRedirectionHandler(documentIdentifier,activities,filter);
handleRedirects(documentIdentifier,redirectHandler);
if (Logging.connectors.isDebugEnabled() && redirectHandler.shouldIndex() ==
false)
Logging.connectors.debug("Web: Not
indexing document '"+documentIdentifier+"' because of redirection");
// For html, we don't want any actions, because we don't do form submission.
ProcessActivityHTMLHandler htmlHandler = new
ProcessActivityHTMLHandler(documentIdentifier,activities,filter,metaRobotsTagsUsage);
handleHTML(documentIdentifier,htmlHandler);
if (Logging.connectors.isDebugEnabled()
&& htmlHandler.shouldIndex() == false)
Logging.connectors.debug("Web: Not indexing document '"+documentIdentifier+"'
because of HTML robots or content tags prohibiting indexing");
ProcessActivityXMLHandler xmlHandler = new
ProcessActivityXMLHandler(documentIdentifier,activities,filter);
handleXML(documentIdentifier,xmlHandler);
if
(Logging.connectors.isDebugEnabled() && xmlHandler.shouldIndex() == false)
Logging.connectors.debug("Web: Not
indexing document '"+documentIdentifier+"' because of XML robots or content
tags prohibiting indexing");
// May add more later for other extraction tasks.
return htmlHandler.shouldIndex() && redirectHandler.shouldIndex() &&
xmlHandler.shouldIndex();
}
{code}
Note that there are three different parsing attempts made: HTML, XML (which is
I believe RSS feeds only at this point) and redirection pages. You could add a
fourth.
Most of these invoke the fuzzyml parser, which is a bottom-up parser with
overrides for specific interesting tags. Even though the sitemap xml is
supposedly well formed, you wouldn't want to bet on it, and the fuzzyml parser
would be a reasonable technology to do parsing of this kind since it is quite
resilient against syntax errors of all kinds.
So the trick would be to identify the tag structure of a sitemap document and
extend the overrides present for the parser the web connector is using to
understand that xml syntax IN ADDITION TO the xhtml it already understands.
> Sitemap xml not detected in version 2.17 webconnector
> -
>
> Key: CONNECTORS-1695
>
[jira] [Commented] (CONNECTORS-1695) Sitemap xml not detected in version 2.17 webconnector
[ https://issues.apache.org/jira/browse/CONNECTORS-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17481420#comment-17481420 ] Karl Wright commented on CONNECTORS-1695: - The problem is that there is no parser in the web connector for the sitemap xml and text/xml is not sufficient indication as to what kind of XML it is to do anything special with it. The web crawler accepts files of this kind but assumes they are xhtml, which doesn't work because these aren't. As for processing the whole file, well certainly. That is what should happen. The file should be read in and parsed and the documents within queued for further processing. > Sitemap xml not detected in version 2.17 webconnector > - > > Key: CONNECTORS-1695 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1695 > Project: ManifoldCF > Issue Type: Bug > Components: Web connector >Affects Versions: ManifoldCF 2.17 >Reporter: DK >Priority: Major > > Trying to index sitemap xml and web connector index the whole xml into solr. > Please fix in version 2.17. > If it is any special config that needs to be taken care, please add here and > add in documentation to make it clear. > > Sitemap.xml: > http://www.sitemaps.org/schemas/sitemap/0.9;> > > https:///sitemap_1.xml > 2022-01-21T16:04:45Z > > > > sitemap_1.xml: > > > https:// > 2018-10-31T11:25:27Z > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1604) Documentation to run Manifold over SSL
[ https://issues.apache.org/jira/browse/CONNECTORS-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480386#comment-17480386 ] Karl Wright commented on CONNECTORS-1604: - Two points: First, the standard configuration files for jetty are included in all example distributions and are provided to Jetty during startup of the webapps. So you should be able to configure Jetty any way you like. We are not Jetty experts, however, and how you do this changes from Jetty release to release. The best we can do is point at Jetty documentation somewhere. Second, the examples contain webapp files which do not need to be deployed on Jetty but can easily be deployed on Tomcat. If you cannot get anywhere with Jetty you always have that option. > Documentation to run Manifold over SSL > -- > > Key: CONNECTORS-1604 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1604 > Project: ManifoldCF > Issue Type: Improvement > Components: Documentation >Reporter: DK >Priority: Major > > Documentation does not talk about running manifold over SSL anywhere. I am > sure lot of users who deployed manifold in prod configured ssl. I tried to > configure SSL without success. Can someone point me to instructions to run > manifold over SSL please? > Thanks -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (CONNECTORS-1665) WebConnector: Add activity records for excluded URLs
[ https://issues.apache.org/jira/browse/CONNECTORS-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1665: --- Assignee: Julien Massiera > WebConnector: Add activity records for excluded URLs > - > > Key: CONNECTORS-1665 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1665 > Project: ManifoldCF > Issue Type: Improvement > Components: Web connector >Affects Versions: ManifoldCF 2.18 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Trivial > Fix For: ManifoldCF 2.19 > > Attachments: patch-CONNECTORS-1665 > > > It would be interesting to add activity records in the WebConnector to keep > track of excluded URLs that match an exclude filter -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1665) WebConnector: Add activity records for excluded URLs
[ https://issues.apache.org/jira/browse/CONNECTORS-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480238#comment-17480238 ] Karl Wright commented on CONNECTORS-1665: - Please go ahead and commit this. > WebConnector: Add activity records for excluded URLs > - > > Key: CONNECTORS-1665 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1665 > Project: ManifoldCF > Issue Type: Improvement > Components: Web connector >Affects Versions: ManifoldCF 2.18 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Trivial > Fix For: ManifoldCF 2.19 > > Attachments: patch-CONNECTORS-1665 > > > It would be interesting to add activity records in the WebConnector to keep > track of excluded URLs that match an exclude filter -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Add logs to repository connectors
Having a second log is pretty non-standard, and obscures the ordering of events that may be pertinent. So I am not thrilled with that idea. However, the current logging system does not make it easy to determine where a log message is coming from, nor can you filter log messages with grep easily. It would be better to include enough information in the ONE manifoldcf.log file so that you know what connector the log message is coming from. We already have individual loggers set up that are "catch all" buckets for events of various kinds, but the logger itself might want to have more information, e.g. the class that did the logging, as an optional argument. Or - we could use log4j's native capabilities. All that is needed to do that is to simply create a static logger in every connector class from which you wish to log, and write directly to that one. Karl On Tue, Jan 18, 2022 at 10:29 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi all, > > > > In order to improve the tracking of documents processing status, and in > particular when something goes wrong with MCF (like hanging processes > without obvious causes), I would like to propose to add specific logs into > each repository connector. One log at the beginning of each > documentIdentifier processing in the processDocuments method, and one at > the > end, so that, at any time, we can easily tell which documents are being > processed. > > To implement that, I was thinking of a common class that would write the > logs, with a custom log level (between INFO and DEBUG), so that the logs > could be easily isolated into a specific file through log4j conf. The class > would be stored in the org.apache.manifoldcf.crawler.system package like > the > Logging class. > > > > What do you think about that ? > > > > Regards, > > Julien > >
Re: Zookeeper locks issue
If you are using existing connectors we are shipping, then all you need to
do is tell us which ones are involved in your entire job pipeline.
If you have any CUSTOM connectors involved, then I need to know what they
are doing, especially if the connect() method may hang.
It sounds like your repository connector is Sharepoint. That is fine, but
it is only one connector. What is your output connector? Do you have any
transformation connections? etc.
Karl
On Tue, Jan 11, 2022 at 4:57 PM Cedric Ulmer
wrote:
> Hi,
>
>
> would you also want to have a look at the disconnect() method ?
>
>
> Cedric
>
>
>
> De : Cedric Ulmer
> Envoyé : jeudi 6 janvier 2022 16:35:55
> À : dev
> Objet : RE: Zookeeper locks issue
>
> Hi Karl,
>
>
> to bounce back on this issue, since you asked about how we initiate the
> connections, here is how we implement the connect() method. We are talking
> here about a sharepoint datasource :
>
>
> public void connect(final ConfigParams configParams) {
> super.connect(configParams);
>
> spVersion =
> configParams.getParameter(SharePointConfiguration.Server.VERSION);
> connectionType =
> configParams.getParameter(SharePointConfiguration.Server.CONNECTION_TYPE);
> sts = configParams.getParameter(SharePointConfiguration.Server.STS);
> protocol =
> configParams.getParameter(SharePointConfiguration.Server.PROTOCOL);
> host = configParams.getParameter(SharePointConfiguration.Server.HOST);
> port = configParams.getParameter(SharePointConfiguration.Server.PORT);
> path = configParams.getParameter(SharePointConfiguration.Server.PATH);
> username =
> configParams.getParameter(SharePointConfiguration.Server.USERNAME);
> password =
> configParams.getObfuscatedParameter(SharePointConfiguration.Server.PASSWORD);
> socketTimeoutString =
> configParams.getParameter(SharePointConfiguration.Server.SOCKET_TIMEOUT);
> connectionTimeoutString =
> configParams.getParameter(SharePointConfiguration.Server.CONNECTION_TIMEOUT);
>
> proxyHost =
> configParams.getParameter(SharePointConfiguration.Server.PROXY_HOST);
> proxyPortStr =
> configParams.getParameter(SharePointConfiguration.Server.PROXY_PORT);
> proxyUser =
> configParams.getParameter(SharePointConfiguration.Server.PROXY_USER);
> proxyPassword =
> configParams.getObfuscatedParameter(SharePointConfiguration.Server.PROXY_PASSWORD);
> proxyDomain =
> configParams.getParameter(SharePointConfiguration.Server.PROXY_DOMAIN);
>
> sharePointUrl = protocol + "://" + host;
> if (port != null && !port.isEmpty() && !port.equals("80") &&
> !port.equals("443")) {
> sharePointUrl += ":" + port;
> }
> sharePointUrl += path;
> if (sharePointUrl.endsWith("/")) {
> sharePointUrl = sharePointUrl.substring(0, sharePointUrl.length() -
> 1);
> }
>
> final String fullDelta =
> configParams.getParameter(SharePointConfiguration.Delta.FULL_DELTA);
> if (fullDelta != null && !fullDelta.isEmpty()) {
> final boolean isFullDelta = Boolean.parseBoolean(fullDelta);
> if (isFullDelta) {
> connectorModel = MODEL_ADD_CHANGE;
> } else {
> connectorModel = MODEL_ADD_CHANGE_DELETE;
> }
> } else {
> connectorModel = MODEL_ADD_CHANGE_DELETE;
> }
>
> }
>
> Regards,
>
>
> Cédric
>
>
> France Labs – Search Experts
> Datafari – Discover our Version 5 released in 2021
> www.datafari.com<http://www.datafari.com/>
>
>
>
> De : Karl Wright
> Envoyé : vendredi 10 décembre 2021 15:32
> À : dev
> Objet : Re: Zookeeper locks issue
>
> You haven't told me anything about the connectors involved in this job.
> For every connector, there are connection pools of limited sizes,
> partitioned by configuration parameters. Throttling is done also on these
> pools, and those require locks.
>
> So if a connector has code for establishing a connection that can hang, it
> can cause carnage. Connection establishment in the connector contract
> involves several methods, which I am sure you know. If any one of those
> can block, that is bad. Instead they should just throw an exception.
>
> Karl
>
>
> On Fri, Dec 10, 2021 at 8:15 AM wrote:
>
> > Hi,
> >
> > Not sure to understand what you mean by establishing a connection, what
> > kind of connection ? If you can explain to me when the agent needs to set
> > and release locks, I'll be able to better investigate.
> >
> > Julien
> >
> > -Messag
Re: New LDAP Mapper Connector
Thank you for doing this Julien. Let's see what others say and if we hear nothing in a few days, onto trunk it goes. Karl On Tue, Jan 11, 2022 at 12:37 PM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi everyone, > > > > I have created the following issue > https://issues.apache.org/jira/browse/CONNECTORS-1692 to develop a new > LDAP > Mapper connector. I have created the corresponding branch and commited the > connector to it. I did an ant and a maven build to make sure everything is > fine. If anyone could check if it is also fine on his side, it would be > greatly appreciated, and we could then merge it to the trunk ! > > > > Regards, > > Julien > > > >
[RESULT] [VOTE] Release Apache ManifoldCF 2.21, RC0
On Mon, Jan 3, 2022 at 12:31 PM Karl Wright wrote: > Four +1's. >72hrs. Vote passes! > Karl > > > On Mon, Jan 3, 2022 at 9:41 AM Piergiorgio Lucidi > wrote: > >> Hi Karl, >> >> you are right, it's something related to system variables for the Maven >> build, so I think that we could fix it in the next version. >> Practically we have Ant tests run correctly, so +1 from me. >> >> I'll raise a new issue to work on this after the release. >> Cheers, >> PJ >> >> >> Il giorno lun 3 gen 2022 alle ore 15:35 Karl Wright >> ha >> scritto: >> >> > Hi Piergiorgio - unless this is a regression, I don't think we should >> hold >> > the release on the maven tests. What do you think? >> > Karl >> > >> > >> > On Mon, Jan 3, 2022 at 9:15 AM Julien Massiera < >> > julien.massi...@francelabs.com> wrote: >> > >> > > Hi all, >> > > >> > > just integrated the RC in Datafari 5. Tested an indexing of 2 websites >> > and >> > > a 10k files share, no problems showed up, so +1 for me >> > > >> > > Julien >> > > >> > > -Message d'origine- >> > > De : Karl Wright >> > > Envoyé : vendredi 31 décembre 2021 19:57 >> > > À : dev >> > > Objet : Re: [VOTE] Release Apache ManifoldCF 2.21, RC0 >> > > >> > > We still need one more vote for the release to be finalized. >> > > >> > > Karl >> > > >> > > >> > > On Wed, Dec 29, 2021 at 9:28 AM Karl Wright >> wrote: >> > > >> > > > Ran tests; +1 from me. >> > > > Karl >> > > > >> > > > >> > > > On Wed, Dec 29, 2021 at 8:00 AM Markus Schuch > > >> > > > wrote: >> > > > >> > > >> Hi, >> > > >> >> > > >> +1 from me >> > > >> >> > > >> Checked out the tag 2.21-rc0 and ran build and hsqldb based tests >> > > >> successfully Did a simple example based test crawl of a webpage >> with >> > > >> , ingest URIs are built correctly Checked the >> logs, >> > > >> found no errors or warnings Checked that mail notifications work >> > > >> (tested with mailhog) Travis ci reports green for the release tag: >> > > >> >> https://app.travis-ci.com/github/apache/manifoldcf/builds/243992093 >> > > >> Checked that Log4j 2.17 is contained in the built distribution >> > > >> >> > > >> Yesterday Log4J 2.17.1 was released due to >> > > >> https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832 >> > > >> It is an RCE vulnerability, but only if the attacker has already >> the >> > > >> capability to modify the log4j configuration. >> > > >> I don't think that we have to respin the release for this one (base >> > > >> cvss score is 6.6). >> > > >> >> > > >> Many thanks for managing the release Karl and many thanks to all >> > > >> contributors. >> > > >> >> > > >> I wish everyone a happy new year 2022. >> > > >> >> > > >> Markus >> > > >> >> > > >> Am 26.12.2021 um 12:30 schrieb Karl Wright: >> > > >> > Hi, >> > > >> > >> > > >> > Please vote on whether to release Apache ManifoldCF 2.21, RC0. >> > > >> > The release candidate can be found at >> > > >> > >> > > >> >> > > >> https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.21 >> > . >> > > >> > There is also a release tag at >> > > >> > >> https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.21-RC0. >> > > >> > >> > > >> > As everyone is aware, this release updates log4j to version 2.17. >> > > >> > It >> > > >> also >> > > >> > fixes numerous other build-related issues on Unix systems. Other >> > > >> changes >> > > >> > are listed in CHANGES.txt, as always. >> > > >> > >> > > >> > Karl >> > > >> > >> > > >> >> > > > >> > > >> > > >> > >> >> >> -- >> Piergiorgio >> >
Re: [VOTE] Release Apache ManifoldCF 2.21, RC0
Four +1's. >72hrs. Vote passes! Karl On Mon, Jan 3, 2022 at 9:41 AM Piergiorgio Lucidi wrote: > Hi Karl, > > you are right, it's something related to system variables for the Maven > build, so I think that we could fix it in the next version. > Practically we have Ant tests run correctly, so +1 from me. > > I'll raise a new issue to work on this after the release. > Cheers, > PJ > > > Il giorno lun 3 gen 2022 alle ore 15:35 Karl Wright > ha > scritto: > > > Hi Piergiorgio - unless this is a regression, I don't think we should > hold > > the release on the maven tests. What do you think? > > Karl > > > > > > On Mon, Jan 3, 2022 at 9:15 AM Julien Massiera < > > julien.massi...@francelabs.com> wrote: > > > > > Hi all, > > > > > > just integrated the RC in Datafari 5. Tested an indexing of 2 websites > > and > > > a 10k files share, no problems showed up, so +1 for me > > > > > > Julien > > > > > > -Message d'origine- > > > De : Karl Wright > > > Envoyé : vendredi 31 décembre 2021 19:57 > > > À : dev > > > Objet : Re: [VOTE] Release Apache ManifoldCF 2.21, RC0 > > > > > > We still need one more vote for the release to be finalized. > > > > > > Karl > > > > > > > > > On Wed, Dec 29, 2021 at 9:28 AM Karl Wright > wrote: > > > > > > > Ran tests; +1 from me. > > > > Karl > > > > > > > > > > > > On Wed, Dec 29, 2021 at 8:00 AM Markus Schuch > > > > wrote: > > > > > > > >> Hi, > > > >> > > > >> +1 from me > > > >> > > > >> Checked out the tag 2.21-rc0 and ran build and hsqldb based tests > > > >> successfully Did a simple example based test crawl of a webpage with > > > >> , ingest URIs are built correctly Checked the logs, > > > >> found no errors or warnings Checked that mail notifications work > > > >> (tested with mailhog) Travis ci reports green for the release tag: > > > >> https://app.travis-ci.com/github/apache/manifoldcf/builds/243992093 > > > >> Checked that Log4j 2.17 is contained in the built distribution > > > >> > > > >> Yesterday Log4J 2.17.1 was released due to > > > >> https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832 > > > >> It is an RCE vulnerability, but only if the attacker has already the > > > >> capability to modify the log4j configuration. > > > >> I don't think that we have to respin the release for this one (base > > > >> cvss score is 6.6). > > > >> > > > >> Many thanks for managing the release Karl and many thanks to all > > > >> contributors. > > > >> > > > >> I wish everyone a happy new year 2022. > > > >> > > > >> Markus > > > >> > > > >> Am 26.12.2021 um 12:30 schrieb Karl Wright: > > > >> > Hi, > > > >> > > > > >> > Please vote on whether to release Apache ManifoldCF 2.21, RC0. > > > >> > The release candidate can be found at > > > >> > > > > >> > > > > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.21 > > . > > > >> > There is also a release tag at > > > >> > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.21-RC0 > . > > > >> > > > > >> > As everyone is aware, this release updates log4j to version 2.17. > > > >> > It > > > >> also > > > >> > fixes numerous other build-related issues on Unix systems. Other > > > >> changes > > > >> > are listed in CHANGES.txt, as always. > > > >> > > > > >> > Karl > > > >> > > > > >> > > > > > > > > > > > > > > > -- > Piergiorgio >
Re: [VOTE] Release Apache ManifoldCF 2.21, RC0
Hi Piergiorgio - unless this is a regression, I don't think we should hold the release on the maven tests. What do you think? Karl On Mon, Jan 3, 2022 at 9:15 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi all, > > just integrated the RC in Datafari 5. Tested an indexing of 2 websites and > a 10k files share, no problems showed up, so +1 for me > > Julien > > -Message d'origine- > De : Karl Wright > Envoyé : vendredi 31 décembre 2021 19:57 > À : dev > Objet : Re: [VOTE] Release Apache ManifoldCF 2.21, RC0 > > We still need one more vote for the release to be finalized. > > Karl > > > On Wed, Dec 29, 2021 at 9:28 AM Karl Wright wrote: > > > Ran tests; +1 from me. > > Karl > > > > > > On Wed, Dec 29, 2021 at 8:00 AM Markus Schuch > > wrote: > > > >> Hi, > >> > >> +1 from me > >> > >> Checked out the tag 2.21-rc0 and ran build and hsqldb based tests > >> successfully Did a simple example based test crawl of a webpage with > >> , ingest URIs are built correctly Checked the logs, > >> found no errors or warnings Checked that mail notifications work > >> (tested with mailhog) Travis ci reports green for the release tag: > >> https://app.travis-ci.com/github/apache/manifoldcf/builds/243992093 > >> Checked that Log4j 2.17 is contained in the built distribution > >> > >> Yesterday Log4J 2.17.1 was released due to > >> https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832 > >> It is an RCE vulnerability, but only if the attacker has already the > >> capability to modify the log4j configuration. > >> I don't think that we have to respin the release for this one (base > >> cvss score is 6.6). > >> > >> Many thanks for managing the release Karl and many thanks to all > >> contributors. > >> > >> I wish everyone a happy new year 2022. > >> > >> Markus > >> > >> Am 26.12.2021 um 12:30 schrieb Karl Wright: > >> > Hi, > >> > > >> > Please vote on whether to release Apache ManifoldCF 2.21, RC0. > >> > The release candidate can be found at > >> > > >> > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.21. > >> > There is also a release tag at > >> > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.21-RC0. > >> > > >> > As everyone is aware, this release updates log4j to version 2.17. > >> > It > >> also > >> > fixes numerous other build-related issues on Unix systems. Other > >> changes > >> > are listed in CHANGES.txt, as always. > >> > > >> > Karl > >> > > >> > > > >
Re: [VOTE] Release Apache ManifoldCF 2.21, RC0
We still need one more vote for the release to be finalized. Karl On Wed, Dec 29, 2021 at 9:28 AM Karl Wright wrote: > Ran tests; +1 from me. > Karl > > > On Wed, Dec 29, 2021 at 8:00 AM Markus Schuch > wrote: > >> Hi, >> >> +1 from me >> >> Checked out the tag 2.21-rc0 and ran build and hsqldb based tests >> successfully >> Did a simple example based test crawl of a webpage with > href="...">, ingest URIs are built correctly >> Checked the logs, found no errors or warnings >> Checked that mail notifications work (tested with mailhog) >> Travis ci reports green for the release tag: >> https://app.travis-ci.com/github/apache/manifoldcf/builds/243992093 >> Checked that Log4j 2.17 is contained in the built distribution >> >> Yesterday Log4J 2.17.1 was released due to >> https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832 >> It is an RCE vulnerability, but only if the attacker has already the >> capability to modify the log4j configuration. >> I don't think that we have to respin the release for this one (base cvss >> score is 6.6). >> >> Many thanks for managing the release Karl and many thanks to all >> contributors. >> >> I wish everyone a happy new year 2022. >> >> Markus >> >> Am 26.12.2021 um 12:30 schrieb Karl Wright: >> > Hi, >> > >> > Please vote on whether to release Apache ManifoldCF 2.21, RC0. >> > The release candidate can be found at >> > >> https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.21. >> > There is also a release tag at >> > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.21-RC0. >> > >> > As everyone is aware, this release updates log4j to version 2.17. It >> also >> > fixes numerous other build-related issues on Unix systems. Other >> changes >> > are listed in CHANGES.txt, as always. >> > >> > Karl >> > >> >
Re: [VOTE] Release Apache ManifoldCF 2.21, RC0
Ran tests; +1 from me. Karl On Wed, Dec 29, 2021 at 8:00 AM Markus Schuch wrote: > Hi, > > +1 from me > > Checked out the tag 2.21-rc0 and ran build and hsqldb based tests > successfully > Did a simple example based test crawl of a webpage with href="...">, ingest URIs are built correctly > Checked the logs, found no errors or warnings > Checked that mail notifications work (tested with mailhog) > Travis ci reports green for the release tag: > https://app.travis-ci.com/github/apache/manifoldcf/builds/243992093 > Checked that Log4j 2.17 is contained in the built distribution > > Yesterday Log4J 2.17.1 was released due to > https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832 > It is an RCE vulnerability, but only if the attacker has already the > capability to modify the log4j configuration. > I don't think that we have to respin the release for this one (base cvss > score is 6.6). > > Many thanks for managing the release Karl and many thanks to all > contributors. > > I wish everyone a happy new year 2022. > > Markus > > Am 26.12.2021 um 12:30 schrieb Karl Wright: > > Hi, > > > > Please vote on whether to release Apache ManifoldCF 2.21, RC0. > > The release candidate can be found at > > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.21 > . > > There is also a release tag at > > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.21-RC0. > > > > As everyone is aware, this release updates log4j to version 2.17. It > also > > fixes numerous other build-related issues on Unix systems. Other changes > > are listed in CHANGES.txt, as always. > > > > Karl > > >
[VOTE] Release Apache ManifoldCF 2.21, RC0
Hi, Please vote on whether to release Apache ManifoldCF 2.21, RC0. The release candidate can be found at https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.21. There is also a release tag at https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.21-RC0. As everyone is aware, this release updates log4j to version 2.17. It also fixes numerous other build-related issues on Unix systems. Other changes are listed in CHANGES.txt, as always. Karl
Fwd: [jira] [Resolved] (CONNECTORS-1691) CMIS integration test fails on linux
Are we now ready for RC0?
Karl
-- Forwarded message -
From: Markus Schuch (Jira)
Date: Thu, Dec 23, 2021 at 7:07 AM
Subject: [jira] [Resolved] (CONNECTORS-1691) CMIS integration test fails on
linux
To:
[
https://issues.apache.org/jira/browse/CONNECTORS-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Schuch resolved CONNECTORS-1691.
---
Resolution: Cannot Reproduce
> CMIS integration test fails on linux
>
>
> Key: CONNECTORS-1691
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1691
> Project: ManifoldCF
> Issue Type: Bug
> Components: CMIS connector, CMIS Output Connector
>Reporter: Markus Schuch
>Assignee: Markus Schuch
>Priority: Major
>
> When running on linux (tested on Ubuntu 18.04.6 LTS or in TravisCI Ubuntu
Xenial) the
{{org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT}}
fails.
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
103.703 s <<< FAILURE! - in
org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT
> [ERROR]
sanityCheck(org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT)
Time elapsed: 103.411 s <<< ERROR!
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Wrong number
of documents stored in the CMIS Target repo - expected 2, saw 1
> at
org.apache.manifoldcf.agents.output.cmisoutput.tests.APISanityHSQLDBIT.sanityCheck(APISanityHSQLDBIT.java:484)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Thanks to all committers for going the extra mile to get past the log4j multiple CVEs and releases
Marcus and Piergiorgio deserve special thanks for all their hard work. I would like soon to spin the next release of ManifoldCF containing these changes. However, for some of the work being looked at, I am unsure whether open problems still exist or everything is resolved. I'd like to wait until resolution before branching, so please keep me informed. Thanks, Karl
[jira] [Comment Edited] (CONNECTORS-1680) WebConnector: Support the Document Base URL element
[
https://issues.apache.org/jira/browse/CONNECTORS-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461575#comment-17461575
]
Karl Wright edited comment on CONNECTORS-1680 at 12/17/21, 5:22 PM:
The baseDocumentIdentifier code is new for adding the ability to set the base
URL for all relative references, a change you requested and I implemented.
All relative document urls should be based on baseDocumentIdentifier, and that
is initialized to be documentIdentifier at the start. Since this is obviously
not happening you are correct in your diagnosis.
was (Author: kwri...@metacarta.com):
The baseDocumentIdentifier code is new for adding the ability to set the base
URL for all relative references, a change you requested and I implemented.
All relative document urls should be based on baseDocumentIdentifier, and that
is initialized to be documentIdentifier at the start.
> WebConnector: Support the Document Base URL element
> ---
>
> Key: CONNECTORS-1680
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1680
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
>Reporter: Markus Schuch
> Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.21
>
>
> HTML allows to specifiy the base URL to use for all relative URLs in a
> document:
> {code:java}
>
>
>
> https://example.org/"/>
> ...
>
>...
> {code}
> [https://developer.mozilla.org/de/docs/Web/HTML/Element/base]
> The Web Connector should respect this element when handling relative links.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1680) WebConnector: Support the Document Base URL element
[
https://issues.apache.org/jira/browse/CONNECTORS-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461575#comment-17461575
]
Karl Wright commented on CONNECTORS-1680:
-
The baseDocumentIdentifier code is new for adding the ability to set the base
URL for all relative references, a change you requested and I implemented.
All relative document urls should be based on baseDocumentIdentifier, and that
is initialized to be documentIdentifier at the start.
> WebConnector: Support the Document Base URL element
> ---
>
> Key: CONNECTORS-1680
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1680
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
>Reporter: Markus Schuch
> Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.21
>
>
> HTML allows to specifiy the base URL to use for all relative URLs in a
> document:
> {code:java}
>
>
>
> https://example.org/"/>
> ...
>
>...
> {code}
> [https://developer.mozilla.org/de/docs/Web/HTML/Element/base]
> The Web Connector should respect this element when handling relative links.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Fwd: [jira] [Commented] (CONNECTORS-1686) Solr Ingester: issues with CursorMark
Marcus, if Julien does not respond soon, please just reverse merge his commit. svn -c- https://svn.apache.org/repos/asf/manifoldcf/trunk Leaving a broken trunk around is not acceptable. Karl -- Forwarded message - From: Markus Schuch (Jira) Date: Wed, Dec 15, 2021 at 12:54 PM Subject: [jira] [Commented] (CONNECTORS-1686) Solr Ingester: issues with CursorMark To: [ https://issues.apache.org/jira/browse/CONNECTORS-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460141#comment-17460141 ] Markus Schuch commented on CONNECTORS-1686: --- trunk does not compile: https://app.travis-ci.com/github/apache/manifoldcf/jobs/552176370#L8015 > Solr Ingester: issues with CursorMark > - > > Key: CONNECTORS-1686 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1686 > Project: ManifoldCF > Issue Type: Bug > Components: Lucene/SOLR connector >Affects Versions: ManifoldCF 2.20 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Major > Fix For: ManifoldCF 2.21 > > > The Solr Ingester connector may have some issues with the response.getNextCursorMark() method when processing requests responses. Indeed, sometimes the response contains errors and/or is malformed, and this method raises an exception that is currently not handled. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1685) Alfresco Webscript Connector: Integration test fails on JDKs having patch level > 255
[
https://issues.apache.org/jira/browse/CONNECTORS-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460023#comment-17460023
]
Karl Wright commented on CONNECTORS-1685:
-
Hi [~piergiorgioluc...@gmail.com], sorry for the late response.
We can perhaps conditionally use the JDK 11 version when running JDK 11, and
the JDK 8 version when running the older versions of JDK 8. But the download
time is huge so not sure we want to subject people to that just to run a test.
I therefore also suggest that the ant job select the version of alfresco to
download based on the java version it's using at the time. We do this kind of
thing already for unix vs. windows for ES.
What do you think?
> Alfresco Webscript Connector: Integration test fails on JDKs having patch
> level > 255
> -
>
> Key: CONNECTORS-1685
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1685
> Project: ManifoldCF
> Issue Type: Bug
> Components: Alfresco webscript connector, Build
>Reporter: Markus Schuch
>Assignee: Piergiorgio Lucidi
>Priority: Major
>
> Currently the {{ant test}} job executed by TravisCI in the github mirror of
> manifoldcf fails
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1685) Alfresco Webscript Connector: Integration test fails on JDKs having patch level > 255
[
https://issues.apache.org/jira/browse/CONNECTORS-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459104#comment-17459104
]
Karl Wright commented on CONNECTORS-1685:
-
I've disabled the test conditionally, when the patch level of jdk8 exceeds 255.
See:
r1895950
Also, I've had a chat with Piergiorgio and he's going to look into updating the
test so it works again on all systems. It may be that no fix will be available
until we turn off JDK 8 support for good though.
> Alfresco Webscript Connector: Integration test fails on JDKs having patch
> level > 255
> -
>
> Key: CONNECTORS-1685
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1685
> Project: ManifoldCF
> Issue Type: Bug
> Components: Alfresco webscript connector, Build
>Reporter: Markus Schuch
>Priority: Major
>
> Currently the {{ant test}} job executed by TravisCI in the github mirror of
> manifoldcf fails
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1685) Alfresco Webscript Connector: Integration test fails on JDKs having patch level > 255
[
https://issues.apache.org/jira/browse/CONNECTORS-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459092#comment-17459092
]
Karl Wright commented on CONNECTORS-1685:
-
[~schuch]
Our original Alfresco guy has been inactive for years now. I have no contacts
there either. I would think any current users of Alfresco would be able to
point us at updates, though. Maybe post to the user list?
On another point: we will need to do a release if for no other reason than to
get the log4j 2.15.0 update out. If this is a release blocker we can annotate
the test as "@Ignore" for now. Let's see if we find anything out in, say,
another week.
> Alfresco Webscript Connector: Integration test fails on JDKs having patch
> level > 255
> -
>
> Key: CONNECTORS-1685
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1685
> Project: ManifoldCF
> Issue Type: Bug
> Components: Alfresco webscript connector, Build
>Reporter: Markus Schuch
>Priority: Major
>
> Currently the {{ant test}} job executed by TravisCI in the github mirror of
> manifoldcf fails
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: Log4j dependency
ManifoldCF framework and connectors use log4j 2.x to dump information to the ManifoldCF log file. Please read the following page: https://logging.apache.org/log4j/2.x/security.html Specifically, this part: 'Descripton: Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default.' In other words, unless you are allowing external people access to the crawler UI or to the API, it's impossible to exploit this in ManifoldCF. However, in the interest of assuring people, we are updating this core dependency to the recommended 2.15.0 anyway. The release is scheduled by the end of December. Karl On Tue, Dec 14, 2021 at 4:41 AM ritika jain wrote: > .Hi All, > > How does manifold.cf use log4j. When I checked pom.xml of ES connector , > it is shown as an *exclusion *of maven dependency. > [image: image.png] > > But when checked in Project's downloaded Dependencies, It shows it being > used and downloaded. > > [image: image.png] > How does manifold use log 4j and how can we change the version of it. > > Thanks > Ritika >
[jira] [Commented] (CONNECTORS-1685) Alfresco Webscript Connector: Integration test fails on JDKs having patch level > 255
[
https://issues.apache.org/jira/browse/CONNECTORS-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458207#comment-17458207
]
Karl Wright commented on CONNECTORS-1685:
-
[~schuch], if there is a version of ICU that is available with the fix, all we
can reasonably do is request that the Alfresco maintainers update to that
version and give us an ETA for the fix.
We should not remove the test but we should be prepared to upgrade the
components used for it when a new component is available.
> Alfresco Webscript Connector: Integration test fails on JDKs having patch
> level > 255
> -
>
> Key: CONNECTORS-1685
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1685
> Project: ManifoldCF
> Issue Type: Bug
> Components: Alfresco webscript connector, Build
>Reporter: Markus Schuch
>Priority: Major
>
> Currently the {{ant test}} job executed by TravisCI in the github mirror of
> manifoldcf fails
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1626) CSWS Authority does not return all user permissions
[ https://issues.apache.org/jira/browse/CONNECTORS-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457491#comment-17457491 ] Karl Wright commented on CONNECTORS-1626: - [~schuch] Please go ahead! > CSWS Authority does not return all user permissions > --- > > Key: CONNECTORS-1626 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1626 > Project: ManifoldCF > Issue Type: Bug > Components: LiveLink connector >Reporter: Markus Schuch >Assignee: Markus Schuch >Priority: Major > Attachments: CONNECTORS-1626.patch > > > Currently the CSWS Authority does return tokens for groups, a user is a > directly a member of. > The CSWS Authority does not return > - tokens for transitive group memberships > - tokens for project group memberships -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Zookeeper locks issue
You haven't told me anything about the connectors involved in this job. For every connector, there are connection pools of limited sizes, partitioned by configuration parameters. Throttling is done also on these pools, and those require locks. So if a connector has code for establishing a connection that can hang, it can cause carnage. Connection establishment in the connector contract involves several methods, which I am sure you know. If any one of those can block, that is bad. Instead they should just throw an exception. Karl On Fri, Dec 10, 2021 at 8:15 AM wrote: > Hi, > > Not sure to understand what you mean by establishing a connection, what > kind of connection ? If you can explain to me when the agent needs to set > and release locks, I'll be able to better investigate. > > Julien > > -Message d'origine- > De : Karl Wright > Envoyé : jeudi 9 décembre 2021 15:37 > À : dev > Objet : Re: Zookeeper locks issue > > The fact that you only see this on one job is pretty clearly evidence that > we are seeing a hang of some kind due something a specific connector or > connection is doing. > > I'm going to have to guess wildly here to focus us on a productive path. > What I want to rule out is a case where the connector hangs while > establishing a connection. If this can happen then I could well believe > there would be a train wreck. Is this something you can confirm or > disprove? > > Karl > > > On Thu, Dec 9, 2021 at 9:07 AM Julien Massiera < > julien.massi...@francelabs.com> wrote: > > > Actually, I have several jobs, but only one job is running at a time, > > and currently the error always happens on the same one. The problem is > > that I can't access the environment in debug mode, I also can't > > activate debug log because I am limited in log size, so the only thing > > I can do, is to add specific logs in specific places in the code to > > try to understand what is happening. Where would you suggest me to add > > log entries to optimise our chances to spot the issue ? > > > > Julien > > > > Le 09/12/2021 à 13:27, Karl Wright a écrit : > > > The large number of connections can happen but usually that means > > something > > > is stuck somewhere and there is a "train wreck" of other locks > > > getting backed up. > > > > > > If this is completely repeatable then I think we have an opportunity > > > to figure out why this is happening. One thing that is clear is > > > that this doesn't happen in other situations or in our integration > > > tests, so that makes it necessary to ask what you may be doing > differently here? > > > > > > I was operating on the assumption that the session just expires from > > > lack of use, but in this case it may well be the other way around: > > > something hangs elsewhere and a lock is held open for a very long > > > time, long enough to exceed the timeout. If you have dozens of jobs > > > running it might be a challenge to do this but if you can winnow it > > > down to a small number the logs may give us a good picture of what is > happening. > > > > > > Karl > > > > > > > > > > > > > > > On Wed, Dec 8, 2021 at 3:55 PM Julien Massiera < > > > julien.massi...@francelabs.com> wrote: > > > > > >> Hi, > > >> > > >> after having increased the session lifetime by 3, the lock error > > >> still happens and the MCF agent hangs, so all my jobs also hang. > > >> > > >> Also, as I said in the other thread today, I notice a very large > > >> amount of simultaneous connections from the agent to Zookeeper > > >> (more than 1000) and I cannot tell if it is normal or not. > > >> > > >> Can we ignore that particular error and avoid to block an entire > > >> MCF > > node ? > > >> > > >> Julien > > >> > > >> Le 07/12/2021 à 22:15, Julien Massiera a écrit : > > >>> Ok that makes sense. But still, I don't understand how the "Can't > > >>> release lock we don't hold" exception can happen, knowing for sure > > >>> that neither the Zookeeper process or the MCF agent process have > > >>> been down and/or restarted. Not sure that increasing the session > > >>> lifetime would solve that particular issue, and since I have no > > >>> use case to easily reproduct it, it is very complicated to debug. > > >>> > > >>> Julien > > >>> > > >>
Re: Zookeeper locks issue
The fact that you only see this on one job is pretty clearly evidence that we are seeing a hang of some kind due something a specific connector or connection is doing. I'm going to have to guess wildly here to focus us on a productive path. What I want to rule out is a case where the connector hangs while establishing a connection. If this can happen then I could well believe there would be a train wreck. Is this something you can confirm or disprove? Karl On Thu, Dec 9, 2021 at 9:07 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Actually, I have several jobs, but only one job is running at a time, > and currently the error always happens on the same one. The problem is > that I can't access the environment in debug mode, I also can't activate > debug log because I am limited in log size, so the only thing I can do, > is to add specific logs in specific places in the code to try to > understand what is happening. Where would you suggest me to add log > entries to optimise our chances to spot the issue ? > > Julien > > Le 09/12/2021 à 13:27, Karl Wright a écrit : > > The large number of connections can happen but usually that means > something > > is stuck somewhere and there is a "train wreck" of other locks getting > > backed up. > > > > If this is completely repeatable then I think we have an opportunity to > > figure out why this is happening. One thing that is clear is that this > > doesn't happen in other situations or in our integration tests, so that > > makes it necessary to ask what you may be doing differently here? > > > > I was operating on the assumption that the session just expires from lack > > of use, but in this case it may well be the other way around: something > > hangs elsewhere and a lock is held open for a very long time, long enough > > to exceed the timeout. If you have dozens of jobs running it might be a > > challenge to do this but if you can winnow it down to a small number the > > logs may give us a good picture of what is happening. > > > > Karl > > > > > > > > > > On Wed, Dec 8, 2021 at 3:55 PM Julien Massiera < > > julien.massi...@francelabs.com> wrote: > > > >> Hi, > >> > >> after having increased the session lifetime by 3, the lock error still > >> happens and the MCF agent hangs, so all my jobs also hang. > >> > >> Also, as I said in the other thread today, I notice a very large amount > >> of simultaneous connections from the agent to Zookeeper (more than 1000) > >> and I cannot tell if it is normal or not. > >> > >> Can we ignore that particular error and avoid to block an entire MCF > node ? > >> > >> Julien > >> > >> Le 07/12/2021 à 22:15, Julien Massiera a écrit : > >>> Ok that makes sense. But still, I don't understand how the "Can't > >>> release lock we don't hold" exception can happen, knowing for sure > >>> that neither the Zookeeper process or the MCF agent process have been > >>> down and/or restarted. Not sure that increasing the session lifetime > >>> would solve that particular issue, and since I have no use case to > >>> easily reproduct it, it is very complicated to debug. > >>> > >>> Julien > >>> > >>> Le 07/12/2021 à 19:08, Karl Wright a écrit : > >>>> What this code is doing is interpreting exceptions back from > Zookeeper. > >>>> There are some kinds of exceptions it interprets as "session has > >>>> expired", > >>>> so it rebuilds the session. > >>>> > >>>> The code is written in such a way that the locks are presumed to > persist > >>>> beyond the session. In fact, if they do not persist beyond the > session, > >>>> there is a risk that proper locks won't be enforced. > >>>> > >>>> If I recall correctly, we have a number of integration tests that > >>>> exercise > >>>> Zookeeper integration that are meant to allow sessions to expire and > be > >>>> re-established. If what you say is true and information is attached > >>>> solely > >>>> to a session, Zookeeper cannot possibly work as the cross-process lock > >>>> mechanism we use it for. And yet it is used not just by us in this > way, > >>>> but by many other projects as well. > >>>> > >>>> So I think that the diagnosis that nodes in Zookeeper have session > >>>> affinity > >>
Re: Zookeeper locks issue
The large number of connections can happen but usually that means something is stuck somewhere and there is a "train wreck" of other locks getting backed up. If this is completely repeatable then I think we have an opportunity to figure out why this is happening. One thing that is clear is that this doesn't happen in other situations or in our integration tests, so that makes it necessary to ask what you may be doing differently here? I was operating on the assumption that the session just expires from lack of use, but in this case it may well be the other way around: something hangs elsewhere and a lock is held open for a very long time, long enough to exceed the timeout. If you have dozens of jobs running it might be a challenge to do this but if you can winnow it down to a small number the logs may give us a good picture of what is happening. Karl On Wed, Dec 8, 2021 at 3:55 PM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi, > > after having increased the session lifetime by 3, the lock error still > happens and the MCF agent hangs, so all my jobs also hang. > > Also, as I said in the other thread today, I notice a very large amount > of simultaneous connections from the agent to Zookeeper (more than 1000) > and I cannot tell if it is normal or not. > > Can we ignore that particular error and avoid to block an entire MCF node ? > > Julien > > Le 07/12/2021 à 22:15, Julien Massiera a écrit : > > Ok that makes sense. But still, I don't understand how the "Can't > > release lock we don't hold" exception can happen, knowing for sure > > that neither the Zookeeper process or the MCF agent process have been > > down and/or restarted. Not sure that increasing the session lifetime > > would solve that particular issue, and since I have no use case to > > easily reproduct it, it is very complicated to debug. > > > > Julien > > > > Le 07/12/2021 à 19:08, Karl Wright a écrit : > >> What this code is doing is interpreting exceptions back from Zookeeper. > >> There are some kinds of exceptions it interprets as "session has > >> expired", > >> so it rebuilds the session. > >> > >> The code is written in such a way that the locks are presumed to persist > >> beyond the session. In fact, if they do not persist beyond the session, > >> there is a risk that proper locks won't be enforced. > >> > >> If I recall correctly, we have a number of integration tests that > >> exercise > >> Zookeeper integration that are meant to allow sessions to expire and be > >> re-established. If what you say is true and information is attached > >> solely > >> to a session, Zookeeper cannot possibly work as the cross-process lock > >> mechanism we use it for. And yet it is used not just by us in this way, > >> but by many other projects as well. > >> > >> So I think that the diagnosis that nodes in Zookeeper have session > >> affinity > >> is not absolutely correct. It may be the case that only one session > >> *owns* > >> a node, and if that session expires then the node goes away. In that > >> case > >> I think the right approach is the modify the zookeeper parameters to > >> increase the session lifetime; I don't see any other way to prevent bad > >> things from happening. Presumably, if a session is created within a > >> process, and the process dies, the session does too. > >> > >> Kar > >> > >> > >> On Tue, Dec 7, 2021 at 11:54 AM Julien Massiera < > >> julien.massi...@francelabs.com> wrote: > >> > >>> Karl, > >>> > >>> I tried to understand the Zookeeper lock logic in the code, and the > >>> only > >>> thing I don't understand is the 'handleEphemeralNodeKeeperException' > >>> method that is called in the catch(KeeperException e) of every > >>> obtain/release lock method of the ZookeeperConnection class. > >>> > >>> This method sets the lockNode param to 'null', recreates a session and > >>> recreates nodes but do not resets the lockNode param at the end. So, as > >>> I understood it, if it happens it may result in the lock release error > >>> that I mentioned because this error is triggered when the lockNode > >>> param > >>> is 'null'. > >>> > >>> The method is in the class > >>> org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection. If you can > >>> take a look and tell me what you think about it, it would be great ! > >>>
Re: Zookeeper locks issue
What this code is doing is interpreting exceptions back from Zookeeper. There are some kinds of exceptions it interprets as "session has expired", so it rebuilds the session. The code is written in such a way that the locks are presumed to persist beyond the session. In fact, if they do not persist beyond the session, there is a risk that proper locks won't be enforced. If I recall correctly, we have a number of integration tests that exercise Zookeeper integration that are meant to allow sessions to expire and be re-established. If what you say is true and information is attached solely to a session, Zookeeper cannot possibly work as the cross-process lock mechanism we use it for. And yet it is used not just by us in this way, but by many other projects as well. So I think that the diagnosis that nodes in Zookeeper have session affinity is not absolutely correct. It may be the case that only one session *owns* a node, and if that session expires then the node goes away. In that case I think the right approach is the modify the zookeeper parameters to increase the session lifetime; I don't see any other way to prevent bad things from happening. Presumably, if a session is created within a process, and the process dies, the session does too. Kar On Tue, Dec 7, 2021 at 11:54 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Karl, > > I tried to understand the Zookeeper lock logic in the code, and the only > thing I don't understand is the 'handleEphemeralNodeKeeperException' > method that is called in the catch(KeeperException e) of every > obtain/release lock method of the ZookeeperConnection class. > > This method sets the lockNode param to 'null', recreates a session and > recreates nodes but do not resets the lockNode param at the end. So, as > I understood it, if it happens it may result in the lock release error > that I mentioned because this error is triggered when the lockNode param > is 'null'. > > The method is in the class > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection. If you can > take a look and tell me what you think about it, it would be great ! > > Thanks, > > Julien > > Le 07/12/2021 à 14:40, Julien Massiera a écrit : > > Yes, I will then try the patch and see if it is working > > > > Regards, > > > > Julien > > > > Le 07/12/2021 à 14:28, Karl Wright a écrit : > >> Yes, this is plausible. But I'm not sure what the solution is. If a > >> zookeeper session disappears, according to the documentation everything > >> associated with that session should also disappear. > >> > >> So I guess we could catch this error and just ignore it, assuming > >> that the > >> session must be gone anyway? > >> > >> Karl > >> > >> > >> On Tue, Dec 7, 2021 at 8:21 AM Julien Massiera < > >> julien.massi...@francelabs.com> wrote: > >> > >>> Hi, > >>> > >>> the Zookeeper lock error mentioned in the before last comment of this > >>> issue https://issues.apache.org/jira/browse/CONNECTORS-1447: > >>> > >>> FATAL 2017-08-04 09:28:25,855 (Agents idle cleanup thread) - Error > >>> tossed: > >>> Can't release lock we don't hold > >>> java.lang.IllegalStateException: Can't release lock we don't hold > >>> at > >>> > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.releaseLock(ZooKeeperConnection.java:815) > > >>> > >>> at > >>> > org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.clearLock(ZooKeeperLockObject.java:218) > > >>> > >>> at > >>> > org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.clearGlobalWriteLockNoWait(ZooKeeperLockObject.java:100) > > >>> > >>> at > >>> > org.apache.manifoldcf.core.lockmanager.LockObject.clearGlobalWriteLock(LockObject.java:160) > > >>> > >>> at > >>> > org.apache.manifoldcf.core.lockmanager.LockObject.leaveWriteLock(LockObject.java:141) > > >>> > >>> at > >>> > org.apache.manifoldcf.core.lockmanager.LockGate.leaveWriteLock(LockGate.java:205) > > >>> > >>> at > >>> > org.apache.manifoldcf.core.lockmanager.BaseLockManager.leaveWrite(BaseLockManager.java:1224) > > >>> > >>> at > >>> > org.apache.manifoldcf.core.lockmanager.BaseLockManager.leaveWriteLock(BaseLockManager.java:771) > > >>> > >>> at > >>> > org.apache.manifoldcf.core.connectorpool.ConnectorPool$Pool.pollAll(ConnectorPool.java:670) > >
Re: Zookeeper locks issue
Yes, this is plausible. But I'm not sure what the solution is. If a zookeeper session disappears, according to the documentation everything associated with that session should also disappear. So I guess we could catch this error and just ignore it, assuming that the session must be gone anyway? Karl On Tue, Dec 7, 2021 at 8:21 AM Julien Massiera < julien.massi...@francelabs.com> wrote: > Hi, > > the Zookeeper lock error mentioned in the before last comment of this > issue https://issues.apache.org/jira/browse/CONNECTORS-1447: > > FATAL 2017-08-04 09:28:25,855 (Agents idle cleanup thread) - Error tossed: > Can't release lock we don't hold > java.lang.IllegalStateException: Can't release lock we don't hold > at > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.releaseLock(ZooKeeperConnection.java:815) > at > org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.clearLock(ZooKeeperLockObject.java:218) > at > org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.clearGlobalWriteLockNoWait(ZooKeeperLockObject.java:100) > at > org.apache.manifoldcf.core.lockmanager.LockObject.clearGlobalWriteLock(LockObject.java:160) > at > org.apache.manifoldcf.core.lockmanager.LockObject.leaveWriteLock(LockObject.java:141) > at > org.apache.manifoldcf.core.lockmanager.LockGate.leaveWriteLock(LockGate.java:205) > at > org.apache.manifoldcf.core.lockmanager.BaseLockManager.leaveWrite(BaseLockManager.java:1224) > at > org.apache.manifoldcf.core.lockmanager.BaseLockManager.leaveWriteLock(BaseLockManager.java:771) > at > org.apache.manifoldcf.core.connectorpool.ConnectorPool$Pool.pollAll(ConnectorPool.java:670) > at > org.apache.manifoldcf.core.connectorpool.ConnectorPool.pollAllConnectors(ConnectorPool.java:338) > at > org.apache.manifoldcf.agents.transformationconnectorpool.TransformationConnectorPool.pollAllConnectors(TransformationConnectorPool.java:121) > at > org.apache.manifoldcf.agents.system.IdleCleanupThread.run(IdleCleanupThread.java:91) > > is still happening in 2021 with the 2.20 version of MCF. > > Karl, you hypothesized that it could be related to Zookeeper being > restarted while the MCF agent is still running, but after some > investigations, my theory is that it is related to re-established > sessions. Locks are not associated to a process but to a session, and it > could happen that when a session is closed accidentally (interrupted by > exceptions etc), it does not correctly release the locks it sets. When a > new session is created by Zookeeper for the same client, the locks > cannot be released because they belong to an old session and the > exception is thrown ! > > Is it something plausible for you ? I have no knowledge on Zookeeper but > if it is something plausible, then it is worth investigating into the > code to see if everything is correctly done to be sure that all locks > are released when a session is closed/interrupted by a problem. > > Julien > > -- > L'absence de virus dans ce courrier électronique a été vérifiée par le > logiciel antivirus Avast. > https://www.avast.com/antivirus >
[jira] [Commented] (CONNECTORS-1681) TikaServiceRmeta: recordActivity can cause Database exception
[
https://issues.apache.org/jira/browse/CONNECTORS-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448749#comment-17448749
]
Karl Wright commented on CONNECTORS-1681:
-
[~julienFL], the database record just needs to not include any non-UTF8
strings. You do not need to limit it to just ASCII. If you read the
description, you will note that the error message says as much: it says you
don't have a valid UTF-8 sequence, and since the input is a Java string, it
must contain codepoints that cannot be represented as UTF-8.
> TikaServiceRmeta: recordActivity can cause Database exception
> -
>
> Key: CONNECTORS-1681
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1681
> Project: ManifoldCF
> Issue Type: Bug
> Components: Tika service connector
>Affects Versions: ManifoldCF 2.20
>Reporter: Julien Massiera
>Assignee: Julien Massiera
>Priority: Major
> Fix For: ManifoldCF 2.21
>
>
> Some files containing non ASCII characters can cause Tika to trigger an
> exception describing the parsing problem.
> As the TikaServiceRmeta connector creates an activity record for any Tika
> exception containing its description (and so that contains the non ASCII char
> in those cases), it causes an SQL exception when MCF tries to insert the
> activity record in Postgres:
> {code:java}
> ERROR 2021-11-24T13:37:00,121 (Worker thread '41') -
> MCF|MCF-agent|apache.manifoldcf.crawlerthreads|Worker thread aborting and
> restarting due to database connection reset: Database exception: SQLException
> doing query (22021): ERROR: invalid byte sequence for encoding "UTF8": 0x00
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (22021): ERROR: invalid byte sequence for
> encoding "UTF8": 0x00 {code}
> So to avoid this, we need to remove any non ASCII chars from the exception
> description before recording the activity
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: Gradual deprecation and removal of NTLM and SPNEGO support
This is not a security issue. The implementation of NTLM is just as secure as the Microsoft implementation. That's not terribly secure but that's inherent in their design. Karl On Sat, Nov 20, 2021 at 7:02 PM larry mccay wrote: > This is a concerning statement and I need some additional information to > determine what sort of risk is inherent in the current implementation. > Perhaps we should take those details off list if this is a security issue. > > I'll need to determine whether there are any workarounds or usage patterns > that we can use to ensure that we are safe. > Currently we do not preemptively send tokens and we use our own delegation > tokens after the first authentication. > > > On Sat, Nov 20, 2021 at 2:58 PM Michael Osipov > wrote: > > > Am 2021-11-20 um 20:46 schrieb larry mccay: > > > Hi - > > > > > > I am the Apache Knox PMC chair and a committer on Hadoop and other > > > ecosystem projects. > > > > > > FYI, Apache Knox is indeed dependent on SPNEGO in httpclient. > > > Knox is a Hadoop ecosystem gateway and as part of the trusted proxy or > > > proxyuser pattern within Hadoop it requires all proxies that dispatch > > > requests on behalf of other users to authenticate via Kerberos/SPNEGO. > > > Knox is not the only proxyuser in the ecosystem and likely not the only > > one > > > that is leveraging HttpClient this way. > > > It is used within a huge number of Hadoop deployments both on-prem and > in > > > the cloud and SPNEGO is critical to the security of these deployments. > > > > If this would be critical for security then you would not rely on it. It > > does not complete the security context for you. As sad as it sounds. > > > > I am currently in a project at work where I try to get rid of the Ping > > Identity/OAuth/OIDC hype and move to to SPNEGO/Kerberos and HttpClient > > is used, in Spring and JMeter. Maybe this well an opportunity to make it > > right, but this will only land in 5.2.x, maybe 5.1.x if the API allows > it. > > > > M > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > >
Re: Gradual deprecation and removal of NTLM and SPNEGO support
These protocols are, unfortunately, still used. However, the projects I know that use them have not yet moved to 5.x of httpcomponents. Other projects I know of that used to use httpcomponents have since upgraded to different http libraries that supported http 2.0 early on. The hint that all it takes is a shove from below to convince other projects to drop NTLM support is, perhaps, not accurate. Projects that maintain NTLM support do so because they are tied to legacy systems that use it. Later improvements, e.g. Kerberos, have also only lightly been supported by HttpComponents, and only with external configuration, which really limits its utility. ManifoldCF, which does much integration with windows systems, supports Kerberos but only in the most hacky way, because there wasn't anything more seamless available. I would therefore counter-propose that Kerberos become a first-class replacement to NTLM before NTLM is discontinued. By first-class, I mean that it is possible to programmatically set up a kerberos connection without an external config file. Maybe this is now possible; if so please correct me. I would love to be able to contribute to this effort, but I fear my day job's responsibilities are so vast and growing that this will be impossible. At best I can maintain the projects I have; new development is out of the question at the moment. Karl On Sat, Nov 20, 2021 at 11:42 AM Oleg Kalnichevski wrote: > Folks > > Presently NTLM & SPNEGO are stated as supported authentication schemes, > by the project which is, quite frankly, not the case. There are partial > implementations in various state of decay contributed some while ago by > contributors long gone with no one on the project both capable and > willing to maintain that code and deal with user reported issues. > > I think we should drop the pretense and start dealing with the problem. > If we cannot adequately support those features we should consider > deprecating and eventually removing them entirely. > > As the first step I would like to propose NTLM & SPNEGO be made an opt- > in feature as of version 5.3. Users would have to explicitly enable > NTLM & SPNEGO support to make HttpClient engage in an NTLM or SPNEGO > handshake. > > This may encourage people vested in NTLM and SPNEGO come forth and help > support those features. > > Also, the deprecation or removal of NTLM would unable us to drop > connection state tracking support and greatly simplify the connection > management APIs. > > Oleg > > > - > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > For additional commands, e-mail: dev-h...@hc.apache.org > >
[jira] [Resolved] (CONNECTORS-1680) WebConnector: Support the Document Base URL element
[
https://issues.apache.org/jira/browse/CONNECTORS-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1680.
-
Resolution: Fixed
r1895187 .
Didn't add anything to the tests for this. [~schuch], please review the commit
and try it out to see if it works as expected.
> WebConnector: Support the Document Base URL element
> ---
>
> Key: CONNECTORS-1680
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1680
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
>Reporter: Markus Schuch
> Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.21
>
>
> HTML allows to specifiy the base URL to use for all relative URLs in a
> document:
> {code:java}
>
>
>
> https://example.org/"/>
> ...
>
>...
> {code}
> [https://developer.mozilla.org/de/docs/Web/HTML/Element/base]
> The Web Connector should respect this element when handling relative links.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (CONNECTORS-1680) WebConnector: Support the Document Base URL element
[
https://issues.apache.org/jira/browse/CONNECTORS-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright updated CONNECTORS-1680:
Fix Version/s: ManifoldCF 2.21
> WebConnector: Support the Document Base URL element
> ---
>
> Key: CONNECTORS-1680
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1680
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
>Reporter: Markus Schuch
> Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.21
>
>
> HTML allows to specifiy the base URL to use for all relative URLs in a
> document:
> {code:java}
>
>
>
> https://example.org/"/>
> ...
>
>...
> {code}
> [https://developer.mozilla.org/de/docs/Web/HTML/Element/base]
> The Web Connector should respect this element when handling relative links.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (CONNECTORS-1680) WebConnector: Support the Document Base URL element
[
https://issues.apache.org/jira/browse/CONNECTORS-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1680:
---
Assignee: Karl Wright
> WebConnector: Support the Document Base URL element
> ---
>
> Key: CONNECTORS-1680
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1680
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
>Reporter: Markus Schuch
> Assignee: Karl Wright
>Priority: Major
>
> HTML allows to specifiy the base URL to use for all relative URLs in a
> document:
> {code:java}
>
>
>
> https://example.org/"/>
> ...
>
>...
> {code}
> [https://developer.mozilla.org/de/docs/Web/HTML/Element/base]
> The Web Connector should respect this element when handling relative links.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1679) HTML Extractor: output has escaped entities
[ https://issues.apache.org/jira/browse/CONNECTORS-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17446210#comment-17446210 ] Karl Wright commented on CONNECTORS-1679: - Patch looks good. Would you like to commit it? > HTML Extractor: output has escaped entities > --- > > Key: CONNECTORS-1679 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1679 > Project: ManifoldCF > Issue Type: Bug > Components: HTML extractor >Affects Versions: ManifoldCF 2.20 >Reporter: Julien Massiera >Priority: Major > Fix For: ManifoldCF 2.21 > > Attachments: patch-CONNECTORS-1679.txt > > > The output of the HTML extractor is generated with escaped entities (eg '&' > becomes '& amp ;'), which is not the wanted behavior as we want this > connector to extract text from HTML as it is -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (CONNECTORS-1679) HTML Extractor: output has escaped entities
[ https://issues.apache.org/jira/browse/CONNECTORS-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1679: --- Assignee: Julien Massiera > HTML Extractor: output has escaped entities > --- > > Key: CONNECTORS-1679 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1679 > Project: ManifoldCF > Issue Type: Bug > Components: HTML extractor >Affects Versions: ManifoldCF 2.20 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Major > Fix For: ManifoldCF 2.21 > > Attachments: patch-CONNECTORS-1679.txt > > > The output of the HTML extractor is generated with escaped entities (eg '&' > becomes '& amp ;'), which is not the wanted behavior as we want this > connector to extract text from HTML as it is -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Manifoldcf background process
The degree of parallelism can be controlled in two ways. The first way is to set the number of worker threads to something reasonable. Usually, this is no more than about 2x the number of processors you have. The second way is to control the number of connections in your jcifs connector to keep it at something reasonable, e.g. 4 (because windows SMB is really not good at handling more than that anyway). These two controls are independent of each other. From your description, it sounds like the parameter you want to set is not the number of worker threads but rather the number of connections. But setting both properly certainly will help. The reason that having a high worker thread count is bad is because you use up some amount of memory for each active thread, and that means if you give too big a value you need to give ManifoldCF way too much memory, and you won't be able to compute it in advance either. Karl On Thu, Nov 18, 2021 at 2:49 AM ritika jain wrote: > Hi All, > > I would like to understand the background process of Manifoldcf windows > shares jobs , and how it processes the path mentioned in the jobs > configuration. > > I am creating a dynamic job via API using PHP which will pick up approx > 70k of documents and a dynamic job with 70k of different paths mentioned > in the job and mention folder-subfolders path otherwise and file name in > filespec. > > My question is, how does manifold work in the background to access all > different folders at a time. Because mostly all files correspond to > different folders. Does manifold loads while fetching all folder > permissions and accessing folder/subfolders files. How does it fetch > permission for one folder say for path 1 and simultaneously fetch different > folder permission/access for say path2. > Does it load the manifold. Because when this job is running then > manifoldcf seems to be under heavy load and it gets really really slow and > has to restart the docker container every 15-20 min. > > How can a job be run efficiently? > > Thanks > Ritika > >
Re: Manifold Job process isssue
; jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) > at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) > at > jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) > at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551) > at jcifs.smb.SmbFile.exists(SmbFile.java:845) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2220) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:610) > at > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) > [Worker thread '10'] WARN jcifs.smb.SmbTransportImpl - Disconnecting > transport while still in use Transport41[homestore.directory.intra/ > 136.231.158.104:445,state=5,signingEnforced=false,usage=5]: > [SmbSession[credentials=svc_EScrawl,targetHost=homestore.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]] > [Worker thread '10'] WARN jcifs.smb.SmbSessionImpl - Logging off session > while still in use > SmbSession[credentials=svc_EScrawl,targetHost=homestore.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINHOMES,service=?,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]] > [Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed > jcifs.util.transport.RequestTimeoutException: Transport44 timedout waiting > for response to > command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=72 > at > jcifs.util.transport.Transport.waitForResponses(Transport.java:365) > at jcifs.util.transport.Transport.sendrecv(Transport.java:232) > at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1 > > > On Tue, Nov 9, 2021 at 6:19 PM Karl Wright wrote: > >> One hour is quite a lot and will wreak havoc on the document queue. >> Karl >> >> >> On Tue, Nov 9, 2021 at 7:08 AM ritika jain >> wrote: >> >>> I have checked, there is only one hour time difference between docker >>> container and docker host >>> >>> On Tue, Nov 9, 2021 at 4:41 PM Karl Wright wrote: >>> >>>> If your docker image's clock is out of sync badly with the real world, >>>> then System.currentTimeMillis() may give bogus values, and ManifoldCF uses >>>> that to manage throttling etc. I don't know if that is the correct >>>> explanation but it's the only thing I can think of. >>>> >>>> Karl >>>> >>>> >>>> On Tue, Nov 9, 2021 at 4:56 AM ritika jain >>>> wrote: >>>> >>>>> >>>>> Hi All, >>>>> >>>>> I am using window shares connector , manifoldcf 2.14 and ES as output. >>>>> I have configured a job to process 60k of documents, Also these documents >>>>> are new and do not have corresponding values in DB and ES index. >>>>> >>>>> So ideally it should process/Index the documents as soon as the job >>>>> starts. >>>>> But Manifoldcf does not process anything for many hours of job start >>>>> up.I have tried restarting the docker container as well. But it didn't >>>>> help >>>>> much. Also logs only correspond to Long running queries. >>>>> >>>>> Why does the manifold behave like that? >>>>> >>>>> Thanks >>>>> Ritika >>>>> >>>>
Re: Manifold Job process isssue
One hour is quite a lot and will wreak havoc on the document queue. Karl On Tue, Nov 9, 2021 at 7:08 AM ritika jain wrote: > I have checked, there is only one hour time difference between docker > container and docker host > > On Tue, Nov 9, 2021 at 4:41 PM Karl Wright wrote: > >> If your docker image's clock is out of sync badly with the real world, >> then System.currentTimeMillis() may give bogus values, and ManifoldCF uses >> that to manage throttling etc. I don't know if that is the correct >> explanation but it's the only thing I can think of. >> >> Karl >> >> >> On Tue, Nov 9, 2021 at 4:56 AM ritika jain >> wrote: >> >>> >>> Hi All, >>> >>> I am using window shares connector , manifoldcf 2.14 and ES as output. I >>> have configured a job to process 60k of documents, Also these documents are >>> new and do not have corresponding values in DB and ES index. >>> >>> So ideally it should process/Index the documents as soon as the job >>> starts. >>> But Manifoldcf does not process anything for many hours of job start >>> up.I have tried restarting the docker container as well. But it didn't help >>> much. Also logs only correspond to Long running queries. >>> >>> Why does the manifold behave like that? >>> >>> Thanks >>> Ritika >>> >>
